### Basics of Machine Learning Series

### Introduction

For intuition and implementation of Binary Logistic Regression refer Classifiction and Logistic Regression and Logistic Regression Model.

Multiclass logistic regression is a extension of the binary classification making use of the **one-vs-all** or **one-vs-rest** classification strategy.

### Intuition

Given a classification problem with **n** distinct classes, train n classifiers, where each classifier draws a decision boundary for one class vs all the other classes. Mathematically,

### Implementation

Below is an implementation for **multiclass logistic regression with linear decision boundary**, where number of classes is 3 and one-vs-all strategy is used.

```
import math
import numpy as np
import matplotlib.pyplot as plt
x_orig = [[0,0], [0,1], [1, 0], [1, 1], [2, 2], [2, 3], [3, 2], [3, 3], [0, 4], [1, 4], [0, 5], [1, 5]]
y_orig = [0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2, 2]
x = np.atleast_2d(x_orig)
y = np.atleast_2d(y_orig).T
def h(X, theta):
return 1 / (1 + np.exp(-mul(X, theta)))
def j(X, y, theta):
return (-1/m) * (mul(y.T, np.log(h(X, theta))) + mul((1-y).T, np.log(1-h(X, theta))))
def update(X, y, theta):
return theta - (alpha/m * mul(X.T, (h(X, theta) - y)))
theta_all = []
for _ in range(3):
theta = np.random.randint(1, 100, size=(3, 1))/ 100
mul = np.matmul
alpha = 0.6
m = len(x)
x = np.atleast_2d(x_orig)
y = np.atleast_2d(y_orig).T
idx_0 = np.where(y!=_)
idx_1 = np.where(y==_)
y[idx_0] = 0
y[idx_1] = 1
X = np.hstack((np.ones((len(x), 1)), x))
prev_j = 10000
curr_j = j(X, y, theta)
tolerance = 0.000001
theta_history = [theta]
cost_history = [curr_j]
while(abs(curr_j - prev_j) > tolerance):
theta = update(X, y, theta)
theta_history.append(theta)
prev_j = curr_j
curr_j = j(X, y, theta)
cost_history.append(curr_j[0][0])
theta_all.append(theta)
print("classifier %d stopping with loss: %.5f" % (_, curr_j[0][0]))
def theta_2(theta, x_range):
return [(-theta[0]/theta[2] - theta[1]/theta[2]*i) for i in x_range]
x_range = np.linspace(-1, 4, 100)
x = np.atleast_2d(x_orig)
y = np.atleast_2d(y_orig).T
fig, ax = plt.subplots()
ax.set_xlim(-1, 4)
ax.set_ylim(-1, 6)
plt.scatter(x[np.where(y == 2), 0], x[np.where(y == 2), 1])
plt.scatter(x[np.where(y == 1), 0], x[np.where(y == 1), 1])
plt.scatter(x[np.where(y == 0), 0], x[np.where(y == 0), 1])
for theta in theta_all:
plt.plot(x_range, theta_2(theta, x_range))
plt.title('Multiclass Logistic Regression')
plt.show()
```

Below is the plot of all the decision boundaries found by the logistic regression.

**Value of \(h_\theta^{(i)}(x)\) is the probability of data point belonging to \(i^{th}\) class** as seen in (1). Keeping this is mind one can decide the precedence of the class based on the values of its corresponding prediction on that data point. So, **the predicted class is the one with maximum value of corresponding hypothesis**. It shown in the plot below.

Similar to the above implementation the classificaiton can be extented to many more classes.

## REFERENCES:

Machine Learning: Coursera - Multiclass Classification: One-vs-All