204 KiB
Linear regression¶
The linear regression is a training procedure based on a linear model. The model makes a prediction by simply computing a weighted sum of the input features, plus a constant term called the bias term (also called the intercept term):
$$ \hat{y}=\theta_0 x_0 + \theta_1 x_1 + \theta_2 x_2 + \cdots + \theta_n x_n$$
This can be writen more easy by using vector notation form for $m$ values. Therefore, the model will become:
$$ \begin{bmatrix} \hat{y}^0 \\ \hat{y}^1\\ \hat{y}^2\\ \vdots \\ \hat{y}^m \end{bmatrix} = \begin{bmatrix} 1 & x_0^0 & x_1^0 & \cdots x_n^0\\ 1 & x_0^1 & x_1^1 & \cdots x_n^1\\ \vdots & \vdots \\ 1 & x_0^m & x_1^m & \cdots x_n^m \end{bmatrix} \begin{bmatrix} \theta_0 \\ \theta_1 \\ \theta_2 \\ \vdots \\ \theta_n \end{bmatrix} $$ Resulting:
$$\hat{y}= h_\theta(x) = x \theta $$
Now that we have our mode, how do we train it?
Please, consider that training the model means adjusting the parameters to reduce the error or minimizing the cost function. The most common performance measure of a regression model is the Mean Square Error (MSE). Therefore, to train a Linear Regression model, you need to find the value of θ that minimizes the MSE:
$$ MSE(X,h_\theta) = \frac{1}{m} \sum_{i=1}^{m} \left(\hat{y}^{(i)}-y^{(i)} \right)^2$$
$$ MSE(X,h_\theta) = \frac{1}{m} \sum_{i=1}^{m} \left( x^{(i)}\theta-y^{(i)} \right)^2$$
$$ MSE(X,h_\theta) = \frac{1}{m} \left( x\theta-y \right)^T \left( x\theta-y \right)$$
The normal equation¶
To find the value of $\theta$ that minimizes the cost function, there is a closed-form solution that gives the result directly. This is called the Normal Equation; and can be find it by derivating the MSE equation as a function of $\theta$ and making it equals to zero:
$$\hat{\theta} = (X^T X)^{-1} X^{T} y $$
$$ Temp = \theta_0 + \theta_1 * t $$
import pandas as pd
df = pd.read_csv('data.csv')
df
import numpy as np
y = df['0']
n = 300
t = np.linspace(0,n-1,n)
X = np.c_[np.ones(len(t)), t]
X
theta = np.linalg.inv(X.T.dot(X)).dot(X.T).dot(y)
theta
import matplotlib.pyplot as plt
Xnew1 = np.linspace(0,300, 20)
Xnew = np.c_[np.ones(len(Xnew1)), Xnew1]
Xnew
ypre = Xnew.dot(theta)
plt.plot(Xnew1, ypre, '*-r', label='model')
plt.plot(t,y, '.k', label='data')
plt.legend()
plt.show()
Polynomial model¶
$$ Temp = \theta_0 + \theta_1 * t + \theta_2 * t^2$$
X = np.c_[np.ones(len(t)), t, t*t]
theta = np.linalg.inv(X.T.dot(X)).dot(X.T).dot(y)
theta
Xnew1 = np.linspace(0,300,50)
Xnew = np.c_[np.ones(len(Xnew1)), Xnew1, Xnew1*Xnew1]
ypred = Xnew.dot(theta)
plt.plot(Xnew1, ypred, '*-r', label='model')
plt.plot(t,y, '.g', label='data')
plt.legend()
plt.show()
Batch Gradient Descent¶
$$\theta_{new} = \theta_{old}-\eta \nabla_{\theta} $$
$$\nabla_{\theta} = \frac{2}{m} X^T (X \theta -y) $$
y = np.array(df['0']).reshape(300,1)
n = 300
t = np.linspace(0,n-1,n)
X = np.c_[np.ones(len(t)), t]
y
np.random.seed(82)
eta = 0.00001 #lerning rate
n_iteration = 1000000
m = len(y)
theta = np.random.randn(2,1)*10
theta
for iterations in range(n_iteration):
gradient = 2/m * X.T.dot(X.dot(theta)- y)
theta = theta - eta*gradient
theta
#array([25.70275643, 0.07850281])
#array([[25.53711216],[ 0.07933242]]) -> 42
#array([[25.53941259],[ 0.0793209 ]]) -> 82
BGD Visualization¶
def plot_gradient_descent(eta):
m =len(y)
theta = np.random.randn(2,1)
plt.plot(t,y,'.b')
n_iteration = 1000000
Xnew1 = np.linspace(0,n-1,n)
Xnew = np.c_[np.ones(len(Xnew1)), Xnew1]
for iterations in range(n_iteration):
if iterations % 100000 == 0:
#print(iterations)
ypre = Xnew.dot(theta)
style = '-r' if iterations > 0 else 'g--'
plt.plot(Xnew1, ypre, style)
gradient = 2/m * Xnew.T.dot(Xnew.dot(theta)- y)
theta = theta - eta*gradient
plt.xlabel('$x_1$', fontsize=18)
#plt.axis([0,300, 15,50])
plt.title(r'$\eta$ = {}'.format(eta), fontsize=16)
np.random.seed(112)
plot_gradient_descent(eta=0.000001)
theta
plt.plot(t,y,'.b')
Xnew1 = np.linspace(0,n-1,n)
Xnew = np.c_[np.ones(len(Xnew1)), Xnew1]
ypre = Xnew.dot(theta)
plt.plot(Xnew1, ypre, '-r')
plt.figure(figsize=(10,4))
plt.subplot(131)
np.random.seed(112)
plot_gradient_descent(eta=0.000001)
plt.subplot(132)
np.random.seed(112)
plot_gradient_descent(eta=0.001)
plt.subplot(133)
np.random.seed(112)
plot_gradient_descent(eta=0.00000001)