You cannot select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

69 KiB

Raw Blame History

None <html lang="en"> <head> </head>

Ordinary Least Squares¶

Introduction¶

The Ordinary Least Squares (OLS) is an important method in machine learning and statistics for several reasons. The OLS is a straightforward and easy-to-understand method for fitting linear models.

The model minimizes the sum of squared differences between the observed $y$ and predicted values $\hat{y}$, making it intuitive to understand.

Additionally, the OLS is the foundation for linear regression, one of the most widely used machine learning and statistics techniques. Linear regression is valuable for modeling relationships between variables when you suspect a linear relationship exists.

While OLS is valuable in many scenarios, it's essential to acknowledge its limitations, especially when dealing with nonlinear relationships or complex data structures.

More advanced machine learning techniques like decision trees, neural networks, or support vector machines may be more appropriate in such cases. However, OLS remains a foundational method with enduring relevance in machine learning and statistics.

Obtaining the OLS general equation¶

The OLS general model $\hat{y}$ is defined by:

$$ \hat{y} = \theta_0+\theta_1 x_1 $$

Applying the partial derivatives with rescpect $\theta_0$ and equaliting to zero:

$$\frac{\partial SSR(\theta_0, \theta_1)}{\partial \theta_0}=0 $$

here SSR is defined as:

$$ \sum_{i=1}^n (y^i - \hat{y}^i)^2 $$

Resulting in:

$$ \theta_0 = \frac{\sum_{i=1}^n y^i}{n} - \frac{\theta_1 \sum_{i=1}^n x^i}{n}$$

$$ \theta_0 = \bar{y} -\theta_1 \bar{x} $$

In a similar way, the partial derivative of SSR with respect of $\theta_1$ will result in:

$$\theta_1 = \frac{\sum_{i=1}^n x^i(y^i-\bar{y}) }{\sum_{i=1}^n x^i(x^i-\bar{x})}$$

Implementing OLS in Python¶

In [3]:

import numpy as np
x = np.linspace(0,4,20)
theta0 = 3.9654
theta1 = 2.5456
y = theta0+theta1*x
y

Out[3]:

array([ 3.9654    ,  4.50131579,  5.03723158,  5.57314737,  6.10906316,
        6.64497895,  7.18089474,  7.71681053,  8.25272632,  8.78864211,
        9.32455789,  9.86047368, 10.39638947, 10.93230526, 11.46822105,
       12.00413684, 12.54005263, 13.07596842, 13.61188421, 14.1478    ])

In [4]:

import matplotlib.pyplot as plt 
plt.plot(x,y, '.k')
plt.show()

No description has been provided for this image

In [5]:

x = 4*np.random.rand(50, 1)
y = theta0 + theta1*x+0.5*np.random.randn(50, 1)
plt.plot(x,y, '*k')
plt.show()

Implementing with `for`¶

$$\theta_1 = \frac{\sum_{i=1}^n x^i(y^i-\bar{y}) }{\sum_{i=1}^n x^i(x^i-\bar{x})}$$

In [6]:

# for implementation for computing theta1:
xAve = x.mean()
yAve = y.mean()
num = 0
den = 0
for i in range(len(x)):
    num = num + x[i]*(y[i]-yAve)
    den = den + x[i]*(x[i]-xAve)
theta1Hat = num/den
print(theta1Hat)

[2.56763627]

In [7]:

# for implementation for theta0:
# $$ \theta_0 = \bar{y} -\theta_1 \bar{x} $$
theta0Hat = yAve - theta1Hat*xAve
print(theta0Hat)
#real values are
#theta0 = 3.9654
#theta1 = 2.5456

[3.99216691]

In [8]:

total = 0
for i in range(len(x)):
    total = total + x[i]
total/len(x)

Out[8]:

array([2.06512069])

Implementing OLS by numpy methods¶

In [9]:

# For theta1:
# $$\theta_1 = \frac{\sum_{i=1}^n x^i(y^i-\bar{y}) }{\sum_{i=1}^n x^i(x^i-\bar{x})}$$
num2 = np.sum(x*(y-y.mean()))
den2 = np.sum(x*(x-x.mean()))
theta1Hat2 = num2/den2
print(theta1Hat2)

# Efficacy --> time

2.5676362738874774

In [10]:

theta0Hat2 = yAve-theta1Hat2*xAve
theta0Hat2

Out[10]:

np.float64(3.992166911745958)

Comparing Model and Data¶

In [11]:

xNew = np.linspace(0,4,20)
yHat = theta0Hat + theta1Hat*xNew
plt.plot(xNew, yHat, '-*r', label="$\hat{y}$")
plt.plot(x,y,'.k', label="data")
plt.legend()
plt.show()

Functions for data and OLS¶

In [12]:

def DataGen(xn: float,n: int, disp,theta0=3.9654,theta1=2.5456):
    x = xn*np.random.rand(n, 1)
    #theta0 = 3.9654
    #theta1 = 2.5456
    y = theta0+theta1*x+disp*np.random.randn(n,1)
    return x,y

In [13]:

x,y = DataGen(9, 100, 1, 0,1)

In [14]:

plt.plot(x,y,'.k')
plt.show()

In [15]:

def MyOLS(x,y):
    # for implementation for computing theta1:
    xAve = x.mean()
    yAve = y.mean()
    num = 0
    den = 0
    for i in range(len(x)):
        num = num + x[i]*(y[i]-yAve)
        den = den + x[i]*(x[i]-xAve)
    theta1Hat = num/den
    theta0Hat = yAve - theta1Hat*xAve
    return theta0Hat, theta1Hat

In [16]:

the0, the1 = MyOLS(x,y)
the1

Out[16]:

array([0.99109086])

</html>

69 KiB Raw Blame History