# Ordinary Least Squares

## Introduction

The Ordinary Least Squares (OLS) is an important method in machine learning and statistics for several reasons. The OLS is a straightforward and easy-to-understand method for fitting linear models. 

The model minimizes the sum of squared differences between the observed $y$ and predicted values $\hat{y}$, making it intuitive to understand. 

Additionally, the OLS is the foundation for linear regression, one of the most widely used machine learning and statistics techniques. Linear regression is valuable for modeling relationships between variables when you suspect a linear relationship exists.

While OLS is valuable in many scenarios, it's essential to acknowledge its limitations, especially when dealing with nonlinear relationships or complex data structures. 

More advanced machine learning techniques like decision trees, neural networks, or support vector machines may be more appropriate in such cases. However, OLS remains a foundational method with enduring relevance in machine learning and statistics.



## Obtaining the OLS general equation

The OLS general model $\hat{y}$ is defined by: 

$$ \hat{y} = \theta_0+\theta_1 x_1 $$

Applying the partial derivatives with rescpect $\theta_0$ and equaliting to zero:

$$\frac{\partial SSR(\theta_0, \theta_1)}{\partial \theta_0}=0 $$

here SSR is defined as:

$$ \sum_{i=1}^n (y^i - \hat{y}^i)^2 $$

Resulting in:

$$ \theta_0 = \frac{\sum_{i=1}^n y^i}{n} - \frac{\theta_1 \sum_{i=1}^n x^i}{n}$$

or 

$$ \theta_0 = \bar{y} -\theta_1 \bar{x} $$

In a similar way, the partial derivative of SSR with respect of $\theta_1$ will result in: 

$$\theta_1 = \frac{\sum_{i=1}^n x^i(y^i-\bar{y}) }{\sum_{i=1}^n x^i(x^i-\bar{x})}$$

## Implementing OLS in Python


```python
import numpy as np
x = np.linspace(0,4,20)
theta0 = 3.9654
theta1 = 2.5456
y = theta0+theta1*x
y
```




    array([ 3.9654    ,  4.50131579,  5.03723158,  5.57314737,  6.10906316,
            6.64497895,  7.18089474,  7.71681053,  8.25272632,  8.78864211,
            9.32455789,  9.86047368, 10.39638947, 10.93230526, 11.46822105,
           12.00413684, 12.54005263, 13.07596842, 13.61188421, 14.1478    ])




```python
import matplotlib.pyplot as plt 
plt.plot(x,y, '.k')
plt.show()
```


    
![png](main_files/main_6_0.png)
    



```python
x = 4*np.random.rand(50, 1)
y = theta0 + theta1*x+0.5*np.random.randn(50, 1)
plt.plot(x,y, '*k')
plt.show()
```


    
![png](main_files/main_7_0.png)
    


## Implementing with `for` 
$$\theta_1 = \frac{\sum_{i=1}^n x^i(y^i-\bar{y}) }{\sum_{i=1}^n x^i(x^i-\bar{x})}$$


```python
# for implementation for computing theta1:
xAve = x.mean()
yAve = y.mean()
num = 0
den = 0
for i in range(len(x)):
    num = num + x[i]*(y[i]-yAve)
    den = den + x[i]*(x[i]-xAve)
theta1Hat = num/den
print(theta1Hat)
```

    [2.56763627]



```python
# for implementation for theta0:
# $$ \theta_0 = \bar{y} -\theta_1 \bar{x} $$
theta0Hat = yAve - theta1Hat*xAve
print(theta0Hat)
#real values are
#theta0 = 3.9654
#theta1 = 2.5456
```

    [3.99216691]



```python
total = 0
for i in range(len(x)):
    total = total + x[i]
total/len(x)
```




    array([2.06512069])



## Implementing OLS by numpy methods


```python
# For theta1:
# $$\theta_1 = \frac{\sum_{i=1}^n x^i(y^i-\bar{y}) }{\sum_{i=1}^n x^i(x^i-\bar{x})}$$
num2 = np.sum(x*(y-y.mean()))
den2 = np.sum(x*(x-x.mean()))
theta1Hat2 = num2/den2
print(theta1Hat2)

# Efficacy --> time

```

    2.5676362738874774



```python
theta0Hat2 = yAve-theta1Hat2*xAve
theta0Hat2
```




    np.float64(3.992166911745958)



# Comparing Model and Data


```python
xNew = np.linspace(0,4,20)
yHat = theta0Hat + theta1Hat*xNew
plt.plot(xNew, yHat, '-*r', label="$\hat{y}$")
plt.plot(x,y,'.k', label="data")
plt.legend()
plt.show()
```


    
![png](main_files/main_16_0.png)
    


# Functions for data and OLS


```python
def DataGen(xn: float,n: int, disp,theta0=3.9654,theta1=2.5456):
    x = xn*np.random.rand(n, 1)
    #theta0 = 3.9654
    #theta1 = 2.5456
    y = theta0+theta1*x+disp*np.random.randn(n,1)
    return x,y
```


```python
x,y = DataGen(9, 100, 1, 0,1)
```


```python
plt.plot(x,y,'.k')
plt.show()
```


    
![png](main_files/main_20_0.png)
    



```python
def MyOLS(x,y):
    # for implementation for computing theta1:
    xAve = x.mean()
    yAve = y.mean()
    num = 0
    den = 0
    for i in range(len(x)):
        num = num + x[i]*(y[i]-yAve)
        den = den + x[i]*(x[i]-xAve)
    theta1Hat = num/den
    theta0Hat = yAve - theta1Hat*xAve
    return theta0Hat, theta1Hat
```


```python
the0, the1 = MyOLS(x,y)
the1
```




    array([0.99109086])