{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "eb335a6d",
   "metadata": {},
   "source": [
    "# Linear regression\n",
    "\n",
    "The linear regression is a training procedure based on a linear model. The model makes a prediction by simply computing a weighted sum of the input features, plus a constant term called the bias term (also called the intercept term, $\\theta_0$):\n",
    "\n",
    "$$ \\hat{y}=\\theta_0 x_0 + \\theta_1 x_1 + \\theta_2 x_2 + \\cdots + \\theta_n x_n$$\n",
    "\n",
    "\n",
    "This can be written more easy by the using vector notation form:\n",
    "\n",
    "$$\\hat{y}= h_\\theta(x) = \\theta^T x$$\n",
    "\n",
    "---\n",
    "**Now that we have our model, how do we train it?**\n",
    "\n",
    "*Please, consider that training the model means adjusting the parameters to reduce the error or minimizing the cost function.* \n",
    "\n",
    "The most common performance measure of a regression model is the **Mean Square Error (MSE)**. Therefore, to train a Linear Regression model, you need to find the value of $\\theta$ that minimizes the MSE:\n",
    "\n",
    "\n",
    "$$ MSE(X,h_\\theta) = \\frac{1}{m} \\sum_{i=1}^{m} \\left( \\theta^T x^{(i)}-y^{(i)} \\right)^2$$\n",
    "\n",
    "\n",
    "$$ MSE(X,h_\\theta) = \\frac{1}{m} \\sum_{i=1}^{m} \\left(\\hat{y}^{(i)}-y^{(i)} \\right)^2$$\n",
    "\n",
    "$$ MSE(X,h_\\theta) = \\frac{1}{m}  \\left(\\hat{y}-y \\right)^2$$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "12cf820d",
   "metadata": {},
   "source": [
    "# The normal equation\n",
    "\n",
    "To find the value of $\\theta$ that minimizes the cost function, there is a closed-form or direct solution that gives the result. This is called the **Normal Equation**; and can be found it by deriving the *MSE* equation as a function of $\\theta$ and making it equals to zero:\n",
    "\n",
    "\\begin{eqnarray*}\n",
    "\t\\frac{\\partial J(\\theta)}{\\partial \\theta} = 0\\\\\n",
    "\t\\frac{\\partial J(\\theta)}{\\partial \\theta}=\\frac{1}{m}\\left(\\theta x-y \\right)^2=\\frac{1}{m}\\left(\\theta x-y \\right)^T\\left(\\theta x-y \\right)\\\\\n",
    "\t\\frac{1}{m}\\left[(\\theta x)^T-y^T \\right] \\left[\\theta x-y \\right]\n",
    "\\end{eqnarray*}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9139b1c9",
   "metadata": {},
   "source": [
    "just considers that:\n",
    "\\begin{align*}\n",
    "\t(A^T)^T = A\\\\\n",
    "\t(A+B)^T = A^T + B^T\\\\\n",
    "\t(kA)^T = kA^T\\\\\n",
    "\t(AB)^T =A^TB^T\n",
    "\\end{align*}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5b069092",
   "metadata": {},
   "source": [
    "just considers that $(\\theta x)^Ty=y^T(\\theta x)$\n",
    "\\begin{align*}\n",
    "\t0=\\frac{\\partial\\,}{\\partial \\theta}\\frac{1}{m}\\left[(\\theta x)^T\\theta x - (\\theta x)^Ty-y^T\\theta x+y^Ty \\right]\\\\\n",
    "\\end{align*}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ae45a858",
   "metadata": {},
   "source": [
    "$$0=\\frac{\\partial\\,}{\\partial \\theta}\\frac{1}{m}\\left[(\\theta x)^T\\theta x - 2(\\theta x)^Ty+y^Ty \\right]$$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ec27d3cd",
   "metadata": {},
   "source": [
    "$$0=\\frac{1}{m}\\left[2\\theta x^T x - 2(x)^Ty+0 \\right]$$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7c8c8cfe",
   "metadata": {},
   "source": [
    "$$0=\\frac{2}{m}\\left[\\theta x^T x - (x)^Ty+0 \\right]$$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c06b74fd",
   "metadata": {},
   "source": [
    "$$\\hat{\\theta} = (X^T X)^{-1} X^{T} y $$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1315356e",
   "metadata": {},
   "source": [
    "## A basic example\n",
    "\n",
    "First try to implement the linear regressor by only using two instances or points, $(x^0, y^0)$ and $(x^1,y^1)$, thus:\n",
    "\n",
    "$$\\hat{y}^{(i)}=\\theta_0 x_0+ \\theta_1 x_1^{(i)}$$\n",
    "\n",
    "but, considers that $x_0=1$, then:\n",
    "\n",
    "$$\\hat{y}^{(i)}=\\theta_0 + \\theta_1 x_1^{(i)}$$\n",
    "\n",
    "then, to write it in vector form\n",
    "\n",
    "$$\n",
    "    \\hat{y}^0=\n",
    "    \\begin{bmatrix}\n",
    "    \\theta_0 & \\theta_1\n",
    "    \\end{bmatrix} \n",
    "    \\begin{bmatrix}\n",
    "    1 \\\\ \n",
    "    x_0\n",
    "    \\end{bmatrix}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "54dda83d",
   "metadata": {},
   "source": [
    "Now, we need a matrix form to manage all data:\n",
    "\n",
    "$$\n",
    "    \\begin{bmatrix}\n",
    "    \\hat{y}^0 \\\\ \n",
    "    \\hat{y}^1\\\\\n",
    "    \\end{bmatrix}\n",
    "    =\n",
    "    \\begin{bmatrix}\n",
    "    \\theta_0 + \\theta_1 x^0_1 \\\\\n",
    "    \\theta_0 + \\theta_1 x^0_1 \\\\\n",
    "    \\end{bmatrix}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d86618e4",
   "metadata": {},
   "source": [
    "$$\n",
    "    \\begin{bmatrix}\n",
    "    \\hat{y}^0 \\\\ \n",
    "    \\hat{y}^1\\\\\n",
    "    \\end{bmatrix}\n",
    "    =\n",
    "    \\begin{bmatrix}\n",
    "    1 &  x^0_1 \\\\\n",
    "    1 &  x^1_1 \\\\\n",
    "    \\end{bmatrix}\n",
    "    \\begin{bmatrix}\n",
    "    \\theta_0 \\\\\n",
    "    \\theta_1 \\\\\n",
    "    \\end{bmatrix}\n",
    "$$\n",
    "\n",
    "Now, let's compute $\\theta$ using the normal equation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "d2670f5b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1.000e+00, 1.000e-03],\n",
       "       [1.000e+00, 2.196e+00]])"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import numpy as np\n",
    "X = np.array([[1,0.001],[1, 2.196]])\n",
    "Xo = np.array([[0.001],[2.196]])\n",
    "y = np.array([4.314825, 10.877373])\n",
    "X"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "d95f739b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([4.31183523, 2.9897713 ])"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "theta =  np.linalg.inv(X.T.dot(X)).dot(X.T).dot(y)\n",
    "theta"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "b241fb39",
   "metadata": {},
   "outputs": [],
   "source": [
    "t0 = theta[0]\n",
    "t1 = theta[1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "152366f0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 9228 characters]\n",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "plt.plot(Xo,y, '*k')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "4f15df79",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 17852 characters]\n",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "Xvec = np.linspace(0,2,2)\n",
    "Xnew = np.c_[np.ones((2, 1)), Xvec]\n",
    "ypre = Xnew.dot(theta)\n",
    "plt.plot(Xo,y, '*k')\n",
    "plt.plot(Xvec, ypre, '-r')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a986cb4d",
   "metadata": {},
   "source": [
    "# My Linear Regressor\n",
    "\n",
    "**Instructions:** Create your own function train (estimate) the model's parameter for a Linear Regressor implementing the *Normal Equation*. Test your model with the next generated data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "5ca2686a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0xffff82286040>]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "[encoded data, 16636 characters]\n",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "np.random.seed(42)\n",
    "Xo = 2 * np.random.rand(100, 1)\n",
    "X = np.c_[np.ones((100, 1)), Xo]  # add x0 = 1 to each instance\n",
    "y = 4 + 3 * Xo + np.random.randn(100, 1)\n",
    "plt.plot(Xo, y, '.k')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "43a9426c",
   "metadata": {},
   "source": [
    "Therefore, the model will become:\n",
    "\n",
    "$$\n",
    "    \\begin{bmatrix}\n",
    "    \\hat{y}^0 \\\\ \n",
    "    \\hat{y}^1\\\\\n",
    "    \\vdots \\\\\n",
    "    \\hat{y}^n\n",
    "    \\end{bmatrix}\n",
    "    =\n",
    "    \\begin{bmatrix}\n",
    "    1 &  x^0_1 \\\\\n",
    "    1 &  x^1_1 \\\\\n",
    "    \\vdots & \\vdots \\\\\n",
    "    1 & x^n_1\n",
    "    \\end{bmatrix}\n",
    "    \\begin{bmatrix}\n",
    "    \\theta_0 \\\\\n",
    "    \\theta_1 \\\\\n",
    "    \\vdots \\\\\n",
    "    \\theta_n\n",
    "    \\end{bmatrix}\n",
    "$$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c07852f4",
   "metadata": {},
   "source": [
    "## Using Sklearn to train the model\n",
    "Python already includes a linear regression function within Scikit-Learn."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "17481af6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(array([4.21509616]), array([[0.        , 2.77011339]]))"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.linear_model import LinearRegression\n",
    "\n",
    "lin_reg = LinearRegression()\n",
    "lin_reg.fit(X, y)\n",
    "lin_reg.intercept_, lin_reg.coef_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "c5e85264",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[4.21509616],\n",
       "       [9.75532293]])"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_new = np.array([[0], [2]])\n",
    "X_new_b = np.c_[np.ones((2, 1)), X_new] #Se agrega x0=1 para cada instancia\n",
    "y_Pred=lin_reg.predict(X_new_b)\n",
    "y_Pred"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "20da996b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0xffff72c2bfa0>"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "[encoded data, 27888 characters]\n",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(Xo,y, '.k', label='data')\n",
    "plt.plot(X_new, y_Pred, '-r', label='SK-model')\n",
    "plt.legend()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "19ad47f6",
   "metadata": {},
   "source": [
    "# Gradient Descent"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1a8ec08d",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
