{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "e22794c1",
   "metadata": {},
   "source": [
    "# Multi-parametric OLS\n",
    "The multiprarmetric linear regression is a training procedure based on a linear model. The model makes a prediction by simply computing a weighted sum of the input features, plus a constant term called the bias term (also called the intercept term):\n",
    "\n",
    "$$ \\hat{y}=\\theta_0 + \\theta_1 x_1 + \\theta_2 x_2 + \\cdots + \\theta_n x_n$$\n",
    "\n",
    "This new model includes $\\theta_n$ unknown parameter. Thus, the model can be writen more easy by using vector notation form for $m$ instances. \n",
    "Therefore, the model will become in matrix form $Y=X\\times\\theta$:\n",
    "\n",
    "$$ \n",
    "\\begin{bmatrix} \n",
    "    \\hat{y}^0 \\\\ \n",
    "    \\hat{y}^1\\\\ \n",
    "    \\hat{y}^2\\\\ \n",
    "    \\vdots \\\\ \n",
    "    \\hat{y}^m \n",
    "\\end{bmatrix} \n",
    "= \n",
    "\\begin{bmatrix} \n",
    "    1 & x_1^0 & x_2^0 & \\cdots & x_n^0\\\\ \n",
    "    1 & x_1^1 & x_2^1 & \\cdots & x_n^1\\\\\n",
    "    1 & x_1^2 & x_2^2 & \\cdots & x_n^2\\\\\n",
    "    \\vdots & \\vdots & \\vdots &\\cdots& \\vdots\\\\\n",
    "    1 & x_0^m & x_1^m & \\cdots &x_n^m \n",
    "\\end{bmatrix} \n",
    "\\begin{bmatrix} \n",
    "    \\theta_0 \\\\ \n",
    "    \\theta_1 \\\\ \n",
    "    \\theta_2 \\\\ \n",
    "    \\vdots \\\\ \n",
    "    \\theta_n \n",
    "\\end{bmatrix} $$ \n",
    "Resulting:\n",
    "\n",
    "$$\\hat{y}= h_\\theta(x) = x \\theta $$\n",
    "\n",
    "Now that we have our model, how do we train it?\n",
    "\n",
    "Please, consider that training the model means adjusting the parameters to reduce the error or minimizing the cost function. \n",
    "The most common performance measure of a regression model is the Mean Square Error (MSE). Therefore, to train a Linear Regression model, you need to find the value of $\\theta$ that minimizes the MSE:\n",
    "\n",
    "$$J = MSE(X,h_\\theta) = \\frac{1}{m} \\sum_{i=1}^{m} \\left(\\hat{y}^{(i)}-y^{(i)} \\right)^2$$\n",
    "\n",
    "$$J = MSE(X,h_\\theta) = \\frac{1}{m} \\sum_{i=1}^{m} \\left( x^{(i)}\\theta-y^{(i)} \\right)^2$$\n",
    "\n",
    "$$J = MSE(X,h_\\theta) = \\frac{1}{m} \\left( x\\theta-y \\right)^T \\left( x\\theta-y \\right)$$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a9bb4e9b",
   "metadata": {},
   "source": [
    "# The normal equation\n",
    "\n",
    "To find the value of $\\theta$ that minimizes the cost function, there is a closed-form solution that gives the result directly. \n",
    "This is called the Normal Equation; and can be find it by derivating the MSE equation as a function of $\\theta$ and making it equals to zero:\n",
    "\n",
    "$$\\hat{\\theta} = (X^T X)^{-1} X^{T} y $$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "6bbfeb7d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>X1</th>\n",
       "      <th>X2</th>\n",
       "      <th>y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>3.745401</td>\n",
       "      <td>0.314292</td>\n",
       "      <td>9.247570</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>9.507143</td>\n",
       "      <td>6.364104</td>\n",
       "      <td>16.257728</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>7.319939</td>\n",
       "      <td>3.143560</td>\n",
       "      <td>16.258844</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>5.986585</td>\n",
       "      <td>5.085707</td>\n",
       "      <td>6.359638</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1.560186</td>\n",
       "      <td>9.075665</td>\n",
       "      <td>-9.739221</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>95</th>\n",
       "      <td>4.937956</td>\n",
       "      <td>3.492096</td>\n",
       "      <td>6.523018</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>96</th>\n",
       "      <td>5.227328</td>\n",
       "      <td>7.259557</td>\n",
       "      <td>28.761328</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>97</th>\n",
       "      <td>4.275410</td>\n",
       "      <td>8.971103</td>\n",
       "      <td>-4.306011</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>98</th>\n",
       "      <td>0.254191</td>\n",
       "      <td>8.870864</td>\n",
       "      <td>-19.500923</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>99</th>\n",
       "      <td>1.078914</td>\n",
       "      <td>7.798755</td>\n",
       "      <td>-10.525044</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>100 rows × 3 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "          X1        X2          y\n",
       "0   3.745401  0.314292   9.247570\n",
       "1   9.507143  6.364104  16.257728\n",
       "2   7.319939  3.143560  16.258844\n",
       "3   5.986585  5.085707   6.359638\n",
       "4   1.560186  9.075665  -9.739221\n",
       "..       ...       ...        ...\n",
       "95  4.937956  3.492096   6.523018\n",
       "96  5.227328  7.259557  28.761328\n",
       "97  4.275410  8.971103  -4.306011\n",
       "98  0.254191  8.870864 -19.500923\n",
       "99  1.078914  7.798755 -10.525044\n",
       "\n",
       "[100 rows x 3 columns]"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "df =  pd.read_csv('dataset1.csv')\n",
    "x1 = df['X1']\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "567f8438",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 14780 characters]",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "x1 = df['X1']\n",
    "x2 = df['X2']\n",
    "y = df['y']\n",
    "plt.plot(x1,y, '.k')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "695dbdbe",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 68028 characters]",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Create figure and 3D axis\n",
    "fig = plt.figure()\n",
    "ax = fig.add_subplot(111, projection='3d')\n",
    "\n",
    "# Scatter plot\n",
    "ax.scatter(x1, x2, y, c='blue', marker='o')\n",
    "ax.view_init(elev=20, azim=60)  # adjust camera\n",
    "\n",
    "# Axis labels\n",
    "ax.set_xlabel(\"X1\")\n",
    "ax.set_ylabel(\"X2\")\n",
    "ax.set_zlabel(\"Y\")\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "92a0e7d7",
   "metadata": {},
   "source": [
    "# The normal equation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "a5ae94cb",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.85319677,  2.98010241, -1.871685  ])"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import numpy as np\n",
    "from numpy.linalg import inv \n",
    "x0 = np.ones(len(x1)) \n",
    "X = np.c_[x0,x1,x2]\n",
    "thetaH = np.dot(inv(X.T@X), X.T@y)\n",
    "thetaH"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "e31f0319",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 71560 characters]",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "x1New = np.linspace(0,10,10)\n",
    "x2New= np.linspace(0,10,10)\n",
    "x1NewG, x2NewG = np.meshgrid(x1New, x2New)\n",
    "yModel = thetaH[0]+thetaH[1]*x1NewG+thetaH[2]*x2NewG\n",
    "\n",
    "fig = plt.figure()\n",
    "ax = fig.add_subplot(111, projection='3d')\n",
    "ax.scatter(x1NewG, x2NewG, yModel, c='red', marker='.')\n",
    "ax.scatter(x1, x2, y, c='black', marker='.')\n",
    "ax.view_init(elev=30, azim=30)  # adjust camera\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6b375645",
   "metadata": {},
   "source": [
    "# Batch Gradient Descent\n",
    "\n",
    "$$\\theta_{new} = \\theta_{old}-\\eta \\nabla_\\theta$$\n",
    "$$\\nabla_\\theta = \\frac{2}{m}X^T(X\\theta-y)$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "604264ba",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 14780 characters]",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(x1,y, '.k', label=\"data\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "544f3777",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[10],\n",
       "       [ 9]])"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#bgd:\n",
    "X = np.c_[np.ones_like(x1), x1]\n",
    "y = np.array(y, ndmin=2).reshape(-1,1)\n",
    "np.random.seed(67)\n",
    "n = 1000\n",
    "m = len(y)\n",
    "eta = 0.0001\n",
    "thetaH = np.array([[10],[9]])\n",
    "thetaH"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "a512e89d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-=-=-=-=-=-=\n",
      "[[9.99064619]\n",
      " [8.94558151]]\n",
      "-=-=-=-=-=-=\n",
      "[[9.2750177 ]\n",
      " [4.93169627]]\n",
      "-=-=-=-=-=-=\n",
      "[[8.85060396]\n",
      " [2.80897556]]\n",
      "-=-=-=-=-=-=\n",
      "[[8.58094375]\n",
      " [1.68958788]]\n",
      "-=-=-=-=-=-=\n",
      "[[8.39363522]\n",
      " [1.10248799]]\n",
      "-=-=-=-=-=-=\n",
      "[[8.25026369]\n",
      " [0.79775963]]\n",
      "-=-=-=-=-=-=\n",
      "[[8.13044659]\n",
      " [0.64280814]]\n",
      "-=-=-=-=-=-=\n",
      "[[8.02336849]\n",
      " [0.56728464]]\n",
      "-=-=-=-=-=-=\n",
      "[[7.92328982]\n",
      " [0.53386527]]\n",
      "-=-=-=-=-=-=\n",
      "[[7.82716409]\n",
      " [0.52274774]]\n",
      "Final values:\n",
      "[[7.73430322]\n",
      " [0.52338189]]\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 56120 characters]",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "thetaH = np.array([[10],[9]])\n",
    "x1New = np.linspace(0,10,2)\n",
    "plt.scatter(x1, y, color=\"k\", s=15, label=\"data\")\n",
    "#yModel = thetaH[0]+thetaH[1]*x1New\n",
    "#plt.plot(x1New, yModel, '-g', label=\"zero\")\n",
    "for iter in range(n):\n",
    "    nabla = 2/m * X.T@(X@thetaH-y)\n",
    "    thetaH = thetaH-eta*nabla\n",
    "    if iter % 100 == 0:\n",
    "        yModel = thetaH[0]+thetaH[1]*x1New\n",
    "        plt.plot(x1New,yModel, label=f\"iter {iter}\")\n",
    "        print(\"-=-=-=-=-=-=\")\n",
    "        print(thetaH)\n",
    "print(\"Final values:\")\n",
    "print(thetaH) \n",
    "plt.xlabel(\"x1\")\n",
    "plt.ylabel(\"y\")\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b1bbdec0",
   "metadata": {},
   "source": [
    "# History"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "id": "a8a864bc",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Initial values\n",
      "[[ 0.47298583]\n",
      " [-0.68142588]]\n",
      "Final values:\n",
      "[[-3.25986811]\n",
      " [ 2.19754312]]\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 47364 characters]",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "np.random.seed(12)\n",
    "thetaH = np.random.randn(2,1)\n",
    "print(\"Initial values\")\n",
    "print(thetaH)\n",
    "thetas = np.zeros((n,2))\n",
    "J_hist = np.zeros((n,1))\n",
    "thetas[0] =  thetaH.ravel()\n",
    "x1New = np.linspace(0,10,2)\n",
    "plt.scatter(x1, y, color=\"k\", s=15, label=\"data\")\n",
    "for iter in range(n):\n",
    "    nabla = 2/m * X.T@(X@thetaH-y)\n",
    "    thetaH = thetaH-eta*nabla\n",
    "    thetas[iter] = thetaH.ravel()\n",
    "    J = (1/m*(X@thetaH - y).T@(X@thetaH - y))\n",
    "    # print(\"J init\")\n",
    "    # print(J)\n",
    "    J_hist[iter] = J\n",
    "    if iter % 100 == 0:\n",
    "        yModel = thetaH[0]+thetaH[1]*x1New\n",
    "        plt.plot(x1New,yModel, label=f\"iter {iter}\")\n",
    "        # print(\"-=-=-=-=-=-=\")\n",
    "        # print(thetaH)\n",
    "print(\"Final values:\")\n",
    "print(thetaH) \n",
    "plt.xlabel(\"x1\")\n",
    "plt.ylabel(\"y\")\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "id": "ad27ce36",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<function matplotlib.pyplot.show(close=None, block=None)>"
      ]
     },
     "execution_count": 100,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "[encoded data, 16776 characters]",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(J_hist,'.r')\n",
    "plt.show"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "id": "49be25e1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 16748 characters]",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(thetas[:,0], '.b', label=r\"$\\theta_0$\")\n",
    "plt.plot(thetas[:,1], '.r', label=r\"$\\theta_1$\")\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "543bbf87",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 14124 characters]",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(thetas[:,0], thetas[:,1], '.k')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "id": "5a62a2bc",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1000, 1)"
      ]
     },
     "execution_count": 115,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "J_hist.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "id": "502253bf",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 54232 characters]",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig = plt.figure()\n",
    "ax = fig.add_subplot(111, projection='3d')\n",
    "ax.scatter(thetas[:,0], thetas[:,1], J_hist[:,0], c='red', marker='.')\n",
    "ax.view_init(elev=30, azim=30)  # adjust camera\n",
    "plt.show()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv (3.13.5)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.13.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
