diff --git a/ANN/main.ipynb b/ANN/main.ipynb index a142a63..78db26e 100644 --- a/ANN/main.ipynb +++ b/ANN/main.ipynb @@ -51,7 +51,7 @@ }, { "cell_type": "code", - "execution_count": 41, + "execution_count": 1, "id": "371dacfd", "metadata": {}, "outputs": [ @@ -100,7 +100,7 @@ }, { "cell_type": "code", - "execution_count": 42, + "execution_count": 2, "id": "d18ac1be", "metadata": {}, "outputs": [], @@ -121,7 +121,7 @@ }, { "cell_type": "code", - "execution_count": 43, + "execution_count": 3, "id": "83a68db8", "metadata": {}, "outputs": [ @@ -149,7 +149,7 @@ }, { "cell_type": "code", - "execution_count": 44, + "execution_count": 4, "id": "7285c021", "metadata": {}, "outputs": [], @@ -178,7 +178,7 @@ }, { "cell_type": "code", - "execution_count": 45, + "execution_count": 5, "id": "b5b34c6e", "metadata": {}, "outputs": [], @@ -188,7 +188,7 @@ }, { "cell_type": "code", - "execution_count": 46, + "execution_count": 6, "id": "5c00cf15", "metadata": {}, "outputs": [ @@ -200,7 +200,7 @@ " [ 0.00512708, -0.12022767, -0.80698188]])" ] }, - "execution_count": 46, + "execution_count": 6, "metadata": {}, "output_type": "execute_result" } @@ -211,7 +211,7 @@ }, { "cell_type": "code", - "execution_count": 47, + "execution_count": 7, "id": "4b0f2e37", "metadata": {}, "outputs": [ @@ -223,7 +223,7 @@ " [-0.12214979, 1.01251548, -0.91386915]])" ] }, - "execution_count": 47, + "execution_count": 7, "metadata": {}, "output_type": "execute_result" } @@ -257,7 +257,7 @@ }, { "cell_type": "code", - "execution_count": 48, + "execution_count": 8, "id": "31b8c4dd", "metadata": {}, "outputs": [], @@ -294,7 +294,7 @@ }, { "cell_type": "code", - "execution_count": 49, + "execution_count": 9, "id": "6d6b0b09", "metadata": {}, "outputs": [ @@ -306,7 +306,7 @@ " [0.48247944]])" ] }, - "execution_count": 49, + "execution_count": 9, "metadata": {}, "output_type": "execute_result" } @@ -327,21 +327,45 @@ }, { "cell_type": "markdown", - "id": "2fb1cf2c", + "id": "c8b04eed", "metadata": {}, "source": [ + "#### Mathematical Derivation of the Cost Function and Gradient\n", "\n", - "The Gradient of Error\n", - "$$ \\frac{\\partial E}{\\partial w_{ho}}= -e_o\\cdot \\sigma \\left(w_{ho} O_h\\right) \\left(1-\\sigma\\left (w_{ho} O_h\\right) \\right) O_h $$\n", + "To thoroughly understand the network's learning mechanics, we must derive the gradient of the error with respect to the synaptic weights. This procedure uses the **Chain Rule** from calculus and establishes the mathematical foundation for the Gradient Descent optimization strategy used in our backpropagation algorithm.\n", "\n", - "Thus,\n", + "**1. The Cost Function (SSE)**\n", + "We define the Total Error ($E$) using the Sum of Squared Errors:\n", + "$$E = \\frac{1}{2} \\sum (T - O_o)^2$$\n", + "Where $T$ represents the target label and $O_o$ is the predicted output. We define the output error as $e_o = (T - O_o)$.\n", "\n", - "$$ \\frac{\\partial E}{\\partial w_{ho}}= -e_o\\cdot O_o \\left(1-O_o \\right) O_h $$" + "**2. The Chain Rule Application**\n", + "To update the weight matrix $w_{ho}$ (connecting the hidden layer to the output layer), we need to determine how a change in $w_{ho}$ impacts the total error $E$. We calculate the partial derivative using the Chain Rule:\n", + "$$\\frac{\\partial E}{\\partial w_{ho}} = \\frac{\\partial E}{\\partial O_o} \\cdot \\frac{\\partial O_o}{\\partial X_o} \\cdot \\frac{\\partial X_o}{\\partial w_{ho}}$$\n", + "*(Note: $X_o = w_{ho} \\cdot O_h$ represents the raw signal entering the output node before activation).*\n", + "\n", + "**3. Solving the Partial Derivatives**\n", + "* **Error derivative:** How the total error changes with respect to the final output.\n", + " $$\\frac{\\partial E}{\\partial O_o} = -(T - O_o) = -e_o$$\n", + "* **Activation derivative:** The derivative of the Sigmoid activation function $\\sigma(X_o)$.\n", + " $$\\frac{\\partial O_o}{\\partial X_o} = \\sigma(X_o)(1 - \\sigma(X_o)) = O_o(1 - O_o)$$\n", + "* **Weight derivative:** How the raw input $X_o$ changes with respect to the weight matrix $w_{ho}$. This evaluates directly to the output of the preceding hidden layer $O_h$.\n", + " $$\\frac{\\partial X_o}{\\partial w_{ho}} = O_h$$\n", + "\n", + "**4. Final Gradient Equation**\n", + "Multiplying these individual derivatives yields the final gradient of the error for $w_{ho}$:\n", + "$$\\frac{\\partial E}{\\partial w_{ho}}= -e_o\\cdot \\sigma \\left(w_{ho} O_h\\right) \\left(1-\\sigma\\left (w_{ho} O_h\\right) \\right) O_h$$\n", + "\n", + "Thus, by substituting the activated output $O_o$, we arrive at the simplified expression:\n", + "$$\\frac{\\partial E}{\\partial w_{ho}}= -e_o\\cdot O_o \\left(1-O_o \\right) O_h$$\n", + "\n", + "This precise formulation dictates the weight update rule programmed in our `backpropagation` method, scaled by the learning rate ($\\eta$) to ensure stable convergence:\n", + "$$w_{ho_{new}} = w_{ho} + \\eta \\cdot e_o \\cdot O_o(1 - O_o) \\cdot O_h^T$$" ] }, { "cell_type": "code", - "execution_count": 50, + "execution_count": 32, "id": "ef1098b0", "metadata": {}, "outputs": [], @@ -397,7 +421,7 @@ }, { "cell_type": "code", - "execution_count": 51, + "execution_count": 33, "id": "deacbee4", "metadata": {}, "outputs": [], @@ -417,7 +441,7 @@ }, { "cell_type": "code", - "execution_count": 52, + "execution_count": 34, "id": "bb581e90", "metadata": {}, "outputs": [ @@ -427,7 +451,7 @@ "" ] }, - "execution_count": 52, + "execution_count": 34, "metadata": {}, "output_type": "execute_result" } @@ -441,7 +465,7 @@ }, { "cell_type": "code", - "execution_count": 53, + "execution_count": 35, "id": "ddb0a6fb", "metadata": {}, "outputs": [ @@ -466,7 +490,7 @@ }, { "cell_type": "code", - "execution_count": 54, + "execution_count": 36, "id": "23314858", "metadata": {}, "outputs": [ @@ -476,7 +500,7 @@ "49999" ] }, - "execution_count": 54, + "execution_count": 36, "metadata": {}, "output_type": "execute_result" } @@ -497,7 +521,7 @@ }, { "cell_type": "code", - "execution_count": 55, + "execution_count": 50, "id": "c4dd5448", "metadata": {}, "outputs": [], @@ -512,14 +536,27 @@ }, { "cell_type": "code", - "execution_count": 56, - "id": "ae0ac96d", + "execution_count": 51, + "id": "c4020634", "metadata": {}, - "outputs": [], + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Epoch 1/5 - Average Loss: 0.0972\n", + "Epoch 2/5 - Average Loss: 0.0559\n", + "Epoch 3/5 - Average Loss: 0.0461\n", + "Epoch 4/5 - Average Loss: 0.0403\n", + "Epoch 5/5 - Average Loss: 0.0361\n" + ] + } + ], "source": [ "# Iterative training loop\n", - "epoch = 1\n", - "for e in range (epoch):\n", + "epochs = 5\n", + "for e in range(epochs):\n", + " total_loss = 0\n", " for record in list: \n", " values = record.split(\",\")\n", " \n", @@ -531,10 +568,17 @@ " target = np.zeros(outNodes) + 0.01\n", " target[index] = 0.99\n", " \n", + " # Calculate loss before updating weights\n", + " output = MyANN.feedforward(data)\n", + " # Using SSE formulation: 0.5 * sum((target - output)^2)\n", + " loss = np.sum(0.5 * (target.reshape(-1, 1) - output)**2)\n", + " total_loss += loss\n", + " \n", + " # Train\n", " MyANN.backpropagation(data, target, learningRate)\n", - " pass\n", - " pass\n", - " " + " \n", + " average_loss = total_loss / len(list)\n", + " print(f\"Epoch {e+1}/{epochs} - Average Loss: {average_loss:.4f}\")" ] }, { @@ -548,7 +592,7 @@ }, { "cell_type": "code", - "execution_count": 57, + "execution_count": 52, "id": "a446646f", "metadata": {}, "outputs": [ @@ -558,7 +602,7 @@ "" ] }, - "execution_count": 57, + "execution_count": 52, "metadata": {}, "output_type": "execute_result" } @@ -572,33 +616,74 @@ }, { "cell_type": "code", - "execution_count": 58, + "execution_count": 53, + "id": "bb895183", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Network Accuracy (Score) on Test Set: 95.27%\n" + ] + } + ], + "source": [ + "#Test Set Evaluation (Network Score)\n", + "scorecard = []\n", + "\n", + "for record in list2:\n", + " values = record.split(\",\")\n", + " correct_label = int(values[0])\n", + " \n", + " # Normalize input\n", + " data = np.asarray(values[1:], dtype=float) / 255.0 * 0.99 + 0.01\n", + " \n", + " # Get network prediction\n", + " outputs = MyANN.feedforward(data)\n", + " # The index of the highest value corresponds to the predicted class\n", + " predicted_label = np.argmax(outputs)\n", + " \n", + " # Append 1 if correct, 0 if incorrect\n", + " if predicted_label == correct_label:\n", + " scorecard.append(1)\n", + " else:\n", + " scorecard.append(0)\n", + "\n", + "scorecard_array = np.asarray(scorecard)\n", + "accuracy = scorecard_array.sum() / scorecard_array.size\n", + "print(f\"Network Accuracy (Score) on Test Set: {accuracy * 100:.2f}%\")" + ] + }, + { + "cell_type": "code", + "execution_count": 54, "id": "682673f4", "metadata": {}, "outputs": [ { "data": { "text/plain": [ - "array([[1.12958417e-03],\n", - " [7.04806122e-03],\n", - " [1.33450332e-03],\n", - " [9.98132913e-01],\n", - " [1.21412650e-04],\n", - " [6.66539345e-03],\n", - " [2.96287176e-05],\n", - " [1.83308214e-03],\n", - " [8.11709452e-04],\n", - " [1.75208890e-04]])" + "array([[3.92210113e-05],\n", + " [9.64025823e-01],\n", + " [1.22524901e-03],\n", + " [1.47886768e-02],\n", + " [1.61851002e-03],\n", + " [3.98689305e-03],\n", + " [7.74585782e-05],\n", + " [2.65175424e-03],\n", + " [4.98386616e-03],\n", + " [2.37478194e-03]])" ] }, - "execution_count": 58, + "execution_count": 54, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Inference on sample 500\n", - "values = list2[500].split(\",\")\n", + "values = list2[700].split(\",\")\n", "data = np.asarray(values[1:], dtype=int)/255*0.99+0.01\n", "\n", "# Display probability vector for the 10 classes\n", @@ -607,13 +692,13 @@ }, { "cell_type": "code", - "execution_count": 59, + "execution_count": 55, "id": "8d5fb4bd", "metadata": {}, "outputs": [ { "data": { - "image/png": "iVBORw0KGgoAAAANSUhEUgAAAaEAAAGdCAYAAAC7EMwUAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjExLjAsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvlcelbwAAAAlwSFlzAAAPYQAAD2EBqD+naQAAGqlJREFUeJzt3Q1MVecdx/E/orz4AqJWbQut2lGJDCoTnPNlkxrr+0vnKnVr10Xj7JK5RJzLaFbXtElJ7Ka2rltcN7PMrWWdc0b2ZnwpvjDWULE6FBaVVahdob4AIgoKd3mehBtvFcq5Xvjfe8/3k5zQczh/7vH03PO7zznPeW6Ex+PxCAAACvpovCgAAAYhBABQQwgBANQQQgAANYQQAEANIQQAUEMIAQDUEEIAADV9JQi1t7fLRx99JIMGDZKIiAjtzQEAOGDGQLhy5Yrcd9990qdPn9ALIRNASUlJ2psBALgLNTU1kpiYqBNCJ06ckHPnzklycrKkpKQ4qjUtoI5/QFxcXA9tIQCgJzQ2NtqGRMe5vFdDqLW1VZYuXSqHDx+W8ePHS2lpqTzxxBPyq1/9qtuX1jrWMwFECAFAaOrOOT/gIbR582YpLi6W48eP22bYyZMnJTMzU6ZPny5PP/10oF8OABDCAt47bvv27ZKTk+O9Dpiamipz5syxywEA6LEQunnzplRUVEh6errPcjNv7hF1pqWlxV5DvHUCAIS/gIZQU1OTtLW1SUJCgs/yoUOHSn19fad1+fn5Eh8f753oGQcA7hDQEIqOjrY/m5ubbwunmJiYTuvy8vKkoaHBO5lecQCA8BfQjgmxsbEycuRIqa6u9llu5seMGdNleHUEGADAPQLeMcF0Qti5c6cd9cC4fv26FBYW2uUAANwqwmPGVwigDz74QLKysmTq1Kkyd+5cKSgokLNnz8rRo0ftvaHuMB0TzL0hc2mO54QAILQ4OYcHvCU0atQoKSsrs6MkFBUVybRp0+wDq90NIACAewS8JRQItIQAIHSptoQAAOguQggAoIYQAgCoIYQAAGoIIQCAGkIIAKCGEAIAqCGEAABqCCEAgBpCCACghhACAKghhAAAagghAIAaQggAoIYQAgCoIYQAAGoIIQCAGkIIAKCGEAIAqCGEAABqCCEAgBpCCACghhACAKghhAAAagghAIAaQggAoIYQAgCoIYQAAGoIIQCAGkIIAKCGEAIAqCGEAABqCCEAgBpCCACghhACAKghhAAAagghAIAaQggAoIYQAgCoIYQAAGoIIQCAGkIIAKCGEAIAqCGEAABqCCEAgBpCCACghhACAKghhAAAagghAIAaQggAoIYQAgCoIYQAAGoIIQCAmr56L41g8vHHHzuuaW9v9+u1Bg4c6LimsLDQcc2pU6cc1+Tn5zuu8Xg84o/58+c7rlm2bJnjmhkzZjiuGTFihOMaIChCqLKyUsrLy32WRUVFycKFCwP9UgCAEBfwENqxY4ds2rRJsrOzvcsGDBhACAEAeudyXHJysg0jAAB6PYSuXbsme/bskZiYGElLS5MhQ4b0xMsAAEJcj/SOq6qqkg0bNkhubq4kJibKxo0bu1y/paVFGhsbfSYAQPgLeAhNnz5dqqurZf/+/XL06FHZtm2brF271s531SMpPj7eOyUlJQV6swAAbgihqVOnSkJCgnf+ySeftJfkdu/e3WlNXl6eNDQ0eKeamppAbxYAwK3PCZnWTV1dXae/j46OthMAwF0C3hKqra31mTetmrKyMsnMzAz0SwEAQlzAW0KLFi2SiRMnSkZGhly4cEG2bNkiqampsmrVqkC/FAAgxAW8JXTw4EF7D6i4uNi2gl5++WUpKSnxa6gWAEB4C3hLyNzbWblypZ0AAOhKhMff0Rd7kHlOyHRmMD3l4uLixM0qKioc15iWp1OmG71T/j7PtWTJEsc1f/rTn/x6LYgMHz7c8W545513HNekpKSwu+H4HM5XOQAA1BBCAAA1hBAAQA0hBABQQwgBANQQQgAANYQQAEANIQQAUEMIAQDUEEIAADWEEABADSEEAFDDAKa96NChQ45rZs6c6bjm5s2bEm5iY2Md17S1tTmu6dPH+eeyuXPnij+OHTvmuOa///2v9IbBgwc7rjFf3eKP/v37+1WH4MUApgCAkMDlOACAGkIIAKCGEAIAqCGEAABqCCEAgBpCCACghhACAKghhAAAagghAIAaQggAoIYQAgCo6av30u7jz4Ca4TgY6aRJkxzXFBQUOK5pbW11XFNdXe24JjMzU/xx9epVxzX333+/9Ibly5c7runXr1+PbAvCGy0hAIAaQggAoIYQAgCoIYQAAGoIIQCAGkIIAKCGEAIAqCGEAABqCCEAgBpCCACghhACAKghhAAAagghAICaCI/H45Eg09jYKPHx8dLQ0CBxcXESLvwZETs9Pd1xzQcffOC4pqioyHFNc3Oz+CMrK8txzYABAyTcHDhwwHHNzJkzJViZ96s/Bg4cGPBtQeicw2kJAQDUEEIAADWEEABADSEEAFBDCAEA1BBCAAA1hBAAQA0hBABQQwgBANQQQgAANYQQAEANIQQAUNNX76Xdp29f57v71KlTjmuqqqoc1zz44IOOayIjIyXcXL9+3XHNf/7zH79ea/Xq1X7VAeL2ltCZM2dkx44dUl1d3ek6J06ckMLCQqmsrLyb7QMAhDFHIVRWViazZs2S2bNnyxNPPCGHDh26bZ3W1lZZvHixZGdny+bNm2XixImyYsUKCcJvjAAAhFIIXbx4UdasWSOnT5/udB0TPMXFxXL8+HHZv3+/lJSUyJtvvim/+93vArG9AAC3hpD5Qi3TCoqIiOh0ne3bt0tOTo4kJiba+dTUVJkzZ45dDgBAj/WOM98cWlFRcdu3gZp5c4+oMy0tLfab+G6dAADhL6Ah1NTUJG1tbZKQkOCzfOjQoVJfX99pXX5+vv0q2I4pKSkpkJsFAHBDCEVHR9ufzc3Nt4VTTExMp3V5eXn2u8g7ppqamkBuFgDADc8JxcbGysiRI2/rum3mx4wZ02V4dQQYAMA9Aj5igumEsHPnTmlvb/c+/GeeFzLLAQDwuyVUV1fn82xQaWmpvcw2evRomTBhgl22fv16ycrKkiVLlsjcuXOloKDAjhSQm5vr5KUAAC7gqCVUW1trQ8VMJmTOnz9v/9uEUYdRo0bZh1pTUlKkqKhIpk2bZn9vOicAAOB3SygtLc0O1/NZTO820+MNAICuMIBpGOqqE4ibmCGknBo7dqzjmg8//FCCWVcPl3dm8uTJjmv69evnuAbgqxwAAGoIIQCAGkIIAKCGEAIAqCGEAABqCCEAgBpCCACghhACAKghhAAAagghAIAaQggAoIYQAgCoYQBThK3du3eH3WCk/njooYcc19z6vWFAT6IlBABQQwgBANQQQgAANYQQAEANIQQAUEMIAQDUEEIAADWEEABADSEEAFBDCAEA1BBCAAA1hBAAQA0DmAJhrrq62nHNvn37HNfU19dLb3nkkUcc1yQnJ/fItuDu0BICAKghhAAAagghAIAaQggAoIYQAgCoIYQAAGoIIQCAGkIIAKCGEAIAqCGEAABqCCEAgBpCCACghhACAKhhFG2ErSlTpjiuGTFihOOa2tpaCWatra2Oa2bNmiXB7Ctf+YrjmgMHDvTItuDu0BICAKghhAAAagghAIAaQggAoIYQAgCoIYQAAGoIIQCAGkIIAKCGEAIAqCGEAABqCCEAgBpCCACghgFMEbbuvfdexzWnTp1yXHPlyhXxR35+vuOarVu3+vVa4ebw4cOOa95//33HNePHj3dcg14IoTNnztj/oRMnTpQHHnjA53eVlZVSXl7usywqKkoWLlzoz0sBAMKYoxAqKyuTvLw8OXv2rJ22b98uTz31lM86O3bskE2bNkl2drZ32YABAwghAMDdhdDFixdlzZo19rtG+vTp/HZScnKyDSMAAAIWQjNnzuzWeteuXZM9e/ZITEyMpKWlyZAhQ5y8DADAJXqkd1xVVZVs2LBBcnNzJTExUTZu3Njl+i0tLdLY2OgzAQDCX8BDaPr06VJdXS379++Xo0ePyrZt22Tt2rV2vqteQvHx8d4pKSkp0JsFAHBDCE2dOlUSEhK8808++aS9JLd79+5Oa0xnh4aGBu9UU1MT6M0CALj1OSHTuqmrq+v099HR0XYCALhLwFtCtbW1PvOmVWO6dmdmZgb6pQAAbmoJmdbMoUOHvPOlpaW2B9zo0aNlwoQJdtmiRYvsQ6wZGRly4cIF2bJli6SmpsqqVasCv/UAAPe0hEwrp6CgwE5LliyR8+fP2/82YdTh4MGD9h5QcXGxbQW9/PLLUlJSIgMHDuyJ7QcAuKUlZMLlsx5CNfd2Vq5caScAALoS4fF4PBJkzHNCpjOD6SkXFxenvTlAj/Dnrdfe3u64xvQ+deqXv/xlrw3k2lu6ekykq0dO0LPncL7KAQCghhACAKghhAAAagghAIAaQggAoIYQAgCoIYQAAGoIIQCAGkIIAKCGEAIAqCGEAABqCCEAQHh/syqA20VERDjeLZGRkY5rNmzY4Lhm2LBhvTJQKkBLCACghhACAKghhAAAagghAIAaQggAoIYQAgCoIYQAAGoIIQCAGkIIAKCGEAIAqCGEAABqCCEAgBoGMAXCXHt7u+Oa5uZmCWb33HOP45qUlJQe2RbcHVpCAAA1hBAAQA0hBABQQwgBANQQQgAANYQQAEANIQQAUEMIAQDUEEIAADWEEABADSEEAFBDCAEA1BBCAAA1jKINhLmNGzc6rnnppZckmBUVFTmuGTlyZI9sC+4OLSEAgBpCCACghhACAKghhAAAagghAIAaQggAoIYQAgCoIYQAAGoIIQCAGkIIAKCGEAIAqCGEAABqGMAUYevGjRu9UuOv8vJyxzU//OEPHdccOnRIgtnYsWMd1zz44IM9si0IgRC6evWqHDt2zL5Z09LSZNiwYXdc78SJE3Lu3DlJTk6WlJSUQGwrAMDNl+NefPFFefjhh+2nsRdeeMF+GnnllVd81mltbZXFixdLdna2bN68WSZOnCgrVqwQj8cT6G0HALipJTR8+HCprKyUQYMG2fldu3bJ448/LtOnT5esrCy7zARPcXGxHD9+XBITE+XkyZOSmZlp13n66ad75l8BAAj/ltCzzz7rDSDDtHiioqLs5bkO27dvl5ycHBtARmpqqsyZM8cuBwAgYL3jjhw5Yi+/maAxbt68KRUVFZKenu6znpk394g609LSIo2NjT4TACD8+R1Cly9fluXLl8u8efNkypQpdllTU5O0tbVJQkKCz7pDhw6V+vr6Tv9Wfn6+xMfHe6ekpCR/NwsAEO4hZFoq5hLb4MGD5c033/Quj46Otj+bm5t91jfhFBMT0+nfy8vLk4aGBu9UU1Pjz2YBAMK9i/aVK1dk9uzZ9tLbvn37JC4uzvu72NhYGTlypFRXV/vUmPkxY8Z0+jdNeHUEGADAPRy1hEyLxgSQuQ+0d+9e2xL6NNNC2rlzp7S3t9v569evS2FhoV0OAIDfLaEFCxbYDgabNm2S/fv3e5ePGzfOTsb69ettd+0lS5bI3LlzpaCgQPr27Su5ublOXgoA4AKOQujee++VWbNmyT/+8Q+f5UuXLvWG0KhRo6SsrEx+/vOfS1FRkUybNs0GkemcAACA3yF0ayeErpjebabHGwAAXWEAU1iXLl1yvCfeeOMNv/ZeVVWV45pPP3vWHW+99ZbjmpKSEsc1uDvmMQ+nTCcohAe+ygEAoIYQAgCoIYQAAGoIIQCAGkIIAKCGEAIAqCGEAABqCCEAgBpCCACghhACAKghhAAAagghAIAaBjANQxcuXHBcs3r1asc1b7/9tuMa3J3IyEjHNR1fMOlE//79HdfMmDFD/LFmzRq/6hAeaAkBANQQQgAANYQQAEANIQQAUEMIAQDUEEIAADWEEABADSEEAFBDCAEA1BBCAAA1hBAAQA0hBABQwwCmYcifAUx37twp4SYiIsJxzezZsx3XHDlyRPzxne98x3HNggULHNecPn3acc0zzzzjuAbwBy0hAIAaQggAoIYQAgCoIYQAAGoIIQCAGkIIAKCGEAIAqCGEAABqCCEAgBpCCACghhACAKghhAAAagghAICaCI/H45Eg09jYKPHx8dLQ0CBxcXHam+MKH3/8ca+MzmzcuHHDcU19fb3jmtWrVzuuOX/+vOOaixcvij+GDBnSKyODA8F8DqclBABQQwgBANQQQgAANYQQAEANIQQAUEMIAQDUEEIAADWEEABADSEEAFBDCAEA1BBCAAA1hBAAQA0DmAIA1AYw7ev0j1+9elWOHTtmR0JOS0uTYcOG+fy+srJSysvLfZZFRUXJwoULnb4UACDMOQqhF198UbZu3SqjR4+WyMhIee+99+SFF16QdevWedfZsWOHbNq0SbKzs73LBgwYQAgBAO4uhIYPH25bOoMGDbLzu3btkscff1ymT58uWVlZ3vWSk5NtGAEAELCOCc8++6w3gIzFixfbS23m8tytrl27Jnv27JGDBw/KpUuXnLwEAMBF7qp33JEjR6S1tVVSU1N9lldVVcmGDRskNzdXEhMTZePGjV3+nZaWFnsj69YJABD+/A6hy5cvy/Lly2XevHkyZcoU73Jzaa66ulr2798vR48elW3btsnatWvtfGfy8/NtT4qOKSkpyd/NAgCEexdt01J57LHHpL29Xfbt2/eZXfDS09NtR4VXX32105aQmW79+yaIutO9DwDgoi7aV65ckdmzZ8vNmze7FUCG2Zi6urpOfx8dHW0nAIC7OLoc19TUZAPI3Afau3evDB48+LZ1amtrfeZramqkrKxMMjMz735rAQDuvRxnLqmZZ4PMc0C3BtC4cePsZEyaNEkmTpwoGRkZcuHCBdmyZYuMHDlSDhw4IAMHDgx4Uw4AEFycnMMdhdDXv/512wr6tKVLl9rJMPd2fvvb38q7774r/fv3t4G0bNky+3BrT/wDAAAuCaHeQggBQOhycg5nFG0AgBpCCACghhACAKghhAAAagghAIAaQggAoIYQAgCoIYQAAGoIIQCAGkIIAKCGEAIAqCGEAABqCCEAgBpCCACghhACAKghhAAAagghAIAaQggAoIYQAgCoIYQAAGoIIQCAGkIIAKCGEAIAqCGEAABqCCEAgJq+EoQ8Ho/92djYqL0pAACHOs7dHefykAuhK1eu2J9JSUnamwIAuItzeXx8fJfrRHi6E1W9rL29XT766CMZNGiQRERE+KSrCaaamhqJi4sTt2I/sB84HnhfBPP5wcSKCaD77rtP+vTpE3otIbPRiYmJnf7e7Fg3h1AH9gP7geOB90Wwnh8+qwXUgY4JAAA1hBAAQE1IhVB0dLT8+Mc/tj/djP3AfuB44H0RLueHoOyYAABwh5BqCQEAwgshBABQQwgBANQE5XNCd9LW1iYlJSVy4cIFGT9+vIwaNUrcxvz7z50757PsnnvukRkzZki4a2hokH379smIESNk6tSpd1zn/Pnz8t5779nnE6ZMmSL9+vUTN+0H83DgX//619tqsrOz7frhwpwLTpw4IR9++KE89NBDMm7cOFceD22fsR9C5XgIiRCqq6uTmTNn2jfg5z73OfnnP/8pzz//vOTl5YmbvPrqq1JaWipZWVneZSkpKWEdQuaNtHbtWvnLX/5iR9KYNGnSHUPoZz/7mfzgBz+wv6+urpbIyEjZu3evPPDAA+KW/WBOusuWLZOFCxdKbGysd/nnP//5oDrp3I09e/bImjVrbM8v80C7+WD2hS98QXbu3CkDBw50zfGwpxv7IWSOB08I+MY3vuHJyMjwNDc32/nCwkLTo89TWlrqcZOcnBzPihUrPG5SW1vr2bp1q6epqcmzZMkSz6JFi25bp6KiwhMZGel566237HxLS4tn8uTJnvnz53vcth/M+6KmpsYTrsx7//Tp0975Tz75xHP//fd71q1b56rjobCb+yEUjoegbwldv35dduzYIa+99po3zefPny8PP/yw/P73v5fMzExxk08++UR27dolw4YNk0ceecSOrxfOhg8fLt/+9re7XKegoMB+ssvJybHzUVFR8t3vfleeeuopuXTpkgwZMkRCXXf2QwdzpcC8V8aOHWvfJ+HEvPdvZd4HX/7yl+X999931fEwvxv7IVSOh6DvmHD69GlpaWmxTchbpaWlyb///W9xm3fffVfeeOMNWbVqlb0v9sc//lHczhwHqampPoPdmuPDXLY6deqUuIkZd/GnP/2pvRw1YcIEWbBgQVh/JYr5kFpcXOxzfnDj8XD9DvshVI6HoA8hcx/I+PSnl6FDh0p9fb24ycqVK+31bXOz8eTJk7J69Wp55plnpKqqStzMHCN3Oj4MNx0jCQkJ9ka8+aBi7hlUVFTIsWPHZN26dRKuzHugublZvv/977v6eFh9h/0QKsdD0IdQx9ATTU1NPsvNfExMjLiJ6YBgLi10+NGPfmQ/3ZneUm5mjpE7HR+Gm44RcwkqIyPDO29uWJsWc2FhoYSj5557Tt5++23bWcN8ZYBbj4fnOtkPoXI8BH0IjRkzxv40LYBbma7KHb9zq759+9prveY+kZuZ7ql3Oj4Mtx8jZij/cDw+TO9Yc4np73//u3zxi1907fHwfBf7IVSOh6APIdOMNjv31nsf5gD717/+JfPmzRO3MPfFLl++7LPsnXfesZcXbu2y7UZz58619wEqKyu9y/7whz/Y7uvhdtLpyv/+9z+feTMs5J///OewOz7Wr19vH1cwJ97Jkye79nhY/xn7IVSOh6DvHWf85Cc/sZeiTO+g9PR0+cUvfmEfPvva174mbnHt2jX50pe+ZPv8mzfT2bNnZcuWLfY5gMcee0zCmXn2obW11T6UZx7QM72fzGXJr371q/b3s2fPth9ITI+h733ve3LmzBn5zW9+E3SXHXp6P/z617+2N6fN8WAuO5kPbuZhxr/97W8SLkwv2Zdeesn+fzbfHGr2gTF48GB7HLjleHitG/shVI6HkBlFu7y8XLZt2yYXL160vTzMtc1QGao8UMwNV/NmMp/yTJfMRx99NOwDyFixYoVcvXrVZ9mAAQPsm6zDjRs37PFhHtozT8h/85vftMeJ2/bDoUOH7MnW9IAyXXK/9a1vhUWX5A6vv/66HD58+Lbl5uusX3nlFdccD693cz+EwvEQMiEEAAg/QX9PCAAQvgghAIAaQggAoIYQAgCoIYQAAGoIIQCAGkIIAKCGEAIAqCGEAABqCCEAgBpCCACghhACAIiW/wPEA9pE02dWAwAAAABJRU5ErkJggg==", + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAaEAAAGdCAYAAAC7EMwUAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjExLjAsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvlcelbwAAAAlwSFlzAAAPYQAAD2EBqD+naQAAGUxJREFUeJzt3Q+QF2X9B/DPAfIn4E5AjRQKKZThgkSBTKEBHRVR8Q8pWlqNjH/GyWaELGnKHJuRBn+JRuU4FTWhRsaQ4/WPhPIfkoOgEgalMsmlDojIHQgcAt/f7M7cDZce8T2/8Nzd9/Wa2Tl3v/vcLuuz+75n99nnW1EoFAoBAAl0SrFRAMgIIQCSEUIAJCOEAEhGCAGQjBACIBkhBEAyQgiAZLpEG7Rv3754/fXXo3fv3lFRUZF6dwAoQjYGwrZt2+LYY4+NTp06tb8QygJo4MCBqXcDgA+gtrY2BgwYkCaEVq9eHa+++moMGTIkhg4dWlTZrAXU+A+orKw8RHsIwKFQX1+fNyQar+WHNYR2794dl112WTz55JNx0kknxYoVK+LSSy+Nn/70pwd9a61xvSyAhBBA+3Qw1/ySh9Ddd98dy5YtixdeeCFvhr344osxatSoGD9+fFx11VWl3hwA7VjJe8fNnz8/pk6d2nQfsLq6Os4999x8OQAcshDas2dPrF27NkaMGNFseTafPSNqSUNDQ34Pcf8JgI6vpCG0ffv22Lt3b/Tp06fZ8n79+sXWrVtbLDdr1qyoqqpqmvSMAygPJQ2hbt265T937NjxnnDq3r17i+VmzpwZdXV1TVPWKw6Ajq+kHRN69OgR/fv3jw0bNjRbns0PHjz4gOHVGGAAlI+Sd0zIOiEsWrQoH/Ugs2vXrqipqcmXA8D+KgrZ+Aol9O9//ztGjx4dY8eOjUmTJsWCBQvilVdeiZUrV+bPhg5G1jEhezaU3ZrznhBA+1LMNbzkLaFBgwbFqlWr8lESHnvssRg3blz+wurBBhAA5aPkLaFS0BICaL+StoQA4GAJIQCSEUIAJCOEAEhGCAGQjBACIBkhBEAyQgiAZIQQAMkIIQCSEUIAJCOEAEhGCAGQjBACIBkhBEAyQgiAZIQQAMkIIQCSEUIAJCOEAEhGCAGQjBACIBkhBEAyQgiAZIQQAMkIIQCSEUIAJCOEAEhGCAGQjBACIBkhBEAyQgiAZIQQAMkIIQCSEUIAJCOEAEhGCAGQjBACIBkhBEAyQgiAZIQQAMkIIQCSEUIAJCOEAEhGCAGQjBACIBkhBEAyQgiAZIQQAMkIIQCSEUIAJCOEAEhGCAGQTJd0m4aDN3v27KIP1ze+8Y2iy1RUVBRd5uqrr47WuPfee4suc8QRR7RqW1A2IbRu3bpYs2ZNs2Vdu3aNyZMnl3pTALRzJQ+hhQsXxpw5c2LChAlNy3r27CmEADg8t+OGDBmShxEAHPYQ2rlzZyxevDi6d+8ew4cPj759+x6KzQDQzh2S3nHr16/PHyRPnz49BgwYEHfdddcB129oaIj6+vpmEwAdX8lDaPz48bFhw4ZYunRprFy5MubNmxczZszI51sya9asqKqqapoGDhxY6t0CoBxCaOzYsdGnT5+m+csvvzy/JffII4+0WGbmzJlRV1fXNNXW1pZ6twAo1/eEstbNpk2bWvy8W7du+QRAeSl5S2jjxo3N5rNWzapVq2LUqFGl3hQA7VzJW0IXXnhhjBkzJkaOHBmbN2+OuXPnRnV1dVx33XWl3hQA7VzJW0KPP/54/gxo2bJleSvojjvuiOXLl0evXr1KvSkA2rmSt4SyZzvXXHNNPgHAgRjAlA6rU6fDM0j8z3/+81aVu+eee4ouYwBTOhpf5QBAMkIIgGSEEADJCCEAkhFCACQjhABIRggBkIwQAiAZIQRAMkIIgGSEEADJCCEAkhFCACQjhABIRggBkIwQAiAZIQRAMkIIgGSEEADJCCEAkhFCACQjhABIRggBkIwQAiAZIQRAMhWFQqEQbUx9fX1UVVVFXV1dVFZWpt4d2oB9+/YVXeboo48uuszWrVvjcHnmmWeKLjNq1KhDsi+Q6hquJQRAMkIIgGSEEADJCCEAkhFCACQjhABIRggBkIwQAiAZIQRAMkIIgGSEEADJCCEAkhFCACTTJd2m4eB16lT830uzZ88uusy1114bh8udd95ZdJn58+cXXaZr165Fl4HDRUsIgGSEEADJCCEAkhFCACQjhABIRggBkIwQAiAZIQRAMkIIgGSEEADJCCEAkhFCACRjAFM6rGHDhkVbtnDhwqLL3HLLLUWXGTlyZNFloE23hF5++eX8BNqwYUOL66xevTpqampi3bp1H2T/AOjAigqhVatWxTnnnBMTJ06MSy+9NJ544on3rLN79+646KKLYsKECXH33XfHmDFjYtq0aVEoFEq53wCUWwi99dZbcdNNN8VLL73U4jpZ8CxbtixeeOGFWLp0aSxfvjwefPDBuP/++0uxvwCUawidddZZeSuooqLigF+6NXXq1BgwYEA+X11dHeeee26rvowLgI6tpL3j9uzZE2vXro0RI0Y0W57NZ8+IWtLQ0BD19fXNJgA6vpKG0Pbt22Pv3r3Rp0+fZsv79esXW7dubbHcrFmzoqqqqmkaOHBgKXcLgHIIoW7duuU/d+zY8Z5w6t69e4vlZs6cGXV1dU1TbW1tKXcLgHJ4T6hHjx7Rv3//93TdzuYHDx58wPBqDDAAykfJR0zIOiEsWrQo9u3bl8/v2rUrf18oWw4ArW4Jbdq0qdm7QStWrMhvsx1//PFxyimn5MtuvfXWGD16dEyZMiUmTZoUCxYsiC5dusT06dOL2RQAZaColtDGjRvzUMmmLGRee+21/L+zMGo0aNCg/KXWoUOHxmOPPRbjxo3LP886JwBAq1tCw4cPP6jxrrLebVmPNwA4EAOY0mFlQ0YV65JLLim6TPYM9HDJRiAplgFMact8lQMAyQghAJIRQgAkI4QASEYIAZCMEAIgGSEEQDJCCIBkhBAAyQghAJIRQgAkI4QASMYApnRYnTt3LrpM9t1Xbdn3vve9osvccMMNh2RfoBS0hABIRggBkIwQAiAZIQRAMkIIgGSEEADJCCEAkhFCACQjhABIRggBkIwQAiAZIQRAMkIIgGSEEADJCCEAkhFCACQjhABIRggBkIwQAiAZIQRAMkIIgGSEEADJCCEAkhFCACQjhABIRggBkEyXdJuGtufkk08uusxDDz10SPYFyoGWEADJCCEAkhFCACQjhABIRggBkIwQAiAZIQRAMkIIgGSEEADJCCEAkhFCACQjhABIpqJQKBSijamvr4+qqqqoq6uLysrK1LtDGcnqXLH69u0bh8txxx1XdJkNGzYckn2BUlzDWzWK9ssvvxzPP/98jBkzJj760Y82+2zdunWxZs2aZsu6du0akydPbs2mAOjAigqhVatWxcyZM+OVV17Jp/nz58eVV17ZbJ2FCxfGnDlzYsKECU3LevbsKYQA+GAh9NZbb8VNN90U55xzTnTq1PLjpCFDhuRhBAAlC6GzzjrroNbbuXNnLF68OLp37x7Dhw8/rPfMASjz3nHr16+P2bNnx/Tp02PAgAFx1113HXD9hoaG/EHW/hMAHV/JQ2j8+PF5b5ylS5fGypUrY968eTFjxox8viWzZs3Ke1I0TgMHDiz1bgFQDiE0duzY6NOnT9P85Zdfnt+Se+SRR1osk3V2yLryNU61tbWl3i0A2qBWddEuVta62bRpU4ufd+vWLZ8AKC8lbwlt3Lix2XzWqsm6do8aNarUmwKgnFpCWWvmiSeeaJpfsWJF3gPu+OOPj1NOOSVfduGFF+YvsY4cOTI2b94cc+fOjerq6rjuuutKv/cAlE9LKGvlLFiwIJ+mTJkSr732Wv7fWRg1evzxx/NnQMuWLctbQXfccUcsX748evXqdSj2H4B2zNhxsB9jx0E7GDsOOqrWdJC54IILWrWtmpqaosts27at6DL//Oc/iy5z4oknFl0GWsNXOQCQjBACIBkhBEAyQgiAZIQQAMkIIQCSEUIAJCOEAEhGCAGQjBACIBkhBEAyQgiAZAxgCvvJvh+rWDfeeGOrjuGf//znVo1OXKwlS5YUXcYAphwuWkIAJCOEAEhGCAGQjBACIBkhBEAyQgiAZIQQAMkIIQCSEUIAJCOEAEhGCAGQjBACIBkDmMIHdOaZZ7aqXN++fYsu88YbbxRdZufOnUWXKRQKRZepqKgougxoCQGQjBACIBkhBEAyQgiAZIQQAMkIIQCSEUIAJCOEAEhGCAGQjBACIBkhBEAyQgiAZIQQAMlUFFozXO4hVl9fH1VVVVFXVxeVlZWpdwcOiQEDBhyWUbRbY8uWLUWXyc5ZKPYariUEQDJCCIBkhBAAyQghAJIRQgAkI4QASEYIAZCMEAIgGSEEQDJCCIBkhBAAyQghAJIRQgAk06XYAu+8804899xz8e6778bw4cPjqKOOet/1Vq9eHa+++moMGTIkhg4dWop9BaCcW0K33357nHDCCXHLLbfEbbfdFh/72MfizjvvbLbO7t2746KLLooJEybE3XffHWPGjIlp06ZFG/zGCADaU0vomGOOiXXr1kXv3r3z+YcffjguvvjiGD9+fIwePTpflgXPsmXL4oUXXsi/L+XFF1+MUaNG5etcddVVh+ZfAUDHbwldf/31TQGUyVo8Xbt2zW/PNZo/f35MnTq16Qu7qqur49xzz82XA0DJOiY89dRT+e23LGgye/bsibVr18aIESOarZfNZ8+IWtLQ0JB/E9/+EwAdX6tD6O23346rr746zjvvvDj99NPzZdu3b4+9e/dGnz59mq3br1+/2Lp1a4u/a9asWflXwTZOAwcObO1uAdDRQyhrqWS32I488sh48MEHm5Z369Yt/7ljx45m62fh1L179xZ/38yZM/PvIm+camtrW7NbAHT0Ltrbtm2LiRMn5rfelixZEpWVlU2f9ejRI/r37x8bNmxoViabHzx4cIu/MwuvxgADoHwU1RLKWjRZAGXPgR599NG8JfTfshbSokWLYt++ffn8rl27oqamJl8OAK1uCV1wwQV5B4M5c+bE0qVLm5YPGzYsnzK33npr3l17ypQpMWnSpFiwYEF06dIlpk+fXsymACgDRYXQRz7ykTjnnHPiT3/6U7Pll112WVMIDRo0KFatWhU//vGP47HHHotx48blQZR1TgCA/VUU2uBQBlnHh6yXXNZJYf9nTtCRNL5LV4w33ngjDoctW7YUXSY7Z6HYa3jRHROA0jjiiCPa7KF84IEHii5zww03HJJ9oWMzijYAyQghAJIRQgAkI4QASEYIAZCMEAIgGSEEQDJCCIBkhBAAyQghAJIRQgAkI4QASMYAppDI008/XXSZM844o+gy//rXv4ous2zZsqLLGMCU1tASAiAZIQRAMkIIgGSEEADJCCEAkhFCACQjhABIRggBkIwQAiAZIQRAMkIIgGSEEADJVBQKhUK0MfX19VFVVRV1dXVRWVmZenegzXj99deLLjNu3Liiy2zZsqXoMmPHjo3WqKmpaVU52q5iruFaQgAkI4QASEYIAZCMEAIgGSEEQDJCCIBkhBAAyQghAJIRQgAkI4QASEYIAZCMEAIgGSEEQDJd0m0aKNaxxx5bdJnHH3+86DL3339/0WX27t1bdBnQEgIgGSEEQDJCCIBkhBAAyQghAJIRQgAkI4QASEYIAZCMEAIgGSEEQDJCCIBkhBAAyVQUCoVCtDH19fVRVVUVdXV1UVlZmXp3ADhE1/CiR9F+55134rnnnot33303hg8fHkcddVSzz9etWxdr1qxptqxr164xefLkYjcFQAdXVAjdfvvtcd9998Xxxx8fnTt3jmeffTZuu+22uPnmm5vWWbhwYcyZMycmTJjQtKxnz55CCIAPFkLHHHNM3tLp3bt3Pv/www/HxRdfHOPHj4/Ro0c3rTdkyJA8jACgZB0Trr/++qYAylx00UX5rbbs9tz+du7cGYsXL86/TGvLli3FbAKAMvKBesc99dRTsXv37qiurm62fP369TF79uyYPn16DBgwIO66664D/p6Ghob8Qdb+EwAdX6tD6O23346rr746zjvvvDj99NOblme35jZs2BBLly6NlStXxrx582LGjBn5fEtmzZqV96RonAYOHNja3QKgo3fRzloqZ599duzbty+WLFnyP7vgjRgxIu+ocM8997TYEsqm/X9/FkS6aAO0P4e0i/a2bdti4sSJsWfPnoMKoEy2M5s2bWrx827duuUTAOWlqNtx27dvzwMoew706KOPxpFHHvmedTZu3Nhsvra2NlatWhWjRo364HsLQPnejstuqWXvBmXvAe0fQMOGDcunzKmnnhpjxoyJkSNHxubNm2Pu3LnRv3//+Mtf/hK9evU6qO0YMQGg/SrmGl5UCH3+85/PW0H/7bLLLsunTPZs55e//GU888wz8aEPfSgPpCuuuCJ/ufVQ/AMAKJMQOlyEEED7Vcw13CjaACQjhABIRggBkIwQAiAZIQRAMkIIgGSEEADJCCEAkhFCACQjhABIRggBkIwQAiAZIQRAMkIIgGSEEADJCCEAkhFCACQjhABIRggBkIwQAiAZIQRAMkIIgGSEEADJCCEAkhFCACTTJdqgQqGQ/6yvr0+9KwAUqfHa3Xgtb3chtG3btvznwIEDU+8KAB/gWl5VVXXAdSoKBxNVh9m+ffvi9ddfj969e0dFRUWzdM2Cqba2NiorK6NcOQ6Og/rgvGjL14csVrIAOvbYY6NTp07tryWU7fSAAQNa/Dw7sOUcQo0cB8dBfXBetNXrw/9qATXSMQGAZIQQAMm0qxDq1q1bfOc738l/ljPHwXFQH5wXHeX60CY7JgBQHtpVSwiAjkUIAZCMEAIgmTb5ntD72bt3byxfvjw2b94cJ510UgwaNCjKTfbvf/XVV5stO/roo+PMM8+Mjq6uri6WLFkSH/7wh2Ps2LHvu85rr70Wzz77bP5+wumnnx5HHHFElNNxyF4O/P3vf/+eMhMmTMjX7yiya8Hq1avjP//5T3z84x+PYcOGlWV92Ps/jkN7qQ/tIoQ2bdoUZ511Vn4CfuITn4inn346vv3tb8fMmTOjnNxzzz2xYsWKGD16dNOyoUOHdugQyk6kGTNmxO9+97t8JI1TTz31fUPohz/8YXz961/PP9+wYUN07tw5Hn300fjoRz8a5XIcsovuFVdcEZMnT44ePXo0Lf/kJz/Zpi46H8TixYvjpptuynt+ZS+0Z3+YnXzyybFo0aLo1atX2dSHxQdxHNpNfSi0A1/4whcKI0eOLOzYsSOfr6mpyXr0FVasWFEoJ1OnTi1MmzatUE42btxYuO+++wrbt28vTJkypXDhhRe+Z521a9cWOnfuXPjVr36Vzzc0NBROO+20wvnnn18ot+OQnRe1tbWFjio791966aWm+TfffLNw3HHHFW6++eayqg81B3kc2kN9aPMtoV27dsXChQvjBz/4QVOan3/++XHCCSfEAw88EKNGjYpy8uabb8bDDz8cRx11VHzqU5/Kx9fryI455pi49tprD7jOggUL8r/spk6dms937do1vvKVr8SVV14ZW7Zsib59+0Z7dzDHoVF2pyA7V0488cT8POlIsnN/f9l58NnPfjaef/75sqoP5x/EcWgv9aHNd0x46aWXoqGhIW9C7m/48OHx97//PcrNM888Ez/5yU/iuuuuy5+L/eY3v4lyl9WD6urqZoPdZvUju231j3/8I8pJNu7i97///fx21CmnnBIXXHBBh/5KlOyP1GXLljW7PpRjfdj1PsehvdSHNh9C2XOgzH//9dKvX7/YunVrlJNrrrkmv7+dPWx88cUX48Ybb4wvfelLsX79+ihnWR15v/qRKac60qdPn/xBfPaHSvbMYO3atfHcc8/FzTffHB1Vdg7s2LEjvva1r5V1fbjxfY5De6kPbT6EGoee2L59e7Pl2Xz37t2jnGQdELJbC42+9a1v5X/dZb2lyllWR96vfmTKqY5kt6BGjhzZNJ89sM5azDU1NdERffOb34yHHnoo76yRfWVAudaHb7ZwHNpLfWjzITR48OD8Z9YC2F/WVbnxs3LVpUuX/F5v9pyonGXdU9+vfmTKvY5kQ/l3xPqR9Y7NbjH98Y9/jE9/+tNlWx++fYDj0F7qQ5sPoawZnR3c/Z99ZBXsb3/7W5x33nlRLrLnYm+//XazZX/961/z2wv7d9kuR5MmTcqfA6xbt65p2a9//eu8+3pHu+gcyBtvvNFsPhsW8re//W2Hqx+33npr/rpCduE97bTTyrY+3Po/jkN7qQ9tvndc5v/+7//yW1FZ76ARI0bEvffem7989rnPfS7Kxc6dO+Mzn/lM3uc/O5leeeWVmDt3bv4ewNlnnx0dWfbuw+7du/OX8rIX9LLeT9ltyUsuuST/fOLEifkfJFmPoa9+9avx8ssvxy9+8Ys2d9vhUB+Hn/3sZ/nD6aw+ZLedsj/cspcZ//CHP0RHkfWS/e53v5v/f86+OTQ7BpkjjzwyrwflUh9+cBDHob3Uh3YzivaaNWti3rx58dZbb+W9PLJ7m+1lqPJSyR64ZidT9lde1iXzjDPO6PABlJk2bVq88847zZb17NkzP8kavfvuu3n9yF7ay96Q/+IXv5jXk3I7Dk888UR+sc16QGVdcr/85S93iC7JjX70ox/Fk08++Z7l2ddZ33nnnWVTH350kMehPdSHdhNCAHQ8bf6ZEAAdlxACIBkhBEAyQgiAZIQQAMkIIQCSEUIAJCOEAEhGCAGQjBACIBkhBEAyQgiASOX/AS7EY3NPwgLQAAAAAElFTkSuQmCC", "text/plain": [ "
" ] @@ -628,6 +713,240 @@ "plt.imshow(image.reshape(28,28), cmap='Grays')\n", "plt.show()" ] + }, + { + "cell_type": "markdown", + "id": "0dabacd4", + "metadata": {}, + "source": [ + "## 6.1 Hyperparameter Tuning: Learning Rate Impact\n", + "To optimize the network's performance, we evaluate the impact of the Learning Rate ($\\eta$) on the final classification accuracy. The network is trained across a sweep of different learning rates `[0.01, 0.1, 0.2, 0.3, 0.6, 0.9]` while keeping the hidden nodes constant (100 nodes). \n", + "\n", + "The results are plotted to identify the optimal step size for the Gradient Descent algorithm, avoiding both slow convergence (values too close to 0) and divergent oscillations (values too close to 1)." + ] + }, + { + "cell_type": "code", + "execution_count": 56, + "id": "b3be2fde", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Starting Learning Rate sweep. This may take a few minutes...\n", + "Training network with Learning Rate: 0.01...\n", + "Performance for LR 0.01: 0.8683\n", + "\n", + "Training network with Learning Rate: 0.1...\n", + "Performance for LR 0.1: 0.9249\n", + "\n", + "Training network with Learning Rate: 0.2...\n", + "Performance for LR 0.2: 0.9274\n", + "\n", + "Training network with Learning Rate: 0.3...\n", + "Performance for LR 0.3: 0.9178\n", + "\n", + "Training network with Learning Rate: 0.6...\n", + "Performance for LR 0.6: 0.8587\n", + "\n", + "Training network with Learning Rate: 0.9...\n", + "Performance for LR 0.9: 0.8326\n", + "\n" + ] + }, + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAscAAAHVCAYAAAAdAFYjAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjExLjAsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvlcelbwAAAAlwSFlzAAAPYQAAD2EBqD+naQAAbd9JREFUeJzt3Qd0FGUXxvELBEIJvffeQXqT3lEB6UWpCkhRUIqAgoCIIApYsCAgvQkIUqSDVOm9996khhoI7Hfu6+5+m5BAEpJs+//O2UN2sjOZmd2QZ9+9c98YFovFIgAAAAAkJucAAAAA+A/hGAAAALAiHAMAAABWhGMAAADAinAMAAAAWBGOAQAAACvCMQAAAGBFOAYAAACsCMcAotSECROkevXqki9fPmnVqhVnG9Hm9ddfl27dunHGAYQL4RjwMn///bfkyZPHfsufP7+UL19eevfuLZcuXYrUnzVv3jx55513pFmzZjJ37lz54osvInX7iHqHDx82r5NffvnF7U73yZMn5cKFC077+fq6d/xdK1y4sNSvX19mzpwZ4W3u3LnTbEt/jwFEDZ8o2i4AF3X37l05cuSIfPnll+YP9ePHj80f3D59+sjkyZNlx44dki5dukj5WQsXLpSMGTPKu+++GynbQ/R7+PCheb1cu3bN7U7/kiVLxNfX12k///Tp03Lr1i17kL1+/bpMmTJFmjdvbn7Pvv7663Bv8/79++b50N9jAFGDcAx4qbRp05oRKFWwYEFJlCiRNGjQQH744QcZOnRopPyMy5cvm+0CzpA1a1ann3gfHx/775kqW7as7N69W7799lvp16+fJE6c2Kn7B+BZlFUAMEqVKmX+PXr0qP2M6IjXW2+9JUWKFJGSJUtKjx495MqVK/bvP3nyxPzh/+6778zH7y1atJBChQrJiBEjzPJ169bJsWPH7B8r//XXX/Z1p0+fLrVr15ZXXnlFKlasaAL5vXv3XrjtadOmyapVq8z39uzZY8o1qlatavZPyzaePn1q1p8zZ45Uq1bN7HuvXr3kwYMHQZ7pU6dOPfORd6NGjWTRokVBHuf4sxYvXiw1atQwj+3QoYMJ/8EFBgbKuHHjzLHp4/TfSZMmicViCfKYX3/9VV577TVz/FWqVDFvSnR5aA4ePGj2Y+rUqc98T0d1tTxm1KhR5r7+LP2Z+rP1+PXnDB8+3Iw6RpV///1XPv30UylXrpx5nvRTiRUrVjxTf24733nz5jWvuY4dO5rn11H//v2lTJky5lMN/YTj1Vdflbp165rvaQlQ3759TQlQ+/btzfHpc6LPTVhqjsOzvm2f9fkpWrSodO7c2ZxrLRXS10pE6bb0udayDxv9vXJ8PdrOob6+HT+Jefvtt83XnTp1sj9W99FGX/96/4033jDb0N8t/X189OhRhPcX8DoWAF5l4cKFmtIsEyZMCLJ869atZnn79u3N/W+++cYSK1Ysy4cffmhZv369ZcWKFZaKFStaMmbMaLl06ZJ5zOPHj806LVu2tFSpUsVse/HixZZp06ZZDh06ZClXrpwlZ86c5mu9+fv7m/W6dOlitj1o0CDL5s2bLZMmTbKkTJnSUrRoUcu9e/eeu+1Zs2ZZ5s2bZ77XtWtXs61NmzaZbSRIkMDSr18/y9ixYy0dO3a0bNy40TJlyhSLn5+fpXPnzkGONyAgwL5fetPH9u7d2xIjRgzL9OnT7Y+z/ayePXuac6Pn4s8//7RkyJDBUqJEiWe2Wa1aNUvChAnN+dNj031u27at5eeffzaPefTokaVGjRqWZMmSWX788UfLtm3bLFOnTrWkTZvWUr9+fcvTp09Dfe7y5s1rKVWq1DPLR44cafbxwIED5v7gwYMt8ePHt/zyyy+WnTt3WlauXGnp27evpV27duF8tVgsu3btMtvWbYbm2LFjZv/1+dPzpa8lfW5jx45tngub69ev28+37uvSpUsttWrVsiROnNhy9uxZ++PeffddS/LkyS3vvPOOZcSIEeb5HTBggPle6tSpLQ0aNLC89tprlrlz51r++ecfS4sWLSwxY8Y0P9dR7ty5LQ0bNgyyLDzrf/rpp2b5Z599Zp5LfV3XrFnTUqFCBUuxYsVeeO70uUqfPv0zy998801zTs+dO2dfpq/34K9H/bn6e/Lrr7+ax9y+fdvsg66rryfbY/W8qidPnljq1atnzueoUaPM8ejvi/7O6n7r9wG8GOEY8DIhhWP946oBVIPh33//bTl8+LAJBRoIHd29e9eSJk0ay3vvvRckwGoYvHHjhv1xtoCnf5Dz588fYgjv379/kOUaOh1D2PO2bQusGnIcffTRR+bxb7/9dpDlvXr1svj6+lru3LnzwvOjQalAgQL2+7afpaHD0YwZM8xy3W+b4cOHm2XLly9/Zrsaih2DrJ5nR2vXrjXL9fkJjQZuxxBso/tbunRp+/1ChQpZGjduHOo+RHY41tdOunTpTHgLft71+Qi+3JE+zylSpLD06dMnSDjW1+L333//zGtKw60+l6dPnw7ypkTfXAV/3kMLx2FZX7+vvwPdu3cPsr4Gdd23iIbjRYsWmcCrvxthoW/ysmTJ8szvSUivEw3R+j39GY62b99u9llfswBejLIKwEvpR8v6kWz27NklderUcvbsWXMVvX4MqyUJ+vFs8AvpEiRIYD6C1gudHNWqVUuSJk1qvx8jRoxQf66tbKFNmzZBluvH8Tly5HimrOF5227SpMkzH1ffuXPHlBEEXx4QEGAukHK0adMm8xG5lmTox/x6PvTYtLTEsQxCNW3aNMh9XUfpxVE2s2fPlpw5c5rWdcHFjh3b/Pv777+bc67n2VGFChUkWbJkz5xbR9oKT7czfvx4+7KtW7fK/v37gzxXGTJkkNWrV5tuIXpBXfB9iExaZrBmzRpp3LjxM/XlWhagz8fGjRvNfX1NablHvXr1TMmJnvMCBQqYxwQvrdDzbyshCP68ly5dWjJnzmy/HydOHLM9x+fiecKy/tKlS83+almRIy33yJQpk4SVY7lEypQpTamLdrHQ10pw27dvN6UeWm6irQ91HS2rOHPmzDNlQSHR15b+LmtJhaNixYqZC2Of99oC8H9ckAd4qa5du5rwEjNmTBPKUqRIYf+e/jFWDRs2NKHE+imT/Y998CvlwxMWLl68GOo6Glgca55ftG0NgY6SJEny3OXaLcBmwYIF5vhbtmwpQ4YMMRco6sVT33zzjQmfWvOs9200XDjSc6YcuzicP3/eXNz4PHpu/f39TShUtnOrNz2vIdUx22i40tpb7XgwbNgwE3Z/++0386bFMbxr/XK7du3M86fdGjRsaSjTOlV9bGTSN1W677NmzZKVK1faXyf6r74hUbZj0prdiRMnyueffy4ff/yxeV709af7Fjz86X7aznFwwZ8LpY/V+vawCMv6tn0O/lqyLXN80/E8yZMnl/nz55vzoW3ltIZa3wBqLXSJEiXsj9Nzp2/q9DnT86MdY/T5/emnn8zzqecyXrx4EX5tXb169bmvLQD/RzgGvJRjt4rgEiZMaP7VIGP72lHwkeG4ceOG+efatnfz5k0T9hxpeA3+85637VixYoVrueNosHYLyJ07tzlGR6GN0IVlmzpy+qKWZ3p8GnxCurDO9v3n0dCro4l6cZaOqs+YMcMEY8f1tEuDXkiogUhHbZcvX24uctNPBrZt2/bckf3wsv1cHeXVfQvttaahTS9U7N69uwnGwS/m008NXvZ5Dz7a/zLr+/n52V+nOhrrSJe9KKiG1K1CR8q1W4WGV/3UQy+ytG3n+++/l/Tp05sLVfUNg43tDUZYn4ssWbKYT35CEtlvjABPRTgG8Ay9On/kyJGmDEE/Lo9M2nlAuypoeNOPlx1HlA8dOhRts+jpR/ka2hxptwwNki9z3nTUWUfwHD+2D/4YHfnVkXrH0fqw0rIWHfnUn6MjzRo6Q+sjnSpVKjM6rjcNeIMGDTIjvaHtW0RoqNX90fZkob3Zsn3ioKPxwc+5ln/oMbgaLZ9Qa9euDXJc2uXi+PHjL/yEIDQahrW/sY4Q6xs0LW+yvR71OXIMxtphwrHDi+ObhpA6m+hrS7u76Ju0yOpVDngjao4BhNgCS0clP/jggyBhUUex/vjjDxOcI+rNN9807ct0Rj6tlVU6UYLW/uooW8+ePaPlGdGaXx1V3bx5s7mvQVP3IVeuXBHepgYdDSYa+m1tujQQagmHfrSutN2ZjkpqK7ATJ04EGTXXkg590/A8Gp7atm0ry5YtM4/X4KZvOBxp+YTWr9ra2mno1xFj/YhfA7PStm66rj72ZegotIY8DbnaMs+x5Ebf7Oj2dTReg59tpN42uq51xlpC4Ar9iIPTc1qpUiWzf9rGzxZgtZ1h8FHu8NJ+4loHrO31dBTa9nrUyXi0ftv2/Gi7wOBlRToyrK8BfTMSnJ7/NGnSmODtWMOtP0NDc2jt6gAERTgGEGLg0TCnI5L6cbl+XKt/pDVc6UfzlStXjvBZ0zpKDdxab6kXyukIl5ZXaDjUYPgy4TQ8Bg4cKHXq1DEfc2sNqQYevZDuZY5NR1D1Ij+tpdXj0FFSrWXVvrPaT1fpz9JArqPG2ptYQ6MGVr0AS0Oj9qZ9EQ3xWgKwb9++EEeNNdS9//77Jqhr8NTzqwFZRyFtH+NrcNYL0M6dOxemY9Nw5diH13bT5Rr29GIv7YutF0/qa0VfM1o6oBe62Wap07pk/bl6DrSEQN8gaLB21YlitHylePHi9tepPjd6vvV8vuzFjdqTW98UakBWOkOllsdob249PxqCNUDra9SRvm4+++wz0xdcz7Njn2N9Hf3zzz9mXd1n3U8Ny/pa1E8zdHsAXiyGtqwIw+MAeAgNSRqINLiFZXYu/S9CSx70Xw00wetVdYQqtBIBvUBNJ3IIbWRQR8f0IiENkyFdfBXatnV0UretIcCxNtV2bFo24FgTaluuYSJ+/PjPnA8N5hp+dORaA6rebB+lh/azNOTpxYMaSELad92u1vzqdkObwlhH4vVjeg2HoV18FhodddZzG/xYHelFY1rOoAE8eA2vPp8ajnUUO6SLzhz3USdMCU3w5+f27dvmfOrrK7T90vPiWGKh5Tt67m37oa8JPe8hjdBqSYPWzgYvz9DXqI5QaycQG91vPfeOJQbhWd9x5FVHjfX1r/XK2pFEXx9a9/08Gki1/CGkbSo9/9opw/H3Q/fB9rrRAH7jxg1zXwOuY8mFPk73WV8DIb0Gdbl+X481IuU7gDcjHAMAEEZaLqNBVeu3tUQGgOchHAMAEAKt69aRX62/1xFjbcXWvHlzU86i9dRasgDA87hEzbF+3KkXj4T1imX9OFPfvevFC6G1XdKPDPUxO3bsMB9JAQAQHloT/uuvv5o6ai2p0JpyLTPRnsQEY8BzOXXkWOvh9GIfvZBD6+a0Puurr74yV8iHRmeD0qb9Ws+n/2FpLeAvv/wSpAH+3r17zX2tE9P/0LRuUd/5a1/RsPamBADAVr+ro8Zao+84WyMAz+TUkWOt2dKwqxeW6EdU2vxcZ+3asmVLqP9Bab9OnWZWg7R+tKXraF9Ux5mNtHWQhm0NzjpyrNvWq+B//PHHaDw6AIAn0Avj9IJMgjHgHZwajrX9jM6oZLtquF69embmIFtbmuB0BFivvu3YsaN9hiOdblOvcHZcR2db0jZRtqlf9Qp1fYwuBwAAAFxuhjwNudpiKHjfxZIlS8quXbtCXMfWqkY/3rLNSa+lGdo2SEeIbbRpu04koO/0dQRZe6pqy6jOnTuHuj9aR+Y4TafWNWsLHe3rGplTrQIAACByaHWwtlrU9oeO7Q7dMhxr8FQaPh3pfdv3grM1jdfSC714T3s3/vTTTya8Oq6jU2hqyNbZqnTEWC/M0wbrwWcacqQN1bXMAwAAAO5Fmzs8r2e7W4Rj2+xCOvLrSLtPaFP00EybNs1cgDdv3jzzWJ2BSWdi0tFk2zsILbWw1RzrtvSEaVjWljz9+vULcbsapLt37x6kkb2GaW0i76qzNwEAAHgzf39/M5GOZsHI4rRwrC1xdPjbFmpt9P7zRng17OrIsd5shg0bZqbctJVraIs3nZrTFrL1Z9WtW1cWLFgQajjWWZRCmsVKSzkIxwAAAK7Hdn1ZZJbAOu2CPJ3C9dVXXzWB1Ubbs2n/yOrVqweZ6tOxBjl4X2PtQqGdKrRjhS3MaujWUWNHOnqs88wDAAAALjdyrHR0V4OwljSUKVNGfvjhBzNHfIcOHYKMCm/evFn2799v7n/55ZemPKJy5cpmXvqBAweaeuLSpUub72sf4/bt28snn3wiT548kWzZspkL8pYuXSqLFy922rECAADA9Tk1HFesWFHWrFlj+g9rv+OCBQvKlClTxM/Pz/6YnDlzyqNHj+z3BwwYIN9//72MGjVKkiRJYlq4acmEI91eqVKlzNSfc+fONfXHmzZtsgdoAAAAwOVmyHP1Am+dDUkvzKPmGAAAwDvymlMnAQEAAABcCeEYAAAAsCIcAwAAAFaEYwAAAMCKcAwAAABYEY4BAAAAK8IxAAAAYEU4BgAAAKwIxwAAAADhGAAAAAiKkWMAAADAinAMAAAAWBGOAQAAACvCMQAAAGBFOAYAAACsCMcAAACAFeEYAAAAsCIcAwAAAFaEYwAAAMCKcAwAAABYEY4BAAAAK8IxAAAAYEU4BgAAAKwIxwAAAIAV4RgAAACwIhwDAAAAVoRjAAAAwIpwDAAAAFgRjgEAAAArwjEAAABgRTgGAAAArHzEye7fvy+LFy+WK1euSMGCBaVixYovXOf8+fOybt06uXXrlpQuXVqKFi0a4uOOHTsma9eulfjx48sbb7whiRMnjoIjAAAAgKdw6sjxpUuXpFChQjJkyBDZsWOHNG7cWFq0aPHcdaZMmSK5c+eW2bNny65du6RmzZry8ccfP/O4vn37SpEiRWTNmjXmVr58eTl58mQUHg0AAADcXQyLxWJx1g9v1aqV7N+/X/755x/x9fU1X2tYnjt3rtSrV++Zx9+7d09SpkwpgwYNkl69epllJ06ckLx588rq1aulXLlyZtnkyZOlXbt2smnTJilevLg9iAcGBkrGjBnDtG/+/v5mpPn27duSKFGiSD1uAAAAvLyoyGtOGzl+8uSJ/PHHH9K6dWsTjFWBAgVMwP39999DLZN48OCBVK1a1b4se/bskiVLFpk5c6Z92YgRI6Rp06b2YKzSpk0b5mAMAAAA7+S0muNz586ZkWAtkXCUJ08e2bp1a4jrZM6cWWLFiiXbtm2z1xlfvnzZbOvAgQPmvm5z37590q1bNzMivXPnTkmXLp1Ur15d/Pz8Qt2fgIAAc3N8J6J0tFlvAAAAcC1RkdGcFo7v3Llj/k2SJEmQ5Xrf9r3gkiZNKgMGDJAPP/xQ9uzZIylSpJA5c+ZIpkyZ7OvcvHlTtFJESys04OoFexMmTJD3339fVqxYIfny5Qtx20OHDjXlGsFpXXOCBAki4YgBAAAQmXRQ1GPCsXaQcByhtdGakeeF0f79+5uL8PQiOy2xmD59ugm1tnBs265+b/v27RIzZkwTlrULhtYpa2eMkOgFfN27d7ff1/3SMgy9qI+aYwAAANcTPEe6dTjWEgmtNdYL6hzp/Zw5cz533ZIlS5qbevr0qWzevFnatGlj7idLlsxctKdhWIOxihEjhlSqVMl0ugiN7out9tmRj4+PuQEAAMC1REVGi+nMg9Hew9OmTTMBV50+fdr0Ja5fv779ccuXL5eJEyfa7x8/fjzIdsaMGWP6Hbdv396+rEGDBqYu2ZHWMefKlSsKjwgAAADuzqmt3HSU+NVXXzWTf+hIsHac0FHjv/76y1x4p7Qlm44Ma5s39f3338v8+fPNSPCRI0fM17/99pvpTmFz9epVs13tZFG2bFnZsmWLuThPSzG0VVxY0MoNAADAtUVFXnNqOFb//vuvzJgxwz5Dnk4EYgvGSif7OHv2rPTo0cO+bOPGjbJy5Upz8Z6OMusFecHdvXvXbPfMmTPm+w0bNpTkyZOHeb8IxwAAAK7NI8OxqyIcAwAAuDaPmgQEAAAAcDWEYwAAAMCKcAwAAABYEY4BAAAAK8IxAAAAYEU4BgAAAKwIxwAAAIAV4RgAAACwIhwDAAAAVoRjAAAAwIpwDAAAAFgRjgEAAAArwjEAAABgRTgGAAAArAjHAAAAgBXhGAAAALAiHAMAAABWhGMAAADAinAMAAAAWBGOAQAAACvCMQAAAGBFOAYAAACsCMcAAACAFeEYAAAAsCIcAwAAAFaEYwAAAMCKcAwAAABYEY4BAAAAK8IxAAAAYEU4BgAAAKwIxwAAAICVjzjZ4cOHZezYsXLlyhUpWLCgdOnSRfz8/J67zuLFi2XZsmVy69YtKV26tLRr107ixIkT4mOXLl0q48aNk9q1a0ubNm2i6CgAAADgCZw6crxz504pVqyY3LhxQ8qXLy9z5swx/wYEBIS6To8ePaRly5aSPn16qVixokyePFlef/11efr06TOPvXDhgnTo0EHWr18vu3fvjuKjAQAAgLuLYbFYLM764TVr1hQfHx8zEqyuX78uGTNmlJEjR0rHjh2fefy1a9ckZcqUMm3aNHnrrbfMstu3b0uGDBnkt99+k8aNG9sfq2G5SpUq0rRpUxkzZoxUqlRJvv322zDvm7+/vyROnNhsP1GiRJFyvAAAAIg8UZHXnDZyrKPDq1evlkaNGtmXJU+eXKpWrSp//fVXiOvoSLDKnTu3fZmekNSpU8uiRYuCPPbzzz+XhAkTSqdOnaLsGAAAAOBZnFZzfPbsWQkMDJRMmTIFWa73165dG+I6uXLlMu8Kfv/9d1OOobZv3y6nT5+WNGnS2B+3bt06U8e8a9euMO+PhnXHcg59J6J0H/UGAAAA1xIVGc1p4dgWROPHjx9kuV6M9/DhwxDXiRcvnkyaNEnatm0rK1askBQpUsiJEyfMRXm27WlpRosWLUw4TpUqVZj3Z+jQoTJo0KBnlmvATpAgQTiPDgAAAFHt3r17nhOOtRxC3bx5M8hyDbdJkiQJdb169epJ5cqVzQV2Dx48kLJly5paY9s6s2bNMqO+WoOsN6Ujy1rXfP78eTPqHDPms9Ukffv2le7du9vv6za0/rlIkSLUHAMAALgg2yf9HhGO9SK6pEmTyt69e023CRu9/8orr7wwWGunCts7ho0bN8qAAQPsF/kFHzHesWOHqVNu1qyZxIgRI8Rt+vr6mltwesGg3gAAAOBaoiKjOe2CPA2pb7/9towfP970K1Zaa7xt2zZTFmHzyy+/BBnRXblypX0IXTtS9OrVy5RivPvuu2ZZ9uzZzUV+jjcN0zly5DBfhxaOAQAAAKcOiQ4ZMsT0Os6bN6+5bdmyRT755BPTgs1GL7jbvHmz/b6WUhQqVMhcnHfs2DH7RB86Cg0AAAC4bZ9jpT9eR4t1hrwCBQpI1qxZnymJ0ElCqlevbl+mdcpbt241dcbFixeXWLFiPfdn6Giz9kfWUB1W9DkGAABwbVGR15wejl0V4RgAAMC1edQkIAAAAICrIRwDAAAAVoRjAAAAwIpwDAAAAFgRjgEAAAArwjEAAABgRTgGAAAArAjHAAAAgBXhGAAAALAiHAMAAABWhGMAAADAinAMAAAAWBGOAQAAACsf2xeAtzh7+YZcu3U3zI9PkcRPMqVJFqX7BAAAXAPhGF4XjHM36C8PHwWGeZ24cXzkyB+DCcgAAHgByirgVXTEODzBWOnjwzPSDAAA3BfhGAAAALAiHAMAAABW1Bx7MS5MAwAACIpw7KU8/cK0J0+eyvXbd+XqjTty9eYd8++VG/6y9+h5Z+8aAABwYYRjL/UyF6Y5IxxbLBa5ez/AGnT97WHXMfw6/qv7qesAAACEB+EYTvPo8X9h2xZor1z3/3/4DSHwPgx4HK7tx4gRQ5InTiCpkiWUVEkTmn/V7yt2hHtfx85bL92aV5U8WdKEe10AAOA+CMeINDpSe+vO/ReM6v5/+U3/++H+GQni+f4/7FoD7//DbyJJbe4nMvc1GPv4xAqy/s7DZyMUjn+Zu87cCmRPJ42rFZNGVYtKvmzpwr0dAADg2gjHCJcFa3fLis0HQxzZ1RHfwCdPw7W9WLFiSsokfvZA6zjKm1qXOdxPmTShCcfO8Oor2WXrgVOy/8RFcxswZqHky5ZWGlctZsJy/uwEZQAAPAHhGOEyaOziFz4msV+8IKE2VdL/Qq7jqK7t+0kTxZeYMV2/o+APHzeTrOmSy59r98icVTtl+eaDcvDkJRl0cpEMGrtI8mZNa0aTNSjr6LKWdAAAAPdDOEa4lHklm+TMmMoebu2ju9bQmzKpn/jGie2RZzVpogTSps6r5qblIwvW7pHZq3bI8s2H5NCpSzJ43GJzy505tb304pWcGQjKAAC4EcIxwmX0x82laJ5MbnvWUiTxMy3pwtvCTtdzlCRhfGlVu4y53b77QBau2yOzV+6Upf8ckCNnrsgX4/8yt5yZUtlLLwrlIigDAODqYljodxUif39/SZw4sdy+fVsSJUoknkYvTCvWYki419sx9VO3DsdRPfmJvwbl9Xtl9sodJigHOITwHBlT2UsviuTOyIgyAAAumNcIx9F4sl2JN4fj6KJBefGGfab0YsmmA0Fa0WVLn8JeelEsb2aCMgAALpLXKKsAokgiv3jSvFZJc7tz76EJynox318b98nJC9fkq0nLzC1r+hTSqMp/I8rF8xGUAQBwJsIxEA0SJogrzWqWMLe79x/KXxv3m9ILDcynLlyTr6csN7fMaZPbSy9K5s/CiDIAANHM6T20fv/9dylbtqzkyJFD6tevL4cOHXru4+/evSv9+/eXkiVLSq5cuaRVq1Zy/vz5II85evSodOrUSQoVKiTFihWTbt26ydWrV6P4SNzzwrTwCOnCNISfX/y40qR6cZn91Xvy78oRMntYB2lSvZjEjxtHzly6LiOmrpDSbYZJljqfSI9Rs2XzvpNMhQ0AQDRxas3xvHnzpEmTJvLDDz9ImTJlZMSIEbJ06VI5cOCApEyZMsR1XnvtNTl37pz8+OOPkiJFChk1apSsWbNG9u7dKwkSJJAnT57IK6+8Il27djXbvH//vvTs2dPUomzbtk3ixo0bpn3z9JpjdfriNanaaZT5iP+DJpWlTd1XI+3CNITf/YePZMnG/ab0Qi/qu/cgwP69jKmTSkNr6UXpglndojc0AABRzeMuyCtatKi5jRs3ztwPDAyUtGnTygcffCCfffbZM48/c+aMZMmSRZYtWyY1atQwyzQMp0mTRgYOHChdunQxy54+fRokPBw7dsyMMmuIrlSpUpj2zRvCsX6kX/vD0eYj/7OLhpr2ZHANDx4+Mt0utPRCg/Ld+/8PyulTJTGlF42qFpNXX8lGUAYAeC1/T7ogTw9m165d0rt37//vjI+PVK1aVdatWxfiOjoKrBImTGhfFitWLIkfP778/fff9nAcfFTt8eP/ugTEju2Zk1NE1PDJy8y/7zUoTzB2MfHixpH6lYuYmwZlnZFPu14sWLdXLly9Jd/NWG1u6VImkYZVipgR5bKFsj83KEdlCzsAADyF08LxhQsXzL866usoderUsmfPnhDXyZkzp2TLlk2GDRsm06dPN2UUOup89uxZSZcuXYjr6MD4J598YmqaS5QoEer+BAQEmJtjeLeNZuvN02zef0rW7TwmsX1iSZfGFT3yGD1FbJ+Y8ka5Auam7eBWbDkkc9fskoXr9srFf2/JD7PWmFvaFImlXqVCpvOFBuVYsWIGCcb5mwwK9+QnB34fQEAGALisqMgvTgvHWvpgdsAn6C7o6K6WSoREHzt//nx57733JHny5OLr6yulSpWSunXrypUrV0JcR4Px6tWrzchynDhxQt2foUOHyqBBg55ZrqPbGsI9Tf+xq82/NYpllctnT8jls87eI4RV2vgi77+RXzrUyCNbD1+U1btOyfp95+TStdvy85x15pY8UTypWCizVC2SRQrlSC3HL9wIVzBW+vj1/2yV3BlT8OQAAFzSvXv3PCcc2y64u3btWpDlej+0i/FUwYIFZdOmTfLw4UNzS5IkiZQvX14yZXp2YgqtQx49erQsWbLE1DY/T9++faV79+5BRo4zZswoRYoU8bia46Nnr8javf+l4SHdmkr+bCGPusP1vVpG5MO2IgGPHsuqbUdk7updsmDdHrnu/0D+WH/Y3FIlTSjlCmeP0Pbz5s3HpC8AAJdl+6TfI8JxqlSpJHPmzLJx40Z588037cs3bNggderUeeH62nVCb1qesXnzZhk7dmyQ7+sosHa/+Ouvv6RcuXIv3J6OQustpNHq4KPb7u67GWtMuUnt8gWlUC5mu/ME+hqtW7GwuT16HCirth42F/PNX7tbrt68I3+s2R3h7Xra6x8A4Dl8ouBvlFP7QekFdOPHjzc1xlpm8f3335v64Q4dOtgf8/HHH0u1atXs97XG2FaTfPnyZWnZsqUULlxY3n77bftjhgwZIt98840JxjqqjP+7fO22TFr8z3/ntlVNTo0HihPbR14rW0B+G9Bariz/Rpb+0FXerFjI2bsFAIBbcOqQUI8ePeTSpUtSunRp03VCyxdmzpwpefPmtT/mxo0bJgTbaI3xu+++K8ePHzdlFfXq1ZPZs2fbO1Ho4/v162c6WDRv3jzIz/v666+fWeZt9MKtgEeBplduucI5nL07iGJ6wWXNMvklZdKE8ufakC90BQAALhKOte3UyJEjTfcJ7U+nk3rEiBHjmUD76NGjIDXHW7dulVu3bpkAHPwiO61B1klCQpIsmXe3pbpz76H8NGetfdQ4+LkGAADwdi5RTKgBN7SL8JImTRricg3BoQXuDBkyROr+eYpx8zfIrTv3JVem1FK3Ah+zAwAABMcctF7iceATGTV9pfm6Z8vqQXrgAgAA4D8kJC8xa/k2OXflpqROnkhavl7a2bsDN7F4/V7T2QQAAG9BOPYCGm6GT15uvu7WrIrE9WUabYTNZ2MWSoNev8iV65HfRxIAAFdEOPYCy/45IPuOXxC/+L7SsWEFZ+8O3IhPrJgy/+/dUqDpIJm7aqezdwcAgChHOPYCtlHjDvXLS9JEnjcVNl4sRRI/iRsnfNff6uMXjuoir+TMINdu3ZVGvcfI2/3Gy43bkT9VJwAAriKGhYLCUKcjTJw4sWkx587TR287cFpKth5qRgBP/jlEMqbx7nZ23uzs5Rsm5IYnUGdKk8zMuPf52EUydOJSefrUImlTJJZx/VrK6+UKRun+AgDgjLzmEq3cEHW+nvLfqPFbtUoSjL2cBl29RWTGvS8615M6FQpJ6wET5MiZK/LGh6OlXb1yMuLDRpLIL16U7C8AAM5AWYUHO37uqsxd/V+daM+WNZy9O3BzpQpklV3T+slHb1U1E8ho3+xXmg+WNduPOHvXAACINIRjDzZy2krzMfjrZQtIwRzpnb078ADx4saRkd2byJpfukvW9CnkzKXrUqXjSOn2zSy5//D/M1kCAOCuCMce6uoNf5mwcJN9qmggMlUslkv2TO8v7zX4r/vJ9zNXS+G3Bss/e09wogEAbo1w7KFG//63PAx4LCXzZ5EKRXM6e3fggRImiCu/fPK2LP2hq6RPlUSOnb0q5dp9LX1Hz5OAR4+dvXsAAER/OL5w4YKsX7/+ZTaBKHD3/kMZ/fsa+6ix1ocCUaVmmfyyb+ZnZuZFLeMZNnGpFG/5pew6fJaTDgDwjnB848YNqVWrlmTIkEEqVPj/pBK1a9cmLLuA3xZskpv+9yVHxlRSr1JhZ+8OvID2z578eVv54+uOkjJpQtl/4qJpIagt4B4HPnH27gEAELXhuGfPnhInThw5c+ZMkOUfffSRfPHFFxHZJCKJBpERU1eYr3u2qC6xYlE5g+hTv3IROfD7AGlYpagEPnkqA8YslFff+UoOnrzI0wAA8NxJQNKkSSM7duyQ9OnTm4/sbZvQBsxp06aV+/fvi7tz10lApi/damYxS5UsoZxe8KXpLgBEN/0/YcaybdLlqxly68598Y3jI0M615MPm1flDRsAwKXzWoSGFXUHEiZMaL52rGfV5T4+zCvizEAyfPIy83XXplUIxnAa/X9BJ57RUeTXXi0gAY8Cpee3c6Rih29M/20AAFxVhMJxyZIlZfbs2UHC8dOnT01JRdmyZSN3DxFmK7Yckj1Hz0uCeL7SqVFFzhycLl3KJLL4u/fNdNN+8X1l454TUqj5YPl5zlr7J04AALiSCA3zfvXVV1KjRg1Zs2aN+QPXq1cvWb58uRw/fpwL8pzINmrcvl45SZY4gTN3BbDTN9Dv1isnVUvmkbaDJsnfO45K52HTZd6aXTK+fyumNQcAuP/IcenSpWXz5s3i6+srBQsWlKVLl0rhwoVl69atUrRo0cjfS7zQjkNnZNXWw6ae86O3q3HG4HKypEshq37+SL7r2VTi+cY2n3QUaDpIJi7cxCgyAMC9L8jzBu52QV6zvmNl1ort0uK1UjJl8DvO3h3guY6euSKtB06QzftOmft1KxSSMZ+8LWlSJObMAQDc74K8J0+eyO7du59Zrsv0e4heJ8//K7NX7TBf92pVg9MPl5crc2rZMO5jGfZ+fYkT20cWrNtjRpFnr/zvdQwAgLNEKBwPHTpU5s6d+8xyXab1yIheI6etNDOT1Xo1v7ySMwOnH25BS4B6t6kl26d8IoVzZZTrt+9Jkz6/mk9Brt+66+zdAwB4qQiVVejMeFpfnC5dumemk9ZuFadPnxZ35y5lFf/evCOZa/eVBwGPZfUv3aVy8dzO3iUg3B49DpQhv/0lQ35bIk+ePJU0yRPJr5+2kDoVCnE2AQCuX1Zx8+bNIP2NbXTZ1av0MI1OP/7+twnGxfNllkrFckXrzwYii5ZWDHqvrmye0FvyZk0rl6/7S93uP8k7gybJ7bsPONEAgGgToXBcqlQp+fbbb59ZPmrUKNMDGdHj3oMAGf37GvP1x61qhviGBXAnxfNlkZ1TP5WeLaub1/OEhZukYNNBsmrrIWfvGgDAS0SorOKff/6RKlWqSJEiRaRChQqmDdO6devMBXmrV6+WMmXKiLtzh7KK0bPWyAdfz5Rs6VPI0T8GMy0vPMqG3cel9YAJcvLCNXO/S+NK8lXXBmaSGwAAXKqsQsOv1hznzJlTFi9eLEuWLJFcuXKZZZ4QjN1BYOATGTFthfm6Z8saBGN4nHKFc8ieGf2lc+P/Znv8cfbfZna9jbuPO3vXAAAejD7HbjpyPHPZNmn+6ThJmTShnFn4pcSLG8fZuwREmRWbD8o7gyfL+Sv/Xe/Qs0V1+bxjXYnrG5uzDgBezN9VRo5ttKfxrVu3nrkhamkZi22q6A+aViYYw+NVL51P9s38TNrUKWNe/19PWS7FWg4xM0MCABCZIhSODx8+LOXLl5e4ceNK0qRJn7khauk00buOnJP4ceNI50b/feQMeLokCePLhAFt5M8RnSV18kRy8OQlKdVmmAwcs1AeBzL5EAAgcvhEZKV27dqZEPzXX39FShh+9OiRGRZPnjx5mDsuPH78WO7cuSPJkiWL1O26A9uocbt65SR5Ej9n7w4QrepWLCSvFsounYdNNzPqDRq7SBau3yuTBraRAjnS82wAAKJ/5HjXrl0yceJEqV69uhQvXvyZW1g9ffpUevToIUmSJJEsWbKYyUXmzZv33HWOHz9ufq7Wleg6uXPnNp0yXna77mLX4bOyYsshcwHeR29VdfbuAE6RIomf/D6sg8z8sp0kS5xAdh4+K8VafinDJy0zk4gAABCt4Thjxozy8OFDeVkjRoyQCRMmyKZNm8wIb+/evaVp06Zy6NChUGuc69SpY8o5/v33X7NO9+7d5Y033pCLFy9GeLvuRGstVdPqxSVLuhTO3h3AqZrWKCH7Zw2Q2uULmln2ev/wh5Rv/7UcO3uFZwYAEH3h+IMPPpCPPvrIXBn4Mn788UdTolG4cGGJGTOmdO3aVTJlyiS//vpriI8/evSoqXf+5JNPxM/vv3KC9957z5RW/PbbbxHerrs4deGa/L5yh/m6V8sazt4dwCWkTZFYFozsIr991koSJogr/+w9aVq+aR9w/RQJAIAoD8eDBw+W2bNnm1CaOnVqSZMmTZBbWOg002fOnJGyZcsGWV6uXDnTLzkk8eLFM//evXs3yGjy/fv3ZfPmzRHerrsYNX2l+ci4Rul8Ujh3RmfvDuAy9JqCtnXLmo4WVUrkNlOq6wQ51bt8K2cuXXf27gEAPP2CvG+++ealf7CWRagUKYKWBuh9LYcIidYPV6xY0dQTjx492jz2u+++M6UTGoojul0VEBBgbja6TRUYGGhuznbt1l0ZN3+D+brH21VdYp8AV5M+ZWJZ8t378ssf66Xv6PmyetsRKdj0cxnxUUNpU7uMR12YCwCQKMlDEQrHLVq0eOkfrOUOIR2UdqGIFStWqOvphXU6ct2tWzd58OCBNGnSRBo1aiSnT59+qe0OHTpUBg0aFOLFhwkSJBBnG79ktxkNy50xuSSUO7J9+3Zn7xLgskpmSSATP64tg6dukH0nr0qHIdNk4vy10vetspIicXxn7x4AIJLcu3dPXCIcR4b06f9ruXT58uUgy69cuWL/Xki0ddzIkSODLCtUqJAUK1bspbbbt29fc3Gf48ixXnhYpEgRp8+Qd//hI5nfb7b5+rMOb0qJEv8dK4DQad+cujUry7czVslnYxbJpgPnpdVXC+WHXk2lSbVijCIDgAfwt37S7xLhWEsU5syZI2fPnn1mlHb+/PkvXF8Dp14wt2LFCtNJQul2Vq1aZS74s7l586bpV6y1zUovsLGNDttGdvfu3SvDhw8P13aD8/X1NbfgfHx8zM2Zpi7ZYMoqsqZPIU2qFxcfn9BHwAH8n/7q9m7zmtQuX0haDZhgWr616D9B/ly7V37q85ZpCQcAcF8+UZDRInRB3tSpU02vYa3znTt3rqnnPXXqlPz555+mzVpY9e/fXyZNmiRjxowxAfedd94xyzt16mR/TK9evaRq1f/38/3888/N47VzxcKFC6VBgwamzKNmzZrh2q67CAx8IiOmrjBf93i7GsEYiID82dPJ5ol9ZGCH2uITK6aZPCR/k4GyYO0ezicA4OXD8bBhw2TGjBkmJKtx48bJ7t27pWfPnuH6qFKDrW5Dg2z9+vXN0PjatWslZcqU9sdoRwzHDhjaQu7AgQPy5ptvypdffmnu64Qk4d2uu/hjzS45eeGaJE+cwFyNDyBiYvvEkgEd6piQnD9bOrl644682eMnaTNwoty6c5/TCgAwYlgsFouEk44Oa7mDtlaLEyeO3Lp1S+LHjy/Xrl2TXLlyyY0bN8TdaaBOnDix6eXsrJpjfWpKtPpSdhw6a0a89A87gJf3MOCxDBizQL6essL8nmVInVTG929l2iQCALw7r0Vo5Fhbntl6DutFbgcPHrT3H9auEIgca7YfMcE4nm9s6dKkMqcViCRxfWPLV10byoZxvSRHxlRy/spNqfn+d9J52HS5e//lZ/8EALivCIVjR9pGTWt++/XrJ/Xq1TO1yIgcwycvM/+++2ZZLhwCosCrhbLL7un95P0mlcz9n+esNbPrrd91jPMNAF4qQuH4n3/+sX89ZMgQady4sZmhTmehGzt2bGTun9fac/ScLPvnoMSMGUO6v80bDiCqJIjnKz983FxW/vShZEqTzNT4V+wwQnp+O0cePHzEiQcALxOhmmNv4Oya4xb9x8u0JVulWY0SMuPLdtH+8wFv5H/3gXQfNVvG/7nR3M+bNa1MGthGSuTP4uxdAwC4cs0xotaZS9dl5vL/ZsDr1aoGpxuIJon84sm4/q1k4agukiZ5Ijl06pKUeecr+eyXBfLoMVO2A4A3iFA41nSuM8qVL19eChQo8MwNL2fU9JXy5MlTqVYyrxTNk4nTCUSz2uVfkf2zBphPbvR3cfC4xVKq9TDZd/wCzwUAeLgITSvStm1b06GiWbNmkiRJksjfKy92/dZdGTtvg/n6Y0aNAadJnsTPlDQ1qFJEOg2dJruPnpNiLYbI5x3rSs8W1ZmQBwA8VIRqjv38/MxEHJkzZxZP5aya4y/GLZb+vyyQwrkyys5pn4ZrUhUAUePKdX/pMGSqLFj334x6pQtmlUkD20quzP9Naw8A8PKaY90JW59jRB69Mv77WWvso8YEY8A1pE6eSOaP6CQTB7aRRAniyuZ9p6TwW4Pluxmr5OnTp87ePQBAJIpQOO7UqZN8/PHHcv8+U65GpkmL/pF/b96RzGmTS+NqxSJ12wBejr5ZbV27jKlFrl4qrzwIeCwfjvhdqnYaJacuXOP0AoA3l1WcOHFCSpQoYWbES5s27TMjnKdPnxZ3F91lFXrRT+6Gn8mJ8//K9z2bygfNqkT5zwQQMfrf5i9z15leyPcfPhK/+L4y8qPG0q5eOT7xAQA3z2sRuiCvdevWkilTJjMzHhfkRY55a3aZYJwscQJ5582ykbRVAFFBBwQ6NaooNUrnkzYDJ8qG3cdNTfK8NbtlbL8Wkj5VUk48AHjTyLHWGx8/flzSp08vnio6R471KdA2UdsOnpbP2r8hg96rG6U/D0Dkfuqjtcef/DRfAh4FSpKE8WX0x83krVolGUUGAG+5IE9DcaxYsSJlByCydsdRE4zj+saW95tU5pQAbiRWrJjSvUV12TWtnxTPl1lu3bkvLfr/Jo0+HmOuIQAAuJeYES2r6Natm0npeHnDJy83/75T91VJmTQhpxRwQzrV9D+/9ZbBHeuKT6yY8seaXZK/ySBTMgUA8PCyitSpU8vVq1clZsyYkiJFimc+Orx8+bK4u+gqq9h77LwUaj5YYsaMIUf/GCzZM6SMsp8FIHrsOnxWWg+caJ9Rr8VrpeT7Xk0laaIEPAUA4IkX5I0YMSJSfjhEvpmywpyGRlWLEowBD1EkTybZNrmvDBq7SL6atEymLtkiq7cfkfH9W0qtVws4e/cAAJEdjgMDA6VNmzYRWRUOzl6+ITOWbTVf92pZg3MDeBDfOLHlyy71pW6FQtJ6wEQ5evaKvNb1B+lQv7x882EjSZggrrN3EQAQWTXH7du3Z1aoSPDt9JUS+OSpVCmRW4rnyxIZmwTgYkoXzCa7pveTbs3/613+67z18krzz82FuAAADwnHefLkkV27uMjkZdz0vye/zttgvv64Vc2X2hYA1xY/bhz5tkdTWf1LdzMD5umL16XSeyPkoxG/m2njAQAeMH108+bNZebMmbJ37145fPhwkBte7Oc5a+XegwB5JWcGM5EAAM9XuXhu2TfzM2lfv5y5/+2MVVLk7S9ky/5Tzt41AMDLdKsI3p0iuAhs0qu6VTwMeCyZ6/SVqzfuyNTB78jbr5WK1O0DcH1LNu6XdwdPlkvXbptuNX1a1zKTAGmtMgDAzbpVnDt3LlJ+uLeavPgfE4wzpUkmTaoXd/buAHCC18oWkP2zBkjXb2bKtCVb5csJS2TRhn0yeVAbKZQrI88JALjTyLE3iKqRY51qNm/jAXLs7FX5tkcT6da8aqRtG4B7mrtqp3QcOk2u3borsX1iyYD2taV365ri48NMpADgFtNH2+zevVtmzJgh06dPN1/jxf5cu9sE46SJ4su7b5bllAGQhlWLyoHfB0i9SoXlceAT6ffzn/LqO8Pl8Gn3n1AJANxNhMLx9evXpUaNGlKkSBF55513pF27dubrmjVrmu8hZDpIrxMCqC6NK4lffPqcAvhPqmSJ5I+vO8qUz9tKYr94su3gaXOx3qhpK2mdCQCuHo67du1qhq+1nduDBw/k/v375utbt25Jt27dIn8vPcT6Xcdk64HT4hvHRz5oWtnZuwPAxejFzi1eL21qkWuWyWcu3u0+arZU7jhSTp7/19m7BwBeIUI1x0mSJJHt27dLjhw5giw/duyYlCxZUm7evCnuLipqWGp/OFoWb9gnHRtWkJ/7vh0p2wTgmfS/5rHz1kv3UXNM28cE8XxlxIeNpEOD8i/sGAQA3sLfVWqOHz16FOIO6LKAgIDI2C+Ps//4BROM9Y9ajxbVnb07AFyc/l/RoUEF2Tujv1QomtMEZL1or9YH38v5K+4/AAEAripC4bhcuXLSs2dPuXfvnn3Z3bt3pUePHlK+fPnI3D+P8c3UFebfhlWKSI6MqZy9OwDcRLYMKWXNL91lVPfGEtc3tizffFAKNB0kUxZv9oie8gDgEWUVhw4dklq1apka4/z585tlBw4cMOUWS5culbx584q7i8xheh3lyVr3Ewl88lS2TuorJfJnibT9BOA9tHtF6wETzLULSrtb/NL3bUmdPHInKgIAby6riHCf44cPH5rpozUU68d/+fLlk2bNmkncuOHrwPD06VPZsmWLXLlyRQoUKPBMHXNI9CLAbdu2mdrmTJkymU4ZEXlMdJ3snt/OkRFTV0ilYrlkzZgeL7UtAN4tMPCJDJ+8TAb+usi0fUuRxE9+7vOWNKpWzNm7BgDeFY41uO7fv998/cUXX0i/fv1e+ofrgegI9NmzZ0243rRpk3zwwQcybNiwUNdZu3atNGrUSNKlSydZsmQxwTpz5sxmxDpp0qRhfkx0nexbd+5Lxjf6yN37AfLXdx+YWbEA4GXtOXpOWg+cKHuOnjf336pVUn7o1UySJU7AyQXgNfydGY59fX3lzp07EidOHDNSHBm1bu+//76sWLFCtm7dag5sw4YNpmZ55cqVUrVqyDPHFStWzIwuz5o1y9zX0o6cOXOa9nL9+/cP82Oi62QPm7hU+o6eJwWyp5O9Mz/jKnMAkebR40D5fOwiGTpxqTx9apG0KRLLuH4t5fVyBTnLALyCfxSEY5+wPrBgwYLy7rvvmlZtavTo0c8NvS+i4XratGnSt29fc1C2C/10+1OnTg01HGs3DB0NttE652TJkpkOGuF5THTQHqXfzVhlvv64VU2CMYBIFSe2j3zRuZ7UqVDI1CIfOXNF3vhwtLSrV860fUvkF48zDgBRFY4nTpxoSinGjBkTKeH4/PnzZkRXyzWCh/DnTUU9atQo6dChg8SPH9+USixfvtwEXx0VDs9jgtNA7diGTt+JqMDAQHOLiEmLNsnl6/6SMXVSaVS1SIS3AwDPUyxPRtk2qY/0/2WhfD9rjYybv0FWbD4o4/q3NNc6AICnCoyCbBXmcKwhdv78+eZrLas4fPjwS/1gHf5WGlodJU+e3ITm0OiIcPbs2WX27NmSLVs2E6QbNmwYZCg9LI8JbujQoTJo0KBnluvMfwkShL+GTz/i/HL8IvN1/bI5Zc/uXeHeBgCER7NymSVX6loyZOp6OXP5hlTv8p00qZRXOtUtLnHjhPm/ewBwG45thSNLhP63/Pnnn1/6B2sNs60/siO9H1rHC+1s8frrr0vFihVNXbItZGsnCh8fH/n666/D9JiQaHlH9+7dg4wcZ8yY0awXkRqWP9fukbNX/SVJwngysEtTSZggfF08ACAiihcXaV63mvQePU/Gztsgv/99SHaduC7jP2spZQpm46QC8Cj+1k/6nR6OtaOEli3EjBmhOUQMba+mYVU7VTg6c+aMGe0NyYULF+T48eNBSjq0XrlGjRqyZs2aMD8mtLBuC+yOdB/1Fh5aT22b9KNzo0qSNLFfuNYHgJeh/+f8+mlLaVilqLw7eLIcO3dVKr03Unq1rCGD3qsjvnFic4IBeASfcGa0sIhQus2TJ48pN3gZGkT1ojstfbC5du2arF69Wt544w37Mu1VrC3YVOrUqSV27NhmEhJHej9DhgxhfkxU27jnhGzed0p84/jIB00rR8vPBIDgapbJL/tnDZCWr5c2pV5fTVomxVt+KbsOBx2UAAD8X4TidqdOnaR58+by+eefm/7E2t4teHgOC+1nrB0qWrZsKWXKlJFx48aZ2fXatGljf4xeALh582bTD1l/Tp8+fcyFgRqkdYRZL7bT72tvYxWWx0S14ZOWmX9bv1FG0qT4rxMHADhDkoTxZfLnbaV+5cLy3pfTZP+Ji1Ky9VDp3+4N6dv2NYntE4snBgBedoY8vSDvecKzyaNHj5pQrDPkaacKDd6OF8D9+uuvcuzYsSC1wtobedmyZXL9+nXTjaJ169aSNWvWINsNy2Oiom/ewZMXJX+TQf9dtDhnkOTKnDrM6wJAVPr35h3pNHS6zF2909wvljeTTB7UVvJlS8eJB+CW/F1l+mhtw/Y80VW+4Ion+51Bk2TCwk3SoHIRmft1xyjdRwAIL/0vf8aybdLlqxlmBk8t//qi05vy0VvVJFasiF9HAgBeHY69QURO9oWrNyVr3U/lceAT2Tyxj5QqEPaRagCIThf/vSXtBk+RJZv2m/tlC2WXiQPbSI6MqXgiAHh1OI7wMIH2ItYZ7gYPHmxfphfpaSs1b/XdjNUmGFcompNgDMClpUuZRBZ/976Zbtovvq+5kLhQ88Hy0+y/vfr/cQCI0Mjx/v37pXr16hIvXjw5deqUvcZYp5fW/sKtWrXyuncit+8+kIxv9JE79x7Kom/flzfKFYyW/QSAl3X64jV55/PJsmb7EXO/Wsm88ttnrSRjmqCTNAGAq3GZkWOdLKNjx45y8uTJZ6aNHjlypHijMXPXmWCcP1s6ee3V/M7eHQAIsyzpUsjKnz6U73s2lXi+sWXl1kNSoOkgmbhwU7gusAYArx051oR+7tw5k9C1K4NtEzqFn04HHRAQIN70TiTg0WNTa3zp2m1Ts9e6dplo208AiExHz1yR1gMnmF7tqk75V+TXT1vQlhKAS3KZkWOdGe/+/fvPtHXTlmtJkiQRbzNtyVYTjNOnSiLNa5Zw9u4AQIRp+8kN4z6WYe/XlzixfWTh+r2Sv+kg+X3Fds4qAK8QoXCsE3J88cUXZsTYFo4vXrwoXbp0kdq1a4s30QtXvp6y3HytrZD0jwkAuDNt6da7TS3ZPuUTKZwro9y4fU+a9h0rzfqOleu37jp79wDA9cLxiBEjZOXKlWZyDQ2HJUqUMDPR3bhxQ4YOHSreZNH6fXL49GVJ7BdP2tcr5+zdAYBIUzBHetkyqY981v4NE5hnrdhuapEXrtvDWQbgsSLc5/jhw4cye/Zs2b59uwnIRYsWlWbNmpkOFt5Uw1Lu3eGmBVKfNrVk6Pv1o3UfASC6bD94WloNmCiHTl0y99vWeVVG9WhiBgYAwFmYBMTFTvbG3celXLuvTSnF6YVfStoUiaNzFwEgWj0MeCz9f/lTRkxdacrqMqZOKhMGtJaqJfPyTADw7gvybFNIf/rpp9KgQQNz69evn1y4cEG8ia3WuNUbpQnGADxeXN/Y8nW3RrJubE/Jlj6FnLtyU6p1/lbe/2qG3Hvg/l2KACDCZRXLli2TN998U3LmzCnFixc3y7Zt2yYnTpyQBQsWmAlCPOWdyNpt+8XPL+Ez3z918Zo0+niM+Xru8PdMn9AUSfwkE03zAXiBu/cfSu8f/pCfZq8197NnSCmTBraRsoVzOHvXAHgR/ygYOY5QOM6bN6+pLx4wYECQ5YMGDZJZs2bJwYMHxVNOthRuIxIrTpjWiRvHR478MZiADMBrrNh8UN4ZPFnOX7lpuhf1bFFdPu9Y14wyA4DXhOP48eOb1m3BexrfunVL0qdPbyYD8cZwrHZM/VSK5skUpfsGAK7k1p378tHI32Xiwn/M/XzZ0srkQW2lWN7Mzt41AB7O31VqjnXkOKTR4QMHDkiePHkiY78AAG4iScL4MmFAG/lzRGdJnTyRHDx5SUq1GSYDxyyUx4FPnL17ABD14bhNmzbSqFEj+emnn0yt8datW83XjRs3lnfeeUcOHz5svwEAvEPdioVk/6wB0qR6MXny5KkMGrtISrUeKvuPe9fF2gDcW4TKKhynjH6RCLZRdjrKKgAg4mYt3yadv5phZtfTdpeDO9aVHi2qm8lEAMCVyyoiNNfxuXPnIuWHAwA8U9MaJaRC0VzSYcgUM5OodraYv3a36WiRM1NqZ+8eAERuOM6QIUNEVgMAeBGdGGnByC4yceEm6Tbid/ln70kp1HywDO/aUDo3rigxYzKKDMD18D8TACDKaBle27plTS1ylRK55UHAY/ng65lSvcu3cubSdc48AJdDOAYARDmdIGnFjx/K6I+bSfy4cWT1tiNSsNnn8tufG9322hQAnolwDACInj84MWNKlyaVZff0fvLqK9nlzr2H8u7gyVLnox/l0rXbPAsAXALhGAAQrfSCvHVje8rwrg1MJ4vFG/ZJ/iYDZeaybYwiA3A6wjEAINppS7derWrKTuusojf970vzT8dJ075j5dqtuzwjAJyGcByJ4sbxkRRJ/CJzkwDg0fJnTyebJ/aRgR1qi0+smDJ75Q4zirxg7R5n7xoALxWhSUC8qan02m37xc8vYZjW0WCsF50AAMJvx6Ez0nrARDlw8qK537p2Gfm2RxMzPTUARNckIITjaDzZAIDnexjwWAaMWSBfT1lh6o8zpE4q4/u3khql83HqAERLXqOsAgDgMuL6xpavujaUDeN6SY6MqeT8lZtS8/3vpNPQaXL3/kNn7x4AL0A4BgC4nFcLZTct395vUsnc/2XuOjO73vpdx5y9awA8HOEYAOCSEsTzlR8+bi4rf/rQXM9x8sI1qdhhhPT8do48ePjI2bsHwEMRjgEALq1qybyyb+Zn8u6bZU0d8oipK6RoiyGy7cBpZ+8aAA/k9Avyrly5ItOnTzf/FixYUJo2bSo+Pj7PXWffvn2yZMkSuXnzpmTKlMmskyzZs10i1q1bJ6tXr5b48eNLs2bNzGPDigvyAMD16IQh7QZPlsvX/U2v5L5takn/dm+YyUQAeB9/T7sg79ixYyYQL126VGLHji0DBgyQWrVqyZMnT0JdZ8yYMVK8eHE5ceKEOQlz5syRHDlyyJEjR+yPefr0qbRs2VIaNWokDx48MLe6devKoUOHounIAABR4Y1yBWX/rAHSvGYJefLkqXwx/i8p1XqY7Dt+wXz/7OUbsvPw2TDf9PEA4DIjxw0aNJBr167J33//LTFjxpSzZ8+aoDt+/HgTbkOSJ08eqVmzpnz33XfmvgZpXUdHj4cNG2aW6fc+/fRT2bNnj2TPnt0su3v3rty/f19SpUoVpn1j5BgAXJtOGKJdLK7fviexfWLJR29Vle9nrpaHjwLDNXnTkT8G06MecFP+njRy/PjxY/nrr7/krbfeMsFYadlDpUqVZP78+aGup+FWg67jdgICAiRNmjT2ZT/99JO8/fbb9mCs/Pz8whyMAQCur3G1YnLg94FSt0IheRz4RIZPXh6uYKz08UxXDcCR04q0dJRYQ61jgFV6f+PGjaGu99tvv0nHjh3ltddek8yZM8uWLVvMqHHnzp3t7yCOHj1qRo615GLHjh2SLl06adiwofk3NLoverPR7ajAwEBzAwC4nuSJ48ucr9rL1CVb5YPhM+VeBLpY8P884L4CoyCjOS0ca4mDSpgw6NTMOiRu+15INPgePnzYlFboSLPWGm/evNmUZ2j4tYXar7/+2ny/bNmysnLlSvnkk09kxYoVUrp06RC3O3ToUBk0aNAzy3ft2iUJEiR4yaMFAESlvKliyedtK0iPn1eGe91Dhw7K07tXo2S/AESte/fueU7N8enTpyVr1qym64RehGfToUMH2bZtmwmlwT169MiUT/To0cOMDCvd/VKlSknu3LllypQppuYkSZIkUqVKFVm1apV93TfeeEMePnwYZNmLRo4zZswo169fZ/poAHADeoFdqTZfhXu9LRN7S9E8Ye9mBMB1aF5Lnjx5pNYcO23kWEd1tQ5YR4Edw7Hez5cvX4jraLs3bd9WokQJ+7IYMWJIsWLFTKBWWpSdPn16s8yR3tfwHBpfX19zC07byr2otRwAwPki+n81/88D7ssnCjKa0y7I04vwtA544sSJZkRX7d2719QbN2nSxP64WbNmyVdf/TcSoKFXR4WXL18eZDR57dq1kj9/fvuy5s2by5o1a+wt4bS1m/Y7LlSoUDQeIQAAANyNU1u5Xb58WSpWrChx4sSRIkWKmO4VWv4wadIk+2PatWtnaor3799v7v/+++/Stm1bKVOmjGTLls0EYy3G1nZwWgZhG2KvXr266W+sNcY6qqzD7Vp7rOuEBa3cAMD9yiqKtRgS7vUKZE8nzWqUkDoVXpGCOdKbTyQBuIeoyGtOnyFPA6zWHdtmyCtXrlyQ72ug1RDdokUL+zJ97Pr16009sHasqFq1qplExJEGZr0A78yZM6aEQx8TUtlEaAjHAOAd4dhRpjTJpE75V0xQrlQsl/jGCfq3BYBr8chw7KoIxwDgHeH403dekz3HzsvKrYflYcBj+/IE8XylRum8JizrzHypkkXOH14Arp3XuNIMAODVGlQpKl90rif3Hz6S1dsOy8J1e2XRhn1y8d9bMm/NbnPTUotSBbKYoFy7POUXgCcjHAMAICLx48YxwVdv+qGqjkRrUF64fq/5evO+U+b26U9/Sua0yaV2uYKUXwAeiLKKUFBWAQDu5ezlG5K7Qf9wTSEdN46PHPljsKk1fp4LV2/K4g37TFAOXn7hF99XapTKJ7XLF6T8Aohm1By7+ckGAER9QL52626YH58iid8Lg3FwWn6xaushE5QXrd8nl67dtn/PsfyiToVCphMG3S+AqEM4jkaEYwDAi2gf/V1HzgUpv3BE+QUQtQjH0YhwDAAILy2/0NHkRRtCL7/QNnGvly1A9wsgEhCOoxHhGADwMii/AKIe4TgaEY4BAJFZfqElFzqqHFr5RZ3yBU2nDCYfAcKOcByNCMcAgKguv9CgvGob5RdARBGOoxHhGADgCuUXpQtmtU8+QvcLICjCcTQiHAMAnFV+YQvKwcsvsqSzTT5SSCoWzSm+cWLzJMGr+UdB610mAYnGkw0AQGSWX9Qsnd/e/SJl0oScXHgdf8Kxe59sAABepvxi5RZb+cVeuXzdP8TyC73lZ/IReAl/wrF7n2wAACK7/EInINGJSIKXX9jqlCm/gCfzJxy798kGACAqnL+i5Rd77eUXAY8C7d+j/AKezJ9w7N4nGwCAqHbvQYCs2no41PKLMgWzSe3yBSm/gEfwJxy798kGACC6yy92HLJ1vwi9/EJvFeh+ATfkTzh275MNAICrll8kTBBXapTKR/cLuBXCsZufbAAAXKn8QrtfLNqwL9TyC20Tp32V6X4BV0U4dvOTDQCAq5dfaPeL3UeDll9kTZ/iv8lHtPtFsVwSJ7aP0/YVcEQ4jkaEYwCAtzp3+YYs3rAv1PKLmqXzmTZxTD4CZyMcu/nJBgDAXcsvzEV9G/bJlVDKL3RUOV+2tGYZEF0Ix9GIcAwAQFCUX8DVEI7d/GQDAOBp5Re2C/pCK7/QEeXXyxWUFEn8nLqv8Ez+tHJz75MNAICnovwCzkA4dvOTDQCAt5RfbD94xowqh9b9wnHyEbpfIKIIx9GIcAwAQOSWX2hQXr2d8gtEHsJxNCIcAwAQNeUXK7T7xbq9snhj0O4XMWNau1+Uf8W0iqP7BV6EcByNCMcAAERP+YWZfGT9Xtlz9HyQ71N+gRchHEcjwjEAANHrrJZfaD/l9fueKb9IpN0vyuQ3M/XR/QI2hONoRDgGAMB57t5/KCu3Hn5h+YVOQJI3K5OPeCt/T2zltm7dOvnxxx/lypUrUrBgQfnkk08kbdq0oT4+MDBQJk6cKH/99ZfcvHlTMmXKJO3bt5dy5cqF+Php06bJDz/8IM2aNZMPP/wwzPtFOAYAwHXKL7Zp94tQyi+yafeLCq9I7XJ0v/A2/p4WjtesWSM1atSQ3r17S5kyZUyIPXr0qOzZs0cSJkwY4jrdu3eXKVOmyPDhwyVLliyydOlSGTFihKxYsUIqV64c5LFHjhyRatWqiR5io0aN5Ntvvw3zvhGOAQBw7fILDcqrtx2RR4+fLb8wk4+ULSDJmXzEo/l7WjguW7asZMyYUWbOnGnu379/34waf/bZZ9KjR48Q18mWLZu8/fbbMnjwYPuyAgUKmJA9cuRI+7KAgAApVaqUfPrppzJkyBCpVKkS4RgAAA8uv1i0Ya9cvXEnSPnFq69kN3XKlF94Jv8oCMcxxUk0CG/evFnq1KljXxY/fnwz0rty5cpQ1ytevLhs27ZNHj9+bO6fPXtWzp07JyVLlgzyOA3XhQsXlsaNG0fhUQAAAGfyix9X6lUqLOM/ayWXlg6XzRP7yKfvvCaFcmWQp08tsmH3cekzep7kbzJIctTrJx+OmCWrth4KMtoMOPIRJ9FAqzVE6dKlC7Jc769atSrU9SZMmCDvvvuupE+f3jz29OnTpsRCa4pt5s+fL0uWLJHdu3eHeX90pFlvju9EbDXOegMAAK6vWJ6M5jawQ21TfrF4w35ZvGGfrNlxVE5euCbfzVhtblp+UaN0XnmjXEF57dX8kjyxn7N3HREQFRnNaeHYNvLr6+sbZHm8ePHs3wuJXryntcpaKpE9e3ZZvny59O3bV4oVK2ZGlTV0d+jQQf78889Q65ZDMnToUBk0aNAzy3ft2iUJEiQI17EBAADXUCJLfCmRpZR83LiobD18UTbuP2duN+88lDmrdplbzBgxpGC2VFKuQEYpVzCjZE6dWGLEiOHsXUcY3Lt3TyKb02qOL168aEZ/Fy5cKLVr17Yv11HhAwcOmJKL4O7cuSPJkyeXn376Sdq1a2dfruvrKLR2sNDw3KdPH8mfP7/9+/v27TN1KJkzZ5aNGzdKrFixwjRyrPXQ169fj7QaFgAA4DrdL3RUWae13nf8wjPdL3REuXa5AlKucA6JE9tpY4l4Ac1rmg0js+bYac+2lkSkSZPG1A87huMtW7ZI+fLlQ1zn7t27ZlRZQ3Xwbe3du9d8rV0pdBTZUatWraRo0aKmlVtIwdg2gh18FFv5+PiYGwAA8BxlC+c0ty/fry9nLl03pRe27hdafvHDrDXmpuUXtV79r/vFa6/S/cLVREVGc2q3Cu0koT2Lt27dagLv3LlzzQV0GpBLlChhL3fYv3+/6VescubMaUaFZ82aZcKs1hxrVwoNwF9//XWIP0cvzKNbBQAACEv3ixVbDtknHwmp+4Vt8pE8WdJQfuGB3SqcOiSqLduOHTsmOXLkMCUM58+fl9GjR9uDsTpx4oTpe2yjoViDsI4Wa6DW9XXkeeDAgU46CgAA4EndL+pXLmJutvKLhev2yML1+2TvsfOm+4Xeev/wh2TPkNIE5drlC0qForkktk/In07DvTh9hjxb/bHOkKchOfhFdCdPnjTlFK+88op9me7yhQsXTD2wzpCXNGnS527fseY4rJgEBAAAONLyC9vkI2u2H31m8hHKL6Kfx00C4soIxwAAIKLlF2UL6eQjlF9ENcJxNCIcAwCAsNDyi60HTpugrKPKwbtf2MovtE65fJGclF9EIsJxNCIcAwCAiKD8IvoQjt38ZAMAAO9y556WXxw0I8raV/nfm8+WX/x3UV/Yu1/ozH/Xbt0N8z6kSOInmdIkE0/kT82xe59sAADgvZ480e4XL1d+ocE4d4P+8vBR2KdNjhvHR478MdgjA7I/4di9TzYAAIDN6YvXZNH6fbJow7PdLxL7xZNaZfKbNnGOk4/sPHxWirUYEu6TuGPqp1I0TyaPO/n+ntbnGAAAwFtlSZdC3m9a2dxCKr+YtWK7uTmWX2TLkNLZu+3xCMcAAABOljBBXGlQpai5afnF1gOnzKiyrfxi/a7j5oaoRzgGAABwIbFixZQyr2Q3tyFd6tnLLzQor95+RAIDnzh7Fz1aTGfvAAAAAF5cfrFsdDdZ/fNHnKooRjgGAABwEwni+Tp7Fzwe4RgAAACwIhwDAAAAVoRjAAAAwIpwDAAAAFgRjgEAAAArwjEAAICbSJHET+LGCd80Ffp4XQ9hwyQgAAAAbiJTmmRy5I/Bcu3W3TCvo8FY10PYEI4BAADciAZdwm7UoawCAAAAsCIcAwAAAIRjAAAAIChGjgEAAAArwjEAAABgRTgGAAAArAjHAAAAgBXhGAAAALAiHAMAAABWhGMAAADAinAMAAAAWBGOAQAAACvCMQAAAOAq4XjUqFGSOXNmiRs3rpQoUUI2bNjw3MffuHFD2rVrJ+nSpZN48eJJ7ty5zTYcrV27VurUqSPJkyeX1KlTS8OGDeXYsWNRfCQAAABwd04Nx+PHj5dPP/1UfvzxR7lw4YJUqVJFatWqJWfPng11nfbt28vmzZtl1apVJigPGzZMevfuLVOnTjXff/LkiQwYMEA6duxoAvGOHTvMsmrVqsndu3ej8egAAADgbmJYLBaLs354njx5pEaNGvL999+b+7orGTNmlJYtW8rQoUNDXCdr1qzSunVrGThwoH1ZoUKFTLAOPoJso2FbR6dXrFhhQnJY+Pv7S+LEieX27duSKFGiCB0fAAAAok5U5DWnjRzrqO+RI0ekUqVK9mUxYsQw9zdt2hTqes2bN5e5c+eadR8+fCgLFiyQU6dOSaNGjUJd599//zX/6skDAAAAQuMjTnL58mXzb8qUKYMsT5UqlWzbti3U9QYPHmzCsI46qzhx4siYMWOkbNmyIT4+MDBQunfvLkWKFJFixYqFut2AgABzc3wnYltfbwAAAHAtUZHRnBaOQ6OlFTqCHJpWrVrJwYMHZe/evZI9e3ZZvny5vPXWW+Ln5/fM6LFuSy/e01FmvdAvZszQB8q1jGPQoEHPLN+1a5ckSJDgJY8KAAAAke3evXueU3OsZRXaTUJLJBo0aGBf3qJFCzl37pzpOBHctWvXzEjznDlzTAcKG61RPn36tKxfv96+TA+rQ4cOpuxizZo1ki9fvufuT0gjx1r/fP36dWqOAQAAXJDmNc2TkVlz7LSR42TJkpk2bBpcbeFYA+3ff/9twm5IYsWKZf4NPrKs69m+Z7uv3So0GK9evfqFwVj5+vqaW3A+Pj7mBgAAANcSFRnNqa3cevbsKb/99pssXrzYjCT36dNHbt26ZYKtjZZFFChQwHydNGlSc8Gelj/s379f7t+/L/PnzzcjyY6jz126dDHLNRjnz5/fKccGAAAA9+PUIVENvjoc3qlTJ7ly5YoULFhQli5datquhWbWrFnSt29f0w9ZSx70sUOGDJEPPvjAXnrx888/m69todpm7Nix5mcCAAAALtfn2JXR5xgAAMC1eVSfYwAAAMDVEI4BAAAAK8IxAAAAYEU4BgAAAKwIxwAAAIAV4RgAAACwIhwDAAAAVoRjAAAAwIpwDAAAAFgRjgEAAAArwjEAAABgRTgGAAAArAjHAAAAgBXhGAAAALAiHAMAAABWhGMAAADAinAMAAAAWBGOAQAAACvCMQAAAGBFOAYAAACsCMcAAACAFeEYAAAAsCIcAwAAAFaEYwAAAMCKcAwAAABYEY4BAAAAK8IxAAAAYEU4BgAAAKwIxwAAAIAV4RgAAACwIhwDAAAArhSOz507J9u3bxd/f/8wPd5iscjJkydlx44dcvXq1UjbLgAAALybU8Pxw4cPpWHDhpI7d25p2bKlpEmTRn744YfnrrN3717Jly+fvPrqq9KhQwfJmjWr2caDBw9earsAAACAU8PxoEGDZOvWrXLixAk5dOiQTJ8+Xbp27SpbtmwJdZ1OnTpJ5syZ5fz582bkWNdbtWqV/Pjjjy+1XQAAAMCp4XjChAnSrl07SZs2rblfr149KVCggFkemn///VdKlCghPj4+5n6mTJkkQ4YMZvnLbBcAAAD4L2E6wcWLF+XKlStSrFixIMtLliwpu3btCnW9zz//XHr27ClZsmQxI8jLly+X+/fvS+fOnV9quwEBAeZmc/v2bfPvjRs3JDAwMMLHCQAAgKhhu65Mr0dz+3CsoVMlT548yHK9b/teSKpUqWKCbt++fc2IsV6Y16dPHzOC/DLbHTp0qCnHCE5rmgEAAOC6rl+/LokTJ3bvcBw7dmz7xXOO9MK6OHHihLiOvit47bXX7DXH+jjtSKFhWUd3+/XrF6HtKg3b3bt3t9+/deuW+Tlnz56NtJPtLu/AMmbMaM5rokSJxFtw3Dzf3oDXOa9zb8Dr3Lte57dv3zYDpMmSJYu0bTotHGsAixkzply4cCHIcr1vGwUOTksmdu7cKV988YU96Op26tatKwsWLDDhOCLbVb6+vuYWnAZjb3qR2egxc9zeg+fbu/B8exeeb+/irc93zJgx3f+CvPjx45t2bBpqbe7duycrV66U6tWr25cdP37cXius7wr04HXU2JGOcqZMmTJc2wUAAABcZuRY6QiwBlYtaShTpozpRZwqVSrTv9hm2LBhsnnzZtm/f7/EixdP2rdvL5988ok8efJEsmXLZi7IW7p0qSxevDhc2wUAAABcKhxXrFhR1qxZY3oUa1/iggULypQpU8TPz8/+mJw5c8qjR4/s9/WxpUqVkmXLlsncuXNNXfCmTZukdOnS4drui2iJxYABA0IstfBkHDfPtzfgdc7r3BvwOud17g18oyCvxbBEZu8LAAAAwI05dRIQAAAAwJUQjgEAAAArwjEAAADgChfkOZOWWh88eNBMHpI/f37x8fGJknVcTUSPQdfRiVG0TZ47ishx67TkR48eNW0C06dPL95y3I8fP5YjR46Yixt0hkhvep2r06dPm3aRBQoUkCRJkognH/f27dufmTBJZx7NkiWLeMPzfefOHTl27Ji5sDv4rKqedtw6McbevXtD/F7evHnd6vgj8nzr61xn1NV1tNNVeC7Qd/fj1pa42gZXn+cYMWKIOzoYzgzy0nnN4oWOHDliyZMnjyVVqlSWjBkzWtKnT2/ZtGlTpK/jaiJyDNOmTbOUKFHCkjRpUouvr6/FHYX3uC9fvmxp27atJXHixJbChQtbkiVLZilTpozlxIkTFk9/vgcOHGgeX6hQIUuGDBnMOgsXLrS4k5f5Xb1y5YolXbp0epGyZcmSJRZPP+7MmTNbcuTIYSlbtqz9Nnr0aIunH/eTJ08sffv2tcSPH9+81vU89OzZ0+LJx713794gz7Pe9LnX1/ratWstnvx8T5w40fw/nitXLkuBAgUsCRIksAwdOtTiTiJy3GPGjDF/x/Lly2fJlCmTWf/QoUMWdzItAhkkMvKaV4bjIkWKWOrUqWMJDAw099977z3zB/HBgweRuo6ricgx9O/f37J582bL2LFj3TYch/e4t23bZpkwYYLl8ePH5v7du3ctVapUMQHZk49bH/fVV1+Z41VPnz61fPzxxyZAPHz40OIuIvq7qsdbq1YtS+/evd0yHEfkuDUU6u+2O4vIcffr18+SPHlyExhtz/2PP/5ocSeR8TepVatWlmzZspnj99TjvnnzpiVWrFiWr7/+2r5s5syZ5nfc9vx76t+xmDFjWmbMmGHu63P84YcfmqCsbw7dRf8IZJDI+N3wunC8c+dO80uhJ9vm3LlzlhgxYljmzZsXaeu4mpc9BncNx5H13OnIg/5HYwvM3nLcf/31l9mOjqZ7+nHrG4Pq1atbLl686HbhOKLHreF42LBh5g+puzzHL3vct27dssSLF88ycuRIi7uKjN/v27dvmze+X375pcWTj1s/8dN1NmzYEGQdXbZ69WqLpx738OHDzWi5o6NHj7rdJwXhzSCR9bfP6y7Is01FXbRo0SA1dmnTprV/LzLWcTWecAzOPO5t27aZukR3qb99meM+deqUrF+/XmbOnCm9evWSrl27SurUqcWTj1snCxo1apRMnDjRLWvyXub5HjJkiLRr106yZ88uFSpUMHWZnnzcOmnUgwcPpE6dOnLx4kXzOK3H9bb/12bMmGEm2GrTpo148nFrfbHOjtu9e3eZP3++/PXXX/Luu+9K7dq1zYRhnnrcyZIlk7t37wZ5bV+4cMH8u2PHDvFUuyLpb77XheMbN25IokSJJHbs2EGW68UI+r3IWsfVeMIxOOu4//77bxkzZoz0799fvOG4Fy5cKL179zZ/TPQijrfffls8+bj1j0fz5s3lp59+knTp0ok7iujzrbNKXb9+XXbv3i1nzpwxyxo3bixPnjwRTz1uDcT6BkhnUC1evLgJh/rmT1/z3vT/2vjx401A1NDg6cfdtm1buXfvnnmzr7dDhw5Jly5dzP9vnnrcDRs2lFSpUkn9+vVl0aJF5s3Q+++/L/HixeNvfhi4xysjEumLK/jV2UpHEuLEiRNp67gaTzgGZxy3vsOuV6+edOvWzfwH6w3HrSPFOrqmowz6H2yVKlVMoPDU4+7Tp48JxdqVZMOGDWYUWR04cED2798vnvx862va9gdX/9B++eWXsnPnTtOlxZP/P9eSwkuXLsnZs2dlz549smrVKhk5cqTMmjVLvOH/tX379plPw9q3by/uJCLHrW/6KleuLO+9957pTKK/16NHjzZvDLZs2SKeetzaaUe70RQrVsy8Efzjjz9k3LhxEjduXBOQPVXsSMo6XheO9aNx/Sjp2rVr9mU6SnLlyhXJlClTpK3jajzhGKL7uDUkVK9e3QSIb775RtxJZDzfOrrWs2dPM+Kybt068dTjTpgwoQlLGpL1pmUGasKECWZ0zZt+v23lM7aPXz3xuG1t6vSjdVuZlLaHeuWVV0w5kTc83/q6zpgxo9SqVUvcSUSOe+XKlaY9ZadOnezL6tata9pzLl68WDz5+dbf5+HDh8uSJUtk9uzZZiT55s2bpk2lp8ocWVnH4mWuX79uiRMnjmX8+PH2ZcuXLzcF3AcOHLAv27p1q+XUqVPhWsfTjtsTLsiL6HHv2rXLXMzQtWtXizuKyHHbulQ42r17t1lH1/WG17m6dOmS212QF1nPt7Zx0wtPz58/b/HU49bOK0mSJDEdaWz0Qltt96StDD39da7Hr506BgwYYHE3ETnuP//803z/7Nmz9u/fu3fP4ufnZ/n+++8tnvx8379/P8h2tF1hmjRp3KrLVlgySFTkNa8Lx7bWIPqfo55s7aGn/Vy1pY0j/Y+yR48e4VrHE4/78OHDlvXr15v2VvqC06/15u/vb/HU49YeifrHo0KFCvbjtd0CAgIsnnrcS5cutVSrVs3y22+/WVasWGH5+eefLVmzZjXnwdYSx1Nf5+4ejiNy3PocV6pUyfwR0edeg6F2cejVq5fFnUTk+da2balTp7aMGzfOPM+NGzc2v/N6Vbunv861jZm+ATpz5ozFHYX3uDUg5s2b11KsWDHLnDlzLAsWLDBdabQHrjt1aInI862tKfW1ruHwo48+ssSNG9eybNkyizs5HIYMEhV5zT0uvY9kgwYNMjN/zZ0718yeohceaaG6o5IlS5rHhGcdTzxuLeLXj6VUiRIlzMfO6tdff5V8+fKJJx63zpCWJ08e81GM7Xht5s2bZ2pTPfG4a9asaerUtJxg+vTpkiZNGnPBll6QFytWLPHk17kjrUsrW7asJE2aVNxJeI+7WrVq5iIf/YhdX/P6Mfuff/5pSoncSUSe786dO5sL0aZOnWrKhnQGLa09dqeZMCP6OtdZ8rTEwF3L6cJ73Fpfu3HjRvn+++9l8uTJZp0iRYqYzjTu0oUnos/3pEmTZOjQoebvVs6cOU0Nsr7W3cmMMGSQqMhrMTQhR+qRAAAAAG7K6y7IAwAAAEJDOAYAAACsCMcAAACAFeEYAAAAsCIcAwAAAFaEYwAAAMCKcAwAAABYEY4BIIK0yfyFCxdc5vzdvHlTZs6cKY8fP3b2rgCA2yIcA0AEtW7dWrZt2+Yy5+/EiRPSvHlzM+ubq7p+/boJ8DpzFQC4IsIxAHiIZMmSSdOmTc0U2K7q2LFjJsA/fPjQ2bsCACHyCXkxACAiNPRt2rRJHjx4IAULFpRMmTIF+f7ixYvlzp07EjNmTEmfPr0UKVJE4sePH+QxOrJauXJlMwK8Z88eyZ07t6RIkUJWr15twu+hQ4fk5MmTkitXLnOzSZo0qdSrV09ix45t7l+9evWF69js3r1bzp07J3ny5JE0adKY/dRtxY0b95nH2rbbpEkT2bFjh5w5c0YqVaokCRIkkD///NM8RvchW7ZsUrhwYYkRI4ZZdvfuXVm5cqW9JMXX11eyZMkipUuXNsuuXbsmW7ZsMT+zaNGi5ngAILoRjgEgkmzYsEEaN25sAnHKlClNSO7YsaN8+eWX9sesWLFCLl++LE+fPpUjR46YOuEFCxaYEGmjI6uvvfaaHD582ITnFi1amECqy6dPny4XL14021+zZo0MGTJEevToEaSsQreZJEkSOXjw4AvXUe3atTOBvFy5cnL8+HETnpcsWSKXLl0yQTk423anTp1qaq718fpGQAP//PnzzWMCAgJk+/bt5lwsXbpUEiZMaML+2rVrzfcXLlwoPj4+Ur58eROOR48eLf369ZMSJUqYc7Nr1y4ZO3asNGzYkNcngOhlAQBESIIECSzz5s0zX/v7+1tSpEhhmTJliv37p0+ftiROnNiyYsWKULfx0UcfWSpVqhRkmf7XXLZsWcuDBw/sy9asWWOWDxw40L5s4sSJlnjx4lkCAgLM/W3btpnH3Lx5M8zrLF++3OLj42PZtWuXua/LK1eubNa7dOlSiPts226PHj2ee350W6VKlQry8//55x+z7p07d+zLNm7caEmUKJHl4MGD9mXz5883y65fv/7cnwEAkY2RYwCIBDoSqiOjWiowe/Zs2+CDKRvQ0dpq1arZH6vlDTpq7O/vL35+frJ169ZnttehQ4cQSxo6depk/1pLGbR84+zZs5IjR45Q9+1568yZM8fsm23kWuuVu3XrZvb5Rbp27Rrich311e3rz8mQIUOIx+do4sSJZvT5wIEDsn//fnPe9KYlKlq2Ub169RfuCwBEFsIxAESC06dPmzIBraV1pDW8mTNnNl9ruYB2uNC63JIlS5oL6LQE4v79+yZYa82uTdq0aUP8ObqOjQZx9aKL2563jtYZa4B3FPx+aILv47///muCrP5bqFAhSZQokbkA70UXCOq50/OgQd1R/fr1JV68eGHaFwCILIRjAIgEGgQ1/Gp9r9behkQvRvvjjz9MbbCtlnfRokVm+X/VFP9nu4gtqmlwvnXrVpBlGlTDIvg+jho1yoRvvUBP3yionj17yt9///3Cc6cjx1r3DADORis3AIgENWrUMGUEepGao0ePHpkuDEovxNML5VKnTm3/fvDR0uhWtmxZE8519NrG1nEivPT4smfPbg/GOhmJXmzoSMtIgo9216pVS1atWmVGmR3pRYhMaAIgujFyDACRQEc+Bw8eLO3btzddGrSsQMsFNPyOGzfOtGKrUqWKaePWqlUr06pN26FpyzRneuedd+Tbb7+VqlWrSps2bUzLtxkzZkRo9PrNN9803Tpy5sxpSi4mT55sSix0ZNhG27tpizbtTFGxYkXJmjWr+bnz5s0zQf2DDz4w6+7du9d0zNBWdrbWdAAQHRg5BoAIatSokbngzOaTTz4xJQRaY7t+/XpTL6ttzDT0KX2s9vHVlmra0kwDtLZ20z7EjgFQ7wdvoZYqVSqz3LFkQ7evy3Q0OqRJQMKyjpZBaMs5HfnWfdPey3qBnOMob3AhbdcWjnXU+cqVK+YNgrax++2336RmzZr2x2hPZx0l1v3Qixj1gjsdadavf/rpJ9NyTi/g01ptvbAveA9oAIhqMbRlRZT/FACAy7px40aQi/Z0VFcvLNRRZADwNpRVAICX00lAtK2bzsSnI74TJkwwJREA4I0YOQYAL6d10OPHjzc9hrWcQ8tFHGfsAwBvQjgGAAAArLggDwAAALAiHAMAAABWhGMAAADAinAMAAAAWBGOAQAAACvCMQAAAGBFOAYAAACsCMcAAACAFeEYAAAAkP/8D89IrvcKizEEAAAAAElFTkSuQmCC", + "text/plain": [ + "
" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "# 6. Hyperparameter Tuning: Learning Rate Sweep\n", + "learning_rates = [0.01, 0.1, 0.2, 0.3, 0.6, 0.9]\n", + "performances = []\n", + "hidden_nodes_baseline = 100\n", + "\n", + "print(\"Starting Learning Rate sweep. This may take a few minutes...\")\n", + "\n", + "for lr in learning_rates:\n", + " print(f\"Training network with Learning Rate: {lr}...\")\n", + " # Initialize a fresh network for each test\n", + " testANN = ann(784, hidden_nodes_baseline, 10)\n", + " \n", + " # Train 1 epoch\n", + " for record in list: \n", + " values = record.split(\",\")\n", + " data = np.asarray(values[1:], dtype=float) / 255.0 * 0.99 + 0.01\n", + " target = np.zeros(10) + 0.01\n", + " target[int(values[0])] = 0.99\n", + " testANN.backpropagation(data, target, lr)\n", + " \n", + " # Evaluate on the Test Set\n", + " score = 0\n", + " for record in list2:\n", + " values = record.split(\",\")\n", + " correct_label = int(values[0])\n", + " data = np.asarray(values[1:], dtype=float) / 255.0 * 0.99 + 0.01\n", + " \n", + " outputs = testANN.feedforward(data)\n", + " if np.argmax(outputs) == correct_label:\n", + " score += 1\n", + " \n", + " # Calculate performance (accuracy as a decimal between 0 and 1)\n", + " performance = score / len(list2)\n", + " performances.append(performance)\n", + " print(f\"Performance for LR {lr}: {performance:.4f}\\n\")\n", + "\n", + "# Plotting the exact graph requested\n", + "plt.figure(figsize=(8, 5))\n", + "plt.plot(learning_rates, performances, marker='s', markersize=8, color='#003366', linewidth=1.5)\n", + "\n", + "# Formatting to match the requested style\n", + "plt.title(\"Performance vs. Learning Rate\")\n", + "plt.xlabel(\"learning rate\")\n", + "plt.ylabel(\"performance\")\n", + "\n", + "# Setting axes limits and ticks\n", + "plt.xlim(0, 1)\n", + "plt.xticks(np.arange(0, 1.1, 0.1))\n", + "plt.ylim(0.8, 0.98)\n", + "plt.yticks(np.arange(0.8, 1.0, 0.02))\n", + "\n", + "# Adding horizontal grid lines\n", + "plt.grid(axis='y', linestyle='-', alpha=0.7)\n", + "\n", + "plt.show()" + ] + }, + { + "cell_type": "markdown", + "id": "7bda99b0", + "metadata": {}, + "source": [ + "## 6.2 Hyperparameter Tuning: Hidden Nodes Capacity\n", + "In this experiment, we evaluate the effect of the network's capacity by varying the number of hidden nodes `[10, 50, 100, 200, 500]`. The Learning Rate is kept constant at $0.2$.\n", + "\n", + "The resulting curve demonstrates the law of diminishing returns in neural network architecture. While increasing the number of nodes initially provides a massive boost in classification performance, the accuracy plateaus after approximately 200 nodes. Beyond this threshold, adding more nodes significantly increases computational cost and memory footprint without yielding proportional accuracy gains." + ] + }, + { + "cell_type": "code", + "execution_count": 25, + "id": "0ead825a", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Starting Hidden Nodes sweep. This will take a while...\n", + "Training network with 10 hidden nodes...\n", + "Performance for 10 nodes: 0.8064\n", + "\n", + "Training network with 50 hidden nodes...\n", + "Performance for 50 nodes: 0.9183\n", + "\n", + "Training network with 100 hidden nodes...\n", + "Performance for 100 nodes: 0.9274\n", + "\n", + "Training network with 200 hidden nodes...\n", + "Performance for 200 nodes: 0.9344\n", + "\n", + "Training network with 500 hidden nodes...\n", + "Performance for 500 nodes: 0.9167\n", + "\n" + ] + }, + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAsoAAAHFCAYAAADrKN8IAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjExLjAsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvlcelbwAAAAlwSFlzAAAPYQAAD2EBqD+naQAARi9JREFUeJzt3Qd4VFX6x/E3ISGEAAm9hl6UIh0EREClKEVQEPnDsggIsgu6Ii7L2hZFQVdxEV1sgAURFRFQEAWWKk0wKCDF0HsnCSEJhMz/eQ/OmEluwmQyYUq+n+cZZ+bOvTd37sjNL2fec06QzWazCQAAAAAnwc5PAQAAABCUAQAAgCzQogwAAABYICgDAAAAFgjKAAAAgAWCMgAAAGCBoAwAAABYICgDAAAAvhqUjx8/LmvXrpW4uDiXtzl8+LBs3rxZ4uPjc7UOAAAA4HNBWUNs7969pWHDhtK2bVuJiYm57jbJycly//33S506deRPf/qTlCtXTqZOnZrjdQAAAACfDcrbt2+Xvn37ysaNG13eZvz48bJp0ybZu3ev7Ny5U2bPni2PPvqo0z5cWQcAAADw2aA8aNAg6dOnj4SGhrq8zcyZM2Xo0KFSvnx587xnz55Sv359szwn6wAAAADZCRE/cuzYMTl58qQ0bdrUaXmLFi0cZRuurGMlJSXF3OzS0tLk3LlzUrJkSQkKCvL4ewEAAEDu2Gw2SUhIkAoVKkhwcHD+DsoaXJWG1/T0uf01V9axMnHiRFOyAQAAAP+iAzhUqlQpfwdle4mGdtZLLykpSQoWLOjyOlbGjRsno0ePdjzXETgqV64s+/fvl2LFinn0fQAAACD3dGSzatWqSdGiRSUv+FVQjo6ONs3qR48edVquzzXUurqOlbCwMHPLqESJEgRlAAAAHxQSci3K5lWZrE+Mo5yd2NhYR21x4cKFpXXr1rJw4ULH64mJibJs2TLp2LGjy+sAAAAAPt2ifOrUKdmzZ4+cPn3aPN+2bZv5y0Bbfu2tv5MmTZINGzaYoeTUhAkTTODVUolWrVqZ8ZHLlCkjw4YNc+zXlXUAAAAAn21R3rJli/zjH/+Q1157Tdq0aSOfffaZea6tv3a1atWSJk2aOJ63a9dOVqxYIQcPHpQpU6ZIvXr1zKx+RYoUydE6AAAAQHaCbDquBiyLwyMjI02nPjrzAQAA5L+85vM1ygAAAIA3EJQBAAAACwRlAAAAwAJBGQAAALBAUAYAAAAsEJQBAAAACwRlAAAAwAJBGQAAALBAUAYAAAAsEJQBAAAACwRlAAAAwAJBGQAAALBAUAYAAAAsEJQBAAAACwRlAAAAwAJBGQAAALBAUAYAAAAsEJQBAAAACwRlAAAAwAJBGQAAALBAUAYAAAAsEJQBAAAACwRlAAAAwAJBGQAAALBAUAYAAAAsEJQBAAAACwRlAAAAwAJBGQAAALBAUAYAAAAsEJQBAAAACwRlAAAAwAJBGQAAALBAUAYAAAAsEJQBAAAACwRlAAAAwAJBGQAAALAQIj7g8uXLEh8fLyVLlpSgoCCXtrly5YokJCRIiRIlMr124cIFuXjxotOy0NBQKVu2rMeOGQAAAIHNqy3KaWlp8sQTT0hUVJRUrVpVKlWqJF999VW228TGxkrHjh2lWLFiZps6derI6tWrndb5xz/+IbVr15Zbb73Vcevfv38evxsAAAAEEq8G5ddee01mzpwp69atMy3KY8eOlb59+8rOnTst17969ap0795dChUqJKdPnzbbjB49Wrp27SrHjh1zWveee+6RI0eOOG7Lli27Qe8KAAAAgcCrQfmtt96SoUOHSqNGjSQ4OFgeffRRqVy5srz77ruW6+/Zs0d27dol//znP6VIkSJm2fDhw035xYwZMzKtf+7cOVPWAQAAAPhNUD516pQcPHhQ2rRp47T8tttuk02bNlluEx4ebu7T1x9rK/OlS5dkw4YNTutqCUe1atVMoG7durX89NNPefI+AAAAEJi81plPSydUqVKlnJbrcy3FsKI1ye3atTN1zW+++aZZd8qUKaYEQ4O3XbNmzUxLc+PGjSUuLk7++te/mrrmHTt2SLly5Sz3nZKSYm52uk+VmppqbgAAAPAteZ3RvBaUtdTC6g3qaBYFChTIcjttKX7hhRfksccek6SkJHnggQekd+/ecuDAAcc6Ws5hFxkZKe+9956ULl1a5s6dKyNHjrTc78SJE2X8+PGZlsfExEhERIRb7xEAAAB5JzExMTCDcsWKFc39iRMnnJafPHnS8ZqV4sWLy+TJk52WNWzYUJo2bZrlNlqyUb58eacwndG4ceNMx8D0LcrR0dGmVVpH2AAAAIBvsVcABFxQ1vCpnfiWLl1qRrqwty4vX75cRo0a5Vjv/PnzpkOefQxkHVLO3hptb/H95Zdf5JVXXnEss9lsTuMxHz9+3NRDV69ePcvjCQsLM7eMQkJCzA0AAAC+Ja8zmlcT4DPPPGNCcvPmzaVVq1by6quvmuUjRoxwrPPkk0+ajnrbt283z59//nnTOtyhQwfZvXu3GSljwIAB0rlzZ/O61hm3b9/ebFevXj05dOiQaS3WMZp1PQAAAMDnh4e77777ZNasWfLhhx9Kr169TPP5qlWrTD2xnQ79lr4D3uOPP2465d17773y0ksvmecffPCB43VtFZ42bZqpR+7Zs6c899xzcuedd8qWLVsooQAAAIDLgmxap4BMNLRrR0AdNYMaZQAAgPyX17zaogwAAAD4KoIyAAAAYIGgDAAAAFggKAPwmBfeXyTBzYebewAA/B0DBAPwCA3Hz7690Dy23z8ztCtnFwDgt2hRBuDRkGynz2lZBgD4M4IyAI+HZDvCMgDAn1F6AcAtySlXTBD+98ffZ7uernMh4ZI8PeQeKVq4kISEFOCMAwD8AhOOZIEJRxDodK6hhMRkOZ9wyQTZ8/GXzGO9N8/tjy/+8dqFhCQ5H59oHqdcTnXr5xYuVFCKRRSSYhHhUqxIIYksEv7Hc3Of7nGRrJcVDOXvfADI7+LzeMIRftPAL7/qf+6dhTJ+eI9831ksNfWqXLiY5BR0nUNv4rVwaxl6L0la2o2fmPNS8mVzO3E2Plf7KRQWmk3Azjp4/xHMry0PKxgiQUFBHnt/AIDAQVCGXwnEkRWSki+bsGtaak2YtXjsaOm1h95E85q2COeWBsXiRQtLVNHCUrxY4T8e//48/eP067331Rp5cca3Lv+c5x7uJqP6dpD4xGSJT0z6/T5Z4i9eexxn7tMttz++6Lx+YlKKo/RDb6fOJeTq/YeGFMi69Tp92LYI2em305ZyAjcABBaCMvxGViMreDsspy9hyFS2kEUJQ/rX3C1hSK9oRCGJKhIuxYtF/B5mw68feotFmG3CCxV062dO+EtPCSsYmmVHvvSef+SP1v+SUUUkt63oF5NSHCH7WsDOEL71sQnY6Zc7h/OESynms7uSelXOxiWaW24EBwdlbsU2Ldv28hLXSkuKFA6T4GD6WQOALyAoIyBGVshtWLaXMDiXKmT12PMlDBqynAJtkfQtuBkDsPNjDbve6iBnP+fZheX0IdkT9L3qudJbbqSlpcnFSylZtnBfL3inD+j6+etN/1/Qmyf+8LGH50inMpLrl5bYl9FxEgByj6AMvw7Jdvq6tgyO6N3OUbaQsV7XuaX3xpQwZA691o/9uRUxu7Ds6ZDsSXq+TbAsEi4ixd3ej7ZKa811liE7wzLLFvDfW8ZTr6aZfer/j3o7Khdy9R4jwsMsW7ivF7LTt4BraKfjJID8ilEvssCoF/4Tkj3J1RIGe0uuJ0oYAvWz8uWQ7Is0cGsZjsshO1NpyR+PtXbbk1ztOJlpBBM6TgLIY4x6gXzl9PkE2b73mOzYe8zcvzNvdY73USIyIkclDPbaXW+WMAQCeyhmRBL3aEdADaR6K1Mid0McXb6SalqkLUN2VqUlXu44eb2QTcdJAN5Ai3IWaFHOW1rysGPfcdkee1R27LsWivW5BuXcGD+8uzz7cDePHSeQn2ntfsKl5BzXb2cM554obcqq42TmgO36mNz+XPIE4BpalOHX9Bfrr/uPO1qJ7aH4+Jm4LFvVqlUoKfVrVJR6NcpLveoVZN3Pe+W/c1dd92fxVT/gWfoNi/nmpVjEDe84GecI4M6vebLjpF5vihYOy35owCyCtwnov2+nHScLFCBwA4GIznz5wI2YoEO/ot25//i1VuK9R2XHXn18TA6dOJflNpXLlTBBuH6NClJPb9XLy83VypsOSOn1v7ullCsVeUNHVgDgnx0n/6jjznpoQHvHSd2ffXluudpx0tECnkVpiZanAPAdBOUA5+kJOrROcffBE7+XShyT7bHX7vcfO2t+6VgpXyryWhhOF4rrViv/+y/NwB5ZAYDnaAuwBlK96XXF0x0nXR+TO3PHSW0s0FtW35bltuPkHwHbtdISZpwEPIOgHMByM0GHdgb67dCpa63D+447OtfFHjmV5ZjBpYsXNa3CjlBcs6J5ntuvbbMLy4RkAN7uOOnq0IBWHSft4VxbyvOq42RkDoYGzDjde3hYaL6fcfJGfCsL30VQzucTdGhnnb1HTqfrUKe1xMdNq7F9TNeMdISIjC3EGohz+wvHFYysAMCX6BjTpaKKmNuN7DiZ1ZTv9o6TnppxUmuvXRka0Cpkp18eEV7QLztOevpbWfgfRr0IwFEvXB17uFzJYtlOoaw9wh1hOF0o1q88tUUGAOBbXOk4aVnHnUXHSU9xtePktdbvrFu9b2THyax+l/JNpm9h1Avk2QQdJ87Gm3v9aq1u9WsjTKQvm4guW5xADAB+JK87TjqFbB/rOJl+RklXSkuy6zjp6reyCHy0KAdYi3Jw8+GSRZ86S9ownLpxml9+JQYA8G32jpOZSkVy0HHSvm1W3356ouNkZLoykgPHzsrPvx257va0LPsGWpSRI9rZICdTPuv6hGQAQF53nCxbMneNTimXr0iClpXkoOPkHzNS/vGapzpO0rKcP9CiHGAtyjkpv+CvYQBAfpNdx8kH//l+jval38qm/fhOnh0rro8WZeRYdmMO2xGSAQD5UXYzTu45dCrH38oisFGYGsBhuV2TWpavEZIBALD+3am/I13B79L8gaAcoK5eTZO9R89kWs4/bAAAcheW+V2afxCUA9TSjb/KkZPnpURkhDz7cFdTR8U/bAAAcheW+V2avzAzX4CasXCduR9wd0tTQ0UdFQAAuevvQ0jOfwjKAejMhYsyf+VW83hwj9bePhwAAPw6LD/3zkLT4MQkI/kPQTkAzV6ySa6kXpUmN1WWhrWjvX04AAD4LQ3HBOT8ixrlAKOzIE1f8IN5POTeNt4+HAAAAL9FUA4wP+06JL/8dkTCCoZIv87NvX04AAAAfougHGBmLLzWmnxfh8aWg6kDAADAT4Ly1KlTpUaNGlKkSBFp1aqVrF+/Ptv1T58+LQ8//LBUrlzZTC3dqVMn2blzZ673GwiSki/L7CU/mseDe1B2AQAA4LdBeebMmTJ27Fh57bXXZO/evdK6dWsTfA8fPpxl/e29994r27dvl6VLl8r+/fulWbNm0qFDBzl//rzb+w0UX63cKhcSLkmV8iXljuZ1vH04AAAAfs2rQfnf//63DB48WHr27Clly5aVV1991bQST5s2zXL9ffv2mZbhSZMmSZ06daRkyZLy4osvSlBQkMyYMcPt/QZa2cVD3VtJcLDXvywAAADwa15LU9oCrCUT2hpsp4FXn69bd22yjIzS0tLMffoQqNvo87Vr17q930Bw4NgZWb5pl3mvg7ozdjIAAIDfjqN8/Phxc1+mTBmn5aVLl5bNmzdbbqM1xw0aNJBnnnlGPvroIylVqpRMnjzZ7Mu+P3f2q1JSUszNLj4+3tynpqaam6+b8fuQcHc0qyMVS0f6xTEDAADkRl7nHZ+bcERbh7UWOavXFi5cKKNHj5ZGjRpJUlKS9OnTR3r37n3d+uPs9qsmTpwo48ePz7Q8JiZGIiJ8e/SIq2lp8t68Vebx7fXKZvsHAQAAQKBITEwMzKBcrlw5xygW6Z06dcrUFWelatWqMm/ePKdlLVu2lJo1a+Zqv+PGjTMBPH2LcnR0tDRu3NjUN/uyZRt3yonziRJVNFxGP9RLCoWFevuQAAAA8py9AiDggnKJEiWkdu3asmrVKrnvvvvMMm3xXblypfTv39/l/eioFtqC+sQTT+Rqv2FhYeaWUUhIiLn5sg8XbTT3/bu0lCIR4d4+HAAAgBsirzOaV4dG0HA7ffp0+e677yQuLk6efvppOXfunDzyyCOOdYYPH27KLOy0JlnX13ribdu2mdKLO+64w5Rf5GS/geJcXKJ8tTLGPB7MlNUAAAAe49Wm0mHDhsmFCxfkoYceMqUR9evXl8WLF5vyCrurV686FWrff//9MmrUKOnVq5dERkZKv379ZMKECU4jYbiy30Axe8kmSbmcKg1rV5LGdaK9fTgAAAABI8iWXQ83H6BDwukhFihQ4IbXvGgQ1xZpX65RbtJ/gsTsPixvjOkrox68w9uHAwAAEDB5zbeLbzOMmQxnMbsOmZBcMDRE/q9LC04PAACAB5FC/djMr69NoNKzfUMpGVXE24cDAAAQUAjKfio55YrM+vbaaBeDe7Tx9uEAAAAEHIKyn1qwaqucj78k0WWLy10tbvb24QAAAAQcgrKfmrHwWtnFoO6tpUABPkYAAABPI2H5oUMnzsnSjTvN40HdWnn7cAAAAAISQdkPffD1OjNkXodmdaR6pdLePhwAAICARFD2MzqutH20iyHMxAcAAJBnCMp+ZuWWPXLg2FmJLBIu93Vo7O3DAQAACFgEZT8zY+EP5r5f5+YSXqigtw8HAAAgYBGU/ciFhEvy5f9izGPGTgYAAMhbBGU/8ul3P5qJRhrUrCjN6lbx9uEAAAAENIKyH5ZdDO7RWoKCgrx9OAAAAAGNoOwnfvntiGz+9aCEhhSQ/ne39PbhAAAABDyCsp+1Jve4vaGULl7U24cDAAAQ8AjKfiDl8hWZtXijeczYyQAAADcGQdkPfL36FzkblygVy0RJp1vrevtwAAAA8gWCsh+YsfDaTHx/7tpKChTgIwMAALgRSF0+7sjJ8/Ldhh3m8UM9Wnv7cAAAAPINgrKP+/Cb9ZKWZpN2TWpLzegy3j4cAACAfIOg7MPS0tKcxk4GAADAjUNQ9mFrYmJl39EzUjSikNx/ZxNvHw4AAEC+QlD2YfbW5Ac7NZOI8DBvHw4AAEC+QlD2UXEXk+SLZVvM48E92nj7cAAAAPIdgrKP+uz7HyUp5YrUrV5eWtav5u3DAQAAyHcIyj4+drK2JgcFBXn7cAAAAPIdgrIP2rH3mGzcvl9CCgTLgLtbevtwAAAA8iWCsg934uvW9hYpW7KYtw8HAAAgXyIo+5jLV1Ll48UbzeMh99KJDwAAwFsIyj5m0dptcvp8gpQrWUy6tKrn7cMBAADItwjKPlp28edurSQkpIC3DwcAACDfIij7kGOnL8jiH7abxw91Z8pqAAAAbyIo+5CPFm2QtDSb3NaoptSpWs7bhwMAAJCvEZR9hM1mc5RdDO5BazIAAIC3EZR9xA8/75XfDp2SiPAw6XNXU28fDgAAQL5HUPYR0xesNfd9OzaTIoULeftwAAAA8j2Csg9ISEyWz5duMY8ZOxkAAMA3eD0oL1iwQO68806pX7++9OvXT2JjY7NdPzk5WSZNmiS33367NGjQQLp27SoLFy50WufZZ5+VmjVrOt3uuece8VWfL90sl5IvS50qZaXVLdW9fTgAAAAQkRBvnoVvvvlGevfuLf/+97+lVatW8vrrr0vbtm1lx44dUqJECcttHn30UVm8eLFMmzZNqlatKkuWLJFevXrJ119/7QjDp06dklq1asnUqVMd24WFhYmvmrFwnbkf3KONBAUFeftwAAAA4O2g/K9//UsGDBggf/vb38zzjz76SMqXL29C8FNPPWW5zdKlS2XIkCHSvXt381xblWfNmiXLli1zajWOiIgwLcm+buf+47Lul71SoECwDOx6q7cPBwAAAN4uvUhISJCffvpJOnfu7FhWsGBBueuuu2TlypVZbte+fXtZsWKFXLx40Tzfvn277Nu3Tzp06OC03tq1a+WWW24xLdT//Oc/zc/zRTN/b03u2qaBlCsV6e3DAQAAgLdblI8cOWLGDi5XznliDX3+yy+/ZLndu+++KwMHDpQyZcqY8oxz587JW2+95WhhVrp87Nix0q5dOzl69Kg8/fTTpsxj8+bNJoxbSUlJMTe7+Ph4c5+ammpueeFK6lX5aNF68/jP3Vrm2c8BAAAIRKl5nJ28FpTT0tLMfWhoqNNyDbJXr17NcrtnnnlGfvjhB/nkk0+kRo0a8v3335u6ZX2sHfzUCy+8IAUKFDCPmzRpIo0bN5Zq1arJnDlzTMi2MnHiRBk/fnym5TExMaaMIy+s+eWQnDyXIMWLFpIyYSkmyAMAAMA1iYmJEpBBuVSpUub+7NmzTsvPnDnjeC2juLg40/Hvgw8+MB34lJZXrF+/3oRjrV9W9pBsV6lSJdPxTzsJZmXcuHEyevRopxbl6OhoE7KLFSsmeeGlz64F44d6tJFbW7bIk58BAAAQqOJ/rwDwyaCsZQ1aH6x1wDlVtmxZE0Q15Pbo0cOxXFuLu3TpYrmNlkZoS3Tx4sWdluvz48ePZ/mzdLsTJ05IVFRUluvoqBhWI2OEhISYm6edOBMni9ddC+5De7bNk58BAAAQyELyOD+51ZlP64I1zGpLrb3cQXXr1k3WrFnj8n4eeeQRef/992XXrl2O+mMN3g8//LBjHe2IZx/NQuuStQVZh5HT1mWl9cxffvmldOzY0Ty/fPmyPPnkk46W6kuXLslf/vIXU87Rt29f8RUfL94gV6+mmXGTb65W3tuHAwAAgAzciuFjxowxtcQHDx6UKlWqOJY//vjjMmHCBPnuu+9c2o92uNNOfQ0bNjR1wFoyoUO96ZBvdjom8qFDhxzPP//8cxk2bJhpkdZOe+fPn5dBgwY5hpPTmmcN8BqotfVZX9f9L1++XKpX943JPLQTY/qxkwEAAOB7gmya2nJIR6bYsmWLVKxY0UyQYd+FtvLqOMjaipvTQmxtpdZtMzahnz592szGp2Ua6SUlJZlt9Fgy1iTbnTx5UiIjI6VQoUJu1bzotvqePF2jvO7nvdJmyCtSuFBBOb7kFSlWJNyj+wcAAMgP4vMwr7ndoqwHU7RoUfM4/UxyutydWhFtTc5qZInSpUtbLg8PDzdBPTva6uyLZiz8wdw/0LEpIRkAAMBHuVWj3KJFC/niiy+cgrKWOWjZRZs2lBJk5+KlZPls6bXRLii7AAAA8F1utSi//PLL0qlTJzNDnpZdaOc5Hc84NjY2R5358qO5y3+Si5dSpGZ0Gbmtke9PsQ0AAJBfudWifOutt8qGDRvMcGra8W7JkiXSqFEj2bRpk5ngA1mbvuBa2cXgHq2dylYAAADgW9wefK5u3boyffp0zx5NgNtz8KSs3RorwcFBMrDrrd4+HAAAAHi6RVnHJN66dWum5bosu+mn87uZX18bEu7u1vWlYhnnSVMAAAAQAEF54sSJZpKPjHSZ1i8js9TUq/LhN+sdZRcAAADwbW4F5bfffltGjBhhOdOezq6HzJas3yHHz8RJqagi0q3tLZwiAACAQAzKOtudVUc0XaYz6SHrsZP/dE9LKRiat/OSAwAAwEtBuWXLlvKf//wn0/LXX3/djLEMZ6fOxcvXq38xjxk7GQAAwD+41bT54osvyh133GHGTL799tvNWMqrV682nfn+97//ef4o/dysxRsl9WqatKhXVerXzH42QQAAAPhxi3KrVq3MmMm1atWSRYsWybfffiu1a9c2y/Q1/EH/iJj+e9kFrckAAAD+w+1iWZ1o5MMPP/Ts0QSgTTsOyK/7jkt4WKg82Lm5tw8HAAAALspVrzIdMzkhISHT8qioqNzsNqDM+H0mvt53NpXIIuHePhwAAADkZenFrl27pG3btlKoUCEpXrx4phuuuZR8WT79/kfzmLGTAQAA8kGL8tChQ00gXrx4McE4G3OXbZGExGSpXrGU3N6kltsfEgAAAPwkKMfExMihQ4ekZMmSnj+iADJj4bUpqx/q3lqCg91qvAcAAICXuJXeoqOjJTk52fNHE0BiD5+SVT/tMZOwDOrOlNUAAAD5IiiPGjVKHn/8cYmLi/P8EQWID76+1prcuVVdqVSWum0AAIB8UXrxwgsvyMmTJ+XLL7+UUqVKZZrO+sSJE5KfXb2aJh98s948ZuxkAACAfBSUX331Vc8fSQD5fsOvcvTUBSkZGSE9br/F24cDAACAGxWUBwwY4M5m+caM32fiG3BPSwkrGOrtwwEAAIAbGIrBw85cuCgLVv3sGO0CAAAA+WxmvnXr1sncuXPNMHGpqalOr82fP1/yq1mLN8iV1KvS9ObK0rB2tLcPBwAAADeyRXnWrFnSsWNHOXXqlKND3/79+2XBggVmtr78ymazOcZOphMfAABAPgzKkyZNkk8//dQEZvX+++/L1q1bZcyYMZlGwMhPtuw8KNtij0qhsFD5vy4tvH04AAAAuNGlF7GxsaZFWYWGhsqlS5ekcOHCMnbsWKldu7bkV/bW5Ps6NJaoooW9fTgAAAC40S3KKSkpEh4ebh5XrFhRfv31V/P44sWLcuXKFcmPkpIvy+wlm8zjwT3oxAcAAJBvO/PZ9e7d2wwXp/fffPONo6U5v5m3IkbiLiZJ1QolpUOzOt4+HAAAAHgjKK9ff23WOfXiiy+aDny67LbbbpPx48dLfh47WYeECw5m1D0AAAB/F2TToRqQSXx8vERGRkpcXJwUK1YsyzP0wvuL5Ll3FoqeRe3IuH/hi1KlfEnOKAAAgI/kNa+VXuRnGpKffXuh43n1iiUJyQAAAAHCraCsqV2HiFu7dq2cP38+0+vbt2+X/BaS1d4jZ8zyZ4Z29dpxAQAAwItB+aGHHjIjXTz44IMSFRUl+Y1VSLazLycsAwAA5MOg/P3338uOHTukSpUqkt9kF5LtCMsAAAD+z63hGbRo2j6Ocn7iSki20/V0fQAAAOSjoDxixAj5+9//bmbky090dIu8XB8AAAB+XnrRr18/ad68ucyePVvKly9vhkVL78CBAy7va/PmzTJt2jQ5efKkNGjQQMaMGSMlS2Y9vFpaWpp8/vnnsnjxYtORsHLlyjJkyBBp0qRJrvbrivHDe7jcomxfHwAAAPkoKP/5z382AVVn5MtNZ75169ZJhw4dTAt1p06d5L///a+0adNGtmzZIhEREZbbPP300yYAP//881K1alVZsmSJ3HrrrbJixQqzrbv7dYW9g54rYfn5R3rQoQ8AACC/TTii9cmxsbFSsWLFXP3w9u3bS4kSJWTevHnmeUJCglSoUEEmTJggjz32mOU2NWrUMKNt6IyAdvXq1ZMuXbrIa6+95vZ+czKA9fVqlQnJAAAA/j/hiFs1yhqQCxQokKsfnJSUJGvWrJGePXs6lhUtWlTuvPNOM6pGVrSMYtu2baYEQ504cUKOHTsmDRs2zNV+c9qyrGHYCiEZAAAgn5deaMvsu+++a1K8Ow4fPmzCbqVKlZyW63Mto8jKhx9+KMOGDZNq1apJdHS07N6925RhDBw4MFf7TUlJMbf0f6Go1NRUc8to3KDO5uf8691vHMv+NaybWW61PgAAADwrrzOXW0H5zTfflFOnTsncuXOlVKlSmTrzaSvv9Vy+fNncZxxmrnDhwo7XrHz88ceyfPlyeeqpp0wZhrYSaxmGlltoa7O7+504caKMHz8+0/KYmJgs65o731JajnZtLO8vipGhXRub59qJEAAAAHkvMTHR94KyvRY4N4oXL27uz50757T87NmzjtesToaOXvGf//xHHnnkEbOsR48e8ttvv5ngvHDhQrf2q8aNGyejR492alHWFuvGjRtnW/PSrFkz+e8zLr1lAAAAeJC9AsCngrI2cw8aNChXP1jrnEuXLi0//fSTdO16bTQJpc916DkrWqit5RE62kV6+nzr1q1u71eFhYWZW0YhISHmBgAAAN+S1xnNrc58Dz/8sKMzXW5orfP06dNNGYfSsZE18Opyu8mTJ5ufp3TkCp02e+bMmY6alOPHj8s333wjrVq1ytF+AQAAgOy4FcNvuukmU7vbtGlTyQ2tCd6xY4fUqlXL1Bvv3LlTXnnlFcd4yOrXX3+VDRs2OJ7PmTNH+vfvb1qRdSxnHQHjtttuMx36crJfAAAAwOPjKOsEHlonrOG0bt26UrBgwUxBOif27NljZtC7+eabTefA9DTkasmFTipip63JOo6z1h1rC3PGES5c2a+3x+UDAABA7uR1XnMrKGcc5SIjN3bpcwjKAAAA+TuvuVV6oWMVAwAAAIHMraCcVakDAAAAEChyNaaGjiShNcRaaqG1yo0aNfLckQEAAAD+FpS1E12/fv1k6dKlUqhQIVOznJSUJJ06dZLZs2dLyZIlPX+kAAAAwA3k1jjKjz76qCma1iHiNCBfunTJPL5w4YI89thjnj9KAAAA4AZza9SLqKgo2bx5s9SsWdNpuU4l3aJFCzl//rz4O0a9AAAAyN95za0W5cuXL1sejC7TKaYBAAAAf+dWUNaZ8MaMGSOJiYmOZRcvXpQnnnhC2rZt68njAwAAAPynM9+UKVOkS5cuUqFCBalXr55ZplNGa0nGkiVLPH2MAAAAgH/UKKvk5GSZM2eOCcg66oUOD/fggw+aUTACATXKAAAAvs1nZuarX7++bN++3TyeMGGCPP300zJo0CCPHxAAAADgVy3KYWFhkpCQIAULFjQtyG42RPsNWpQBAAB8m8+0KDdo0ECGDBlihn9Tb775Zpbrjhw50jNHBwAAAPh6i7KWXWi5RWxsrKlLrlOnTpbr7tq1S/wdLcoAAAD5O6+51ZmP0gsAAAB4m09OODJt2jSPHwgAAADgS9wKyqNGjZK0tDTPHw0AAADgz0H5pptukpiYGM8fDQAAAODPM/ONGDFC+vXrJ88//7yZaESHjMsYpAEAAAB/5nZnvuwEwhjLjHoBAADg23xmHOX0Dh8+7PEDAQAAAHyJW0G5UqVKnj8SAAAAwN8786kLFy7IJ598Ii+88IJjmXbwYzQMAAAA5NsaZZ2lr2PHjhIeHi779+931CTrFNft2rWTgQMHir+jRhkAAMC3+eSEI6NHj5ZHHnlE9u3b57R85MiRMnnyZE8dGwAAAOBfLcqa3LVDnyb39NNZJyYmSokSJSQlJUX8HS3KAAAAvs0nW5SDg4Pl0qVLmYaK++233yQqKspzRwcAAAB4iVtBuUuXLjJhwgTTkmwPyseOHZO//vWv0q1bN08fIwAAAOAfQfm1116TZcuWSZUqVcwoF82bN5fq1avLuXPnZOLEiZ4/SgAAAMAfapRVcnKyfPHFF7J582YTlps0aSIPPvigGQkjEFCjDAAAkL/zmttBOdARlAEAAHybT05hrY4cOSLTpk2TnTt3mud169aVESNGSMWKFT15fAAAAID/1Ch/9913UrNmTVm4cKFJ8XqbP3++WbZ06VLPHyUAAABwg7lVenHzzTebeuTnnnvOafn48ePls88+k19//VX8HaUXAAAAvs0na5QLFy5shoPLOGbyhQsXTOmFTjzi7wjKAAAAvs0nJxzRFmWrVuMdO3bITTfd5InjAgAAALzKrc58gwYNkt69e8vTTz9txlDWRmkdJk4nIXnqqadk165djnWvF5wPHTokH374oZw8eVIaNGhg9h0WFpbl+iNHjjRD02XUtm1b+fOf/2wez5o1S1auXOn0urZ0a2kIAAAAkGdB+dFHHzX3OhOfVZBNL7vKDm2VbtOmjbRv315uvfVWeeONN+Sjjz4yITc0NNRyGw3mV65ccRp9QwNwy5YtHcvWrl0rW7dulUceecSxrESJEjl8lwAAAMjP3ArKhw8f9sgPHzt2rJmoZN68eWYq7IEDB0q1atXk448/lsGDB1tuY281tnv++eelSJEipnNhelWrVpWhQ4d65DgBAACQ/7gVlCtVqpTrH3z58mUzzNxbb71lQrIqX768dOjQQb7++ussg3LG1uqZM2eakFy0aFGn13bv3m1avrXAW8syOnXqlOtjBgAAQP7h9oQjuaW1yVpCoS2/6WmL8po1a1zax/Lly+XAgQPy8MMPOy3X4F29enWz76NHj8r9999vbh988EGW+0pJSTG39L0oVWpqqrkBAADAt+R1RvNaUE5KSjL3GVuC9bn9teuZPn263HLLLdKiRQun5c8++6xpnbbTkHzbbbeZlucuXbpY7mvixImWnf1iYmIkIiLCpeMBAADAjZPXQxJ7LSjbx7o7f/680/Jz5865NA6ervfVV1/Jq6++mum19CFZtW7dWqKjo2XdunVZBuVx48bJ6NGjnVqUdZvGjRvnybh8AAAAyB17BUDABWUNoRpAdeSLu+++22ks5vr16193ex0CTkssBgwY4NLP01bq7Ebg0CHprIalCwkJMTcAAAD4lrzOaG5NOOKRHxwcLH379pUZM2Y4ms1//PFH2bBhg/Tr18+xno6xnHGqbKXb9enTJ9PsgFqrsmjRIqdl7733npw+fdopkAMAAAA+GZTVSy+9ZP4SaNiwodx3331y1113yYgRI5wCrXbs+/LLL52208lNfv7550yd+JS2Mmvtsu5TA7eO0/z444/L66+/bkowAAAAAFcE2bKrR7gBdOSLFStWOGbma9SokdPrOnnIqVOnTJC205Csnex0Fr+s6PBwOulI8eLFzVjNpUqV8qm5wwEAAJA7eZ3XvB6UfRVBGQAAIH/nNa+WXgAAAAC+iqAMAAAAWCAoAwAAABYIygAAAIAFgjIAAABggaAMAAAAWCAoAwAAABYIygAAAIAFgjIAAABggaAMAAAAWCAoAwAAABYIygAAAIAFgjIAAABggaAMAAAAWCAoAwAAABYIygAAAIAFgjIAAABggaAMAAAAWCAoAwAAABYIygAAAIAFgjIAAABggaAMAAAAWCAoAwAAABYIygAAAIAFgjIAAABggaAMAAAAWCAoAwAAABYIygAAAIAFgjIAAABggaAMAAAAWCAoAwAAABYIygAAAIAFgjIAAABggaAMAAAAWCAoAwAAABYIygAAAICFEPGyhIQEmT9/vpw8eVIaNGggnTt3znb9V155RS5fvpxpeePGjaVr165u7xcAAADwmRblI0eOmBA7depU2bt3rzz00EPSp08fsdlsWW6TkpIiycnJjtvRo0flmWeekd27d+dqvwAAAEB6QTYvpsf+/fvLnj17ZN26dRIaGmrCbr169WTOnDnSu3dvl/YxefJkGTdunAnMpUqV8th+4+PjJTIyUuLi4qRYsWK5ep8AAADwvLzOa15rUb569aopjRg4cKAJs6pOnTrStm1bmTt3rsv7mTFjhvTq1csRkj21XwAAAORvXqtRPnTokFy6dElq167ttFyfb9y40aV9bNiwQXbs2CFTpkzJ9X61pENv6f9CUampqeYGAAAA35LXGc1rQfnixYvmXpvL04uKinK8dj3Tp0+X6tWryx133JHr/U6cOFHGjx+faXlMTIxERES4dDwAAAC4cRITEwMzKNvDp73l1k5rTFwJpnpiPvvsM1OfHBQUlOv96n5Gjx7teK7bR0dHm9E0qFEGAADwPRnzXsAE5cqVK0t4eLjExsZKp06dHMt/++03U1N8PZ9//rkkJSXJoEGDPLLfsLAwc8soJCTE3AAAAOBb8jqjBXvzjXXv3l0+/vhjR32JDuW2evVque+++xzrLVq0SN59913Lsotu3bpJ+fLl3dovAAAA4LPDwx04cEDatGlj6oybN29uRqVo2LChLFiwQIKDr2X4oUOHmk5727dvd2ynw73ddNNNJkTfc889bu33ehgeDgAAwLfldV7zak1B1apVTQD+4osvzAx6b7zxhvTo0cMpzGqrsYbc9M6cOSPPPfecdOnSxe39AgAAAD7bouzLaFEGAADwbQE74QgAAADgywjKAAAAgAWCMgAAAGCBoAwAAABYICgDAAAAFgjKAAAAgAWCMgAAAGCBoAwAAABYICgDAAAAFgjKAAAAgAWCMgAAAGCBoAwAAABYICgDAAAAFgjKAAAAgAWCMgAAAGCBoAwAAABYICgDAAAAFgjKAAAAgAWCMgAAAGCBoAwAAABYICgDAAAAFgjKAAAAgAWCMgAAAGCBoAwAAABYICgDAAAAFgjKAAAAgAWCMgAAAGCBoAwAAABYICgDAAAAFgjKAAAAgAWCMgAAAGCBoAwAAABYICgDAAAAFgjKAAAAAEEZAAAAcA0tygAAAICFEPGyq1evyrp16+TkyZPSoEEDqVOnjkvbnT17VtavXy+FCxeW2267TQoWLOh4bdOmTbJnzx6n9aOioqRbt24eP34AAAAEJq8G5fPnz0vnzp3lxIkTUq9ePVm7dq0MHz5cXn311Wy3mzJlijz11FPSsmVLE5SffPJJWbBggVSqVMm8PmPGDPn222+lbdu2jm2io6MJygAAAPCPoKxhNz4+Xnbs2CFFixY1LcvaOqzhuWPHjpbbzJ8/X0aPHi1LlixxrLN7925JSUlxWq958+Yya9asG/I+AAAAEHi8VqNss9nk008/lSFDhpiQrFq3bi0tWrSQTz75JMvtJk6cKL169XIK0lquUaNGDaf1zp07J/PmzZPly5ebxwAAAIBftCgfPnxYLly4IPXr13darnXKP/30k+U2SUlJsnnzZhOuY2Nj5eeff5YKFSqY1uOQEOe3oq3UH3zwgRw9etTUK7/++usydOjQLI9HW6TTt0prS7dKTU01NwAAAPiWvM5oXgvKcXFx5r548eJOy0uUKOF4zaoDX1pamqk/fvnll6Vhw4YSExNj6pQXL14sVapUMev93//9n6ljDgsLM8+nTp0qI0aMMIFat8mqpXr8+PGZluv+IyIicv1+AQAA4FmJiYmSl4JsWgPhBdoiXKtWLVm6dKncddddjuV/+ctfZM2aNbJt27ZM25w5c0ZKly5tWp11ZItChQrJ5cuXTclG1apVZe7cuVn+PN1uzJgxMnbsWJdblLUDoIbzYsWK5fr9AgAAwLM0r5UsWdI0suZFXvNai7KG0NDQUDl48KDTcn1evXp1y21KlSplhnnTzn4akpUOC3f33Xdft+Oetgpr0M6Ktj7bW6DT05KOjGUdAAAA8L68zmhe68ynofTOO++Uzz//3LHs9OnT8r///U+6du3qWLZhwwb55ptvHM+7d+8uv/76q9O+9Lm97EJLM3S4ufR+/PFHOXTokBlODgAAAPDp0gv1yy+/SJs2bcz4xq1atTLjH2sr8w8//OCYQEQ74GlY3r59u6PFWQOvtirrths3bjSjZGgJh46bfOXKFVOHrOUcOjazBuT//ve/0r59e/nyyy8lODjY5ab8yMjIPGvKBwAAQO7kdV7z6hTWt9xyi2zdulWqVatmRrAYNGiQrFq1ymmWPQ3Q2opspy3Huk3NmjVNnbI+37lzp2NyEQ3aW7ZsMaNp6Hr6d8Ds2bPlq6++cjkkAwAAAF5tUfZltCgDAAD4toBuUQYAAAB8FUEZAAAAsEBQBgAAACwQlAEAAAALBGUAAADAAkEZAAAAsEBQBgAAACwQlAEAAAALBGUAAACAoAwAAAC4hhZlAAAAwAJBGQAAALBAUAYAAAAsEJQBAAAACwRlAAAAwAJBGQAAALBAUAYAAAAsEJQBAAAACwRlAAAAwAJBGQAAALBAUAYAAAAsEJQBAAAACwRlAAAAwAJBGQAAALBAUAYAAAAsEJQBAAAACwRlAAAAwAJBGQAAALBAUAYAAAAsEJQBAAAACwRlAAAAwAJBGQAAALBAUAYAAAAsEJQBAAAACwRlAAAAwAJBGQAAAPDVoHz48GHZvHmzxMfHu7zN1atXZfv27bJv3z6P7hcAAADwelBOTk6W+++/X+rUqSN/+tOfpFy5cjJ16tTrbjdv3jypVKmS3HvvvebWsWNHOXPmTK73CwAAANiFiBeNHz9eNm3aJHv37pXy5cvL/PnzpVevXtKiRQtp2bKl5TarV6+WPn36yHvvvSeDBw82y1auXCknT56UUqVKub1fAAAAIL0gm81mEy/Rlt4RI0bIc88951jWoEEDadOmjbz99tuW29xxxx0SGhoq3333nUf3m5GWa0RGRkpcXJwUK1YsR+8LAAAAeS+v85rXWpSPHTtmWoGbNm3qtFxbfWNiYiy3SUlJkbVr18rkyZMlISFB9uzZIxUqVDCtxrnZr33ferPTE67OnTsnqampbr9PAAAA5A17P7S8avf1WlDWAKpKlizptFyf21/LSOuQr1y5Ir/88oupP9aW49jYWGnVqpXMnj3baduc7FdNnDjRlGxkVK1aNbfeHwAAAG6Ms2fPmpblgAnKWj5h73iXXlJSkhQsWDDbbZYuXSpbt26VMmXKmBOjQXnMmDEyc+ZMt/arxo0bJ6NHj3Y8v3DhglSpUkUOHTqUJyceufvrMTo62oxqQlmMb+Gz8W18Pr6Lz8Z38dn4Nq0AqFy5spQoUSJP9u+1oKxBJzg4WI4ePeq0XJ/rG7ainfUiIiLkvvvuMyHZ3lKsnfs+++wzt/erwsLCzC0jDcmEMd+knwufjW/is/FtfD6+i8/Gd/HZ+DbNfnmyX/GSwoULS+vWrWXhwoWOZYmJibJs2TIz3JudllbYa4v1JOhrGUPwkSNHpHTp0jnaLwAAAOCzw8NNmDDBhFcte9DyCR3rWFuKhw0b5lhn0qRJsmHDBjO5iHr++efN6BU6ooXeb9y40dQnf/755znaLwAAAOCzE460a9dOVqxYIQcPHpQpU6ZIvXr1zKgWRYoUcaxTq1YtadKkidMwb+vWrTOtyK+88ors379fVq1aZcZJzsl+r0fLMDSMW5VjwLv4bHwXn41v4/PxXXw2vovPJn9/Pl4dRxkAAADwVV5tUQYAAAB8FUEZAAAAsEBQBgAAAHxt1AtfpZOY7Nu3z4zJrLP/4cbQcnkdClCHAWzUqJHlOjoz444dO0zR/k033SRBQUFurYOc0XO6e/duM465TsST1XiVe/fuNZP13HzzzWaoRnfXQc6cOnXKMVZ8xllJc3Jd49qXd3bu3GnOb/PmzTN1OuK6duP9+uuvmWbrLV68uOn8nxHXNe/Rc68TyOnvC6vfOzfkuqad+fCHZ5991hYWFmarW7euuR8yZIjt6tWrnKI8NnnyZFvt2rVtUVFRtoYNG1qus2rVKlu5cuVsVapUsZUqVcpWv3592759+3K8DlyXlJRk+/vf/24rUaKEOZcVKlQwn9MPP/zgtN65c+ds7dq1sxUtWtRWq1YtW7FixWxz5szJ8TrImZiYGHNO9XNp3LixLTw83PbAAw/YLl26lOPrGte+vPPzzz/bChcurB3nbfv373d6jeuad3Tt2tX8u2nTpo3j9sQTTzitw3XNe/Ta1qBBA1uZMmVsTZs2tdWrV8+2detWr1zXCMrpzJ8/3xYaGuoIAbt27bJFRkbapkyZ4v6njetKTU21/e1vfzPnWy9UVkE5ISHBVrp0advo0aPN8ytXrtjuuusuW6tWrXK0DnLmxIkTtpdfftmcW/tnNWzYMPNHSHJysmO9AQMGmCAdFxdnnk+dOtVWsGBBp1DgyjrIma+++sr2448/Op4fOnTIVrJkSdsLL7yQo+sa1768k5iYaH5J6x+cGYMy1zXvBuXHHnss23W4rnmH/t7R69iIESPM7xwVGxtr+/bbb71yXSMop9OjRw9bly5dnE7Q0KFDs2zhhOdlFZRnz55tK1CggO3MmTOOZcuWLTO/eHbu3OnyOsi9zZs3m3OqrWT2X/YaeN9//33HOnpx0zBtD2yurAPPaNGihe2RRx7J0XWNa1/eeeihh2wjR460rVixIlNQ5rrm3aCsrYv6h+bBgwdtaWlpTq9zXfOecePGmZbk9I0xGd3I6xqd+dLR+timTZs6laa0aNHCzAqoNWTw7mdTtWpVp/pL/Wzsr7m6DnLvxx9/lAIFCphzba/1u3z5stO/HX1dJwqyn3dX1oF7rl69aiZUWrZsmfzjH/8wkzGNGjUqR9c1rn15Y86cOWb2WJ0cywrXNe+aNWuWDB061PSJ0RrY9evXO17juuY9y5cvN7Mra02y/hvRieXS0tKc1rmR1zWCcjpa2J+xI4w+119E8fHxrn/KuCGfTdGiRSU0NNTRIcOVdZD7jhVPPfWUPPbYY1KsWDHHeVdW/3bSfzbXWwfuSUpKMgF5zJgxMnXqVBk4cKCZ0TQn1zWufXnzb0X/YJk9e7aEh4dbrsN1zXsGDBhgOsFu3bpVjh8/Li1btpSePXuajl/2z0ZxXbvxjh07JpcuXZL69evLQw89JK1atTKzMv/8889eua4RlNPRQKW9KzP+ElIFCxbMyeeMG/DZpKammpv9s3FlHeTu4tW5c2dp3bq1TJo0yemzUVb/dtJ/NtdbB+4pUqSIaVHWX/jaCqbB7O9//3uOrmtc+zzv4YcfNv9eEhISzOezbds2s3zz5s2mB35W553r2o3x4IMPOv7Y11FIpkyZYoLz//73P8dno7iu3Xh67hctWiSffPKJua4dOnRIateubT4zb1zXCMrp6LBXOsRSevo8KirKtEzCu5+NBrX0M67bn+uQWK6uA/foeezQoYPUqVNHvvzyS8cvEft5V1b/dtJ/NtdbB7mn5/mBBx6Qb7/9NkfXNa59nlemTBk5cOCAae3X27vvvmuWv/zyy7JgwQLHeee65hsiIyOlUKFCjn8rXNe8p2rVqtKsWTNzs4fawYMHy65du+TMmTM3/LpGUE5Ha2IWL15smuXt9IKmy+Fd+hnoP5D0NWT62eg4vG3atHF5HeScfi2pIblGjRoyb968TH+J16xZU6pVqyYLFy50LNM62S1btjj+7biyDnIuMTEx07LY2Finrxtdua5x7cub+mRtSbbftCxGffHFF/L44487zjvXtRtP+0toy316q1evNq2P+nW/4rrmPZ07dza/d9LXJevvC/3dY/8W4IZe13LU9S/AHTt2zPS07N27t23hwoW24cOHm7Evt23b5u1DyxdjJq5Zs8b24IMP2mrWrGke680+NIzq06ePrXr16rZPP/3U9vbbb9uKFClie+mll5z248o6cN2FCxdsN998s61GjRpmBBH756K38+fPO9bT8ZBDQkJsEydOtM2bN8/WrFkzW/PmzZ0+P1fWQc5oj24dJ/Trr7+2LViwwPTo1nO8ePHiHF3XuPblPatRLxTXtRvv8OHDtkaNGtnefPNN23fffWd74403zL+Re+65x2n0C65r3hEfH29+5/Tv398MCffOO++Y4eLGjh3rletakP4nd9k/sGjvSu2hrLOQ6VfC+pd/w4YNvX1YAU8L9n/77bdMy5csWWJqMO2tAFpHpr37taasT58+8qc//clpfVfWgeu0dXLQoEGWr7322mumA4ydft0/ffp0M+ueLtc6Wf06Mz1X1oHrtMPL22+/bVrDtPVF6/iGDx/u1JnP1esa1768pT3wtXPf3LlznWYH47rmvc6Wb731lpnFtWzZstKpUyfp379/pplcua55h9aL6zVLa5RLlSol9957r6lRTv/53KjrGkEZAAAAsECNMgAAAGCBoAwAAABYICgDAAAAFgjKAAAAgAWCMgAAAGCBoAwAAABYICgDAAAAFgjKAGAxNbROQXzx4kWfOzc6Mc9XX30l3333nVvHrRNc6Do66UtWdPISXefs2bO5WsfXBcJ7AJC3CMoAkMHp06elX79+cuLECZ86NzNmzJDmzZvLrFmzZMWKFW4dd3x8vFnnwIED2YZpXcdqtsycrOPrAuE9AMhbIXm8fwCAh7z33nsyduxYGTdunNv70Knd+/btK8WLF+dzAYDroEUZgM9JX0Kwb98+WbRokfz8889O65w/f96sc+XKFcey1NRUp6/S0+8nNjZWvv76a6f9/PLLL7JgwQLzWlb27Nljttu2bVuWLbRaBqG3kydPZvk+tm/fLvPmzZODBw9m+bO0JXjhwoVmXwkJCY7l+h51P/v375e9e/eaxzt37sz2HGZ13jQo9+zZUyIjI52Wa6uqngs9zqy4so6r5yOr48soJ9tkdf48+R7sn8eaNWvMsRw5ciTb4wfg52wA4GP2799v08tT9+7dbXXr1rV17drVVqRIEduoUaMc6/z4449mnfPnzzuWJSQkmGXr16932k+7du1sDRo0sN1999220NBQ2+jRo239+vWzNWzY0NalSxdbWFiY7cMPP8z083X9mjVr2jp16mQrXLiw7bHHHnM6zrlz59qKFy9uu/32281+IiMjbW+++Wam/XTr1s1Wp04dW58+fWxr1qyxfM9Tp061hYeH29q3b29r3LixrUSJErYVK1aY1xITE219+/Y156BZs2bm8YIFC9w6b6dPnzbrxMTEOJZNmjTJnIM777zTVr9+ffO+059HV9dx9Xxkd3zuvKfrnT9PvofDhw/bqlevbrbv0aOHeTx+/Pgsjx+AfyMoA/A59nA0ePBgW1pamlm2cuVKW1BQkO3gwYM5DspDhw517GfatGlm2ciRIx3b/fvf/7ZFR0dn+vl33XWX7fLly2bZxo0bbcHBwbbVq1eb5/v27bNFRESY47LTYypUqJBt586dTvt54IEHbFevXs3y/f72228mwM+ZM8exTINg1apVbcnJyY5lNWrUMMefm/OWMSjv2bPHVqBAAdvXX39tnut2+kdE+vPoyjo5OR/ZHZ8778mV8+ep9/Cvf/3L1rp1a8fr+rnOnz8/y88EgH+j9AKAzxo+fLgEBQWZx7fddpsEBwebUoicGjZsmGM/rVq1ciyz02WHDx+WpKQkp+3+9re/SWhoqHncokULad++vXz++efm+aeffiolS5Y0Hei++OILs1xLA0qUKGG+lk/vr3/9qzn2rHz55ZcSHR1taoftnnrqKdPhbuPGjXl63vRn16pVS7p162ae63ZPPvlkjtfJyflw53PNbhtXzp+n3kN4eLgp7Tl69Kh5rsdx7733ZnvsAPwXnfkA+CwNKHYFChSQkJAQSU5OzvF+0ndc0xrdrJalpKSYIGRXtWpVp/1Uq1bNUWOsIUxHTZg7d67TOm3btpVSpUo5LStfvny2x6f7rF69utOysmXLSkRERLY1zZ44b4cOHbJ8nzldJyfnw53PNbttXDl/nnoPI0aMkK1bt5rQXbduXenYsaOMHDlSKlasmO3xA/BPBGUAfsneQqtj4dq5E6Kzox0GMz63B6ZixYqZsK0dza7H3hKaFd1nxpZjDe2XLl3KFDI9TVtQf/rpp2zftyvr5OR8eJor589T76Fo0aIye/Zs08nwhx9+kKlTp0qzZs1Mh1AN5gACC6UXAPySvQUv/YgVVmML58b8+fMdj+Pi4mT58uXSpk0b87xLly5m5IlVq1Y5baOjLWQ14kJWtJRAR3LQES3stFWzUKFC0qRJk1y/j+v9bA2QWnpip6Nz5HQdT56PnHLl/HnqPdhLLjQUd+rUSd544w0z2oY7Lf8AfB8tygD8kn61rkFlyJAh8uijj8rx48fl448/9ujP0P3pkHM33XSTGcNY62AHDhxoXrvrrrtMnbPWvI4aNUpq1Kghu3fvNrPmff/996bl0VX6Pu655x7zNb7WRWsof/nll+WZZ54x7zMvde7cWW6//XbzfvR9aOibOXNmjtfx5PnIKVfOn6few7Rp02T9+vXm50VFRcknn3wit9xyi9SuXTvP3h8A76FFGYDP0dY67ZiVMVw98MADTrWg2iI4YMAA2bBhg+l0t3LlSrOd/et2q/3o1+u6rHDhwk71r7qsYMGCTtutXr1aKlSoIJs2bTIdtrRDl71zn3rnnXdMy6WO8btu3ToTyjRE2Wtfs3ofVrSzmXZA0/rXY8eOyWeffZZpYhENcDVr1szVebOacETHiX744Ydly5YtphVW30P68+jqOu6ej4yfa07fk6vnzxPvYcKECSaA6x9m+v9dr169zP8XWjMNIPAE6dAX3j4IAAAAwNfQogwAAABYICgDAAAAFgjKAAAAgAWCMgAAAGCBoAwAAABYICgDAAAAFgjKAAAAgAWCMgAAAGCBoAwAAABYICgDAAAAFgjKAAAAgAWCMgAAACCZ/T/MuP7UaGgkngAAAABJRU5ErkJggg==", + "text/plain": [ + "
" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "# 6.2 Hyperparameter Tuning: Hidden Nodes Sweep\n", + "hidden_nodes_options = [10, 50, 100, 200, 500]\n", + "performances_hn = []\n", + "optimal_lr = 0.2 # Fixed learning rate from previous experiment\n", + "\n", + "print(\"Starting Hidden Nodes sweep. This will take a while...\")\n", + "\n", + "for hn in hidden_nodes_options:\n", + " print(f\"Training network with {hn} hidden nodes...\")\n", + " testANN = ann(784, hn, 10)\n", + " \n", + " # Train 1 epoch\n", + " for record in list: \n", + " values = record.split(\",\")\n", + " data = np.asarray(values[1:], dtype=float) / 255.0 * 0.99 + 0.01\n", + " target = np.zeros(10) + 0.01\n", + " target[int(values[0])] = 0.99\n", + " testANN.backpropagation(data, target, optimal_lr)\n", + " \n", + " # Evaluate on the Test Set\n", + " score = 0\n", + " for record in list2:\n", + " values = record.split(\",\")\n", + " correct_label = int(values[0])\n", + " data = np.asarray(values[1:], dtype=float) / 255.0 * 0.99 + 0.01\n", + " \n", + " outputs = testANN.feedforward(data)\n", + " if np.argmax(outputs) == correct_label:\n", + " score += 1\n", + " \n", + " # Calculate performance\n", + " performance = score / len(list2)\n", + " performances_hn.append(performance)\n", + " print(f\"Performance for {hn} nodes: {performance:.4f}\\n\")\n", + "\n", + "# Plotting the exact graph requested\n", + "plt.figure(figsize=(8, 5))\n", + "# marker='D' creates the diamond shapes seen in the reference image\n", + "plt.plot(hidden_nodes_options, performances_hn, marker='D', markersize=6, color='#003366', linewidth=1.5)\n", + "\n", + "# Formatting axes\n", + "plt.xlabel(\"number of hidden nodes\")\n", + "plt.ylabel(\"performance\")\n", + "\n", + "# Setting axes limits and ticks to match the image\n", + "plt.xlim(0, 600)\n", + "plt.xticks(np.arange(0, 601, 100))\n", + "plt.ylim(0.6, 1.0)\n", + "plt.yticks(np.arange(0.6, 1.05, 0.05))\n", + "\n", + "# Adding horizontal grid lines\n", + "plt.grid(axis='y', linestyle='-', alpha=0.7)\n", + "\n", + "plt.show()" + ] + }, + { + "cell_type": "markdown", + "id": "e1ce2d95", + "metadata": {}, + "source": [ + "## 7. Model Selection and Architecture Evaluation\n", + "\n", + "**Which is the best ANN and how do you select which network is better?**\n", + "The best network is the one that achieves the highest accuracy on the Test Set while keeping the Cost Function (Loss) to a minimum, without falling into Overfitting. It is selected by comparing different combinations of hyperparameters (Hidden nodes and Learning Rate), evaluating them with data the network never saw during training. The winning network is the one that best generalizes to new data, not the one that memorizes the training data.\n", + "\n", + "## 8. Microcontroller Deployment Strategy\n", + "\n", + "**How do we implement it on a microcontroller?**\n", + "Once the network is trained on the computer, we extract the final weight matrices ($W_{ih}$ and $W_{ho}$) and export them as constant arrays (`const float`) in C/C++ language. On the microcontroller, only the inference stage (Feedforward) is programmed (matrix multiplication and the sigmoid function). The Backpropagation algorithm is completely omitted, which saves the microcontroller's limited memory and processing capacity, allowing for real-time signal processing." + ] } ], "metadata": { diff --git a/README.md b/README.md index 10f6897..9b20f2d 100644 --- a/README.md +++ b/README.md @@ -197,13 +197,37 @@ MyANN.feedforward([0.1, 0.2, 0.3]) ### 2.3 Backpropagation (Training) The core learning algorithm. It calculates the prediction error, propagates it backward, and dynamically updates the weight matrices using gradient descent and the chain rule. +#### Mathematical Derivation of the Cost Function and Gradient -The Gradient of Error -$$ \frac{\partial E}{\partial w_{ho}}= -e_o\cdot \sigma \left(w_{ho} O_h\right) \left(1-\sigma\left (w_{ho} O_h\right) \right) O_h $$ +To thoroughly understand the network's learning mechanics, we must derive the gradient of the error with respect to the synaptic weights. This procedure uses the **Chain Rule** from calculus and establishes the mathematical foundation for the Gradient Descent optimization strategy used in our backpropagation algorithm. -Thus, +**1. The Cost Function (SSE)** +We define the Total Error ($E$) using the Sum of Squared Errors: +$$E = \frac{1}{2} \sum (T - O_o)^2$$ +Where $T$ represents the target label and $O_o$ is the predicted output. We define the output error as $e_o = (T - O_o)$. -$$ \frac{\partial E}{\partial w_{ho}}= -e_o\cdot O_o \left(1-O_o \right) O_h $$ +**2. The Chain Rule Application** +To update the weight matrix $w_{ho}$ (connecting the hidden layer to the output layer), we need to determine how a change in $w_{ho}$ impacts the total error $E$. We calculate the partial derivative using the Chain Rule: +$$\frac{\partial E}{\partial w_{ho}} = \frac{\partial E}{\partial O_o} \cdot \frac{\partial O_o}{\partial X_o} \cdot \frac{\partial X_o}{\partial w_{ho}}$$ +*(Note: $X_o = w_{ho} \cdot O_h$ represents the raw signal entering the output node before activation).* + +**3. Solving the Partial Derivatives** +* **Error derivative:** How the total error changes with respect to the final output. + $$\frac{\partial E}{\partial O_o} = -(T - O_o) = -e_o$$ +* **Activation derivative:** The derivative of the Sigmoid activation function $\sigma(X_o)$. + $$\frac{\partial O_o}{\partial X_o} = \sigma(X_o)(1 - \sigma(X_o)) = O_o(1 - O_o)$$ +* **Weight derivative:** How the raw input $X_o$ changes with respect to the weight matrix $w_{ho}$. This evaluates directly to the output of the preceding hidden layer $O_h$. + $$\frac{\partial X_o}{\partial w_{ho}} = O_h$$ + +**4. Final Gradient Equation** +Multiplying these individual derivatives yields the final gradient of the error for $w_{ho}$: +$$\frac{\partial E}{\partial w_{ho}}= -e_o\cdot \sigma \left(w_{ho} O_h\right) \left(1-\sigma\left (w_{ho} O_h\right) \right) O_h$$ + +Thus, by substituting the activated output $O_o$, we arrive at the simplified expression: +$$\frac{\partial E}{\partial w_{ho}}= -e_o\cdot O_o \left(1-O_o \right) O_h$$ + +This precise formulation dictates the weight update rule programmed in our `backpropagation` method, scaled by the learning rate ($\eta$) to ensure stable convergence: +$$w_{ho_{new}} = w_{ho} + \eta \cdot e_o \cdot O_o(1 - O_o) \cdot O_h^T$$ ```python @@ -324,8 +348,9 @@ MyANN = ann(inputNodes, hiddenNodes, outNodes) ```python # Iterative training loop -epoch = 1 -for e in range (epoch): +epochs = 5 +for e in range(epochs): + total_loss = 0 for record in list: values = record.split(",") @@ -337,12 +362,26 @@ for e in range (epoch): target = np.zeros(outNodes) + 0.01 target[index] = 0.99 + # Calculate loss before updating weights + output = MyANN.feedforward(data) + # Using SSE formulation: 0.5 * sum((target - output)^2) + loss = np.sum(0.5 * (target.reshape(-1, 1) - output)**2) + total_loss += loss + + # Train MyANN.backpropagation(data, target, learningRate) - pass - pass + average_loss = total_loss / len(list) + print(f"Epoch {e+1}/{epochs} - Average Loss: {average_loss:.4f}") ``` + Epoch 1/5 - Average Loss: 0.0972 + Epoch 2/5 - Average Loss: 0.0559 + Epoch 3/5 - Average Loss: 0.0461 + Epoch 4/5 - Average Loss: 0.0403 + Epoch 5/5 - Average Loss: 0.0361 + + ## 5. Validation & Inference Evaluating model performance using unseen test data. A new sample is normalized and processed to extract the final prediction vector, which is then visually compared to the ground truth image. @@ -362,9 +401,40 @@ file2.close +```python +#Test Set Evaluation (Network Score) +scorecard = [] + +for record in list2: + values = record.split(",") + correct_label = int(values[0]) + + # Normalize input + data = np.asarray(values[1:], dtype=float) / 255.0 * 0.99 + 0.01 + + # Get network prediction + outputs = MyANN.feedforward(data) + # The index of the highest value corresponds to the predicted class + predicted_label = np.argmax(outputs) + + # Append 1 if correct, 0 if incorrect + if predicted_label == correct_label: + scorecard.append(1) + else: + scorecard.append(0) + +scorecard_array = np.asarray(scorecard) +accuracy = scorecard_array.sum() / scorecard_array.size +print(f"Network Accuracy (Score) on Test Set: {accuracy * 100:.2f}%") +``` + + Network Accuracy (Score) on Test Set: 95.27% + + + ```python # Inference on sample 500 -values = list2[500].split(",") +values = list2[700].split(",") data = np.asarray(values[1:], dtype=int)/255*0.99+0.01 # Display probability vector for the 10 classes @@ -374,16 +444,16 @@ MyANN.feedforward(data) - array([[1.12958417e-03], - [7.04806122e-03], - [1.33450332e-03], - [9.98132913e-01], - [1.21412650e-04], - [6.66539345e-03], - [2.96287176e-05], - [1.83308214e-03], - [8.11709452e-04], - [1.75208890e-04]]) + array([[3.92210113e-05], + [9.64025823e-01], + [1.22524901e-03], + [1.47886768e-02], + [1.61851002e-03], + [3.98689305e-03], + [7.74585782e-05], + [2.65175424e-03], + [4.98386616e-03], + [2.37478194e-03]]) @@ -397,6 +467,194 @@ plt.show() -![png](README_files/README_32_0.png) +![png](README_files/README_33_0.png) + + + +## 6.1 Hyperparameter Tuning: Learning Rate Impact +To optimize the network's performance, we evaluate the impact of the Learning Rate ($\eta$) on the final classification accuracy. The network is trained across a sweep of different learning rates `[0.01, 0.1, 0.2, 0.3, 0.6, 0.9]` while keeping the hidden nodes constant (100 nodes). + +The results are plotted to identify the optimal step size for the Gradient Descent algorithm, avoiding both slow convergence (values too close to 0) and divergent oscillations (values too close to 1). + + +```python +# 6. Hyperparameter Tuning: Learning Rate Sweep +learning_rates = [0.01, 0.1, 0.2, 0.3, 0.6, 0.9] +performances = [] +hidden_nodes_baseline = 100 + +print("Starting Learning Rate sweep. This may take a few minutes...") + +for lr in learning_rates: + print(f"Training network with Learning Rate: {lr}...") + # Initialize a fresh network for each test + testANN = ann(784, hidden_nodes_baseline, 10) + + # Train 1 epoch + for record in list: + values = record.split(",") + data = np.asarray(values[1:], dtype=float) / 255.0 * 0.99 + 0.01 + target = np.zeros(10) + 0.01 + target[int(values[0])] = 0.99 + testANN.backpropagation(data, target, lr) + + # Evaluate on the Test Set + score = 0 + for record in list2: + values = record.split(",") + correct_label = int(values[0]) + data = np.asarray(values[1:], dtype=float) / 255.0 * 0.99 + 0.01 + + outputs = testANN.feedforward(data) + if np.argmax(outputs) == correct_label: + score += 1 + + # Calculate performance (accuracy as a decimal between 0 and 1) + performance = score / len(list2) + performances.append(performance) + print(f"Performance for LR {lr}: {performance:.4f}\n") + +# Plotting the exact graph requested +plt.figure(figsize=(8, 5)) +plt.plot(learning_rates, performances, marker='s', markersize=8, color='#003366', linewidth=1.5) + +# Formatting to match the requested style +plt.title("Performance vs. Learning Rate") +plt.xlabel("learning rate") +plt.ylabel("performance") + +# Setting axes limits and ticks +plt.xlim(0, 1) +plt.xticks(np.arange(0, 1.1, 0.1)) +plt.ylim(0.8, 0.98) +plt.yticks(np.arange(0.8, 1.0, 0.02)) + +# Adding horizontal grid lines +plt.grid(axis='y', linestyle='-', alpha=0.7) + +plt.show() +``` + + Starting Learning Rate sweep. This may take a few minutes... + Training network with Learning Rate: 0.01... + Performance for LR 0.01: 0.8683 + + Training network with Learning Rate: 0.1... + Performance for LR 0.1: 0.9249 + + Training network with Learning Rate: 0.2... + Performance for LR 0.2: 0.9274 + + Training network with Learning Rate: 0.3... + Performance for LR 0.3: 0.9178 + Training network with Learning Rate: 0.6... + Performance for LR 0.6: 0.8587 + + Training network with Learning Rate: 0.9... + Performance for LR 0.9: 0.8326 + + + + + +![png](README_files/README_35_1.png) + + + +## 6.2 Hyperparameter Tuning: Hidden Nodes Capacity +In this experiment, we evaluate the effect of the network's capacity by varying the number of hidden nodes `[10, 50, 100, 200, 500]`. The Learning Rate is kept constant at $0.2$. + +The resulting curve demonstrates the law of diminishing returns in neural network architecture. While increasing the number of nodes initially provides a massive boost in classification performance, the accuracy plateaus after approximately 200 nodes. Beyond this threshold, adding more nodes significantly increases computational cost and memory footprint without yielding proportional accuracy gains. + + +```python +# 6.2 Hyperparameter Tuning: Hidden Nodes Sweep +hidden_nodes_options = [10, 50, 100, 200, 500] +performances_hn = [] +optimal_lr = 0.2 # Fixed learning rate from previous experiment + +print("Starting Hidden Nodes sweep. This will take a while...") + +for hn in hidden_nodes_options: + print(f"Training network with {hn} hidden nodes...") + testANN = ann(784, hn, 10) + + # Train 1 epoch + for record in list: + values = record.split(",") + data = np.asarray(values[1:], dtype=float) / 255.0 * 0.99 + 0.01 + target = np.zeros(10) + 0.01 + target[int(values[0])] = 0.99 + testANN.backpropagation(data, target, optimal_lr) + + # Evaluate on the Test Set + score = 0 + for record in list2: + values = record.split(",") + correct_label = int(values[0]) + data = np.asarray(values[1:], dtype=float) / 255.0 * 0.99 + 0.01 + + outputs = testANN.feedforward(data) + if np.argmax(outputs) == correct_label: + score += 1 + + # Calculate performance + performance = score / len(list2) + performances_hn.append(performance) + print(f"Performance for {hn} nodes: {performance:.4f}\n") + +# Plotting the exact graph requested +plt.figure(figsize=(8, 5)) +# marker='D' creates the diamond shapes seen in the reference image +plt.plot(hidden_nodes_options, performances_hn, marker='D', markersize=6, color='#003366', linewidth=1.5) + +# Formatting axes +plt.xlabel("number of hidden nodes") +plt.ylabel("performance") + +# Setting axes limits and ticks to match the image +plt.xlim(0, 600) +plt.xticks(np.arange(0, 601, 100)) +plt.ylim(0.6, 1.0) +plt.yticks(np.arange(0.6, 1.05, 0.05)) + +# Adding horizontal grid lines +plt.grid(axis='y', linestyle='-', alpha=0.7) + +plt.show() +``` + + Starting Hidden Nodes sweep. This will take a while... + Training network with 10 hidden nodes... + Performance for 10 nodes: 0.8064 + + Training network with 50 hidden nodes... + Performance for 50 nodes: 0.9183 + + Training network with 100 hidden nodes... + Performance for 100 nodes: 0.9274 + + Training network with 200 hidden nodes... + Performance for 200 nodes: 0.9344 + + Training network with 500 hidden nodes... + Performance for 500 nodes: 0.9167 + + + + + +![png](README_files/README_37_1.png) + + + +## 7. Model Selection and Architecture Evaluation + +**Which is the best ANN and how do you select which network is better?** +The best network is the one that achieves the highest accuracy on the Test Set while keeping the Cost Function (Loss) to a minimum, without falling into Overfitting. It is selected by comparing different combinations of hyperparameters (Hidden nodes and Learning Rate), evaluating them with data the network never saw during training. The winning network is the one that best generalizes to new data, not the one that memorizes the training data. + +## 8. Microcontroller Deployment Strategy +**How do we implement it on a microcontroller?** +Once the network is trained on the computer, we extract the final weight matrices ($W_{ih}$ and $W_{ho}$) and export them as constant arrays (`const float`) in C/C++ language. On the microcontroller, only the inference stage (Feedforward) is programmed (matrix multiplication and the sigmoid function). The Backpropagation algorithm is completely omitted, which saves the microcontroller's limited memory and processing capacity, allowing for real-time signal processing. diff --git a/README_files/README_32_0.png b/README_files/README_32_0.png deleted file mode 100644 index 55c3ad2..0000000 Binary files a/README_files/README_32_0.png and /dev/null differ diff --git a/README_files/README_33_0.png b/README_files/README_33_0.png new file mode 100644 index 0000000..4264ffc Binary files /dev/null and b/README_files/README_33_0.png differ diff --git a/README_files/README_35_1.png b/README_files/README_35_1.png new file mode 100644 index 0000000..a308e5f Binary files /dev/null and b/README_files/README_35_1.png differ diff --git a/README_files/README_37_1.png b/README_files/README_37_1.png new file mode 100644 index 0000000..35a5236 Binary files /dev/null and b/README_files/README_37_1.png differ