001_micrograd/micrograd.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "e556cca0",
   "metadata": {},
   "source": [
    "# micrograd \n",
    "----\n",
    "* Inspired by Andrej Karpathy's [\"The spelled-out intro to neural networks & backpropagation: building micrograd\"](https://www.youtube.com/watch?v=VMj-3S1tku0&t=4822s)\n",
    "\n",
    "* Follow-up *micrograd* exercises to practice can be found in this [notebook](https://colab.research.google.com/drive/1FPTx1RXtBfc4MaTkf7viZZD4U2F9gtKN?usp=sharing) by Andrej Karpathy.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "efa0b203",
   "metadata": {},
   "source": [
    "# Table of Contents\n",
    "------------------\n",
    "- [0. Introduction](#0)<br>\n",
    "    - [0.1. Derivatives: An Overview](#001)\n",
    "- [1. Micrograd: An Autograd Engine](#1)\n",
    "- [2. Micrograd: DAG Visualization](#2)\n",
    "- [3. Micrograd: Backpropagation](#3)\n",
    "    - [3.1. Manual Backpropagation Example 001: Simple Expression](#301)\n",
    "    - [3.2. Preview of a Single Optimization Step](#302)\n",
    "- [4. Micrograd: Neurons](#4)\n",
    "    - [4.1. Biological Neuron vs. Artificial Neuron](#401)\n",
    "    - [4.2. Perceptron](#402)\n",
    "    - [4.3. Activation Functions](#403)\n",
    "    - [4.4. Manual Backpropagation Example 002: A Neuron](#404)\n",
    "- [5. Micrograd: Implementing Backward Function for Each Operation](#5)\n",
    "- [6. Micrograd: Fixing a Backprop Bug when One Node is Used Multiple Times](#6)\n",
    "    - [6.1. Multivariate Chain Rule for Derivatives](#601)\n",
    "- [7. Micrograd: Adding More Mathematical Operations](#7)\n",
    "    - [7.1. Breaking up `tanh` Function into its Parts](#701)\n",
    "- [8. Micrograd: A `PyTorch` Comparison](#8)\n",
    "- [9. Micrograd: Building a NN Library (Multi-Layer Perceptron, MLP)](#9)\n",
    "    - [9.1. Neural Network architectures](#901)\n",
    "- [10. Micrograd: Creating a Tiny Dataset & Writing the Loss Function](#10)\n",
    "- [11. Micrograd: Neural Net Parameters](#11)\n",
    "- [12. Micrograd: Manual Gradient Descent Optimization & Training the MLP](#12)\n",
    "- [13. Micrograd: Summary](#13)\n",
    "\n",
    "\n",
    "\n",
    "# Appendix\n",
    "---------------\n",
    "## Figures\n",
    "- [A1. L-R: A Cartoon Diagram of a Biological Neuron & its Mathematical Model.](#a1)\n",
    "- [A2. A Single-Layer Perceptron structure.](#a2)\n",
    "- [A3. Topological Sort.](#a3)\n",
    "- [A4. L-R: A Diagram of Two Multi-Layer Perceptrons (MLP): a 2-layer NN, a 3-layer NN.](#a4)\n",
    "\n",
    "\n",
    "## Tables\n",
    "- [B1. Activation Functions](#b1)\n",
    "\n",
    "## Equations\n",
    "- [C1. Mathematical definition of f(x) being differentiable at x](#c1)\n",
    "- [C2. Mathematical representation of a NN weighted sum function](#c2)\n",
    "- [C3. Chain rule for multivariable functions: example](#c3)\n",
    "- [C4. Mean-Squared Error (MSE)](#c4)\n",
    "\n",
    "\n",
    "## Definitions\n",
    "- [1. backpropagation](#d1)\n",
    "- [2. micrograd](#d2)\n",
    "- [3. gradient descent](#d3)\n",
    "\n",
    "\n",
    "## [References](#r1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b23dc28d",
   "metadata": {},
   "source": [
    "-----------\n",
    "<br><br>\n",
    "# 0. Introduction<a id=\"0\"></a>\n",
    "---------------------------------\n",
    "This is the most step-by-step spelled-out explanation of backpropagation and training of neural networks. It only assumes basic knowledge of Python and a vague recollection of calculus from high school. \n",
    "\n",
    "This tutorial covers an introduction into the basic mechanisms of neural networks (NNs). NNs are very common in image classification, text sequence recognition and other applications. We're going to cover some fundamental questions about NNs:\n",
    "1. What are neural networks?\n",
    "    - NNs are mathematical functions with inputs and an output.\n",
    "2. What does it mean to \"train\" neural networks? \n",
    "    - It means tuning the parameters of the NN model to minimize the loss function so as to get the predicted values as close as possible to the target values (ground truth).\n",
    "3. How do we tune/manipulate neural networks? \n",
    "    - Here comes the backpropagation algorithm, which is useful for training neural networks.\n",
    "\n",
    "We implement a tiny autograd engine called **Micrograd** to learn and understand neural networks and backpropagation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "1a370efb",
   "metadata": {},
   "outputs": [],
   "source": [
    "import math\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b3d73d21",
   "metadata": {},
   "source": [
    "## 0.1. Derivatives: An Overview<a id=\"001\"></a>\n",
    "------\n",
    "Let’s define a simple function and try to plot it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "c0ce22d9",
   "metadata": {},
   "outputs": [],
   "source": [
    "def f(x):\n",
    "    return 3*x**2 - 4*x + 5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "bf2ee07c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "20.0"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "f(3.0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "a08a7b3e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([-5.  , -4.75, -4.5 , -4.25, -4.  , -3.75, -3.5 , -3.25, -3.  ,\n",
       "       -2.75, -2.5 , -2.25, -2.  , -1.75, -1.5 , -1.25, -1.  , -0.75,\n",
       "       -0.5 , -0.25,  0.  ,  0.25,  0.5 ,  0.75,  1.  ,  1.25,  1.5 ,\n",
       "        1.75,  2.  ,  2.25,  2.5 ,  2.75,  3.  ,  3.25,  3.5 ,  3.75,\n",
       "        4.  ,  4.25,  4.5 ,  4.75])"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "xs = np.arange(-5, 5, 0.25)\n",
    "xs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "38d634f2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([100.    ,  91.6875,  83.75  ,  76.1875,  69.    ,  62.1875,\n",
       "        55.75  ,  49.6875,  44.    ,  38.6875,  33.75  ,  29.1875,\n",
       "        25.    ,  21.1875,  17.75  ,  14.6875,  12.    ,   9.6875,\n",
       "         7.75  ,   6.1875,   5.    ,   4.1875,   3.75  ,   3.6875,\n",
       "         4.    ,   4.6875,   5.75  ,   7.1875,   9.    ,  11.1875,\n",
       "        13.75  ,  16.6875,  20.    ,  23.6875,  27.75  ,  32.1875,\n",
       "        37.    ,  42.1875,  47.75  ,  53.6875])"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ys = f(xs)\n",
    "ys"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "be53aa2c",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD4CAYAAAAXUaZHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nO3dd3jV5f3/8ec7e0EGCTsQIAgyBCSkiBvcVUTqwqo4WtRaV+nX0mWXVauto3XUDVbrwFE3KjLEARhAZiCEHQiQEJIwErLu3x85+KOVEbI+Z7we18VFzuecw+d1rvZ65fY+9+f+mHMOEREJLmFeBxARkeanchcRCUIqdxGRIKRyFxEJQip3EZEgFOF1AIDU1FSXkZHhdQwRkYCyYMGCYudc2sGe84tyz8jIICcnx+sYIiIBxcw2HOo5TcuIiAQhlbuISBBSuYuIBCGVu4hIEFK5i4gEoSOWu5k9Z2bbzWzZAcdSzOwTM1vt+zvZd9zM7O9mlm9mS8zs+JYMLyIiB9eQkftk4Jz/OTYJ+NQ51xv41PcY4Fygt+/PBOCJ5okpIiJH44jl7pz7DCj5n8MXAlN8P08Bxhxw/AVXby6QZGadmivs/1paUMZfpq1E2xaLiPy3xs65d3DOFfp+3gp08P3cBdh0wOsKfMe+w8wmmFmOmeUUFRU1KsSiTTt5YtYacjbsbNT7RUSCVZO/UHX1w+ajHjo7555yzmU557LS0g569ewRXTI0neS4SJ6cvbZR7xcRCVaNLfdt+6dbfH9v9x3fDKQf8LquvmMtIjYqnKtOyGB67jbyt+9qqdOIiAScxpb7O8B438/jgbcPOH61b9XMcKDsgOmbFjH+hO5ER4Tx9GfrWvI0IiIBpSFLIV8GvgL6mFmBmV0P3AecaWargTN8jwE+ANYC+cDTwE9aJPUB2iVEc0lWV95atJnt5ZUtfToRkYBwxF0hnXPjDvHUqIO81gE3NzXU0frRST15ad5GJn+5njvP6dvapxcR8TtBcYVqRmo85/TvyItzN7B7X43XcUREPBcU5Q4w4ZSelFfW8OrXm478YhGRIBc05T6kWzLZGSk89/k6qmvrvI4jIuKpoCl3qB+9by6t4IOlLbpAR0TE7wVVuY/s255eafH8c/ZabUkgIiEtqMo9LMyYcEpPcgvL+Ty/2Os4IiKeCapyBxgzpAtpbaJ56jNtSSAioSvoyj06IpxrRmQwZ3Uxy7eUeR1HRMQTQVfuAFd+rzvxUeE8rdG7iISooCz3xLhILs/uxrtLCtlcWuF1HBGRVheU5Q5w3Uk9AHjuc20oJiKhJ2jLvUtSLKMHdebl+RvZuafK6zgiIq0qaMsd4MZTe7G3qpbnv1zvdRQRkVYV1OXep2MbzurXgclfrNOGYiISUoK63AFuPj2T8soaXpy7wesoIiKtJujLfVB6Eif3TuWZOeuorK71Oo6ISKsI+nIH+MlpmRTv3sdrOdoOWERCQ0iU+/CeKQztnsyTs9dqO2ARCQkhUe5mxk9Pz2RzaQX/WbTZ6zgiIi0uJMod4LQ+afTr1JYnZq2htk7bAYtIcAuZcjczbj49k7XFe5i2bKvXcUREWlTIlDvAOQM60jMtnkdn5utmHiIS1EKq3MPDjJtO7UVuYTkzV233Oo6ISIsJqXKH+pt5dEmK5dEZGr2LSPAKuXKPDA/jhlN7snBjKXPXlngdR0SkRYRcuQNcmpVOakI0j8/K9zqKiEiLCMlyj4kM50cn92DO6mIWbyr1Oo6ISLMLyXIHuHJ4dxJjI/nHjNVeRxERaXYhW+4J0RFcf1IPpuduZ2mBbqQtIsElZMsd4NoTM0iMjeSRT/O8jiIi0qxCutzbxETy45PrR+9LCjT3LiLBI6TLHWD8iAyS4iJ5eLrm3kUkeDSp3M3sDjNbbmbLzOxlM4sxsx5mNs/M8s3sVTOLaq6wLaF+9N6TGSu3841WzohIkGh0uZtZF+BWIMs5NwAIBy4H/gI85JzLBHYC1zdH0Ja0f/T+yHTNvYtIcGjqtEwEEGtmEUAcUAiMBF73PT8FGNPEc7S4hOgIfnxyT2auKmLRxp1exxERabJGl7tzbjPwV2Aj9aVeBiwASp1zNb6XFQBdDvZ+M5tgZjlmllNUVNTYGM1m/IgMkjX3LiJBoinTMsnAhUAPoDMQD5zT0Pc7555yzmU557LS0tIaG6PZJERHMOGUXszOK2KhRu8iEuCaMi1zBrDOOVfknKsG3gROBJJ80zQAXYGAua/d1Sd0JyU+SqN3EQl4TSn3jcBwM4szMwNGASuAmcDFvteMB95uWsTWEx8dwYRTevJZXhELNmj0LiKBqylz7vOo/+J0IbDU9289BfwC+JmZ5QPtgGebIWerufqE7rSLj+JhrZwRkQDWpNUyzrnfOef6OucGOOeucs7tc86tdc5lO+cynXOXOOf2NVfY1hAXFcENp/ZkzupictZrv3cRCUwhf4XqwVw5vDupCZp7F5HApXI/iLioCG44pRef5xfztUbvIhKAVO6HcOXw7qS1ieaBaat0r1URCTgq90OIjQrn1pGZzF9fwqw87y+yEhE5Gir3w7hsWDfSU2J5YNoq6uo0eheRwKFyP4yoiDAmntmHFYXlvL+00Os4IiINpnI/gtGDOtO3Yxv+9vEqqmvrvI4jItIgKvcjCAsz/u/sPqzfsZepOQVexxERaRCVewOM7Nueod2TeeTTPCqra72OIyJyRCr3BjAzfnFOX7aV72PKl+u9jiMickQq9wbK7pHCaX3SeHzWGsoqqr2OIyJyWCr3o/B/Z/ehrKKapz9b63UUEZHDUrkfhf6dE7lgUGee/XwdRbsCaj80EQkxKvej9LMzj6Gqto5HZ2hTMRHxXyr3o9QjNZ7LhqXz7/kb2VSy1+s4IiIHpXJvhFtH9ibMjIc+0Q09RMQ/qdwboWNiDNecmMFb32xm5dZyr+OIiHyHyr2Rbjq1F21jIrnng5VeRxER+Q6VeyMlxUVxy8hMPssrYra2BBYRP6Nyb4KrT8ige7s47nk/l1ptCSwifkTl3gRREWFMOqcvq7bt4rWcTV7HERH5lsq9ic4Z0JFhGcn87eM8du+r8TqOiAigcm8yM+PX3+9H8e59PDl7jddxREQAlXuzGJyexOhBnXl6zloKyyq8jiMionJvLv93dh/qHDzw0Sqvo4iIqNybS3pKHNeemMGbCzezbHOZ13FEJMSp3JvRzadnkhIfxd3vr8A5LY0UEe+o3JtR25hIbj+jN3PXljA9d7vXcUQkhKncm9m47G70TIvn3g9zqa6t8zqOiIQolXsziwwP41fnHsvaoj28PH+j13FEJESp3FvAqGPbc0LPdjz0SR5le3W/VRFpfSr3FmBm/Ob8YymrqOah6drzXURaX5PK3cySzOx1M1tpZrlmdoKZpZjZJ2a22vd3cnOFDST9Oydyxfe68a+5G7Tnu4i0uqaO3B8Bpjnn+gKDgFxgEvCpc6438KnvcUiaeGYf2sRE8Lu3l2tppIi0qkaXu5klAqcAzwI456qcc6XAhcAU38umAGOaGjJQJcdH8X9n92HeuhLeW1LodRwRCSFNGbn3AIqA581skZk9Y2bxQAfn3P4m2wp0ONibzWyCmeWYWU5RUfDe7OLyYd0Y0KUt93yQyx7tGikiraQp5R4BHA884ZwbAuzhf6ZgXP1cxEHnI5xzTznnspxzWWlpaU2I4d/Cw4w/jO5PYVklj83M9zqOiISIppR7AVDgnJvne/w69WW/zcw6Afj+DvlLNYd2T2HskC48M2cd64v3eB1HREJAo8vdObcV2GRmfXyHRgErgHeA8b5j44G3m5QwSEw6ty9REWH88b0VXkcRkRDQ1NUytwAvmdkSYDBwD3AfcKaZrQbO8D0Oee3bxnDrqExmrNzOjJXbvI4jIkEuoilvds59A2Qd5KlRTfl3g9U1I3rw6teb+MO7KxjRK5WYyHCvI4lIkNIVqq0oKiKM34/uz4Yde3n283VexxGRIKZyb2Un907j7P4deHRGPltKdUs+EWkZKncP/Ob7/ahzjj9/kOt1FBEJUip3D6SnxPGT0zJ5f0khs1aF/EpREWkBKneP3HhaT3qmxfPbt5dRUVXrdRwRCTIqd49ER4Rzz0UD2VRSwSOfrvY6jogEGZW7h4b3bMclQ7vyzJy12hZYJMQ453hk+moKy1pmYYXK3WO/Ou9Y2sZG8ss3l1JXp22BRULFmws389D0PKbntsz3bip3jyXHR/Hb849l0cZSXpq3wes4ItIKinfv40/vr2Bo92R+mN2tRc6hcvcDYwZ34cTMdtw/bRXbyiu9jiMiLeyP765g775a7hs7kLAwa5FzqNz9gJnx5zED2Vdbxx/eXe51HBFpQTNWbuOdxVu4+fRMendo02LnUbn7iYzUeG4dmckHS7fyaa42FhMJRrv31fCbt5ZxTIcEbjqtV4ueS+XuRyac0otjOiRw19vLddcmkSD0149WUVheyb1jjyMqomXrV+XuR6IiwrjnooFsLq3g4el5XscRkWa0YMNOpny1nvEnZDC0e3KLn0/l7meyMlIYl92N575Yz7LNZV7HEZFmUFVTx6Q3ltCpbQw/P7vPkd/QDFTufmjSOX1JjoviF28sobq2zus4ItJET8xaw+rtu7n7ogEkRDfpNhoNpnL3Q4lxkdw9pj/Lt5TzxKw1XscRkSZYvW0Xj85czehBnRnZt0OrnVfl7qfOGdCJ0YM6848Zq8kt1NYEIoGors4x6c2lxEdHcNcF/Vr13Cp3P/aH0f1JjI1i4muLNT0jEoBemreBBRt28tvv9yM1IbpVz61y92PJ8VH8+aIBrCgs57GZ+V7HEZGjsKlkL/d9uJKTe6cy9vgurX5+lbufO7t/Ry4c3JlHZ+SzfItWz4gEgro6x8+nLsbMuHfsQMxaZouBw1G5B4DfX9CfpLgofj51CVU1mp4R8XfPfbGOeetKuOuCfnRNjvMkg8o9ACTHR3HPRQPI1fSMiN/L376L+z9axRnHtueSoV09y6FyDxBn9e/ImMGdeWxmvi5uEvFT1bV1/Oy1xcRHhXOPR9Mx+6ncA8jvR/cnOT6Kn09drOkZET/0+Mw1LCko488XDaR9mxhPs6jcA0hSXBT3XjSQlVt38egM3XdVxJ8s21zGP2as5sLBnTlvYCev46jcA80Z/TowdkgXHpu1hqUFmp4R8QeV1bX87LVvaJcQxR9HD/A6DqByD0i/u6A/qQlR3P7qIiqqar2OIxLyHvokj7xtu/nLD44jMS7S6ziAyj0gJcZF8uClg1lbvIc/vb/C6zgiIe3r9SU8NWctV3yvG6f1ae91nG+p3APUiZmpTDi5J/+et5Fpy7Z6HUckJO3ZV8PE1xbTNTmWX593rNdx/ovKPYBNPKsPA7q0ZdKbS9haphtri7S2u9/PZdPOvfztksHEt9JWvg2lcg9gURFhPHL5EPZV1zFx6jfU1TmvI4mEjA+WFvLy/I1MOKUn2T1SvI7zHU0udzMLN7NFZvae73EPM5tnZvlm9qqZRTU9phxKr7QEfndBP77I38HTc9Z6HUckJGwq2csv3ljC4PQkfn5W69xZ6Wg1x8j9NiD3gMd/AR5yzmUCO4Hrm+EcchiXDUvn3AEd+evHq3T1qkgLq66t47ZXFoGDf4wbQmS4f06ANCmVmXUFvg8843tswEjgdd9LpgBjmnIOObL9O8+lJkRz68uL2FtV43UkkaD10Cd5LNxYyj1jB5Ke4s2mYA3R1F85DwN3AvuvhW8HlDrn9rdLAXDQjYzNbIKZ5ZhZTlFRURNjSFJcFA9eOph1O/bwx3e1PFKkJXy+upgnZq9hXHY6Fwzq7HWcw2p0uZvZ+cB259yCxrzfOfeUcy7LOZeVlpbW2BhygBN6teOmU3vxyteb+HBpoddxRIJK0a593PHaN2SmJXDX+f29jnNETRm5nwiMNrP1wCvUT8c8AiSZ2f41QV2BzU1KKEfljjOPYVDXRCa9uZQtpRVexxEJCnV1jolTF1NeUc0/rhhCbFS415GOqNHl7pz7pXOuq3MuA7gcmOGc+yEwE7jY97LxwNtNTikNFhlevzyyts5x878XavdIkWbw9Jy1fJZXxF0X9KNvx7Zex2mQlvia9xfAz8wsn/o5+Gdb4BxyGBmp8dx/8XEs2ljKPR/kHvkNInJI32wq5YGPVnHugI5ckd3N6zgN1iyXVDnnZgGzfD+vBbKb49+VxjtvYCeuP6kHz36+juO7JzPaz7/8EfFH5ZXV3PLyQjq0jeG+scd5evONo+WfCzSlWUw6ty9Z3ZOZ9MYS8rfv8jqOSEBxznHn1CVsKa3k7+OG+M1ujw2lcg9ikeFhPHrF8cRFhXPjiwvZs0/r30Ua6onZa5i2fCu/PLcvQ7snex3nqKncg1zHxBj+fvkQ1hbt5pdvLsU57T8jciSf5RXx149WccGgzlx/Ug+v4zSKyj0EjMhMZeJZfXhn8RZe+GqD13FE/Nqmkr3c+soierdvw19+4O1NrptC5R4ibjq1F6P6tufu91ewcONOr+OI+KWKqlpu+NcC6uocT141lLgo/9rG92io3ENEWJjx4KWD6ZgYw09fWkjJniqvI4n4Feccv35rKblby3nk8iFkpMZ7HalJVO4hJDEukid+OJTiPVXc9soiarX/u8i3XvhqA28u2szto47h9L7+c7u8xlK5h5gBXRL54+j+zFldzL26wEkEqL8P6p/eW8Govu25ZWSm13GaReBOKEmjXZ7djdzCcp75fB29OyRw2bDAuepOpLltK6/kJy8tJD0ljgcvG0xYWGB+gfq/NHIPUb89vx8n907lN/9Zxry1O7yOI+KJqpo6fvJS/TUg/7xyKImxgXWh0uGo3ENUhO8Cp/SUOG58cQEbd+z1OpJIq9r/BeqCDTu5/+Lj6NOxjdeRmpXKPYQlxkby7Phh1Dm4fsrX7Kqs9jqSSKt5fNYapi4o4NZRvTn/uODbe0nlHuJ6pMbzxA+PZ23xHm59WStoJDS8u3gLD3y0igsHd+aOM3p7HadFqNyFEZmp/GF0f2auKtIKGgl6CzbsZOLUxQzLSOYvPwisnR6PhlbLCABXDu/O6m27tIJGgtrGHXuZ8EIOnRJjePKqLGIi/f+OSo2lkbt8SytoJJiV7a3m2snzqalzPH/NMFLio7yO1KJU7vKtiPAwHh1Xv4LmhhcXaA94CRpVNXXc9NICNpbs5cmrhtIzLcHrSC1O5S7/JTEukuevGUZEWBhXPzufwjLdZFsCm3OO3/xnKV+u2cF9Y49jeM92XkdqFSp3+Y7u7eKZfO0wyitruPrZ+ZTu1SZjEriemL2G13IKuHVkJj8Y2tXrOK1G5S4HNaBLIk9dPZQNO/Zy/ZQcKqpqvY4kctReX1DA/dN8Sx7PPMbrOK1K5S6HNKJXKg9fPpiFG3fy038vpLq2zutIIg02bdlW7nx9MSdlpnL/xcG75PFQVO5yWOcN7MQfLxzApyu36zZ9EjA+X13MrS8vYlB6Ek9eNZToiOBd8ngoWucuR3TV8O4U7drH3z9dTWpCNJPO7et1JJFDWrBhJxP+lUPPtHgmX5NNfHRo1lxofmo5anec0Zvi3fv45+w1pCZE8aOTe3odSeQ7cgvLufb5+bRvE80L12eTGBc8uzweLZW7NIiZ8acLB1Cyu4q738+lXUIUFw0JnZUH4v/WF+/hqmfnExcVwb+u/x7t28R4HclTmnOXBgsPMx6+fDAn9GzHxNcW887iLV5HEgGgsKyCHz4zjzrnePFH2aSnxHkdyXMqdzkqMZHhPDM+i6yMFG5/ZRHvquDFYzt27+PKZ+ZRVlHNlGuzyWwfXPuyN5bKXY5afHQEz18zjKzuKdymghcP7dxTxfjn51Ows4Jnx2cxsGui15H8hspdGiU+OoLnr60v+Ntf/Yb3lqjgpXUV797HuKfnkrdtN/+8aijfC5FtBRpK5S6Ntr/gj++WxG2vfMP7Swq9jiQhYnt5JZc/NZf1O/bw7PgsTu/T3utIfkflLk0SHx3B5GuzOb5bEre+skgFLy1uS2kFlz75FVtKK5h8bTYn907zOpJfUrlLk9WP4LMZkq6Cl5a1qWQvlz75FTt2V/Gv67NDZofHxmh0uZtZupnNNLMVZrbczG7zHU8xs0/MbLXv7+Tmiyv+KiE6gsnX/f+C1zJJaW7rivdw2ZNfsauyhpd+/D2Gdk/xOpJfa8rIvQaY6JzrBwwHbjazfsAk4FPnXG/gU99jCQH7C35o92Rue2URk79Y53UkCRL523dx2ZNfUVlTx8s/Hs5xXZO8juT3Gl3uzrlC59xC38+7gFygC3AhMMX3sinAmKaGlMCREB3BC9dlc+axHfj9uyu4f9pKbTYmTZJbWM5lT87FAa9OGE6/zm29jhQQmmXO3cwygCHAPKCDc27/pOtWoMMh3jPBzHLMLKeoqKg5YoifiIkM54krhzIuuxuPz1rDna8voUbbBUsjfJlfzKVPfkVkeBivThhO7w66QKmhmlzuZpYAvAHc7pwrP/A5Vz9kO+iwzTn3lHMuyzmXlZamb7uDTXiYcc9FA7htVG+mLijghn8t0A0/5Ki8saCA8c/Pp1NiDG/8ZERI3Pe0OTWp3M0skvpif8k596bv8DYz6+R7vhOwvWkRJVCZGXeceQx3jxnAzFXb+eEzc9m5R7fsk8NzzvHI9NVMnLqYYRkpTL1xBF2SYr2OFXCaslrGgGeBXOfcgwc89Q4w3vfzeODtxseTYHDl8O48/sPjWbalnEt865NFDqa6to47X1/CQ9PzGHt8FyZfm01ibOhu29sUTRm5nwhcBYw0s298f84D7gPONLPVwBm+xxLizhnQiReuy2ZbWSVjH/+S5VvKvI4kfqa8spprn/+aqQsKuHVUb/52ySCiInQpTmOZP6xkyMrKcjk5OV7HkFaQW1jOdZO/ZufeKu6/eBCjB3X2OpL4gS2lFVw3+Wvyt+/mnrEDuTQr3etIAcHMFjjnsg72nH4tSqs6tlNb3vnpSQzsksitLy/i3g9zqa3zfoAh3lm2uYyLHv+CzTvrtxNQsTcPlbu0urQ20bz0o+FcObwbT85ey7WTv6Zsb7XXscQDr369kbFPfEmYGVNvOoGTeqd6HSloqNzFE1ERYdw9ZiD3jh3IV2uKGf3Y5+Rt2+V1LGklldW13Pn6Yn7xxlKyM1J475aT6NtRFyc1J5W7eGpcdjdemTCcvVW1jHnsC6Yt06ZjwW7jjr2MffxLXssp4JaRmUy5Lpt2CdFexwo6Knfx3NDu9SO3Yzq04cYXF/K3j1dpHj5ITV+xjfP/MYfNpRU8d00WE8/qQ3iYeR0rKKncxS90aBvDqzcM59KsrvxjRj6XP/UVm0r2eh1LmklNbR33T1vJj17IoVu7ON675SRG9j3oziTSTFTu4jeiI8L5yw+O48FLB5FbuIvzHpnDW4sKtPFYgNteXsnVz83n8VlrGJedzus3jiA9Jc7rWEEvwusAIgcyM8Ye35VhGSn87LVvuOPVxcxYWcTdYwboSsUA45zjncVbuOvt5VRW1/LAxcdxiZY5thqVu/il9JQ4XplwAk/Myufh6atZsL6EBy8brDvvBIji3fv4zVvLmLZ8K0O6JfHXSwbRSxt/tSpNy4jfCg8zfjqyN2/cNILoyHDGPT2X+z5cSVWNtg/2Zx8sLeSshz5jxsrtTDq3L6/fOELF7gGN3MXvDUpP4r1bTuLu91fwz9lrmJ1XxJ8vGsDx3XQHR3+yc08Vv317Ge8tKeS4ron87ZJB2n/dQ9pbRgLKx8u3ctfby9laXsm47HTuPLsvyfFRXscKeR8v38qv3lpGWUUVt43qzY2n9iIiXBMDLe1we8to5C4B5az+HRmRmcoj0/N47ov1TFu2lUnn9uWSoemEab10q1tTtJs/v5/LjJXb6depLf+6PptjO+lKU3+gkbsErJVby/nNW8vI2bCT47slcfeYgbq/Zispq6jm75+uZsqX64mNDOeWUZlcM6KHtuhtZYcbuavcJaDV1TneWFjAvR+upHRvFeNHZHDHmcfQNkbLJltCbZ3j5fkbefCTPHbureKyrHQmntWHtDbaPsALmpaRoBUWZlySlc6Z/TrwwEermPzlev6zaDM3nNqLq0/oTlyU/i/eXL7ML+aP761g5dZdZPdI4a7z+zGgS6LXseQQNHKXoLKkoJS/fpzHZ3lFpCZEceOpvbhyeHdiIsO9jhawlhaU8fcZq/lkxTa6Jsfyq/OO5dwBHam/06Z4SdMyEnJy1pfw0PQ8vsjfQfs20dx8eiaXZ6cTHaGSbwjnHPPXlfDozHzmrC6mTUwEN5zSkx+d3FO/KP2Iyl1C1ty1O3jw4zzmry+hU2IMPx2ZycVDu6rkD8E5x6xVRTw2M5+cDTtJTYji+pN6cuXwbrTR9xh+R+UuIc05xxf5O/jbJ6tYtLGUdvFRXDYsnXHZ3bSBlU9tnePDZYU8NnMNuYXldEmK5YZTe3JpVrpG6n5M5S5Cfcl/nl/MlC83MGPlNhxwep/2XDm8G6ce0z4k9xXfVLKXqQsKeGNBAZtLK+iZFs9PTsvkwsGdidRFSH5P5S7yPzaXVvDK/I28PH8Txbv30TU5liu+141Ls9JJDfK7Au2tquGDpVt5fcEm5q4twQxOykzliuxunNW/Y0j+kgtUKneRQ6iqqePjFVt5ce4G5q4tITLcGNErlbP6d+DMYzvQvm2M1xGbhXOOnA07mZqzifeXFLKnqpaMdnFcPLQrY4/vSuekWK8jSiOo3EUaIH/7Ll79ehMfLd/GRt9doIZ0S+Ksfh05q3+HgNvZcOeeKubkFzN7VRFzVhexfdc+4qLC+f7ATlySlc6wjGQtZwxwKneRo+CcI2/bbj5evpWPV2xj6eYyAHqlxXPGsR0Y2j2Zwd2SaN/Gv0b1NbV1LC4oZXZeMbPzilhSUIpzkBgbycm9UxnZtz1n9+9IfLQu7AoWKneRJthcWsH0Fdv4eMVW5q0tocZ38+4uSbEMSk9kcHoSg9OTGdClbatdEVtZXUv+9t3kFpaTW7iLlVvLWba5jPLKGsIMBqcnccoxaZx6TBrHdU3SPHqQUrmLNJOKqlqWbynjm02l3/4p2FkB1N9cpEdqPF2SYumcFLdkKDwAAARQSURBVEPnxFg6JcXSOTGGzkmxdEyMadCyQucc5ZU1lOypomTPPnbsrqJkTxVFu/aRt303KwvLWVu8h1rfL5mYyDD6dGxLv05tOSkzlRMz25EUp22QQ4H2lhFpJrFR4WRlpJCVkfLtsaJd+1i8qZTFBaWs2rqLwrJKlm8po3h31XffHxlORJgREW6Eh4UREWaE739sxu59NezcW0V17cEHXV2TYzm2U1vOHdCRvp3a0rdjG7q3i9fIXL5D5S7SRGltojmjXwfO6Nfhv45XVteytaySLWUVbCmtpLC0gvLKamrqHLV1jupaR21d3bePa+ocCVERpCRE0S4+ihTfn3bx0d8e0wVF0lAqd5EWEhMZTkZqPBmp8V5HkRCkS9BERIKQyl1EJAi1SLmb2TlmtsrM8s1sUkucQ0REDq3Zy93MwoHHgHOBfsA4M+vX3OcREZFDa4mRezaQ75xb65yrAl4BLmyB84iIyCG0RLl3ATYd8LjAd+y/mNkEM8sxs5yioqIWiCEiEro8+0LVOfeUcy7LOZeVlpbmVQwRkaDUEuW+GUg/4HFX3zEREWklzb63jJlFAHnAKOpL/WvgCufc8sO8pwjY0KxBWk8qUOx1CA/oc4eeUP3s/vy5uzvnDjr10exXqDrnaszsp8BHQDjw3OGK3feegJ2XMbOcQ23cE8z0uUNPqH72QP3cLbL9gHPuA+CDlvi3RUTkyHSFqohIEFK5N91TXgfwiD536AnVzx6Qn9svbtYhIiLNSyN3EZEgpHIXEQlCKvdmYmYTzcyZWarXWVqLmT1gZivNbImZvWVmSV5nakmhuNupmaWb2UwzW2Fmy83sNq8ztTYzCzezRWb2ntdZjobKvRmYWTpwFrDR6yyt7BNggHPuOOovXPulx3laTAjvdloDTHTO9QOGAzeHyOc+0G1ArtchjpbKvXk8BNwJhNS30865j51zNb6Hc6nfaiJYheRup865QufcQt/Pu6gvue9sBBiszKwr8H3gGa+zHC2VexOZ2YXAZufcYq+zeOw64EOvQ7SgBu12GszMLAMYAszzNkmrepj6gVud10GOlm6Q3QBmNh3oeJCnfg38ivopmaB0uM/unHvb95pfU/+f7y+1ZjZpPWaWALwB3O6cK/c6T2sws/OB7c65BWZ2mtd5jpbKvQGcc2cc7LiZDQR6AIvNDOqnJRaaWbZzbmsrRmwxh/rs+5nZNcD5wCgX3BdNhOxup2YWSX2xv+Sce9PrPK3oRGC0mZ0HxABtzexF59yVHudqEF3E1IzMbD2Q5Zzz1x3kmpWZnQM8CJzqnAvqO640ZrfTYGD1o5YpQIlz7nav83jFN3L/uXPufK+zNJTm3KUpHgXaAJ+Y2Tdm9k+vA7UU3xfH+3c7zQVeC/Zi9zkRuAoY6fvf+BvfSFb8nEbuIiJBSCN3EZEgpHIXEQlCKncRkSCkchcRCUIqdxGRIKRyFxEJQip3EZEg9P8AHOgo1Uw70q4AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(xs, ys)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c4916e64",
   "metadata": {},
   "source": [
    "We want to calculate the derivative of the function at each point. In this example, we can explicitly calculate the derivative of the function. However, this approach is not effective when it comes to neural networks. What is a derivative?<br> \n",
    "\n",
    "On a base level, this is a derivative:\n",
    "\n",
    "*“Lets say I have a function y which is dependent on the value of x. If I nudge x just a tiny bit at a certain point, what would be the rate of change of y in response to that?”*\n",
    "\n",
    "<u>Interpretation</u>: Derivatives indicate the rate of change of a function with respect to that variable surrounding an infinitesimally small region near a particular point:\n",
    "Mathematically, a [derivative](https://en.wikipedia.org/wiki/Derivative) is shown below: \n",
    "<a id=\"c1\"></a>\n",
    "![mathematical definition of f(x) being differentiable at x](_imgs/differentiable_function.png)<br>\n",
    "**Eq. 1: Mathematical definition of f(x) being differentiable at x.** ([Source](https://cs231n.github.io/optimization-2/))<br>\n",
    "\n",
    "\n",
    "We give a slight bump to the variable x and check how the function responds to it. In eq. 1, we calculate the rate of change by dividing the difference between `f(x)` and `f(x+h)` (gotten from nudging `f(x)` by a tiny amount, `h`) by `h`, how much we nudged. Derivative shows the sensitivity of a function to a change in its variables. We will do a small numerical approximation of the derivative below.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "dfa50950",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "14.00000009255109"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "h = 0.00000001\n",
    "x = 3.0\n",
    "(f(x + h) - f(x))/h"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "ffed1099",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-22.00000039920269"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "h = 0.00000001\n",
    "x = -3.0\n",
    "(f(x + h) - f(x))/h"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "83c5ebbf",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2.999378523327323e-06"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "h = 0.000001\n",
    "x = 2/3\n",
    "(f(x + h) - f(x))/h"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7f98a4ea",
   "metadata": {},
   "source": [
    "As seen above, if we take smaller and smaller h values, we can converge to the real gradient of **14, -22 and 0** @ **x = 3, -3 and 2/3** respectively. If the h value is infinitesimally small, then the gradient will be zero. <br>\n",
    "At an optimal point (**x = 2/3**), the gradient is zero! The function won't change in any direction as a result of changing the input at that point.<br><br>\n",
    "Suppose we have multiple input variables. Let's calculate the derivative of `d` WRT `c`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "7ed0b5be",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4.0\n"
     ]
    }
   ],
   "source": [
    "# lets get more complex\n",
    "a = 2.0\n",
    "b = -3.0\n",
    "c = 10.0\n",
    "d = a*b + c\n",
    "print(d)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "bad8b68b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "d1 4.0\n",
      "d2 4.0001\n",
      "slope 0.9999999999976694\n"
     ]
    }
   ],
   "source": [
    "h = 0.0001\n",
    "\n",
    "# inputs\n",
    "a = 2.0\n",
    "b = -3.0\n",
    "c = 10.0\n",
    "\n",
    "d1 = a*b + c\n",
    "c += h\n",
    "d2 = a*b + c\n",
    "\n",
    "print('d1', d1)\n",
    "print('d2', d2)\n",
    "print('slope', (d2 - d1)/h)\n",
    "      "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e8ac1acd",
   "metadata": {},
   "source": [
    "Since **c** is a constant number, an infinitesimal or small change (*h*) in **c** will lead to that exact same change in **d**. Thus, the gradient/derivative is **one**. We can do the same for **a** and **b** to understand how **d** changes.\n",
    "\n",
    "Next, lets discuss Micrograd, its creation, implementation and management. To deploy a neural network and leverage the Micrograd engine, we need separate data structures. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4a2f48ee",
   "metadata": {},
   "source": [
    "---------\n",
    "<br><br>\n",
    "# 1. Micrograd: An Autograd Engine<a id=\"1\"></a>\n",
    "-------------------------------------------------------\n",
    "<a id=\"d2\"></a>\n",
    "<u>Micrograd:</u> A tiny **autograd** (automatic gradient) engine that implements **backpropagation** (reverse-mode autodiff) over a dynamically built **DAG** (Directed Acyclic Graph) and a **small NNs library** on  top of it with a **PyTorch-like API**. It's a scalar-valued autograd-engine. Both python objects, the DAG and NN library, are tiny with ~100 and ~50 lines of code respectively. The DAG only operates over scalar values, so e.g. we chop up each neuron into all of its individual tiny adds and multiplies. However, this is enough to build up entire deep neural nets doing binary classification, as the demo notebook shows. <br><br>\n",
    "Objectives for building **micrograd**:\n",
    "1. Pedagogical/educational purposes\n",
    "2. Understand neural networks (NNs)\n",
    "3. Understand backpropagation (chain-rule & differentiation)\n",
    "4. Understand the math being the same for scalars vs tensors <br>\n",
    "    a. parallelism in computers runs tensors (array of scalars)<br>\n",
    "    b. efficiency of using tensors\n",
    "5. Needed to train NNs\n",
    "\n",
    "In this notebook, we are going to build Micrograd - a minimalistic auto-differentiation (**autodiff**) engine in python. An autodiff/autograd algorithm forms the basis of modern neural network libraries like `PyTorch` and `Tensorflow` and works by automatically calculating gradients in computational graphs (DAGs such as NNs). This tutorial is inspired by this lecture - <u>\"The spelled-out intro to neural networks and backpropagation: building micrograd\"</u> - in Andrej Karpathy's [Neural Networks: Zero to Hero](https://karpathy.ai/zero-to-hero.html) lecture series. The full version of Micrograd (Andrej Karpathy) can be found on [github](https://github.com/karpathy/micrograd).\n",
    "\n",
    "First thing we need to do is figure out a way to represent numbers and mathematical operations in Python to allow for different calculations to be done. In essence, we need a simple robust data structure to serve as a building block and connective tissue for our Micrograd engine.\n",
    "\n",
    "Let’s develop a simple data structure (a core `Value` object) to manage the micrograd."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "0065a63a",
   "metadata": {},
   "outputs": [],
   "source": [
    "class Value:\n",
    "    \"\"\" stores a single scalar value and its gradient \"\"\"\n",
    "\n",
    "    def __init__(self, data, _children=(), _op='', label=''):\n",
    "        self.data = data\n",
    "        self.grad = 0.0\n",
    "        # internal variables used for autograd graph construction\n",
    "        self._backward = lambda: None\n",
    "        self._prev = set(_children)\n",
    "        self._op = _op # the op that produced this node, for graphviz / debugging / etc\n",
    "        self.label = label\n",
    "        \n",
    "    def __repr__(self):\n",
    "        return f\"Value(data={self.data}, grad={self.grad})\"\n",
    "    \n",
    "    def __add__(self, other):\n",
    "        other = other if isinstance(other, Value) else Value(other)\n",
    "        out = Value(self.data + other.data, (self, other), '+')\n",
    "        \n",
    "        def _backward():\n",
    "            self.grad += 1.0 * out.grad\n",
    "            other.grad += 1.0 * out.grad\n",
    "        out._backward = _backward\n",
    "\n",
    "        return out\n",
    "    \n",
    "    def __mul__(self, other):\n",
    "        other = other if isinstance(other, Value) else Value(other)\n",
    "        out = Value(self.data * other.data, (self, other), '*')\n",
    "        \n",
    "        def _backward():\n",
    "            self.grad += other.data * out.grad\n",
    "            other.grad += self.data * out.grad\n",
    "        out._backward = _backward\n",
    "        \n",
    "        return out\n",
    "    \n",
    "    def __pow__(self, other):\n",
    "        assert isinstance(other, (int, float)), \"only supporting int/float powers for now\"\n",
    "        out = Value(self.data**other, (self,), f'**{other}')\n",
    "\n",
    "        def _backward():\n",
    "            self.grad += (other * self.data**(other-1)) * out.grad\n",
    "        out._backward = _backward\n",
    "\n",
    "        return out\n",
    "    \n",
    "    def relu(self):\n",
    "        out = Value(0 if self.data < 0 else self.data, (self,), 'ReLU')\n",
    "\n",
    "        def _backward():\n",
    "            self.grad += (out.data > 0) * out.grad\n",
    "        out._backward = _backward\n",
    "\n",
    "        return out\n",
    "    \n",
    "    def tanh(self):\n",
    "        x = self.data\n",
    "        t = (math.exp(2*x) - 1)/(math.exp(2*x) + 1)\n",
    "        out = Value(t, (self,), 'tanh')\n",
    "        \n",
    "        def _backward():\n",
    "            self.grad += (1 - t**2) * out.grad\n",
    "        out._backward = _backward\n",
    "        \n",
    "        return out\n",
    "    \n",
    "    def exp(self):\n",
    "        x = self.data\n",
    "        t = math.exp(x)\n",
    "        out = Value(t, (self,), 'exp')\n",
    "        \n",
    "        def _backward():\n",
    "            self.grad += t * out.grad\n",
    "        out._backward = _backward\n",
    "        \n",
    "        return out\n",
    "    \n",
    "    def backward(self):\n",
    "        \n",
    "        # build the topological ordered list of all the children in the DAG\n",
    "        topo = []\n",
    "        visited = set()\n",
    "        def build_topo(v):\n",
    "            if v not in visited:\n",
    "                visited.add(v)\n",
    "                for child in v._prev:\n",
    "                    build_topo(child)\n",
    "                topo.append(v)\n",
    "        build_topo(self)\n",
    " \n",
    "        # We must go through the topogical ordered list in the reverse order as per backpropagation\n",
    "        # We must go through one variable at a time and\n",
    "        # then apply the chain rule to get its gradient\n",
    "        self.grad = 1.0\n",
    "        for node in reversed(topo):\n",
    "            node._backward()\n",
    "    \n",
    "    def __neg__(self): # -self\n",
    "        return self * -1\n",
    "\n",
    "    def __radd__(self, other): # other + self\n",
    "        return self + other\n",
    "\n",
    "    def __sub__(self, other): # self - other\n",
    "        return self + (-other)\n",
    "\n",
    "    def __rsub__(self, other): # other - self\n",
    "        return other + (-self)\n",
    "\n",
    "    def __rmul__(self, other): # other * self\n",
    "        return self * other\n",
    "\n",
    "    def __truediv__(self, other): # self / other\n",
    "        return self * other**-1\n",
    "\n",
    "    def __rtruediv__(self, other): # other / self\n",
    "        return other * self**-1\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "e33589af",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Value(data=-8.0, grad=0.0)"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a = Value(2.0,label='a')\n",
    "b = Value(-3.0,label='b')\n",
    "c = Value(10.0,label='c')\n",
    "\n",
    "e = a*b; e.label ='e'\n",
    "d = e+c; d.label = 'd'\n",
    "f = Value(-2.0, label='f')\n",
    "L = d * f\n",
    "L.label = 'L'\n",
    "L"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "65789cfe",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The input nodes of L are {Value(data=4.0, grad=0.0), Value(data=-2.0, grad=0.0)}\n",
      "The label of L is L\n"
     ]
    }
   ],
   "source": [
    "print(f\"The input nodes of L are {L._prev}\")\n",
    "print(f\"The label of L is {L.label}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "2a26ea0b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'*'"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "L._op"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "60fdbc64",
   "metadata": {},
   "source": [
    "This class simply wraps the value and creates an object. Python uses `__repr__` function to internally represent the object. Without `__repr__`, the print function will print out the location of the object.<br>\n",
    "\n",
    "We define addition using `__add__`. Whenever Python sees **a+b**, it internally executes `a.__add__(b)`. Other mathematical operators can also be defined in the same way.<br>\n",
    "\n",
    "When initializing objects, the `_children` tuple is empty. However, when adding or multiplying, the previous elements are added to `_prev` set. Moreover, we need to know how a value was created (keep track of the mathematical operator involved). Thus, we introduce the `_op` attribute.\n",
    "\n",
    "We add the `label`attribute to store the variable name/letter.\n",
    "\n",
    "Now we need to visualize the graph DAG structure. Thus, we introduce the following code.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "99ee719e",
   "metadata": {},
   "source": [
    "-------------\n",
    "<br><br>\n",
    "# 2. Micrograd: DAG Visualization<a id=\"2\"></a>\n",
    "-------------------------------------------------------\n",
    "\n",
    "We need a function to help us visualize the computational graph built by the `Value` object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "ae017cf0",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/Users/CEO/anaconda3/envs/py36/bin/python\n"
     ]
    }
   ],
   "source": [
    "import sys\n",
    "print(sys.executable)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "c83667fe",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('0.19.1', (10, 0, 1))"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import graphviz\n",
    "graphviz.__version__, graphviz.version()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "1e802e87",
   "metadata": {},
   "outputs": [],
   "source": [
    "from graphviz import Digraph\n",
    "\n",
    "def trace(root):\n",
    "    \"\"\"\n",
    "    builds a set of all nodes and edges in a graph\n",
    "    \"\"\"\n",
    "    nodes, edges = set(), set()\n",
    "    def build(v):\n",
    "        if v not in nodes:\n",
    "            nodes.add(v)\n",
    "            for child in v._prev:\n",
    "                edges.add((child, v))\n",
    "                build(child)\n",
    "    build(root)\n",
    "    return nodes, edges\n",
    "\n",
    "def draw_dot(root):\n",
    "    dot = Digraph(format='svg', graph_attr={'rankdir': 'LR'}) # LR = left to right\n",
    "    nodes, edges = trace(root)    \n",
    "    for n in nodes:\n",
    "        uid = str(id(n))\n",
    "        # for any value in the graph,create a rectangular ('record') node for it\n",
    "        dot.node(name = uid, label = \"{ %s | data %.4f | grad %.4f }\" % (n.label, n.data, n.grad), shape='record')\n",
    "        if n._op:\n",
    "            # if this value is a result of some operation, create an op node for it\n",
    "            dot.node(name = uid + n._op, label=n._op)\n",
    "            # and connect this node to it\n",
    "            dot.edge(uid + n._op, uid)\n",
    "    \n",
    "    for n1, n2 in edges:\n",
    "        # connect n1 to the op node of n2\n",
    "        dot.edge(str(id(n1)), str(id(n2)) + n2._op)\n",
    "    \n",
    "    return dot"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2800485f",
   "metadata": {},
   "source": [
    "We created two functions: \n",
    "- `trace`: This function builds and stores a set of all the nodes and edges in a graph via recursion.\n",
    "- `draw_dot`: This function takes a `Value` object and builds the computational graph associated with it. First, it recursively finds all the nodes and edges using the `trace` function and then continuously connect and draw these elements using the `graphviz` library.\n",
    "\n",
    "So far, we've created a `Value` class that allows us to create a simple computational graph built of mathematical operators. We are also able to visualize this graph using the `graphviz` library. However to learn neural network parameters, we need to be able to calculate the gradients for each parameter in the computational graph with respect to the output node. Therefore, we need to understand backpropagation."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e7303eb1",
   "metadata": {},
   "source": [
    "-------------\n",
    "<br><br>\n",
    "# 3. Micrograd: Backpropagation<a id=\"3\"></a>\n",
    "---------------------------------\n",
    "<a id=\"d1\"></a>\n",
    "**Backpropagation** is far more general than the neural networks (NN). It is a gradient estimation mathematical machine used to train NNs. The gradient estimate is used by the optimization algorithm to compute the network parameter updates. It is also known as the [reverse mode of automatic differentiation](https://en.wikipedia.org/wiki/Backpropagation). Backpropagation is an algorithm created to efficiently evaluate the gradient of a loss function *wrt* the weights of a NN. \n",
    "\n",
    "It functions via iterative tuning of the NN weights (chain rule and differentiation implementation) to minimize the loss function and subsequently improve the accuracy of the NN. On a base level, it entails computing the gradient one layer at a time & iterating backward from the last layer to avoid redundant calculations of intermediate terms in the chain rule. Backprop is the mathematical core of any modern deep NN library like `PyTorch` or `JAX`.<br>\n",
    "\n",
    "\n",
    "As a first step, let's calculate the gradients for each parameter of a simple expression manually below:\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "213f92c6",
   "metadata": {},
   "source": [
    "-------------\n",
    "<br><br>\n",
    "## 3.1. Manual Backpropagation example 001: simple expression<a id=\"301\"></a>\n",
    "-------------------------------------------------------\n",
    "In backpropagation, we <u>recursively apply the chain rule backward through the computational graph</u>. \n",
    "\n",
    "Our function has 7 parameters: `a`, `b`, `c`, `d`, `e`, `f`, and `L`. Let us consider the gradients for each parameter with respect to the output `L`."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b8b9b929",
   "metadata": {},
   "source": [
    "Functions:\n",
    "\n",
    "\\begin{align*}\n",
    "a & = 2.0, \\quad b = -3.0, \\quad c = 10.0, \\quad f = -2.0 \\\\\n",
    "e & = a \\cdot b \\\\\n",
    "d & = e + c \\\\\n",
    "L & = d \\cdot f = 8.0 \\\\\n",
    "\\end{align*}\n",
    "\n",
    "\\begin{align*}\n",
    "(a, b) & \\rightarrow e \\\\\n",
    "(e, c) & \\rightarrow d \\\\\n",
    "(d, f) & \\rightarrow L \\\\\n",
    "\\end{align*}\n",
    "Given:\n",
    "\\begin{align*}\n",
    "\\frac{dL}{dd} & = f = -2.0 \\\\\n",
    "\\frac{dL}{df} & = d = e + c = (a)(b) + c = (2)(-3) + 10 = 4.0 \\\\\n",
    "\\frac{dd}{dc} & = 1.0 \\\\\n",
    "\\frac{dd}{de} & = 1.0 \\\\\n",
    "\\frac{de}{da} & = b = -3.0 \\\\\n",
    "\\frac{de}{db} & = a = 2.0 \\\\\n",
    "\\end{align*}\n",
    "\n",
    "\n",
    "We want:\n",
    "\\begin{align*}\n",
    "\\frac{dL}{dc} & = \\frac{dL}{dd} \\cdot \\frac{dd}{dc} = -2.0 \\cdot 1.0 = -2.0 \\\\\n",
    "\\frac{dL}{de} & = \\frac{dL}{dd} \\cdot \\frac{dd}{de} = -2.0 \\cdot 1.0 = -2.0 \\\\\n",
    "\\frac{dL}{da} & = \\frac{dL}{de} \\cdot \\frac{de}{da} = \\frac{dL}{de} \\cdot b = -2.0 \\cdot -3.0 = 6.0 \\\\\n",
    "\\frac{dL}{db} & = \\frac{dL}{de} \\cdot \\frac{de}{db} = \\frac{dL}{de} \\cdot a = -2.0 \\cdot 2.0 = -4.0 \\\\\n",
    "\\end{align*}\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "8c516fa8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Value(data=-8.0, grad=0.0)"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a = Value(2.0,label='a')\n",
    "b = Value(-3.0,label='b')\n",
    "c = Value(10.0,label='c')\n",
    "\n",
    "e = a*b; e.label ='e'\n",
    "d = e+c; d.label = 'd'\n",
    "f = Value(-2.0, label='f')\n",
    "L = d * f\n",
    "L.label = 'L'\n",
    "L"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "3b6e4b63",
   "metadata": {},
   "outputs": [],
   "source": [
    "L.grad = 1.0                  # dL/dL\n",
    "f.grad = d.data               # dL/df\n",
    "d.grad = f.data               # dL/dd\n",
    "c.grad = e.grad = d.grad      # dL/dc, dL/de\n",
    "a.grad = e.grad * b.data      # dL/da\n",
    "b.grad = e.grad * a.data      # dL/db"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "c48d0a91",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 10.0.1 (20240210.2158)\n",
       " -->\n",
       "<!-- Pages: 1 -->\n",
       "<svg width=\"1151pt\" height=\"128pt\"\n",
       " viewBox=\"0.00 0.00 1151.00 128.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 124)\">\n",
       "<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-124 1147,-124 1147,4 -4,4\"/>\n",
       "<!-- 140523648766024 -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>140523648766024</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"953.25,-54.5 953.25,-90.5 1143,-90.5 1143,-54.5 953.25,-54.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"965.38\" y=\"-67.7\" font-family=\"Times,serif\" font-size=\"14.00\">L</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"977.5,-55 977.5,-90.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"1019.62\" y=\"-67.7\" font-family=\"Times,serif\" font-size=\"14.00\">data &#45;8.0000</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"1061.75,-55 1061.75,-90.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"1102.38\" y=\"-67.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 1.0000</text>\n",
       "</g>\n",
       "<!-- 140523648766024* -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>140523648766024*</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"890.25\" cy=\"-72.5\" rx=\"27\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"890.25\" y=\"-67.45\" font-family=\"Times,serif\" font-size=\"14.00\">*</text>\n",
       "</g>\n",
       "<!-- 140523648766024*&#45;&gt;140523648766024 -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>140523648766024*&#45;&gt;140523648766024</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M917.69,-72.5C924.84,-72.5 933.03,-72.5 941.74,-72.5\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"941.51,-76 951.51,-72.5 941.51,-69 941.51,-76\"/>\n",
       "</g>\n",
       "<!-- 140523648766136 -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>140523648766136</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"640.12,-82.5 640.12,-118.5 826.12,-118.5 826.12,-82.5 640.12,-82.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"650.38\" y=\"-95.7\" font-family=\"Times,serif\" font-size=\"14.00\">f</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"660.62,-83 660.62,-118.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"702.75\" y=\"-95.7\" font-family=\"Times,serif\" font-size=\"14.00\">data &#45;2.0000</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"744.88,-83 744.88,-118.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"785.5\" y=\"-95.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 4.0000</text>\n",
       "</g>\n",
       "<!-- 140523648766136&#45;&gt;140523648766024* -->\n",
       "<g id=\"edge4\" class=\"edge\">\n",
       "<title>140523648766136&#45;&gt;140523648766024*</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M826.6,-83.81C835.76,-82.16 844.62,-80.56 852.67,-79.1\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"853.09,-82.58 862.31,-77.36 851.84,-75.7 853.09,-82.58\"/>\n",
       "</g>\n",
       "<!-- 140523648766192 -->\n",
       "<g id=\"node4\" class=\"node\">\n",
       "<title>140523648766192</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"319.88,-55.5 319.88,-91.5 511.88,-91.5 511.88,-55.5 319.88,-55.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"330.88\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">e</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"341.88,-56 341.88,-91.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"384\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">data &#45;6.0000</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"426.12,-56 426.12,-91.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"469\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad &#45;2.0000</text>\n",
       "</g>\n",
       "<!-- 140523648766248+ -->\n",
       "<g id=\"node7\" class=\"node\">\n",
       "<title>140523648766248+</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"576\" cy=\"-45.5\" rx=\"27\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"576\" y=\"-40.45\" font-family=\"Times,serif\" font-size=\"14.00\">+</text>\n",
       "</g>\n",
       "<!-- 140523648766192&#45;&gt;140523648766248+ -->\n",
       "<g id=\"edge8\" class=\"edge\">\n",
       "<title>140523648766192&#45;&gt;140523648766248+</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M512.05,-56.65C521.28,-55.01 530.2,-53.43 538.29,-52\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"538.74,-55.47 547.98,-50.28 537.52,-48.58 538.74,-55.47\"/>\n",
       "</g>\n",
       "<!-- 140523648766192* -->\n",
       "<g id=\"node5\" class=\"node\">\n",
       "<title>140523648766192*</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"255.75\" cy=\"-73.5\" rx=\"27\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"255.75\" y=\"-68.45\" font-family=\"Times,serif\" font-size=\"14.00\">*</text>\n",
       "</g>\n",
       "<!-- 140523648766192*&#45;&gt;140523648766192 -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>140523648766192*&#45;&gt;140523648766192</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M283.22,-73.5C290.59,-73.5 299.09,-73.5 308.13,-73.5\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"307.89,-77 317.89,-73.5 307.89,-70 307.89,-77\"/>\n",
       "</g>\n",
       "<!-- 140523648766248 -->\n",
       "<g id=\"node6\" class=\"node\">\n",
       "<title>140523648766248</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"639,-27.5 639,-63.5 827.25,-63.5 827.25,-27.5 639,-27.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"650.38\" y=\"-40.7\" font-family=\"Times,serif\" font-size=\"14.00\">d</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"661.75,-28 661.75,-63.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"701.62\" y=\"-40.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 4.0000</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"741.5,-28 741.5,-63.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"784.38\" y=\"-40.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad &#45;2.0000</text>\n",
       "</g>\n",
       "<!-- 140523648766248&#45;&gt;140523648766024* -->\n",
       "<g id=\"edge6\" class=\"edge\">\n",
       "<title>140523648766248&#45;&gt;140523648766024*</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M827.49,-61.75C836.27,-63.28 844.77,-64.76 852.51,-66.1\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"851.66,-69.51 862.11,-67.78 852.86,-62.61 851.66,-69.51\"/>\n",
       "</g>\n",
       "<!-- 140523648766248+&#45;&gt;140523648766248 -->\n",
       "<g id=\"edge3\" class=\"edge\">\n",
       "<title>140523648766248+&#45;&gt;140523648766248</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M603.31,-45.5C610.49,-45.5 618.72,-45.5 627.47,-45.5\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"627.29,-49 637.29,-45.5 627.29,-42 627.29,-49\"/>\n",
       "</g>\n",
       "<!-- 140523647396216 -->\n",
       "<g id=\"node8\" class=\"node\">\n",
       "<title>140523647396216</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"4.88,-83.5 4.88,-119.5 187.88,-119.5 187.88,-83.5 4.88,-83.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"15.88\" y=\"-96.7\" font-family=\"Times,serif\" font-size=\"14.00\">a</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"26.88,-84 26.88,-119.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"66.75\" y=\"-96.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 2.0000</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"106.62,-84 106.62,-119.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"147.25\" y=\"-96.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 6.0000</text>\n",
       "</g>\n",
       "<!-- 140523647396216&#45;&gt;140523648766192* -->\n",
       "<g id=\"edge7\" class=\"edge\">\n",
       "<title>140523647396216&#45;&gt;140523648766192*</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M188.01,-85.37C198.67,-83.48 209.04,-81.63 218.3,-79.99\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"218.65,-83.48 227.88,-78.28 217.42,-76.59 218.65,-83.48\"/>\n",
       "</g>\n",
       "<!-- 140523647395768 -->\n",
       "<g id=\"node9\" class=\"node\">\n",
       "<title>140523647395768</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"318.75,-0.5 318.75,-36.5 513,-36.5 513,-0.5 318.75,-0.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"329.75\" y=\"-13.7\" font-family=\"Times,serif\" font-size=\"14.00\">c</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"340.75,-1 340.75,-36.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"384\" y=\"-13.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 10.0000</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"427.25,-1 427.25,-36.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"470.12\" y=\"-13.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad &#45;2.0000</text>\n",
       "</g>\n",
       "<!-- 140523647395768&#45;&gt;140523648766248+ -->\n",
       "<g id=\"edge5\" class=\"edge\">\n",
       "<title>140523647395768&#45;&gt;140523648766248+</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M513.41,-34.98C522.1,-36.47 530.5,-37.9 538.15,-39.21\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"537.55,-42.66 548,-40.89 538.73,-35.76 537.55,-42.66\"/>\n",
       "</g>\n",
       "<!-- 140523647397840 -->\n",
       "<g id=\"node10\" class=\"node\">\n",
       "<title>140523647397840</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"0,-28.5 0,-64.5 192.75,-64.5 192.75,-28.5 0,-28.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"11.38\" y=\"-41.7\" font-family=\"Times,serif\" font-size=\"14.00\">b</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"22.75,-29 22.75,-64.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"64.88\" y=\"-41.7\" font-family=\"Times,serif\" font-size=\"14.00\">data &#45;3.0000</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"107,-29 107,-64.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"149.88\" y=\"-41.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad &#45;4.0000</text>\n",
       "</g>\n",
       "<!-- 140523647397840&#45;&gt;140523648766192* -->\n",
       "<g id=\"edge9\" class=\"edge\">\n",
       "<title>140523647397840&#45;&gt;140523648766192*</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M193,-62.91C201.81,-64.42 210.32,-65.88 218.06,-67.2\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"217.21,-70.61 227.66,-68.85 218.4,-63.71 217.21,-70.61\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.graphs.Digraph at 0x7fce3629f4a8>"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "draw_dot(L)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ee9d5c5b",
   "metadata": {},
   "source": [
    "----\n",
    "The `grad_check` function below is used to understand how changes in the parameters (`input: var`) create a response in the output. It basically gives us the slope of L with respect to each parameter. We can change the step size, h, if we want but the default value is `h=0.001`.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "9e9ed770",
   "metadata": {},
   "outputs": [],
   "source": [
    "def grad_check(var, h=0.001): #inline gradient check\n",
    "    \"\"\"\n",
    "    parameters:\n",
    "    \n",
    "    var {string} : a,b,c,d,e,f\n",
    "    h {float}: step change\n",
    "    \n",
    "    adjust a,b,c,d,e,f by h and see how it affects L \n",
    "    estimate dL/di, where i =  a,b,c,d,e,f\n",
    "    \"\"\"\n",
    "    a = Value(2.0,label='a')\n",
    "    b = Value(-3.0,label='b')\n",
    "    c = Value(10.0,label='c')\n",
    "    e = a*b; e.label ='e'\n",
    "    d = e+c; d.label = 'd'\n",
    "    f = Value(-2.0, label='f')\n",
    "    L = d * f\n",
    "    L.label = 'L'\n",
    "    L1 = L.data\n",
    "    \n",
    "    if var=='a':\n",
    "        a = Value(2.0,label='a')\n",
    "        a.data += h\n",
    "        b = Value(-3.0,label='b')\n",
    "        c = Value(10.0,label='c')\n",
    "        e = a*b; e.label ='e'\n",
    "        d = e+c; d.label = 'd'\n",
    "        f = Value(-2.0, label='f')\n",
    "        L = d * f \n",
    "        L.label = 'L'\n",
    "        L2 = L.data\n",
    "    elif var=='b':\n",
    "        a = Value(2.0,label='a')\n",
    "        b = Value(-3.0,label='b')\n",
    "        b.data += h\n",
    "        c = Value(10.0,label='c')\n",
    "        e = a*b; e.label ='e'\n",
    "        d = e+c; d.label = 'd'\n",
    "        f = Value(-2.0, label='f')\n",
    "        L = d * f \n",
    "        L.label = 'L'\n",
    "        L2 = L.data\n",
    "    elif var=='c':\n",
    "        a = Value(2.0,label='a')\n",
    "        b = Value(-3.0,label='b')\n",
    "        c = Value(10.0,label='c')\n",
    "        c.data += h\n",
    "        e = a*b; e.label ='e'\n",
    "        d = e+c; d.label = 'd'\n",
    "        f = Value(-2.0, label='f')\n",
    "        L = d * f \n",
    "        L.label = 'L'\n",
    "        L2 = L.data\n",
    "    elif var=='d':\n",
    "        a = Value(2.0,label='a')\n",
    "        b = Value(-3.0,label='b')\n",
    "        c = Value(10.0,label='c')\n",
    "        e = a*b; e.label ='e'\n",
    "        d = e+c; d.label = 'd'\n",
    "        d.data += h\n",
    "        f = Value(-2.0, label='f')\n",
    "        L = d * f \n",
    "        L.label = 'L'\n",
    "        L2 = L.data\n",
    "    elif var=='e':\n",
    "        a = Value(2.0,label='a')\n",
    "        b = Value(-3.0,label='b')\n",
    "        c = Value(10.0,label='c')\n",
    "        e = a*b; e.label ='e'\n",
    "        e.data += h\n",
    "        d = e+c; d.label = 'd'\n",
    "        f = Value(-2.0, label='f')\n",
    "        L = d * f\n",
    "        L.label = 'L'\n",
    "        L2 = L.data\n",
    "    elif var=='f':\n",
    "        a = Value(2.0,label='a')\n",
    "        b = Value(-3.0,label='b')\n",
    "        c = Value(10.0,label='c')\n",
    "        e = a*b; e.label ='e'\n",
    "        d = e+c; d.label = 'd'\n",
    "        f = Value(-2.0, label='f')\n",
    "        f.data += h\n",
    "        L = d * f\n",
    "        L.label = 'L'\n",
    "        L2 = L.data\n",
    "    elif var=='L':\n",
    "        a = Value(2.0,label='a')\n",
    "        b = Value(-3.0,label='b')\n",
    "        c = Value(10.0,label='c')\n",
    "        e = a*b; e.label ='e'\n",
    "        d = e+c; d.label = 'd'\n",
    "        f = Value(-2.0, label='f')\n",
    "        L = d * f\n",
    "        L.data += h\n",
    "        L.label = 'L'\n",
    "        L2 = L.data\n",
    "    else:\n",
    "        print(\"The variable is NOT recognized !!! Ensure its one of these (case-sensitive): a, b, c, d, e, f, L\")\n",
    "        return None\n",
    "    \n",
    "    return (L2 - L1)/h"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "303e0106",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dL/dL = 1.000\n",
      "dL/dd = -2.000\n",
      "dL/df = 4.000\n",
      "dL/dc = -2.000\n",
      "dL/de = -2.000\n",
      "dL/da = 6.000\n",
      "dL/db = -4.000\n"
     ]
    }
   ],
   "source": [
    "print(f\"dL/dL = {grad_check('L'):.3f}\")\n",
    "print(f\"dL/dd = {grad_check('d'):.3f}\")\n",
    "print(f\"dL/df = {grad_check('f'):.3f}\")\n",
    "print(f\"dL/dc = {grad_check('c'):.3f}\")\n",
    "print(f\"dL/de = {grad_check('e'):.3f}\")\n",
    "print(f\"dL/da = {grad_check('a'):.3f}\")\n",
    "print(f\"dL/db = {grad_check('b'):.3f}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "7f77a596",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The variable is NOT recognized !!! Ensure its one of these (case-sensitive): a, b, c, d, e, f, L\n"
     ]
    }
   ],
   "source": [
    "grad_check('df')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "94e45d5f",
   "metadata": {},
   "source": [
    "-------------\n",
    "<br><br>\n",
    "## 3.2. Preview of a Single Optimization Step<a id=\"302\"></a>\n",
    "-------------------------------------------------------\n",
    "If we nudge the variables in the direction of their gradients, it will increase L. This is the concept of **optimization**. Let’s try.\n",
    "\n",
    "Now L=8.0000. Let’s change the input variables, in the direction of their gradient."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "661bdd94",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-7.286496\n"
     ]
    }
   ],
   "source": [
    "a.data += 0.01 * a.grad\n",
    "b.data += 0.01 * b.grad\n",
    "c.data += 0.01 * c.grad\n",
    "f.data += 0.01 * f.grad\n",
    "\n",
    "e = a * b \n",
    "d = e + c\n",
    "L = d * f\n",
    "\n",
    "print(L.data)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "80796474",
   "metadata": {},
   "source": [
    "After running the forward pass, `L=-7.286`. Here, 0.01 is the step size. Indeed we see that after we optimized our parameters using the gradients we manually calculated, and ran the forward pass of our network again, we managed to increase the value of L."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4d8da56b",
   "metadata": {},
   "source": [
    "-------------\n",
    "<br><br>\n",
    "# 4. Micrograd: Neurons<a id=\"4\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "88ae14ef",
   "metadata": {},
   "source": [
    "-------------\n",
    "<br><br>\n",
    "## 4.1. Biological Neuron vs. Artificial Neuron<a id=\"401\"></a>\n",
    "---------------------------------------\n",
    " \n",
    "<a id=\"a1\"></a>\n",
    "<p align=\"center\">\n",
    "    <img src=\"_imgs/biological_neuron.png\" width=480 height=240 align=\"left\"> \n",
    "    <img src=\"_imgs/neuron_mathematical_model.jpeg\" width=360 height=240 align=\"center\"> \n",
    "</p>\n",
    "\n",
    "**Fig 1. L-R: A Cartoon Diagram of a Biological Neuron & its Mathematical Model.** ([Source](https://cs231n.github.io/neural-networks-1/))\n",
    "\n",
    "\n",
    "The figure above provides a side-by-side drawing of a biological and artificial neuron. The biological neuron is analogous to artificial neurons in the following terms:\n",
    "\n",
    "\n",
    "|Biological Neuron  |Artificial Neuron          |\n",
    "|-------------------|---------------------------|\n",
    "|Cell Nucleus (Soma)|Node                       |\n",
    "|Dendrites          |Input                      |\n",
    "|Synapse            |Weights or interconnections|\n",
    "|Axon               |Output                     |\n",
    "\n",
    "## Artificial Neuron at a Glance\n",
    "-----------------------------\n",
    "\n",
    "The artificial neuron has the following characteristics:\n",
    "\n",
    "*   A neuron is a mathematical function modeled on the working of biological neurons\n",
    "*   It is an elementary unit in an artificial neural network\n",
    "*   One or more inputs are separately weighted\n",
    "*   Inputs are summed and passed through a nonlinear function to produce output\n",
    "*   Every neuron holds an internal state called activation signal\n",
    "*   Each connection link carries information about the input signal\n",
    "*   Every neuron is connected to another neuron via connection link\n",
    "\n",
    "\n",
    "In the next section, let us talk about perceptrons."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2238b908",
   "metadata": {},
   "source": [
    "-------------\n",
    "<br><br>\n",
    "## 4.2. [Perceptron](https://www.simplilearn.com/tutorials/deep-learning-tutorial/perceptron#perceptron)<a id=\"402\"></a>\n",
    "------------\n",
    "\n",
    "Perceptron was introduced by Frank Rosenblatt in 1957. He proposed a Perceptron learning rule based on the original MCP neuron. A Perceptron is an algorithm for supervised learning of binary classifiers. This algorithm enables neurons to learn and processes elements in the training set one at a time.\n",
    "\n",
    "\n",
    "## Basic Components of Perceptron\n",
    "------------------------------\n",
    "\n",
    "Perceptron is a type of artificial neural network, which is a fundamental concept in machine learning. The basic components of a perceptron are:\n",
    "\n",
    "1.  <u>Input Layer</u>: The input layer consists of one or more input neurons, which receive input signals from the external world or from other layers of the neural network.\n",
    "2.  <u>Weights</u>: Each input neuron is associated with a weight, which represents the strength of the connection between the input neuron and the output neuron.\n",
    "3.  <u>Bias</u>: A bias term is added to the input layer to provide the perceptron with additional flexibility in modeling complex patterns in the input data.\n",
    "4.  <u>Activation Function</u>: The activation function determines the output of the perceptron based on the weighted sum of the inputs and the bias term. Common activation functions used in perceptrons include the step function, sigmoid function, and ReLU function.\n",
    "5.  <u>Output</u>: The output of the perceptron is a single binary value, either 0 or 1, which indicates the class or category to which the input data belongs.\n",
    "6.  <u>Training Algorithm</u>: The perceptron is typically trained using a supervised learning algorithm such as the perceptron learning algorithm or backpropagation. During training, the weights and biases of the perceptron are adjusted to minimize the error between the predicted output and the true output for a given set of training examples.\n",
    "7.  Overall, the perceptron is a simple yet powerful algorithm that can be used to perform binary classification tasks and has paved the way for more complex neural networks used in deep learning today.\n",
    "\n",
    "## Types of Perceptron models\n",
    "--------------------------\n",
    "\n",
    "We have already discussed the types of Perceptron models in the Introduction. Here, we shall give a more profound look at this:\n",
    "\n",
    "1.  <u>Single Layer Perceptron model</u>: One of the easiest ANN (Artificial Neural Networks) types consists of a feed-forward network and includes a threshold transfer inside the model. The main objective of the single-layer perceptron model is to analyze the linearly separable objects with binary outcomes. A Single-layer perceptron can learn only linearly separable patterns.\n",
    "2.  <u>Multi-Layered Perceptron **(MLP)** model</u>: It is mainly similar to a single-layer perceptron model but has more hidden layers.\n",
    "\n",
    "<u>Forward Stage</u>: From the input layer in the on stage, activation functions begin and terminate on the output layer.\n",
    "\n",
    "<u>Backward Stage</u>: In the backward stage, weight and bias values are modified per the model's requirement. The backstage removed the error between the actual output and demands originating backward on the output layer. A multilayer perceptron model has a greater processing power and can process linear and non-linear patterns. Further, it also implements logic gates such as AND, OR, XOR, XNOR, and NOR.\n",
    "\n",
    "\n",
    "\n",
    "## [What is the Perceptron Model in Machine Learning?](https://www.simplilearn.com/tutorials/deep-learning-tutorial/perceptron#what_is_the_perceptron_model_in_machine_learning)\n",
    "-------------------------------------------------\n",
    "\n",
    "A machine-based algorithm used for supervised learning of various binary sorting tasks is called Perceptron. Furthermore, Perceptron also has an essential role as an Artificial Neuron or Neural link in detecting certain input data computations in business intelligence. A perceptron model is also classified as one of the best and most specific types of Artificial Neural networks. Being a supervised learning algorithm of binary classifiers, we can also consider it a single-layer neural network with four main parameters: **input values, weights and Bias, net sum, and an activation function.**\n",
    "\n",
    "## How Does Perceptron Work?\n",
    "-------------------------\n",
    "\n",
    "As discussed earlier, Perceptron is considered a single-layer neural link with four main parameters. The perceptron model begins with multiplying all input values and their weights, then adds these values to create the weighted sum. Further, this weighted sum is applied to the activation function to obtain the desired output. This activation function is also known as the step function and is represented by f.\n",
    "\n",
    "<a id=\"a2\"></a>\n",
    "![Perceptron_work](_imgs/perceptron.png)\n",
    "**Fig 2. A single-layer Perceptron structure.** ([Source](https://towardsdatascience.com/what-the-hell-is-perceptron-626217814f53))\n",
    "\n",
    "Mathematical representation of a single-layer neural network (**perceptron**) as shown in the figure above\n",
    "1. All the inputs x are multiplied with their weights w\n",
    "2. Add all the multiplied values including bias, (b or w_0) and call them Weighted Sum (W_s).\n",
    "$$\n",
    "W_s = \\sum_{i=1}^{n} w_i.x_i + b \\\\ \n",
    "where\\ b = w_0 \\\\\n",
    "$$\n",
    "3. Apply the correct Activation Function to that weighted sum.<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "373f2729",
   "metadata": {},
   "source": [
    "-------------\n",
    "<br><br>\n",
    "## 4.3. Activation Functions<a id=\"403\"></a>\n",
    "--------\n",
    "The **[activation function](https://en.wikipedia.org/wiki/Activation_function)** of a node in an artificial neural network (ANN) is a function that calculates the output of the node based on its individual inputs and their weights. Nontrivial problems can be solved using only a few nodes if the activation function is *nonlinear*.\n",
    "\n",
    "<a id=\"b1\"></a>\n",
    "Below is a table of different activation functions.\n",
    "![Different Activation Functions](_imgs/activation_function_full.png)\n",
    "**Table 1: Activation Functions.** ([Source](https://en.wikipedia.org/wiki/Activation_function#Table_of_activation_functions))<br><br>\n",
    "\n",
    "Now let's look into an example of manual backpropagation using a neuron."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "91c79139",
   "metadata": {},
   "source": [
    "-------------\n",
    "<br><br>\n",
    "## 4.4. Micrograd: Manual Backpropagation Example 002: A Neuron<a id=\"404\"></a>\n",
    "-----------------------------"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a003f4e9",
   "metadata": {},
   "source": [
    "Let us discuss the basic structure of an artificial neuron. **Inputs, x** are multiplied by the corresponding **weights, w** and fed to the neuron. Internally, they are summed and then there is an extra term called **bias, b** which is added to yields the **Weighted sum, Ws**. The result is passed through the **activation function, f** (sigmoid, ReLU, tanh, Leaky ReLU, etc.) which yields an output. \n",
    "\n",
    "<a id=\"c2\"></a>\n",
    "$$\n",
    "W_s = \\sum_{i=1}^{n} w_i.x_i + b\n",
    "$$\n",
    "**Eq. 2: Mathematical representation of a NN weighted sum function.**<br>\n",
    "\n",
    "Now, let us implement an artificial neuron and understand how to use backpropagation to tune its weights in order to train the neuron optimally. In the example below, we use the **tanh** activation function, which yields output in the range of -1 to 1. We can implement arbitrary complex functions if we know the local derivative. No need to break it down into atomic pieces. We will implement the `tanh` function under the class `Value` object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "38c70e54",
   "metadata": {},
   "outputs": [],
   "source": [
    "# inputs x1,x2\n",
    "x1 = Value(2.0,label='x1')\n",
    "x2 = Value(0.0,label='x2')\n",
    "# weights w1, w2\n",
    "w1 = Value(-3.0,label='w1')\n",
    "w2 = Value(1.0,label='w2')\n",
    "# bias of the neuron\n",
    "b = Value(6.8813735870195432,label='b')\n",
    "# x1*w1 + x2*w2 + b\n",
    "x1w1 = x1*w1; x1w1.label ='x1w1'\n",
    "x2w2 = x2*w2; x2w2.label ='x2w2'\n",
    "x1w1x2w2 = x1w1+x2w2; x1w1x2w2.label='x1w1 + x2w2'\n",
    "n = x1w1x2w2+b; n.label='n'\n",
    "o = n.tanh() \n",
    "o.label = 'o'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8a9cdddf",
   "metadata": {},
   "source": [
    "Now we have implemented a complete neuron with two inputs: `x1` and `x2`.\n",
    "\n",
    "Now when backpropagating, we need to find the local derivative of the `tanh` function.<br><br><br>\n",
    "$$\n",
    "o = \\tanh(n) \\\\\n",
    "\\frac{do}{dn} = 1 - \\tanh^2(n) \\\\\n",
    "$$\n",
    "\n",
    "Next, we can go through the exercise of manually calculating the `grad` values for each of the parameters: `x1`, `x2`, `w1`, `w2`, `x1w1`, `x2w2`, `x1w1 + x2w2`, `b` and `n`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "b2596a46",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAD4CAYAAADhNOGaAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nO3deXRc9Xn/8fej3ba828grXsAJtsEsFmYLILM6YXFCgABNYgLEaQ50S9MGSn+kJW1Dmp6StqElLiGQJuCQhWLAwdjGCqRg4wW8yJtsGYxsy5u8yZK1zfP7Y67IICRvM5o7o/t5nTNn7vK9M58vY+bRvd9755q7IyIi0ZUTdgAREQmXCoGISMSpEIiIRJwKgYhIxKkQiIhEXF7YAU7GoEGDfPTo0WHHOGGHDx+mV69eYcdIO/U7WqLab8j8vi9fvnyPuw9uvzwrC8Ho0aNZtmxZ2DFOWHl5OWVlZWHHSDv1O1qi2m/I/L6b2fsdLdehIRGRiFMhEBGJOBUCEZGIUyEQEYk4FQIRkYhLSSEwsyfNbJeZrelkvZnZv5vZJjNbZWbnJaybYWaVwWNGKvKIiMjxS9UewVPAtKOs/zQwLnjMBP4LwMwGAN8GLgCmAN82s/4pyiQiIschJdcRuPvrZjb6KE2mAz/1+G9eLzazfmY2FCgD5rt7LYCZzSdeUJ5NRS4RiQ53p7ElRmNLjObWGE0Jz02tMVpanZZY23PwaI3REnNiMSfm0Orx6daYE3PH25Z5fL0H6x0+nI+/N8Tc2VzVxDo248S3bcvV1gbAP8ybkJ2O23ysITDj4tEMLC5M3X840ndB2XDgg4T56mBZZ8s/xsxmEt+boKSkhPLy8i4J2pXq6uqyMney1O9oOZl+uztHWuFQk3Ow0TnUHH+ua3YaWqChxalvcY60QH1zvG1Tq9MUPDfGoLm13RdoWDauT/lLWsL0kKZtDCtO7fBu1lxZ7O6zgFkApaWlnslX73Um06867Crqd7R01G93Z9ehRt7fW0/1vnqq9zXwQW38uXp/PbsONtLYEuvw9XJzjN5FefQuyqe4MJ++vfIYXphHj4JceuTHH0X5OfTIz6UwP5fCvBwK8nIoyM0hPzc+HX82cnNyyM8xcnOMvNwc8oLp3Bwjx4zcHILn+HxOjpFj8WXxB1jbMwYGFqw34PdvvMFll12GBd/cFrT7cJ625faR+ba2ievSKV2FYBswMmF+RLBsG/HDQ4nLy9OUSUS6QEvMWV9zkLXbD7Jux0HW7ohP76tv/ki7U3oXMqJ/D84d2Z8hfYsY2KuAgcWFDCwuYFCv+HO/nvn0yM8N5cvxZBTmGT0KcsOOccLSVQjmAPeZ2WziA8MH3H2Hmc0D/ilhgPga4IE0ZRKRFIjFnHU1B3mjcg9vVO5mSVU9La++AUBBXg5nDOnNtROHcMaQ3owZXMyI/j0Y3q8HRfnZ94XZXaWkEJjZs8T/sh9kZtXEzwTKB3D3x4G5wGeATUA98JVgXa2ZfQdYGrzUw20DxyKSuQ7UN7Nw/c7gy38Pe+oaAThjSG+uGJnHdRedyfihfRg7qBd5ubpcKdOl6qyh24+x3oF7O1n3JPBkKnKISNdxd1Zs3cfPl2zl5VU7aGyJMbBXAZeOG8Sl4wbzqXGDKOlTFB8jOKfDcz4kQ2XNYLGIhONAfTPPv1PNs29/wIadhyguzOOW0hHcMnkkZw3vS05Odhy/l86pEIhIh/bWNfLogo38clk1jS0xzh7Zj+99/iyunzSMXoX66uhO9GmKyEc0t8b46Vvv84MFG2loauWW0hF88cJRTBzWN+xo0kVUCETkQ+UbdvGdl9ayefdhLvvEYB66fjynn9I77FjSxVQIRISq3XX8w8vreG39LsYM6sWTd5Yy9ZOnZM35+5IcFQKRiHtp1Xa++cuV5Ofk8OBnxjPj4tEU5OmUzyhRIRCJqFjMeXTBRv7jtU2UjurPf/7ReZzSpyjsWBICFQKRCDrc2MI3nnuXeRU7+ULpSL7z2TO1FxBhKgQiEfNBbT1f/ekyNu48xLdvmMCdF4/WWEDEqRCIRMiSqr18/ecraGmN8fRdU7h03OCwI0kGUCEQiYg3Kndz11NLGTmgJ098uZSxg4vDjiQZQoVAJALW7TjI13+2gtMGF/OLr11E3x75YUeSDKLRIZFurubAEb7yk6UUF+bxk6+cryIgH6M9ApFu7NCRZr7y1FLqGlt47msXMbRvj7AjSQZSIRDppppbY9z7zDts3HmIn9x5PhOG9Qk7kmQoHRoS6Ybcnf/3v2t4feNuvvu5s7jsEzo7SDqXkkJgZtPMbIOZbTKz+ztY/6iZvRs8NprZ/oR1rQnr5qQij0jU/Wf5ZmYv/YA/ueJ0bj1/5LE3kEhL+tCQmeUCjwFXA9XAUjOb4+5r29q4+18ktP8T4NyEl2hw93OSzSEicfPX7uT78zZw07nD+cbVnwg7jmSBVOwRTAE2uXuVuzcBs4HpR2l/O/BsCt5XRNo50NDMg8+vZsLQPjzy+Um6YliOi8VvJ5zEC5jdDExz93uC+S8BF7j7fR20HQUsBka4e2uwrAV4F2gBHnH3/+3kfWYCMwFKSkomz549O6ncYairq6O4OHoX8ajf6fPkmkZ+v62Fhy4sYnTf3LS+d5uoft6Q+X2fOnXqcncvbb883WcN3Qb8qq0IBEa5+zYzGwu8Zmar3X1z+w3dfRYwC6C0tNTLysrSEjiVysvLycbcyVK/0+PNzXt4/ZUlfO3ysdz56fFpe9/2ovp5Q/b2PRWHhrYBiaNRI4JlHbmNdoeF3H1b8FwFlPPR8QMROQ5Hmlt54DerGTWwJ39+pcYF5MSkohAsBcaZ2RgzKyD+Zf+xs3/M7AygP/BWwrL+ZlYYTA8CLgHWtt9WRI7u0QUbeX9vPd+96Sx6FIRzSEiyV9KHhty9xczuA+YBucCT7l5hZg8Dy9y9rSjcBsz2jw5KjAd+ZGYx4kXpkcSzjUTk2NZsO8ATb2zhtvNHcvFpg8KOI1koJWME7j4XmNtu2UPt5v+ug+3eBM5KRQaRKGpujfHXv1rFwF4FPPCZ8MYFJLvpJyZEsth/v1HF2h0HefyLk/VjcnLS9BMTIlmqancdP1hQybSJQ5h25pCw40gWUyEQyVL/NHcdhXk5PDx9YthRJMupEIhkoTXbDrBg3S6+eulYTulTFHYcyXIqBCJZ6N8XVtK7KI8ZF48OO4p0AyoEIllm7faDvLp2J3ddMkYDxJISKgQiWeY/Xqukd2Eed10yJuwo0k2oEIhkkfU1B/ntmhruvGQ0fXtqb0BSQ4VAJIv8x2ubKC7M4+5PaW9AUkeFQCRLVO48xNzVO5hx8Sj69SwIO450IyoEIlni31/bRI/8XO7+1Niwo0g3o0IgkgU27arjpVXb+fJFoxnQS3sDkloqBCJZ4IevVVKUl8tXL9XYgKSeCoFIhqvaXcecldv58kWjGFhcGHYc6YZUCEQy3GOLNlOQl8M9l2psQLpGSgqBmU0zsw1mtsnM7u9g/Z1mttvM3g0e9ySsm2FmlcFjRiryiHQXtYebeHHldr5QOpLBvbU3IF0j6fsRmFku8BhwNVANLDWzOR3caewX7n5fu20HAN8GSgEHlgfb7ks2l0h38JsV1TS1xrjjglFhR5FuLBV7BFOATe5e5e5NwGxg+nFuey0w391rgy//+cC0FGQSyXruzjNvb2XyqP58ckjvsONIN5aKQjAc+CBhvjpY1t7nzWyVmf3KzEae4LYikbNkSy1Vuw9z+5RTw44i3Vy6blX5IvCsuzea2deAp4ErTuQFzGwmMBOgpKSE8vLylIfsanV1dVmZO1nq98l5fOUReuZBn/2VlJdvSl2wLhbVzxuyt++pKATbgJEJ8yOCZR9y970Js08A/5ywbVm7bcs7ehN3nwXMAigtLfWysrKOmmW08vJysjF3stTvE1d7uIkV8xdyxwWjuebK7LoDWVQ/b8jevqfi0NBSYJyZjTGzAuA2YE5iAzMbmjB7I7AumJ4HXGNm/c2sP3BNsEwk0n69PD5IrMNCkg5J7xG4e4uZ3Uf8CzwXeNLdK8zsYWCZu88B/tTMbgRagFrgzmDbWjP7DvFiAvCwu9cmm0kkm7k7z2qQWNIoJWME7j4XmNtu2UMJ0w8AD3Sy7ZPAk6nIIdIdLNlSS9Wew/zL1NPDjiIRoSuLRTLMM0u20qcoj+snDT12Y5EUUCEQySC1h5t4ZU0NN503gqL83LDjSESoEIhkEA0SSxhUCEQyhAaJJSwqBCIZYnFVfJD4Du0NSJqpEIhkiGffjg8SX6dBYkkzFQKRDHCgoZlX1tTwuXOHa5BY0k6FQCQDzKuooak1xk3njQg7ikSQCoFIBnhx5XZOHdCTSSP6hh1FIkiFQCRke+oaeXPzXm44eyhmFnYciSAVApGQ/XZNDa0x54azh4UdRSJKhUAkZC+u3M64U4r5ZImuHZBwqBCIhGjHgQaWvlfLDWcP02EhCY0KgUiIXl61A3d0WEhCpUIgEqIXV27nrOF9GTOoV9hRJMJUCERC8v7ew6ysPsANZ+tKYglXSgqBmU0zsw1mtsnM7u9g/TfMbK2ZrTKzhWY2KmFdq5m9GzzmtN9WpLt6adUOAK6bpMNCEq6k71BmZrnAY8DVQDWw1MzmuPvahGbvAKXuXm9mXyd+8/ovBOsa3P2cZHOIZJsXV26ndFR/hvfrEXYUibhU7BFMATa5e5W7NwGzgemJDdx9kbvXB7OLAV1HL5G2cech1tcc0iCxZIRU3LN4OPBBwnw1cMFR2t8N/DZhvsjMlhG/sf0j7v6/HW1kZjOBmQAlJSWUl5cnkzkUdXV1WZk7Wer3x/2msgkD+h6qorz8vXTG6nJR/bwhe/uekpvXHy8z+yJQClyesHiUu28zs7HAa2a22t03t9/W3WcBswBKS0u9rKwsHZFTqry8nGzMnSz1+6Pcnb9f9jsuPr0Pn732wvQH62JR/bwhe/ueikND24CRCfMjgmUfYWZXAQ8CN7p7Y9tyd98WPFcB5cC5KcgkkrEqth9ky57D3KBBYskQqSgES4FxZjbGzAqA24CPnP1jZucCPyJeBHYlLO9vZoXB9CDgEiBxkFmk23lx5XbycoxpZw4JO4oIkIJDQ+7eYmb3AfOAXOBJd68ws4eBZe4+B/g+UAz8MriMfqu73wiMB35kZjHiRemRdmcbiXQrsZjz0qodXPaJwfTrWRB2HBEgRWME7j4XmNtu2UMJ01d1st2bwFmpyCCSDd75YD/b9jfwzWs/EXYUkQ/pymKRNJpXUUN+rnHl+JKwo4h8SIVAJE3cnXkVNVx82iD6FOWHHUfkQyoEImmyvuYQ7++t1yCxZBwVApE0mVdRgxlcpcNCkmFUCETS5JU1NZw/agCDexeGHUXkI1QIRNLg/b2HWV9ziGsmam9AMo8KgUgazKuoAeDaiRofkMyjQiCSBq+sqWHisD6MHNAz7CgiH6NCINLFdh08woqt+5mmvQHJUCoEIl1s3tqdAFyr00YlQ6kQiHSxVytqGDuoF+NOKQ47ikiHVAhEutCB+mbe2ryXayYOIfjBRZGMo0Ig0oUWrt9JS8x1NbFkNBUCkS70ypoahvQpYtLwvmFHEemUCoFIF2lscV6v3M21E0vIydFhIclcKgQiXWT1nlaONMd0EZlkvJQUAjObZmYbzGyTmd3fwfpCM/tFsH6JmY1OWPdAsHyDmV2bijwimWD5rhb69cxnypgBYUcROaqkC4GZ5QKPAZ8GJgC3m9mEds3uBva5++nAo8D3gm0nEL/H8URgGvCfweuJZLWmlhjv7mrlqvEl5OVqx1syWyr+hU4BNrl7lbs3AbOB6e3aTAeeDqZ/BVxp8XPppgOz3b3R3bcAm4LXE8lqi6v20tCCriaWrJCKexYPBz5ImK8GLuisTXCz+wPAwGD54nbbDu/oTcxsJjAToKSkhPLy8hRET6+6urqszJ2sKPb7qYpGCnKc2I61lO9aF3actIri590mW/uekpvXp4O7zwJmAZSWlnpZWVm4gU5CeXk52Zg7WVHrd2vM+ebvF3L2Kc41V04NO07aRe3zTpStfU/FoaFtwMiE+RHBsg7bmFke0BfYe5zbimSVd7buY09dI5NPyZq/syTiUlEIlgLjzGyMmRUQH/yd067NHGBGMH0z8Jq7e7D8tuCsojHAOODtFGQSCc28ihoKcnM4+xSd9yDZIek/WYJj/vcB84Bc4El3rzCzh4Fl7j4H+DHwP2a2CaglXiwI2j0HrAVagHvdvTXZTCJhcXdeqajh4tMH0iOvPuw4IsclJfuu7j4XmNtu2UMJ00eAWzrZ9h+Bf0xFDpGwrdtxiA9qG7i37HSorwo7jshx0QnOIin0SkUNOQZXTdC9iSV7qBCIpNCrFTWUjh7AoOLCsKOIHDcVApEUeW/PYdbXHNJvC0nWUSEQSZF5FTUAXDtRh4Uku6gQiKTIKxU1nDm8DyP69ww7isgJUSEQSYGaA0d4Z+t+/baQZCUVApEUmL+27bCQCoFkHxUCkRR4paKGsYN7cfopxWFHETlhKgQiSdpf38TiqlqunTiE+K+ri2QXFQKRJC1Yt4vWmGt8QLKWCoFIkuZV1DC0bxGTRvQNO4rISVEhEElCfVMLr2/crcNCktVUCESS8LsNu2lsiXGNLiKTLKZCIJKE366poX/PfKaMHhB2FJGTpkIgcpLqm1qYv3Yn084cSl6u/leS7KV/vSIn6bX1u2hobuWGs4eGHUUkKUkVAjMbYGbzzawyeO7fQZtzzOwtM6sws1Vm9oWEdU+Z2RYzezd4nJNMHpF0enHldgb3LuSCMQPDjiKSlGT3CO4HFrr7OGBhMN9ePfBld58ITAN+YGb9Etb/lbufEzzeTTKPSFocPNLMog27ue6soeTm6GwhyW7JFoLpwNPB9NPAZ9s3cPeN7l4ZTG8HdgGDk3xfkVDNr9hJU0uMG88ZFnYUkaSZu5/8xmb73b1fMG3Avrb5TtpPIV4wJrp7zMyeAi4CGgn2KNy9sZNtZwIzAUpKSibPnj37pHOHpa6ujuLi6P0WTXfs978uO8K2uhj/cnmPTq8f6I79Ph5R7Tdkft+nTp263N1LP7bC3Y/6ABYAazp4TAf2t2u77yivMxTYAFzYbpkBhcQLxEPHyuPuTJ482bPRokWLwo4Qiu7W7711jX7aAy/7d+euO2q77tbv4xXVfrtnft+BZd7Bd2resSqIu1/V2Toz22lmQ919h5kNJX7Yp6N2fYCXgQfdfXHCa+8IJhvN7CfAN4+VRyRsr6ypoSXmOltIuo1kxwjmADOC6RnAC+0bmFkB8DzwU3f/Vbt1Q4NnIz6+sCbJPCJdbs7KbYwd3IsJQ/uEHUUkJZItBI8AV5tZJXBVMI+ZlZrZE0GbW4HLgDs7OE3052a2GlgNDAL+Ick8Il1q58EjLNlSyw2Thum3haTbOOahoaNx973AlR0sXwbcE0z/DPhZJ9tfkcz7i6Tby6t24A43nK2zhaT70JXFIifgxVXbmTC0j+5EJt2KCoHIcfqgtp53tu7X3oB0OyoEIsfppVXxk9yun6SzhaR7USEQOU4vrtzOuaf2Y+SAnmFHEUkpFQKR47BpVx1rdxzkhkk6LCTdjwqByHF4adV2zOA6HRaSbkiFQOQY3J0X3t3OBWMGUNKnKOw4IimnQiByDIuratmy5zC3TB4ZdhSRLqFCIHIMz769lT5FeTosJN2WCoHIUdQebuKVNTXcdN4IivJzw44j0iVUCESO4tfLq2lqjXHHBaeGHUWky6gQiHTC3Xn27a2UjurPJ0p6hx1HpMuoEIh0YnFVLVV7DnP7FO0NSPemQiDSCQ0SS1SoEIh0QIPEEiVJFQIzG2Bm882sMnju30m71oSb0sxJWD7GzJaY2SYz+0VwNzOR0LUNEuuwkERBsnsE9wML3X0csDCY70iDu58TPG5MWP494FF3Px3YB9ydZB6RpLUNEk8e1Z9PDtEgsXR/yRaC6cDTwfTTxO87fFyC+xRfAbTdx/iEthfpKku2xAeJ79DegEREsoWgxN13BNM1QEkn7YrMbJmZLTazti/7gcB+d28J5quB4UnmEUnaM0s0SCzRcsx7FpvZAmBIB6seTJxxdzcz7+RlRrn7NjMbC7wW3LD+wIkENbOZwEyAkpISysvLT2TzjFBXV5eVuZOVTf0+1OTMXVVP2cg8Fv/fG0m9Vjb1O5Wi2m/I4r67+0k/gA3A0GB6KLDhOLZ5CrgZMGAPkBcsvwiYdzzvO3nyZM9GixYtCjtCKLKp37N+t9lHfeslX7/jYNKvlU39TqWo9ts98/sOLPMOvlOTPTQ0B5gRTM8AXmjfwMz6m1lhMD0IuARYG4RaFBSFTrcXSRfXILFEVLKF4BHgajOrBK4K5jGzUjN7ImgzHlhmZiuJf/E/4u5rg3XfAr5hZpuIjxn8OMk8Iidt/tqdVO05zJcuHBV2FJG0OuYYwdG4+17gyg6WLwPuCabfBM7qZPsqYEoyGURSwd35t4WVjBrYUzenl8jRlcUiwGvrd1Gx/SD3Tj2dvFz9byHRon/xEnltewMjB/Tgc+fqDGaJHhUCibzyDbtZVX2A+6aeTr72BiSC9K9eIs3d+cHCSob368FN540IO45IKFQIJNJer9zDyg/2c6/2BiTC9C9fIsvd+bcFGxnerwc3T9begESXCoFE1u837WHF1v18vew0CvL0v4JEl/71SyTF9wYqGdq3iFtKtTcg0aZCIJH01ua9LHt/H18vO43CPN2BTKJNhUAi6QcLKynpU8itpSPDjiISOhUCiZzyDbt4e0stf3z5abofsQgqBBIxhxtbePD5NYwd3Ev3IxYJJPWjcyLZ5l9e3cC2/Q0897WLtDcgEtAegUTGO1v38dSb7/HFC09lypgBYccRyRgqBBIJTS0x7v/1aob0KeJb084IO45IRtGhIYmE/yrfzIadh/jxjFJ6F+WHHUckoyS1R2BmA8xsvplVBs/9O2gz1czeTXgcMbPPBuueMrMtCevOSSaPSEcqdx7ih4squeHsYVw5viTsOCIZJ9lDQ/cDC919HLAwmP8Id1/k7ue4+znAFUA98GpCk79qW+/u7yaZR+QjWmPOt369il6FeXz7hglhxxHJSMkWgunA08H008Bnj9H+ZuC37l6f5PuKHJf/ees9Vmzdz0PXT2BQcWHYcUQykrn7yW9stt/d+wXTBuxrm++k/WvAv7r7S8H8U8BFQCPBHoW7N3ay7UxgJkBJScnk2bNnn3TusNTV1VFcXBx2jLQLq997GmI8+PsGxvXP5S8nFxL/J5o++ryjJ9P7PnXq1OXuXtp++TELgZktAIZ0sOpB4OnEL34z2+fuHxsnCNYNBVYBw9y9OWFZDVAAzAI2u/vDx+pMaWmpL1u27FjNMk55eTllZWVhx0i7MPp9uLGFW3/0Fu/tOcy8v7iMEf17pvX9QZ93FGV6382sw0JwzLOG3P2qo7zoTjMb6u47gi/1XUd5qVuB59uKQPDaO4LJRjP7CfDNY+UROZaW1hj3PbOC9TWHeOLLpaEUAZFskuwYwRxgRjA9A3jhKG1vB55NXBAUj7bDSp8F1iSZRyLO3XloTgWLNuzm4ekTmXrGKWFHEsl4yRaCR4CrzawSuCqYx8xKzeyJtkZmNhoYCfyu3fY/N7PVwGpgEPAPSeaRiHv8d1U8s2Qrf3z5afzRBaPCjiOSFZK6oMzd9wJXdrB8GXBPwvx7wPAO2l2RzPuLJJqzcjvfe2U9N5w9jL++9pNhxxHJGvqJCekW3t5SyzefW8n5o/vz/ZsnkZOT3jOERLKZCoFkvc276/jqT5cxYkAP/vvLpfpVUZETpEIgWW3pe7Xc+vhb5OUYT905hX49C8KOJJJ1VAgka/1i6Vbu+O/F9OmRzy++dhGnDtRpoiInQ78+KlmnpTXGP85dx0/+7z0uHTeIH95+Hn176hdFRU6WCoFklQP1zdz37AreqNzDXZeM4W8+cwZ5udqxFUmGCoFkjU276rjn6aVs29/AP39+EreePzLsSCLdggqBZLyGplZ+9PpmHv/dZnoV5PHMVy/k/NG61aRIqqgQSMZyd15atYPvzl3H9gNHuG7SUP72uvEM7dsj7Ggi3YoKgWSkNdsO8PcvVrD0vX1MGNqHR79wDheMHRh2LJFuSYVAMsqGmkP8+PdV/HJ5NQN6FvDdm87i1tKR5OpKYZEuo0IgoWtoauXl1Tt49u2tLH9/HwW5Odx1yRj+9Mpx9O2h00JFupoKgYRmQ80hnn17K79ZUc3BIy2MHdyLv71uPDedN4IBvXSFsEi6qBBI2hw80sxbm/fyRuVu3qjcw/t76ynIzeHTZw3h9imncsGYAWm/naSIqBBIF6o93MTa7Qd5YVMTj61/kxVb99Mac3oW5HLxaQO5+1NjuH7SMP31LxIyFQJJWn1TC9X7GqjcWcfaHQdYt+MQa7cfpObgEQAMOGtEjD++fCyXjhvMeaf2pyBPVwOLZIqkCoGZ3QL8HTAemBLckKajdtOAfwNygSfcve1OZmOA2cBAYDnwJXdvSiaTpE4s5uxvaKb2cCN76prYW9fEnrpGth9ooLq2gep99VTva2Dv4T98ZLk5xumDi7notIFMGNqH8UP7sH/Laq6/5lMh9kREjibZPYI1wE3AjzprYGa5wGPA1UA1sNTM5rj7WuB7wKPuPtvMHgfuBv4ryUzdjrsTc2iNOTGPP1piTkur0xKL0dLqtMac5tYYza3x58aWGM2tMZpagkdrjIamVhqaWznS3EpDUytHWlqpb2rl0JEWDh1ppq6xJZiOz++rb6Y15h/LU5Cbw/D+PRjRvwfXDOvLiGB67KBixpUUf+x+AOXbdNxfJJMle6vKdcCxBvimAJvcvSpoOxuYbmbrgCuAO4J2TxPfu+iyQvDg86tZsqUWiH+5Jvr4191HV7Stb9vuD/Nt6/3D+Q+XJbR1hyONjeT/34JgffzL3T2+ZSzmuBN80fPhF378yz+JTh9FXo7RIz+X3kV59C7Kp7gojwG9Chg1sBfFhXkM7FXAwOICBhYXMqhX/HlArwIG9irQHcBEupF0jBEMBz5ImK8GLiB+OGi/u7ckLP/YfY3bmNlMYCZASUkJ5bNQ1YIAAAQPSURBVOXlJxyksbaJ/jmxP7zmcW7X1q59vbN2ExZMJLZP3KSlR4yC/FYscV3wnBM8mxmGfTifY394mMVvIJFj8UMwOQa5bY8cyDEjLwfyDPJyID8nmM+Jf+kX5kJBLhTmGvnBsj+IAU3BI0EzsA+a9sEO4o8TVVdXd1KfV7ZTv6MnW/t+zEJgZguAIR2setDdX0h9pI65+yxgFkBpaamXlZWd8GucxCYpVV5ezsnkznbqd7REtd+QvX0/ZiFw96uSfI9tQOLvBY8Ilu0F+plZXrBX0LZcRETSKB3n8C0FxpnZGDMrAG4D5nj8APoi4Oag3QwgbXsYIiISl1QhMLPPmVk1cBHwspnNC5YPM7O5AMFf+/cB84B1wHPuXhG8xLeAb5jZJuJjBj9OJo+IiJy4ZM8aeh54voPl24HPJMzPBeZ20K6K+FlFIiISEl3eKSIScSoEIiIRp0IgIhJxKgQiIhFn7X9qIRuY2W7g/bBznIRBwJ6wQ4RA/Y6WqPYbMr/vo9x9cPuFWVkIspWZLXP30rBzpJv6HS1R7Tdkb991aEhEJOJUCEREIk6FIL1mhR0gJOp3tES135ClfdcYgYhIxGmPQEQk4lQIREQiToUgJGb2l2bmZjYo7CzpYGbfN7P1ZrbKzJ43s35hZ+pKZjbNzDaY2SYzuz/sPOlgZiPNbJGZrTWzCjP7s7AzpZOZ5ZrZO2b2UthZTpQKQQjMbCRwDbA17CxpNB84090nARuBB0LO02XMLBd4DPg0MAG43cwmhJsqLVqAv3T3CcCFwL0R6XebPyP+U/tZR4UgHI8Cfw1EZqTe3V9NuD/1YuJ3pOuupgCb3L3K3ZuA2cD0kDN1OXff4e4rgulDxL8UO70PeXdiZiOA64Anws5yMlQI0szMpgPb3H1l2FlCdBfw27BDdKHhwAcJ89VE5AuxjZmNBs4FloSbJG1+QPyPu1jYQU5GUjemkY6Z2QJgSAerHgT+hvhhoW7naP129xeCNg8SP4Tw83Rmk/Qxs2Lg18Cfu/vBsPN0NTO7Htjl7svNrCzsPCdDhaALuPtVHS03s7OAMcBKM4P44ZEVZjbF3WvSGLFLdNbvNmZ2J3A9cKV37wtYtgEjE+ZHBMu6PTPLJ14Efu7uvwk7T5pcAtxoZp8BioA+ZvYzd/9iyLmOmy4oC5GZvQeUunsm/1phSpjZNOBfgcvdfXfYebqSmeURHxC/kngBWArckXCv7m7J4n/dPA3Uuvufh50nDMEewTfd/fqws5wIjRFIuvwQ6A3MN7N3zezxsAN1lWBQ/D5gHvEB0+e6exEIXAJ8Cbgi+IzfDf5KlgynPQIRkYjTHoGISMSpEIiIRJwKgYhIxKkQiIhEnAqBiEjEqRCIiEScCoGISMT9f4wnEA+Ihc5dAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# tanh activation function\n",
    "plt.plot(np.arange(-5,5,0.2), np.tanh(np.arange(-5,5,0.2)))\n",
    "plt.grid()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "d78864cf",
   "metadata": {},
   "outputs": [],
   "source": [
    "o.grad = 1.0\n",
    "n.grad = 1 - o.data**2 # o = tanh(n); do/dn = 1 - tanh(n)**2 = 1 - o**2\n",
    "\n",
    "#addition simply distributes the derivative\n",
    "b.grad = n.grad\n",
    "x1w1x2w2.grad = n.grad\n",
    "x1w1.grad = x1w1x2w2.grad\n",
    "x2w2.grad = x1w1x2w2.grad\n",
    "\n",
    "#we only need the derivatives of the weights\n",
    "x2.grad = x2w2.grad * w2.data\n",
    "w2.grad = x2w2.grad * x2.data\n",
    "\n",
    "x1.grad = x1w1.grad * w1.data\n",
    "w1.grad = x1w1.grad * x1.data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "3cfffe39",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "do/do = 1.000\n",
      "do/dn = 0.500\n",
      "do/db = 0.500\n",
      "do/d(x1w1 + x2w2) = 0.500\n",
      "do/d(x1w1) = 0.500\n",
      "do/d(x2w2) = 0.500\n",
      "do/x2 = 0.500\n",
      "do/w2 = 0.000\n",
      "do/x1 = -1.500\n",
      "do/w1 = 1.000\n"
     ]
    }
   ],
   "source": [
    "print(f\"do/do = {o.grad:.3f}\")\n",
    "print(f\"do/dn = {n.grad:.3f}\")\n",
    "print(f\"do/db = {b.grad:.3f}\")\n",
    "print(f\"do/d(x1w1 + x2w2) = {x1w1x2w2.grad:.3f}\")\n",
    "print(f\"do/d(x1w1) = {x1w1.grad:.3f}\")\n",
    "print(f\"do/d(x2w2) = {x2w2.grad:.3f}\")\n",
    "print(f\"do/x2 = {x2.grad:.3f}\")\n",
    "print(f\"do/w2 = {w2.grad:.3f}\")\n",
    "print(f\"do/x1 = {x1.grad:.3f}\")\n",
    "print(f\"do/w1 = {w1.grad:.3f}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "341cb696",
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 10.0.1 (20240210.2158)\n",
       " -->\n",
       "<!-- Pages: 1 -->\n",
       "<svg width=\"1546pt\" height=\"210pt\"\n",
       " viewBox=\"0.00 0.00 1546.25 210.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 206)\">\n",
       "<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-206 1542.25,-206 1542.25,4 -4,4\"/>\n",
       "<!-- 140523648916000 -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>140523648916000</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"1044.75,-109.5 1044.75,-145.5 1228.5,-145.5 1228.5,-109.5 1044.75,-109.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"1056.12\" y=\"-122.7\" font-family=\"Times,serif\" font-size=\"14.00\">n</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"1067.5,-110 1067.5,-145.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"1107.38\" y=\"-122.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 0.8814</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"1147.25,-110 1147.25,-145.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"1187.88\" y=\"-122.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.5000</text>\n",
       "</g>\n",
       "<!-- 140523648916280tanh -->\n",
       "<g id=\"node10\" class=\"node\">\n",
       "<title>140523648916280tanh</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"1291.5\" cy=\"-127.5\" rx=\"27\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"1291.5\" y=\"-122.45\" font-family=\"Times,serif\" font-size=\"14.00\">tanh</text>\n",
       "</g>\n",
       "<!-- 140523648916000&#45;&gt;140523648916280tanh -->\n",
       "<g id=\"edge7\" class=\"edge\">\n",
       "<title>140523648916000&#45;&gt;140523648916280tanh</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M1228.76,-127.5C1237.14,-127.5 1245.27,-127.5 1252.73,-127.5\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1252.7,-131 1262.7,-127.5 1252.7,-124 1252.7,-131\"/>\n",
       "</g>\n",
       "<!-- 140523648916000+ -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>140523648916000+</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"981.75\" cy=\"-127.5\" rx=\"27\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"981.75\" y=\"-122.45\" font-family=\"Times,serif\" font-size=\"14.00\">+</text>\n",
       "</g>\n",
       "<!-- 140523648916000+&#45;&gt;140523648916000 -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>140523648916000+&#45;&gt;140523648916000</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M1009.03,-127.5C1016.17,-127.5 1024.36,-127.5 1033.07,-127.5\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1032.81,-131 1042.81,-127.5 1032.81,-124 1032.81,-131\"/>\n",
       "</g>\n",
       "<!-- 140523648916056 -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>140523648916056</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"698.25,-137.5 698.25,-173.5 882,-173.5 882,-137.5 698.25,-137.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"709.62\" y=\"-150.7\" font-family=\"Times,serif\" font-size=\"14.00\">b</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"721,-138 721,-173.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"760.88\" y=\"-150.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 6.8814</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"800.75,-138 800.75,-173.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"841.38\" y=\"-150.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.5000</text>\n",
       "</g>\n",
       "<!-- 140523648916056&#45;&gt;140523648916000+ -->\n",
       "<g id=\"edge14\" class=\"edge\">\n",
       "<title>140523648916056&#45;&gt;140523648916000+</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M882.33,-142.03C904.17,-138.81 926.27,-135.54 943.98,-132.93\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"944.33,-136.42 953.71,-131.49 943.31,-129.49 944.33,-136.42\"/>\n",
       "</g>\n",
       "<!-- 140523648916112 -->\n",
       "<g id=\"node4\" class=\"node\">\n",
       "<title>140523648916112</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"324,-110.5 324,-146.5 535.5,-146.5 535.5,-110.5 324,-110.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"347\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">x1w1</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"370,-111 370,-146.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"412.12\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">data &#45;6.0000</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"454.25,-111 454.25,-146.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"494.88\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.5000</text>\n",
       "</g>\n",
       "<!-- 140523648915888+ -->\n",
       "<g id=\"node14\" class=\"node\">\n",
       "<title>140523648915888+</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"598.5\" cy=\"-100.5\" rx=\"27\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"598.5\" y=\"-95.45\" font-family=\"Times,serif\" font-size=\"14.00\">+</text>\n",
       "</g>\n",
       "<!-- 140523648916112&#45;&gt;140523648915888+ -->\n",
       "<g id=\"edge8\" class=\"edge\">\n",
       "<title>140523648916112&#45;&gt;140523648915888+</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M535.88,-110.85C544.59,-109.38 552.96,-107.98 560.56,-106.7\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"561.06,-110.17 570.34,-105.06 559.9,-103.26 561.06,-110.17\"/>\n",
       "</g>\n",
       "<!-- 140523648916112* -->\n",
       "<g id=\"node5\" class=\"node\">\n",
       "<title>140523648916112*</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"261\" cy=\"-128.5\" rx=\"27\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"261\" y=\"-123.45\" font-family=\"Times,serif\" font-size=\"14.00\">*</text>\n",
       "</g>\n",
       "<!-- 140523648916112*&#45;&gt;140523648916112 -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>140523648916112*&#45;&gt;140523648916112</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M288.42,-128.5C295.52,-128.5 303.68,-128.5 312.42,-128.5\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"312.24,-132 322.24,-128.5 312.24,-125 312.24,-132\"/>\n",
       "</g>\n",
       "<!-- 140523648916224 -->\n",
       "<g id=\"node6\" class=\"node\">\n",
       "<title>140523648916224</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"326.25,-55.5 326.25,-91.5 533.25,-91.5 533.25,-55.5 326.25,-55.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"349.25\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">x2w2</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"372.25,-56 372.25,-91.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"412.12\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 0.0000</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"452,-56 452,-91.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"492.62\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.5000</text>\n",
       "</g>\n",
       "<!-- 140523648916224&#45;&gt;140523648915888+ -->\n",
       "<g id=\"edge10\" class=\"edge\">\n",
       "<title>140523648916224&#45;&gt;140523648915888+</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M533.51,-90.14C543.16,-91.7 552.45,-93.21 560.82,-94.56\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"560.05,-97.98 570.48,-96.12 561.16,-91.07 560.05,-97.98\"/>\n",
       "</g>\n",
       "<!-- 140523648916224* -->\n",
       "<g id=\"node7\" class=\"node\">\n",
       "<title>140523648916224*</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"261\" cy=\"-73.5\" rx=\"27\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"261\" y=\"-68.45\" font-family=\"Times,serif\" font-size=\"14.00\">*</text>\n",
       "</g>\n",
       "<!-- 140523648916224*&#45;&gt;140523648916224 -->\n",
       "<g id=\"edge3\" class=\"edge\">\n",
       "<title>140523648916224*&#45;&gt;140523648916224</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M288.42,-73.5C296.08,-73.5 304.96,-73.5 314.47,-73.5\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"314.37,-77 324.37,-73.5 314.37,-70 314.37,-77\"/>\n",
       "</g>\n",
       "<!-- 140523648915720 -->\n",
       "<g id=\"node8\" class=\"node\">\n",
       "<title>140523648915720</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"3.75,-55.5 3.75,-91.5 194.25,-91.5 194.25,-55.5 3.75,-55.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"18.5\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">x2</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"33.25,-56 33.25,-91.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"73.12\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 0.0000</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"113,-56 113,-91.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"153.62\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.5000</text>\n",
       "</g>\n",
       "<!-- 140523648915720&#45;&gt;140523648916224* -->\n",
       "<g id=\"edge6\" class=\"edge\">\n",
       "<title>140523648915720&#45;&gt;140523648916224*</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M194.46,-73.5C204.21,-73.5 213.66,-73.5 222.21,-73.5\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"222.12,-77 232.12,-73.5 222.12,-70 222.12,-77\"/>\n",
       "</g>\n",
       "<!-- 140523648916280 -->\n",
       "<g id=\"node9\" class=\"node\">\n",
       "<title>140523648916280</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"1354.5,-109.5 1354.5,-145.5 1538.25,-145.5 1538.25,-109.5 1354.5,-109.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"1365.88\" y=\"-122.7\" font-family=\"Times,serif\" font-size=\"14.00\">o</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"1377.25,-110 1377.25,-145.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"1417.12\" y=\"-122.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 0.7071</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"1457,-110 1457,-145.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"1497.62\" y=\"-122.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 1.0000</text>\n",
       "</g>\n",
       "<!-- 140523648916280tanh&#45;&gt;140523648916280 -->\n",
       "<g id=\"edge4\" class=\"edge\">\n",
       "<title>140523648916280tanh&#45;&gt;140523648916280</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M1318.78,-127.5C1325.92,-127.5 1334.11,-127.5 1342.82,-127.5\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1342.56,-131 1352.56,-127.5 1342.56,-124 1342.56,-131\"/>\n",
       "</g>\n",
       "<!-- 140523648915776 -->\n",
       "<g id=\"node11\" class=\"node\">\n",
       "<title>140523648915776</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"1.5,-165.5 1.5,-201.5 196.5,-201.5 196.5,-165.5 1.5,-165.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"16.25\" y=\"-178.7\" font-family=\"Times,serif\" font-size=\"14.00\">x1</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"31,-166 31,-201.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"70.88\" y=\"-178.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 2.0000</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"110.75,-166 110.75,-201.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"153.62\" y=\"-178.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad &#45;1.5000</text>\n",
       "</g>\n",
       "<!-- 140523648915776&#45;&gt;140523648916112* -->\n",
       "<g id=\"edge11\" class=\"edge\">\n",
       "<title>140523648915776&#45;&gt;140523648916112*</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M172.12,-165.01C180.9,-162.35 189.7,-159.5 198,-156.5 208.09,-152.86 218.82,-148.27 228.47,-143.88\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"229.86,-147.09 237.45,-139.69 226.9,-140.75 229.86,-147.09\"/>\n",
       "</g>\n",
       "<!-- 140523648915832 -->\n",
       "<g id=\"node12\" class=\"node\">\n",
       "<title>140523648915832</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"0,-110.5 0,-146.5 198,-146.5 198,-110.5 0,-110.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"16.25\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">w1</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"32.5,-111 32.5,-146.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"74.62\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">data &#45;3.0000</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"116.75,-111 116.75,-146.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"157.38\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 1.0000</text>\n",
       "</g>\n",
       "<!-- 140523648915832&#45;&gt;140523648916112* -->\n",
       "<g id=\"edge9\" class=\"edge\">\n",
       "<title>140523648915832&#45;&gt;140523648916112*</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M198.14,-128.5C206.61,-128.5 214.8,-128.5 222.29,-128.5\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"222.29,-132 232.29,-128.5 222.29,-125 222.29,-132\"/>\n",
       "</g>\n",
       "<!-- 140523648915888 -->\n",
       "<g id=\"node13\" class=\"node\">\n",
       "<title>140523648915888</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"661.5,-82.5 661.5,-118.5 918.75,-118.5 918.75,-82.5 661.5,-82.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"707.38\" y=\"-95.7\" font-family=\"Times,serif\" font-size=\"14.00\">x1w1 + x2w2</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"753.25,-83 753.25,-118.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"795.38\" y=\"-95.7\" font-family=\"Times,serif\" font-size=\"14.00\">data &#45;6.0000</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"837.5,-83 837.5,-118.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"878.12\" y=\"-95.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.5000</text>\n",
       "</g>\n",
       "<!-- 140523648915888&#45;&gt;140523648916000+ -->\n",
       "<g id=\"edge13\" class=\"edge\">\n",
       "<title>140523648915888&#45;&gt;140523648916000+</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M919.18,-118.73C927.81,-119.96 936.02,-121.13 943.46,-122.19\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"942.96,-125.65 953.35,-123.6 943.94,-118.72 942.96,-125.65\"/>\n",
       "</g>\n",
       "<!-- 140523648915888+&#45;&gt;140523648915888 -->\n",
       "<g id=\"edge5\" class=\"edge\">\n",
       "<title>140523648915888+&#45;&gt;140523648915888</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M625.87,-100.5C632.88,-100.5 640.96,-100.5 649.7,-100.5\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"649.57,-104 659.57,-100.5 649.57,-97 649.57,-104\"/>\n",
       "</g>\n",
       "<!-- 140523648915944 -->\n",
       "<g id=\"node15\" class=\"node\">\n",
       "<title>140523648915944</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"2.25,-0.5 2.25,-36.5 195.75,-36.5 195.75,-0.5 2.25,-0.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"18.5\" y=\"-13.7\" font-family=\"Times,serif\" font-size=\"14.00\">w2</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"34.75,-1 34.75,-36.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"74.62\" y=\"-13.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 1.0000</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"114.5,-1 114.5,-36.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"155.12\" y=\"-13.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
       "</g>\n",
       "<!-- 140523648915944&#45;&gt;140523648916224* -->\n",
       "<g id=\"edge12\" class=\"edge\">\n",
       "<title>140523648915944&#45;&gt;140523648916224*</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M168.91,-36.94C178.74,-39.93 188.67,-43.15 198,-46.5 207.96,-50.07 218.58,-54.47 228.18,-58.68\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"226.54,-61.78 237.1,-62.67 229.4,-55.39 226.54,-61.78\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.graphs.Digraph at 0x7fce3631aa58>"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "draw_dot(o)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "68eb8c69",
   "metadata": {},
   "source": [
    "Now, we understand how to do backpropagation. However, we cannot do this manual backpropagation to set the gradient for each parameter for highly complex neural networks like transformers with billions of parameters (**GPT-4 has ~1.76 trillion parameters**). This leads us to how to implement backpropagation automatically."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5cca99b9",
   "metadata": {},
   "source": [
    "-------------\n",
    "<br><br>\n",
    "# 5. Micrograd: Implementing Backward Function for Each Operation<a id=\"5\"></a>\n",
    "-----------------\n",
    "We are going to add `_backward` function to all the mathematical operations sub-functions (`__add__`, `__mul__`, `__pow__`, `tanh` and `exp`) in the class `Value` object. Now, there's no need to calculate the individual backpropagation gradient values manually. We can directly call the `_backward` method in the right order! This automatically calculates the gradients for each computational node."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "7900a1d1",
   "metadata": {},
   "outputs": [],
   "source": [
    "# inputs x1,x2\n",
    "x1 = Value(2.0,label='x1')\n",
    "x2 = Value(0.0,label='x2')\n",
    "# weights w1, w2\n",
    "w1 = Value(-3.0,label='w1')\n",
    "w2 = Value(1.0,label='w2')\n",
    "# bias of the neuron\n",
    "b = Value(6.8813735870195432,label='b')\n",
    "# x1*w1 + x2*w2 + b\n",
    "x1w1 = x1*w1; x1w1.label ='x1w1'\n",
    "x2w2 = x2*w2; x2w2.label ='x2w2'\n",
    "x1w1x2w2 = x1w1+x2w2; x1w1x2w2.label='x1w1 + x2w2'\n",
    "n = x1w1x2w2+b; n.label='n'\n",
    "o = n.tanh() \n",
    "o.label = 'o'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "5d6e4a2d",
   "metadata": {},
   "outputs": [],
   "source": [
    "o.grad = 1.0           #do/do (initialize to 1: agrees with the derivative too)\n",
    "\n",
    "o._backward()          #do/dn\n",
    "n._backward()          #do/db = do/d(x1w1+x2w2)\n",
    "b._backward()          #b is a leaf node, so it does not have a _backward() function\n",
    "x1w1x2w2._backward()   #do/d(x1w1) = do/d(x2w2)\n",
    "x2w2._backward()       #do/d(x2), do/d(w2)\n",
    "x1w1._backward()       #do/d(x1), do/d(w1)\n",
    "#x1, w1, x2, w2 are leaf nodes, therefore they don't have _backward() functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "4b978d47",
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 10.0.1 (20240210.2158)\n",
       " -->\n",
       "<!-- Pages: 1 -->\n",
       "<svg width=\"1546pt\" height=\"210pt\"\n",
       " viewBox=\"0.00 0.00 1546.25 210.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 206)\">\n",
       "<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-206 1542.25,-206 1542.25,4 -4,4\"/>\n",
       "<!-- 140523649072040 -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>140523649072040</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"2.25,-165.5 2.25,-201.5 195.75,-201.5 195.75,-165.5 2.25,-165.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"18.5\" y=\"-178.7\" font-family=\"Times,serif\" font-size=\"14.00\">w2</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"34.75,-166 34.75,-201.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"74.62\" y=\"-178.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 1.0000</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"114.5,-166 114.5,-201.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"155.12\" y=\"-178.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
       "</g>\n",
       "<!-- 140523649180112* -->\n",
       "<g id=\"node15\" class=\"node\">\n",
       "<title>140523649180112*</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"261\" cy=\"-128.5\" rx=\"27\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"261\" y=\"-123.45\" font-family=\"Times,serif\" font-size=\"14.00\">*</text>\n",
       "</g>\n",
       "<!-- 140523649072040&#45;&gt;140523649180112* -->\n",
       "<g id=\"edge9\" class=\"edge\">\n",
       "<title>140523649072040&#45;&gt;140523649180112*</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M172.12,-165.01C180.9,-162.35 189.7,-159.5 198,-156.5 208.09,-152.86 218.82,-148.27 228.47,-143.88\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"229.86,-147.09 237.45,-139.69 226.9,-140.75 229.86,-147.09\"/>\n",
       "</g>\n",
       "<!-- 140523649179720 -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>140523649179720</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"1044.75,-54.5 1044.75,-90.5 1228.5,-90.5 1228.5,-54.5 1044.75,-54.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"1056.12\" y=\"-67.7\" font-family=\"Times,serif\" font-size=\"14.00\">n</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"1067.5,-55 1067.5,-90.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"1107.38\" y=\"-67.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 0.8814</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"1147.25,-55 1147.25,-90.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"1187.88\" y=\"-67.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.5000</text>\n",
       "</g>\n",
       "<!-- 140523649183136tanh -->\n",
       "<g id=\"node11\" class=\"node\">\n",
       "<title>140523649183136tanh</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"1291.5\" cy=\"-72.5\" rx=\"27\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"1291.5\" y=\"-67.45\" font-family=\"Times,serif\" font-size=\"14.00\">tanh</text>\n",
       "</g>\n",
       "<!-- 140523649179720&#45;&gt;140523649183136tanh -->\n",
       "<g id=\"edge12\" class=\"edge\">\n",
       "<title>140523649179720&#45;&gt;140523649183136tanh</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M1228.76,-72.5C1237.14,-72.5 1245.27,-72.5 1252.73,-72.5\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1252.7,-76 1262.7,-72.5 1252.7,-69 1252.7,-76\"/>\n",
       "</g>\n",
       "<!-- 140523649179720+ -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>140523649179720+</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"981.75\" cy=\"-72.5\" rx=\"27\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"981.75\" y=\"-67.45\" font-family=\"Times,serif\" font-size=\"14.00\">+</text>\n",
       "</g>\n",
       "<!-- 140523649179720+&#45;&gt;140523649179720 -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>140523649179720+&#45;&gt;140523649179720</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M1009.03,-72.5C1016.17,-72.5 1024.36,-72.5 1033.07,-72.5\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1032.81,-76 1042.81,-72.5 1032.81,-69 1032.81,-76\"/>\n",
       "</g>\n",
       "<!-- 140523649181288 -->\n",
       "<g id=\"node4\" class=\"node\">\n",
       "<title>140523649181288</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"661.5,-82.5 661.5,-118.5 918.75,-118.5 918.75,-82.5 661.5,-82.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"707.38\" y=\"-95.7\" font-family=\"Times,serif\" font-size=\"14.00\">x1w1 + x2w2</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"753.25,-83 753.25,-118.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"795.38\" y=\"-95.7\" font-family=\"Times,serif\" font-size=\"14.00\">data &#45;6.0000</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"837.5,-83 837.5,-118.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"878.12\" y=\"-95.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.5000</text>\n",
       "</g>\n",
       "<!-- 140523649181288&#45;&gt;140523649179720+ -->\n",
       "<g id=\"edge7\" class=\"edge\">\n",
       "<title>140523649181288&#45;&gt;140523649179720+</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M916.29,-82.02C926.15,-80.56 935.53,-79.18 943.92,-77.94\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"944.19,-81.44 953.58,-76.51 943.17,-74.51 944.19,-81.44\"/>\n",
       "</g>\n",
       "<!-- 140523649181288+ -->\n",
       "<g id=\"node5\" class=\"node\">\n",
       "<title>140523649181288+</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"598.5\" cy=\"-100.5\" rx=\"27\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"598.5\" y=\"-95.45\" font-family=\"Times,serif\" font-size=\"14.00\">+</text>\n",
       "</g>\n",
       "<!-- 140523649181288+&#45;&gt;140523649181288 -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>140523649181288+&#45;&gt;140523649181288</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M625.87,-100.5C632.88,-100.5 640.96,-100.5 649.7,-100.5\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"649.57,-104 659.57,-100.5 649.57,-97 649.57,-104\"/>\n",
       "</g>\n",
       "<!-- 140523649244856 -->\n",
       "<g id=\"node6\" class=\"node\">\n",
       "<title>140523649244856</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"1.5,-55.5 1.5,-91.5 196.5,-91.5 196.5,-55.5 1.5,-55.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"16.25\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">x1</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"31,-56 31,-91.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"70.88\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 2.0000</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"110.75,-56 110.75,-91.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"153.62\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad &#45;1.5000</text>\n",
       "</g>\n",
       "<!-- 140523649181624* -->\n",
       "<g id=\"node13\" class=\"node\">\n",
       "<title>140523649181624*</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"261\" cy=\"-73.5\" rx=\"27\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"261\" y=\"-68.45\" font-family=\"Times,serif\" font-size=\"14.00\">*</text>\n",
       "</g>\n",
       "<!-- 140523649244856&#45;&gt;140523649181624* -->\n",
       "<g id=\"edge14\" class=\"edge\">\n",
       "<title>140523649244856&#45;&gt;140523649181624*</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M196.76,-73.5C205.77,-73.5 214.47,-73.5 222.4,-73.5\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"222.25,-77 232.25,-73.5 222.25,-70 222.25,-77\"/>\n",
       "</g>\n",
       "<!-- 140523649356576 -->\n",
       "<g id=\"node7\" class=\"node\">\n",
       "<title>140523649356576</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"3.75,-110.5 3.75,-146.5 194.25,-146.5 194.25,-110.5 3.75,-110.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"18.5\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">x2</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"33.25,-111 33.25,-146.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"73.12\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 0.0000</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"113,-111 113,-146.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"153.62\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.5000</text>\n",
       "</g>\n",
       "<!-- 140523649356576&#45;&gt;140523649180112* -->\n",
       "<g id=\"edge13\" class=\"edge\">\n",
       "<title>140523649356576&#45;&gt;140523649180112*</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M194.46,-128.5C204.21,-128.5 213.66,-128.5 222.21,-128.5\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"222.12,-132 232.12,-128.5 222.12,-125 222.12,-132\"/>\n",
       "</g>\n",
       "<!-- 140523649183024 -->\n",
       "<g id=\"node8\" class=\"node\">\n",
       "<title>140523649183024</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"698.25,-27.5 698.25,-63.5 882,-63.5 882,-27.5 698.25,-27.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"709.62\" y=\"-40.7\" font-family=\"Times,serif\" font-size=\"14.00\">b</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"721,-28 721,-63.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"760.88\" y=\"-40.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 6.8814</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"800.75,-28 800.75,-63.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"841.38\" y=\"-40.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.5000</text>\n",
       "</g>\n",
       "<!-- 140523649183024&#45;&gt;140523649179720+ -->\n",
       "<g id=\"edge10\" class=\"edge\">\n",
       "<title>140523649183024&#45;&gt;140523649179720+</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M882.33,-58.49C904.17,-61.6 926.27,-64.74 943.98,-67.26\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"943.32,-70.71 953.71,-68.65 944.3,-63.78 943.32,-70.71\"/>\n",
       "</g>\n",
       "<!-- 140523649245024 -->\n",
       "<g id=\"node9\" class=\"node\">\n",
       "<title>140523649245024</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"0,-0.5 0,-36.5 198,-36.5 198,-0.5 0,-0.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"16.25\" y=\"-13.7\" font-family=\"Times,serif\" font-size=\"14.00\">w1</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"32.5,-1 32.5,-36.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"74.62\" y=\"-13.7\" font-family=\"Times,serif\" font-size=\"14.00\">data &#45;3.0000</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"116.75,-1 116.75,-36.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"157.38\" y=\"-13.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 1.0000</text>\n",
       "</g>\n",
       "<!-- 140523649245024&#45;&gt;140523649181624* -->\n",
       "<g id=\"edge8\" class=\"edge\">\n",
       "<title>140523649245024&#45;&gt;140523649181624*</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M168.91,-36.94C178.74,-39.93 188.67,-43.15 198,-46.5 207.96,-50.07 218.58,-54.47 228.18,-58.68\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"226.54,-61.78 237.1,-62.67 229.4,-55.39 226.54,-61.78\"/>\n",
       "</g>\n",
       "<!-- 140523649183136 -->\n",
       "<g id=\"node10\" class=\"node\">\n",
       "<title>140523649183136</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"1354.5,-54.5 1354.5,-90.5 1538.25,-90.5 1538.25,-54.5 1354.5,-54.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"1365.88\" y=\"-67.7\" font-family=\"Times,serif\" font-size=\"14.00\">o</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"1377.25,-55 1377.25,-90.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"1417.12\" y=\"-67.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 0.7071</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"1457,-55 1457,-90.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"1497.62\" y=\"-67.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 1.0000</text>\n",
       "</g>\n",
       "<!-- 140523649183136tanh&#45;&gt;140523649183136 -->\n",
       "<g id=\"edge3\" class=\"edge\">\n",
       "<title>140523649183136tanh&#45;&gt;140523649183136</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M1318.78,-72.5C1325.92,-72.5 1334.11,-72.5 1342.82,-72.5\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1342.56,-76 1352.56,-72.5 1342.56,-69 1342.56,-76\"/>\n",
       "</g>\n",
       "<!-- 140523649181624 -->\n",
       "<g id=\"node12\" class=\"node\">\n",
       "<title>140523649181624</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"324,-55.5 324,-91.5 535.5,-91.5 535.5,-55.5 324,-55.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"347\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">x1w1</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"370,-56 370,-91.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"412.12\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">data &#45;6.0000</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"454.25,-56 454.25,-91.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"494.88\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.5000</text>\n",
       "</g>\n",
       "<!-- 140523649181624&#45;&gt;140523649181288+ -->\n",
       "<g id=\"edge11\" class=\"edge\">\n",
       "<title>140523649181624&#45;&gt;140523649181288+</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M535.88,-90.52C544.59,-91.93 552.96,-93.29 560.56,-94.52\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"559.91,-97.96 570.34,-96.1 561.03,-91.05 559.91,-97.96\"/>\n",
       "</g>\n",
       "<!-- 140523649181624*&#45;&gt;140523649181624 -->\n",
       "<g id=\"edge4\" class=\"edge\">\n",
       "<title>140523649181624*&#45;&gt;140523649181624</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M288.42,-73.5C295.52,-73.5 303.68,-73.5 312.42,-73.5\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"312.24,-77 322.24,-73.5 312.24,-70 312.24,-77\"/>\n",
       "</g>\n",
       "<!-- 140523649180112 -->\n",
       "<g id=\"node14\" class=\"node\">\n",
       "<title>140523649180112</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"326.25,-110.5 326.25,-146.5 533.25,-146.5 533.25,-110.5 326.25,-110.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"349.25\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">x2w2</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"372.25,-111 372.25,-146.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"412.12\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 0.0000</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"452,-111 452,-146.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"492.62\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.5000</text>\n",
       "</g>\n",
       "<!-- 140523649180112&#45;&gt;140523649181288+ -->\n",
       "<g id=\"edge6\" class=\"edge\">\n",
       "<title>140523649180112&#45;&gt;140523649181288+</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M533.51,-111.25C543.16,-109.62 552.45,-108.06 560.82,-106.66\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"561.2,-110.15 570.48,-105.04 560.04,-103.24 561.2,-110.15\"/>\n",
       "</g>\n",
       "<!-- 140523649180112*&#45;&gt;140523649180112 -->\n",
       "<g id=\"edge5\" class=\"edge\">\n",
       "<title>140523649180112*&#45;&gt;140523649180112</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M288.42,-128.5C296.08,-128.5 304.96,-128.5 314.47,-128.5\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"314.37,-132 324.37,-128.5 314.37,-125 314.37,-132\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.graphs.Digraph at 0x7fce36344400>"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "draw_dot(o)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ef782f39",
   "metadata": {},
   "source": [
    "Note that we had to set `o.grad = 1.0` manually. Why? Because the placeholder gradient value for each node is 0. The `_backward` function always depends on the previous gradient in the computational graph and if it was 0 then all gradients will turn out to be 0 (think about it).\n",
    "\n",
    "But do we have to call the `_backward` function manually? Of course not! We can use a loop here, but notice that we need to iterate over the nodes in a certain order. We can’t call `_backward` on a node that has neighbors connected via outgoing edges whose gradients haven’t been calculated already. For this, we need something called topological sorting.\n",
    "\n",
    "To calculate the gradient at a node in the neural network, its dependencies must be calculated before it. This can be achieved using **topological sort**. A layout of the graph structure whereby all the edges go only from left to right (L-R). In computer science, a [topological sort](https://en.wikipedia.org/wiki/Topological_sorting) or topological ordering of a directed acyclic graph  **(DAG)** is a linear ordering of its vertices such that for every **directed edge UV** from **vertex U** to **vertex V**, <u>**U** comes before **V**</u> in the ordering. As a rule, **cyclic graphs** <u>don't have valid topological orderings.</u>\n",
    "\n",
    "\n",
    "<a id=\"a3\"></a>\n",
    "![Toplogical Sort](_imgs/topological_sort.png)\n",
    "\n",
    "**Fig 3. Topological Sort.** ([Source](https://medium.com/@konduruharish/topological-sort-in-typescript-and-c-6d5ecc4bad95))<br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e60d6526",
   "metadata": {},
   "source": [
    "For the directed graph represented in the above figure, this is one of the topological sort for the DAG:\n",
    "* A → B → C → D → G → F → E. \n",
    "\n",
    "To put it simply, in a topological order of vertices, there will be no edge pointing backward in its linear representation of all vertices. \n",
    "Until now in the `Value` class, we have devised mathematical operation functions with their individual `_backward` methods. Each operation can backpropagate the flowing gradients to its children only.\n",
    "Now, we can create a global `backward` method in `Value` class that leverages topological sorting via an internal local `build_topo` function to iterate over all such mathematical functions in a DAG (built equation) and back-propagate the gradient over the entire computational graph. This function works by:\n",
    "\n",
    "- sorting the DAG in a topological order using `build_topo`\n",
    "- storing the topologically sorted list\n",
    "- setting the `grad` of the last node to 1.0\n",
    "- iterating over the topologically sorted list in reverse order (backpropagation: reverse differentiation) and applying the `_backward` method of each mathematical operation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "b8df5ea8",
   "metadata": {},
   "outputs": [],
   "source": [
    "# inputs x1,x2\n",
    "x1 = Value(2.0,label='x1')\n",
    "x2 = Value(0.0,label='x2')\n",
    "# weights w1, w2\n",
    "w1 = Value(-3.0,label='w1')\n",
    "w2 = Value(1.0,label='w2')\n",
    "# bias of the neuron\n",
    "b = Value(6.8813735870195432,label='b')\n",
    "# x1*w1 + x2*w2 + b\n",
    "x1w1 = x1*w1; x1w1.label ='x1w1'\n",
    "x2w2 = x2*w2; x2w2.label ='x2w2'\n",
    "x1w1x2w2 = x1w1+x2w2; x1w1x2w2.label='x1w1 + x2w2'\n",
    "n = x1w1x2w2+b; n.label='n'\n",
    "o = n.tanh() \n",
    "o.label = 'o'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "c0ca0d69",
   "metadata": {},
   "outputs": [],
   "source": [
    "o.backward()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "c7674784",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 10.0.1 (20240210.2158)\n",
       " -->\n",
       "<!-- Pages: 1 -->\n",
       "<svg width=\"1546pt\" height=\"210pt\"\n",
       " viewBox=\"0.00 0.00 1546.25 210.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 206)\">\n",
       "<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-206 1542.25,-206 1542.25,4 -4,4\"/>\n",
       "<!-- 140523649295416 -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>140523649295416</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"324,-55.5 324,-91.5 535.5,-91.5 535.5,-55.5 324,-55.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"347\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">x1w1</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"370,-56 370,-91.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"412.12\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">data &#45;6.0000</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"454.25,-56 454.25,-91.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"494.88\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.5000</text>\n",
       "</g>\n",
       "<!-- 140523649297264+ -->\n",
       "<g id=\"node12\" class=\"node\">\n",
       "<title>140523649297264+</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"598.5\" cy=\"-100.5\" rx=\"27\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"598.5\" y=\"-95.45\" font-family=\"Times,serif\" font-size=\"14.00\">+</text>\n",
       "</g>\n",
       "<!-- 140523649295416&#45;&gt;140523649297264+ -->\n",
       "<g id=\"edge11\" class=\"edge\">\n",
       "<title>140523649295416&#45;&gt;140523649297264+</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M535.88,-90.52C544.59,-91.93 552.96,-93.29 560.56,-94.52\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"559.91,-97.96 570.34,-96.1 561.03,-91.05 559.91,-97.96\"/>\n",
       "</g>\n",
       "<!-- 140523649295416* -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>140523649295416*</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"261\" cy=\"-73.5\" rx=\"27\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"261\" y=\"-68.45\" font-family=\"Times,serif\" font-size=\"14.00\">*</text>\n",
       "</g>\n",
       "<!-- 140523649295416*&#45;&gt;140523649295416 -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>140523649295416*&#45;&gt;140523649295416</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M288.42,-73.5C295.52,-73.5 303.68,-73.5 312.42,-73.5\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"312.24,-77 322.24,-73.5 312.24,-70 312.24,-77\"/>\n",
       "</g>\n",
       "<!-- 140523649296984 -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>140523649296984</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"698.25,-137.5 698.25,-173.5 882,-173.5 882,-137.5 698.25,-137.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"709.62\" y=\"-150.7\" font-family=\"Times,serif\" font-size=\"14.00\">b</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"721,-138 721,-173.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"760.88\" y=\"-150.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 6.8814</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"800.75,-138 800.75,-173.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"841.38\" y=\"-150.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.5000</text>\n",
       "</g>\n",
       "<!-- 140523649297376+ -->\n",
       "<g id=\"node15\" class=\"node\">\n",
       "<title>140523649297376+</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"981.75\" cy=\"-127.5\" rx=\"27\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"981.75\" y=\"-122.45\" font-family=\"Times,serif\" font-size=\"14.00\">+</text>\n",
       "</g>\n",
       "<!-- 140523649296984&#45;&gt;140523649297376+ -->\n",
       "<g id=\"edge6\" class=\"edge\">\n",
       "<title>140523649296984&#45;&gt;140523649297376+</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M882.33,-142.03C904.17,-138.81 926.27,-135.54 943.98,-132.93\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"944.33,-136.42 953.71,-131.49 943.31,-129.49 944.33,-136.42\"/>\n",
       "</g>\n",
       "<!-- 140523649297544 -->\n",
       "<g id=\"node4\" class=\"node\">\n",
       "<title>140523649297544</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"1.5,-55.5 1.5,-91.5 196.5,-91.5 196.5,-55.5 1.5,-55.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"16.25\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">x1</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"31,-56 31,-91.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"70.88\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 2.0000</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"110.75,-56 110.75,-91.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"153.62\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad &#45;1.5000</text>\n",
       "</g>\n",
       "<!-- 140523649297544&#45;&gt;140523649295416* -->\n",
       "<g id=\"edge9\" class=\"edge\">\n",
       "<title>140523649297544&#45;&gt;140523649295416*</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M196.76,-73.5C205.77,-73.5 214.47,-73.5 222.4,-73.5\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"222.25,-77 232.25,-73.5 222.25,-70 222.25,-77\"/>\n",
       "</g>\n",
       "<!-- 140523649297600 -->\n",
       "<g id=\"node5\" class=\"node\">\n",
       "<title>140523649297600</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"3.75,-165.5 3.75,-201.5 194.25,-201.5 194.25,-165.5 3.75,-165.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"18.5\" y=\"-178.7\" font-family=\"Times,serif\" font-size=\"14.00\">x2</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"33.25,-166 33.25,-201.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"73.12\" y=\"-178.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 0.0000</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"113,-166 113,-201.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"153.62\" y=\"-178.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.5000</text>\n",
       "</g>\n",
       "<!-- 140523649295696* -->\n",
       "<g id=\"node10\" class=\"node\">\n",
       "<title>140523649295696*</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"261\" cy=\"-128.5\" rx=\"27\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"261\" y=\"-123.45\" font-family=\"Times,serif\" font-size=\"14.00\">*</text>\n",
       "</g>\n",
       "<!-- 140523649297600&#45;&gt;140523649295696* -->\n",
       "<g id=\"edge7\" class=\"edge\">\n",
       "<title>140523649297600&#45;&gt;140523649295696*</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M172.12,-165.01C180.9,-162.35 189.7,-159.5 198,-156.5 208.09,-152.86 218.82,-148.27 228.47,-143.88\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"229.86,-147.09 237.45,-139.69 226.9,-140.75 229.86,-147.09\"/>\n",
       "</g>\n",
       "<!-- 140523649297152 -->\n",
       "<g id=\"node6\" class=\"node\">\n",
       "<title>140523649297152</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"0,-0.5 0,-36.5 198,-36.5 198,-0.5 0,-0.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"16.25\" y=\"-13.7\" font-family=\"Times,serif\" font-size=\"14.00\">w1</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"32.5,-1 32.5,-36.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"74.62\" y=\"-13.7\" font-family=\"Times,serif\" font-size=\"14.00\">data &#45;3.0000</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"116.75,-1 116.75,-36.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"157.38\" y=\"-13.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 1.0000</text>\n",
       "</g>\n",
       "<!-- 140523649297152&#45;&gt;140523649295416* -->\n",
       "<g id=\"edge12\" class=\"edge\">\n",
       "<title>140523649297152&#45;&gt;140523649295416*</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M168.91,-36.94C178.74,-39.93 188.67,-43.15 198,-46.5 207.96,-50.07 218.58,-54.47 228.18,-58.68\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"226.54,-61.78 237.1,-62.67 229.4,-55.39 226.54,-61.78\"/>\n",
       "</g>\n",
       "<!-- 140523649294632 -->\n",
       "<g id=\"node7\" class=\"node\">\n",
       "<title>140523649294632</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"1354.5,-109.5 1354.5,-145.5 1538.25,-145.5 1538.25,-109.5 1354.5,-109.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"1365.88\" y=\"-122.7\" font-family=\"Times,serif\" font-size=\"14.00\">o</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"1377.25,-110 1377.25,-145.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"1417.12\" y=\"-122.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 0.7071</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"1457,-110 1457,-145.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"1497.62\" y=\"-122.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 1.0000</text>\n",
       "</g>\n",
       "<!-- 140523649294632tanh -->\n",
       "<g id=\"node8\" class=\"node\">\n",
       "<title>140523649294632tanh</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"1291.5\" cy=\"-127.5\" rx=\"27\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"1291.5\" y=\"-122.45\" font-family=\"Times,serif\" font-size=\"14.00\">tanh</text>\n",
       "</g>\n",
       "<!-- 140523649294632tanh&#45;&gt;140523649294632 -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>140523649294632tanh&#45;&gt;140523649294632</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M1318.78,-127.5C1325.92,-127.5 1334.11,-127.5 1342.82,-127.5\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1342.56,-131 1352.56,-127.5 1342.56,-124 1342.56,-131\"/>\n",
       "</g>\n",
       "<!-- 140523649295696 -->\n",
       "<g id=\"node9\" class=\"node\">\n",
       "<title>140523649295696</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"326.25,-110.5 326.25,-146.5 533.25,-146.5 533.25,-110.5 326.25,-110.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"349.25\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">x2w2</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"372.25,-111 372.25,-146.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"412.12\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 0.0000</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"452,-111 452,-146.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"492.62\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.5000</text>\n",
       "</g>\n",
       "<!-- 140523649295696&#45;&gt;140523649297264+ -->\n",
       "<g id=\"edge13\" class=\"edge\">\n",
       "<title>140523649295696&#45;&gt;140523649297264+</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M533.51,-111.25C543.16,-109.62 552.45,-108.06 560.82,-106.66\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"561.2,-110.15 570.48,-105.04 560.04,-103.24 561.2,-110.15\"/>\n",
       "</g>\n",
       "<!-- 140523649295696*&#45;&gt;140523649295696 -->\n",
       "<g id=\"edge3\" class=\"edge\">\n",
       "<title>140523649295696*&#45;&gt;140523649295696</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M288.42,-128.5C296.08,-128.5 304.96,-128.5 314.47,-128.5\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"314.37,-132 324.37,-128.5 314.37,-125 314.37,-132\"/>\n",
       "</g>\n",
       "<!-- 140523649297264 -->\n",
       "<g id=\"node11\" class=\"node\">\n",
       "<title>140523649297264</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"661.5,-82.5 661.5,-118.5 918.75,-118.5 918.75,-82.5 661.5,-82.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"707.38\" y=\"-95.7\" font-family=\"Times,serif\" font-size=\"14.00\">x1w1 + x2w2</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"753.25,-83 753.25,-118.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"795.38\" y=\"-95.7\" font-family=\"Times,serif\" font-size=\"14.00\">data &#45;6.0000</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"837.5,-83 837.5,-118.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"878.12\" y=\"-95.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.5000</text>\n",
       "</g>\n",
       "<!-- 140523649297264&#45;&gt;140523649297376+ -->\n",
       "<g id=\"edge8\" class=\"edge\">\n",
       "<title>140523649297264&#45;&gt;140523649297376+</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M919.18,-118.73C927.81,-119.96 936.02,-121.13 943.46,-122.19\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"942.96,-125.65 953.35,-123.6 943.94,-118.72 942.96,-125.65\"/>\n",
       "</g>\n",
       "<!-- 140523649297264+&#45;&gt;140523649297264 -->\n",
       "<g id=\"edge4\" class=\"edge\">\n",
       "<title>140523649297264+&#45;&gt;140523649297264</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M625.87,-100.5C632.88,-100.5 640.96,-100.5 649.7,-100.5\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"649.57,-104 659.57,-100.5 649.57,-97 649.57,-104\"/>\n",
       "</g>\n",
       "<!-- 140523649297320 -->\n",
       "<g id=\"node13\" class=\"node\">\n",
       "<title>140523649297320</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"2.25,-110.5 2.25,-146.5 195.75,-146.5 195.75,-110.5 2.25,-110.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"18.5\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">w2</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"34.75,-111 34.75,-146.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"74.62\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 1.0000</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"114.5,-111 114.5,-146.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"155.12\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
       "</g>\n",
       "<!-- 140523649297320&#45;&gt;140523649295696* -->\n",
       "<g id=\"edge10\" class=\"edge\">\n",
       "<title>140523649297320&#45;&gt;140523649295696*</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M195.84,-128.5C205.14,-128.5 214.15,-128.5 222.32,-128.5\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"222.13,-132 232.13,-128.5 222.13,-125 222.13,-132\"/>\n",
       "</g>\n",
       "<!-- 140523649297376 -->\n",
       "<g id=\"node14\" class=\"node\">\n",
       "<title>140523649297376</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"1044.75,-109.5 1044.75,-145.5 1228.5,-145.5 1228.5,-109.5 1044.75,-109.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"1056.12\" y=\"-122.7\" font-family=\"Times,serif\" font-size=\"14.00\">n</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"1067.5,-110 1067.5,-145.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"1107.38\" y=\"-122.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 0.8814</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"1147.25,-110 1147.25,-145.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"1187.88\" y=\"-122.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.5000</text>\n",
       "</g>\n",
       "<!-- 140523649297376&#45;&gt;140523649294632tanh -->\n",
       "<g id=\"edge14\" class=\"edge\">\n",
       "<title>140523649297376&#45;&gt;140523649294632tanh</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M1228.76,-127.5C1237.14,-127.5 1245.27,-127.5 1252.73,-127.5\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1252.7,-131 1262.7,-127.5 1252.7,-124 1252.7,-131\"/>\n",
       "</g>\n",
       "<!-- 140523649297376+&#45;&gt;140523649297376 -->\n",
       "<g id=\"edge5\" class=\"edge\">\n",
       "<title>140523649297376+&#45;&gt;140523649297376</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M1009.03,-127.5C1016.17,-127.5 1024.36,-127.5 1033.07,-127.5\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1032.81,-131 1042.81,-127.5 1032.81,-124 1032.81,-131\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.graphs.Digraph at 0x7fce3633dfd0>"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "draw_dot(o)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "804aa2f4",
   "metadata": {},
   "source": [
    "-------------\n",
    "<br><br>\n",
    "# 6. Micrograd: Fixing a Backprop Bug when One Node is Used Multiple Times<a id=\"6\"></a>\n",
    "-----------------\n",
    "\n",
    "Problem arises when a variable/node in the DAG is used more than once due to our current implementation of `Value`. Let's see in the example below  \n",
    "\n",
    "<u>Note:</u> Since the class `Value` object is updated in a cell above, the example below will give the right answer upon restarting the notebook after the class `Value` object has been updated to account for **accumulation of gradients**. This example was to show the initial bug issue prior to the update of the `Value` class object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "9026231a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 10.0.1 (20240210.2158)\n",
       " -->\n",
       "<!-- Pages: 1 -->\n",
       "<svg width=\"501pt\" height=\"45pt\"\n",
       " viewBox=\"0.00 0.00 500.75 45.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 41)\">\n",
       "<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-41 496.75,-41 496.75,4 -4,4\"/>\n",
       "<!-- 140523649357976 -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>140523649357976</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"0,-0.5 0,-36.5 183,-36.5 183,-0.5 0,-0.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"11\" y=\"-13.7\" font-family=\"Times,serif\" font-size=\"14.00\">a</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"22,-1 22,-36.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"61.88\" y=\"-13.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 3.0000</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"101.75,-1 101.75,-36.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"142.38\" y=\"-13.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 2.0000</text>\n",
       "</g>\n",
       "<!-- 140523649358816+ -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>140523649358816+</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"246\" cy=\"-18.5\" rx=\"27\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"246\" y=\"-13.45\" font-family=\"Times,serif\" font-size=\"14.00\">+</text>\n",
       "</g>\n",
       "<!-- 140523649357976&#45;&gt;140523649358816+ -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>140523649357976&#45;&gt;140523649358816+</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M183.41,-18.5C191.77,-18.5 199.88,-18.5 207.32,-18.5\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"207.26,-22 217.26,-18.5 207.26,-15 207.26,-22\"/>\n",
       "</g>\n",
       "<!-- 140523649358816 -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>140523649358816</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"309,-0.5 309,-36.5 492.75,-36.5 492.75,-0.5 309,-0.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"320.38\" y=\"-13.7\" font-family=\"Times,serif\" font-size=\"14.00\">b</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"331.75,-1 331.75,-36.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"371.62\" y=\"-13.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 6.0000</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"411.5,-1 411.5,-36.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"452.12\" y=\"-13.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 1.0000</text>\n",
       "</g>\n",
       "<!-- 140523649358816+&#45;&gt;140523649358816 -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>140523649358816+&#45;&gt;140523649358816</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M273.28,-18.5C280.42,-18.5 288.61,-18.5 297.32,-18.5\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"297.06,-22 307.06,-18.5 297.06,-15 297.06,-22\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.graphs.Digraph at 0x7fce3633dac8>"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a = Value(3.0, label = 'a')\n",
    "b = a + a ; b.label ='b'\n",
    "b.backward()\n",
    "draw_dot(b)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4fdd539f",
   "metadata": {},
   "source": [
    "The gradient of b WRT a (`a.grad` or `db/da`) in the cell above should be 2 but the answer above is 1 which is wrong. We have just propagated the gradient of the output to the input. We have given only one object as the input. Thus, the gradient machine ignores the second instance of the same input object. **We will get a wrong answer whenever we use a variable more than once unless we fix the bug.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "ca04d7ef",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 10.0.1 (20240210.2158)\n",
       " -->\n",
       "<!-- Pages: 1 -->\n",
       "<svg width=\"826pt\" height=\"100pt\"\n",
       " viewBox=\"0.00 0.00 826.25 100.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 96)\">\n",
       "<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-96 822.25,-96 822.25,4 -4,4\"/>\n",
       "<!-- 140523649181848 -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>140523649181848</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"1.88,-55.5 1.88,-91.5 190.12,-91.5 190.12,-55.5 1.88,-55.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"13.25\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">b</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"24.62,-56 24.62,-91.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"64.5\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 3.0000</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"104.38,-56 104.38,-91.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"147.25\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad &#45;8.0000</text>\n",
       "</g>\n",
       "<!-- 140523649298216+ -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>140523649298216+</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"255\" cy=\"-73.5\" rx=\"27\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"255\" y=\"-68.45\" font-family=\"Times,serif\" font-size=\"14.00\">+</text>\n",
       "</g>\n",
       "<!-- 140523649181848&#45;&gt;140523649298216+ -->\n",
       "<g id=\"edge9\" class=\"edge\">\n",
       "<title>140523649181848&#45;&gt;140523649298216+</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M190.59,-73.5C199.58,-73.5 208.29,-73.5 216.23,-73.5\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"216.11,-77 226.11,-73.5 216.11,-70 216.11,-77\"/>\n",
       "</g>\n",
       "<!-- 140523647396720* -->\n",
       "<g id=\"node5\" class=\"node\">\n",
       "<title>140523647396720*</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"255\" cy=\"-18.5\" rx=\"27\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"255\" y=\"-13.45\" font-family=\"Times,serif\" font-size=\"14.00\">*</text>\n",
       "</g>\n",
       "<!-- 140523649181848&#45;&gt;140523647396720* -->\n",
       "<g id=\"edge6\" class=\"edge\">\n",
       "<title>140523649181848&#45;&gt;140523647396720*</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M166.39,-55.06C175.08,-52.38 183.79,-49.5 192,-46.5 201.99,-42.85 212.63,-38.3 222.22,-33.93\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"223.55,-37.17 231.15,-29.78 220.6,-30.83 223.55,-37.17\"/>\n",
       "</g>\n",
       "<!-- 140523649298216 -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>140523649298216</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"318.38,-55.5 318.38,-91.5 505.88,-91.5 505.88,-55.5 318.38,-55.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"329.38\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">e</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"340.38,-56 340.38,-91.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"380.25\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 1.0000</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"420.12,-56 420.12,-91.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"463\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad &#45;6.0000</text>\n",
       "</g>\n",
       "<!-- 140523649295304* -->\n",
       "<g id=\"node8\" class=\"node\">\n",
       "<title>140523649295304*</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"569.25\" cy=\"-45.5\" rx=\"27\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"569.25\" y=\"-40.45\" font-family=\"Times,serif\" font-size=\"14.00\">*</text>\n",
       "</g>\n",
       "<!-- 140523649298216&#45;&gt;140523649295304* -->\n",
       "<g id=\"edge7\" class=\"edge\">\n",
       "<title>140523649298216&#45;&gt;140523649295304*</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M506.05,-56.73C514.96,-55.12 523.59,-53.56 531.44,-52.14\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"531.99,-55.6 541.21,-50.38 530.75,-48.71 531.99,-55.6\"/>\n",
       "</g>\n",
       "<!-- 140523649298216+&#45;&gt;140523649298216 -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>140523649298216+&#45;&gt;140523649298216</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M282.31,-73.5C289.55,-73.5 297.86,-73.5 306.7,-73.5\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"306.62,-77 316.62,-73.5 306.62,-70 306.62,-77\"/>\n",
       "</g>\n",
       "<!-- 140523647396720 -->\n",
       "<g id=\"node4\" class=\"node\">\n",
       "<title>140523647396720</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"318,-0.5 318,-36.5 506.25,-36.5 506.25,-0.5 318,-0.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"329.38\" y=\"-13.7\" font-family=\"Times,serif\" font-size=\"14.00\">d</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"340.75,-1 340.75,-36.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"382.88\" y=\"-13.7\" font-family=\"Times,serif\" font-size=\"14.00\">data &#45;6.0000</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"425,-1 425,-36.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"465.62\" y=\"-13.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 1.0000</text>\n",
       "</g>\n",
       "<!-- 140523647396720&#45;&gt;140523649295304* -->\n",
       "<g id=\"edge5\" class=\"edge\">\n",
       "<title>140523647396720&#45;&gt;140523649295304*</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M506.49,-34.75C515.27,-36.28 523.77,-37.76 531.51,-39.1\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"530.66,-42.51 541.11,-40.78 531.86,-35.61 530.66,-42.51\"/>\n",
       "</g>\n",
       "<!-- 140523647396720*&#45;&gt;140523647396720 -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>140523647396720*&#45;&gt;140523647396720</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M282.31,-18.5C289.49,-18.5 297.72,-18.5 306.47,-18.5\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"306.29,-22 316.29,-18.5 306.29,-15 306.29,-22\"/>\n",
       "</g>\n",
       "<!-- 140523647395712 -->\n",
       "<g id=\"node6\" class=\"node\">\n",
       "<title>140523647395712</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"0,-0.5 0,-36.5 192,-36.5 192,-0.5 0,-0.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"11\" y=\"-13.7\" font-family=\"Times,serif\" font-size=\"14.00\">a</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"22,-1 22,-36.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"64.12\" y=\"-13.7\" font-family=\"Times,serif\" font-size=\"14.00\">data &#45;2.0000</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"106.25,-1 106.25,-36.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"149.12\" y=\"-13.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad &#45;3.0000</text>\n",
       "</g>\n",
       "<!-- 140523647395712&#45;&gt;140523649298216+ -->\n",
       "<g id=\"edge4\" class=\"edge\">\n",
       "<title>140523647395712&#45;&gt;140523649298216+</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M163.53,-36.96C173.14,-39.96 182.87,-43.18 192,-46.5 201.86,-50.08 212.39,-54.46 221.92,-58.63\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"220.23,-61.71 230.79,-62.59 223.09,-55.32 220.23,-61.71\"/>\n",
       "</g>\n",
       "<!-- 140523647395712&#45;&gt;140523647396720* -->\n",
       "<g id=\"edge8\" class=\"edge\">\n",
       "<title>140523647395712&#45;&gt;140523647396720*</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M192.4,-18.5C200.73,-18.5 208.79,-18.5 216.18,-18.5\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"216.05,-22 226.05,-18.5 216.05,-15 216.05,-22\"/>\n",
       "</g>\n",
       "<!-- 140523649295304 -->\n",
       "<g id=\"node7\" class=\"node\">\n",
       "<title>140523649295304</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"632.25,-27.5 632.25,-63.5 818.25,-63.5 818.25,-27.5 632.25,-27.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"642.5\" y=\"-40.7\" font-family=\"Times,serif\" font-size=\"14.00\">f</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"652.75,-28 652.75,-63.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"694.88\" y=\"-40.7\" font-family=\"Times,serif\" font-size=\"14.00\">data &#45;6.0000</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"737,-28 737,-63.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"777.62\" y=\"-40.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 1.0000</text>\n",
       "</g>\n",
       "<!-- 140523649295304*&#45;&gt;140523649295304 -->\n",
       "<g id=\"edge3\" class=\"edge\">\n",
       "<title>140523649295304*&#45;&gt;140523649295304</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M596.72,-45.5C603.86,-45.5 612.03,-45.5 620.71,-45.5\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"620.44,-49 630.44,-45.5 620.44,-42 620.44,-49\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.graphs.Digraph at 0x7fce3632ef98>"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a = Value(-2.0,label='a')\n",
    "b = Value(3.0,label='b')\n",
    "\n",
    "\n",
    "d = a * b   ; d.label ='d'\n",
    "e = a + b   ; e.label = 'e'\n",
    "f = d * e   ; f.label='f'\n",
    "\n",
    "f.backward()\n",
    "\n",
    "draw_dot(f)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aa55e1d3",
   "metadata": {},
   "source": [
    "----------\n",
    "<br><br>\n",
    "## 6.1.  Multivariate Chain Rule for Derivatives<a id=\"601\"></a>\n",
    "-----\n",
    "The solution is that the gradients must be **accumulated** as we propagate from the output to each parameter node. This is derived from the [**multivariate chain rule.**](https://en.wikipedia.org/wiki/Chain_rule#Example) Let's show an [example below](https://machinelearningmastery.com/the-chain-rule-of-calculus-even-more-functions/).<br><br>\n",
    "$$\n",
    "h=g(r,s,t)=r^2-rs+t^3 \\\\\n",
    "r = x \\cos y \\\\ s = xe^y \\\\ t = x+y \\\\\n",
    "$$\n",
    "\n",
    "<a id=\"c3\"></a>\n",
    "Implementing the chain rule here requires the computation of partial derivatives, since we are working with multiple independent variables. Furthermore, $r$, $s$ and $t$ will also act as our intermediate variables. The formulae that we will be working with, defined with respect to each input, are the following:\n",
    "![multivariate chain rule](_imgs/multivariate_chain_rule.png)\n",
    "\n",
    "From these formulae, we can see that we will now need to find nine different partial derivatives. Again, we proceed to substitute these terms in the formulae for $\\partial h/\\partial x$ and $\\partial h/\\partial y$:\n",
    "![multivariate chain rule1](_imgs/multivariate_chain_rule1.png)\n",
    "\n",
    "And subsequently substitute for $r$, $s$ and $t$ to find the derivatives:\n",
    "![multivariate chain rule2](_imgs/multivariate_chain_rule2.png)\n",
    "\n",
    "Which may be simplified a little further (hint: apply the trigonometric identity $2\\sin y\\cos y=\\sin 2y$ to $\\partial h/\\partial y$):\n",
    "![multivariate chain rule3](_imgs/multivariate_chain_rule3.png)<br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c361f1da",
   "metadata": {},
   "source": [
    "In backpropagation, the gradient of a certain node will be calculated additively — the effects of all calculations will add up. In the class `Value` object, we replace `=` with `+=` in the `self.grad` and `other.grad` calculations under the local `_backward` method in `__add__`, `__mul__`, and `tanh` functions to reflect this accumulation of gradients that follows the multivariate chain rule. Now, we should get the **correct answer for the gradients** for the examples below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "88cb561b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 10.0.1 (20240210.2158)\n",
       " -->\n",
       "<!-- Pages: 1 -->\n",
       "<svg width=\"501pt\" height=\"45pt\"\n",
       " viewBox=\"0.00 0.00 500.75 45.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 41)\">\n",
       "<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-41 496.75,-41 496.75,4 -4,4\"/>\n",
       "<!-- 140523649357360 -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>140523649357360</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"309,-0.5 309,-36.5 492.75,-36.5 492.75,-0.5 309,-0.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"320.38\" y=\"-13.7\" font-family=\"Times,serif\" font-size=\"14.00\">b</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"331.75,-1 331.75,-36.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"371.62\" y=\"-13.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 6.0000</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"411.5,-1 411.5,-36.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"452.12\" y=\"-13.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 1.0000</text>\n",
       "</g>\n",
       "<!-- 140523649357360+ -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>140523649357360+</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"246\" cy=\"-18.5\" rx=\"27\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"246\" y=\"-13.45\" font-family=\"Times,serif\" font-size=\"14.00\">+</text>\n",
       "</g>\n",
       "<!-- 140523649357360+&#45;&gt;140523649357360 -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>140523649357360+&#45;&gt;140523649357360</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M273.28,-18.5C280.42,-18.5 288.61,-18.5 297.32,-18.5\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"297.06,-22 307.06,-18.5 297.06,-15 297.06,-22\"/>\n",
       "</g>\n",
       "<!-- 140523649357808 -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>140523649357808</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"0,-0.5 0,-36.5 183,-36.5 183,-0.5 0,-0.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"11\" y=\"-13.7\" font-family=\"Times,serif\" font-size=\"14.00\">a</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"22,-1 22,-36.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"61.88\" y=\"-13.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 3.0000</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"101.75,-1 101.75,-36.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"142.38\" y=\"-13.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 2.0000</text>\n",
       "</g>\n",
       "<!-- 140523649357808&#45;&gt;140523649357360+ -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>140523649357808&#45;&gt;140523649357360+</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M183.41,-18.5C191.77,-18.5 199.88,-18.5 207.32,-18.5\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"207.26,-22 217.26,-18.5 207.26,-15 207.26,-22\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.graphs.Digraph at 0x7fce3633d9b0>"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a = Value(3.0, label = 'a')\n",
    "b = a + a ; b.label ='b'\n",
    "b.backward()\n",
    "draw_dot(b)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "ed1517e7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 10.0.1 (20240210.2158)\n",
       " -->\n",
       "<!-- Pages: 1 -->\n",
       "<svg width=\"826pt\" height=\"100pt\"\n",
       " viewBox=\"0.00 0.00 826.25 100.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 96)\">\n",
       "<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-96 822.25,-96 822.25,4 -4,4\"/>\n",
       "<!-- 140523649297992 -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>140523649297992</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"632.25,-27.5 632.25,-63.5 818.25,-63.5 818.25,-27.5 632.25,-27.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"642.5\" y=\"-40.7\" font-family=\"Times,serif\" font-size=\"14.00\">f</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"652.75,-28 652.75,-63.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"694.88\" y=\"-40.7\" font-family=\"Times,serif\" font-size=\"14.00\">data &#45;6.0000</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"737,-28 737,-63.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"777.62\" y=\"-40.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 1.0000</text>\n",
       "</g>\n",
       "<!-- 140523649297992* -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>140523649297992*</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"569.25\" cy=\"-45.5\" rx=\"27\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"569.25\" y=\"-40.45\" font-family=\"Times,serif\" font-size=\"14.00\">*</text>\n",
       "</g>\n",
       "<!-- 140523649297992*&#45;&gt;140523649297992 -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>140523649297992*&#45;&gt;140523649297992</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M596.72,-45.5C603.86,-45.5 612.03,-45.5 620.71,-45.5\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"620.44,-49 630.44,-45.5 620.44,-42 620.44,-49\"/>\n",
       "</g>\n",
       "<!-- 140523649296536 -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>140523649296536</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"1.88,-55.5 1.88,-91.5 190.12,-91.5 190.12,-55.5 1.88,-55.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"13.25\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">b</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"24.62,-56 24.62,-91.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"64.5\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 3.0000</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"104.38,-56 104.38,-91.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"147.25\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad &#45;8.0000</text>\n",
       "</g>\n",
       "<!-- 140523649295584+ -->\n",
       "<g id=\"node5\" class=\"node\">\n",
       "<title>140523649295584+</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"255\" cy=\"-73.5\" rx=\"27\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"255\" y=\"-68.45\" font-family=\"Times,serif\" font-size=\"14.00\">+</text>\n",
       "</g>\n",
       "<!-- 140523649296536&#45;&gt;140523649295584+ -->\n",
       "<g id=\"edge4\" class=\"edge\">\n",
       "<title>140523649296536&#45;&gt;140523649295584+</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M190.59,-73.5C199.58,-73.5 208.29,-73.5 216.23,-73.5\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"216.11,-77 226.11,-73.5 216.11,-70 216.11,-77\"/>\n",
       "</g>\n",
       "<!-- 140523649298272* -->\n",
       "<g id=\"node7\" class=\"node\">\n",
       "<title>140523649298272*</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"255\" cy=\"-18.5\" rx=\"27\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"255\" y=\"-13.45\" font-family=\"Times,serif\" font-size=\"14.00\">*</text>\n",
       "</g>\n",
       "<!-- 140523649296536&#45;&gt;140523649298272* -->\n",
       "<g id=\"edge7\" class=\"edge\">\n",
       "<title>140523649296536&#45;&gt;140523649298272*</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M166.39,-55.06C175.08,-52.38 183.79,-49.5 192,-46.5 201.99,-42.85 212.63,-38.3 222.22,-33.93\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"223.55,-37.17 231.15,-29.78 220.6,-30.83 223.55,-37.17\"/>\n",
       "</g>\n",
       "<!-- 140523649295584 -->\n",
       "<g id=\"node4\" class=\"node\">\n",
       "<title>140523649295584</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"318.38,-55.5 318.38,-91.5 505.88,-91.5 505.88,-55.5 318.38,-55.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"329.38\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">e</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"340.38,-56 340.38,-91.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"380.25\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 1.0000</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"420.12,-56 420.12,-91.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"463\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad &#45;6.0000</text>\n",
       "</g>\n",
       "<!-- 140523649295584&#45;&gt;140523649297992* -->\n",
       "<g id=\"edge5\" class=\"edge\">\n",
       "<title>140523649295584&#45;&gt;140523649297992*</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M506.05,-56.73C514.96,-55.12 523.59,-53.56 531.44,-52.14\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"531.99,-55.6 541.21,-50.38 530.75,-48.71 531.99,-55.6\"/>\n",
       "</g>\n",
       "<!-- 140523649295584+&#45;&gt;140523649295584 -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>140523649295584+&#45;&gt;140523649295584</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M282.31,-73.5C289.55,-73.5 297.86,-73.5 306.7,-73.5\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"306.62,-77 316.62,-73.5 306.62,-70 306.62,-77\"/>\n",
       "</g>\n",
       "<!-- 140523649298272 -->\n",
       "<g id=\"node6\" class=\"node\">\n",
       "<title>140523649298272</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"318,-0.5 318,-36.5 506.25,-36.5 506.25,-0.5 318,-0.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"329.38\" y=\"-13.7\" font-family=\"Times,serif\" font-size=\"14.00\">d</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"340.75,-1 340.75,-36.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"382.88\" y=\"-13.7\" font-family=\"Times,serif\" font-size=\"14.00\">data &#45;6.0000</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"425,-1 425,-36.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"465.62\" y=\"-13.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 1.0000</text>\n",
       "</g>\n",
       "<!-- 140523649298272&#45;&gt;140523649297992* -->\n",
       "<g id=\"edge8\" class=\"edge\">\n",
       "<title>140523649298272&#45;&gt;140523649297992*</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M506.49,-34.75C515.27,-36.28 523.77,-37.76 531.51,-39.1\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"530.66,-42.51 541.11,-40.78 531.86,-35.61 530.66,-42.51\"/>\n",
       "</g>\n",
       "<!-- 140523649298272*&#45;&gt;140523649298272 -->\n",
       "<g id=\"edge3\" class=\"edge\">\n",
       "<title>140523649298272*&#45;&gt;140523649298272</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M282.31,-18.5C289.49,-18.5 297.72,-18.5 306.47,-18.5\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"306.29,-22 316.29,-18.5 306.29,-15 306.29,-22\"/>\n",
       "</g>\n",
       "<!-- 140523649295864 -->\n",
       "<g id=\"node8\" class=\"node\">\n",
       "<title>140523649295864</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"0,-0.5 0,-36.5 192,-36.5 192,-0.5 0,-0.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"11\" y=\"-13.7\" font-family=\"Times,serif\" font-size=\"14.00\">a</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"22,-1 22,-36.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"64.12\" y=\"-13.7\" font-family=\"Times,serif\" font-size=\"14.00\">data &#45;2.0000</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"106.25,-1 106.25,-36.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"149.12\" y=\"-13.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad &#45;3.0000</text>\n",
       "</g>\n",
       "<!-- 140523649295864&#45;&gt;140523649295584+ -->\n",
       "<g id=\"edge6\" class=\"edge\">\n",
       "<title>140523649295864&#45;&gt;140523649295584+</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M163.53,-36.96C173.14,-39.96 182.87,-43.18 192,-46.5 201.86,-50.08 212.39,-54.46 221.92,-58.63\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"220.23,-61.71 230.79,-62.59 223.09,-55.32 220.23,-61.71\"/>\n",
       "</g>\n",
       "<!-- 140523649295864&#45;&gt;140523649298272* -->\n",
       "<g id=\"edge9\" class=\"edge\">\n",
       "<title>140523649295864&#45;&gt;140523649298272*</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M192.4,-18.5C200.73,-18.5 208.79,-18.5 216.18,-18.5\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"216.05,-22 226.05,-18.5 216.05,-15 216.05,-22\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.graphs.Digraph at 0x7fce3632ecf8>"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a = Value(-2.0,label='a')\n",
    "b = Value(3.0,label='b')\n",
    "\n",
    "\n",
    "d = a * b   ; d.label ='d'\n",
    "e = a + b   ; e.label = 'e'\n",
    "f = d * e   ; f.label='f'\n",
    "\n",
    "f.backward()\n",
    "\n",
    "draw_dot(f)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b3dbce81",
   "metadata": {},
   "source": [
    "-------------\n",
    "<br><br>\n",
    "# 7. Micrograd: Adding More Mathematical Operations<a id=\"7\"></a>\n",
    "-----------------\n",
    "\n",
    "Now let’s further develop our `Value` class object by adding more functionality and operations. Operations that are added include `__rmul__`, `__pow__`, `__truediv__`, `__neg__`, `__sub__`, `exp` and other operators needed.<br><br>\n",
    "Also, we can handle constants like 1 in **a + 1** by adding the following line of code (which takes care of 1 by wrapping it as a `Value` object) to the `__add__` function:`other = other if isinstance(other, Value) else Value(other)`.<br><br>\n",
    "We can handle **1 + a** by calling the `__radd__` operator to avoid pythonic issues from default calling of `__add__`(`1.__add__(a)`) which doesn't know how to deal with `Value` objects that are user-defined. If python fails to call `__add__` for some reason and `__radd__` is available, it will swap the operands of `__add__` and call `__radd__`. Thus, **1 + a** will become **a + 1** and python will call this instead. The same logic applies for `__mul__` and `__rmul__`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "b5a25716",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Value(data=3.0, grad=0.0)"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a = Value(2.0)\n",
    "b = Value(4.0)\n",
    "a+1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "6b07e0f3",
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Value(data=2.0, grad=0.0)"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a*1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "5e65c660",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Value(data=4.0, grad=0.0)"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "2*a"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "e0794e75",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Value(data=7.38905609893065, grad=0.0)"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a.exp()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "e8bd78f3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Value(data=0.5, grad=0.0)"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a/b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "078adac8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Value(data=0.5, grad=0.0)"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a * 1/b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "f6d65ed2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Value(data=0.5, grad=0.0)"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a * (b**-1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "ba4fc9ee",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Value(data=-2.0, grad=0.0)"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a-b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "711f7bdd",
   "metadata": {},
   "outputs": [],
   "source": [
    "# inputs x1,x2\n",
    "x1 = Value(2.0,label='x1')\n",
    "x2 = Value(0.0,label='x2')\n",
    "# weights w1, w2\n",
    "w1 = Value(-3.0,label='w1')\n",
    "w2 = Value(1.0,label='w2')\n",
    "# bias of the neuron\n",
    "b = Value(6.8813735870195432,label='b')\n",
    "# x1*w1 + x2*w2 + b\n",
    "x1w1 = x1*w1; x1w1.label ='x1w1'\n",
    "x2w2 = x2*w2; x2w2.label ='x2w2'\n",
    "x1w1x2w2 = x1w1+x2w2; x1w1x2w2.label='x1w1 + x2w2'\n",
    "n = x1w1x2w2+b; n.label='n'\n",
    "o = n.tanh() \n",
    "o.label = 'o'\n",
    "o.backward()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "f60bf2fd",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 10.0.1 (20240210.2158)\n",
       " -->\n",
       "<!-- Pages: 1 -->\n",
       "<svg width=\"1546pt\" height=\"210pt\"\n",
       " viewBox=\"0.00 0.00 1546.25 210.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 206)\">\n",
       "<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-206 1542.25,-206 1542.25,4 -4,4\"/>\n",
       "<!-- 140523649534496 -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>140523649534496</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"1354.5,-54.5 1354.5,-90.5 1538.25,-90.5 1538.25,-54.5 1354.5,-54.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"1365.88\" y=\"-67.7\" font-family=\"Times,serif\" font-size=\"14.00\">o</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"1377.25,-55 1377.25,-90.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"1417.12\" y=\"-67.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 0.7071</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"1457,-55 1457,-90.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"1497.62\" y=\"-67.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 1.0000</text>\n",
       "</g>\n",
       "<!-- 140523649534496tanh -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>140523649534496tanh</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"1291.5\" cy=\"-72.5\" rx=\"27\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"1291.5\" y=\"-67.45\" font-family=\"Times,serif\" font-size=\"14.00\">tanh</text>\n",
       "</g>\n",
       "<!-- 140523649534496tanh&#45;&gt;140523649534496 -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>140523649534496tanh&#45;&gt;140523649534496</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M1318.78,-72.5C1325.92,-72.5 1334.11,-72.5 1342.82,-72.5\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1342.56,-76 1352.56,-72.5 1342.56,-69 1342.56,-76\"/>\n",
       "</g>\n",
       "<!-- 140523649533992 -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>140523649533992</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"1.5,-165.5 1.5,-201.5 196.5,-201.5 196.5,-165.5 1.5,-165.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"16.25\" y=\"-178.7\" font-family=\"Times,serif\" font-size=\"14.00\">x1</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"31,-166 31,-201.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"70.88\" y=\"-178.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 2.0000</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"110.75,-166 110.75,-201.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"153.62\" y=\"-178.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad &#45;1.5000</text>\n",
       "</g>\n",
       "<!-- 140523649534328* -->\n",
       "<g id=\"node12\" class=\"node\">\n",
       "<title>140523649534328*</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"261\" cy=\"-128.5\" rx=\"27\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"261\" y=\"-123.45\" font-family=\"Times,serif\" font-size=\"14.00\">*</text>\n",
       "</g>\n",
       "<!-- 140523649533992&#45;&gt;140523649534328* -->\n",
       "<g id=\"edge9\" class=\"edge\">\n",
       "<title>140523649533992&#45;&gt;140523649534328*</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M172.12,-165.01C180.9,-162.35 189.7,-159.5 198,-156.5 208.09,-152.86 218.82,-148.27 228.47,-143.88\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"229.86,-147.09 237.45,-139.69 226.9,-140.75 229.86,-147.09\"/>\n",
       "</g>\n",
       "<!-- 140523649534048 -->\n",
       "<g id=\"node4\" class=\"node\">\n",
       "<title>140523649534048</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"0,-110.5 0,-146.5 198,-146.5 198,-110.5 0,-110.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"16.25\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">w1</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"32.5,-111 32.5,-146.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"74.62\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">data &#45;3.0000</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"116.75,-111 116.75,-146.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"157.38\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 1.0000</text>\n",
       "</g>\n",
       "<!-- 140523649534048&#45;&gt;140523649534328* -->\n",
       "<g id=\"edge11\" class=\"edge\">\n",
       "<title>140523649534048&#45;&gt;140523649534328*</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M198.14,-128.5C206.61,-128.5 214.8,-128.5 222.29,-128.5\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"222.29,-132 232.29,-128.5 222.29,-125 222.29,-132\"/>\n",
       "</g>\n",
       "<!-- 140523649534104 -->\n",
       "<g id=\"node5\" class=\"node\">\n",
       "<title>140523649534104</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"661.5,-82.5 661.5,-118.5 918.75,-118.5 918.75,-82.5 661.5,-82.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"707.38\" y=\"-95.7\" font-family=\"Times,serif\" font-size=\"14.00\">x1w1 + x2w2</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"753.25,-83 753.25,-118.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"795.38\" y=\"-95.7\" font-family=\"Times,serif\" font-size=\"14.00\">data &#45;6.0000</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"837.5,-83 837.5,-118.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"878.12\" y=\"-95.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.5000</text>\n",
       "</g>\n",
       "<!-- 140523649534216+ -->\n",
       "<g id=\"node9\" class=\"node\">\n",
       "<title>140523649534216+</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"981.75\" cy=\"-72.5\" rx=\"27\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"981.75\" y=\"-67.45\" font-family=\"Times,serif\" font-size=\"14.00\">+</text>\n",
       "</g>\n",
       "<!-- 140523649534104&#45;&gt;140523649534216+ -->\n",
       "<g id=\"edge10\" class=\"edge\">\n",
       "<title>140523649534104&#45;&gt;140523649534216+</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M916.29,-82.02C926.15,-80.56 935.53,-79.18 943.92,-77.94\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"944.19,-81.44 953.58,-76.51 943.17,-74.51 944.19,-81.44\"/>\n",
       "</g>\n",
       "<!-- 140523649534104+ -->\n",
       "<g id=\"node6\" class=\"node\">\n",
       "<title>140523649534104+</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"598.5\" cy=\"-100.5\" rx=\"27\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"598.5\" y=\"-95.45\" font-family=\"Times,serif\" font-size=\"14.00\">+</text>\n",
       "</g>\n",
       "<!-- 140523649534104+&#45;&gt;140523649534104 -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>140523649534104+&#45;&gt;140523649534104</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M625.87,-100.5C632.88,-100.5 640.96,-100.5 649.7,-100.5\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"649.57,-104 659.57,-100.5 649.57,-97 649.57,-104\"/>\n",
       "</g>\n",
       "<!-- 140523649534160 -->\n",
       "<g id=\"node7\" class=\"node\">\n",
       "<title>140523649534160</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"2.25,-55.5 2.25,-91.5 195.75,-91.5 195.75,-55.5 2.25,-55.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"18.5\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">w2</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"34.75,-56 34.75,-91.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"74.62\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 1.0000</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"114.5,-56 114.5,-91.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"155.12\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
       "</g>\n",
       "<!-- 140523649534440* -->\n",
       "<g id=\"node14\" class=\"node\">\n",
       "<title>140523649534440*</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"261\" cy=\"-73.5\" rx=\"27\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"261\" y=\"-68.45\" font-family=\"Times,serif\" font-size=\"14.00\">*</text>\n",
       "</g>\n",
       "<!-- 140523649534160&#45;&gt;140523649534440* -->\n",
       "<g id=\"edge12\" class=\"edge\">\n",
       "<title>140523649534160&#45;&gt;140523649534440*</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M195.84,-73.5C205.14,-73.5 214.15,-73.5 222.32,-73.5\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"222.13,-77 232.13,-73.5 222.13,-70 222.13,-77\"/>\n",
       "</g>\n",
       "<!-- 140523649534216 -->\n",
       "<g id=\"node8\" class=\"node\">\n",
       "<title>140523649534216</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"1044.75,-54.5 1044.75,-90.5 1228.5,-90.5 1228.5,-54.5 1044.75,-54.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"1056.12\" y=\"-67.7\" font-family=\"Times,serif\" font-size=\"14.00\">n</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"1067.5,-55 1067.5,-90.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"1107.38\" y=\"-67.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 0.8814</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"1147.25,-55 1147.25,-90.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"1187.88\" y=\"-67.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.5000</text>\n",
       "</g>\n",
       "<!-- 140523649534216&#45;&gt;140523649534496tanh -->\n",
       "<g id=\"edge7\" class=\"edge\">\n",
       "<title>140523649534216&#45;&gt;140523649534496tanh</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M1228.76,-72.5C1237.14,-72.5 1245.27,-72.5 1252.73,-72.5\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1252.7,-76 1262.7,-72.5 1252.7,-69 1252.7,-76\"/>\n",
       "</g>\n",
       "<!-- 140523649534216+&#45;&gt;140523649534216 -->\n",
       "<g id=\"edge3\" class=\"edge\">\n",
       "<title>140523649534216+&#45;&gt;140523649534216</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M1009.03,-72.5C1016.17,-72.5 1024.36,-72.5 1033.07,-72.5\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1032.81,-76 1042.81,-72.5 1032.81,-69 1032.81,-76\"/>\n",
       "</g>\n",
       "<!-- 140523649534272 -->\n",
       "<g id=\"node10\" class=\"node\">\n",
       "<title>140523649534272</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"698.25,-27.5 698.25,-63.5 882,-63.5 882,-27.5 698.25,-27.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"709.62\" y=\"-40.7\" font-family=\"Times,serif\" font-size=\"14.00\">b</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"721,-28 721,-63.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"760.88\" y=\"-40.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 6.8814</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"800.75,-28 800.75,-63.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"841.38\" y=\"-40.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.5000</text>\n",
       "</g>\n",
       "<!-- 140523649534272&#45;&gt;140523649534216+ -->\n",
       "<g id=\"edge14\" class=\"edge\">\n",
       "<title>140523649534272&#45;&gt;140523649534216+</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M882.33,-58.49C904.17,-61.6 926.27,-64.74 943.98,-67.26\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"943.32,-70.71 953.71,-68.65 944.3,-63.78 943.32,-70.71\"/>\n",
       "</g>\n",
       "<!-- 140523649534328 -->\n",
       "<g id=\"node11\" class=\"node\">\n",
       "<title>140523649534328</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"324,-110.5 324,-146.5 535.5,-146.5 535.5,-110.5 324,-110.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"347\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">x1w1</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"370,-111 370,-146.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"412.12\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">data &#45;6.0000</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"454.25,-111 454.25,-146.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"494.88\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.5000</text>\n",
       "</g>\n",
       "<!-- 140523649534328&#45;&gt;140523649534104+ -->\n",
       "<g id=\"edge8\" class=\"edge\">\n",
       "<title>140523649534328&#45;&gt;140523649534104+</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M535.88,-110.85C544.59,-109.38 552.96,-107.98 560.56,-106.7\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"561.06,-110.17 570.34,-105.06 559.9,-103.26 561.06,-110.17\"/>\n",
       "</g>\n",
       "<!-- 140523649534328*&#45;&gt;140523649534328 -->\n",
       "<g id=\"edge4\" class=\"edge\">\n",
       "<title>140523649534328*&#45;&gt;140523649534328</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M288.42,-128.5C295.52,-128.5 303.68,-128.5 312.42,-128.5\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"312.24,-132 322.24,-128.5 312.24,-125 312.24,-132\"/>\n",
       "</g>\n",
       "<!-- 140523649534440 -->\n",
       "<g id=\"node13\" class=\"node\">\n",
       "<title>140523649534440</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"326.25,-55.5 326.25,-91.5 533.25,-91.5 533.25,-55.5 326.25,-55.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"349.25\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">x2w2</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"372.25,-56 372.25,-91.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"412.12\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 0.0000</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"452,-56 452,-91.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"492.62\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.5000</text>\n",
       "</g>\n",
       "<!-- 140523649534440&#45;&gt;140523649534104+ -->\n",
       "<g id=\"edge6\" class=\"edge\">\n",
       "<title>140523649534440&#45;&gt;140523649534104+</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M533.51,-90.14C543.16,-91.7 552.45,-93.21 560.82,-94.56\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"560.05,-97.98 570.48,-96.12 561.16,-91.07 560.05,-97.98\"/>\n",
       "</g>\n",
       "<!-- 140523649534440*&#45;&gt;140523649534440 -->\n",
       "<g id=\"edge5\" class=\"edge\">\n",
       "<title>140523649534440*&#45;&gt;140523649534440</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M288.42,-73.5C296.08,-73.5 304.96,-73.5 314.47,-73.5\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"314.37,-77 324.37,-73.5 314.37,-70 314.37,-77\"/>\n",
       "</g>\n",
       "<!-- 140523649533936 -->\n",
       "<g id=\"node15\" class=\"node\">\n",
       "<title>140523649533936</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"3.75,-0.5 3.75,-36.5 194.25,-36.5 194.25,-0.5 3.75,-0.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"18.5\" y=\"-13.7\" font-family=\"Times,serif\" font-size=\"14.00\">x2</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"33.25,-1 33.25,-36.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"73.12\" y=\"-13.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 0.0000</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"113,-1 113,-36.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"153.62\" y=\"-13.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.5000</text>\n",
       "</g>\n",
       "<!-- 140523649533936&#45;&gt;140523649534440* -->\n",
       "<g id=\"edge13\" class=\"edge\">\n",
       "<title>140523649533936&#45;&gt;140523649534440*</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M172.12,-36.99C180.9,-39.65 189.7,-42.5 198,-45.5 208.09,-49.14 218.82,-53.73 228.47,-58.12\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"226.9,-61.25 237.45,-62.31 229.86,-54.91 226.9,-61.25\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.graphs.Digraph at 0x7fce36352978>"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "draw_dot(o)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2d250996",
   "metadata": {},
   "source": [
    "-------------\n",
    "<br><br>\n",
    "## 7.1. Breaking up `tanh` Function into its Parts<a id=\"701\"></a>\n",
    "-----------------------------\n",
    "Currently, our `Value` class supports the `tanh` operation as a complete block. What if we wanted to break it into its constituting parts by explicitly including the tanh calculation in a computational graph?\n",
    "\n",
    "To achieve `tanh` implementation, we need the following operators: \n",
    "1. exponentiation (`exp`)\n",
    "2. power (`__pow__`)\n",
    "3. division (`__truediv__`, `__rdiv__`)\n",
    "4. multiplication (`__mul__`, `__rmul__`) \n",
    "5. subtraction (`__neg__`, `__sub__`)\n",
    "6. addition (`__add__`, `__radd__`)\n",
    "\n",
    "Most of the listed operators above have already been added to our `Value` object. The rest will be implemented in our `Value` object. Now, we can break up the `tanh` function into its parts, such that the complete calculation is visualized in the computational DAG as shown below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "58caa73f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# inputs x1,x2\n",
    "x1 = Value(2.0,label='x1')\n",
    "x2 = Value(0.0,label='x2')\n",
    "# weights w1, w2\n",
    "w1 = Value(-3.0,label='w1')\n",
    "w2 = Value(1.0,label='w2')\n",
    "# bias of the neuron\n",
    "b = Value(6.8813735870195432,label='b')\n",
    "# x1*w1 + x2*w2 + b\n",
    "x1w1 = x1*w1; x1w1.label ='x1w1'\n",
    "x2w2 = x2*w2; x2w2.label ='x2w2'\n",
    "x1w1x2w2 = x1w1+x2w2; x1w1x2w2.label='x1w1 + x2w2'\n",
    "n = x1w1x2w2+b; n.label='n'\n",
    "# -----------\n",
    "e = (2*n).exp()\n",
    "o = (e - 1)/(e + 1)\n",
    "# ------------\n",
    "o.label = 'o'\n",
    "o.backward()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "id": "0d028287",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 10.0.1 (20240210.2158)\n",
       " -->\n",
       "<!-- Pages: 1 -->\n",
       "<svg width=\"2909pt\" height=\"210pt\"\n",
       " viewBox=\"0.00 0.00 2909.00 210.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 206)\">\n",
       "<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-206 2905,-206 2905,4 -4,4\"/>\n",
       "<!-- 140523649975296 -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>140523649975296</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"1661.25,-136.5 1661.25,-172.5 1846.5,-172.5 1846.5,-136.5 1661.25,-136.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"1671.12\" y=\"-149.7\" font-family=\"Times,serif\" font-size=\"14.00\"> </text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"1681,-137 1681,-172.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"1723.12\" y=\"-149.7\" font-family=\"Times,serif\" font-size=\"14.00\">data &#45;1.0000</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"1765.25,-137 1765.25,-172.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"1805.88\" y=\"-149.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.1464</text>\n",
       "</g>\n",
       "<!-- 140523649975520+ -->\n",
       "<g id=\"node19\" class=\"node\">\n",
       "<title>140523649975520+</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"1909.5\" cy=\"-126.5\" rx=\"27\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"1909.5\" y=\"-121.45\" font-family=\"Times,serif\" font-size=\"14.00\">+</text>\n",
       "</g>\n",
       "<!-- 140523649975296&#45;&gt;140523649975520+ -->\n",
       "<g id=\"edge13\" class=\"edge\">\n",
       "<title>140523649975296&#45;&gt;140523649975520+</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M1846.9,-137.73C1855.64,-136.13 1864.11,-134.59 1871.83,-133.18\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1872.2,-136.67 1881.41,-131.44 1870.94,-129.79 1872.2,-136.67\"/>\n",
       "</g>\n",
       "<!-- 140523649975856 -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>140523649975856</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"1972.5,-53.5 1972.5,-89.5 2157.75,-89.5 2157.75,-53.5 1972.5,-53.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"1982.38\" y=\"-66.7\" font-family=\"Times,serif\" font-size=\"14.00\"> </text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"1992.25,-54 1992.25,-89.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"2032.12\" y=\"-66.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 6.8284</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"2072,-54 2072,-89.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"2114.88\" y=\"-66.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad &#45;0.1036</text>\n",
       "</g>\n",
       "<!-- 140523649975912**&#45;1 -->\n",
       "<g id=\"node8\" class=\"node\">\n",
       "<title>140523649975912**&#45;1</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"2284.12\" cy=\"-71.5\" rx=\"27.81\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"2284.12\" y=\"-66.45\" font-family=\"Times,serif\" font-size=\"14.00\">**&#45;1</text>\n",
       "</g>\n",
       "<!-- 140523649975856&#45;&gt;140523649975912**&#45;1 -->\n",
       "<g id=\"edge22\" class=\"edge\">\n",
       "<title>140523649975856&#45;&gt;140523649975912**&#45;1</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M2158.15,-71.5C2188.39,-71.5 2220.38,-71.5 2244.42,-71.5\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"2244.37,-75 2254.37,-71.5 2244.37,-68 2244.37,-75\"/>\n",
       "</g>\n",
       "<!-- 140523649975856+ -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>140523649975856+</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"1909.5\" cy=\"-71.5\" rx=\"27\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"1909.5\" y=\"-66.45\" font-family=\"Times,serif\" font-size=\"14.00\">+</text>\n",
       "</g>\n",
       "<!-- 140523649975856+&#45;&gt;140523649975856 -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>140523649975856+&#45;&gt;140523649975856</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M1936.91,-71.5C1943.95,-71.5 1952,-71.5 1960.55,-71.5\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1960.54,-75 1970.54,-71.5 1960.54,-68 1960.54,-75\"/>\n",
       "</g>\n",
       "<!-- 140523649975352 -->\n",
       "<g id=\"node4\" class=\"node\">\n",
       "<title>140523649975352</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"326.25,-110.5 326.25,-146.5 533.25,-146.5 533.25,-110.5 326.25,-110.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"349.25\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">x2w2</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"372.25,-111 372.25,-146.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"412.12\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 0.0000</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"452,-111 452,-146.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"492.62\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.5000</text>\n",
       "</g>\n",
       "<!-- 140523649975016+ -->\n",
       "<g id=\"node15\" class=\"node\">\n",
       "<title>140523649975016+</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"598.5\" cy=\"-100.5\" rx=\"27\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"598.5\" y=\"-95.45\" font-family=\"Times,serif\" font-size=\"14.00\">+</text>\n",
       "</g>\n",
       "<!-- 140523649975352&#45;&gt;140523649975016+ -->\n",
       "<g id=\"edge28\" class=\"edge\">\n",
       "<title>140523649975352&#45;&gt;140523649975016+</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M533.51,-111.25C543.16,-109.62 552.45,-108.06 560.82,-106.66\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"561.2,-110.15 570.48,-105.04 560.04,-103.24 561.2,-110.15\"/>\n",
       "</g>\n",
       "<!-- 140523649975352* -->\n",
       "<g id=\"node5\" class=\"node\">\n",
       "<title>140523649975352*</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"261\" cy=\"-128.5\" rx=\"27\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"261\" y=\"-123.45\" font-family=\"Times,serif\" font-size=\"14.00\">*</text>\n",
       "</g>\n",
       "<!-- 140523649975352*&#45;&gt;140523649975352 -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>140523649975352*&#45;&gt;140523649975352</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M288.42,-128.5C296.08,-128.5 304.96,-128.5 314.47,-128.5\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"314.37,-132 324.37,-128.5 314.37,-125 314.37,-132\"/>\n",
       "</g>\n",
       "<!-- 140523649974848 -->\n",
       "<g id=\"node6\" class=\"node\">\n",
       "<title>140523649974848</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"3.75,-165.5 3.75,-201.5 194.25,-201.5 194.25,-165.5 3.75,-165.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"18.5\" y=\"-178.7\" font-family=\"Times,serif\" font-size=\"14.00\">x2</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"33.25,-166 33.25,-201.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"73.12\" y=\"-178.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 0.0000</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"113,-166 113,-201.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"153.62\" y=\"-178.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.5000</text>\n",
       "</g>\n",
       "<!-- 140523649974848&#45;&gt;140523649975352* -->\n",
       "<g id=\"edge27\" class=\"edge\">\n",
       "<title>140523649974848&#45;&gt;140523649975352*</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M172.12,-165.01C180.9,-162.35 189.7,-159.5 198,-156.5 208.09,-152.86 218.82,-148.27 228.47,-143.88\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"229.86,-147.09 237.45,-139.69 226.9,-140.75 229.86,-147.09\"/>\n",
       "</g>\n",
       "<!-- 140523649975912 -->\n",
       "<g id=\"node7\" class=\"node\">\n",
       "<title>140523649975912</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"2410.5,-57.5 2410.5,-93.5 2591.25,-93.5 2591.25,-57.5 2410.5,-57.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"2420.38\" y=\"-70.7\" font-family=\"Times,serif\" font-size=\"14.00\"> </text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"2430.25,-58 2430.25,-93.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"2470.12\" y=\"-70.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 0.1464</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"2510,-58 2510,-93.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"2550.62\" y=\"-70.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 4.8284</text>\n",
       "</g>\n",
       "<!-- 140523649976024* -->\n",
       "<g id=\"node13\" class=\"node\">\n",
       "<title>140523649976024*</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"2654.25\" cy=\"-98.5\" rx=\"27\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"2654.25\" y=\"-93.45\" font-family=\"Times,serif\" font-size=\"14.00\">*</text>\n",
       "</g>\n",
       "<!-- 140523649975912&#45;&gt;140523649976024* -->\n",
       "<g id=\"edge23\" class=\"edge\">\n",
       "<title>140523649975912&#45;&gt;140523649976024*</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M2591.68,-89.15C2600.29,-90.45 2608.64,-91.72 2616.28,-92.88\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"2615.71,-96.34 2626.12,-94.38 2616.76,-89.41 2615.71,-96.34\"/>\n",
       "</g>\n",
       "<!-- 140523649975912**&#45;1&#45;&gt;140523649975912 -->\n",
       "<g id=\"edge3\" class=\"edge\">\n",
       "<title>140523649975912**&#45;1&#45;&gt;140523649975912</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M2312.32,-72.01C2334.43,-72.42 2367.08,-73.03 2398.97,-73.62\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"2398.62,-77.11 2408.68,-73.8 2398.75,-70.12 2398.62,-77.11\"/>\n",
       "</g>\n",
       "<!-- 140523649974904 -->\n",
       "<g id=\"node9\" class=\"node\">\n",
       "<title>140523649974904</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"1.5,-55.5 1.5,-91.5 196.5,-91.5 196.5,-55.5 1.5,-55.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"16.25\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">x1</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"31,-56 31,-91.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"70.88\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 2.0000</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"110.75,-56 110.75,-91.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"153.62\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad &#45;1.5000</text>\n",
       "</g>\n",
       "<!-- 140523649975240* -->\n",
       "<g id=\"node17\" class=\"node\">\n",
       "<title>140523649975240*</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"261\" cy=\"-73.5\" rx=\"27\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"261\" y=\"-68.45\" font-family=\"Times,serif\" font-size=\"14.00\">*</text>\n",
       "</g>\n",
       "<!-- 140523649974904&#45;&gt;140523649975240* -->\n",
       "<g id=\"edge17\" class=\"edge\">\n",
       "<title>140523649974904&#45;&gt;140523649975240*</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M196.76,-73.5C205.77,-73.5 214.47,-73.5 222.4,-73.5\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"222.25,-77 232.25,-73.5 222.25,-70 222.25,-77\"/>\n",
       "</g>\n",
       "<!-- 140523649975408 -->\n",
       "<g id=\"node10\" class=\"node\">\n",
       "<title>140523649975408</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"1046.25,-109.5 1046.25,-145.5 1227,-145.5 1227,-109.5 1046.25,-109.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"1056.12\" y=\"-122.7\" font-family=\"Times,serif\" font-size=\"14.00\"> </text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"1066,-110 1066,-145.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"1105.88\" y=\"-122.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 2.0000</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"1145.75,-110 1145.75,-145.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"1186.38\" y=\"-122.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.2203</text>\n",
       "</g>\n",
       "<!-- 140523649975576* -->\n",
       "<g id=\"node21\" class=\"node\">\n",
       "<title>140523649975576*</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"1291.5\" cy=\"-99.5\" rx=\"27\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"1291.5\" y=\"-94.45\" font-family=\"Times,serif\" font-size=\"14.00\">*</text>\n",
       "</g>\n",
       "<!-- 140523649975408&#45;&gt;140523649975576* -->\n",
       "<g id=\"edge20\" class=\"edge\">\n",
       "<title>140523649975408&#45;&gt;140523649975576*</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M1227.44,-111.05C1236.69,-109.35 1245.68,-107.71 1253.83,-106.22\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1254.42,-109.67 1263.62,-104.42 1253.16,-102.78 1254.42,-109.67\"/>\n",
       "</g>\n",
       "<!-- 140523649974960 -->\n",
       "<g id=\"node11\" class=\"node\">\n",
       "<title>140523649974960</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"0,-0.5 0,-36.5 198,-36.5 198,-0.5 0,-0.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"16.25\" y=\"-13.7\" font-family=\"Times,serif\" font-size=\"14.00\">w1</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"32.5,-1 32.5,-36.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"74.62\" y=\"-13.7\" font-family=\"Times,serif\" font-size=\"14.00\">data &#45;3.0000</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"116.75,-1 116.75,-36.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"157.38\" y=\"-13.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 1.0000</text>\n",
       "</g>\n",
       "<!-- 140523649974960&#45;&gt;140523649975240* -->\n",
       "<g id=\"edge14\" class=\"edge\">\n",
       "<title>140523649974960&#45;&gt;140523649975240*</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M172.12,-36.99C180.9,-39.65 189.7,-42.5 198,-45.5 208.09,-49.14 218.82,-53.73 228.47,-58.12\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"226.9,-61.25 237.45,-62.31 229.86,-54.91 226.9,-61.25\"/>\n",
       "</g>\n",
       "<!-- 140523649976024 -->\n",
       "<g id=\"node12\" class=\"node\">\n",
       "<title>140523649976024</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"2717.25,-80.5 2717.25,-116.5 2901,-116.5 2901,-80.5 2717.25,-80.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"2728.62\" y=\"-93.7\" font-family=\"Times,serif\" font-size=\"14.00\">o</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"2740,-81 2740,-116.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"2779.88\" y=\"-93.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 0.7071</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"2819.75,-81 2819.75,-116.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"2860.38\" y=\"-93.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 1.0000</text>\n",
       "</g>\n",
       "<!-- 140523649976024*&#45;&gt;140523649976024 -->\n",
       "<g id=\"edge4\" class=\"edge\">\n",
       "<title>140523649976024*&#45;&gt;140523649976024</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M2681.53,-98.5C2688.67,-98.5 2696.86,-98.5 2705.57,-98.5\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"2705.31,-102 2715.31,-98.5 2705.31,-95 2705.31,-102\"/>\n",
       "</g>\n",
       "<!-- 140523649975016 -->\n",
       "<g id=\"node14\" class=\"node\">\n",
       "<title>140523649975016</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"661.5,-82.5 661.5,-118.5 918.75,-118.5 918.75,-82.5 661.5,-82.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"707.38\" y=\"-95.7\" font-family=\"Times,serif\" font-size=\"14.00\">x1w1 + x2w2</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"753.25,-83 753.25,-118.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"795.38\" y=\"-95.7\" font-family=\"Times,serif\" font-size=\"14.00\">data &#45;6.0000</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"837.5,-83 837.5,-118.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"878.12\" y=\"-95.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.5000</text>\n",
       "</g>\n",
       "<!-- 140523649975128+ -->\n",
       "<g id=\"node26\" class=\"node\">\n",
       "<title>140523649975128+</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"981.75\" cy=\"-72.5\" rx=\"27\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"981.75\" y=\"-67.45\" font-family=\"Times,serif\" font-size=\"14.00\">+</text>\n",
       "</g>\n",
       "<!-- 140523649975016&#45;&gt;140523649975128+ -->\n",
       "<g id=\"edge26\" class=\"edge\">\n",
       "<title>140523649975016&#45;&gt;140523649975128+</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M916.29,-82.02C926.15,-80.56 935.53,-79.18 943.92,-77.94\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"944.19,-81.44 953.58,-76.51 943.17,-74.51 944.19,-81.44\"/>\n",
       "</g>\n",
       "<!-- 140523649975016+&#45;&gt;140523649975016 -->\n",
       "<g id=\"edge5\" class=\"edge\">\n",
       "<title>140523649975016+&#45;&gt;140523649975016</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M625.87,-100.5C632.88,-100.5 640.96,-100.5 649.7,-100.5\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"649.57,-104 659.57,-100.5 649.57,-97 649.57,-104\"/>\n",
       "</g>\n",
       "<!-- 140523649975240 -->\n",
       "<g id=\"node16\" class=\"node\">\n",
       "<title>140523649975240</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"324,-55.5 324,-91.5 535.5,-91.5 535.5,-55.5 324,-55.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"347\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">x1w1</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"370,-56 370,-91.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"412.12\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">data &#45;6.0000</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"454.25,-56 454.25,-91.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"494.88\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.5000</text>\n",
       "</g>\n",
       "<!-- 140523649975240&#45;&gt;140523649975016+ -->\n",
       "<g id=\"edge21\" class=\"edge\">\n",
       "<title>140523649975240&#45;&gt;140523649975016+</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M535.88,-90.52C544.59,-91.93 552.96,-93.29 560.56,-94.52\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"559.91,-97.96 570.34,-96.1 561.03,-91.05 559.91,-97.96\"/>\n",
       "</g>\n",
       "<!-- 140523649975240*&#45;&gt;140523649975240 -->\n",
       "<g id=\"edge6\" class=\"edge\">\n",
       "<title>140523649975240*&#45;&gt;140523649975240</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M288.42,-73.5C295.52,-73.5 303.68,-73.5 312.42,-73.5\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"312.24,-77 322.24,-73.5 312.24,-70 312.24,-77\"/>\n",
       "</g>\n",
       "<!-- 140523649975520 -->\n",
       "<g id=\"node18\" class=\"node\">\n",
       "<title>140523649975520</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"2193.75,-108.5 2193.75,-144.5 2374.5,-144.5 2374.5,-108.5 2193.75,-108.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"2203.62\" y=\"-121.7\" font-family=\"Times,serif\" font-size=\"14.00\"> </text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"2213.5,-109 2213.5,-144.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"2253.38\" y=\"-121.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 4.8284</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"2293.25,-109 2293.25,-144.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"2333.88\" y=\"-121.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.1464</text>\n",
       "</g>\n",
       "<!-- 140523649975520&#45;&gt;140523649976024* -->\n",
       "<g id=\"edge19\" class=\"edge\">\n",
       "<title>140523649975520&#45;&gt;140523649976024*</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M2374.88,-119.89C2435.9,-115.36 2518.45,-109.17 2591.25,-103.5 2599.13,-102.89 2607.58,-102.21 2615.62,-101.57\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"2615.81,-105.06 2625.5,-100.77 2615.25,-98.09 2615.81,-105.06\"/>\n",
       "</g>\n",
       "<!-- 140523649975520+&#45;&gt;140523649975520 -->\n",
       "<g id=\"edge7\" class=\"edge\">\n",
       "<title>140523649975520+&#45;&gt;140523649975520</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M1937,-126.5C1987.26,-126.5 2098.82,-126.5 2181.78,-126.5\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"2181.77,-130 2191.77,-126.5 2181.77,-123 2181.77,-130\"/>\n",
       "</g>\n",
       "<!-- 140523649975576 -->\n",
       "<g id=\"node20\" class=\"node\">\n",
       "<title>140523649975576</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"1354.5,-81.5 1354.5,-117.5 1535.25,-117.5 1535.25,-81.5 1354.5,-81.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"1364.38\" y=\"-94.7\" font-family=\"Times,serif\" font-size=\"14.00\"> </text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"1374.25,-82 1374.25,-117.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"1414.12\" y=\"-94.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 1.7627</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"1454,-82 1454,-117.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"1494.62\" y=\"-94.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.2500</text>\n",
       "</g>\n",
       "<!-- 140523649975632exp -->\n",
       "<g id=\"node24\" class=\"node\">\n",
       "<title>140523649975632exp</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"1598.25\" cy=\"-99.5\" rx=\"27\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"1598.25\" y=\"-94.45\" font-family=\"Times,serif\" font-size=\"14.00\">exp</text>\n",
       "</g>\n",
       "<!-- 140523649975576&#45;&gt;140523649975632exp -->\n",
       "<g id=\"edge18\" class=\"edge\">\n",
       "<title>140523649975576&#45;&gt;140523649975632exp</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M1535.68,-99.5C1544.02,-99.5 1552.12,-99.5 1559.56,-99.5\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1559.5,-103 1569.5,-99.5 1559.5,-96 1559.5,-103\"/>\n",
       "</g>\n",
       "<!-- 140523649975576*&#45;&gt;140523649975576 -->\n",
       "<g id=\"edge8\" class=\"edge\">\n",
       "<title>140523649975576*&#45;&gt;140523649975576</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M1318.87,-99.5C1326.01,-99.5 1334.2,-99.5 1342.89,-99.5\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1342.61,-103 1352.61,-99.5 1342.61,-96 1342.61,-103\"/>\n",
       "</g>\n",
       "<!-- 140523649975072 -->\n",
       "<g id=\"node22\" class=\"node\">\n",
       "<title>140523649975072</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"2.25,-110.5 2.25,-146.5 195.75,-146.5 195.75,-110.5 2.25,-110.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"18.5\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">w2</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"34.75,-111 34.75,-146.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"74.62\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 1.0000</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"114.5,-111 114.5,-146.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"155.12\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
       "</g>\n",
       "<!-- 140523649975072&#45;&gt;140523649975352* -->\n",
       "<g id=\"edge12\" class=\"edge\">\n",
       "<title>140523649975072&#45;&gt;140523649975352*</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M195.84,-128.5C205.14,-128.5 214.15,-128.5 222.32,-128.5\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"222.13,-132 232.13,-128.5 222.13,-125 222.13,-132\"/>\n",
       "</g>\n",
       "<!-- 140523649975632 -->\n",
       "<g id=\"node23\" class=\"node\">\n",
       "<title>140523649975632</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"1663.5,-81.5 1663.5,-117.5 1844.25,-117.5 1844.25,-81.5 1663.5,-81.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"1673.38\" y=\"-94.7\" font-family=\"Times,serif\" font-size=\"14.00\"> </text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"1683.25,-82 1683.25,-117.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"1723.12\" y=\"-94.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 5.8284</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"1763,-82 1763,-117.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"1803.62\" y=\"-94.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0429</text>\n",
       "</g>\n",
       "<!-- 140523649975632&#45;&gt;140523649975856+ -->\n",
       "<g id=\"edge16\" class=\"edge\">\n",
       "<title>140523649975632&#45;&gt;140523649975856+</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M1844.68,-83.13C1854.23,-81.39 1863.51,-79.7 1871.89,-78.17\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1872.39,-81.64 1881.6,-76.4 1871.14,-74.75 1872.39,-81.64\"/>\n",
       "</g>\n",
       "<!-- 140523649975632&#45;&gt;140523649975520+ -->\n",
       "<g id=\"edge11\" class=\"edge\">\n",
       "<title>140523649975632&#45;&gt;140523649975520+</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M1844.68,-115.28C1854.23,-116.96 1863.51,-118.59 1871.89,-120.07\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1871.14,-123.49 1881.6,-121.77 1872.36,-116.59 1871.14,-123.49\"/>\n",
       "</g>\n",
       "<!-- 140523649975632exp&#45;&gt;140523649975632 -->\n",
       "<g id=\"edge9\" class=\"edge\">\n",
       "<title>140523649975632exp&#45;&gt;140523649975632</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M1625.66,-99.5C1633.37,-99.5 1642.29,-99.5 1651.76,-99.5\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1651.57,-103 1661.57,-99.5 1651.57,-96 1651.57,-103\"/>\n",
       "</g>\n",
       "<!-- 140523649975128 -->\n",
       "<g id=\"node25\" class=\"node\">\n",
       "<title>140523649975128</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"1044.75,-54.5 1044.75,-90.5 1228.5,-90.5 1228.5,-54.5 1044.75,-54.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"1056.12\" y=\"-67.7\" font-family=\"Times,serif\" font-size=\"14.00\">n</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"1067.5,-55 1067.5,-90.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"1107.38\" y=\"-67.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 0.8814</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"1147.25,-55 1147.25,-90.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"1187.88\" y=\"-67.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.5000</text>\n",
       "</g>\n",
       "<!-- 140523649975128&#45;&gt;140523649975576* -->\n",
       "<g id=\"edge25\" class=\"edge\">\n",
       "<title>140523649975128&#45;&gt;140523649975576*</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M1228.76,-88.6C1237.59,-90.16 1246.15,-91.67 1253.94,-93.04\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1253.17,-96.46 1263.63,-94.75 1254.39,-89.57 1253.17,-96.46\"/>\n",
       "</g>\n",
       "<!-- 140523649975128+&#45;&gt;140523649975128 -->\n",
       "<g id=\"edge10\" class=\"edge\">\n",
       "<title>140523649975128+&#45;&gt;140523649975128</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M1009.03,-72.5C1016.17,-72.5 1024.36,-72.5 1033.07,-72.5\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1032.81,-76 1042.81,-72.5 1032.81,-69 1032.81,-76\"/>\n",
       "</g>\n",
       "<!-- 140523649975184 -->\n",
       "<g id=\"node27\" class=\"node\">\n",
       "<title>140523649975184</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"698.25,-27.5 698.25,-63.5 882,-63.5 882,-27.5 698.25,-27.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"709.62\" y=\"-40.7\" font-family=\"Times,serif\" font-size=\"14.00\">b</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"721,-28 721,-63.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"760.88\" y=\"-40.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 6.8814</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"800.75,-28 800.75,-63.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"841.38\" y=\"-40.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.5000</text>\n",
       "</g>\n",
       "<!-- 140523649975184&#45;&gt;140523649975128+ -->\n",
       "<g id=\"edge24\" class=\"edge\">\n",
       "<title>140523649975184&#45;&gt;140523649975128+</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M882.33,-58.49C904.17,-61.6 926.27,-64.74 943.98,-67.26\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"943.32,-70.71 953.71,-68.65 944.3,-63.78 943.32,-70.71\"/>\n",
       "</g>\n",
       "<!-- 140523649975744 -->\n",
       "<g id=\"node28\" class=\"node\">\n",
       "<title>140523649975744</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"1661.25,-26.5 1661.25,-62.5 1846.5,-62.5 1846.5,-26.5 1661.25,-26.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"1671.12\" y=\"-39.7\" font-family=\"Times,serif\" font-size=\"14.00\"> </text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"1681,-27 1681,-62.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"1720.88\" y=\"-39.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 1.0000</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"1760.75,-27 1760.75,-62.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"1803.62\" y=\"-39.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad &#45;0.1036</text>\n",
       "</g>\n",
       "<!-- 140523649975744&#45;&gt;140523649975856+ -->\n",
       "<g id=\"edge15\" class=\"edge\">\n",
       "<title>140523649975744&#45;&gt;140523649975856+</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M1846.9,-60.67C1855.64,-62.21 1864.11,-63.7 1871.83,-65.05\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1870.95,-68.45 1881.41,-66.74 1872.16,-61.56 1870.95,-68.45\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.graphs.Digraph at 0x7fce36368438>"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "draw_dot(o)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8414c185",
   "metadata": {},
   "source": [
    "<br><br>\n",
    "# 8. Micrograd: A [`PyTorch`](https://pytorch.org/tutorials/) Comparison<a id=\"8\"></a>\n",
    "-----------------\n",
    "\n",
    "We can use [**PyTorch**](https://pytorch.org/docs/stable/index.html) to run the same architecture we implemented. We developed our own <u>autograd computational graph object</u> (`Value`) and validated its functionality as a data structure representation and a backpropagation supportive engine for arbitrarily complex DAGs. \n",
    "- PyTorch uses <u>n-dimensional tensors</u> as their basic data structure. Thus, to define a scalar, we use one-dimensional tensors.\n",
    "\n",
    "- **PyTorch** uses <u>single precision values</u> (`torch.float32`). However, **Python** uses <u>double-precision values</u> (`torch.float64`). Thus, for consistency, we use the `.double()` function to typecast PyTorch values.\n",
    "\n",
    "- PyTorch by default does not calculate the gradients. Thus, we must explicitly state that the variables require gradients and we must calculate them. <br>\n",
    "    1. `var.requires_grad=True`: This indicates that the gradients with respect to **var** will be computed during backpropagation.\n",
    "    2. `var.grad_item()`: Returns the value of the `var.grad` tensor as a standard Python number. This only works for tensors with one element.<br>\n",
    "\n",
    "To validate our implementation of Micrograd, we will implement the exact same computational graph from before in Pytorch and confirm that the results are identical. Run the following Python code cells below to see PyTorch in action. <br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "id": "c50dfdfc",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/Users/CEO/anaconda3/envs/py36/bin/python\n"
     ]
    }
   ],
   "source": [
    "import sys\n",
    "print(sys.executable)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "id": "38b08bd5",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "id": "f4f001c4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor([[1., 2., 3.],\n",
       "         [4., 5., 6.]]),\n",
       " torch.Size([2, 3]))"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "torch.Tensor([[1, 2, 3], [4, 5, 6]]),\\\n",
    "torch.Tensor([[1, 2, 3], [4, 5, 6]]).shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "id": "982bd723",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(torch.float32, torch.float64)"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "torch.tensor([2.0]).dtype,\\\n",
    "torch.tensor([2.0]).double().dtype "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "id": "a967adaa",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.7071066904050358\n",
      "-----\n",
      "x2 0.5000001283844369\n",
      "w2 0.0\n",
      "x1 -1.5000003851533106\n",
      "w1 1.0000002567688737\n"
     ]
    }
   ],
   "source": [
    "x1 = torch.tensor([2.0]).double()     ;x1.requires_grad=True\n",
    "w1 = torch.tensor([-3.0]).double()    ;w1.requires_grad=True\n",
    "x2 = torch.tensor([0.0]).double()     ;x2.requires_grad=True\n",
    "w2 = torch.tensor([1.0]).double()     ;w2.requires_grad=True\n",
    "b  = torch.tensor([6.8813735870195432]).double()     ;b.requires_grad=True\n",
    "n = w1*x1 + w2*x2 + b\n",
    "o = torch.tanh(n)\n",
    "\n",
    "print(o.data.item())\n",
    "o.backward()\n",
    "\n",
    "print('-----')\n",
    "print('x2', x2.grad.item())\n",
    "print('w2', w2.grad.item())\n",
    "print('x1', x1.grad.item())\n",
    "print('w1', w1.grad.item())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "id": "b5fe20f8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor([0.7071], dtype=torch.float64, grad_fn=<TanhBackward0>),\n",
       " 0.7071066904050358,\n",
       " 0.7071066904050358)"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "o, o.item(), o.data.item()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1e95b0d4",
   "metadata": {},
   "source": [
    "-------------\n",
    "<br><br>\n",
    "# 9. Micrograd: Building a NN Library (Multi-Layer Perceptron, MLP)<a id=\"9\"></a>\n",
    "-----------------"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "939861b7",
   "metadata": {},
   "source": [
    "## 9.1. [Neural Network architectures (cs231n)](https://cs231n.github.io/neural-networks-1/)<a id=\"901\"></a>\n",
    "-------\n",
    "### Layer-wise organization\n",
    "-------\n",
    "**Neural Networks as neurons in graphs**. Neural Networks are modeled as collections of neurons that are connected in an acyclic graph. In other words, the outputs of some neurons can become inputs to other neurons. Cycles are not allowed since that would imply an infinite loop in the forward pass of a network. Instead of an amorphous blobs of connected neurons, Neural Network models are often organized into distinct layers of neurons. For regular neural networks, the most common layer type is the **fully-connected layer** in which neurons between two adjacent layers are fully pairwise connected, but neurons within a single layer share no connections. Below are two example Neural Network topologies that use a stack of fully-connected layers:\n",
    "\n",
    "<a id=\"a4\"></a>\n",
    "<p align=\"center\">\n",
    "    <img src=\"_imgs/MLP_2layer.jpeg\" width=340 height=240 align=\"left\"> \n",
    "    <img src=\"_imgs/MLP_3layer.jpeg\" width=480 height=240 align=\"center\"> \n",
    "</p>\n",
    "\n",
    "\n",
    "**Fig 4. L-R: A Diagram of Two Multi-Layer Perceptrons (MLP): a 2-layer NN, a 3-layer NN.** ([Source](https://cs231n.github.io/neural-networks-1/))<br>\n",
    "\n",
    "Left: A **2-layer NN** (one hidden layer of 4 neurons (or units) and one output layer with 2 neurons), and three inputs. <br>Right: A **3-layer NN** with three inputs, two hidden layers of 4 neurons each and one output layer. <br>Notice that in both cases there are connections (synapses) between neurons across layers, but not within a layer. <br><br>\n",
    "\n",
    "**Naming conventions.** Notice that when we say N-layer neural network, we do not count the input layer. Therefore, a single-layer neural network describes a network with no hidden layers (input directly mapped to output). In that sense, you can sometimes hear people say that logistic regression or SVMs are simply a special case of single-layer Neural Networks. You may also hear these networks interchangeably referred to as _“Artificial Neural Networks”_ (ANN) or _“Multi-Layer Perceptrons”_ (MLP). Many people do not like the analogies between Neural Networks and real brains and prefer to refer to neurons as _units_.\n",
    "\n",
    "**Output layer.** Unlike all layers in a Neural Network, the output layer neurons most commonly do not have an activation function (or you can think of them as having a linear identity activation function). This is because the last output layer is usually taken to represent the class scores (e.g. in classification), which are arbitrary real-valued numbers, or some kind of real-valued target (e.g. in regression).\n",
    "\n",
    "**Sizing neural networks**. The two metrics that people commonly use to measure the size of neural networks are the number of neurons, or more commonly the number of parameters. Working with the two example networks in the above picture:\n",
    "\n",
    "*   The first network (left) has 4 + 2 = 6 neurons (not counting the inputs), \\[3 x 4\\] + \\[4 x 2\\] = 20 weights and 4 + 2 = 6 biases, for a total of 26 learnable parameters.\n",
    "*   The second network (right) has 4 + 4 + 1 = 9 neurons, \\[3 x 4\\] + \\[4 x 4\\] + \\[4 x 1\\] = 12 + 16 + 4 = 32 weights and 4 + 4 + 1 = 9 biases, for a total of 41 learnable parameters.\n",
    "\n",
    "To give you some context, modern Convolutional Networks contain on orders of 100 million parameters and are usually made up of approximately 10-20 layers (hence _deep learning_). However, as we will see the number of _effective_ connections is significantly greater due to parameter sharing. More on this in the Convolutional Neural Networks module.<br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5760a260",
   "metadata": {},
   "source": [
    "Now let’s develop the machinery to create a small neural network.\n",
    "First, we define a class to implement a single neuron. Then we build the layer. And finally, we create an MLP. The breakdown of each class object is stated below:\n",
    "\n",
    "- Class `Neuron` contains code for a computational unit that receives `nin` inputs and outputs the result of a **tanh** operation on a weighted sum of the inputs plus a bias term.\n",
    "\n",
    "\n",
    "- Class `Layer` aggregates several neurons such that it takes certain data as input (of size `nin`) and passes it through `nout` neurons. We do a small check in the end where if the output contains only one neuron we return it as-is and not in a list.\n",
    "\n",
    "\n",
    "- Class `MLP` is the final abstraction in which we aggregate `nout` instances of `Layer` by passing the input data `x` through each of them in succession.\n",
    "\n",
    "For each class, we added a `parameters` function that collects all trainable parameters. This will come in handy later when we train the network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "id": "54b0fd9d",
   "metadata": {},
   "outputs": [],
   "source": [
    "import random\n",
    "\n",
    "class Neuron:\n",
    "    \"\"\" Defines a single neuron\"\"\"\n",
    "\n",
    "    def __init__(self, nin):\n",
    "        \"\"\" Assign random weights and biases to the inputs \n",
    "        Parameters\n",
    "        ----------\n",
    "        nin: int\n",
    "            inputs to the neuron\n",
    "        \"\"\"\n",
    "        self.w = [Value(random.uniform(-1,1)) for _ in range(nin)]\n",
    "        self.b = Value(random.uniform(-1,1))\n",
    "\n",
    "    def __call__(self, x): # w * x + b\n",
    "        \"\"\" Calculate the output of the neuron\n",
    "        Parameters\n",
    "        ----------\n",
    "        x: list\n",
    "            input values to the neuron\n",
    "\n",
    "        Returns\n",
    "        ----------\n",
    "        integer\n",
    "            output of the neuron\n",
    "        \"\"\"\n",
    "        act = sum((wi*xi for wi, xi in zip(self.w, x)),  self.b)\n",
    "        out = act.tanh()\n",
    "        return out\n",
    "    \n",
    "    def parameters(self):\n",
    "        return self.w + [self.b]\n",
    "    \n",
    "    \n",
    "class Layer:\n",
    "    \"\"\" Defines a layer of neurons \"\"\"\n",
    "\n",
    "    def __init__(self, nin, nout):\n",
    "        \"\"\" Initialize the neurons of the layer\n",
    "        Parameters\n",
    "        ----------\n",
    "        nin: int\n",
    "            number of inputs to the layer\n",
    "        nout: int\n",
    "            number of neurons in the layer\n",
    "        \"\"\"\n",
    "        self.neurons = [Neuron(nin) for _ in range(nout)]\n",
    "  \n",
    "    def __call__(self, x):\n",
    "        \"\"\" Calculate the outputs of a layer\n",
    "        Parameters\n",
    "        ----------\n",
    "        x: list\n",
    "            input values to the neuron\n",
    "\n",
    "        Returns\n",
    "        ----------\n",
    "        int\n",
    "            outputs of the neuron\n",
    "        \"\"\"\n",
    "        outs = [n(x) for n in self.neurons]\n",
    "        return outs[0] if len(outs) == 1 else outs\n",
    "    \n",
    "    def parameters(self):\n",
    "        return [param for neuron in self.neurons for param in neuron.parameters()]\n",
    "    \n",
    "    \n",
    "class MLP:\n",
    "    \"\"\" Defines a multilayer perceptron\"\"\"\n",
    "    \n",
    "    def __init__(self, nin, nouts): # nout define the sizes of all layers\n",
    "        \"\"\" Initialize the layers of MLP\n",
    "        Parameters\n",
    "        ----------\n",
    "        nin: int\n",
    "            number of inputs to the MLP\n",
    "        nouts: list\n",
    "            number of neurons in each layer\n",
    "        \"\"\"\n",
    "        sz = [nin] + nouts\n",
    "        self.layers= [Layer(sz[i], sz[i+1]) for i in range(len(nouts))]\n",
    "\n",
    "    def __call__(self, x):\n",
    "        \"\"\" Calculate the outputs of a layer sequentially\n",
    "        Parameters\n",
    "        ----------\n",
    "        x: list\n",
    "            input values to the MLP\n",
    "\n",
    "        Returns\n",
    "        ----------\n",
    "        list\n",
    "            outputs of the MLP \n",
    "        \"\"\"\n",
    "        for layer in self.layers:\n",
    "            x = layer(x)\n",
    "        return x\n",
    "    \n",
    "    def parameters(self):\n",
    "        return [param for layer in self.layers for param in layer.parameters()]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "id": "a59013c4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Value(data=0.9105848567129079, grad=0.0)"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = [2.0, 3.0]\n",
    "a = Neuron(2)\n",
    "a(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "id": "f85277e9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Value(data=0.9835475868478987, grad=0.0),\n",
       " Value(data=-0.18080027900666776, grad=0.0),\n",
       " Value(data=-0.36304820207442917, grad=0.0)]"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = [2.0, 3.0]\n",
    "b =  Layer(2, 3)\n",
    "b(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "id": "89cf08e7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Value(data=-0.03132176728538012, grad=0.0)"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# inputs to the nn\n",
    "x = [2.0, 3.0, -1.0]\n",
    "\n",
    "# initialize the MLP\n",
    "n = MLP(3, [4, 4, 1])\n",
    "\n",
    "# run the NN\n",
    "n(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "id": "630f8167",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "41"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(n.parameters())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "89f66451",
   "metadata": {},
   "source": [
    "-----------\n",
    "<br><br>\n",
    "# 10. Micrograd: Creating a Tiny Dataset & Writing the Loss Function<a id=\"10\"></a>\n",
    "-----------------\n",
    "Now let’s create a tiny data set. We have **inputs** and **desired targets**. We will run the neural network in the forward direction. This is a **small, simple binary classification problem**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "id": "1235b14d",
   "metadata": {},
   "outputs": [],
   "source": [
    "xs =[\n",
    "    [2.0, 3.0, -1.0],\n",
    "    [3.0, -1.0, 0.5],\n",
    "    [0.5, 1.0, 1.0],\n",
    "    [1.0, 1.0, -1.0]\n",
    "]\n",
    "\n",
    "#target values (simple binary classification)\n",
    "ys =[1.0, -1.0, -1.0, 1.0]\n",
    "\n",
    "#run the neural network on all the inputs\n",
    "# ypred = [n(x) for x in xs]\n",
    "# ypred"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "71116067",
   "metadata": {},
   "source": [
    "<a id=\"c4\"></a>\n",
    "We want the neural network to generate predicted values that are as close to the target values as possible. We should tune the weights. But how???\n",
    "\n",
    "- **Backpropagation !!!**\n",
    "\n",
    "Before that, we must calculate a single number (loss) to measure the network's performance. There are various loss functions (mean squared error, binary cross-entropy loss, etc.) that can be used. We use the mean squared error (MSE) function as the loss function. Our goal is to minimize the loss function so as to improve model performance.\n",
    "\n",
    "![MSE](_imgs/MSE.png)\n",
    "**Eq. 4: Mean Squared Error (MSE)** ([Source](https://en.wikipedia.org/wiki/Mean_squared_error))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "id": "b7d4630f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Value(data=5.625114897319604, grad=0.0)\n"
     ]
    }
   ],
   "source": [
    "ypred = [n(x) for x in xs]\n",
    "loss = 0\n",
    "for ygt, yout in zip(ys, ypred):\n",
    "    loss += (yout - ygt)**2 \n",
    "print(loss)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "id": "bd1363e8",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'loss = sum((yout - ygt)**2 for ygt, yout in zip(ys, ypred)) # ygt = y_ground_truth, yout = y_output\\nloss'"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "'''loss = sum((yout - ygt)**2 for ygt, yout in zip(ys, ypred)) # ygt = y_ground_truth, yout = y_output\n",
    "loss''' #it affects operations due to Value object representation of parameters (error: int + Value)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "21ce8317",
   "metadata": {},
   "source": [
    "-----------\n",
    "<br><br>\n",
    "# 11. Micrograd: Neural Net Parameters<a id=\"11\"></a>\n",
    "-----------------\n",
    "Now, we've calculated the loss function, and all the gradients of the loss function. However, the gradients WRT inputs are only useful if the inputs are assumed to be fixed. We need to collect all the parameters we can change. We can add a `parameter` method to the `neuron`, `layer`, and `MLP` class modules above which collects all trainable parameters. This will come in handy later when we train the network.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "id": "7e72a2d4",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.0"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "n.layers[0].neurons[0].w[0].grad"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "id": "aea022a1",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-0.3188397214186136"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "n.layers[0].neurons[0].w[0].data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d733f29a",
   "metadata": {},
   "source": [
    "We need to update parameters according to their gradient information. **The gradient** is a vector that points towards the increasing direction of loss. Thus, we must move in the opposite direction of the gradient.\n",
    "\n",
    "\n",
    "Finally! We are going to train our neural network to fit the dummy data points we showed before. To train our net we need to repeat the **forward pass -> loss calculation -> backward pass** procedure. We do it manually in the cells below. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "id": "c8429c1d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Value(data=5.625114897319604, grad=0.0)\n"
     ]
    }
   ],
   "source": [
    "ypred = [n(x) for x in xs]\n",
    "loss = 0\n",
    "for ygt, yout in zip(ys, ypred):\n",
    "    loss += (yout - ygt)**2 \n",
    "print(loss)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "id": "d2eef746",
   "metadata": {},
   "outputs": [],
   "source": [
    "loss.backward()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "id": "27237345",
   "metadata": {},
   "outputs": [],
   "source": [
    "for p in n.parameters():\n",
    "    p.data += -0.05 * p.grad"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "id": "30b2d531",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Value(data=0.5438762632690177, grad=0.0),\n",
       " Value(data=0.5956186061958746, grad=0.0),\n",
       " Value(data=-0.4498238385421293, grad=0.0),\n",
       " Value(data=0.5037095702051143, grad=0.0)]"
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ypred = [n(x) for x in xs]\n",
    "ypred"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "898349ef",
   "metadata": {},
   "source": [
    "-----------\n",
    "<br><br><a id=\"12\"></a>\n",
    "# 12. Micrograd: Manual Gradient Descent Optimization & Training the MLP\n",
    "-----------------\n",
    "\n",
    "<a id=\"d3\"></a><u>Gradient Descent:</u><br>\n",
    "Now, we encounter gradient descent (G.D.) for optimization. [Gradient descent](https://en.wikipedia.org/wiki/Gradient_descent) is a method for unconstrained mathematical optimization. It is a first-order iterative algorithm for finding a local minimum of a differentiable multivariate function.\n",
    "\n",
    "The idea is to take repeated steps in the opposite direction of the gradient (or approximate gradient) of the function at the current point, because this is the direction of steepest descent.It is particularly useful in machine learning for minimizing the cost or loss function. To reach the lowest point on the surface we start taking steps along the direction of the steepest downward slope. This is what the gradient descent algorithm achieves during each training epoch or iteration.<br><br>\n",
    "<u>Steps for implementing gradient descent:</u>\n",
    "\n",
    "Iterate a certain number of times (**n_iter**): \n",
    "1. <u>Forward Pass</u>: We compute the predictions from the inputs using `MLP`. Then we compute the loss from the target and predicted values using Mean-Squared Error (MSE) loss function.\n",
    "2. <u>Backward Pass</u>: We reset the gradients of the NN to zero and perform backpropagation on our NN MLP model.\n",
    "3. <u>Parameter Update</u>: We update the parameters (weights and biases) of our `MLP` with the gradient information.\n",
    "\n",
    " \n",
    "Neural networks can be trained by:\n",
    "1. Constructing an instance with the desired architecture\n",
    "2. Calling on inputs to make predictions\n",
    "3. Calculating loss between predictions and targets\n",
    "4. Calling a method to calculate gradients\n",
    "5. Calling a method to update parameters\n",
    "6. Repeating steps 2-5 over many batches of training data (**Gradient Descent Optimization**)\n",
    "\n",
    "Instead of manually training the MLP like we did in the previous section, we can write a simple `train_MLP` function that implements the training loop for us."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "id": "b73fadcc",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Value(data=-0.6041503111538816, grad=0.0)"
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = [2.0, 3.0, -1.0]\n",
    "n = MLP(3, [4, 4, 1])\n",
    "n(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "id": "e1dd7c41",
   "metadata": {},
   "outputs": [],
   "source": [
    "xs =[\n",
    "    [2.0, 3.0, -1.0],\n",
    "    [3.0, -1.0, 0.5],\n",
    "    [0.5, 1.0, 1.0],\n",
    "    [1.0, 1.0, -1.0]\n",
    "]\n",
    "\n",
    "#target values (simple binary classification)\n",
    "ys =[1.0, -1.0, -1.0, 1.0] "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "id": "cb9642d4",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "def train_MLP(n_iter=20, learning_rate=0.01):\n",
    "    losses, steps = [], []\n",
    "    step = 0\n",
    "    for k in range(n_iter):\n",
    "        step += 1\n",
    "        steps.append(step)\n",
    "        # forward pass\n",
    "        ypred = [n(x) for x in xs]\n",
    "        loss = 0\n",
    "        for ygt, yout in zip(ys, ypred):\n",
    "            loss += (yout - ygt)**2 \n",
    "        total_loss = loss\n",
    "        losses.append(total_loss.data)\n",
    "\n",
    "        # backward pass -> reset gradients to zero so as to\n",
    "        # ensure the actual backward pass accumulates the loss derivatives into the gradients\n",
    "        for p in n.parameters():\n",
    "            p.grad = 0.0\n",
    "        total_loss.backward()\n",
    "\n",
    "        # update parameters\n",
    "        for p in n.parameters():\n",
    "            p.data += -1 * learning_rate * p.grad\n",
    "\n",
    "        print(k, total_loss.data)\n",
    "    print(ypred)\n",
    "    return [steps, losses]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "id": "ea5c2c9c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 5.485189247043687\n",
      "1 3.3421234365691754\n",
      "2 2.584049020397665\n",
      "3 1.9610887856041734\n",
      "4 1.4208451979993688\n",
      "5 0.9652687705981344\n",
      "6 0.6646346234265398\n",
      "7 0.47428422790045377\n",
      "8 0.3615123589435135\n",
      "9 0.2870300719366895\n",
      "10 0.23489049651030347\n",
      "11 0.19691204612563087\n",
      "12 0.16831557271479816\n",
      "13 0.14617933935615032\n",
      "14 0.12864178050918224\n",
      "15 0.1144722088955907\n",
      "16 0.10283027925617594\n",
      "17 0.09312577909436154\n",
      "18 0.08493384323191937\n",
      "19 0.07794195214308396\n",
      "20 0.071915798959111\n",
      "21 0.066676715938482\n",
      "22 0.06208638656923887\n",
      "23 0.05803626867652136\n",
      "24 0.0544401349246291\n",
      "25 0.051228719740150565\n",
      "26 0.048345816790658386\n",
      "27 0.04574539276626418\n",
      "28 0.043389424538740215\n",
      "29 0.041246258705737185\n",
      "30 0.03928935342896755\n",
      "31 0.037496303502279935\n",
      "32 0.03584807765559297\n",
      "33 0.03432841658324537\n",
      "34 0.032923353888887105\n",
      "35 0.03162083189798568\n",
      "36 0.03041039131962961\n",
      "37 0.029282918859381307\n",
      "38 0.028230440651596557\n",
      "39 0.027245952177094315\n",
      "[Value(data=0.9311099191634069, grad=-0.13778016167318619), Value(data=-0.9800438376407024, grad=0.03991232471859529), Value(data=-0.8834365990471585, grad=0.2331268019056829), Value(data=0.90772414139322, grad=-0.18455171721356)]\n"
     ]
    }
   ],
   "source": [
    "_train_loss = train_MLP(40, 0.05)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "id": "158e5b82",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Value(data=0.9322752068220904, grad=0.0),\n",
       " Value(data=-0.9804343270784147, grad=0.0),\n",
       " Value(data=-0.885414663220842, grad=0.0),\n",
       " Value(data=0.9093136457027569, grad=0.0)]"
      ]
     },
     "execution_count": 77,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ypred = [n(x) for x in xs]\n",
    "ypred"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "id": "b04eb474",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXgAAAEWCAYAAABsY4yMAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nO3de5xU9X3/8ddnZmdnYC9cd7ktCKiAioBmxaiJom2sES+pxlRLUk3aqmkaG5NUk/TXX83vYR7VpDWpj9Ym5mZajTGpMfUWY4woRqIIKgQUvMASUWEXUHaXhb3MfH5/zJl1wQV3lz17Zs+8nw/nMWfOnMtnj8t7zn7ne77H3B0REYmfRNQFiIhIOBTwIiIxpYAXEYkpBbyISEwp4EVEYkoBLyISUwp4iSUz+6WZXTrYy4oMJ6Z+8FIszKy1x8uRQDuQDV5f4e53DH1VA2dmi4Db3b0u6lqkNJVFXYBIgbtXFqbNrAH4K3d/ZP/lzKzM3buGsjaR4UhNNFL0zGyRmW0xs2vNbCvwQzMbY2b3m1mTmb0VTNf1WOcxM/urYPoyM/utmf1LsOwmM/vwAJedYWbLzKzFzB4xs/8ws9sH8DMdFez3bTNbZ2bn9XjvbDN7IdjH62b2xWD++ODnfNvMdprZE2amf8NyQPrlkOFiIjAWOAy4nPzv7g+D19OAPcC/H2T9E4ENwHjg68D3zcwGsOyPgRXAOOA64BP9/UHMLAXcBzwM1AKfBe4ws9nBIt8n3yRVBcwFHg3mfwHYAtQAE4CvAGpjlQNSwMtwkQP+yd3b3X2Pu+9w97vdvc3dW4CvAacdZP3N7v5dd88CPwImkQ/JPi9rZtOAE4D/6+4d7v5b4N4B/CzvByqBG4LtPArcD1wSvN8JHG1m1e7+lrs/22P+JOAwd+909ydcX6LJQSjgZbhocve9hRdmNtLMvmNmm82sGVgGjDaz5AHW31qYcPe2YLKyn8tOBnb2mAfwWj9/DoLtvObuuR7zNgNTgukLgbOBzWb2uJmdFMz/BvAK8LCZbTSzLw1g31JCFPAyXOx/pvoFYDZwortXA6cG8w/U7DIY3gTGmtnIHvOmDmA7bwBT92s/nwa8DuDuz7j7+eSbb34B/DSY3+LuX3D3mcB5wOfN7I8GsH8pEQp4Ga6qyLe7v21mY4F/CnuH7r4ZWAlcZ2blwZn1ue+1npllej7It+G3AdeYWSroTnku8JNgu0vMbJS7dwLN5JunMLNzzOyI4PuAXeS7kOZ63akICngZvr4FjAC2A08BDw3RfpcAJwE7gOuBu8j31z+QKeQ/iHo+ppIP9A+Tr/8W4C/cfX2wzieAhqDp6cpgnwBHAo8ArcDvgFvcfemg/WQSO7rQSeQQmNldwHp3D/0vCJH+0hm8SD+Y2QlmdriZJczsLOB88u3kIkVHV7KK9M9E4Ofk+8FvAT7t7s9FW5JI79REIyISU2qiERGJqaJqohk/frxPnz496jJERIaNVatWbXf3mt7eK6qAnz59OitXroy6DBGRYcPMNh/oPTXRiIjElAJeRCSmFPAiIjFVVG3wIlJcOjs72bJlC3v37n3vhSVUmUyGuro6UqlUn9dRwIvIAW3ZsoWqqiqmT5/Oge+PImFzd3bs2MGWLVuYMWNGn9dTE42IHNDevXsZN26cwj1iZsa4ceP6/ZeUAl5EDkrhXhwG8v9h2Ad8Luf8+6Mv8/hLTVGXIiJSVIZ9wCcSxneWbWTp+saoSxGRQbZjxw4WLFjAggULmDhxIlOmTOl+3dHRcdB1V65cyVVXXfWe+zj55JMHpdbHHnuMc845Z1C2NVhi8SXrhOoM25r1Lb9I3IwbN47nn38egOuuu47Kykq++MUvdr/f1dVFWVnvMVZfX099ff177mP58uWDU2wRGvZn8AATqtMKeJEScdlll3HllVdy4okncs0117BixQpOOukkjjvuOE4++WQ2bNgA7HtGfd111/GpT32KRYsWMXPmTG6++ebu7VVWVnYvv2jRIj760Y8yZ84clixZQmG03QcffJA5c+bwvve9j6uuuqpfZ+p33nknxx57LHPnzuXaa68FIJvNctlllzF37lyOPfZYvvnNbwJw8803c/TRRzNv3jwuvvjiQz5WsTiDr63KsGLTzqjLEIm1r963jhfeaB7UbR49uZp/OveYfq+3ZcsWli9fTjKZpLm5mSeeeIKysjIeeeQRvvKVr3D33Xe/a53169ezdOlSWlpamD17Np/+9Kff1af8ueeeY926dUyePJlTTjmFJ598kvr6eq644gqWLVvGjBkzuOSSS/pc5xtvvMG1117LqlWrGDNmDGeeeSa/+MUvmDp1Kq+//jpr164F4O233wbghhtuYNOmTaTT6e55hyIWZ/C11WmaWtrR2PYipeGiiy4imUwCsGvXLi666CLmzp3L1Vdfzbp163pdZ/HixaTTacaPH09tbS3btm171zILFy6krq6ORCLBggULaGhoYP369cycObO7/3l/Av6ZZ55h0aJF1NTUUFZWxpIlS1i2bBkzZ85k48aNfPazn+Whhx6iuroagHnz5rFkyRJuv/32AzY99UcszuAnVGXoyOZ4u62TMRXlUZcjEksDOdMOS0VFRff0P/7jP3L66adzzz330NDQwKJFi3pdJ51Od08nk0m6uroGtMxgGDNmDKtXr+ZXv/oV3/72t/npT3/KD37wAx544AGWLVvGfffdx9e+9jV+//vfH1LQx+IMfkJ1BoBtLWqHFyk1u3btYsqUKQDcdtttg7792bNns3HjRhoaGgC46667+rzuwoULefzxx9m+fTvZbJY777yT0047je3bt5PL5bjwwgu5/vrrefbZZ8nlcrz22mucfvrp3HjjjezatYvW1tZDqj0WZ/C11flP3W3N7cyZGHExIjKkrrnmGi699FKuv/56Fi9ePOjbHzFiBLfccgtnnXUWFRUVnHDCCQdc9je/+Q11dXXdr3/2s59xww03cPrpp+PuLF68mPPPP5/Vq1fzyU9+klwuB8A///M/k81m+fjHP86uXbtwd6666ipGjx59SLUX1T1Z6+vrfSA3/PjDjjZO/cZSvvHReVxUPzWEykRK04svvshRRx0VdRmRa21tpbKyEnfnM5/5DEceeSRXX331kNfR2/8PM1vl7r32B41FE03hDL6xpT3iSkQkjr773e+yYMECjjnmGHbt2sUVV1wRdUl9EosmmkwqyagRKfWFF5FQXH311ZGcsR+qWJzBA9RWpWls1hm8yGArpmbcUjaQ/w+xCfgJ1Rn1ohEZZJlMhh07dijkI1YYDz6TyfRrvVg00UC+Hf7pjbujLkMkVurq6tiyZQtNTRqtNWqFOzr1R2wCfkJ1hsaWvbi7xq8WGSSpVKpfdxCS4hJqwJtZA9ACZIGuA3XlGQy1VWk6s85bbZ2M1dWsIiJDcgZ/urtvD3sn3VezNu9VwIuIEKsvWQtXs+qLVhERCD/gHXjYzFaZ2eW9LWBml5vZSjNbeShf5NRW5c/g1VVSRCQv7ID/gLsfD3wY+IyZnbr/Au5+q7vXu3t9TU3NgHdUU1W4mlVn8CIiEHLAu/vrwXMjcA+wMKx9ZVJJRo9MsU1n8CIiQIgBb2YVZlZVmAbOBNaGtT/IjwuvNngRkbwwe9FMAO4J+qSXAT9294dC3B+11Wm2acAxEREgxIB3943A/LC235vaqgyvNobeI1NEZFiITTdJyHeVbGxpJ5fTuBkiIjEL+AxdOWdnW0fUpYiIRC5WAV9b6CqpnjQiIjELeN18W0SkW6wCvjBcQaO6SoqIxCvgC1ez6mInEZGYBXy6LMmYkSkNVyAiQswCHoJb9+kMXkQkfgFfW51RG7yICDEM+AlVaZ3Bi4gQw4CvrU7T1KqrWUVEYhfwE6ozZHPOjt26mlVESlvsAr5wZycNGywipS52Ad99sZO6SopIiYtdwBeGK9B4NCJS6mIX8DWVuppVRARiGPDlZQnGVZRrwDERKXmxC3jQxU4iIhDXgK/K39lJRKSUxTLgJ1Sn1U1SREpeTAM+Q1NLO1ldzSoiJSyWAV9bnSHnsGO3mmlEpHTFM+B1b1YRkXgG/IRqDVcgIhLTgNfFTiIisQz48ZVpzDQejYiUtlgGfCoZXM2qM3gRKWGhB7yZJc3sOTO7P+x99VRbpatZRaS0DcUZ/N8BLw7BfvYxoTqt8WhEpKSFGvBmVgcsBr4X5n56kz+DVxONiJSusM/gvwVcA+QOtICZXW5mK81sZVNT06DteEJ1mu2t7XRlD7hrEZFYCy3gzewcoNHdVx1sOXe/1d3r3b2+pqZm0Pb/ztWsujeriJSmMM/gTwHOM7MG4CfAGWZ2e4j724cudhKRUhdawLv7l929zt2nAxcDj7r7x8Pa3/40XIGIlLpY9oOHHmfw6kkjIiWqbCh24u6PAY8Nxb4KxleWY6bhCkSkdMX2DL4smWB8ZVoXO4lIyYptwINu3ScipS3WAT+hOqNeNCJSsmIe8Gm1wYtIyYp1wNdWZdixW1ezikhpinfAV6dxh+2tuppVREpPrAN+QpWuZhWR0hXvgNdwBSJSwmIe8MFwBeoqKSIlKNYBP64yTcLQxU4iUpJiHfDJhDG+Ul0lRaQ0xTrgIbjYSQOOiUgJKoGAT2vIYBEpSbEP+JqqDI06gxeREhT7gM/fm7WDTl3NKiIlpgQCPt8XvkldJUWkxMQ+4Ltv3aeAF5ESE/uAL5zBb921J+JKRESGVuwD/vCaSsrLEqzY9FbUpYiIDKnYB/yI8iQnzhjLYy81Rl2KiMiQin3AAyyaXcvGpt28trMt6lJERIZMiQR8DQCPvdQUcSUiIkOnJAJ+5vgKpo4dweMb1EwjIqWjJALezFg0q5blr+6gvSsbdTkiIkOiJAIe4LRZNbR1ZFnZoN40IlIaSibgTz5iHOXJBI+pmUZESkTJBPzI8jIWzhjLYxv0RauIlIbQAt7MMma2wsxWm9k6M/tqWPvqq0Wza3i5sZXX39ZVrSISf2GewbcDZ7j7fGABcJaZvT/E/b2n7u6SaqYRkRIQWsB7XmvwMhU8PKz99cXhNZVMGT2Cx9VMIyIlINQ2eDNLmtnzQCPwa3d/updlLjezlWa2sqkp3OA1M06bXcOTr2yno0vjw4tIvPUp4M2swswSwfQsMzvPzFLvtZ67Z919AVAHLDSzub0sc6u717t7fU1NTX/r77dFs2rY3ZFl5eadoe9LRCRKfT2DXwZkzGwK8DDwCeC2vu7E3d8GlgJn9bfAwXbyEeNJJU3NNCISe30NeHP3NuAC4BZ3vwg45qArmNWY2ehgegTwIWD9oRQ7GCrTZZwwfSyPa1waEYm5Pge8mZ0ELAEeCOYl32OdScBSM1sDPEO+Df7+gZU5uE6bVcP6rS28qZuAiEiM9TXgPwd8GbjH3deZ2UzyTS4H5O5r3P04d5/n7nPd/f8darGDZdHsWgA104hIrPUp4N39cXc/z91vDL5s3e7uV4VcW2hmTahk0qiMrmoVkVjray+aH5tZtZlVAGuBF8zs78MtLTxmxqKgu2RnVt0lRSSe+tpEc7S7NwMfAX4JzCDfk2bYOm1WLS3tXTy7WaNLikg89TXgU0G/948A97p7JxFflXqoTjliHGUJ012eRCS2+hrw3wEagApgmZkdBjSHVdRQqMqkeN9hY9QOLyKx1dcvWW929ynufnYwxsxm4PSQawvdotm1vPhmM9ua90ZdiojIoOvrl6yjzOymwpgxZvav5M/mh7XC6JK66ElE4qivTTQ/AFqAjwWPZuCHYRU1VOZMrGJCdVr94UUklsr6uNzh7n5hj9dfDUaJHNbMjNNm1fDQ2q10ZXOUJUvmBlciUgL6mmh7zOwDhRdmdgoQi+v8z5hTS/PeLp7epNElRSRe+noGfyXwX2Y2Knj9FnBpOCUNrUWza6koT3Lf6jc45YjxUZcjIjJo+tqLZnVw6715wDx3Pw44I9TKhkgmleTMYybyy7VbdRMQEYmVfjU6u3tzcEUrwOdDqCcS586fxK49nTzxsr5sFZH4OJRvFW3QqojYB46oYfTIFPetfiPqUkREBs2hBPywHqqgp/KyBB+eO5GHX9jGno5s1OWIiAyKgwa8mbWYWXMvjxZg8hDVOCTOnT+Zto4sj65vjLoUEZFBcdCAd/cqd6/u5VHl7n3tgTMsnDhjHLVVae5d/XrUpYiIDApd2RNIJozF8yaxdEMTzXs7oy5HROSQKeB7OG/+ZDq6cjy8blvUpYiIHDIFfA8Lpo5m6tgR3KveNCISAwr4HsyMc+dN5slXtrOjtT3qckREDokCfj/nzp9MNuc8uHZr1KWIiBwSBfx+5kys4sjaSl30JCLDngJ+P2bGufMn80zDTt7cFYsBM0WkRCnge3Hu/Mm4wwNr3oy6FBGRAVPA92LG+AqOnTJKvWlEZFhTwB/AefMns2bLLjZt3x11KSIiAxJawJvZVDNbamYvmNk6M/u7sPYVhsXzJgFwv87iRWSYCvMMvgv4grsfDbwf+IyZHR3i/gbV5NEjWDh9LPeufgP32AycKSIlJLSAd/c33f3ZYLoFeBGYEtb+wnDu/Em83NjKhm0tUZciItJvQ9IGb2bTgeOAp3t573IzW2lmK5uaiuuOSmcfO4lkwrj3eTXTiMjwE3rAm1klcDfwuR63++vm7re6e72719fU1IRdTr+Mq0xzyhHjuXf1G+RyaqYRkeEl1IA3sxT5cL/D3X8e5r7CcsFxU9jy1h5+t3FH1KWIiPRLmL1oDPg+8KK73xTWfsJ21tyJjBqR4scr/hB1KSIi/RLmGfwpwCeAM8zs+eBxdoj7C0UmleTC4+t4eN1WtmuESREZRsLsRfNbdzd3n+fuC4LHg2HtL0yXLJxKZ9a5e9WWqEsREekzXcnaB0dOqOKE6WO4c8Uf1CdeRIYNBXwf/fmJ02jY0aYvW0Vk2FDA99GH505i1IgUd654LepSRET6RAHfR5lUkguOn8Kv1m7V7fxEZFhQwPfDJQun0ZHNcfez+rJVRIqfAr4fZk2oov6wMdy54jV92SoiRU8B30+XLJzGpu27eWrjzqhLERE5KAV8Py2eN4nqTBl36spWESlyCvh+yn/ZWsdDa7eyc3dH1OWIiByQAn4ALl44lY5sjp/ry1YRKWIK+AGYM7Ga46eN5se6slVEipgCfoAuWTiNjU27WbFJX7aKSHFSwA/QOfMmU5Up0zDCIlK0FPADNKI8yQXHTeGXv9/KW/qyVUSKkAL+EFxyoq5sFZHipYA/BHMmVnPctNH891ObyeqerSJSZBTwh+jyD85k8442Hvz9m1GXIiKyDwX8IfqTYyYys6aCWx57VV0mRaSoKOAPUSJhXHna4bz4ZjOPvdQUdTkiIt0U8IPgIwumMGlUhv9c+mrUpYiIdFPAD4LysgR//cGZrGjYyTMNuvBJRIqDAn6QXLxwKmNGprhl6StRlyIiAijgB83I8jI+ecoMlm5o4oU3mqMuR0REAT+YLj1pOhXlSf7zcbXFi0j0FPCDaNTIFB9//2E8sOYNGrbvjrocESlxCvhB9pcfmEFZMsF3lm2MuhQRKXEK+EFWW53ho++r4+5VW9jWvDfqckSkhCngQ3DFqTPpyuX4/m83RV2KiJSw0ALezH5gZo1mtjasfRSrw8ZVcM68ydzx1GZ2tXVGXY6IlKgwz+BvA84KcftF7dOLDmd3R5Yf/a4h6lJEpESFFvDuvgwo2cs6j5pUzRlzavnhk5to6+iKuhwRKUGRt8Gb2eVmttLMVjY1xWuwrr9ZdDhvtXXyX7/bHHUpIlKCIg94d7/V3evdvb6mpibqcgZV/fSx/NGcWm769Uts2NoSdTkiUmIiD/i4u+HCeVRnyrjqzufY25mNuhwRKSEK+JDVVKX5xkXz2bCthRt+uT7qckSkhITZTfJO4HfAbDPbYmZ/Gda+it3ps2v55CnTuW15A4+u3xZ1OSJSIsLsRXOJu09y95S717n798Pa13Bw7VlzmDOxir//2RoaW3SFq4iET000QySTSnLzJcfR2t7F3/9sDbmc7t8qIuFSwA+hWROq+D+Lj+Lxl5r44fKGqMsRkZhTwA+xj7//MP74qFpu/OV63RhEREKlgB9iZsaNF85j1MgUV/3kOfZ0qOukiIRDAR+BcZVpbvrYfF5pbOVrD74QdTkiElMK+Ih88Mga/vqDM7j9qT9wx9MaykBEBl9Z1AWUsi/+yWxe2tbKP9yzlm279nL1h2ZhZlGXJSIxoTP4CKXLknzv0no+Vl/HzY++whd/tobObC7qskQkJnQGH7FUMsGNF85j8ugRfOuRl2ls2cstS46nKpOKujQRGeZ0Bl8EzIzP/fEsvn7hPJa/uoM/+85Tup+riBwyBXwR+dgJU/nepfU07NjNBbcs55VGDTEsIgOngC8yp8+u5a7LT6K9K8cFtyxnxaaSvSmWiBwiBXwROrZuFPf8zcmMr0qz5HtP8dX71rG9tT3qskRkmFHAF6mpY0dy95Un86fHTeFHyxs49etL+deHN7BrT2fUpYnIMGHuxTOqYX19va9cuTLqMorOq02t3PTrl3hgzZuMGpHiytMO57KTpzOiPBl1aSISMTNb5e71vb6ngB8+1r6+i5t+/RKPrm+kpirNZ884gotPmEZ5mf4QEylVCviYWdmwk6//agMrNu1kYnWG8xZM5tx5k5k7pVpXwoqUGAV8DLk7T7y8nR8tb2DZy010Zp0Z4ys4d94kzp0/mSMnVEVdoogMAQV8zL3d1sFDa7dy35o3+N2rO8g5zJlYxbnzJ/Mnx0zk8JoKndmLxJQCvoQ0tuzlwTVvct+aN1m1+S0AxleWs3DGWBZOH8vCGeOYPbGKZEKBLxIHCvgSteWtNn778nZWbNrJ05t28vrbewCoypRxwvSxLJwxlnl1o5gzsZqxFeURVysiA6GAFyAf+M807GTFpvzj1abd3e/VVqWZPbGKOROrmD2xmjkTqziitpJMSl0xRYrZwQJeo0mWkLoxI6kbM5I/Pa4OgO2t7bzwRjMbtrawfmsLG7Y181+/20x7V37I4oTB5NEjmDZ2JNPGjmRq8Fx4jB6ZUtu+SBFTwJew8ZVpTp1Vw6mzarrndWVzNOxoY8PWFjZsa+EPO3bzh51tPPJi47uGS6hMlzGhOs2E6gwTqzPUVmeYUJ3unq6tSjOuspyR5fo1E4mC/uXJPsqSCY6oreSI2koWM2mf99o6unht5x7+sLON13a28dpbbWxr3su25nZWNOyksbmdjl5uWJJJJRg7spyxleWMrUgzrqKcsRXljBmZYtSIFNXBY1SPR3UmpQu4RA6RAl76bGR5GbMnVjF7Yu997N2dt9o62da8l63Ne2lqbmdnWwc7d3ewo7WDnbvb2bm7g41Nrezc3UFbR/ag+8ukElSmU1Smk1RmyqhMl1GZTlGVKaMinaQiXcbIVH56RHmSivKyfZ5HpJLdz5lUgkwqSbosoWYlKRkKeBk0ZsbY4Oz8qEnV77l8e1eW5j1d7NrTya49nTQHz4VHa3sXLXu7aG3vonVvJ7vbs7z+9h5a2ztp3dvF7o4sHV39u8WhGUHg58O+8JwuS5Dunk6STiVIJxOUlwWPntM9XycTpJIJUt3zLP86mSCVzE+XJfLzyxL55VIJoyyZoCxplCWC+UnTB48MulAD3szOAv4NSALfc/cbwtyfDC/psiQ1VUlqqtID3kZXNkdbZ5Y9HVl2t3fR1pGlrSPL7o4u2juz7OnMsrczx56OwnT+saczS3tnjvauHO1dWdq7cuztzNLa3sWO1g7au7J0ZHN0dOWX6QgeXbnwep0lE0YyYaQKz8kEyUT+QyCZNFKJRPcyZUkjmUiQNCgL5ufnGUmz7uX2eQTzE8E2Ez2Wy09D0vLvF54L83suaxa8b7xrmcJyCct/4Bfe6562nu+9s37C3lnfCtPd2wToZZmEYbyzjPVc18AwLNhGYTmz/Ie89dieBevGUWgBb2ZJ4D+ADwFbgGfM7F53fyGsfUrpKUsmqE4mqB6ie9hmc05nNh/6ndng0eV0ZLN0dOXf6yjMzzpdPaY7szm6sk5HNkdXNv9h0ZUrLOP5befyy2RzTlcul3/Oevey2Vx+2Vz363eWa+/Kds/rfvh+r3NOzt9ZNxcsk8vRvWypygf/vqFvwfyeHxIUlkvs+8EB9u5t8M6Hh9m75werMb4izU+vPGnQf6Ywz+AXAq+4+0YAM/sJcD6ggJdhK38Wm4z19QHdob9f8Pec7073B4Y7PZZ3ck5+Oli/e9qD94Jl3Avbo3td7143/37PbbmD8842C8s6PdaHd21jn9cE2+meX9jWO+s6hXXpnr//toF9awqWp3v779RWWCb4r8d672y/OhNOFIcZ8FOA13q83gKcuP9CZnY5cDnAtGnTQixHRPoikTASxLPJotRE3g/N3W9193p3r6+pqXnvFUREpE/CDPjXgak9XtcF80REZAiEGfDPAEea2QwzKwcuBu4NcX8iItJDaG3w7t5lZn8L/Ip8N8kfuPu6sPYnIiL7CrUfvLs/CDwY5j5ERKR3kX/JKiIi4VDAi4jElAJeRCSmiuqOTmbWBGw+wNvjge1DWE5/qLaBUW0Do9oGJq61HebuvV5EVFQBfzBmtvJAt6WKmmobGNU2MKptYEqxNjXRiIjElAJeRCSmhlPA3xp1AQeh2gZGtQ2MahuYkqtt2LTBi4hI/wynM3gREekHBbyISEwVfcCb2VlmtsHMXjGzL0Vdz/7MrMHMfm9mz5vZyohr+YGZNZrZ2h7zxprZr83s5eB5TBHVdp2ZvR4cu+fN7OyIaptqZkvN7AUzW2dmfxfMj/TYHaSuYjluGTNbYWarg/q+GsyfYWZPB/9m7wpGky2W2m4zs009jt2Coa4tqCNpZs+Z2f3B63COmRduV1WED/KjUL4KzATKgdXA0VHXtV+NDcD4qOsIajkVOB5Y22Pe14EvBdNfAm4sotquA75YBMdtEnB8MF0FvAQcHfWxO0hdxXLcDKgMplPA08D7gZ8CFwfzvw18uohquw34aBEcu88DPwbuD16HcsyK/Qy++76u7t4BFO7rKr1w92XAzv1mnw/8KJj+EfCRIS0qcIDaioK7v+nuzwbTLcCL5NMo0IcAAATBSURBVG85GemxO0hdRcHzWoOXqeDhwBnA/wTzI/mdO0htkTOzOmAx8L3gtRHSMSv2gO/tvq5F8wsecOBhM1sV3F+22Exw9zeD6a3AhCiL6cXfmtmaoAknkuajnsxsOnAc+TO+ojl2+9UFRXLcgqaG54FG4Nfk/+J+2927gkUi+ze7f23uXjh2XwuO3TfNLB1Bad8CrgFywetxhHTMij3gh4MPuPvxwIeBz5jZqVEXdCCe//uvKM5iAv8JHA4sAN4E/jXKYsysErgb+Jy7N/d8L8pj10tdRXPc3D3r7gvI35JzITAnqlr2t39tZjYX+DL5Gk8AxgLXDmVNZnYO0Ojuq4Zif8Ue8EV/X1d3fz14bgTuIf9LXky2mdkkgOC5MeJ6urn7tuAfYQ74LhEeOzNLkQ/RO9z958HsyI9db3UV03ErcPe3gaXAScBoMyvcTCjyf7M9ajsraPZyd28HfsjQH7tTgPPMrIF8k/MZwL8R0jEr9oAv6vu6mlmFmVUVpoEzgbUHX2vI3QtcGkxfCvxvhLXsoxCegT8lomMXtIF+H3jR3W/q8Vakx+5AdRXRcasxs9HB9AjgQ+S/J1gKfDRYLJLfuQPUtr7HB7aRb+ce0mPn7l929zp3n04+zx519yWEdcyi/ja5D982n02+98CrwD9EXc9+tc0k37NnNbAu6vqAO8n/yd5Jvh3vL8m37/0GeBl4BBhbRLX9N/B7YA35MJ0UUW0fIN/8sgZ4PnicHfWxO0hdxXLc5gHPBXWsBf5vMH8msAJ4BfgZkC6i2h4Njt1a4HaCnjYRHb9FvNOLJpRjpqEKRERiqtibaEREZIAU8CIiMaWAFxGJKQW8iEhMKeBFRGJKAS+xYWatwfN0M/vzQd72V/Z7vXwwty8SBgW8xNF0oF8B3+MqwgPZJ+Dd/eR+1iQy5BTwEkc3AB8Mxvu+Ohh06htm9kwwyNQVAGa2yMyeMLN7gReCeb8IBo5bVxg8zsxuAEYE27sjmFf4a8GCba+1/H0B/qzHth8zs/8xs/Vmdkdw9SRmdoPlx3hfY2b/MuRHR0rGe521iAxHXyI/Xvo5AEFQ73L3E4LRA580s4eDZY8H5rr7puD1p9x9Z3B5+zNmdre7f8nM/tbzA1ft7wLyg37NB8YH6ywL3jsOOAZ4A3gSOMXMXiQ/vMAcd/fC5fQiYdAZvJSCM4G/CIaOfZr8EARHBu+t6BHuAFeZ2WrgKfID3R3JwX0AuNPzg39tAx4nP1JhYdtbPD8o2PPkm452AXuB75vZBUDbIf90IgeggJdSYMBn3X1B8Jjh7oUz+N3dC5ktAv4YOMnd55MfyyRzCPtt7zGdBco8P+b3QvI3dzgHeOgQti9yUAp4iaMW8re4K/gV8Olg6F3MbFYw+uf+RgFvuXubmc0hf4u3gs7C+vt5AvizoJ2/hvytCVccqLBgbPdR7v4gcDX5ph2RUKgNXuJoDZANmlpuIz/e9nTg2eCLziZ6vyXaQ8CVQTv5BvLNNAW3AmvM7FnPD+9acA/5MdBXkx/58Rp33xp8QPSmCvhfM8uQ/8vi8wP7EUXem0aTFBGJKTXRiIjElAJeRCSmFPAiIjGlgBcRiSkFvIhITCngRURiSgEvIhJT/x/92+NXE82XogAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Retrieve each dictionary's values\n",
    "train_values = _train_loss[1]\n",
    " \n",
    "# Generate a sequence of integers to represent the iteration numbers\n",
    "n_iter = _train_loss[0]\n",
    " \n",
    "# Plot and label the training and validation loss values\n",
    "plt.plot(n_iter, train_values, label='Training Loss')\n",
    " \n",
    "# Add in a title and axes labels\n",
    "plt.title('Training Loss')\n",
    "plt.xlabel('Iterations')\n",
    "plt.ylabel('Loss')\n",
    " \n",
    "# Set the tick locations\n",
    "# plt.xticks(np.arange(0, 21, 2))\n",
    " \n",
    "# Display the plot\n",
    "plt.legend(loc='best')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8099c67b",
   "metadata": {},
   "source": [
    "A [common bug](https://twitter.com/karpathy/status/1013244313327681536?lang=en) encountered during training of a neural network is <u>forgetting to clear the gradient information of the parameters from the computational graph</u>. Remember that as we calculate the gradients for the different components of the graph, we use `+=` instead of `=` to accumulate gradient information because a certain node may be used more than once in the graph and to get its correct gradient we need to sum up all of the contributions to the gradient.\n",
    "\n",
    "In the manual implementation in the prior section, we forgot to clear the gradient value, so the gradient will get accumulated through subsequent forward passes — which were performed with different network weights! This makes no sense, of course, so as a simple fix, we need to set the gradients to zero or use `.zero_grad()` in PyTorch before `.backward`.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e1145329",
   "metadata": {},
   "source": [
    "-----------\n",
    "<br><br>\n",
    "# 13. Micrograd: Summary<a id=\"13\"></a>\n",
    "-----------------"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "823c4552",
   "metadata": {},
   "source": [
    "The summary below encapsulates the concepts discussed, covering the basics of neural networks, and their training process. Furthermore, the building and implementation of **Micrograd**, an autograd engine, helps in the understanding & reinforcement of the fundamental principles behind neural networks and backpropagation. Summary points include:\n",
    "\n",
    "\n",
    "- **Neural Networks (NNs)**: \n",
    "\n",
    "  - Neural networks are mathematical expressions used for processing data.\n",
    "\n",
    "  - They are composed of simple mathematical operations.\n",
    "\n",
    "  - In the case of a multi-layer perceptron, they take input data along with weights and parameters.\n",
    "\n",
    "- **Forward Pass and Loss Function**:\n",
    "\n",
    "  - The forward pass involves applying the neural net's mathematical expression to the input data.\n",
    "\n",
    "  - This is followed by evaluating a loss function, which measures the accuracy of predictions.\n",
    "\n",
    "  - Lower loss indicates better performance, where predictions align closely with targets.\n",
    "\n",
    "- **Backpropagation and Gradient Descent**:\n",
    "\n",
    "  - Backpropagation is used to compute gradients, indicating how each parameter affects the loss.\n",
    "\n",
    "  - Gradients guide the adjustment of parameters to minimize the loss.\n",
    "\n",
    "  - Gradient descent iteratively updates parameters to minimize the loss function.\n",
    "\n",
    "- **Power and Complexity**:\n",
    "\n",
    "  - Neural networks possess significant power, capable of addressing complex problems.\n",
    "\n",
    "  - They can have billions or even trillions of parameters, forming a complex network.\n",
    "\n",
    "  - Despite complexity, they operate on fundamental principles of gradient descent and loss minimization.\n",
    "\n",
    "- **NNs Emergent Properties**:\n",
    "\n",
    "  - Neurons within neural networks exhibit emergent properties when solving challenging problems.\n",
    "\n",
    "  - Properties arise from interactions within the network during training.\n",
    "\n",
    "- **Application in Language Modeling**:\n",
    "\n",
    "  - Neural networks are extensively used in language modeling tasks like predicting the next word in a sequence.\n",
    "\n",
    "  - Even large-scale models like GPT utilize the same fundamental principles of neural network training.\n",
    "\n",
    "  - Although more complex, they rely on gradient descent and loss minimization.\n",
    "\n",
    "- **Fundamental Principles**:\n",
    "\n",
    "  - While specific implementations may vary, the fundamental principles of neural network training remain consistent.\n",
    "\n",
    "  - Updates may include variations in gradient descent techniques (stochastic G.D., etc.) and loss functions (cross_entropy_loss, etc.) tailored to specific tasks.\n",
    "\n",
    "  - Despite variations, the core concepts of neural network training remain universal.\n",
    "\n",
    "- **Understanding and Implementation**:\n",
    "\n",
    "  - By understanding the foundational principles of neural networks, one can grasp their workings intuitively.\n",
    "\n",
    "  - Learning about neural networks' inner workings allows for better comprehension and implementation of related concepts.<br><br>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a9154496",
   "metadata": {},
   "source": [
    "-----\n",
    "<br><br><a id=\"r1\"></a>\n",
    "# References\n",
    "----\n",
    "1. \"<u>The spelled-out intro to neural networks and backpropagation: building micrograd</u>\" [youtube video](https://www.youtube.com/watch?v=VMj-3S1tku0), Aug 2022.\n",
    "2. Andrej Karpathy **Micrograd** [github repo](https://github.com/karpathy/micrograd).\n",
    "3. Andrej Karpathy  **Neural Networks: Zero to Hero** [github repo](https://github.com/karpathy/nn-zero-to-hero/tree/master) ([notebooks](https://github.com/karpathy/nn-zero-to-hero/tree/master/lectures/micrograd) to follow video tutorial with).\n",
    "4. Andrej Karpathy **Micrograd** follow-up [excercises](https://colab.research.google.com/drive/1FPTx1RXtBfc4MaTkf7viZZD4U2F9gtKN?usp=sharing) for practice.\n",
    "5. Article: \"<u>Backpropagation with Andrej Karpathy</u>\" - Kavishka Abeywardana, Pt [1](https://medium.com/thedeephub/backpropagation-with-andrej-karpathy-part-1-916bc9bb54d8), [2](https://medium.com/thedeephub/backpropagation-with-andrej-karpathy-part-2-291a70069dd6), [3](https://medium.com/@kdwa2404/backpropagation-with-andrej-karpathy-part-3-717642087c34), Feb 2024.\n",
    "6. \"<u>Back(prop) to Basics — Fundamentals of Neural Networks</u>\" - Elad Rapaport, [article](https://medium.com/@erap129/back-prop-to-basics-fundamentals-of-neural-networks-d54cd8fc61e4), May 2023.\n",
    "7. \"<u>What is Perceptron: A Beginners Guide</u>\" - Mayank Banoula, [article](https://www.simplilearn.com/tutorials/deep-learning-tutorial/perceptron#:~:text=The%20perceptron%20model%20begins%20with,and%20is%20represented%20by%20'f.), May 2023.\n",
    "8. \"<u>What the Hell is Perceptron? The Fundamentals of Neural Networks</u>\" - Sagar Sharma, [article](https://towardsdatascience.com/what-the-hell-is-perceptron-626217814f53), Sept 2017.\n",
    "9. \"<u>Topological Sort: In Typescript and C#</u>\" - Harish Reddy Konduru, [article](https://medium.com/@konduruharish/topological-sort-in-typescript-and-c-6d5ecc4bad95), June 2020.\n",
    "10. \"<u>Automatic Differentiation Part 2: Implementation Using Micrograd</u>\" - Aritra Roy Gosthipaty & Ritwik Raha, [article](https://pyimagesearch.com/2022/12/26/automatic-differentiation-part-2-implementation-using-micrograd/), Dec 2022.\n",
    "11. \"<u>The Chain Rule of Calculus – Even More Functions</u>\" - Stefania Cristina, [article](https://machinelearningmastery.com/the-chain-rule-of-calculus-even-more-functions/), March 2022.\n",
    "12. CS231n Convolutional Neural Networks for Visual Recognition [course website](https://cs231n.github.io/)\n",
    "13. CS231n \"<u>Neural Networks Part 1: Setting up the Architecture</u>\" [lecture notes](https://cs231n.github.io/neural-networks-1/)\n",
    "14. Wikipedia: [derivative](https://en.wikipedia.org/wiki/Derivative), [backpropagation](https://en.wikipedia.org/wiki/Backpropagation), [activation function](https://en.wikipedia.org/wiki/Activation_function), [topological sorting](https://en.wikipedia.org/wiki/Topological_sorting), [multivariate chain rule](https://en.wikipedia.org/wiki/Chain_rule#Example), [gradient descent](https://en.wikipedia.org/wiki/Gradient_descent)\n",
    "15. PyTorch Documentation [link](https://pytorch.org/docs/stable/index.html).\n",
    "16. OpenAI [chatGPT](https://chat.openai.com/): Helped in preparing [**Section 13. Micrograd: Summary**](#13) notes\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}