adding dropout figures and some more exposition

P03-C03-mlp-dropout-scratch.ipynb

@@ -107,9 +107,26 @@
     "\n",
     "To prevent the neural network from depending too much on any exact activation pathway,\n",
     "Hinton and Srivastava proposed randomly *dropping out* (i.e. setting to $0$) \n",
-    "the values of half of the nodes in every layer during the forward pass.\n",
+    "the hidden nodes in every layer with probability $.5$.\n",
+    "Given a network with $n$ nodes we are sampling uniformly at random from the $2^n$ \n",
+    "networks in which a subset of the nodes are turned off. \n",
     "\n",
-    "[**add figures, complete exposition**]\n",
+    "![](./img/dropout.png)\n",
+    "\n",
+    "One intuition here is that because the nodes to drop out are chosen randomly on every pass,\n",
+    "the representations in each layer can't depend on the exact values taken by nodes in the previous layer. \n",
+    "\n",
+    "## Making predictions with dropout models\n",
+    "\n",
+    "However, when it comes time to make predictions, \n",
+    "we want to use the full representational power of our model. \n",
+    "In other words, we don't want to drop out activations at test time.\n",
+    "One principled way to justify the use of all nodes simultaneously,\n",
+    "despite not training in this fasion,\n",
+    "is that it's a form of model averaging. \n",
+    "At each layer we average the representations of all of the $2^n$ dropout networks.\n",
+    "Because each node has a $.5$ probability of being on during training, \n",
+    "its vote is scaled by $.5$ when we use all nodes at prediction time\n",
     "\n"
    ]
   },