0.GettingStartedWithTensorFlow1.x-I.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Getting started with TensorFlow - I"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_Notes compiled from Chapter 9: Hands-On Machine Learning with Scikit-Learn and TensorFlow_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We start by creating a simple graph in Tensor Flow."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "def simple_graph():\n",
    "    \n",
    "    # creating the computational graph. Variables are not initialized, and no computation is performed yet.\n",
    "    x = tf.Variable(3, name=\"x\")\n",
    "    y = tf.Variable(4, name=\"y\")\n",
    "    f = x*x*y + y + 2\n",
    "\n",
    "    # To evaluate the graph constructed above, we open a tensorflow session and use it to initialize variables and evaluate f.\n",
    "    with tf.Session() as sess:\n",
    "        x.initializer.run()\n",
    "        y.initializer.run()\n",
    "        result = f.eval()\n",
    "    \n",
    "    return\n",
    "\n",
    "simple_graph()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A TensorFlow session takes care of placing the operations onto devices(CPUs and GPUs) and running them, and it holds all the variable values."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "NOTE: Calling x.initializer.run() is equivalent to calling tf.get_default_session().run(x.initializer)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  Instead of manually running the initializer for every single variable, we can use the global_variables_initializer() as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = tf.Variable(3, name=\"x\")\n",
    "y = tf.Variable(4, name=\"y\")\n",
    "f = x*x*y + y + 2\n",
    "\n",
    "init = tf.global_variables_initializer()\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    init.run()\n",
    "    result = f.eval()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### In Tensorflow, any node that is created is automatically added to the default graph."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x1 = tf.Variable(2)\n",
    "x1.graph is tf.get_default_graph()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Attributes and methods of the default graph:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['_ControlDependenciesController', '__class__', '__delattr__', '__dict__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__le__', '__lt__', '__module__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', '_add_device_to_stack', '_add_function', '_add_new_tf_operations', '_add_op', '_apply_device_functions', '_as_graph_def', '_as_graph_element_locked', '_attr_scope', '_attr_scope_map', '_building_function', '_c_graph', '_check_not_finalized', '_collections', '_colocate_with_for_gradient', '_colocation_stack', '_container', '_control_dependencies_for_inputs', '_control_dependencies_stack', '_control_flow_context', '_copy_functions_to_graph_def', '_create_op_from_tf_operation', '_create_op_helper', '_current_control_dependencies', '_default_original_op', '_device_function_stack', '_device_functions_outer_to_inner', '_distribution_strategy_stack', '_finalized', '_functions', '_get_control_flow_context', '_get_function', '_get_op_def', '_get_operation_by_name_unsafe', '_get_operation_by_tf_operation', '_get_tensor_by_tf_output', '_gradient_override_map', '_graph_colocation_stack', '_graph_control_dependencies_stack', '_graph_def_versions', '_graph_device_function_stack', '_graph_key', '_group_lock', '_handle_deleters', '_handle_feeders', '_handle_movers', '_handle_readers', '_is_function', '_kernel_label_map', '_last_id', '_last_loss_reduction', '_lock', '_make_colocation_conflict_message', '_mutation_lock', '_name_stack', '_names_in_use', '_next_id', '_next_id_counter', '_nodes_by_id', '_nodes_by_name', '_op_to_kernel_label_map', '_original_op', '_pop_control_dependencies_controller', '_push_control_dependencies_controller', '_record_op_seen_by_control_dependencies', '_registered_ops', '_scoped_c_graph', '_seed', '_session_run_lock', '_set_control_flow_context', '_snapshot_colocation_stack_metadata', '_snapshot_device_function_stack_metadata', '_stack_state_is_thread_local', '_tf_api_names', '_tf_api_names_v1', '_thread_local', '_unfeedable_tensors', '_unfetchable_ops', '_unsafe_unfinalize', '_variable_creator_scope', '_variable_creator_stack', '_version', 'add_to_collection', 'add_to_collections', 'as_default', 'as_graph_def', 'as_graph_element', 'building_function', 'clear_collection', 'collections', 'colocate_with', 'container', 'control_dependencies', 'create_op', 'device', 'finalize', 'finalized', 'get_all_collection_keys', 'get_collection', 'get_collection_ref', 'get_name_scope', 'get_operation_by_name', 'get_operations', 'get_tensor_by_name', 'gradient_override_map', 'graph_def_versions', 'is_feedable', 'is_fetchable', 'name_scope', 'prevent_feeding', 'prevent_fetching', 'seed', 'switch_to_thread_local', 'unique_name', 'version']\n"
     ]
    }
   ],
   "source": [
    "print(dir(tf.get_default_graph()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<tf.Operation 'Variable/initial_value' type=Const>,\n",
       " <tf.Operation 'Variable' type=VariableV2>,\n",
       " <tf.Operation 'Variable/Assign' type=Assign>,\n",
       " <tf.Operation 'Variable/read' type=Identity>,\n",
       " <tf.Operation 'x/initial_value' type=Const>,\n",
       " <tf.Operation 'x' type=VariableV2>,\n",
       " <tf.Operation 'x/Assign' type=Assign>,\n",
       " <tf.Operation 'x/read' type=Identity>,\n",
       " <tf.Operation 'y/initial_value' type=Const>,\n",
       " <tf.Operation 'y' type=VariableV2>,\n",
       " <tf.Operation 'y/Assign' type=Assign>,\n",
       " <tf.Operation 'y/read' type=Identity>,\n",
       " <tf.Operation 'mul' type=Mul>,\n",
       " <tf.Operation 'mul_1' type=Mul>,\n",
       " <tf.Operation 'add' type=Add>,\n",
       " <tf.Operation 'add_1/y' type=Const>,\n",
       " <tf.Operation 'add_1' type=Add>,\n",
       " <tf.Operation 'x_1/initial_value' type=Const>,\n",
       " <tf.Operation 'x_1' type=VariableV2>,\n",
       " <tf.Operation 'x_1/Assign' type=Assign>,\n",
       " <tf.Operation 'x_1/read' type=Identity>,\n",
       " <tf.Operation 'y_1/initial_value' type=Const>,\n",
       " <tf.Operation 'y_1' type=VariableV2>,\n",
       " <tf.Operation 'y_1/Assign' type=Assign>,\n",
       " <tf.Operation 'y_1/read' type=Identity>,\n",
       " <tf.Operation 'mul_2' type=Mul>,\n",
       " <tf.Operation 'mul_3' type=Mul>,\n",
       " <tf.Operation 'add_2' type=Add>,\n",
       " <tf.Operation 'add_3/y' type=Const>,\n",
       " <tf.Operation 'add_3' type=Add>,\n",
       " <tf.Operation 'init' type=NoOp>,\n",
       " <tf.Operation 'Variable_1/initial_value' type=Const>,\n",
       " <tf.Operation 'Variable_1' type=VariableV2>,\n",
       " <tf.Operation 'Variable_1/Assign' type=Assign>,\n",
       " <tf.Operation 'Variable_1/read' type=Identity>]"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.get_default_graph().get_operations()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Running the same commands in jupyter notebook leads to addition of duplicate nodes in the default graph as we can see above. A conveninet solution to this is to <font color=blue>reset the default graph</font> as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[]"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.reset_default_graph()\n",
    "tf.get_default_graph().get_operations()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Lifecycle of a Node Value"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10\n",
      "15\n"
     ]
    }
   ],
   "source": [
    "w = tf.constant(3)\n",
    "x = w + 2\n",
    "y = x + 5\n",
    "z = x * 3\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    print(y.eval())\n",
    "    print(z.eval())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When we evaluate a node in tensorflow, it automatically evaluates its dependencies first. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the above snippet, we define a simple graph consisting of w, x, y, and z nodes. Then we start the session and run the graph to evaluate node y. TensorFlow detects that node y is dependent on x, which in turn depends on w. So the order of evaluation is w, then x and then y."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However, to evaluate z, it again evaluates x, and w. It does not reuse the previous evaluation of w and x."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To evaluate y and z efficiently, without evaluating w and x twice, we evaluate w and x in just one graph run as shown below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10\n",
      "15\n"
     ]
    }
   ],
   "source": [
    "with tf.Session() as sess:\n",
    "    y_val, z_val = sess.run([y, z])\n",
    "    print(y_val)\n",
    "    print(z_val)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In a single process TensorFlow, multiple sessions do not share any state, even if they reuse the same graph. Each session has its own copy of every variable. A variable starts its life when its initializer is run, and it ends when the session is closed."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### TensorFlow Operations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "TensorFlow operations can take any number of inputs and produce any number of outpts. Constants and variables take no input, and they are called _source operations_."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv",
   "language": "python",
   "name": ".venv"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}