intermediate-stuff.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# A slightly deeper dive\n",
    "Now we'll try to cover some \"intermediate\" features of the language.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1. Python magic methods\n",
    "\n",
    "Before we look at magic methods, here is a quick overview of the different types of method naming conventions :\n",
    "\n",
    "```\n",
    "1. _method    : To prevent automatic import due to a \"from xyz import *\" statement.\n",
    "2. __method   : To mark as a private method.\n",
    "3. method_    : To deal with reserved words\n",
    "4. __method__ : magic functions, hooks that are triggered on various builtin operators and functions.\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "False\n",
      "A foo\n",
      "__foo\n",
      "doo\n"
     ]
    }
   ],
   "source": [
    "class A(object):\n",
    "    def __foo(self):\n",
    "        print \"A foo\"\n",
    "    def class_(self):\n",
    "        self.__foo()\n",
    "        print self.__foo.__name__\n",
    "    def __doo__(self):\n",
    "        print \"doo\"\n",
    "        \n",
    "a = A()\n",
    "print hasattr(a, '__foo') # where has this gone?\n",
    "a.class_()\n",
    "a.__doo__()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So we can see here that we cannot access the private method outside the class, this is due to name mangling. The members can be inspected using the builtin `dir` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['_A__foo', '__class__', '__delattr__', '__dict__', '__doc__', '__doo__', '__format__', '__getattribute__', '__hash__', '__init__', '__module__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', 'class_']\n"
     ]
    }
   ],
   "source": [
    "print dir(a)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we see, the name has changed to `_A_foo` and that is the only reason it is \"private\" if we explicitly call it by it's mangaled name, it is very much accessible."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "A foo\n"
     ]
    }
   ],
   "source": [
    "a._A__foo()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It's worth noting that the `__magic_method__` format does not do anything special unless we use the predefined names. It is also strongly advised that you only override the builtin magic methods and not re define your own as I had just done previously!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x : 4, y : 7\n",
      "False\n",
      "True\n"
     ]
    }
   ],
   "source": [
    "class P(object):\n",
    "    def __init__(self, x, y):\n",
    "        self.x = x\n",
    "        self.y = y\n",
    "    \n",
    "    def __add__(self, other):\n",
    "        return P(self.x + other.x, self.y + other.y)\n",
    "    \n",
    "    def __gt__(self, other):\n",
    "        #if both x and y component is greater than the other object's x and y\n",
    "        return (self.x > other.x) and (self.y > other.y)\n",
    "    \n",
    "    def __str__(self):\n",
    "        return \"x : %s, y : %s\" % (self.x, self.y)\n",
    "    \n",
    "p1 = P(0,0)\n",
    "p2 = P(3,4)\n",
    "p3 = P(1,3)\n",
    "\n",
    "print p3 + p2\n",
    "print p1 > p2\n",
    "print p2 > p1\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can even add stuff like **slicing capabilities**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "5\n",
      "-4\n",
      "[2, 3, 4]\n"
     ]
    }
   ],
   "source": [
    "class Seq(object):\n",
    "    def __getitem__(self, i):\n",
    "        if type(i) is slice:\n",
    "            # this has edge case issues, but just a demo!\n",
    "            return list(range(i.start, i.stop))\n",
    "        else:\n",
    "            return i\n",
    "\n",
    "s = Seq()\n",
    "print s[5]\n",
    "print s[-4]\n",
    "print s[2:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note** We have covered a very small subset of all the \"magic\" functions! Please do have a look at the official python docs for the exhaustive reference!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. Comprehension\n",
    "\n",
    "Python comprehensions gives us interesting ways to populate builtin data structures, *in terms of expressions*, as mathematicians would do. Comprehensions are a paradigm borrowed from functional languages, and provides a great deal of syntactic sugar. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0, 1, 2, 3, 4]\n",
      "[0, 1, 4, 9, 16]\n"
     ]
    }
   ],
   "source": [
    "l = [i for i in range(0,5)]\n",
    "l2 = [i*i for i in range(0,5)]\n",
    "print l\n",
    "print l2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also define slightly more complex expressions with the use of if statements and nested loops."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0, 2, 4]\n"
     ]
    }
   ],
   "source": [
    "l = [i for i in range(0,5) if i % 2 ==0]\n",
    "print l"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[(1, 0), (2, 0), (2, 1), (3, 0), (3, 1), (3, 2), (4, 0), (4, 1), (4, 2), (4, 3)]\n"
     ]
    }
   ],
   "source": [
    "# get all combinations where x > y and x, y < 5\n",
    "xy = [ (x, y) for x in range (0,5) for y in range (0, 5) if x > y]\n",
    "print xy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['H', 'E', 'L', 'L', 'O']\n"
     ]
    }
   ],
   "source": [
    "# we can even call functions\n",
    "l = [x.upper() for x in \"hello\"]\n",
    "print l\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[['hello', 'HELLO', 5],\n",
       " ['how', 'HOW', 3],\n",
       " ['are', 'ARE', 3],\n",
       " ['you', 'YOU', 3],\n",
       " ['doing?', 'DOING?', 6]]"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# creating lists of lists is also a synch\n",
    "gre = \"hello how are you doing?\"\n",
    "[[s.lower(), s.upper(), len(s)] for s in gre.split()]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[0, 1, 2], [1, 2, 3], [2, 3, 4]]\n"
     ]
    }
   ],
   "source": [
    "# nested comprehensions - we can do it, but it may not be very readable\n",
    "matrix = [[i+x for i in range(3)] for x in range(3)]\n",
    "print matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{0: 0, 1: 1, 2: 4, 3: 9, 4: 16}\n"
     ]
    }
   ],
   "source": [
    "# we can also have a comprehension for dicts\n",
    "d = {x : x**2 for x in range(5)}\n",
    "print d"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3. Functional parts\n",
    "\n",
    "There are a lot of concepts borrowed from functional languages to make the code look more elegant. List comprehension was just scratching the surface. We have builtin helpers such as `lambda, filter, zip, map, all, any` to help us write cleaner code! Other than the builtin components we have functools (which I won't be covering) which even helps us with partial functions and currying!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0, 1, 4, 9, 16]\n",
      "[0, 1, 4, 9, 16]\n"
     ]
    }
   ],
   "source": [
    "# lambda is used when you need anonymous functions defined as an expression\n",
    "# in this example you could define a function and pass it to foo, or use the lambda\n",
    "# in this case the lambda is neater.\n",
    "# lambdas can take in n number of params, and the body is a single expression that is also the return value\n",
    "\n",
    "def foo(list_, func):\n",
    "    l = []\n",
    "    for i in list_:\n",
    "        l.append(func(i))\n",
    "    return l\n",
    "\n",
    "def sq(i):\n",
    "    return i**2\n",
    "\n",
    "l = [i for i in range(5)]\n",
    "print foo(l, sq)\n",
    "print foo(l, lambda x : x**2)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x : 1\n",
      "x : 2\n",
      "x : 3\n",
      "x : 4\n",
      "x : 5\n"
     ]
    }
   ],
   "source": [
    "class P(object):\n",
    "    def __init__(self, x):\n",
    "        self.x = x\n",
    "    def __str__(self):\n",
    "        return \"x : %s\" % self.x\n",
    "\n",
    "l = [P(5), P(2), P(1), P(4), P(3)]\n",
    "l.sort(cmp=lambda x, y: x.x - y.x)\n",
    "for p in l : print p # [str(p) for p in l]\n",
    "    \n",
    "# there are many more complex and cryptic ways to use (exploit) lambdas, \n",
    "#     you can search for it online if you are interested\n",
    "# check lambda with multiple args\n",
    "# lambda *x : sys.stdout.write(\" \".join(map(str, x)))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0, 2, 4, 6, 8] <type 'list'>\n"
     ]
    }
   ],
   "source": [
    "# filter is a function that takes an interable and a callable \n",
    "#  applies the function to each element,i.e. ret = func(element)\n",
    "#  and returns a list with elements for which 'ret' was true\n",
    "\n",
    "l = range(0,10)\n",
    "l = filter(lambda x : x%2==0, l)\n",
    "print l, type(l)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[(1, 0, 1), (2, 4, 'm'), (3, 6, 7)]\n"
     ]
    }
   ],
   "source": [
    "# zip is to sew together a bunch of iterables\n",
    "# the list generated is of the minimum size of all the iterators that have gone in!\n",
    "\n",
    "a = [1,2,3,4,5]\n",
    "b = (0,4,6,7)\n",
    "c = {1:'a', 7:'b', 'm':'v'}\n",
    "\n",
    "print zip(a,b,c)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Though zip looks trivial it is a fairly important operation for mathematical algorithms - matrices, curve fitting, interpolation, pattern recognition, that sort of thing. Also very important in engineering applications like digital signal processing where much of what you do is combine multiple signals or apply linear transforms to them - both are based on the sample index, hence, zip it.\n",
    "\n",
    "It would be a pain to reimplement it everytime taking care of all edge cases, etc!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]\n"
     ]
    }
   ],
   "source": [
    "# map - takes in a iterable and callable - applies the callable to each element of the iterable\n",
    "#  returns a new list with each element being the return value of \"callable(elem)\"\n",
    "\n",
    "print map(lambda x: x**2, range(10))\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "hello how are you 1234"
     ]
    }
   ],
   "source": [
    "# map is extremely useful as a shorthand for \"applying\" a function across an iterable,\n",
    "#  especially in conjunction with lambda\n",
    "import sys\n",
    "my_print = lambda *x : sys.stdout.write(\" \".join(map(str, x)))\n",
    "my_print(\"hello\", \"how are you\", 1234)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 4. Iterables\n",
    "\n",
    "Lists, dicts, tuples are iterables. i.e. we can \"iterate\" through them! Any object that supports the iterator protocol is a iterator. The iterator protocol states that the object should override the `__iter__` magic method that returns an object that has a `.next()` method and raises a `StopIteration exception`. \n",
    "\n",
    "There are 4 key ways to create an iterable - \n",
    "\n",
    "```\n",
    "1. Iterators - classes that override __iter__ and next()\n",
    "2. Generator functions - functions that yield\n",
    "3. Generator expressions\n",
    "4. overriding the __getitem__ magic method.\n",
    "```\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1\n",
      "2\n",
      "3\n",
      "4\n",
      "5\n"
     ]
    }
   ],
   "source": [
    "# 1. Iterators\n",
    "\n",
    "class myitr(object):\n",
    "    def __init__(self, ulimit=5):\n",
    "        self.limit = ulimit\n",
    "    def __iter__(self):\n",
    "        self.index = 0\n",
    "        return self\n",
    "    def next(self):\n",
    "        if self.index < self.limit:\n",
    "            self.index += 1\n",
    "            return self.index\n",
    "        else:\n",
    "            raise StopIteration\n",
    "            \n",
    "itr = myitr()\n",
    "for i in itr:\n",
    "    print i"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0\n",
      "1\n",
      "2\n",
      "3\n",
      "4\n"
     ]
    }
   ],
   "source": [
    "# 2. Generators\n",
    "\n",
    "def gen(lim):\n",
    "    i = 0\n",
    "    while i < lim:\n",
    "        yield i\n",
    "        i = i + 1\n",
    "        \n",
    "for i in gen(5):\n",
    "    print i"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0\n",
      "1\n",
      "4\n",
      "9\n",
      "16\n"
     ]
    }
   ],
   "source": [
    "# 3. Generator expression\n",
    "\n",
    "def seq(num):\n",
    "    return (i**2 for i in range(num))\n",
    "\n",
    "for i in seq(5):\n",
    "    print i\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0\n",
      "1\n",
      "2\n",
      "3\n",
      "4\n"
     ]
    }
   ],
   "source": [
    "# 4. Overriding __getitem__\n",
    "\n",
    "class Itr(object):\n",
    "    def __init__(self, x):\n",
    "        self.x = x\n",
    "        \n",
    "    def __getitem__(self, index):\n",
    "        if index < self.x:\n",
    "            return index\n",
    "        else:\n",
    "            raise StopIteration\n",
    "\n",
    "\n",
    "for i in Itr(5):\n",
    "    print i"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 5. Decorators\n",
    "\n",
    "Before we start with decorators, we need to know a bit about closures. A closure is a function object that remembers values in enclosing scopes regardless of whether those scopes are still present in memory. Most common case is when we define a function within a function, and return the inner function. If the inner function definition uses variables / values in the outer function, it maintains the references to those even after it is returned (and no longer in the scope of the outer function)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4 9\n",
      "8 27\n"
     ]
    }
   ],
   "source": [
    "# closure example - raised_to_power returns a fn that takes a variable and raises to the power 'n'\n",
    "#  'n' is passed only once - while defining the function!\n",
    "\n",
    "def raised_to_power(n):\n",
    "    def fn(x):\n",
    "        return x**n\n",
    "    return fn\n",
    "    \n",
    "p2 = raised_to_power(2)\n",
    "p3 = raised_to_power(3)\n",
    "\n",
    "print p2(2), p2(3) # still remembers that n=2\n",
    "print p3(2), p3(3) # still remembers that n=3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "8\n",
      "8\n",
      "8\n",
      "8\n"
     ]
    }
   ],
   "source": [
    "# have to be cautious!\n",
    "\n",
    "def power_list(n):\n",
    "    '''returns list of fn, each raises to power i, where i : 0 --> n'''\n",
    "    fn_list = []\n",
    "\n",
    "    def fn(x):\n",
    "        return x**i\n",
    "    \n",
    "    for i in range(n):\n",
    "        # doesn't matter if fn was defined here either\n",
    "        fn_list.append(fn)\n",
    "\n",
    "    return fn_list\n",
    "\n",
    "for j in power_list(4):\n",
    "    print j(2) # prints 2 power 3, 4 times\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "entring function foo\n",
      "x :  4\n",
      "exiting function foo\n"
     ]
    }
   ],
   "source": [
    "# decorator is just a nicer way of defining a closure - more syntactic sugar\n",
    "\n",
    "def deco(fn):\n",
    "    def new_fn(*args, **kwargs):\n",
    "        print \"entring function\", fn.__name__\n",
    "        ret = fn(*args, **kwargs)\n",
    "        print \"exiting function\", fn.__name__\n",
    "    return new_fn\n",
    "\n",
    "@deco\n",
    "def foo(x):\n",
    "    print \"x : \", x\n",
    "    \n",
    "foo(4)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<h1> Hello Nutanix! </h1>\n"
     ]
    }
   ],
   "source": [
    "# Another example\n",
    "\n",
    "def add_h1(fn):\n",
    "    def nf(pram):\n",
    "        return \"<h1> \" + fn(pram) + \" </h1>\"\n",
    "    return nf\n",
    "\n",
    "@add_h1\n",
    "def greet(name):\n",
    "    return \"Hello {0}!\".format(name)\n",
    "\n",
    "print greet(\"Nutanix\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<h3> Hello Nutanix! </h3>\n",
      "<h2> <h4> Hello Nutanix! </h4> </h2>\n"
     ]
    }
   ],
   "source": [
    "# decorator that takes parameter\n",
    "\n",
    "def add_h(num):\n",
    "    def deco(fn):\n",
    "        # this is the decorator for a specific 'h'\n",
    "        def nf(pram):\n",
    "            return \"<h%s> \"%num + fn(pram) + \" </h%s>\"%num\n",
    "        return nf\n",
    "    return deco\n",
    "\n",
    "@add_h(3)\n",
    "def greet(name):\n",
    "    return \"Hello {0}!\".format(name)\n",
    "print greet(\"Nutanix\")\n",
    "\n",
    "\n",
    "# we can have multiple decorators as well\n",
    "@add_h(2)\n",
    "@add_h(4)\n",
    "def greet2(name):\n",
    "    return \"Hello {0}!\".format(name)\n",
    "\n",
    "print greet2(\"Nutanix\")\n",
    "        "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 6. More OO\n",
    "\n",
    "Let's take another look at classes and OO in python. We'll start with multiple inheritance (or mixins).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Mixins\n",
    "\n",
    "Let's start with this inheritance model -\n",
    "```\n",
    "    A\n",
    "   / \\\n",
    "  B   C\n",
    "   \\ /\n",
    "    D\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "D.init\n",
      "B.foo\n",
      "E.init\n",
      "C.foo\n"
     ]
    }
   ],
   "source": [
    "class A(object):\n",
    "    def __init__(self):\n",
    "        print \"A.init\"        \n",
    "    def foo(self):\n",
    "        print \"A.foo\"\n",
    "        \n",
    "class B(A):\n",
    "    def __init__(self):\n",
    "        print \"B.init\"\n",
    "    def foo(self):\n",
    "        print \"B.foo\"\n",
    "\n",
    "class C(A):\n",
    "    def __init__(self):\n",
    "        print \"C.init\"\n",
    "    def foo(self):\n",
    "        print \"C.foo\"\n",
    "        \n",
    "class D(B, C):\n",
    "    def __init__(self):\n",
    "        print \"D.init\"\n",
    "    #def foo(self):\n",
    "    #    print \"D.foo\"\n",
    "    \n",
    "class E(C, B):\n",
    "    def __init__(self):\n",
    "        print \"E.init\"\n",
    "\n",
    "d = D()\n",
    "d.foo() \n",
    "\n",
    "e = E()\n",
    "e.foo()\n",
    "\n",
    "# we see that fn lookup's happen in the order of declaration of parent in the child's definition."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What if the mixin is slightly more complex? (Note, how ever complex stuff gets - which it shouldn't, python will never let you create a circular dependency!\n",
    "\n",
    "```\n",
    "       A\n",
    "     /  \\\n",
    "     B  C\n",
    "     | /|\n",
    "     D/ |\n",
    "     |  |\n",
    "      \\ |\n",
    "       E\n",
    "       \n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "E.init\n",
      "D.foo\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(__main__.E, __main__.D, __main__.C, __main__.A, object)"
      ]
     },
     "execution_count": 111,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "class A(object):\n",
    "    def __init__(self):\n",
    "        print \"A.init\"        \n",
    "    def foo(self):\n",
    "        print \"A.foo\"\n",
    "        \n",
    "class B(A):\n",
    "    def __init__(self):\n",
    "        print \"B.init\"\n",
    "    def foo(self):\n",
    "        print \"B.foo\"\n",
    "\n",
    "class C(A):\n",
    "    def __init__(self):\n",
    "        print \"C.init\"\n",
    "    def foo(self):\n",
    "        print \"C.foo\"\n",
    "        \n",
    "class D(C):\n",
    "    def __init__(self):\n",
    "        print \"D.init\"\n",
    "    def foo(self):\n",
    "        print \"D.foo\"\n",
    "    \n",
    "class E(D, C): # you can't have (C, D) - TypeError: Cannot create a consistent MRO\n",
    "    def __init__(self):\n",
    "        print \"E.init\"\n",
    "\n",
    "e = E()\n",
    "e.foo()\n",
    "E.__mro__\n",
    "\n",
    "# so what's mro - (explain in live session)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**NOTE** MRO is also the reason why `super()` is called in the manner it is. You need both the class and the object to traverse the next parent in the MRO!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Attribute access hooks\n",
    "Next let's have a look at two magic functions which deal with object variable access, `__getattr__` and `__setattr__`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 127,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "X : 3\n",
      "getattr val : y <type 'str'>\n",
      "Y : y\n"
     ]
    }
   ],
   "source": [
    "class A(object):\n",
    "    def __init__(self, x):\n",
    "        self.x = x\n",
    "    def __getattr__(self, val):\n",
    "        print \"getattr val :\", val, type(val)\n",
    "        return val\n",
    "\n",
    "a = A(3)\n",
    "print \"X :\", a.x # getattr not called for x\n",
    "ret = a.y\n",
    "print \"Y :\", ret\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "setattr\n",
      "3\n",
      "getattr\n",
      "y\n",
      "setattr\n",
      "setattr\n"
     ]
    }
   ],
   "source": [
    "class A(object):\n",
    "    def __init__(self, x):\n",
    "        self.x = x\n",
    "    def __getattr__(self, val):\n",
    "        print \"getattr\"\n",
    "        return val\n",
    "    def __setattr__(self, name, val):\n",
    "        print \"setattr\"\n",
    "        if name == 'x':\n",
    "            self.__dict__[name] = val\n",
    "\n",
    "a = A(3)\n",
    "print a.x\n",
    "print a.y\n",
    "\n",
    "# setattr is called for both\n",
    "a.y = 5\n",
    "a.x = 5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Callable objects\n",
    "\n",
    "You can make an object callable (a functor) by overriding the magic `__call__` method. You can call the object like a function and the `__call__` method will be called instead. This is useful when you want to have more complex functionality (like state) plus data, but want to keep the syntactic sugar / simplicity of a function!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 132,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "here!\n",
      "15\n"
     ]
    }
   ],
   "source": [
    "class MulBy(object):\n",
    "    def __init__(self, x):\n",
    "        self.x = x\n",
    "    def __call__(self, n):\n",
    "        print \"here!\"\n",
    "        return self.x * n\n",
    "    \n",
    "m = MulBy(5)\n",
    "print m(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### \\__new\\__ vs \\__init\\__\n",
    "\n",
    "Until now we never bothered to see how / when the python objects were created. The `__init__` function just deals with handling the initialization of the object, the actual creation happens within `__new__`, which can be overridden.\n",
    "\n",
    "From the python mailing list\n",
    "\n",
    "```\n",
    "Use __new__ when you need to control the creation of a new instance.\n",
    "Use __init__ when you need to control initialization of a new instance.\n",
    "\n",
    "__new__ is the first step of instance creation.  It's called first,\n",
    "and is responsible for returning a new instance of your class.  In\n",
    "contrast, __init__ doesn't return anything; it's only responsible for\n",
    "initializing the instance after it's been created.\n",
    "```\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 137,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "new\n",
      "(1, 2, 3) {'a': 4}\n",
      "init\n",
      "(1, 2, 3) {'a': 4}\n"
     ]
    }
   ],
   "source": [
    "class X(object):\n",
    "    def __new__(cls, *args, **kwargs):\n",
    "        print \"new\"\n",
    "        print args, kwargs\n",
    "        return object.__new__(cls)\n",
    "        \n",
    "    def __init__(self, *args, **kwargs):\n",
    "        print \"init\"\n",
    "        print args, kwargs\n",
    "        \n",
    "x = X(1,2,3,a=4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**This approach is rather useful for the factory design pattern!**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 150,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "New linux vm. state : off\n",
      "<class '__main__.LinuxVM'>\n",
      "linux ops\n",
      "\n",
      "New windows vm. state : on\n",
      "<class '__main__.WindowsVM'>\n",
      "windows ops\n"
     ]
    }
   ],
   "source": [
    "class WindowsVM(object):\n",
    "    def __init__(self, state=\"off\"):\n",
    "        print \"New windows vm. state : %s\" %state\n",
    "    def operation(self):\n",
    "        print \"windows ops\"\n",
    "        \n",
    "class LinuxVM(object):\n",
    "    def __init__(self, state=\"off\"):\n",
    "        print \"New linux vm. state : %s\" %state\n",
    "    def operation(self):\n",
    "        print \"linux ops\"\n",
    "\n",
    "class VM(object):\n",
    "    MAP = {\"Linux\" : LinuxVM, \"Windows\": WindowsVM}\n",
    "    \n",
    "    def __new__(self, vm_type, state=\"off\"):\n",
    "        # return object.__new__(VM.MAP[vm_type]) #--doesn't call init of other class\n",
    "        vm = object.__new__(VM.MAP[vm_type])\n",
    "        vm.__init__(state)\n",
    "        return vm\n",
    "\n",
    "\n",
    "vm1 = VM(\"Linux\")\n",
    "print type(vm1)\n",
    "vm1.operation()\n",
    "print \"\"\n",
    "vm2 = VM(\"Windows\", state=\"on\")\n",
    "print type(vm2)\n",
    "vm2.operation()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 7. Properties\n",
    "\n",
    "Properties are ways of adding behaviour to instance variable access, i.e. trigger a function when a variable is being accessed. This is most commonly used for getters and setters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 159,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "setx\n",
      "getx\n",
      "5\n",
      "delx\n",
      "<property object at 0x10fd42c58>\n"
     ]
    }
   ],
   "source": [
    "# simple example \n",
    "class C(object):\n",
    "    def __init__(self):\n",
    "        self._x = None\n",
    "\n",
    "    def getx(self):\n",
    "        print \"getx\"\n",
    "        return self._x\n",
    "    \n",
    "    def setx(self, value):\n",
    "        print \"setx\"\n",
    "        self._x = value\n",
    "        \n",
    "    def delx(self):\n",
    "        print \"delx\"\n",
    "        del self._x\n",
    "        \n",
    "    x = property(getx, setx, delx, \"I'm the 'x' property.\")\n",
    "    \n",
    "c = C()\n",
    "c.x = 5 # so when we use 'x' variable of a C object, the getters and setters are being called!\n",
    "print c.x\n",
    "del c.x\n",
    "\n",
    "print C.x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 168,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "5\n"
     ]
    }
   ],
   "source": [
    "# the same properties can be used in form of decorators!\n",
    "class M(object):\n",
    "    def __init__(self):\n",
    "        self._x = None\n",
    "\n",
    "    @property\n",
    "    def x(self):\n",
    "        print \"getx\"\n",
    "        return self._x\n",
    "    \n",
    "    @x.setter\n",
    "    def x(self, value):\n",
    "        print \"setx\"\n",
    "        self._x = value\n",
    "    \n",
    "    @x.deleter\n",
    "    def x(self):\n",
    "        print \"delx\"\n",
    "        del self._x\n",
    "\n",
    "m = C()\n",
    "m.x = 5 \n",
    "print m.x\n",
    "del m.x\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So how does this magic happen? how do properties work? It so happens that properties are data discriptors! Discriptors are objects that have a `__get__`, `__set__`, `__del__` method. When accessed as a member variable, the corresponding function gets called. Property is a class that implements this discriptor interface, there is nothing more to it!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 169,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# This is a pure python implementation of property\n",
    "\n",
    "class Property(object):\n",
    "    \"Emulate PyProperty_Type() in Objects/descrobject.c\"\n",
    "\n",
    "    def __init__(self, fget=None, fset=None, fdel=None, doc=None):\n",
    "        self.fget = fget\n",
    "        self.fset = fset\n",
    "        self.fdel = fdel\n",
    "        if doc is None and fget is not None:\n",
    "            doc = fget.__doc__\n",
    "        self.__doc__ = doc\n",
    "\n",
    "    def __get__(self, obj, objtype=None):\n",
    "        if obj is None:\n",
    "            return self\n",
    "        if self.fget is None:\n",
    "            raise AttributeError(\"unreadable attribute\")\n",
    "        return self.fget(obj)\n",
    "\n",
    "    def __set__(self, obj, value):\n",
    "        if self.fset is None:\n",
    "            raise AttributeError(\"can't set attribute\")\n",
    "        self.fset(obj, value)\n",
    "\n",
    "    def __delete__(self, obj):\n",
    "        if self.fdel is None:\n",
    "            raise AttributeError(\"can't delete attribute\")\n",
    "        self.fdel(obj)\n",
    "\n",
    "    def getter(self, fget):\n",
    "        return type(self)(fget, self.fset, self.fdel, self.__doc__)\n",
    "\n",
    "    def setter(self, fset):\n",
    "        return type(self)(self.fget, fset, self.fdel, self.__doc__)\n",
    "\n",
    "    def deleter(self, fdel):\n",
    "        return type(self)(self.fget, self.fset, fdel, self.__doc__)\n",
    "    \n",
    "# during the live session, explain how this maps to the previous decorator version of property."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 8. Metaclasses\n",
    "\n",
    "Meta classes are classes that create new classes (or rather a class whose instances are classes themselves!). They are useful when you want to dynamically create your own types. Eg. when you have to create classes based on a description file (XML) - like in the case of some libraries built over WSDL (PyVmomi). Or in the case when you want to dynamically mix two or more types of classes to create a new one. (eg. a VM type, a OS type and an interface type - used in NuTest framework developed by the automation team). \n",
    "\n",
    "Another reason is to enforce some checking / have restrictions in the user defined classes. Like in the case of Abstract Base Class (ABCMeta). The meta class which creates the user defined class can run some pre checks (whether certain function are defined, etc) and some pre processing (adding new methods, etc) if required!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 175,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "In new\n",
      "Me\n",
      "(<type 'object'>,)\n",
      "{'__module__': '__main__', 'foo': <function foo at 0x10fd0f5f0>, '__metaclass__': <class '__main__.MyMet'>}\n",
      "In init\n",
      "<class '__main__.Me'>\n",
      "('Me', (<type 'object'>,), {'__module__': '__main__', 'foo': <function foo at 0x10fd0f5f0>, '__metaclass__': <class '__main__.MyMet'>})\n",
      "{}\n",
      "I'm foo\n"
     ]
    }
   ],
   "source": [
    "class MyMet(type):\n",
    "    \"\"\"Here we see that MyMet doesn't inherit 'object' but rather 'type' class - the builtin metaclass\n",
    "    \"\"\"\n",
    "    def __new__(cls, name, bases, attrs):\n",
    "        \"\"\"\n",
    "        Args:\n",
    "          name (str) : name of the new class being created\n",
    "          bases (tuple) : tuple of the classes which are the parents of cls\n",
    "          attrs (dict) : the attributes that belong to the class\n",
    "        \"\"\"\n",
    "        print \"In new\"\n",
    "        print name\n",
    "        print bases\n",
    "        print attrs\n",
    "        return super(MyMet, cls).__new__(cls, name, bases, attrs)\n",
    "        \n",
    "    def __init__(self, *args, **kwargs):\n",
    "        print \"In init\"\n",
    "        print self\n",
    "        print args\n",
    "        print kwargs\n",
    "        \n",
    "class Me(object):\n",
    "    __metaclass__ = MyMet\n",
    "    \n",
    "    def foo(self):\n",
    "        print \"I'm foo\"\n",
    "    \n",
    "\n",
    "m = Me()\n",
    "m.foo()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this case we see that 'm' which is an instance of 'Me' works as expected. Here we're using the metaclass to just print out the flow, but we can do much more!\n",
    "\n",
    "Also if you note we see that the args to `__init__` are the same as args to `__new__`, which is again as expected."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 193,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "In new\n",
      "Me\n",
      "(<type 'object'>,)\n",
      "{'__module__': '__main__', '__metaclass__': <class '__main__.MyMet'>}\n",
      "In init\n",
      "<class '__main__.Me'>\n",
      "Me\n",
      "(<type 'object'>,)\n",
      "{'__module__': '__main__', 'foo': <function foo at 0x10fd0ff50>, '__metaclass__': <class '__main__.MyMet'>}\n",
      "\n",
      "-------------------------------\n",
      "\n",
      "<class '__main__.MyMet'>\n",
      "I'm foo\n",
      "I'm bar\n",
      "in test\n"
     ]
    }
   ],
   "source": [
    "class MyMet(type):\n",
    "    \"\"\"Here we see that MyMet doesn't inherit 'object' but rather 'type' class - the builtin metaclass\n",
    "    \"\"\"\n",
    "    def __new__(cls, name, bases, attrs):\n",
    "        \"\"\"\n",
    "        Args:\n",
    "          name (str) : name of the new class being created\n",
    "          bases (tuple) : tuple of the classes which are the parents of cls\n",
    "          attrs (dict) : the attributes that belong to the class\n",
    "        \"\"\"\n",
    "        print \"In new\"\n",
    "        print name\n",
    "        print bases\n",
    "        print attrs\n",
    "        def foo(self):\n",
    "            print \"I'm foo\"\n",
    "        attrs['foo'] = foo\n",
    "        return super(MyMet, cls).__new__(cls, name, bases, attrs)\n",
    "        \n",
    "    def __init__(self, name, bases, attrs):\n",
    "        print \"In init\"\n",
    "        print self # actually the object being created\n",
    "        print name\n",
    "        print bases\n",
    "        print attrs\n",
    "        def bar(self):\n",
    "            print \"I'm bar\"\n",
    "        setattr(self, \"bar\", bar)\n",
    "        \n",
    "    def test(self):\n",
    "        print \"in test\"\n",
    "        \n",
    "    #def __call__(self):\n",
    "    #    print \"self :\", self\n",
    "    # Note : If I override call here, then I have to explicitly call self.__new__\n",
    "    #        otherwise it is completely skipped. Normally a class calls type's __call__\n",
    "    #        which re-routes it to __new__ of the class\n",
    "         \n",
    "        \n",
    "class Me(object):\n",
    "    __metaclass__ = MyMet\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "print \"\\n-------------------------------\\n\"\n",
    "\n",
    "m = Me()\n",
    "print type(Me) # not of type 'type' anymore!\n",
    "m.foo()\n",
    "m.bar()\n",
    "# print m.test --attribute error\n",
    "Me.test()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note**\n",
    "\n",
    "What the `__metaclass__` does is it tells the interpreter to parse the class in question, get the name, the attribute dictionary and the base classes and create it using a 'type' type, in this case, the MyMet class. In it's most primitive form that's how classes are created, using the 'type' inbuilt class. We use this a lot to dynamically mix classes in NuTest!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 198,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "init A\n",
      "<class '__main__.C'>\n",
      "C <type 'type'>\n",
      "<__main__.C object at 0x10fd290d0>\n"
     ]
    }
   ],
   "source": [
    "class A(object):\n",
    "    def __init__(self):\n",
    "        print \"init A\"\n",
    "    def foo(self):\n",
    "        print \"foo A\"\n",
    "    def bar(self):\n",
    "        print \"bar A\"\n",
    "        \n",
    "class B(object):\n",
    "    def __init__(self):\n",
    "        print \"init B\"\n",
    "    def doo(self):\n",
    "        print \"doo B\"\n",
    "    def bar(self):\n",
    "        print \"bar B\"\n",
    "        \n",
    "def test(self):\n",
    "    print \"Self : \", self\n",
    "\n",
    "Cls = type(\"C\", (A,B), {\"test\": test})\n",
    "\n",
    "c = Cls()\n",
    "print Cls\n",
    "print Cls.__name__, type(Cls)\n",
    "print c"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 196,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "foo A\n",
      "bar A\n",
      "doo B\n",
      "Self :  <__main__.C object at 0x10fcff750>\n"
     ]
    }
   ],
   "source": [
    "c.foo()\n",
    "c.bar()\n",
    "c.doo()\n",
    "c.test()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}