Session 0 - IPython Notebooks and Viewing Data in Python.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "![alt text](http://datascience.uci.edu/wp-content/uploads/sites/2/2014/09/data_science_logo_with_image1.png 'UCI_data_science')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preliminaries\n",
    "\n",
    "1.  If you haven't do so already, download and **install the [Anaconda Scientific Python Distribution version 2.7](https://store.continuum.io/cshop/anaconda/)**.  If it offers to make itself your default Python distribution, allow it.\n",
    "1. Whether you've just installed Anaconda, or you have done so previously, you should now **update Anaconda** to the latest version of the distribution.  It changes a lot so do this today even if you did recently.\n",
    " 1. Open a terminal or command prompt.\n",
    " 1. Type ```conda update conda``` and press enter or return.  Confirm that you'd like it to make any changes that it offers.\n",
    " 1. Type ```conda update anaconda``` and press enter or return.  Confirm that you'd like it to make any changes that it offers.\n",
    "1. **Download the code repository**.  \n",
    " 1. Go to [bit.ly/uci_predictive](http://bit.ly/uci_predictive) and click the \"download zip\" button on the right to download a zip file containing this entire repository.\n",
    " 1. Unzip that file into a directory you know how to find; you'll need it several times throughout the day.  \n",
    "1. **Start an ipython notebook server**.\n",
    " 1. Open a terminal and type ```ipython notebook```.  Navigate to the directory where you unzipped this repository.\n",
    " 1. Open \"Test Notebook.ipynb\".\n",
    " 1. Click \"Cell\" at the top of the opened notebook, followed by \"Run All\" and ensure that 1) there are no errors and that 2) the output from the first cell is the same as that in the second.  If it doesn't match, raise your hand.\n",
    " 1. If everything looks good, close the browser tab containing the test notebook but keep open the tab listing all the other notebooks."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Schedule for Today\n",
    "\n",
    "|Start Time | Session |\n",
    "|-----------|---------|\n",
    "|8:30am     | Check In|\n",
    "|9:00am     | **The IPython Notebook and Pandas** |\n",
    "|10:30am    | Coffee & Bagels|\n",
    "|10:45am    | **Linear Regression and Predictive Modeling** |\n",
    "|12:30pm    | Lunch|\n",
    "|1:00pm     | **Out of Sample Prediction** |\n",
    "|2:45pm     | Afternoon break|\n",
    "|3:00pm     | **Logistic Regression** |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Predictive Modeling with Python - _IPython Notebooks and Viewing Data in Python_\n",
    "#### Author: Kevin Bache"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Outline\n",
    "1. IPython and IPython Notebooks\n",
    "1. Numpy\n",
    "1. Pandas"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Python and IPython\n",
    "* `python` is a programming language and also the name of the program that runs scripts written in that language.\n",
    "* If you're running scripts from the command line you can use either `ipython` with something like `ipython my_script.py` or `python` with something like `python my_script.py`\n",
    "* If you're using the command line interpreter interactively to load and explore data, try out a new package, etc. always use `ipython` over `python`.  This is because `ipython` has a bunch of features like tab completion, inline help, and easy access to shell commands which are just plain great (more on these in a bit)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## IPython Notebook\n",
    "* IPython notebook is an interactive front-end to ipython which lets you combine snippets of python code with explanations, images, videos, whatever.  \n",
    "* It's also really convenient for conveying experimental results.\n",
    "* http://nbviewer.ipython.org\n",
    "\n",
    "### <span style=\"color:red\">Self-Driven IPython Notebook Exercise #1</span>\n",
    "1. Start a terminal window and cd to the directory where you stored the course files\n",
    "1. Start the IPython Notebook server with the command `ipython notebook`.  The IPython notebook server runs your python code behind the scenes and renders the output into the notebook\n",
    "1. Create a new notebook by clicking New (top right) >> Python 2 Notebook"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Notebook Concepts\n",
    "* **Cells** -- That grey box is called a cell.  An IPython notebook is nothing but a series of cells.  \n",
    "* **Selecting** -- You can tell if you have a cell selected because it will have a thin, black box around it.\n",
    "* **Running a Cell** -- Running a cell displays its output.  You can run a cell by pressing **`shift + enter`** while it's selected (or click the play button toward the top of the screen). \n",
    "* **Modes** -- There are two different ways of having a cell selected:\n",
    "  * **Command Mode** -- Lets you delete a cell and change its type (more on this in a second).\n",
    "  * **Edit Mode** -- Lets you change the contents of a cell.\n",
    "\n",
    "### Aside: Keyboard Shortcuts That I Use A Lot\n",
    "* (When describing keyboard shortcuts, `+` means 'press at the same time', `,` means 'press after'\n",
    "* **`Enter`** -- Run this cell and make a new one after it\n",
    "* **`Esc`** -- Stop editing this cell\n",
    "* **`Option + Enter`** -- Run this cell and make a new cell after it (Note: this is OSX specific.  Check help >> keyboard shortcuts to find your operating system's version)\n",
    "* **`Shift + Enter`** -- Run this cell and don't make a new one after it\n",
    "* **`Up Arrow`** and **`Down Arrow`**  -- Navigate between cells (must be in command mode)\n",
    "* **`Esc, m, Enter`** -- Convert the current cell to markdown and start editing it again\n",
    "* **`Esc, y, Enter`** -- Convert the current cell to a code cell and start editing it again\n",
    "* **`Esc, d, d`** -- Delete the current cell\n",
    "* **`Esc, a`** -- Create a new cell above the current one\n",
    "* **`Esc, b`** -- Create a new cell below the current one\n",
    "* **`Command + /`** -- Toggle comments in Python code (OSX)\n",
    "* **`Ctrl + /`** -- Toggle comments in Python code (Linux / Windows)\n",
    "\n",
    "### <span style=\"color:red\">Self-Driven IPython Notebook Exercise #2</span>\n",
    "1. Click Help >> User Interface Tour and take the tour\n",
    "1. Click Help >> Keyboard Shortcuts.  Mice are for suckers."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### More Notebook Concepts\n",
    "* **Cell Types** -- There are 3 types of cells: python, markdown, and raw.\n",
    "  * **Python Cells** -- Contain python code. Running a python cell displays its output. Press **`y`** in command mode convert any selected cell into a python cell. All cells start their lives as python cells.\n",
    "  * **Markdown Cells** -- Contain formatted text, lists, links, etc. Press **`m`** in command mode to convert the selected cell into a markdown cell.\n",
    "  * **Raw Cells** -- Useful for a few advanced corner cases.  We won't deal with these at all today.\n",
    "  \n",
    "### <span style=\"color:red\">Self-Driven IPython Notebook Exercise #3</span>:\n",
    "1. Partner up with someone next to you\n",
    "  1. Create a code cell\n",
    "    1. IPython notebooks offer tab completion.  Try typing: ```a<tab>``` \n",
    "    1. IPython notebooks offer inline help.  Try typing: ```?abs``` and then running the cell\n",
    "  1. Create and render a markdown cell which contains \n",
    "    1. Bold text\n",
    "    1. A nested numbered list\n",
    "    1. A working link to UCI's website\n",
    "    1. An image\n",
    "    1. Some rendered LaTex. \n",
    "  1. **Hint**: The following websites might be useful:\n",
    "    1. Check out this [Markdown Cheat Sheet](https://github.com/adam-p/markdown-here/wiki/Markdown-Cheatsheet).  Markdown is a set of simple commands for formatting text to make it pretty.  It isn't specific to IPython Notebooks; it's used all over (for example, a lot of blogging platforms let you write your content in markdown because it's easy and HTML is a pain in the butt).\n",
    "    1. Check out this [stackoverflow post](http://stackoverflow.com/questions/13208286/how-to-write-latex-in-ipython-notebook) about using LaTex in IPython Notebooks."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Numpy\n",
    "Numpy is the main package that you'll use for doing scientific computing in Python.  Numpy provides a multidimensional array datatype called `ndarray` which can do things like vector and matrix computations.\n",
    "\n",
    "### Resources:\n",
    "* [Official Numpy Tutorial](http://wiki.scipy.org/Tentative_NumPy_Tutorial)\n",
    "* [Numpy, R, Matlab Cheat Sheet](http://mathesaurus.sourceforge.net/matlab-python-xref.pdf)\n",
    "* [Another Numpy, R, Matlab Cheat Sheet](http://sebastianraschka.com/Articles/2014_matrix_cheatsheet_table.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# you don't have to rename numpy to np but it's customary to do so\n",
    "import numpy as np\n",
    "\n",
    "# you can create a 1-d array with a list of numbers\n",
    "a = np.array([1, 4, 6])\n",
    "print 'a:'\n",
    "print a\n",
    "print 'a.shape:', a.shape\n",
    "print \n",
    "\n",
    "# you can create a 2-d array with a list of lists of numbers\n",
    "b = np.array([[6, 7], [3, 1], [4, 0]])\n",
    "print 'b:'\n",
    "print b\n",
    "print 'b.shape:', b.shape\n",
    "print"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# you can create an array of ones\n",
    "print 'np.ones(3, 4):'\n",
    "print np.ones((3, 4))\n",
    "print\n",
    "\n",
    "# you can create an array of zeros\n",
    "print 'np.zeros(2, 5):'\n",
    "print np.zeros((2, 5))\n",
    "print\n",
    "\n",
    "# you can create an array which of a range of numbers and reshape it\n",
    "print 'np.arange(6):'\n",
    "print np.arange(6)\n",
    "print \n",
    "print 'np.arange(6).reshape(2, 3):'\n",
    "print np.arange(6).reshape(2, 3)\n",
    "print\n",
    "\n",
    "# you can take the transpose of a matrix with .transpose or .T\n",
    "print 'b and b.T:'\n",
    "print b\n",
    "print \n",
    "print b.T\n",
    "print "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# you can iterate over rows\n",
    "i = 0\n",
    "for this_row in b:\n",
    "    print 'row', i, ': ', this_row\n",
    "    i += 1 \n",
    "print \n",
    "    \n",
    "# you can access sections of an array with slices\n",
    "print 'first two rows of the first column of b:'\n",
    "print b[:2, 0]\n",
    "print"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# you can concatenate arrays in various ways:\n",
    "print 'np.hstack([b, b]):'\n",
    "print np.hstack([b, b])\n",
    "print\n",
    "\n",
    "print 'np.vstack([b, b]):'\n",
    "print np.vstack([b, b])\n",
    "print"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# note that you get an error if you pass in print 'np.hstack(b, b):'\n",
    "print np.hstack(b, b)\n",
    "print\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# you can perform matrix multiplication with np.dot()\n",
    "c = np.dot(a, b)\n",
    "print 'c = np.dot(a, b):'\n",
    "print c\n",
    "print\n",
    "\n",
    "# if a is already a numpy array, then you can also use this chained \n",
    "# matrix multiplication notation.  use whichever looks cleaner in \n",
    "# context\n",
    "print 'a.dot(b):'\n",
    "print a.dot(b)\n",
    "print\n",
    "\n",
    "\n",
    "# you can perform element-wise multiplication with * \n",
    "d = b * b\n",
    "print 'd = b * b:'\n",
    "print d\n",
    "print\n",
    "\n",
    "a.dot(b)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Arrays and Matrices\n",
    "In addition to arrays which can have any number of dimensions, Numpy also has a `matrix` data type which always has exactly 2.  **DO NOT USE `matrix`**.  \n",
    "\n",
    "The original intention behind this data type was to make Numpy feel a bit more like Matlab, mainly by making the `*` operator perform matrix multiplication so you don't have to use `np.dot`.  But `matrix` isn't as well developed by the Numpy people as `array` is.  `matrix` is slower and using it will sometimes throw errors in other people's code because everyone expects you to use `array`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# you can convert a 1-d array to a 2-d array with np.newaxis\n",
    "print 'a:'\n",
    "print a\n",
    "print 'a.shape:', a.shape\n",
    "print \n",
    "print 'a[np.newaxis] is a 2-d row vector:'\n",
    "print a[np.newaxis]\n",
    "print 'a[np.newaxis].shape:', a[np.newaxis].shape\n",
    "print\n",
    "\n",
    "print 'a[np.newaxis].T: is a 2-d column vector:'\n",
    "print a[np.newaxis].T\n",
    "print 'a[np.newaxis].T.shape:', a[np.newaxis].T.shape\n",
    "print\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# numpy provides a ton of other functions for working with matrices\n",
    "m = np.array([[1, 2],[3, 4]])\n",
    "m_inverse = np.linalg.inv(m)\n",
    "print 'inverse of [[1, 2],[3, 4]]:'\n",
    "print m_inverse\n",
    "print\n",
    "\n",
    "print 'm.dot(m_inverse):'\n",
    "print m.dot(m_inverse)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# and for doing all kinds of sciency type stuff.  like generating random numbers:\n",
    "np.random.seed(5678)\n",
    "n = np.random.randn(3, 4)\n",
    "print 'a matrix with random entries drawn from a Normal(0, 1) distribution:'\n",
    "print n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <span style=\"color:red\">Self-Driven Numpy Exercise</span>\n",
    "1. In the cell below, add a column of ones to the matrix `X_no_constant`.  This is a common task in linear regression and general linear modeling and something that you'll have to be able to do later today.  \n",
    "1. Multiply your new matrix by the `betas` vector below to make a vector called `y`\n",
    "1. You'll know you've got it when the cell prints '\\*\\*\\*\\*\\*\\* Tests passed! \\*\\*\\*\\*\\*\\*' at the bottom.\n",
    "\n",
    "Specificically, given a matrix:\n",
    "\n",
    "\\begin{equation*}\n",
    "\\qquad\n",
    "\\mathbf{X_{NoConstant}} = \n",
    "\\left( \\begin{array}{ccc}\n",
    "x_{1,1} & x_{1,2} & \\dots & x_{1,D} \\\\\n",
    "x_{2,1} & x_{2,2} & \\dots & x_{2,D} \\\\\n",
    "\\vdots & \\vdots & \\ddots & \\vdots \\\\\n",
    "x_{i,1} & x_{i,2} & \\dots & x_{i,D} \\\\\n",
    "\\vdots & \\vdots & \\ddots & \\vdots \\\\\n",
    "x_{N,1} & x_{N,2} & \\dots & x_{N,D} \\\\\n",
    "\\end{array} \\right)\n",
    "\\qquad\n",
    "\\end{equation*}\n",
    "\n",
    "We want to convert it to:\n",
    "\\begin{equation*}\n",
    "\\qquad\n",
    "\\mathbf{X} = \n",
    "\\left( \\begin{array}{ccc}\n",
    "1 & x_{1,1} & x_{1,2} & \\dots & x_{1,D} \\\\\n",
    "1 & x_{2,1} & x_{2,2} & \\dots & x_{2,D} \\\\\n",
    "\\vdots & \\vdots & \\vdots & \\ddots & \\vdots \\\\\n",
    "1 & x_{i,1} & x_{i,2} & \\dots & x_{i,D} \\\\\n",
    "\\vdots & \\vdots & \\ddots & \\vdots \\\\\n",
    "1 & x_{N,1} & x_{N,2} & \\dots & x_{N,D} \\\\\n",
    "\\end{array} \\right)\n",
    "\\qquad\n",
    "\\end{equation*}\n",
    "\n",
    "So that if we have a vector of regression coefficients like this:\n",
    "\n",
    "\\begin{equation*}\n",
    "\\qquad\n",
    "\\beta = \\left( \\begin{array}{ccc}\n",
    "\\beta_0 \\\\\n",
    "\\beta_1 \\\\\n",
    "\\vdots \\\\\n",
    "\\beta_j \\\\\n",
    "\\vdots \\\\\n",
    "\\beta_D\n",
    "\\end{array} \\right)\n",
    "\\end{equation*}\n",
    "\n",
    "We can do this:\n",
    "\n",
    "\\begin{equation*}\n",
    "\\mathbf{y} \\equiv \\mathbf{X} \\mathbf{\\beta} \n",
    "\\end{equation*}\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "np.random.seed(3333)\n",
    "n_data = 10 # number of data points. i.e. N\n",
    "n_dim = 5   # number of dimensions of each datapoint.  i.e. D\n",
    "\n",
    "betas = np.random.randn(n_dim + 1)\n",
    "\n",
    "X_no_constant = np.random.randn(n_data, n_dim)\n",
    "print 'X_no_constant:'\n",
    "print X_no_constant\n",
    "print \n",
    "\n",
    "# INSERT YOUR CODE HERE!\n",
    "\n",
    "# Tests:\n",
    "y_expected = np.array([-0.41518357, -9.34696153, 5.08980544, \n",
    "                       -0.26983873, -1.47667864, 1.96580794, \n",
    "                       6.87009791, -2.07784135, -0.7726816, \n",
    "                       -2.74954984])\n",
    "np.testing.assert_allclose(y, y_expected)\n",
    "print '****** Tests passed! ******'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Pandas\n",
    "Pandas is a python package which adds some useful data analysis features to numpy arrays.  Most importantly, it contains a `DataFrame` data type like the r `dataframe`: a set of named columns organized into something like a 2d array.  Pandas is great.\n",
    "\n",
    "### Resources:\n",
    "* [10 Minutes to Pandas](http://pandas.pydata.org/pandas-docs/dev/10min.html)\n",
    "* [Pandas Data Structures Tutorial](http://pandas.pydata.org/pandas-docs/stable/dsintro.html)\n",
    "* [Merge, Join, Concatenate Tutorial](http://pandas.pydata.org/pandas-docs/dev/merging.html)\n",
    "* [Another Numpy/Pandas Tutorial](https://www.kaggle.com/c/titanic/details/getting-started-with-python-ii)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# like with numpy, you don't have to rename pandas to pd, but it's customary to do so\n",
    "import pandas as pd\n",
    "\n",
    "b = np.array([[6, 7], [3, 1], [4, 0]])\n",
    "df = pd.DataFrame(data=b,  columns=['Weight', 'Height'])\n",
    "print 'b:'\n",
    "print b\n",
    "print \n",
    "print 'DataFame version of b:'\n",
    "print df\n",
    "print"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Pandas can save and load CSV files.  \n",
    "# Python can do this too, but with Pandas, you get a DataFrame \n",
    "# at the end which understands things like column headings\n",
    "baseball = pd.read_csv('data/baseball.dat.txt')\n",
    "\n",
    "# A Dataframe's .head() method shows its first 5 rows\n",
    "baseball.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# you can see all the column names\n",
    "print 'baseball.keys():'\n",
    "print baseball.keys()\n",
    "print\n",
    "\n",
    "print 'baseball.Salary:'\n",
    "print baseball.Salary\n",
    "print \n",
    "print \"baseball['Salary']:\"\n",
    "print baseball['Salary']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "baseball.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "baseball.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "baseball"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# You can perform queries on your data frame.  \n",
    "# This statement gives you a True/False vector telling you \n",
    "# whether the player in each row has a salary over $1 Million\n",
    "millionaire_indices = baseball['Salary'] > 1000\n",
    "print millionaire_indices"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# you can use the query indices to look at a subset of your original dataframe\n",
    "print 'baseball.shape:', baseball.shape\n",
    "print \"baseball[millionaire_indices].shape:\", baseball[millionaire_indices].shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# you can look at a subset of rows and columns at the same time\n",
    "print \"baseball[millionaire_indices][['Salary', 'AVG', 'Runs', 'Name']]:\"\n",
    "baseball[millionaire_indices][['Salary', 'AVG', 'Runs', 'Name']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pandas Joins - If you have time\n",
    "The real magic with a Pandas DataFrame comes from the merge method which can match up the rows and columns from two DataFrames and combine their data.  Let's load another file which has shoesize for just a few players"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# load shoe size data\n",
    "shoe_size_df = pd.read_csv('data/baseball2.dat.txt')\n",
    "shoe_size_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "merged = pd.merge(baseball, shoe_size_df, on=['Name'])\n",
    "merged"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "merged_outer = pd.merge(baseball, shoe_size_df, on=['Name'], how='outer')\n",
    "merged_outer.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### <span style=\"color:red\">Self-Driven Pandas Exercise</span>\n",
    "1. Partner up with someone next to you.  Then, on one of your computers:\n",
    "  1. Prepend a column of ones to the dataframe `X_df` below.  Name the new column 'const'.\n",
    "  1. Again, matrix multiply `X_df` by the `betas` vector and assign the result to an new variable: `y_new`\n",
    "  1. You'll know you've got it when the cell prints '\\*\\*\\*\\*\\*\\* Tests passed! \\*\\*\\*\\*\\*\\*' at the bottom.\n",
    "\n",
    "  **Hint**: This stackoverflow post may be useful: http://stackoverflow.com/questions/13148429/how-to-change-the-order-of-dataframe-columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "np.random.seed(3333)\n",
    "n_data = 10 # number of data points. i.e. N\n",
    "n_dim = 5   # number of dimensions of each datapoint.  i.e. D\n",
    "\n",
    "betas = np.random.randn(n_dim + 1)\n",
    "\n",
    "X_df = pd.DataFrame(data=np.random.randn(n_data, n_dim))\n",
    "\n",
    "# INSERT YOUR CODE HERE!\n",
    "\n",
    "# Tests:\n",
    "assert 'const' in X_df.keys(), 'The new column must be called \"const\"'\n",
    "assert np.all(X_df.shape == (n_data, n_dim+1))\n",
    "assert len(y_new == n_data)\n",
    "print '****** Tests passed! ******'"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Raw Cell Format",
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}