2_Intro to numpy.ipynb

{
  "cells": [
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "# Section 2: Introduction to NumPy\n\nNumPy is one of the two most important libraries in Python for data science, along with pandas (which we will cover in the next section). NumPy is a crucial library for effectively loading, storing, and manipulating in-memory data in Python, all of which will be at the heart of what you do with data science in Python.\n\nDatasets come from a huge range of sources and in a wide range of formats, such as text documents, images, sound clips, numerical measurements, and nearly anything else. Despite this variety, however, the start of data science is to think of all data fundamentally as arrays of numbers.\n\nFor example, the words in documents can be represented as the numbers that encode letters in computers or even the frequency of particular words in a collection of documents. Digital images can be thought of as two-dimensional arrays of numbers representing pixel brightness or color. Sound files can be represented as one-dimensional arrays of frequency versus time. However, no matter what form our data takes, in order to analyze it, our first step will be to transform it into arrays of numbers—which is where NumPy comes in (and pandas down the road).\n\nNumPy is short for *Numerical Python*, and it provides an efficient means of storing and operating on dense data buffers in Python. Array-oriented computing in Python goes back to 1995 with the Numeric library. Scientific programming in Python took off over the next 10 years, but the collections of libraries splintered. The NumPy project began in 2005 as a means of bringing the Numeric and NumArray projects together around a single array-based framework.\n\nLet's get started exploring NumPy! Our first step will be to import NumPy using `np` as an alias:"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "import numpy as np",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "Get used to this convention — it's a common convention in Python, and it's the way we will use and refer to NumPy throughout the rest of this course."
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "## Built-In Help\n\nThere's a lot to learn about NumPy, and it can be tough to remember it all the first time through. Don't worry! IPython — the underlying program that enables notebooks like this one to interact with Python—has you covered.\n\nFirst off, IPython gives you the ability to quickly explore the contents of a package like NumPy by using the tab-completion feature. So, if you want to see all of the functions available with NumPy, type this:\n\n```ipython\nIn [ ]: np.<TAB>\n```\nWhen you do so, a drop-down menu will appear next to the `np.`\n\n> **Exercise**"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "# Place your cursor after the period and press <TAB>:\nnp.",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "From the drop-down menu, you can select any function to run. Better still, you can select any function and view the built-in help documentation for it. For example, to view the documentation for the NumPy `add()` function, you can run this code:\n\n```ipython\nIn [ ]: np.add?\n```\nTry this with a few different functions. Remember, these functions are just like the ones you wrote in the previous section; the documentation will help explain what parameters you can (or should) provide to the function, in addition to what output you can expect.\n\n> **Exercise**"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "# Replace 'add' below with a few different NumPy function names and look over the documentation:\nnp.add?",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "For more detailed documentation (along with additional tutorials and other resources), visit [www.numpy.org](http://www.numpy.org).\n\nNow that you know how to quickly get help while you are working on your own, let's return to storing data in arrays."
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "## NumPy arrays: a specialized data structure for analysis\n\n> **Learning goal:** By the end of this subsection, you should have a basic understanding of what NumPy arrays are and how they differ from the other Python data structures you have studied thus far.\n\nWe started the discussion in this section by noting that data science starts by representing data as arrays of numbers.\n\n\"Wait!\" you might be thinking. \"Can't we just use Python lists for that?\"\n\nDepending on the data, yes, you could (and you will use lists as a part of working with data in Python). But to see what we might want to use a specialized data structure for, let's look a little more closely at lists."
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "### Lists in Python\n\nPython lists can hold just one kind of object. Let's use one to create a list of just integers:"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "myList = list(range(10))\nmyList",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "Remember list comprehension? We can use it to probe the data types of items in a list:"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "[type(item) for item in myList]",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "Of course, a really handy feature of Python lists is that they can hold heterogeneous types of data in a single list object:"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "myList2 = [True, \"2\", 3.0, 4]\n[type(item) for item in myList2]",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "However, this flexibility comes at a price. Each item in a list is really a separate Python object (the list is an object itself, true, but mostly it is an object that serves as a container for the memory pointers to the constituent objects). That means that each item in a list must contain its own type info, reference count, and other information. All of this information can become expensive in terms of memory and performance if we are dealing with hundreds of thousands or millions of items in a list. Moreover, for many uses in data science, our arrays just store a single type of data (such as integers or floats), which means that all of the object-related information for items in such an array would be redundant. It can be much more efficient to store data in a fixed-type array.\n\n<img align=\"left\" style=\"padding-right:10px;\" src=\"Graphics/array_vs_list.png\">"
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "### Fixed-type arrays in Python\n\nAt the level of implementation by the computer, the `ndarray` that is part of the NumPy package contains a single pointer to one contiguous block of data. This is efficient memory-wise and computationally. Better still, NumPy provides efficient *operations* on data stored in `ndarray` objects.\n\n(Note that we will pretty much use “array,” “NumPy array,” and “ndarray” interchangeably throughout this section to refer to the ndarray object.)"
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "#### Creating NumPy arrays method 1: using Python lists\n\nThere are multiple ways to create arrays in NumPy. Let's start by using our good old familiar Python lists. We will use the `np.array()` function to do this (remember, we imported NumPy as '`np`'):"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "# Create an integer array:\nnp.array([1, 4, 2, 5, 3])",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "Remember that, unlike Python lists, NumPy constrains arrays to contain a single type. So, if data types fed into a NumPy array do not match, NumPy will attempt to *upcast* them if possible. To see what we mean, here NumPy upcasts integers to floats:"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "np.array([3.14, 4, 2, 3])",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "> **Exercise**"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "# What happens if you construct an array using a list that contains a combination of integers, floats, and strings?\n",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "If you want to explicitly set the data type of your array when you create it, you can use the `dtype` keyword:"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "np.array([1, 2, 3, 4], dtype='float32')",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "> **Exercise**"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "# Try this using a different dtype.\n# Remember that you can always refer to the documentation with the command np.array.\n",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "Most usefully for a lot of applications in data science, NumPy arrays can explicitly be multidimensional (like matrices or tensors). Here's one way of creating a multidimensional array using a list of lists:"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "# nested lists result in multi-dimensional arrays\nnp.array([range(i, i + 3) for i in [2, 4, 6]])",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "The inner lists in a list of lists are treated as rows of the two-dimensional array you created."
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "#### Creating NumPy arrays method 2: building from scratch\n\nIn practice, it is often more efficient to create arrays from scratch using functions built into NumPy, particularly for larger arrays. Here are a few examples; these example will help introduce you to several useful NumPy functions."
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "# Create an integer array of length 10 filled with zeros\nnp.zeros(10, dtype=int)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "# Create a 3x5 floating-point array filled with ones\nnp.ones((3, 5), dtype=float)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "# Create a 3x5 array filled with 3.14\n# The first number in the tuple gives the number of rows\n# The second number in the tuple sets the number of columns\nnp.full((3, 5), 3.14)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "# Create an array filled with a linear sequence\n# Starting at 0, ending at 20, stepping by 2\n# (this is similar to the built-in Python range() function)\nnp.arange(0, 20, 2)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "# Create an array of five values evenly spaced between 0 and 1\nnp.linspace(0, 1, 5)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "# Create a 3x3 array of uniformly distributed\n# random values between 0 and 1\nnp.random.random((3, 3))",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "# Create a 3x3 array of normally distributed random values\n# with mean 0 and standard deviation 1\nnp.random.normal(0, 1, (3, 3))",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "# Create a 3x3 array of random integers in the interval [0, 10)\nnp.random.randint(0, 10, (3, 3))",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "# Create a 3x3 identity matrix\nnp.eye(3)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "# Create an uninitialized array of three integers\n# The values will be whatever happens to already exist at that memory location\nnp.empty(3)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "Now take a couple of minutes to go back and play with these code snippets, changing the parameters. These functions are the bread-and-butter of creating NumPy arrays and you will want to become comfortable with them."
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "Below is a table listing out several of the array-creation functions in NumPy.\n\n| Function      | Description |\n|:--------------|:------------|\n| `array`       | Converts input data (list, tuple, array, or other sequence type) to an ndarray either |\n|               | by inferring a dtype or explicitly specifying a dtype. Copies the input data by default. |\n| `asarray`     | Converts input to ndarray, but does not copy if the input is already an ndarray. |\n| `arange`      | Similar to the built-in `range()` function but returns an ndarray instead of a list. |\n| `ones`, `ones_like` | Produces an array of all 1s with the given shape and dtype. |\n|               | `ones_like` takes another array and produces a ones-array of the same shape and dtype. |\n| `zeros`, `zeros_like` | Similar to `ones` and `ones_like` but producing arrays of 0s instead. |\n| `empty`, `empty_like` | Creates new arrays by allocating new memory, but does not populate with any values \n|               | like `ones` and `zeros`. |\n| `full`, `full_like` | Produces an array of the given shape and dtype with all values set to the indicated “fill value.” |\n|               | `full_like` takes another array and produces a a filled array of the same shape and dtype. |\n| `eye`, `identity` | Create a square $N \\times N$ identity matrix (1s on the diagonal and 0s elsewhere) |"
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "### NumPy data types\n\nThe standard NumPy data types are listed in the following table. Note that when constructing an array, they can be specified using a string:\n\n```python\nnp.zeros(8, dtype='int16')\n```\n\nOr they can be specified directly using the NumPy object:\n\n```python\nnp.zeros(8, dtype=np.int16)\n```"
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "| Data type\t    | Description |\n|:--------------|:------------|\n| ``bool_``     | Boolean (True or False) stored as a byte |\n| ``int_``      | Default integer type (same as C ``long``; normally either ``int64`` or ``int32``)| \n| ``intc``      | Identical to C ``int`` (normally ``int32`` or ``int64``)| \n| ``intp``      | Integer used for indexing (same as C ``ssize_t``; normally either ``int32`` or ``int64``)| \n| ``int8``      | Byte (-128 to 127)| \n| ``int16``     | Integer (-32768 to 32767)|\n| ``int32``     | Integer (-2147483648 to 2147483647)|\n| ``int64``     | Integer (-9223372036854775808 to 9223372036854775807)| \n| ``uint8``     | Unsigned integer (0 to 255)| \n| ``uint16``    | Unsigned integer (0 to 65535)| \n| ``uint32``    | Unsigned integer (0 to 4294967295)| \n| ``uint64``    | Unsigned integer (0 to 18446744073709551615)| \n| ``float_``    | Shorthand for ``float64``.| \n| ``float16``   | Half-precision float: sign bit, 5 bits exponent, 10 bits mantissa| \n| ``float32``   | Single-precision float: sign bit, 8 bits exponent, 23 bits mantissa| \n| ``float64``   | Double-precision float: sign bit, 11 bits exponent, 52 bits mantissa| \n| ``complex_``  | Shorthand for ``complex128``.| \n| ``complex64`` | Complex number, represented by two 32-bit floats| \n| ``complex128``| Complex number, represented by two 64-bit floats| "
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "If these data types seem a lot like those in C, that's because NumPy is built in C.\n\n> **Takeaway:** NumPy arrays are a data structure similar to Python lists that provide high performance when storing and working on large amounts of homogeneous data—precisely the kind of data that you will encounter frequently in doing data science. NumPy arrays support many data types beyond those discussed in this course. With all of that said, however, don’t worry about memorizing all of the NumPy dtypes. **It’s often just necessary to care about the general kind of data you’re dealing with: floating point, integer, Boolean, string, or general Python object.**"
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "## Working with NumPy arrays: the basics\n\n> **Learning goal:** By the end of this subsection, you should be comfortable working with NumPy arrays in basic ways.\n\nNow that you know how to create arrays in NumPy, you need to get comfortable manipulating them for two reasons. First, you will work with NumPy arrays as part of your exploration of data science. Second, our other important Python data-science tool, pandas, is actually built around NumPy. Getting good at working with NumPy arrays will pay dividends in the next section and beyond: NumPy arrays are the building blocks for the `Series` and `DataFrame` data structures in the Python pandas library and you will use them *a lot* in data science. To get comfortable with array manipulation, we will cover five specifics:\n- **Arrays attributes**: Assessing the size, shape, and data types of arrays\n- **Indexing arrays**: Getting and setting the value of individual array elements\n- **Slicing arrays**: Getting and setting smaller subarrays within a larger array\n- **Reshaping arrays**: Changing the shape of a given array\n- **Joining and splitting arrays**: Combining multiple arrays into one and splitting one array into multiple arrays"
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "### Array attributes\nFirst, let's look at some array attributes. We'll start by defining three arrays filled with random numbers: one one-dimensional, another two-dimensional, and the last three-dimensional. Because we will be using NumPy's random number generator, we will set a *seed* value in order to ensure that you get the same random arrays each time you run this code:"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "import numpy as np\nnp.random.seed(0)  # seed for reproducibility\n\na1 = np.random.randint(10, size=6)  # One-dimensional array\na2 = np.random.randint(10, size=(3, 4))  # Two-dimensional array\na3 = np.random.randint(10, size=(3, 4, 5))  # Three-dimensional array",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "Each array has attributes ``ndim`` (the number of dimensions of an array), ``shape`` (the size of each dimension of an array), and ``size`` (the total number of elements in an array).\n\n> **Exercise:**"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "# Change the values in this code snippet to look at the attributes for a1, a2, and a3:\nprint(\"a3 ndim: \", a3.ndim)\nprint(\"a3 shape:\", a3.shape)\nprint(\"a3 size: \", a3.size)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "Another useful array attribute is the `dtype`, which we already encountered earlier in this section as a means of determining the type of data in an array:"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "print(\"dtype:\", a3.dtype)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "> **Exercise:**"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "# Explore the dtype for the other arrays.\n# What dtypes do you predict them to have?\nprint(\"dtype:\", a3.dtype)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "### Indexing arrays\n\nIndexing in NumPy is pretty similar to indexing lists in standard Python. In fact, indices in one-dimensional arrays work exactly as they do with Python lists:"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "a1",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "a1[0]",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "a1[4]",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "As with regular Python lists, in order to index from the end of the array, you can use negative indices:"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "a1[-1]",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "a1[-2]",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "> **Exercise:**"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "# Do multidimensional NumPy arrays work like Python lists of lists?\n# Try a few combinations like a2[1][1] or a3[0][2][1] and see what comes back\n",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "You might have noticed that we can treat multidimensional arrays like lists of lists. But a more common means of accessing items in multidimensional arrays is to use a comma-separated tuple of indices.\n\n(Yes, we realize that these comma-separated tuples use square brackets rather than the parentheses the name might suggest, but they are nevertheless referred to as tuples.)"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "a2",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "a2[0, 0]",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "a2[2, 0]",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "a2[2, -1]",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "You can also modify values by use of this same comma-separated index notation:"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "a2[0, 0] = 12\na2",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "Remember, once defined, NumPy arrays have a fixed data type. So, if you attempt to insert a float into an integer array, the value will be silently truncated."
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "a1[0] = 3.14159\na1",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "> **Exercise:**"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "# What happens if you try to insert a string into a1?\n# Hint: try both a string like '3' and one like 'three'\n",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "### Slicing arrays\nSimilar to how you can use square brackets to access individual array elements, you can also use them to access subarrays. You do this with the *slice* notation, marked by the colon (`:`) character. NumPy slicing syntax follows that of the standard Python list; so, to access a slice of an array `a`, use this notation:\n``` python\na[start:stop:step]\n```\nIf any of these are unspecified, they default to the values ``start=0``, ``stop=``*``size of dimension``*, ``step=1``.\nLet's take a look at accessing sub-arrays in one dimension and in multiple dimensions."
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "#### One-dimensional slices"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "a = np.arange(10)\na",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "a[:5]  # first five elements",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "a[5:]  # elements after index 5",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "a[4:7]  # middle sub-array",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "a[::2]  # every other element",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "a[1::2]  # every other element, starting at index 1",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "> **Exercise:**"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "# How would you access the *last* five elements of array a?\n# How about every other element of the last five elements of a?\n# Hint: Think back to list indexing in Python\n",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "Be careful when using negative values for ``step``. When ``step`` has a negative value, the defaults for ``start`` and ``stop`` are swapped and you can use this functionality to reverse an array:"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "a[::-1]  # all elements, reversed",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "a[5::-2]  # reversed every other from index 5",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "> **Exercise:**"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "# How can you create a slice that contains every third element of a\n# descending from the second-to-last element to the second element of a?\n",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "#### Multidimensional slices\n\nMultidimensional slices use the same slice notation of one-dimensional subarrays mixed with the comma-separated notation of multidimensional arrays. Some examples will help illustrate this."
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "a2",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "a2[:2, :3]  # two rows, three columns",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "a2[:3, ::2]  # all rows, every other column",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "Finally, subarray dimensions can even be reversed together:"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "a2[::-1, ::-1]",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "> **Exercise:**"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "# Now try to show 2 rows and 4 columns with every other element?\n",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "#### Accessing array rows and columns\nOne thing you will often need to do in manipulating data is accessing a single row or column in an array. You can do this through a combination of indexing and slicing; specifically by using an empty slice marked by a single colon (``:``). Again, some examples will help illustrate this."
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "print(a2[:, 0])  # first column of x2",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "print(a2[0, :])  # first row of x2",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "In the case of row access, the empty slice can be omitted for a more compact syntax:"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "print(a2[0])  # equivalent to a2[0, :]",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "> **Exercise:**"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "# How would you access the third column of a3?\n# How about the third row of a3?\n",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "#### Slices are no-copy views\nIt's important to know that slicing produces *views* of array data, not *copies*. This is a **huge** difference between NumPy array slicing and Python list slicing. With Python lists, slices are only shallow copies of lists; if you modify a copy, it doesn't affect the parent list. When you modify a NumPy subarray, you modify the original list. Be careful: this can have ramifications when you are trying to just work with a small part of a large dataset and you don’t want to change the whole thing. Let's look more closely."
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "print(a2)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "Extract a $2 \\times 2$ subarray from `a2`:"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "a2_sub = a2[:2, :2]\nprint(a2_sub)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "Now modify this subarray:"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "a2_sub[0, 0] = 99\nprint(a2_sub)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "`a2` is now modified as well:"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "print(a2)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "> **Exercise:**"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "# Now try reversing the column and row order of a2_sub\n# Does a2 look the way you expected it would after that manipulation?\n",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "The fact that slicing produces views rather than copies is useful for data-science work. As you work with large datasets, you will often find that it is easier to access and manipulate pieces of those datasets rather than copying them entirely."
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "#### Copying arrays\nInstead of just creating views, sometimes it is necessary to copy the data in one array to another. When you need to do this, use the `copy()` method:"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "a2_sub_copy = a2[:2, :2].copy()\nprint(a2_sub_copy)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "If we now modify this subarray, the original array is not touched:"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "a2_sub_copy[0, 0] = 42\nprint(a2_sub_copy)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "print(a2)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "### Reshaping arrays\nAnother way in which you will need to manipulate arrays is by reshaping them. This involves changing the number and size of dimensions of an array. This kind of manipulation can be important in getting your data to meet the expectations of machine learning programs or APIs.\n\nThe most flexible way of doing this kind of manipulation is with the `reshape` method. For example, if you want to put the numbers 1 through 9 in a $3 \\times 3$ grid, you can do the following:"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "grid = np.arange(1, 10).reshape((3, 3))\nprint(grid)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "Another common manipulation you will do in data science is converting one-dimensional arrays into two-dimensional row or column matrices. This can be a common necessity when doing linear algebra for machine learning. While you can do this by means of the `reshape` method, an easier way is to use the `newaxis` keyword in a slice operation:"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "a = np.array([1, 2, 3])\n\n# row vector via reshape\na.reshape((1, 3))",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "# row vector via newaxis\na[np.newaxis, :]",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "# column vector via reshape\na.reshape((3, 1))",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "# column vector via newaxis\na[:, np.newaxis]",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "You will see this type of transformation a lot in the remainder of this course."
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "### Joining and splitting arrays\n\nAnother common data-manipulation need in data science is combining multiple datasets; learning first how to do this with NumPy arrays will help you in the next section when we do this with more complex data structures. You will many times also need to split a single array into multiple arrays."
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "#### Joining arrays\nTo join arrays in NumPy, you will most often use `np.concatenate`, which is the method we will cover here. If you find yourself in the future needing to specifically join arrays in mixed dimensions (a rarer case), read the documentation on `np.vstack`, `np.hstack`, and `np.dstack`."
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "##### `np.concatenate()`\n\n`np.concatenate` takes a tuple or list of arrays as its first argument:"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "a = np.array([1, 2, 3])\nb = np.array([3, 2, 1])\nnp.concatenate([a, b])",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "You can also concatenate more than two arrays at once:"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "c = [99, 99, 99]\nprint(np.concatenate([a, b, c]))",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "`np.concatenate` can also be used for two-dimensional arrays:"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "grid = np.array([[1, 2, 3],\n                 [4, 5, 6]])",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "# concatenate along the first axis, which is the default\nnp.concatenate([grid, grid])",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "> **Exercise:**"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "# Recall that axes are zero-indexed in NumPy.\n# What do you predict np.concatenate([grid, grid], axis=1) will produce?\n",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "#### Splitting arrays\nIn order to split arrays into multiple smaller arrays, you can use the functions ``np.split``, ``np.hsplit``, ``np.vsplit``, and ``np.dsplit``.  As above, we will only cover the most commonly used function (`np.split`) in this course."
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "##### `np.split()`\nLet's first examine the case of a one-dimensional array:"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "a = [1, 2, 3, 99, 99, 3, 2, 1]\na1, a2, a3 = np.split(a, [3, 5])\nprint(a1, a2, a3)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "Notice that *N* split-points produces to *N + 1* subarrays. In this case it has formed the subarray `a2` with `a[3]` and `a[4]` (the element just before position 5 [remember how Python indexing goes], the second input in the tuple) as elements. `a1` and `a3` pick up the leftover portions from the original array `a`."
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "> **Exercise:**"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "grid = np.arange(16).reshape((4, 4))\ngrid",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "# What does np.split(grid, [1, 2]) produce?\n# What about np.split(grid, [1, 2], axis=1)?\n",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "> **Takeaway:** Manipulating datasets is a fundamental part of preparing data for analysis. The skills you learned and practiced here will form building blocks for the most sophisticated data-manipulation you will learn in later sections in this course."
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "## Fancy indexing\n\nSo far, we have explored how to access and modify portions of arrays using simple indices like `arr[0]`) and slices like `arr[:5]`. Now it is time for fancy indexing, in which we pass an array of indices to an array in order to access or modify multiple array elements at the same time.\n\nLet's try it out:"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "rand = np.random.RandomState(42)\n\narr = rand.randint(100, size=10)\nprint(arr)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "Suppose you need to access three different elements. Using the tools you currently have, your code might look something like this:"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "[arr[3], arr[7], arr[2]]",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "With fancy indexing, you can pass a single list or array of indices to do the same thing:"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "ind = [3, 7, 4]\narr[ind]",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "Another useful aspect of fancy indexing is that the shape of the output array reflects the shape of the *index arrays* you supply, rather than the shape of the array you are accessing. This is handy because there will be many times in a data scientist's life when they want to grab data from an array in a particular manner, such as to pass it to a machine learning API. Let's examine this property with an example:"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "ind = np.array([[3, 7],\n                [4, 5]])\narr[ind]",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "`arr` is a one-dimensional array, but `ind`, your index array, is a $2 \\times 2$ array, and that is the shape the results comes back in.\n\n> **Exercise:**"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "# What happens when your index array is bigger than the target array?\n# Hint: you could use a large one-dimensional array or something fancier like ind = np.arange(0, 12).reshape((6, 2))\n",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "Fancy indexing also works in multiple dimensions:"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "arr2 = np.arange(12).reshape((3, 4))\narr2",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "As with standard indexing, the first index refers to the row and the second to the column:"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "row = np.array([0, 1, 2])\ncol = np.array([2, 1, 3])\narr2[row, col]",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "What did you actually get as your final result here? The first value in the result array is `arr2[0, 2]`, the second one is `arr2[1, 1]`, and the third one is `arr2[2, 3]`.\n\nThe pairing of indices in fancy indexing follows all the same broadcasting rules we covered earlier. Thus, if you combine a column vector and a row vector within the indices, you get a two-dimensional result:"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "arr2[row[:, np.newaxis], col]",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "Here, each row value is matched with each column vector, exactly as we saw in broadcasting of arithmetic operations.\n\n> **Exercise:**"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "# Now try broadcasting this on your own.\n# What do you get with row[:, np.newaxis] * col? \n# Or row[:, np.newaxis] * row? col[:, np.newaxis] * row?\n# What about col[:, np.newaxis] * row?\n# Hint: think back to the broadcast rules\n",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "**The big takeaway:** It is always important to remember that fancy indexing returns values reflected by the *broadcasted shape of the indices*, and not the shape of the array being indexed."
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "### Combined indexing\n\nYou can also combine fancy indexing with the other indexing schemes you have learned. Consider `arr2` again:"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "print(arr2)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "Now combine fancy and simple indices:"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "arr2[2, [2, 0, 1]]",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "What did you get back? The elements at positions 2, 0, and 1 of row 2 (the third row).\n\nYou can also combine fancy indexing with slicing:"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "arr2[1:, [2, 0, 1]]",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "Again, consider what you got back as output: the elements at positions 2, 0, and 1 of each row after the first one (so the second and third rows).\n\nOf course, you can also combine fancy indexing with masking:"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "mask = np.array([1, 0, 1, 0], dtype=bool)\narr2[row[:, np.newaxis], mask]",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "### Modifying values using fancy indexing\n\nFancy indexing is, of course, not just for accessing parts of an array, but also for modifying parts of an array:"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "ind = np.arange(10)\narr = np.array([2, 1, 8, 4])\nind[arr] = 99\nprint(ind)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "You can also use a ufunc here and subtract 10 from each element of the array:"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "ind[arr] -= 10\nprint(ind)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "Be cautious when using repeated indices with operations like these. They might not always produce the results you expect. For example:"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "ind = np.zeros(10)\nind[[0, 0]] = [4, 6]\nprint(ind)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "Where did the 4 go? The result of this operation is to first assign `ind[0] = 4`, followed by `ind[0] = 6`. So the result is that `ind[0]` contains the value 6.\n\nBut not every operation repeats the way you might think it should:"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "arr = [2, 3, 3, 4, 4, 4]\nind[arr] += 1\nind",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "We might have expected that `ind[3]` would contain the value 2 and `ind[4]` would contain the value 3. After all, that is how many times each index is repeated. So what happened?\n\nThis happened because `ind[arr] += 1` is really shorthand for `ind[arr] = ind[arr] + 1`. `ind[arr] + 1` is evaluated, and then the result is assigned to the indices in `ind`. So, similar to the previous example, this is not augmentation that happens multiple times, but an assignment, which can lead to potentially counterintuitive results.\n\nBut what if you want an operation to repeat? To do this, use the `at()` method of ufuncs:"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "ind = np.zeros(10)\nnp.add.at(ind, arr, 1)\nprint(ind)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "> **Exercise:**"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "# What does np.subtract.at(ind, arr, 1) give you?\n# Play around with some of the other ufuncs we have seen.\n",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "> **Takeaway:** Fancy indexing enables you to select and manipulate several array members at once. This type of programmatic data manipulation is common in data science: often what you want to do with your data you want to do on several data points at once."
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "## Sorting arrays\n\nSo far we have just worried about accessing and modifying NumPy arrays. Another huge thing you will need to do as a data scientist is sort array data. Sorting is often an important means of teasing out the structure in data (such as outlying data points).\n\nAlthough you could use Python's built-in `sort` and `sorted` functions, they will not work nearly as efficiently as NumPy's `np.sort` function.\n\n`np.sort` returns a sorted version of an array without modifying the input:"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "a = np.array([2, 1, 4, 3, 5])\nnp.sort(a)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "In order to sort the array in-place, use the `sort` method directly on arrays:"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "a.sort()\nprint(a)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "A related function is `argsort`, which returns the *indices* of the sorted elements rather than the elements themselves:"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "a = np.array([2, 1, 4, 3, 5])\nb = np.argsort(a)\nprint(b)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "The first element of this result gives the index of the smallest element, the second value gives the index of the second smallest, and so on. These indices can then be used (via fancy indexing) to reconstruct the sorted array:"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "a[b]",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "### Sorting along rows or columns\n\nA useful feature of NumPy's sorting algorithms is the ability to sort along specific rows or columns of a multidimensional array using the `axis` argument. For example:"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "rand = np.random.RandomState(42)\ntable = rand.randint(0, 10, (4, 6))\nprint(table)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "# Sort each column of the table\nnp.sort(table, axis=0)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "# Sort each row of the table\nnp.sort(table, axis=1)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "Bear in mind that this treats each row or column as an independent array; any relationships between the row or column values will be lost doing this kind of sorting."
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "## Partial sorting: partitioning\n\nSometimes you don't need to sort an entire array, you just need to find the *k* smallest values in the array (often when looking at the distance of data points from one another). NumPy supplies this functionality through the `np.partition` function. `np.partition` takes an array and a number *k*; the result is a new array with the smallest *k* values to the left of the partition, and the remaining values to the right (in arbitrary order):"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "arr = np.array([7, 2, 3, 1, 6, 5, 4])\nnp.partition(arr, 3)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "Note that the first three values in the resulting array are the three smallest in the array, and the remaining array positions contain the remaining values. Within the two partitions, the elements have arbitrary order.\n\nSimilarly to sorting, we can partition along an arbitrary axis of a multidimensional array:"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "np.partition(table, 2, axis=1)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "The result is an array where the first two slots in each row contain the smallest values from that row, with the remaining values filling the remaining slots.\n\nFinally, just as there is an `np.argsort` that computes indices of the sort, there is an `np.argpartition` that computes indices of the partition. We'll see this in action in the following section when we discuss pandas.\n\n> **Takeaway:** Sorting your data is a fundamental means of exploring it and answering questions about it. The sorting algorithms in NumPy provide you with a fast, computationally efficient way of doing this on large amounts of data and with fine-grain control."
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "## Efficient computation on NumPy arrays: Universal functions\n\n> **Learning goal:** By the end of this subsection, you should have a basic understanding of what NumPy universal functions are and how (and why) to use them.\n\nSome of the properties that make Python great to work with for data science (its dynamic, interpreted nature, for example) can also make it slow. This is particularly true with looping. These small performance hits can add up to minutes (or longer) when dealing with truly huge datasets.\n\nWhen we first examined loops in the previous section, you probably didn't notice any delay: the loops were short enough that Python’s relatively slow looping wasn’t an issue. Consider this function, which calculates the reciprocal for an array of numbers:"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "import numpy as np\nnp.random.seed(0)\n\ndef compute_reciprocals(values):\n    output = np.empty(len(values))\n    for i in range(len(values)):\n        output[i] = 1.0 / values[i]\n    return output\n        \nvalues = np.random.randint(1, 10, size=5)\ncompute_reciprocals(values)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "Running this loop, it was probably difficult to even discern that execution wasn't instantaneous.\n\nBut let’s try it on a much larger array. To empirically do this, we'll time this with IPython's `%timeit` magic command."
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "big_array = np.random.randint(1, 100, size=1000000)\n%timeit compute_reciprocals(big_array)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "You certainly noticed that delay. The slowness of this looping becomes noticeable when we repeat many small operations many times.\n\nThe performance bottleneck is not the operations themselves, but the type-checking and function dispatches that Python performs on each cycle of the loop. In the case of the `compute_reciprocals` function above, each time Python computes the reciprocal, it first examines the object's type and does a dynamic lookup of the correct function to use for that type. Such is life with interpreted code. However, were we working with compiled code instead (such as in C), the object-type specification would be known before the code executes, and the result could be computed much more efficiently. This is where NumPy universal functions come into play."
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "### Ufuncs\n\nUniversal functions in NumPy (often shortened to *ufuncs*) provide a statically typed, compiled function for many of the operations that we will need to run in the course of manipulating and analyzing data.\n\nLet's examine what this means in practice. Let's find the reciprocals of `big_array` again, this time using a built-in NumPy division ufunc on the array:"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "%timeit (1.0 / big_array)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "That’s orders of magnitude better.\n\nUfuncs can be used between a scalar and an array and between arrays of arbitrary dimensions.\n\nComputations vectorized by ufuncs are almost always more efficient than doing the same computation using Python loops. This is especially true on large arrays. When possible, try to use ufuncs when operating on NumPy arrays, rather than using ordinary Python loops.\n\nUfuncs come in two flavors: *unary ufuncs*, which use a single input, and *binary ufuncs*, which operate on two inputs. The common ufuncs we'll look at here encompass both kinds."
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "#### Array arithmetic\n\nMany NumPy ufuncs use Python's native arithmetic operators, so you can use the standard addition, subtraction, multiplication, and division operators that we covered in the first section:"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "a = np.arange(4)\nprint(\"a     =\", a)\nprint(\"a + 5 =\", a + 5)\nprint(\"a - 5 =\", a - 5)\nprint(\"a * 2 =\", a * 2)\nprint(\"a / 2 =\", a / 2)\nprint(\"a // 2 =\", a // 2)  # floor division",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "There are also ufuncs for negation, exponentiation, and the modulo operation:"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "print(\"-a     = \", -a)\nprint(\"a ** 2 = \", a ** 2)\nprint(\"a % 2  = \", a % 2)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "You can also combine these ufuncs using the standard order of operations:"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "-(0.5*a + 1) ** 2",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "The Python operators are not actually the ufuncs, but are rather wrappers around functions built into NumPy. So the `+` operator is actually a wrapper for the `add` function:"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "np.add(a, 2)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "Here is a cheat sheet for the equivalencies between Python operators and NumPy ufuncs:\n\n| Operator\t    | Equivalent ufunc    | Description                           |\n|:--------------|:--------------------|:--------------------------------------|\n|``+``          |``np.add``           |Addition (e.g., ``1 + 1 = 2``)         |\n|``-``          |``np.subtract``      |Subtraction (e.g., ``3 - 2 = 1``)      |\n|``-``          |``np.negative``      |Unary negation (e.g., ``-2``)          |\n|``*``          |``np.multiply``      |Multiplication (e.g., ``2 * 3 = 6``)   |\n|``/``          |``np.divide``        |Division (e.g., ``3 / 2 = 1.5``)       |\n|``//``         |``np.floor_divide``  |Floor division (e.g., ``3 // 2 = 1``)  |\n|``**``         |``np.power``         |Exponentiation (e.g., ``2 ** 3 = 8``)  |\n|``%``          |``np.mod``           |Modulus/remainder (e.g., ``9 % 4 = 1``)|\n\nPython Boolean operators also work; we will explore those later in this section."
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "#### Absolute value\n\nNumPy also understands Python's built-in absolute value function:"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "a = np.array([-2, -1, 0, 1, 2])\nabs(a)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "This corresponds to the NumPy ufunc `np.absolute` (which is also available under the alias `np.abs`):"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "np.absolute(a)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "np.abs(a)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "#### Exponents and logarithms\n\nYou will need to use exponents and logarithms a lot in data science; these are some of the most common data transformations for machine learning and statistical work."
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "a = [1, 2, 3]\nprint(\"a     =\", a)\nprint(\"e^a   =\", np.exp(a))\nprint(\"2^a   =\", np.exp2(a))\nprint(\"3^a   =\", np.power(3, a))",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "The basic `np.log` gives the natural logarithm; if you need to compute base-2 or base-10 logarithms, NumPy also provides those:"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "a = [1, 2, 4, 10]\nprint(\"a        =\", a)\nprint(\"ln(a)    =\", np.log(a))\nprint(\"log2(a)  =\", np.log2(a))\nprint(\"log10(a) =\", np.log10(a))",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "There are also some specialized versions of these ufuncs to help maintain precision when dealing with very small inputs:"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "a = [0, 0.001, 0.01, 0.1]\nprint(\"exp(a) - 1 =\", np.expm1(a))\nprint(\"log(1 + a) =\", np.log1p(a))",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "These functions give more precise values than if you were to use the raw `np.log` or `np.exp` on very small values of `a`."
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "#### Specialized ufuncs\n\nNumPy has many other ufuncs. Another source for specialized and obscure ufuncs is the submodule `scipy.special`. If you need to compute some specialized mathematical or statistical function on your data, chances are it is implemented in `scipy.special`."
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "from scipy import special",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "# Gamma functions (generalized factorials) and related functions\na = [1, 5, 10]\nprint(\"gamma(a)     =\", special.gamma(a))\nprint(\"ln|gamma(a)| =\", special.gammaln(a))\nprint(\"beta(a, 2)   =\", special.beta(a, 2))",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "> **Takeaway:** Universal functions in NumPy provide you with computational functions that are faster than regular Python functions, particularly when working on large datasets that are common in data science. This speed is important because it can make you more efficient as a data scientist and it makes a broader range of inquiries into your data tractable in terms of time and computational resources."
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "## Aggregations\n\n> **Learning goal:** By the end of this subsection, you should be comfortable aggregating data in NumPy.\n\nOne of the first things you will find yourself doing with most datasets is computing the summary statistics for the data in order to get a general overview of your data before exploring it further. These summary statistics include the mean and standard deviation, in addition to other aggregates, such as the sum, product, median, minimum and maximum, or quantiles of the data.\n\nNumPy has fast built-in aggregation functions for working on arrays that are the subject of this sub-section."
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "### Summing the values of an array\n\nYou can use the built-in Python `sum` function to sum up the values in an array."
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "import numpy as np",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "myList = np.random.random(100)\nsum(myList)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "If you guessed that there is also a built-in NumPy function for this, you guessed correctly:"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "np.sum(myList)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "And if you guessed that the NumPy version is faster, you are doubly correct:"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "large_array = np.random.rand(1000000)\n%timeit sum(large_array)\n%timeit np.sum(large_array)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "For all their similarity, bear in mind that `sum` and `np.sum` are not identical; their optional arguments have different meanings, and `np.sum` is aware of multiple array dimensions."
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "### Minimum and maximum\n\nJust as Python has built-in `min` and `max` functions, NumPy has similar, vectorized versions:"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "np.min(large_array), np.max(large_array)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "You can also use `min`, `max`, and `sum` (and several other NumPy aggregates) as methods of the array object itself:"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "print(large_array.min(), large_array.max(), large_array.sum())",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "### Multidimensional aggregates\n\nBecause you will often treat the rows and columns of two-dimensional arrays differently (treating columns as variables and rows as observations of those variables, for example), it can often be desirable to aggregate array data along a row or column. Let's consider a two-dimensional array:"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "md = np.random.random((3, 4))\nprint(md)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "Unless you specify otherwise, each NumPy aggregation function will compute the aggregate for the entire array. Hence:"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "md.sum()",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "Aggregation functions take an additional argument specifying the *axis* along which to compute the aggregation. For example, we can find the minimum value within each column by specifying `axis=0`:"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "md.min(axis=0)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "> **Exercise:**"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "# What do you get when you try md.max(axis=1)?\n",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "Remember that the `axis` keyword specifies the *dimension of the array that is to be collapsed*, not the dimension that will be returned. Thus specifying `axis=0` means that the first axis will be the one collapsed: for two-dimensional arrays, this means that values within each column will be aggregated."
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "### Other aggregation functions\n\nThe table below lists other aggregation functions in NumPy. Most NumPy aggregates have a '`NaN`-safe' version, which computes the result while ignoring missing values marked by the `NaN` value.\n\n|Function Name      |   NaN-safe Version  | Description                                   |\n|:------------------|:--------------------|:----------------------------------------------|\n| ``np.sum``        | ``np.nansum``       | Compute sum of elements                       |\n| ``np.prod``       | ``np.nanprod``      | Compute product of elements                   |\n| ``np.mean``       | ``np.nanmean``      | Compute mean of elements                      |\n| ``np.std``        | ``np.nanstd``       | Compute standard deviation                    |\n| ``np.var``        | ``np.nanvar``       | Compute variance                              |\n| ``np.min``        | ``np.nanmin``       | Find minimum value                            |\n| ``np.max``        | ``np.nanmax``       | Find maximum value                            |\n| ``np.argmin``     | ``np.nanargmin``    | Find index of minimum value                   |\n| ``np.argmax``     | ``np.nanargmax``    | Find index of maximum value                   |\n| ``np.median``     | ``np.nanmedian``    | Compute median of elements                    |\n| ``np.percentile`` | ``np.nanpercentile``| Compute rank-based statistics of elements     |\n| ``np.any``        | N/A                 | Evaluate whether any elements are true        |\n| ``np.all``        | N/A                 | Evaluate whether all elements are true        |\n\nWe will see these aggregates often throughout the rest of the course."
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "> **Takeaway:** Aggregation is the primary means you will use to explore you data, not just when using NumPy, but particularly in conjunction with pandas, the Python library you will learn about in the next section, which builds off of NumPy and thus off of everything you have learned thus far."
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "## Computation on arrays with broadcasting\n\n> **Learning goal:** By the end of this subsection, you should have a basic understanding of how broadcasting works in NumPy (and why NumPy uses it).\n\nAnother means of vectorizing operations is to use NumPy's *broadcasting* functionality: creating rules for applying binary ufuncs like addition, subtraction, or multiplication on arrays of different sizes.\n\nBefore, when we performed binary operations on arrays of the same size, those operations were performed on an element-by-element basis."
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "first_array = np.array([3, 6, 8, 1])\nsecond_array = np.array([4, 5, 7, 2])\nfirst_array + second_array",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "Broadcasting enables you to perform these types of binary operations on arrays of different sizes. Thus, you could just as easily add a scalar (which is really just a zero-dimensional array) to an array:"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "first_array + 5",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "Similarly, you can add a one-dimensional array to a two-dimensional array:"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "one_dim_array = np.ones((1))\none_dim_array",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "two_dim_array = np.ones((2, 2))\ntwo_dim_array",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "one_dim_array + two_dim_array",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "So far, so easy. But you can use broadcasting on arrays in more complicated ways. Consider this example:"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "horizontal_array = np.arange(3)\nvertical_array = np.arange(3)[:, np.newaxis]\n\nprint(horizontal_array)\nprint(vertical_array)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "horizontal_array + vertical_array",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "### Rules of broadcasting\nBroadcasting ollows a set of rules to determine the interaction between the two arrays:\n- **Rule 1**: If the two arrays differ in their number of dimensions, the shape of the one with fewer dimensions is *padded* with ones on its leading (left) side.\n- **Rule 2**: If the shape of the two arrays does not match in any dimension, the array with shape equal to 1 in that dimension is stretched to match the other shape.\n- **Rule 3**: If, in any dimension, the sizes disagree and neither is equal to 1, NumPy raises an error.\n\nLet's see these rules in action to better understand them."
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "#### Broadcasting example 1\n\nLet's look at adding a two-dimensional array to a one-dimensional array:"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "two_dim_array = np.ones((2, 3))\none_dim_array = np.arange(3)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "Let's consider an operation on these two arrays. The shape of the arrays are:\n\n- `two_dim_array.shape = (2, 3)`\n- `one_dim_array.shape = (3,)`\n\nWe see by rule 1 that the array `one_dim_array` has fewer dimensions, so we pad it on the left with ones:\n\n- `two_dim_array.shape -> (2, 3)`\n- `one_dim_array.shape -> (1, 3)`\n\nBy rule 2, we now see that the first dimension disagrees, so we stretch this dimension to match:\n\n- `two_dim_array.shape -> (2, 3)`\n- `one_dim_array.shape -> (2, 3)`\n\nThe shapes match, and we see that the final shape will be `(2, 3)`:"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "two_dim_array + one_dim_array",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "> **Exercise:**"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "# Flip this around. Try adding these with two_dim_array = np.ones((3, 2)) \n# and one_dim_array = np.arange(3)[:, np.newaxis].\n# What do you get?\n",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "#### Broadcasting example 2\n\nLet's examine what happens when both arrays need to be broadcast:"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "vertical_array = np.arange(3).reshape((3, 1))\nhorizontal_array = np.arange(3)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "Again, we'll start by writing out the shape of the arrays:\n\n- `vertical_array.shape = (3, 1)`\n- `horizontal_array.shape = (3,)`\n\nRule 1 says we must pad the shape of `horizontal_array ` with ones:\n\n- `vertical_array.shape -> (3, 1)`\n- `horizontal_array.shape -> (1, 3)`\n\nAnd rule 2 tells us that we upgrade each of these ones to match the corresponding size of the other array:\n\n- `vertical_array.shape -> (3, 3)`\n- `horizontal_array.shape -> (3, 3)`\n\nBecause the result matches, these shapes are compatible. We can see this here:"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "vertical_array + horizontal_array",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "#### Broadcasting example 3\n\nHere's what happens with incompatible arrays:"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "M = np.ones((3, 2))\ni = np.arange(3)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "This is just a slightly different situation than in the first example: the matrix ``M`` is transposed.\nHow does this affect the calculation? The shape of the arrays are:\n\n- ``M.shape = (3, 2)``\n- ``i.shape = (3,)``\n\nAgain, rule 1 tells us that we must pad the shape of ``i`` with ones:\n\n- ``M.shape -> (3, 2)``\n- ``i.shape -> (1, 3)``\n\nBy rule 2, the first dimension of ``i`` is stretched to match that of ``M``:\n\n- ``M.shape -> (3, 2)``\n- ``i.shape -> (3, 3)``\n\nNow we hit Rule 3: the final shapes do not match and the two arrays are incompatible:"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "M + i",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "### Broadcasting in practice\nUfuncs enable you to avoid using slow Python loops; broadcasting builds on that.\n\nA common data practice is to *center* an array of data. For example, if we have an array of 10 observations, each of which consists of three values (called features in this context), we might want to center that data so that we have the differences from the mean rather than the raw data itself. Doing this can help us better compare the different values.\n\nWe'll store this in a $10 \\times 3$ array:"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "T = np.random.random((10, 3))\nT",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "Now compute the mean of each feature using the ``mean`` aggregate across the first dimension:"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "Tmean = T.mean(0)\nTmean",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "Finally, center ``T`` by subtracting the mean. (This is a broadcasting operation.)"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "T_centered = T - Tmean\nT_centered",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "This is not just faster, but easier than writing a loop to do this."
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "> **Takeaway:** The data you will work with in data science invariably comes in different shapes and sizes (at least in terms of the arrays in which you work with that data). The broadcasting functionality in NumPy enables you to use binary functions on irregularly fitting data in a predictable way."
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "## Comparisons, masks, and Boolean logic in NumPy\n\n> **Learning goal:** By the end of this subsection, you should be comfortable with and understand how to use Boolean masking in NumPy in order to answer basic questions about your data.\n\n*Masking* is when you want to manipulate or count or extract values in an array based on a criterion. For example, counting all the values in an array greater than a certain value is an example of masking. Boolean masking is often the most efficient way to accomplish these types of tasks in NumPy and it plays a large part in cleaning and otherwise preparing data for analysis (which we will cover in a later section)."
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "### Example: Counting Rainy Days\n\nLet's see masking in practice by examining the monthly rainfall statistics for Seattle. The data is in a CSV file from data.gov. To load the data, we will use pandas, which we will formally introduce later."
    },
    {
      "metadata": {
        "scrolled": true,
        "trusted": true
      },
      "cell_type": "code",
      "source": "import numpy as np\nimport pandas as pd\n\n# Use pandas to extract rainfall as a NumPy array\nrainfall_2003 = pd.read_csv('data/Observed_Monthly_Rain_Gauge_Accumulations_-_Oct_2002_to_May_2017.csv')['RG01'][ 2:14].values\nrainfall_2003",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "Let’s break down what we just did in the code cell above. The rainfall data contains monthly rainfall totals from several rain gauges around the city of Seattle; we selected the first one. From that gauge, we then selected the relevant months for the first full calendar year in the dataset, 2003. That range of months started at the third row of the CSV file (remember, Python zero-indexes!) and ran through the thirteenth row, hence `2:14]`.\n\nYou now have an array containing 12 values, each of which records the monthly rainfall in inches from January to December 2003.\n\nCommonly in data science, you will want to take a quick first exploratory look at the data. In this case, a bar chart is a good way to do this. To generate this bar chart, we will use Matplotlib, another important data-science tool that we will introduce formally later in the course. (This also brings up another widely used Python convention you should adopt: `import matplotlib.pyplot as plt`.)"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "%matplotlib inline\nimport matplotlib.pyplot as plt",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "plt.bar(np.arange(1, len(rainfall_2003) + 1), rainfall_2003)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "To briefly interpret the code snippet above, we passed two parameters to the bar function in pyplot: the first defining the index for the x-axis and the second defining the data to use for the bars (the y-axis). To create the index, we use the NumPy function `arange` to create a sequence of numbers (this is the same `arange` we encountered earlier in this section). We know that the length of our array is 12, but it can be a good habit to get into to programmatically pass the length of an array in case it changes or you don’t know it with specificity. We also added 1 to both the start and the end of the `arange` to accommodate for Python zero-indexing (because there is no “month-zero” in the calendar).\n\nLooking at the chart above (and as residents can attest), Seattle can have lovely, sunny summers. However, this is only a first glimpse of the data. There are still several questions we would like to answer, such as in how many months did it rain, or what was the average precipitation in those months? We would use masking to answer those questions. (We will also return to this example dataset to demonstrate concepts throughout the rest of this section.) Before we dive deeper in explaining what masking is, we should briefly touch on comparison operators in NumPy."
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "### Comparison operators as ufuncs\n\nIn addition to the computational operators as ufuncs that we have already encountered, NumPy also implements comparison operators such as `<` (less than) and `>` (greater than) as element-wise ufuncs. All of the standard Python comparison operations are available:"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "simple_array = np.array([1, 2, 3, 4, 5])",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "simple_array < 2  # less than",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "simple_array >= 4  # greater than or equal",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "simple_array == 2  # equal",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "It is also possible to do an element-wise comparison of two arrays, and to include compound expressions:"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "(2 * simple_array) == (simple_array ** 2)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "As with the arithmetic operators, these comparison operators are wrappers for the NumPy ufuncs: when you write ``x < 3``, NumPy actually uses ``np.less(x, 3)``. Here is a summary of the comparison operators and their equivalent ufuncs:\n\n| Operator\t    | Equivalent ufunc    || Operator\t   | Equivalent ufunc    |\n|:--------------|:--------------------||:--------------|:--------------------|\n|``==``         |``np.equal``         ||``!=``         |``np.not_equal``     |\n|``<``          |``np.less``          ||``<=``         |``np.less_equal``    |\n|``>``          |``np.greater``       ||``>=``         |``np.greater_equal`` |"
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "Just like the arithmetic ufuncs, the comparison ufuncs work on arrays of any size and shape."
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "rand = np.random.RandomState(0)\ntwo_dim_array = rand.randint(10, size=(3, 4))\ntwo_dim_array",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "two_dim_array < 6",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "The result is a Boolean array, and NumPy provides a number of straightforward patterns for working with these Boolean results."
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "## Working with Boolean arrays\n\nGiven a Boolean array, there are a host of useful operations you can do.\nWe'll work with `two_dim_array`, the two-dimensional array we created earlier."
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "print(two_dim_array)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "### Counting entries\n\nTo count the number of ``True`` entries in a Boolean array, ``np.count_nonzero`` is useful:"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "# how many values less than 6?\nnp.count_nonzero(two_dim_array < 6)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "We see that there are eight array entries that are less than 6.\nAnother way to get at this information is to use ``np.sum``; in this case, ``False`` is interpreted as ``0``, and ``True`` is interpreted as ``1``:"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "np.sum(two_dim_array < 5)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "The benefit of `sum()` is that, like with other NumPy aggregation functions, this summation can be done along rows or columns as well:"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "# how many values less than 5 in each row?\nnp.sum(two_dim_array < 5, axis=1)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "This counts the number of values less than 5 in each row of the matrix.\n\nIf we're interested in quickly checking whether any or all the values are true, we can use (you guessed it) ``np.any`` or ``np.all``:"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "# Are there any values less than zero?\nnp.any(two_dim_array < 0)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "> **Exercise:**"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "# Now check to see if all values less than 10?\n# Hint: use np.all()\n",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "``np.all`` and ``np.any`` can be used along particular axes as well. For example:"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "# are all values in each row less than 7?\nnp.all(two_dim_array < 7, axis=1)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "Here, all the elements in the first and third rows are less than 7, while this is not the case for the second row.\n\n**A reminder:** Python has built-in `sum()`, `any()`, and `all()` functions. These have a different syntax than the NumPy versions, and, in particular, will fail or produce unintended results when used on multidimensional arrays. Be sure that you are using `np.sum()`, `np.any()`, and `np.all()` for these examples."
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "### Boolean operators\n\nWe've already seen how we might count, say, all months with rain less than four inches, or all months with more than two inches of rain. But what if we want to know about all months with rain less than four inches and greater than one inch? This is accomplished through Python's *bitwise logic operators*, `&`, `|`, `^`, and `~`. Like with the standard arithmetic operators, NumPy overloads these as ufuncs which work element-wise on (usually Boolean) arrays.\n\nFor example, we can address this sort of compound question as follows:"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "np.sum((rainfall_2003 > 0.5) & (rainfall_2003 < 1))",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "So we see that there are two months with rainfall between 0.5 and 1.0 inches.\nNote that the parentheses here are important—because of operator-precedence rules, with parentheses removed, this expression would be evaluated as follows, which results in an error:"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "rainfall_2003 > (0.5 & rainfall_2003) < 1",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "Using the equivalence of *A AND B and NOT (NOT A OR NOT B)* (which you might remember if you've taken an introductory logic course), we can compute the same result in a different manner:"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "np.sum(~((rainfall_2003 <= 0.5) | (rainfall_2003 >= 1)))",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "Combining comparison operators and Boolean operators on arrays can lead to a wide range of efficient logical operations.\n\nThe following table summarizes the bitwise Boolean operators and their equivalent ufuncs:"
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "| Operator\t    | Equivalent ufunc    || Operator\t   | Equivalent ufunc    |\n|:--------------|:--------------------||:--------------|:--------------------|\n|``&``          |``np.bitwise_and``   ||&#124;         |``np.bitwise_or``    |\n|``^``          |``np.bitwise_xor``   ||``~``          |``np.bitwise_not``   |"
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "Using these tools, you can start to answer the types of questions we listed above about the Seattle rainfall data. Here are some examples of results we can compute when combining masking with aggregations:"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "print(\"Number of months without rain:\", np.sum(rainfall_2003 == 0))\nprint(\"Number of months with rain:   \", np.sum(rainfall_2003 != 0))\nprint(\"Months with more than 1 inch: \", np.sum(rainfall_2003 > 1))\nprint(\"Rainy months with < 1 inch:   \", np.sum((rainfall_2003 > 0) &\n                                              (rainfall_2003 < 1)))",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "## Boolean arrays as masks\n\nIn the prior section, we looked at aggregates computed directly on Boolean arrays.\nA more powerful pattern is to use Boolean arrays as masks, to select particular subsets of the data themselves.\nReturning to our `two_dim_array` array from before, suppose we want an array of all values in the array that are less than 5:"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "two_dim_array",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "You can obtain a Boolean array for this condition easily:"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "two_dim_array < 5",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "Now, to *select* these values from the array, you can simply index on this Boolean array. This is the *masking* operation:"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "two_dim_array[two_dim_array < 5]",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "What is returned is a one-dimensional array filled with all the values that meet your condition. Put another way, these are all the values in positions at which the mask array is ``True``.\n\nYou can use masking as a way to compute some relevant statistics on the Seattle rain data:"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "# Construct a mask of all rainy months\nrainy = (rainfall_2003 > 0)\n\n# Construct a mask of all summer months (June through September)\nmonths = np.arange(1, 13)\nsummer = (months > 5) & (months < 10)\n\nprint(\"Median precip in rainy months in 2003 (inches):   \", \n      np.median(rainfall_2003[rainy]))\nprint(\"Median precip in summer months in 2003 (inches):  \", \n      np.median(rainfall_2003[summer]))\nprint(\"Maximum precip in summer months in 2003 (inches): \", \n      np.max(rainfall_2003[summer]))\nprint(\"Median precip in non-summer rainy months (inches):\", \n      np.median(rainfall_2003[rainy & ~summer]))",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "> **Takeaway:** By combining Boolean operations, masking operations, and aggregates, you can quickly answer questions similar to those we posed about the Seattle rainfall data about any dataset. Operations like these will form the basis for the data exploration and preparation for analysis that will by our primary concerns in the following sections."
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "### Resources\n- [NumPy](https://www.numpy.org/)\n- [NumPy Reference](https://docs.scipy.org/doc/numpy/reference/)\n"
    }
  ],
  "metadata": {
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3",
      "language": "python"
    },
    "name": "_merged"
  },
  "nbformat": 4,
  "nbformat_minor": 1
}