PSet1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# CS 224D: Assignment #1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1 - Softmax"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### a - Prove that the softmax is invariant to constant offsets in the input, that is, for any input vector $x$ and any constant $c$, \n",
    "$$\n",
    "softmax(x) = softmax(x + c)\n",
    "$$\n",
    "where $x + c$ means adding the constant $c$ to every dimension of $x$. Remember that: \n",
    "\n",
    "$$\n",
    "softmax(x)_i = \\frac{e^{x_i}}{\\sum_je^{x_j}}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "a - anwer: Adding a constant $c$ changes both the numerator and the denominator the same way, which does not affect the result after division. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Remark\n",
    "It is usually computationnaly difficult to compute sums of exponentials with low arguments. Hence, we will use this property to normalize the arguments  and set $c = -max_i x_i$, ie that we substract the maximum element from all elements of $x$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### b - Given an input matrix of N rows and d columns, compute the softmax predictions for each row. Write your implementation in q1_softmax.py. You may test by executing python q1_softmax.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import random"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def softmax(x):\n",
    "    \"\"\"\n",
    "    Compute the softmax function for each row of the input x.\n",
    "\n",
    "    It is crucial that this function is optimized for speed because\n",
    "    it will be used frequently in later code.\n",
    "    You might find numpy functions np.exp, np.sum, np.reshape,\n",
    "    np.max, and numpy broadcasting useful for this task. (numpy\n",
    "    broadcasting documentation:\n",
    "    http://docs.scipy.org/doc/numpy/user/basics.broadcasting.html)\n",
    "\n",
    "    You should also make sure that your code works for one\n",
    "    dimensional inputs (treat the vector as a row), you might find\n",
    "    it helpful for your later problems.\n",
    "\n",
    "    You must implement the optimization in problem 1(a) of the \n",
    "    written assignment!\n",
    "    \"\"\"\n",
    "    \n",
    "    \"\"\"\n",
    "    Moreover, to be as general as possible, we suppose that x is a 2D array, \n",
    "    N rows and d columns\n",
    "    \"\"\"\n",
    "    \n",
    "    \"\"\"\n",
    "    Remark: we must also consider the specific case where N = 1\n",
    "    \"\"\"\n",
    "\n",
    "    ### YOUR CODE HERE\n",
    "    \n",
    "    # Case N > 1\n",
    "    if len(x.shape) > 1:\n",
    "        \n",
    "        ## normalization part\n",
    "\n",
    "        # First define the column containing the max of each row in x\n",
    "        max_row = x.max(axis = 1).reshape((x.shape[0], 1))\n",
    "        # Then remove this max to the dataset\n",
    "        x -= max_row\n",
    "\n",
    "        ## computation of softmax\n",
    "\n",
    "        x = np.exp(x)\n",
    "        denominator = x.sum(axis = 1).reshape(x.shape[0], 1)\n",
    "        x /= denominator\n",
    "        \n",
    "    else:\n",
    "        \n",
    "        ## normalization part\n",
    "        \n",
    "        max_row = x.max()\n",
    "        x -= max_row\n",
    "        x = np.exp(x)\n",
    "        denominator = x.sum()\n",
    "        x /= denominator\n",
    "    \n",
    "    ### END YOUR CODE\n",
    "    \n",
    "    return x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 0.09003057,  0.24472847,  0.66524096],\n",
       "       [ 0.04661262,  0.01714783,  0.93623955]])"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Toy example\n",
    "# How to visually check? --> For each row, the sum must be equal to 1\n",
    "softmax(np.array([[1,2, 3], [2, 1, 5]]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's test this function with the test function give in the assignment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Running basic tests...\n",
      "[ 0.26894142  0.73105858]\n",
      "[[ 0.26894142  0.73105858]\n",
      " [ 0.26894142  0.73105858]]\n",
      "[[ 0.73105858  0.26894142]]\n",
      "You should verify these results!\n",
      "\n"
     ]
    }
   ],
   "source": [
    "def test_softmax_basic():\n",
    "    \"\"\"\n",
    "    Some simple tests to get you started. \n",
    "    Warning: these are not exhaustive.\n",
    "    \"\"\"\n",
    "    print(\"Running basic tests...\")\n",
    "    test1 = softmax(np.array([1,2]))\n",
    "    print(test1)\n",
    "    assert np.amax(np.fabs(test1 - np.array(\n",
    "        [0.26894142,  0.73105858]))) <= 1e-6\n",
    "\n",
    "    test2 = softmax(np.array([[1001,1002],[3,4]]))\n",
    "    print(test2)\n",
    "    assert np.amax(np.fabs(test2 - np.array(\n",
    "        [[0.26894142, 0.73105858], [0.26894142, 0.73105858]]))) <= 1e-6\n",
    "\n",
    "    test3 = softmax(np.array([[-1001,-1002]]))\n",
    "    print(test3)\n",
    "    assert np.amax(np.fabs(test3 - np.array(\n",
    "        [0.73105858, 0.26894142]))) <= 1e-6\n",
    "\n",
    "    print(\"You should verify these results!\\n\")\n",
    "test_softmax_basic()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## 2 - Neural Network Basics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "#### a - Derive the gradients of the sigmoid function and show that it can be rewritten as a function of the function value (ie in some expression where only $\\sigma (x)$, but not $x$, ispresent). Assume that the input $x$ is a sclarar for this question. Recall, the sigmoid function is: \n",
    "\n",
    "$$\n",
    "\\sigma (x) = \\frac{1}{1 + e^{-x}}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "a - Answer: let's compute the gradient of $\\sigma$ (1-D function): \n",
    "\n",
    "$$\n",
    "\\sigma'(x) = \\frac{e^{-x}}{(1 + e^{-x})^2} = \\sigma(x)(1 - \\sigma(x))\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "#### b - Derive the gradient with regard to the inputs of a softmax function when cross entropy loss is used for evaluation, ie find the gradients with respect to the softmax input vector $\\theta$, when the prediction is made by $\\widehat y = softmax(\\theta)$. Remember the cross entropy function is: \n",
    "\n",
    "$$\n",
    "CE(y, \\widehat y) = -\\sum_iy_ilog(\\widehat y_i)\n",
    "$$ \n",
    "\n",
    "where $y$ is the one-hot label vector, and $\\widehat y$ is the predicted probability vector for all classes (Hint: you might want to consider the fact many elements of $y$ are zeros, and assume that only the $k$-th dimension of $y$ is one)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "b - Answer: We assume here that $y$ is an only word vector. As proposed, let's assume that the only non-zero value for $y$ is for the $k$-th element. We have then: \n",
    "\n",
    "$$\n",
    "CE(y, \\widehat y) = -log(\\widehat y_k) = -log(softmax(\\theta)_k) = -log(\\frac{e^{\\theta_k}}{\\sum_i e^{\\theta_i}}) = -\\theta_k + log(\\sum_i e^{\\theta_i})\n",
    "$$\n",
    "\n",
    "Now, let's differentiate this result w.r.t. $\\theta_j$ for a certain $j$-th coordinate of $\\theta$.\n",
    "\n",
    "We have: \n",
    "\n",
    "$$\n",
    "\\frac{\\partial CE(y, \\widehat y)}{\\partial \\theta_j} = -1 + \\frac{e^{\\theta_j}}{\\sum_i e^{\\theta_i}},  if j = k\n",
    "$$ \n",
    "\n",
    "and \n",
    "\n",
    "$$\n",
    "\\frac{\\partial CE(y, \\widehat y)}{\\partial \\theta_j} = \\frac{e^{\\theta_j}}{\\sum_i e^{\\theta_i}}, if j \\neq k\n",
    "$$\n",
    "\n",
    "We can rewrite this result as: \n",
    "\n",
    "$$\n",
    "\\frac{\\partial CE(y, \\widehat y)}{\\partial \\theta} = \\widehat y - y\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### c - Derive the gradients w.r.t. the inputs $x$ to an one-hidden-layer neural network (that is, find $\\frac{\\partial J}{\\partial x}$ where $J$ is the cost function for the neural network). The neural network employs sigmoid activation function for the hidden layer, and softmax for the outpu layer. Assume the one-hot label vector is $y$, and cross-entropy cost is used. Feel free to use $\\sigma'(x)$ as the shorthand for sigmoid gradient, and feel free to define any variables whenever you see fit.\n",
    "\n",
    "Recall that the forward propagation is as follows: \n",
    "\n",
    "$$\n",
    "h = sigmoid(xW_1 + b_1)\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\widehat y = softmax(hW_2 + b_2)\n",
    "$$\n",
    "\n",
    "Note that here we're assuming that the input vector (thus the hidden variables and output probabilities) is a row vector to be consistent with the programming assignment. When we apply the sigmoid function to a vector, we are applying it to each of the elements of that vector. $W_i$ and $b_i$ ($i = 1, 2$) are the weights and biases, respectively, of the two layers."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "c - Answer:\n",
    "\n",
    "Let's write $\\theta = hW_2 + b_2$\n",
    "\n",
    "We know from $Part1$ that: \n",
    "\n",
    "$$\n",
    "\\frac{\\partial J}{\\partial \\theta} = \\widehat y - y\n",
    "$$\n",
    "\n",
    "Then, using the chain rule, we have: \n",
    "\n",
    "$$\n",
    "\\frac{\\partial J}{\\partial h} = \\frac{\\partial J}{\\partial \\theta} \\frac{\\partial \\theta}{\\partial h} = (\\widehat y - y)W_2^T\n",
    "$$\n",
    "\n",
    "Therefore, still using the chain rule, we obtain: \n",
    "\n",
    "$$\n",
    "\\frac{\\partial J}{\\partial x} = \\frac{\\partial J}{\\partial h} \\frac{\\partial h}{\\partial x} = (\\widehat y - y)W_2^TW_1^T\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Remark\n",
    "\n",
    "Why $W^T$ and not $W$ appears in the derivatives?\n",
    "\n",
    "Take this example: \n",
    "\n",
    "$u^Tv$ is a real number, given that $u$ and $v$ are vectors of the same dimensions. We have: \n",
    "\n",
    "$$\n",
    "\\frac{\\partial u^Tv}{\\partial v} = u\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\frac{\\partial u^Tv}{\\partial u} = v\n",
    "$$\n",
    "\n",
    "Deriving a real number w.r.t. a vector gives a vector of the same dimension. Hence, if we derive w.r.t. the transpose of this vector, we should respect the dimensiosn and hence we have: \n",
    "\n",
    "$$\n",
    "\\frac{\\partial u^Tv}{\\partial u^T} = v^T\n",
    "$$\n",
    "\n",
    "In the previous example take $h = u^T$ and be convinced."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### d - How many parameters are there in this neural network, assuming the input is $D_x$-dimensional, the output is $D_y$-dimensional, and theere are $H$ hidden units?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "d - Answer: \n",
    "\n",
    "In $W_1$ we have $D_x . H$ corefficients, in $W_2$ we have $H . D_y$ coefficients. Then, $b_1$ contains $H$ elements and $b_2$ contains $D_y$ elements.\n",
    "\n",
    "Hence, the total number of parameters used in this neural network is: \n",
    "\n",
    "$\\#_{parameters} = (D_x + 1) . H + (H + 1) . D_y$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### e - Fill in the implementation for the sigmoid implementation and its gradient in q2_sigmoid.py. Test your implementation using python q2_sigmoid.py. Again, thoroughly test your codes as the provided tests may not be exhaustive."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def sigmoid(x):\n",
    "    \"\"\"\n",
    "    Compute the sigmoid function for the input here.\n",
    "    \"\"\"\n",
    "    \n",
    "    ### YOUR CODE HERE\n",
    "    x = 1 / (1 + np.exp(-x))\n",
    "    ### END YOUR CODE\n",
    "    \n",
    "    return x\n",
    "\n",
    "def sigmoid_grad(f):\n",
    "    \"\"\"\n",
    "    Compute the gradient for the sigmoid function here. Note that\n",
    "    for this implementation, the input f should be the sigmoid\n",
    "    function value of your original input x. \n",
    "    \"\"\"\n",
    "    \n",
    "    ### YOUR CODE HERE\n",
    "    f = f*(1 - f)\n",
    "    ### END YOUR CODE\n",
    "    \n",
    "    return f\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 0.73105858,  0.88079708],\n",
       "       [ 0.99966465,  0.5       ]])"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = np.array([[1, 2], [8, 0]])\n",
    "sigmoid(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's test these functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Running basic tests...\n",
      "[[ 0.73105858  0.88079708]\n",
      " [ 0.26894142  0.11920292]]\n",
      "[[ 0.19661193  0.10499359]\n",
      " [ 0.19661193  0.10499359]]\n",
      "You should verify these results!\n",
      "\n"
     ]
    }
   ],
   "source": [
    "def test_sigmoid_basic():\n",
    "    \"\"\"\n",
    "    Some simple tests to get you started. \n",
    "    Warning: these are not exhaustive.\n",
    "    \"\"\"\n",
    "    print(\"Running basic tests...\")\n",
    "    x = np.array([[1, 2], [-1, -2]])\n",
    "    f = sigmoid(x)\n",
    "    g = sigmoid_grad(f)\n",
    "    print(f)\n",
    "    assert np.amax(f - np.array([[0.73105858, 0.88079708], \n",
    "        [0.26894142, 0.11920292]])) <= 1e-6\n",
    "    print(g)\n",
    "    assert np.amax(g - np.array([[0.19661193, 0.10499359],\n",
    "        [0.19661193, 0.10499359]])) <= 1e-6\n",
    "    print(\"You should verify these results!\\n\")\n",
    "test_sigmoid_basic()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### f - To make debugging easier, we will now implement a gradient checker. Fill in the implementation for gradchecker_naive in q2_gradcheck.py. Test your code using python q2_gradcheck.py."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# First implement a gradient checker by filling in the following functions\n",
    "def gradcheck_naive(f, x):\n",
    "    \"\"\" \n",
    "    Gradient check for a function f \n",
    "    - f should be a function that takes a single argument and outputs the cost and its gradients\n",
    "    - x is the point (numpy array) to check the gradient at\n",
    "    \"\"\" \n",
    "\n",
    "    rndstate = random.getstate()\n",
    "    random.setstate(rndstate)  \n",
    "    fx, grad = f(x) # Evaluate function value at original point\n",
    "    h = 1e-4\n",
    "\n",
    "    # Iterate over all indexes in x\n",
    "    it = np.nditer(x, flags=['multi_index'], op_flags=['readwrite'])\n",
    "    while not it.finished:\n",
    "        ix = it.multi_index\n",
    "\n",
    "        ### try modifying x[ix] with h defined above to compute numerical gradients\n",
    "        ### make sure you call random.setstate(rndstate) before calling f(x) each time, this will make it \n",
    "        ### possible to test cost functions with built in randomness later\n",
    "        \n",
    "        ### YOUR CODE HERE:\n",
    "        \n",
    "        x[ix] += h # Go to a h-step forward\n",
    "        random.setstate(rndstate)  \n",
    "        f_sup, _ = f(x) # evaluate f on x + h, the superior value\n",
    "        x[ix] -= 2 * h  # Go to a h-step backward compared to initial x\n",
    "        random.setstate(rndstate)  \n",
    "        f_inf, _ = f(x) # Evaluate f on x - h, the inferior value\n",
    "        x[ix] += h\n",
    "        numgrad = (f_sup - f_inf) / (2*h) # Evaluate the gradient as the tangent at the initial point x\n",
    "        \n",
    "        ### END YOUR CODE\n",
    "        \n",
    "                \n",
    "        # Compare gradients\n",
    "        reldiff = abs(numgrad - grad[ix]) / max(1, abs(numgrad), abs(grad[ix]))\n",
    "        if reldiff > 1e-5:\n",
    "            print(\"Gradient check failed.\")\n",
    "            print(\"First gradient error found at index %s\" % str(ix))\n",
    "            print(\"Your gradient: %f \\t Numerical gradient: %f\" % (grad[ix], numgrad))\n",
    "            return\n",
    "    \n",
    "        it.iternext() # Step to next dimension\n",
    "\n",
    "    print(\"Gradient check passed!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's use the sanity check"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Running sanity checks...\n",
      "Gradient check passed!\n",
      "Gradient check passed!\n",
      "Gradient check passed!\n",
      "\n"
     ]
    }
   ],
   "source": [
    "def sanity_check():\n",
    "    \"\"\"\n",
    "    Some basic sanity checks.\n",
    "    \"\"\"\n",
    "    quad = lambda x: (np.sum(x ** 2), x * 2)\n",
    "\n",
    "    print(\"Running sanity checks...\")\n",
    "    gradcheck_naive(quad, np.array(123.456))      # scalar test\n",
    "    gradcheck_naive(quad, np.random.randn(3,))    # 1-D test\n",
    "    gradcheck_naive(quad, np.random.randn(4,5))   # 2-D test\n",
    "    print(\"\")\n",
    "sanity_check()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### g - Now, implement the forward and backward passes for neural network with one sigmoid hidden layer. Fill in your imlementation in q2_neural.py. Sanity check your implementation with q2_neural.py."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from q1_softmax import softmax\n",
    "from q2_sigmoid import sigmoid, sigmoid_grad\n",
    "from q2_gradcheck import gradcheck_naive\n",
    "\n",
    "def forward_backward_prop(data, labels, params, dimensions):\n",
    "    \"\"\" \n",
    "    Forward and backward propagation for a two-layer sigmoidal network \n",
    "    \n",
    "    Compute the forward propagation and for the cross entropy cost,\n",
    "    and backward propagation for the gradients for all parameters.\n",
    "    \"\"\"\n",
    "\n",
    "    ### Unpack network parameters (do not modify)\n",
    "    ofs = 0\n",
    "    Dx, H, Dy = (dimensions[0], dimensions[1], dimensions[2])\n",
    "\n",
    "    W1 = np.reshape(params[ofs:ofs+ Dx * H], (Dx, H))\n",
    "    ofs += Dx * H\n",
    "    b1 = np.reshape(params[ofs:ofs + H], (1, H))\n",
    "    ofs += H\n",
    "    W2 = np.reshape(params[ofs:ofs + H * Dy], (H, Dy))\n",
    "    ofs += H * Dy\n",
    "    b2 = np.reshape(params[ofs:ofs + Dy], (1, Dy))\n",
    "\n",
    "    ### YOUR CODE HERE: forward propagation\n",
    "    h = sigmoid(data.dot(W1) + b1) # First sigmoid layer\n",
    "    pred = softmax(h.dot(W2) + b2) # Then pass it to the softmax\n",
    "    cost = -np.sum(labels * np.log(pred)) # Finally compute the cost\n",
    "    ### END YOUR CODE\n",
    "    \n",
    "    ### YOUR CODE HERE: backward propagation\n",
    "    delta_0 = pred - labels # Corresponds to the derivative w.r.t. theta like in question 2b\n",
    "    gradW2 = h.T.dot(delta_0) # Derive w.r.t. W2\n",
    "    gradb2 = np.sum(delta_0, axis = 0) # here, gradb2 is the product of a lign of 1's (gradient of b1 w.r.t. to b1)\n",
    "                                         # and delta_0\n",
    "    delta = delta_0.dot(W2.T)*sigmoid_grad(h) # Useful for 2 following lines so we compute it \n",
    "                                              # only once\n",
    "    gradW1 = data.T.dot(delta) # Derive w.r.t. W1\n",
    "    gradb1 = np.sum(delta, axis = 0) # Same reason as for gradb2\n",
    "    \n",
    "    ### END YOUR CODE\n",
    "    \n",
    "    ### Stack gradients (do not modify)\n",
    "    grad = np.concatenate((gradW1.flatten(), gradb1.flatten(), \n",
    "        gradW2.flatten(), gradb2.flatten()))\n",
    "    \n",
    "    return cost, grad\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's pass a sanity check"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Running sanity check...\n",
      "Gradient check passed!\n"
     ]
    }
   ],
   "source": [
    "def sanity_check():\n",
    "    \"\"\"\n",
    "    Set up fake data and parameters for the neural network, and test using \n",
    "    gradcheck.\n",
    "    \"\"\"\n",
    "    print(\"Running sanity check...\")\n",
    "\n",
    "    N = 20\n",
    "    dimensions = [10, 5, 10]\n",
    "    data = np.random.randn(N, dimensions[0])   # each row will be a datum\n",
    "    labels = np.zeros((N, dimensions[2]))\n",
    "    for i in range(N):\n",
    "        labels[i,random.randint(0,dimensions[2]-1)] = 1\n",
    "    \n",
    "    params = np.random.randn((dimensions[0] + 1) * dimensions[1] + (\n",
    "        dimensions[1] + 1) * dimensions[2], )\n",
    "\n",
    "    gradcheck_naive(lambda params: forward_backward_prop(data, labels, params,\n",
    "        dimensions), params)\n",
    "sanity_check()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3 - Word2vec"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### a - Assume you are given a predicted word vector $v_c$, corresponding to the center word $c$ for skipgram, and word prediction is made with the softmax function found in $word2vec$ models: \n",
    "\n",
    "$$\n",
    "\\widehat y_o = p(o|c) = \\frac{exp(u_o^Tv_c)}{\\sum_{w = 1}^{W}exp(u_w^Tv_c)}\n",
    "$$\n",
    "\n",
    "where $w$ denotes the $w$-th word and $u_w$ ($w = 1, ..., W$) are the \"output\" word vectors for all words in the vocabulary. Assume cross entropy cost is applied to this prediction and word $o$ is the expected word (the $o$-th element of the one-hot label vector is one), derive the gradients with respecte to $v_c$.\n",
    "\n",
    "Hint: it will be helpful to use notation from Question 2. For instance, letting $\\widehat y$ be the vector of softmax predictions for every word, $y$ as the expected word vectror, and the loss function\n",
    "\n",
    "$$\n",
    "J_{softmax-CE}(o, v_c, U) = CE(y, \\widehat y)\n",
    "$$\n",
    "\n",
    "where $U = [u_1, u_2, ..., u_w]$ is the matrix of all the output vectors. Make sure you state the orientation of your vectors and matrices."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "a - Answer: \n",
    "\n",
    "The question asks to compute the following: \n",
    "\n",
    "$$\n",
    "\\frac{\\partial -log(\\frac{exp(u_o^Tv_c)}{\\sum_{w = 1}^{W}exp(u_w^Tv_c)})}{\\partial v_c} = \\frac{\\partial -u_o^Tv_c}{\\partial v_c} + \\frac{log(\\sum_{w = 1}^W exp(u_w^Tv_c))}{\\partial v_c} = -u_o + \\frac{\\sum_{i = 1}^W u_iexp(u_i^Tv_c)}{\\sum_{w = 1}^W ex^(u_w^Tv_c)}\n",
    "$$\n",
    "\n",
    "We can rearrange this expression as: \n",
    "\n",
    "$$\n",
    "\\frac{\\partial -log(\\frac{exp(u_o^Tv_c)}{\\sum_{w = 1}^{W}exp(u_w^Tv_c)})}{\\partial v_c} = -u_o + \\sum_{i = 1}^W u_i p(i | c)\n",
    "$$\n",
    "\n",
    "which may be rewritten as: \n",
    "\n",
    "$$\n",
    "\\frac{\\partial J}{\\partial v_c} = -u_o + \\sum_{w = 1}^W \\widehat y_w u_w\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### b - As in the previous part, derive the gradients for the \"ouput\" word vectors $u_w$'s (including $u_o$)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "b - Answer: \n",
    "\n",
    "We have: \n",
    "\n",
    "$$\n",
    "\\frac{\\partial J}{\\partial u_k} = \\frac{\\partial -u_o^Tv_c}{\\partial u_k} + \\frac{log(\\sum_w exp(u_w^Tv_c))}{\\partial u_k}\n",
    "$$\n",
    "\n",
    "If $k = o$, we have: \n",
    "\n",
    "$$\n",
    "\\frac{\\partial J}{\\partial u_k} = -v_c + v_c p(o | c) = v_c(\\widehat y_o - 1)\n",
    "$$\n",
    "\n",
    "If $k \\neq o$ we have: \n",
    "\n",
    "$$\n",
    "\\frac{\\partial J}{\\partial u_k} = v_c p(k | c) = v_c\\widehat y_k\n",
    "$$\n",
    "\n",
    "This may be rewritten as: \n",
    "\n",
    "$$\n",
    "\\frac{\\partial J}{\\partial U} = v_c(\\widehat y - y)^T\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### c - Repeat part (a) and (b) assuming we are using the negative sampling loss for the predicted vectors $v_c$, and the expected output word is $o$. Assume that K negative samples (words) are drawn, and they are $1, ..., K$, respectively for simplicity of notation ($o \\notin {1, ..., K}$). Again, for a given word $o$, denote its output vector as $u_o$. The negative sampling loss function in this case is: \n",
    "\n",
    "$$\n",
    "J_{neg-sample}(o, v_c, U) = -log(\\sigma(u_o^Tv_c)) - \\sum_{k = 1}^K log(\\sigma(-u_k^Tv_c))\n",
    "$$\n",
    "\n",
    "where $\\sigma(.)$ is the sigmoid function.\n",
    "\n",
    "After you've done this, describe with one sentence why this cost function is much more efficient to compute that softmax-CE loss (you could provide a speed-up ratio, ie the runtime of the softmax-CE loss divided by the runtime of the negative sampling-loss).\n",
    "\n",
    "Note: the cost function here is the negative of what Mikolov et al had in their original paper, because we are doing a minimization instead of a maximization in our code."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "c - Answer: \n",
    "\n",
    "- We first compute the gradient w.r.t. $v_c$\n",
    "\n",
    "$$\n",
    "\\frac{\\partial J_{neg-sample}}{\\partial v_c} = \\frac{\\partial -log(\\sigma(u_o^Tv_c))}{\\partial v_c} + \\frac{\\partial -\\sum_{k = 1}^K log(\\sigma(-u_k^Tv_c))}{\\partial v_c}\n",
    "$$\n",
    "\n",
    "We use the property of the derivative of the sigmoid to compute the result as: \n",
    "\n",
    "$$\n",
    "\\frac{\\partial J_{neg-sample}}{\\partial v_c} = u_o(\\sigma(u_o^Tv_c) - 1) - \\sum_{k = 1}^K u_k(\\sigma(-u_k^Tv_c) - 1)\n",
    "$$\n",
    "\n",
    "- Then, we compute the gradient w.r.t. $u_o$\n",
    "\n",
    "We have: \n",
    "\n",
    "$$\\frac{\\partial J_{negative-sample}}{\\partial u_o} = v_c(\\sigma (u_o^Tv_c) - 1)\n",
    "$$  \n",
    "(we remind that $o \\notin {1, ..., K}$)\n",
    "\n",
    "- Finally, we compute the gradient w.r.t. $u_k$ with $k \\in {1, ..., K}$.\n",
    "\n",
    "We have: \n",
    "$$\\frac{\\partial J_{negative-sample}}{\\partial u_k} = -v_c(\\sigma(-u_k^Tv_c) - 1)\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This cost function is interesting in the fact that, at each iteration in the optimization algorithm that we will use, while computing a gradient, we need to compute small sums, with $K$ terms; on the contrary, with cross entropy loss function, we need to compute softmax output, which consists in sums over the whole vocabulary!! \n",
    "\n",
    "Hence, the implementation with negative sampling will be more efficient in terms of running time."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### d - Derive gradients for all of the word vectors for skip-grtam and CBOW given the previous parts and given a set of context words $[word_{c - m}, ..., word_{c - 1}, word_c, word-{c + 1}, ..., word_{c + m}]$, where $m$ is the context size. Denote the \"input\" and \"output\" word vectors for $word_k$ as $v_k$ and $u_k$ respectively.\n",
    "\n",
    "Hint: feel free to use $F(o, v_c)$ (where $o$ is the expected word as a placeholder for the $J_{Softmax-CE}(o, v_c, ...)$ or $J_{neg-sample}(o, v_c, ...)$) cost functions in this part - you'll see that this is a useful abstraction for the coding part. That is, your solution may contain terms of the form $\\frac{\\partial F(o, v_c)}{\\partial ...}$.\n",
    "\n",
    "Recall that for skip-gram, the cost for a context centered around c is: \n",
    "\n",
    "$$\n",
    "J_{skip-gram}(word_{c - m, ..., c + m}) = \\sum_{-m \\leqslant j \\leqslant m, j \\neq 0}F(w_{c + j}, v_c)\n",
    "$$\n",
    "\n",
    "where $w_{c + j}$ refers to the word at the $j$-th index from center.\n",
    "\n",
    "CBOW is slightly different. Instead of using $v_c$ as the predicted vector, we use $\\widehat v$ defined below. For (a simpler variant of) CBOW, we sum up the input word vectors in the context\n",
    "\n",
    "$$\n",
    "\\widehat v = \\sum_{-m \\leqslant j \\leqslant m, j \\neq 0}v_{c + j}\n",
    "$$\n",
    "\n",
    "then the CBOW cost is \n",
    "\n",
    "$$\n",
    "J_{CBOW}(word_{c - m, ..., c + m}) = F(w_c, \\widehat v)\n",
    "$$\n",
    "\n",
    "Note: To be consistent with the $\\widehat v$ notation such as for the code portion, for skip-gram $\\widehat v = v_c$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "d - Answer: \n",
    "\n",
    "- We have for Skip-Gram model \n",
    "\n",
    "$$\n",
    "    \\frac{\\partial J_{skip-gram}(word_{c - m, ..., c + m})}{\\partial U} = \\sum_{-m \\leqslant j \\leqslant m, j \\neq 0}\\frac{\\partial F(w_{c + j}, v_c)}{\\partial U}\n",
    "    $$\n",
    "    \n",
    "$$\n",
    "    \\frac{\\partial J_{skip-gram}(word_{c - m, ..., c + m})}{\\partial v_c} = \\sum_{-m \\leqslant j \\leqslant m, j \\neq 0}\\frac{\\partial F(w_{c + j}, v_c)}{\\partial v_c}\n",
    "    $$\n",
    "    \n",
    "$$\n",
    "    \\frac{\\partial J_{skip-gram}(word_{c - m, ..., c + m})}{\\partial v_j} = 0, \\forall j \\neq c\n",
    "    $$\n",
    "    \n",
    "    \n",
    "- For CBOW model, we have: \n",
    "\n",
    "$$\n",
    "    \\frac{\\partial J_{CBOW}(word_{c - m, ..., c + m})}{\\partial U} = \\frac{\\partial F(w_{c}, \\widehat v)}{\\partial U}\n",
    "    $$\n",
    "    \n",
    "$$\n",
    "    \\frac{\\partial J_{CBOW}(word_{c - m, ..., c + m})}{\\partial v_j} = \\frac{\\partial F(w_{c}, \\widehat v)}{\\partial \\widehat v} \\forall j, v_j \\in context\n",
    "    $$\n",
    "    \n",
    "$$\n",
    "    \\frac{\\partial J_{CBOW}(word_{c - m, ..., c + m})}{\\partial v_j} = 0, \\forall j, v_j \\notin context.\n",
    "    $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### e - In this part, you will implement $word2vec$ models and train your own word vectors with stochastic gradient descent (SGD). First, write a helper function to normalize rows of a matrix in q3_word2vec.py. In the same file, fill in the implementation for the softmax and negative sampling cost and gradient functions. Then, fill in the implementation of the cost and gradient functions for the skip-gram model. When you are done, test your implementation by running python q3_word2vec.py.\n",
    "\n",
    "Note: If you choose not to implement CBOW (part h), simply remove the NotImplementedError so that your tests will complete."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, let's code the normalization function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 0.4472136 ,  0.89442719],\n",
       "       [ 0.92847669,  0.37139068]])"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Each row should be normalized, ie their squared values sum to 1\n",
    "x = np.array([[1, 2], [5, 2]])\n",
    "x / np.sqrt(np.sum(x**2, axis = 1)).reshape((x.shape[0], 1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from q1_softmax import softmax\n",
    "from q2_gradcheck import gradcheck_naive\n",
    "from q2_sigmoid import sigmoid, sigmoid_grad\n",
    "\n",
    "def normalizeRows(x):\n",
    "    \"\"\" Row normalization function \"\"\"\n",
    "    # Implement a function that normalizes each row of a matrix to have unit length\n",
    "    \n",
    "    ### YOUR CODE HERE\n",
    "    \n",
    "    # Create the array of sums of squares by row\n",
    "    aux = np.sqrt(np.sum(x**2, axis = 1))\n",
    "    # Shape it to a column\n",
    "    aux = aux.reshape((x.shape[0], 1))\n",
    "    \n",
    "    # Normalize the rows\n",
    "    x /= aux\n",
    "    \n",
    "    ### END YOUR CODE\n",
    "    \n",
    "    return x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's test this function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Testing normalizeRows...\n",
      "[[ 0.6         0.8       ]\n",
      " [ 0.4472136   0.89442719]]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "def test_normalize_rows():\n",
    "    print(\"Testing normalizeRows...\")\n",
    "    x = normalizeRows(np.array([[3.0,4.0],[1, 2]])) \n",
    "    # the result should be [[0.6, 0.8], [0.4472, 0.8944]]\n",
    "    print(x)\n",
    "    assert (x.all() == np.array([[0.6, 0.8], [0.4472, 0.8944]]).all())\n",
    "    print(\"\")\n",
    "test_normalize_rows()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then, let's implement the softmax cost and gradient functions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def softmaxCostAndGradient(predicted, target, outputVectors, dataset):\n",
    "    \"\"\" Softmax cost function for word2vec models \"\"\"\n",
    "    \n",
    "    # Implement the cost and gradients for one predicted word vector  \n",
    "    # and one target word vector as a building block for word2vec     \n",
    "    # models, assuming the softmax prediction function and cross      \n",
    "    # entropy loss.                                                   \n",
    "    \n",
    "    # Inputs:                                                         \n",
    "    # - predicted: numpy ndarray, predicted word vector (\\hat{v} in \n",
    "    #   the written component or \\hat{r} in an earlier version)\n",
    "    # - target: integer, the index of the target word               \n",
    "    # - outputVectors: \"output\" vectors (as rows) for all tokens     \n",
    "    # - dataset: needed for negative sampling, unused here.         \n",
    "    \n",
    "    # Outputs:                                                        \n",
    "    # - cost: cross entropy cost for the softmax word prediction    \n",
    "    # - gradPred: the gradient with respect to the predicted word   \n",
    "    #        vector                                                \n",
    "    # - grad: the gradient with respect to all the other word        \n",
    "    #        vectors                           .                    \n",
    "    \n",
    "    # We will not provide starter code for this function, but feel    \n",
    "    # free to reference the code you previously wrote for this        \n",
    "    # assignment!                                                  \n",
    "    \n",
    "    ### YOUR CODE HERE\n",
    "    \n",
    "    proba = softmax(predicted.dot(outputVectors.T))\n",
    "    \n",
    "    # Compte cost\n",
    "    cost = -np.log(proba[target])\n",
    "    \n",
    "    # Compute the delta_0\n",
    "    delta_0 = proba\n",
    "    delta_0[target] -= 1 # to be consistent with the formulas we demonstrated in Part 1\n",
    "    \n",
    "    # Use previous expression for gradients\n",
    "    grad = delta_0.reshape((delta_0.shape[0], 1)) * predicted.reshape((1, predicted.shape[0]))\n",
    "    gradPred = (delta_0.reshape((1, delta_0.shape[0])).dot(outputVectors)).flatten()\n",
    "    \n",
    "    \n",
    "    ### END YOUR CODE\n",
    "    \n",
    "    return cost, gradPred, grad"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And now, let's do the same for CBOW."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def negSamplingCostAndGradient(predicted, target, outputVectors, dataset, \n",
    "    K=10):\n",
    "    \"\"\" Negative sampling cost function for word2vec models \"\"\"\n",
    "\n",
    "    # Implement the cost and gradients for one predicted word vector  \n",
    "    # and one target word vector as a building block for word2vec     \n",
    "    # models, using the negative sampling technique. K is the sample  \n",
    "    # size. You might want to use dataset.sampleTokenIdx() to sample  \n",
    "    # a random word index. \n",
    "    # \n",
    "    # Note: See test_word2vec below for dataset's initialization.\n",
    "    #                                       \n",
    "    # Input/Output Specifications: same as softmaxCostAndGradient     \n",
    "    # We will not provide starter code for this function, but feel    \n",
    "    # free to reference the code you previously wrote for this        \n",
    "    # assignment!\n",
    "    \n",
    "    ### YOUR CODE HERE\n",
    "    \n",
    "    # Create arrays of zeros for grad and gradPred, to be filled\n",
    "    grad = np.zeros(outputVectors.shape)\n",
    "    gradPred = np.zeros(predicted.shape)\n",
    "    \n",
    "    indices = [target]\n",
    "    for k in range(K):\n",
    "        newidx = dataset.sampleTokenIdx()\n",
    "        while newidx == target:\n",
    "            newidx = dataset.sampleTokenIdx()\n",
    "        indices += [newidx]\n",
    "        \n",
    "    labels = np.array([1] + [-1 for k in range(K)])\n",
    "    vecs = outputVectors[indices,:]\n",
    "    \n",
    "    t = sigmoid(vecs.dot(predicted) * labels)\n",
    "    cost = -np.sum(np.log(t))\n",
    "    \n",
    "    delta = labels * (t - 1)\n",
    "    gradPred = delta.reshape((1,K+1)).dot(vecs).flatten()\n",
    "    gradtemp = delta.reshape((K+1,1)).dot(predicted.reshape(\n",
    "        (1,predicted.shape[0])))\n",
    "    for k in range(K+1):\n",
    "        grad[indices[k]] += gradtemp[k,:]\n",
    "    \n",
    "    ### END YOUR CODE\n",
    "    \n",
    "    return cost, gradPred, grad\n",
    "\n",
    "## I have taken the implementation from Socher"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's implement the cost and gradient functions for the skip-gram model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# From Socher\n",
    "def skipgram(currentWord, C, contextWords, tokens, inputVectors, outputVectors, \n",
    "    dataset, word2vecCostAndGradient = softmaxCostAndGradient):\n",
    "    \"\"\" Skip-gram model in word2vec \"\"\"\n",
    "\n",
    "    # Implement the skip-gram model in this function.\n",
    "\n",
    "    # Inputs:                                                         \n",
    "    # - currrentWord: a string of the current center word           \n",
    "    # - C: integer, context size                                    \n",
    "    # - contextWords: list of no more than 2*C strings, the context words                                               \n",
    "    # - tokens: a dictionary that maps words to their indices in    \n",
    "    #      the word vector list                                \n",
    "    # - inputVectors: \"input\" word vectors (as rows) for all tokens           \n",
    "    # - outputVectors: \"output\" word vectors (as rows) for all tokens         \n",
    "    # - word2vecCostAndGradient: the cost and gradient function for \n",
    "    #      a prediction vector given the target word vectors,  \n",
    "    #      could be one of the two cost functions you          \n",
    "    #      implemented above\n",
    "\n",
    "    # Outputs:                                                        \n",
    "    # - cost: the cost function value for the skip-gram model       \n",
    "    # - grad: the gradient with respect to the word vectors         \n",
    "    # We will not provide starter code for this function, but feel    \n",
    "    # free to reference the code you previously wrote for this        \n",
    "    # assignment!\n",
    "\n",
    "    ### YOUR CODE HERE\n",
    "    \n",
    "    currentI = tokens[currentWord]\n",
    "    predicted = inputVectors[currentI, :]\n",
    "    \n",
    "    cost = 0.0\n",
    "    gradIn = np.zeros(inputVectors.shape)\n",
    "    gradOut = np.zeros(outputVectors.shape)\n",
    "    for cwd in contextWords:\n",
    "        idx = tokens[cwd]\n",
    "        cc, gp, gg = word2vecCostAndGradient(predicted, idx, outputVectors, dataset)\n",
    "        cost += cc\n",
    "        gradOut += gg\n",
    "        gradIn[currentI, :] += gp    \n",
    "    \n",
    "    ### END YOUR CODE\n",
    "    \n",
    "    return cost, gradIn, gradOut"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, let's implement the cost and gradient functions for the CBOW model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# From Socher\n",
    "def cbow(currentWord, C, contextWords, tokens, inputVectors, outputVectors, \n",
    "    dataset, word2vecCostAndGradient = softmaxCostAndGradient):\n",
    "    \"\"\" CBOW model in word2vec \"\"\"\n",
    "\n",
    "    # Implement the continuous bag-of-words model in this function.            \n",
    "    # Input/Output specifications: same as the skip-gram model        \n",
    "    # We will not provide starter code for this function, but feel    \n",
    "    # free to reference the code you previously wrote for this        \n",
    "    # assignment!\n",
    "\n",
    "    #################################################################\n",
    "    # IMPLEMENTING CBOW IS EXTRA CREDIT, DERIVATIONS IN THE WRIITEN #\n",
    "    # ASSIGNMENT ARE NOT!                                           #  \n",
    "    #################################################################\n",
    "    \n",
    "    cost = 0\n",
    "    gradIn = np.zeros(inputVectors.shape)\n",
    "    gradOut = np.zeros(outputVectors.shape)\n",
    "\n",
    "    ### YOUR CODE HERE\n",
    "    \n",
    "    D = inputVectors.shape[1]\n",
    "    predicted = np.zeros((D,))\n",
    "    \n",
    "    indices = [tokens[cwd] for cwd in contextWords]\n",
    "    for idx in indices:\n",
    "        predicted += inputVectors[idx, :]\n",
    "    \n",
    "    cost, gp, gradOut = word2vecCostAndGradient(predicted, tokens[currentWord], outputVectors, dataset)\n",
    "    gradIn = np.zeros(inputVectors.shape)\n",
    "    for idx in indices:\n",
    "        gradIn[idx, :] += gp\n",
    "    \n",
    "    ### END YOUR CODE\n",
    "    \n",
    "    return cost, gradIn, gradOut"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's run some test functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Testing normalizeRows...\n",
      "[[ 0.6         0.8       ]\n",
      " [ 0.4472136   0.89442719]]\n",
      "\n",
      "==== Gradient check for skip-gram ====\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\Peter martigny\\Anaconda3\\lib\\site-packages\\ipykernel\\__main__.py:6: VisibleDeprecationWarning: using a non-integer number instead of an integer will result in an error in the future\n",
      "C:\\Users\\Peter martigny\\Anaconda3\\lib\\site-packages\\ipykernel\\__main__.py:7: VisibleDeprecationWarning: using a non-integer number instead of an integer will result in an error in the future\n",
      "C:\\Users\\Peter martigny\\Anaconda3\\lib\\site-packages\\ipykernel\\__main__.py:19: VisibleDeprecationWarning: using a non-integer number instead of an integer will result in an error in the future\n",
      "C:\\Users\\Peter martigny\\Anaconda3\\lib\\site-packages\\ipykernel\\__main__.py:20: VisibleDeprecationWarning: using a non-integer number instead of an integer will result in an error in the future\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Gradient check passed!\n",
      "Gradient check passed!\n",
      "\n",
      "==== Gradient check for CBOW      ====\n",
      "Gradient check passed!\n",
      "Gradient check passed!\n",
      "\n",
      "=== Results ===\n",
      "(11.166109001533981, array([[ 0.        ,  0.        ,  0.        ],\n",
      "       [ 0.        ,  0.        ,  0.        ],\n",
      "       [-1.26947339, -1.36873189,  2.45158957],\n",
      "       [ 0.        ,  0.        ,  0.        ],\n",
      "       [ 0.        ,  0.        ,  0.        ]]), array([[-0.41045956,  0.18834851,  1.43272264],\n",
      "       [ 0.38202831, -0.17530219, -1.33348241],\n",
      "       [ 0.07009355, -0.03216399, -0.24466386],\n",
      "       [ 0.09472154, -0.04346509, -0.33062865],\n",
      "       [-0.13638384,  0.06258276,  0.47605228]]))\n",
      "(14.093692760899629, array([[ 0.        ,  0.        ,  0.        ],\n",
      "       [ 0.        ,  0.        ,  0.        ],\n",
      "       [-3.86802836, -1.12713967, -1.52668625],\n",
      "       [ 0.        ,  0.        ,  0.        ],\n",
      "       [ 0.        ,  0.        ,  0.        ]]), array([[-0.11265089,  0.05169237,  0.39321163],\n",
      "       [-0.22716495,  0.10423969,  0.79292674],\n",
      "       [-0.79674766,  0.36560539,  2.78107395],\n",
      "       [-0.31602611,  0.14501561,  1.10309954],\n",
      "       [-0.80620296,  0.36994417,  2.81407799]]))\n",
      "(0.79899580109066504, array([[ 0.23330542, -0.51643128, -0.8281311 ],\n",
      "       [ 0.11665271, -0.25821564, -0.41406555],\n",
      "       [ 0.11665271, -0.25821564, -0.41406555],\n",
      "       [ 0.        ,  0.        ,  0.        ],\n",
      "       [ 0.        ,  0.        ,  0.        ]]), array([[ 0.80954933,  0.21962514, -0.54095764],\n",
      "       [-0.03556575, -0.00964874,  0.02376577],\n",
      "       [-0.13016109, -0.0353118 ,  0.08697634],\n",
      "       [-0.1650812 , -0.04478539,  0.11031068],\n",
      "       [-0.47874129, -0.1298792 ,  0.31990485]]))\n",
      "(7.8955932035991392, array([[-2.98873309, -3.38440688, -2.62676289],\n",
      "       [-1.49436655, -1.69220344, -1.31338145],\n",
      "       [-1.49436655, -1.69220344, -1.31338145],\n",
      "       [ 0.        ,  0.        ,  0.        ],\n",
      "       [ 0.        ,  0.        ,  0.        ]]), array([[ 0.21992784,  0.0596649 , -0.14696034],\n",
      "       [-1.37825047, -0.37390982,  0.92097553],\n",
      "       [-0.77702167, -0.21080061,  0.51922198],\n",
      "       [-2.58955401, -0.7025281 ,  1.73039366],\n",
      "       [-2.36749007, -0.64228369,  1.58200593]]))\n"
     ]
    }
   ],
   "source": [
    "def word2vec_sgd_wrapper(word2vecModel, tokens, wordVectors, dataset, C, word2vecCostAndGradient = softmaxCostAndGradient):\n",
    "    batchsize = 50\n",
    "    cost = 0.0\n",
    "    grad = np.zeros(wordVectors.shape)\n",
    "    N = wordVectors.shape[0]\n",
    "    inputVectors = wordVectors[:N/2,:]\n",
    "    outputVectors = wordVectors[N/2:,:]\n",
    "    for i in range(batchsize):\n",
    "        C1 = random.randint(1,C)\n",
    "        centerword, context = dataset.getRandomContext(C1)\n",
    "        \n",
    "        if word2vecModel == skipgram:\n",
    "            denom = 1\n",
    "        else:\n",
    "            denom = 1\n",
    "        \n",
    "        c, gin, gout = word2vecModel(centerword, C1, context, tokens, inputVectors, outputVectors, dataset, word2vecCostAndGradient)\n",
    "        cost += c / batchsize / denom\n",
    "        grad[:N/2, :] += gin / batchsize / denom\n",
    "        grad[N/2:, :] += gout / batchsize / denom\n",
    "        \n",
    "    return cost, grad\n",
    "\n",
    "def test_word2vec():\n",
    "    # Interface to the dataset for negative sampling\n",
    "    dataset = type('dummy', (), {})()\n",
    "    def dummySampleTokenIdx():\n",
    "        return random.randint(0, 4)\n",
    "\n",
    "    def getRandomContext(C):\n",
    "        tokens = [\"a\", \"b\", \"c\", \"d\", \"e\"]\n",
    "        return tokens[random.randint(0,4)], [tokens[random.randint(0,4)] \\\n",
    "           for i in range(2*C)]\n",
    "    dataset.sampleTokenIdx = dummySampleTokenIdx\n",
    "    dataset.getRandomContext = getRandomContext\n",
    "\n",
    "    random.seed(31415)\n",
    "    np.random.seed(9265)\n",
    "    dummy_vectors = normalizeRows(np.random.randn(10,3))\n",
    "    dummy_tokens = dict([(\"a\",0), (\"b\",1), (\"c\",2),(\"d\",3),(\"e\",4)])\n",
    "    print(\"==== Gradient check for skip-gram ====\")\n",
    "    gradcheck_naive(lambda vec: word2vec_sgd_wrapper(skipgram, dummy_tokens, vec, dataset, 5), dummy_vectors)\n",
    "    gradcheck_naive(lambda vec: word2vec_sgd_wrapper(skipgram, dummy_tokens, vec, dataset, 5, negSamplingCostAndGradient), dummy_vectors)\n",
    "    print(\"\\n==== Gradient check for CBOW      ====\")\n",
    "    gradcheck_naive(lambda vec: word2vec_sgd_wrapper(cbow, dummy_tokens, vec, dataset, 5), dummy_vectors)\n",
    "    gradcheck_naive(lambda vec: word2vec_sgd_wrapper(cbow, dummy_tokens, vec, dataset, 5, negSamplingCostAndGradient), dummy_vectors)\n",
    "\n",
    "    print(\"\\n=== Results ===\")\n",
    "    print(skipgram(\"c\", 3, [\"a\", \"b\", \"e\", \"d\", \"b\", \"c\"], dummy_tokens, dummy_vectors[:5,:], dummy_vectors[5:,:], dataset))\n",
    "    print(skipgram(\"c\", 1, [\"a\", \"b\"], dummy_tokens, dummy_vectors[:5,:], dummy_vectors[5:,:], dataset, negSamplingCostAndGradient))\n",
    "    print(cbow(\"a\", 2, [\"a\", \"b\", \"c\", \"a\"], dummy_tokens, dummy_vectors[:5,:], dummy_vectors[5:,:], dataset))\n",
    "    print(cbow(\"a\", 2, [\"a\", \"b\", \"a\", \"c\"], dummy_tokens, dummy_vectors[:5,:], dummy_vectors[5:,:], dataset, negSamplingCostAndGradient))\n",
    "    \n",
    "test_normalize_rows()\n",
    "test_word2vec()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### f - Complete the implementation for your SGD optimizer in q3_sgd.py. Test your implementation by running python q3_sgd.py."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Save parameters every a few SGD iterations as fail-safe\n",
    "SAVE_PARAMS_EVERY = 1000\n",
    "\n",
    "import glob\n",
    "import random\n",
    "import numpy as np\n",
    "import os.path as op\n",
    "import pickle\n",
    "\n",
    "def load_saved_params():\n",
    "    \"\"\" A helper function that loads previously saved parameters and resets iteration start \"\"\"\n",
    "    st = 0\n",
    "    for f in glob.glob(\"saved_params_*.npy\"):\n",
    "        iter = int(op.splitext(op.basename(f))[0].split(\"_\")[2])\n",
    "        if (iter > st):\n",
    "            st = iter\n",
    "            \n",
    "    if st > 0:\n",
    "        with open(\"saved_params_%d.npy\" % st, \"r\") as f:\n",
    "            params = pickle.load(f)\n",
    "            state = pickle.load(f)\n",
    "        return st, params, state\n",
    "    else:\n",
    "        return st, None, None\n",
    "    \n",
    "def save_params(iter, params):\n",
    "    with open(\"saved_params_%d.npy\" % iter, \"w\") as f:\n",
    "        pickle.dump(params, f)\n",
    "        pickle.dump(random.getstate(), f)\n",
    "\n",
    "def sgd(f, x0, step, iterations, postprocessing = None, useSaved = False, PRINT_EVERY=10):\n",
    "    \"\"\" Stochastic Gradient Descent \"\"\"\n",
    "    # Implement the stochastic gradient descent method in this        \n",
    "    # function.                                                       \n",
    "    \n",
    "    # Inputs:                                                         \n",
    "    # - f: the function to optimize, it should take a single        \n",
    "    #     argument and yield two outputs, a cost and the gradient  \n",
    "    #     with respect to the arguments                            \n",
    "    # - x0: the initial point to start SGD from                     \n",
    "    # - step: the step size for SGD                                 \n",
    "    # - iterations: total iterations to run SGD for                 \n",
    "    # - postprocessing: postprocessing function for the parameters  \n",
    "    #     if necessary. In the case of word2vec we will need to    \n",
    "    #     normalize the word vectors to have unit length.          \n",
    "    # - PRINT_EVERY: specifies every how many iterations to output  \n",
    "\n",
    "    # Output:                                                         \n",
    "    # - x: the parameter value after SGD finishes  \n",
    "    \n",
    "    # Anneal learning rate every several iterations\n",
    "    ANNEAL_EVERY = 20000\n",
    "    \n",
    "    if useSaved:\n",
    "        start_iter, oldx, state = load_saved_params()\n",
    "        if start_iter > 0:\n",
    "            x0 = oldx;\n",
    "            step *= 0.5 ** (start_iter / ANNEAL_EVERY)\n",
    "            \n",
    "        if state:\n",
    "            random.setstate(state)\n",
    "    else:\n",
    "        start_iter = 0\n",
    "    \n",
    "    x = x0\n",
    "    \n",
    "    if not postprocessing:\n",
    "        postprocessing = lambda x: x\n",
    "    \n",
    "    expcost = None\n",
    "    \n",
    "    for iter in range(start_iter + 1, iterations + 1):\n",
    "        ### Don't forget to apply the postprocessing after every iteration!\n",
    "        ### You might want to print the progress every few iterations.\n",
    "\n",
    "        cost = None\n",
    "        ### YOUR CODE HERE\n",
    "\n",
    "        cost, grad = f(x)\n",
    "        x -= step * grad\n",
    "        x = postprocessing(x)\n",
    "        \n",
    "        ### END YOUR CODE\n",
    "        \n",
    "        if iter % PRINT_EVERY == 0:\n",
    "            if not expcost:\n",
    "                expcost = cost\n",
    "            else:\n",
    "                expcost = .95 * expcost + .05 * cost\n",
    "            print(\"iter %d: %f\" % (iter, expcost))\n",
    "        \n",
    "        if iter % SAVE_PARAMS_EVERY == 0 and useSaved:\n",
    "            save_params(iter, x)\n",
    "            \n",
    "        if iter % ANNEAL_EVERY == 0:\n",
    "            step *= 0.5\n",
    "    \n",
    "    return x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's do a sanity check"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Running sanity checks...\n",
      "iter 100: 0.004578\n",
      "iter 200: 0.004353\n",
      "iter 300: 0.004136\n",
      "iter 400: 0.003929\n",
      "iter 500: 0.003733\n",
      "iter 600: 0.003546\n",
      "iter 700: 0.003369\n",
      "iter 800: 0.003200\n",
      "iter 900: 0.003040\n",
      "iter 1000: 0.002888\n",
      "test 1 result: 8.414836786079764e-10\n",
      "iter 100: 0.000000\n",
      "iter 200: 0.000000\n",
      "iter 300: 0.000000\n",
      "iter 400: 0.000000\n",
      "iter 500: 0.000000\n",
      "iter 600: 0.000000\n",
      "iter 700: 0.000000\n",
      "iter 800: 0.000000\n",
      "iter 900: 0.000000\n",
      "iter 1000: 0.000000\n",
      "test 2 result: 0.0\n",
      "iter 100: 0.041205\n",
      "iter 200: 0.039181\n",
      "iter 300: 0.037222\n",
      "iter 400: 0.035361\n",
      "iter 500: 0.033593\n",
      "iter 600: 0.031913\n",
      "iter 700: 0.030318\n",
      "iter 800: 0.028802\n",
      "iter 900: 0.027362\n",
      "iter 1000: 0.025994\n",
      "test 3 result: -2.524451035823933e-09\n",
      "\n"
     ]
    }
   ],
   "source": [
    "def sanity_check():\n",
    "    quad = lambda x: (np.sum(x ** 2), x * 2)\n",
    "\n",
    "    print(\"Running sanity checks...\")\n",
    "    t1 = sgd(quad, 0.5, 0.01, 1000, PRINT_EVERY=100)\n",
    "    print(\"test 1 result:\", t1)\n",
    "    assert abs(t1) <= 1e-6\n",
    "\n",
    "    t2 = sgd(quad, 0.0, 0.01, 1000, PRINT_EVERY=100)\n",
    "    print(\"test 2 result:\", t2)\n",
    "    assert abs(t2) <= 1e-6\n",
    "\n",
    "    t3 = sgd(quad, -1.5, 0.01, 1000, PRINT_EVERY=100)\n",
    "    print(\"test 3 result:\", t3)\n",
    "    assert abs(t3) <= 1e-6\n",
    "    \n",
    "    print(\"\")\n",
    "    \n",
    "sanity_check()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### g - Showtime! Now, we are going to load some real data and train word vectors with everything you just implemented! We are going to use the Stanford Sentiment Treebank (SST) dataset to train word vectors, and later apply them to a simple sentiment analysis task. There is no additional code to write for this part; just run pythonq3_run.py\n",
    "\n",
    "Note: The training process may t ake a long time depending on the efficiency of your implementation (an efficient implementation takes approximately an hour). Plan accordingly!\n",
    "\n",
    "When the script finishes, a visualization for your word vectors will appear. It will also be saved as q3_word_vectors.png in your project directory. Include the plot in your homework write up. Briefly explain in at most three sentences what you see in the plot."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "sanity check: cost at convergence should be around or below 10\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAioAAAFoCAYAAAB5SNBJAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAAPYQAAD2EBqD+naQAAIABJREFUeJzs3Xt8zuX/wPHXdZ+GmuU8sy3LIYS08XVIfX1T+YoKzWEU\nIhLli0hKkUOiEKIcShTTSMq5ow4/RJtDSTnkNGcRY4f7c9/39fvjntvu2WzYvd2b9/PxuB/tc32u\n6/pcn0+437tOH6W1RgghhBDCH5kKugFCCCGEENmRQEUIIYQQfstS0A3wBaWUGQnCfMWltXYWdCOE\nEELcGIpcoKKUMlORUEpiK+i2FEnnsCulEiVYEUIIkR+KXKACmCiJjYY4CcRR0I0pUpKw8DM2jmIC\nJFARQgjhc0UxUHELxEE5CVR8wFzQDRBCCHHjkHkcQgghhPBbEqgIIYQQwm9JoCKEEEIIv3VjBSpj\nWMx4RhR0M67ba2xkMj0KuhlCCCGEr91YgUpOxrCYaUQXdDNy9CAtaceCPK3zAxrxGhvztE4hhBDi\nOhXdVT9FWWPO+KhmefGTEEIIv3IjBioW3mAMqTyGwkEJ5jGEt7LMOYlenKcjLm5F8Q8BfEkbxlCD\nFACm0p7TvEZFnuM4I3ARgo2v6cQAVvAwZ3geTSDF+ZRBvIo1vd5ptOMfnsJJFRTJWPk/7uVVmnIa\ngHFMIo0O6bk1oAAI5zF68DOvsZGSzGIgHwAwkkRKMZjz3I9BM0wcpTyj6MNXnnt5jwc5wSu4CMHC\nJoJYwt9MoS01uJPzef6UhRBCiDxw4w39pNABhUE9HqIMw7nA00yhU/pZ7x4FhZNQhtOQfxNKf+zc\nzTJezlRjcU7Sg+o8za3EYHA3C3mf8/yH2nShIs+RwuPMorWnhMZCBcbTgObcxpM4CeV7JnvOt+YV\n7uPO9E89ijMHxUmqsifb+/qHQdzC50RxHwF8yzHeIZ6SAKwgjGPMpBirqcf9lGIhp3nxsvsVQggh\n/MyN16Ni4jBDeS39aB8TqMVZegOLGO7pxXC72GPhdoQZTOAkbwDDM6RbuIuhtCIRgDdYQSqP0ZW6\n3EYqsJcxrCeJJsAKAPoTl6F8Ih/zKntYyW6KUY1U6nABuADADFqSwuPcRgfu5e9s76sEn9CP5QDU\nYBwL6EkC9YjiB37nCUzs4QVe99z3m9TgAv095XuwEWic8wMUQggh8s+NF6hYSfA6Lkk8yfTGAM/Q\nzEVzuIdj9MNJVTSBuJ+XjQMEcCtp6blSPEEKgJlTmDiUHqRcTDuJi7Ke4wXUYT+DcHAHmiAu9mxt\npxLV2OvJt5DanGAqpXmJrpnandlN7PT8XI1UIInU9Gsa3IaVbV75A9maHgoJIYQQfuvGG/rJrVVU\nIpEPsbKD23iKu2hBaV4C4B+vkMbIVFKjLtu6X6PTn/VuirGHhZg4x630405aUil9qbGR4UWK31GO\n3cylOB9n6oHJmimLa8r/XyGEEIXcjdejYnCX1/E5ojCx77LelP3UBRRDGe1Je4tHr/v6m6iK5hbq\nM44HOAbANOp55UnExk98gJldDPIMU107K3tJ5T9eaUmZrimEEEL4oRsjUImnFNv5N5pAXITzBlMo\nyyqSqEYyPQnkPZZS26uMGQtg5S1ephQbOUttLvAkAL9Ri70k4yAUMHuVdVEeTTGvNE0p4CaWUhsX\nJQEH8QzlICs4TwRneQqAs1RlKWb+YDAuwglhKAtp6qmnBOcw4wJsOKnodY00wjPdgxmDUJZSm7Js\n4iBP8waTKMsazlOVZLoAsIOa7E1fxZQTB2ZOYgNKKaXkhY9CCHFjKAZUBtZqrbOfK+kjSuuitfBD\nKWXldipzP2metyfPpQ2nmU4ylwZDDNyLfq1AQDaV2dM/Gvc7g61AKnBzelkj/TgwQ5k0wAHclCHt\nYhhQPMO10zLUa0vPUyL9+DxZr8cpjju0PJ9e5uJAUVKGc2RIK8aleTeO9LZevKYlvQ0X7yU3dHqZ\nzINdQgghbgRdtNYL8/uiN0aPyk0kchqmvT+NatWqXTHriBEjeO216x9t8Xex82JZ/flq5i+df9m5\n7J6Bw+HAnmQnpGwIVmvmsbKiZeDAgUyePDnnjEXYjf4MbvT7B3kGIM8A4KmnnmLLli0A+wvi+jdG\noGJxr8CpVq0aderUuWLWkiVL5pinMJo3Zx71IutRqnQpNm3YxLK4ZfR4ukeW95rdMzAMg7SzaUSE\nRhT5QCUoKIjIyMiCbkaButGfwY1+/yDPAOQZAAQGeoYNUq+Uz1dujEBFsG/vPqa+OZV//vmHSqGV\n6NO/D88OeragmyWEEEJckQQqN4iR40YyctzIgm6GEEIIcVVknw0hhBBC+C0JVDJp06ZNQTehwMkz\ngJiYmIJuQoG70Z/BjX7/IM8A5BkAtGjRokCvf2MsT15Kbfaxds2aNUVyomx+uZEm0wohhHBLSEgg\nKioKIEprfeXXufhA0Z2jkpTh3hyY0e7ltYYhm4BcK4dD9ngTQgiRv4pioOLiHHZ+xoZ7azM4iY00\nsCfZSTubduXS4opsFhsmk4wYCiGEyB9FLlDRWjuVUokc9Zp/UwogpGwIEaERBdSyosFkMmE2mwu6\nGUIIIW4QRS5QAXewAjgvHl98L43VapW5FUIIIUQhIn34QgghhPBbEqgIIYQQwm9JoCKEEEIIvyWB\nihBCCCH8lgQqQgghhPBbEqgIIYQQwm9JoCKEEEIIv+XzQEUp1U8ptU8plaKU2qiUanCFvG2VUl8q\npU4opc4qpdYrpR7MIl97pdTO9Dq3KaVa+vYuhBBCCFEQfBqoKKU6AhOBEcBdwDZgrVKqbDZF7gW+\nBFoCkcB3wHKl1J0Z6mwCLARmA/WAz4FlSqlavroPIYQQQhQMX/eoDARmaq3na63/APoAyUCPrDJr\nrQdqrd/SWsdrrfdqrV8GdgMPZ8jWH1ittZ6ktf5Ta/0qkAA869tbEUIIIUR+81mgopSyAlHANxfT\ntNYa+BponMs6FBAInM6Q3Di9jozW5rZOIYQQQhQevuxRKYv77cXHM6UfB4JzWccQ4CYgLkNa8HXW\nKYQQQohCwm9fSqiU6gy8AjyitT5V0O0RQgghRP7zZaByCvcbjCtkSq8AHLtSQaVUJ2AWEK21/i7T\n6WPXUifAwIEDCQoK8kqLiYkhJiYmp6JCCCFEkRcbG0tsbKxX2tmzZwuoNW7KPW3ER5UrtRH4WWv9\nv/RjBRwEpmqt38ymTAwwB+iotV6RxflFQHGt9aMZ0v4P2Ka17ptNnZFAfHx8PJGRkdd7W0IIIcQN\nIyEhgaioKIAorXVCfl/f10M/k4APlVLxwCbcq4BKAB8CKKXGASFa627px53Tz/UHNiulLvacpGit\nz6X/PAVYp5QaBKwEYnBP2u3l43sRQgghRD7z6fJkrXUcMBgYBWwB6gIttNYn07MEA2EZivTCPQF3\nOnAkw+ftDHVuADoDvYGtQDvgUa317768FyGEEELkP59PptVazwBmZHPuyUzH/8llnZ8Cn15/64QQ\nQgjhz+RdP0IIIYTwWxKoCCGEEMJvSaAihBBCCL8lgYoQQggh/JYEKkIIIYTwWxKoCCGEEMJvSaAi\nhBBCCL8lgYoQQggh/JYEKkIIIYTwWxKoCCGEEMJvSaAihBBCCL8lgYoQQggh/JYEKkIIIYTwWxKo\nCCGEEMJvSaAihBBCCL8lgYoQQggh/JYEKkIIIYTwWxKoCCGEEMJvSaAihBBCCL8lgYoQQggh/JYE\nKkIIIYTwWxKoCCGEEMJvSaAihBBCCL8lgYoQQggh/JYEKkIIIYTwW5aCboAQQgghcs/pdOJyufLt\neoZhXPzRopSyAi6ttTO/ri+BihBCCFFIOJ1ODh05hN1hz7drHjl1BKxAIBUpxxnOYVdKJeZXsCKB\nihBCCFFIuFwu7A475pvMWCz58xVuC7RBABCMnfo4+RkbRzEB+RKo+HyOilKqn1Jqn1IqRSm1USnV\n4Ap5g5VSC5RSfyqlnEqpSVnk6aaUcqWfd6V/kn17F0IIIYT/sFgsWK3WfPlYLBZQgAUngTjy/V59\nWblSqiMwEegNbAIGAmuVUtW11qeyKBIAnABGp+fNzlmgOu5HB6DzrNFCCCFEActuHophGBgOA5OR\nf2thPHNUXFg4hwUDC+75Krmt4rrmtPi632ggMFNrPR9AKdUHaAX0ACZkzqy1PpBeBqVUzyvUq7XW\nJ/O+uUIIIUTButI8FMMwSDyeiO2CDYs1f4Z+jp8+7o4WzJTnBOexEsBtgDWXvSvXOafFZ3eZPjM4\nCnj9YprWWiulvgYaX2f1Nyul9uMeukoAXtJa/36ddQohhBAF7krzUEyGCVuyjYDAgHwLVKw3W8EG\nlMGgCmkYQEXSuCUXgUoSluud0+LLuywLmIHjmdKPA7dfR71/4u6R2Q4EAUOA9UqpWlrrI9dRrxBC\nCOE3Ls5DySrdbDVfFqj0ju5Njdo1GDRyUN62w5w+R8WMk+K4sOLkFhyUy/V8FfP1XL/Qbfimtd6o\ntf5Ya71da/0j0A44CTxdwE0TQggh/Fbv6N6sWLwi2/OzJs7itYGv5WOLcseXPSqncHfzVMiUXgE4\nllcX0Vo7lFJbgKo55R04cCBBQUFeaTExMcTExORVc4QQQohCK/FwIgMGDPBKO3/+fAG1xs1ngYrW\n2lBKxQPNgS8AlHuKcHNgal5dRyllAuoAK3PKO3nyZCIjI/Pq0kIIIYRfcTgcTHh5Ais/XYnFYiG6\nazTPvPBMrsuHVgplxOQRXml/7PyDLl265HVTc83XQz+TgF5Kqa5KqRrAe0AJ4EMApdQ4pdS8jAWU\nUncqpeoBNwPl0o9rZjj/ilLqAaVUhFLqLmABEA7M8fG9CCGEEAWuf+/+vD3+7SzPrYhbgcVq4aNV\nHzFk9BAWzFrAsoXLAMi4nHjWtFnEtCkcowk+nTKstY5TSpUFRuEe8tkKtMiwtDgYCMtUbAuX9kWJ\nBDoDB4Db0tNKAbPSy54B4oHGWus/fHUfQgghRGEQXCmYP/b+wfZt22ndtjW7d+5m9AujKVWhFDMX\nz/TKm3kflN7P9766i62iEpv4mZGEXm+7r8Tna5u01jOAGdmcezKLtCv28mitBwF5O6VZCCGEKAJq\nR9Ym8USi57huVF1wgdY+2xfV5xuuyrt+hBBCiELG5XAxYdQEVn6RPhclJjrLfGOGjwHg+b7Pg4KQ\n0BCWf7Pcc37V56t4d8q7nDt3jib3NOHVsa9SvERxwB3czJ01l7gFcXAB2Mw4jjCDUNZnuESut6e9\nVoVuebIQQghxo1v++XLMVjMfLfmIIcOH8PHcjzl14hS/bfnNa0inRcsWYIJRE0bx1fqv+GjJR55z\nhw4cYt0365g6eypTZ00lYXMCc2fN9Zx//733Wf35anr27QnFgVKs4gCj2EXdDE2RHhUhhBBCeKtY\nsSLPD3segPDK4ez+czeLZi7CrMw0/ndj6tStw5rP1vB57OdghZsDb6Z0mdJedWitGTV+FMWKFwOg\n1aOt2LRhE30H9MWwG8ydOZf3PnwPa4DV3a1RlR9JIYSTtAFW8hCHeeiyeaZ5TnpUhBBCiELmjjvv\n8DquW68uaalptIpuRVpqGl1bdWXC8Al07tXZvf19FiyGhf91/Z/nuEy5Mpz5+wwAhw4eIjUllb49\n+tKzU0/30M9G5nKeljgI8dV9ZdnO/LyYyF52b8oUvmEymTCbr2tXZyGE8CuWkhZeGPMCSileHPei\nJ33O+1nv3uEwHNRvUt9zrJTCpd3fQ8kXkgGYNnsaZ8+dZdDAQRDOUMpy4LIX4/iYBCp+4EpvyhS+\nYbPYCAsJk2BFCFEo7di+w+t4+9bthN8aftmSYwCr1XrZL8L2NDtGmsHjTz+eZf23Vb2NgIAAjh4+\nSkS1CPf4SylOcAdH0Nn10fiGBCp+4EpvyhR5z+FwYL9gx+VySaAihCiUjh09xuQ3JtOuYzt27tjJ\nJx9/wvMvPZ9l3oqVKrJp/Sbq3lUXm81GYMlAbAE2qtxVxbPCJ7MSN5Xg8R6PM3HcRDo+0RFcwHEq\n8xmRJJNK1fzbZFW+Ff1Idm/KFHnPeW1vGxdCiAKnlKLVw+lzUdp3xWw206V7F9q2b5tl/oFDBzJ5\n/GSWxi2lfHB5r+XJV9J3QF9Kly7NgnkLIAXYz1Bs7CCQj/PwdnKkfLgJjN9QSkUC8fHx8X75rh/D\nMNiXuI+AoAAJVPKBYRiknU0jIjRCnrcQwu9c6TvBMAwOHjmIraQt3/798rzrpw4x1GUvv2GjKoco\nhyPHwiex8DUB/Ml+rbVxLdeXVT9CCCGE8FsSqAghhBDCb0mgUghEt4pm5LCRBd2MPJd4MJHQoFB+\n/+33gm6KEEIUKk6HE8Mw8uXjcDjc+886MJNKvq9AkMm0RUB0q2g6dulI+87tC7opV6VSWCW27tl6\n2W6J12vSuEkkHkpk0oxJeVqvEEIUNJPJhM1sw55sz7dFAcZ5A9KAE1ixYcaJnQDybeMvCVREgVFK\nUbZc2YJuhhBCFBpms5mQiiFe+6L4esPQ0ydOgwE4OEVZjmLDRRomTmYzKhOAi5J5F8jI0E8h4XQ4\nGT54ODXDalInog5vjnkz27yz3pnF/Y3vp1rFajSo1YCXBr3k2WUQIG5BHLXCa/H9N9/TrEEzqodU\n5/F2j3PyxElPnoHPDKRn5568N+09IqtHUrtybV5+/mWczksR/Nl/ztK/d3/uCL+DqsFVeeKxJ9i3\ndx8AKckp1AitwaovVnm1bc2KNVSrWI3kC8mXDf1s+GkDoUGh/PT9Tzz074eoGlyVRx94lL/2/OVV\nx9sT3ubOKndSM6wmQwcMZdxr43iw6YPX/nCFEKIQMZvNWK1WrFYrJpOJk6dOcvTkUZ99Tv1zCqyA\nhbKcoiJHqMQewrL9bKci5/IuvpBApZCIWxiHxWph5XcrGT1hNLOmzyJ2fizAZTsRms1mRr85mnWb\n1jFl5hTW/7iesa+O9cqTkpzCzGkzeWfOOyxds5TDiYcZ9fIorzzrf1zPwX0HWbxyMVNmTiFuYZz7\ndd/pBvQZwG/bfuPDxR+y/JvlaK15IvoJnE4nxUsU55HHHuGTjz/xvo8FcTzc9mFK3FQiy7YDTBg9\ngZFvjGTND2uwWCw83+/SJkZLP1nKtInTGD56OKvWrSK4YjDz58zPsh4hhCjMHA5HjvNH0tLSSE5L\nRts05uJmn3xUceV+X1AZXFTBccVPKOCiOMexcRILSdc/ciNDP4VEpdBKjBw3EnBvbbxzx05mT59N\nTNcYFq9Y7JW35zM9L5ULq8SQ4UMYNnAYYydeClYcDgfjp4wn7Fb3iy+79+7OlAlTvOq5pdQtjJ04\nFqUUVapVoXmL5vy07idiusbw156/+Gr1V3zx9RdENnDvTTNtzjQa1GrAmhVraPVoKzp37UybB9tw\n8sRJypUvx9+n/ubbL78lbsWlYCfzPj5KKV4c8SL/avwvAPoN7Ee3Dt2w2+3YbDbmzppLl25dPPNx\nBg4dyA/f/kBy8qUeo0HDBl3TMxZCCH9gMpmwWWzYL+Q8D8UwDOxJdmzYUFbf/MKmU7V76OcsJhJz\niBvSMHOYAPYTgDV94u057HDtQ0ESqBQSF4OBi6L+FcWsd2ahtb6sN+GH735g+uTp7Nm1h/NJ591b\nxqfZSU1NpVgx9+u8i5co7glSACoEV+DUyVNe9VSvUd2r7vIVyvPnzj8B2LN7D1arlbvq3+U5X6p0\nKapUq8LuP3cDUC+qHtVqVGPxwsX0HdCXJYuWEBoe6glCslOjVo1L1wwuD8Cpk6cIqRTCX7v/onuv\n7l7560XVY/2P669YpxBCFBZms5mwkLBczTsxDAMUBJT03YahZ0+dBQfg5AS3cuiKmf/Bwm4C+Iv9\n4NkQzqW1vuaZvxKoFDGJBxN5suOTdOvVjRdffZFbSt3Cpg2bGPzsYAy74QlUrBbvP9BKqct6NzL/\noVdKXfWErc5dOzNvzjz6DujL4gWL6fREpxzLZLzuxUBJu4r+DspCCHGR2WzO9bvIrBarZ86KL3jq\nNeGgTA670bogvSfFca070WYmc1QKiS2/bPE6jt8UT0SViMt6U7Zv3Y7WmlfHvspd9e8iokoER48c\nzfP2VKteDYfDQcLmBE/a6b9Ps3f3XqrXqO5Ja9exHYmHEvngvQ/Y/eduomOir+u6t1W7jW0J27zS\nMh8LIYQoOiRQKSQuTnbdu3svyxYvY+6suTzV96nL8lW+rTKGYfD+u+9zcP9BlsQu4eO5ef/+qIgq\nETz40IO80P8FNm/czI5fd9C/V39CKoXQolULT76gW4Jo2bolY14Zw7+b/5vgisFXrDerd09lTOvx\ndA8Wzl/I4oWL2bd3H29PeJudO3bKZFohhMhCUdgwVAKVQkApRXSnaFJTUml9X2uGDxlOr7696Nyt\n82V5a9WuxYjXRzBjygyaN27O50s+56WRL/mkXZPfnUydenXo3qE7bR5sgzIp5i+ef1l3ZaeunbDb\n7VkO+2QOMLIKODKmte3Qlueef44xr4yh5b9bcvjQYdp3bk9AQEAe3ZUQQtw4oltFs3jh4pwzXvQB\njXiNjb5r0eXk7cl+oKi/PXlJ7BJGvTyKhF0JWCx5Py0q5tEYygeXZ8rMKTlnRt6eLIQoOnL6/ohu\nFU3turU9q0azOp/Tzua//vor//3vfyGCFvzDzRxiMiNonGXmPHhbcmbSoyJ8JiUlhf1/7WfG2zN4\noscTeRKkpKSkMOudWez6Yxd7du3hrbFv8dP3P9GhS4c8aLEQQhQ917thaGpKaj629nKy6kf4zLtv\nv8vUt6bSuGljnh30bJ7UqZTi26++ZdrEaaSlpVGlWhXmLJjD3ffenSf1CyFEURO3MI6YrjGs/G4l\n27dsZ0j/IYSGhxLTNSbbDUPDbw3nwP4DvDToJU6fPl1ALXeToR8/UNSHfvyNDP0IIYqK3Az9nD51\nmm9//taTNm7kOL5a/ZVXWnZWfr6SIc8N4azzrHvopx2/XbGAD4Z+pEdFCCGEKMLyYsNQSuRni73J\nHBUhhBBCeDYMvaPOHcz5eA5rfljD6xNfL+hm+T5QUUr1U0rtU0qlKKU2KqUaXCFvsFJqgVLqT6WU\nUyk1KZt87ZVSO9Pr3KaUaum7OxBCCCEKL3/bMPRq+XToRynVEZgI9AY2AQOBtUqp6lrrU1kUCQBO\nAKPT82ZVZxNgITAUWAl0AZYppe7SWv+e93eRfxyOK+9MLPKGPGchxI3k4oahXbp34detvzJ31tws\nlytn3DD0gZYPsGnDJp9sGHq1fD1HZSAwU2s9H0Ap1QdoBfQAJmTOrLU+kF4GpVTPzOfT9QdWa60v\n9ra8qpR6AHgW6Ju3zc8fV/OmTJE3bBYbJpOMfAohirbMG4aazeZcbRj6xqg3aNSkES+NfIn/Pf2/\nAmj5JT4LVJRSViAK8Axwaa21UupryGajmNxpjLuXJqO1wKPXUWeBupo3ZYq8YTKZcv3CLyGEKKwW\nr7i06+zrk3Keb/JU36cuez1LtVrV3Bu+FRBf9qiUBczA8Uzpx4Hbr6Pe4GzqvPJLZPzc1bwpUwgh\nhLhR3FDLkwcOHEhQUJBXWkxMDDExMQXUIiGEEMJ/LFu2jGXLlnmlnTt3roBa4+bLQOUU4AQqZEqv\nABy7jnqPXWudkydP9ssN34QQQgh/0KZNG9q0aeOV5nnXTwHx2WzC9B3p4oHmF9OUey1Uc2D9dVS9\nIWOd6R5ITxdCCCFEEeLroZ9JwIdKqXguLU8uAXwIoJQaB4RorbtdLKCUuhNQwM1AufRju9Z6Z3qW\nKcA6pdQg3MuTY3BP2u3l43sRQgghRD7zaaCitY5TSpUFRuEentkKtNBan0zPEgyEZSq2Bbj4AqJI\noDNwALgtvc4NSqnOwNj0z27g0cK+h4oQQgghLufzybRa6xnAjGzOPZlFWo7DUVrrT4FPr791Qggh\nROHny40sHQ6Hu/vAgZmTOcQNSXkfV9xQq36EEEKIoiQ/Ngy1J9khDTiGja8JyLHAOexAnm0MJoGK\nEEIIUUjlx4ahZ06cAQM4zVFOsz8XRVxa6zyLmiRQEUIIIQoxX28YarVaL/7oSF/Rm6/kZSdCCCGE\n8FsSqAghhBDCb0mgIoQQQgi/JYGKEEIIIfyWBCpCCCGE8FsSqAghhBDCb0mgIoQQQgi/JYGKEEII\nIfyWBCpCCCGE8FuyM63IN06n06fbPBd2JpPJp7tLCiFEYSSBisgXTqeTQ0cOYXfYC7opfstmsREW\nEibBihBCZCCBirgmV9s7YhgGF1IvYClhwWyRL+LMnA4nF5IvkJaWlvG9GjmSXhghRFEngYq4atfS\nO2IYBonHE7EF2rBY5Y9dZg7D4X6VuuaqAhXphRFCFHXyjSGumsvlwu6wY77JjMWSuz9CJsOELdlG\nQGCABCpZMBvuQCMgKCDXgYrD4cB+wY7L5ZJARQhRZMk3hrhmFovlqn77t1gsmK1mCVSyoNGe53k1\nz9SJ04etEkKIgifLk4UQQgjhtyRQEUIIIYTfkkBF+IXe0b2ZNHJSrvPXr1Sf79d+75O2PNzwYWLn\nxPqkbiGEEFdHAhWR56JbRTNy2Mg8rbN3dG9WLF6Rp3VmtHzpcpo1aAbAR6s/ot3j7a65rm0J2+j4\ncEca3tGQwc8OBiB+QzwPN3w42zJzps/hyZgnr/maQghRVEmgIvJd7yd6s+Iz3wUd10y7/3NL6VsI\nKBZwzdVMGjeJGrVqsPK7lYwcN9KTrpS6YrmM5yeNm8SgvoOuuQ1CCFFUSKAi/IbD4WDCyxP4d41/\n07x2c96d8O4V8588fpL+j/enyW1NeKTxI3yz8huv83v+2EOfDn1oclsT7rvjPsa+MJaU5BTP+ZED\nRvJ8j+f5YOoHTHx5IuePnwcuH/qpX6k+yxYuY3DPwdxd5W7aNm3LD1/+4HWt79d+T5u729Dktib8\nEf8HVmXlv3f993ofiRBC3PAkUBE+4XQ4GT54ODXDalInog4TX5+YZb63x7/NgKcHALAibgW7d+3m\nvPM8bR9vy4JZC1i2cBmPPvAof5/8GwCtNbPfmY3WmnHDxrFn3x5eHP8iLdu2ZNgzw9j8f5uJuj2K\ntSvW8ngCsqSxAAAgAElEQVTLx4nfHE9w1WD6vNCHn3/8mQkvTwDgi6Vf8PWar1m3dh2fxn5K2+5t\nKVGmhFfb1n2zji7tuqC15vVhr2MOMBP7VSx333c3w58dTlT1KJbELqFP1z4MenIQASUCSDOn4VAO\nPvvoM7TWrFm+huVLl/O/p/93Wd1Rt0fl9WMXQogiRwIV4RNxC+OwWC2s/G4loyeMZs67c1ixzD3c\nk3GII7JBJNsStoGG4ErBBFUIolTZUmibpmOPjsx/dz6HDx1m6sdTad2+NQs/XMiCeQtQStGyXUta\ntm3J66+9TuuOral1Zy2Wxy0HYNLoSdhsNhatWUS1mtX4aP5HDBk9hJVLVvJ/3/8fo18eTUSVCEqV\nKcXjTz/OsiXLMFsvbZqWeCiREUNH0KVbF5RStGjTgp1/7mTt6rX0e7EfyReSwQmz3pmFTdmoXLUy\nU+ZP4auNX3FzqZtp/J/GmEwmmrdoDoA1wMoXG7/wekY5DQUJIYSQQEX4SKXQSowcN5Lbqt5Gm/Zt\n6N6rO4sXLAZg5vyZtG7bGoDI+pGcP3+e5ORkakfWJmFzAk/0eIL4n+OpG1WXxAOJlKtQjkphlQD4\naO5HdO/VHYAmzZrw3ODnuL3G7cTOi6VuVF0O7TsEQNVqVal1Zy2q3F6FPv37cPTwUcoGl8XlcrHg\n/QXcfe/dVKlWhdvvuJ2Y7jE0atrIq/3rf1zPk72f5KFHHwKg2YPNeOZ/z/Dpok8pXqI4NwXeBBpa\nPtwSm8XGnQ3uJLhiMKXLlEYpRUT1CACsttxv3pbRoGGDmDQj96ughBCiqJJARfhEZIPIy44PHTzk\nmbB60c0lb6Z6zeoknUvi3Nlz2Gw22nVsx66du7Db7WiX9tR14fwFTh4/yZ2Rd3rVUTeqLn/99ZdX\nWqlSpTw/ly1XFq01/5z5B4AjR45Q+87aABQvUdxdx111vcqfPH6S2TNm0/SupmitGT54OKNfGc3f\np/4mLS3N0xtSs3bNa3k8QgghcsnngYpSqp9Sap9SKkUptVEp1SCH/M2UUvFKqVSl1C6lVLdM57sp\npVxKKWf6f11KqWTf3oXwpah/RZF0LolfE34lskEkgSUDqVylMt+s+gaz1Uz9hvWzLPdr/K+XHYdF\nhAEQWjmUXTt2kZqSisIdVOz6bRcmswlbMVuObbLb7fR5rg+LPl+EUorBLw8mbnkcn639jICASyuC\nihcvzq1VbmXntp1e5RP3J3p+NplM6EwRmsNw5NgGIYQQPg5UlFIdgYnACOAuYBuwVilVNpv8lYEV\nwDfAncAUYI5S6oFMWc8CwRk+t/qg+eI6bPlli9dxwuYEwsLDIItpGVENojifdJ4zp87wz7F/OLD3\nAKWDSvPtym8xlEH9Bu5A5aabb6JchXJsTdgKwNcrvuaLRV+w+afNJP+dzI5tO2jdoTVKKZre35SA\nYgGMGDCCfbv3gQNiZ8fSOro11W6vxm/bfvNqw/Yt272OKwRXYP++/YSGhwLuXpnQ8FDPcUaPPfEY\n+/fuZ+rYqRz86yBGssG2n7cB7nkopUqX4sL5C6SmpnrK/LHzj6t8okIIcWPydY/KQGCm1nq+1voP\noA+QDPTIJv8zwF9a6xe01n9qracDS9LryUhrrU9qrU+kf0767A7ENTmceJhRL49i7+69LFu8jHlz\n5hHdJTrLvJENInE6nbjMLkqVLkXXVl3Ztn4bLrOLcqHlCKsc5snbtWdX5s2eB0D77u2ZPWU2u7fs\n5tSxU4x7dxxhlcPQWmMLsPHOwnc4d+YcfTr0gVSoVa8WQ8YModMTnVj/43r27t5LSnIKn3z0CRt/\n2ui5hlKKJvc0YeWylcx+ZzYoOHH8BGtXrmXG2zM8eS4KCQth/KzxrFu9jk73d8JINrinxT0A2Gw2\nat9Zm2LFizFt4jQSDyayevlq/9xHRggh/JDPAhWllBWIwt07ArijC+BroHE2xRqln89obRb5b1ZK\n7VdKHVRKLVNK1cqjZos8oJQiulM0qSmptL6vNcOHDKdnn5483CbrnVkDSwZSPao6ZcPKMvadsXz/\nx/es3LwSVUxR/1/ewz4xXWPo0r0L5auVZ+7cuQQGBzJ98XRWbV5F81bNPdcHqHJ7Fd6Ne5e1W9ai\nAhWPP/M4xUsUp069OgwfM5zTyafZunMrP2/4maf6PuW5xhcbv2DY68N4e+bbbFy/kYCyAUyfNp2F\n8xYSUikEgO9+/w5luxSs3PvAvSz9aSnr/1rPTWVvIvlCMuUrlsdqs1IyqCRj3hzD+h/W0/GRjny5\n6kv6PNcnT5+5EEIUVcodO/igYqUqAoeBxlrrnzOkjwfu1VpfFqwopf4EPtBaj8+Q1hL3cFAJrXWa\nUqoRUBXYDgQBQ4B7gVpa6yPZtCUSiI+PjycyMjKrLOIqGIbBvsR9BAQFYLXmblWLYRgcPHIQW0lb\nrssUJovnLeaOencQVCqIrZu28uYrb9KpRyf6DMldQGIYBvZzdsJDwq/qmaadTSMiNKJIPlMhhH9I\nSEggKioKIEprnZDf17fk9wWvl9Z6I+Dpp1dKbQB2Ak/jnguTrYEDBxIUFOSVFhMTQ0xMjA9aKm4k\nh/Yd4v0p73Pun3MEVwqm6zNd6f5s94JulhBCXJXY2FhiY71fynr27NkCao2bLwOVU4ATqJApvQJw\nLJsyx7LJf05rnZZVAa21Qym1BXcvyxVNnjxZelSETwwaOYhBI+XdPEKIwi2rX94z9KgUCJ/NUdFa\nG0A80PximnJPHmgOrM+m2IaM+dM9mJ6eJaWUCagDHL2e9gohhBDC//h66GcS8KFSKh7YhHv1Tgng\nQwCl1DggRGt9ca+U94B+6fNYPsAdtEQDD12sUCn1Cu6hnz3ALcALQDgwx8f3IkSecDqdaJf33DCn\n4cThcGAYRq7rMQzD8/EnJpMJs9mcc0YhhMgFnwYqWuu49D1TRuEewtkKtMiwnDgYCMuQf79SqhUw\nGegPJAI9tdYZVwKVAmallz2Du9emcfryZyH8mtPp5PiJ4xhO7+DC4XBgJBmgwWLN3V9Lh8OBPckO\nCqwW/5lMa7PYCAsJk2BFCJEnfD6ZVms9A5iRzbkns0j7Afey5uzqGwTIZIBCyulwFnQTCpTTcJKS\nloKpmAmz5dIXuclpwqIt2Erach2omA13+YCSuV995WsOhwP7BTsul0sCFSFEnih0q36E/3A4cr8N\nvNPpRKFIO5flnOgbhmEYpJxJwXazDZfF5XXOZrZhMpk8W/7nyOTufbFarX4TqAA4ubGDUSFE3pJA\nRVw1k8mEzWLDfsF+VV9KZYqVweVy5ZyxCDMMA6O4ga345T0nyqRwJDtwkPsA0GZxBzdCCFFUSaAi\nrprZbCYsJOyGDzquhWEYWG3WPBuukYmrQoiiTgIVcU3MZrN8QV4jq8Xqd8M1Qgjhr6TPWAg/Et0q\nmpHDRhbItRMPJhIaFMrvv/0OwIafNhAaFErSuaQCaY8QQoAEKkIUKtGtolm8cLHP6s/4VujMxxt+\n2kCjOo18dm0hhMiKBCpCCI+cXlKaOZARQghfk0BFCD/jdDgZPng4NcNqUieiDm+OeTPLfKOHj6Zb\nh26e49nTZxMaFMr333zvSbu73t0s+miR53jhvIU0a9CMKuWr0KxBM+bNmee7GxFCiDwggYoQfiZu\nYRwWq4WV361k9ITRzJo+i9j57reZZuzRaHR3I37Z+IunF+Tn9T9TpmwZ1v/kfpXW0SNHObj/IE3u\naQLA0k+WMnHcRIaNHMb3v3zPiyNe5K2xb7Ekdkk+36EQQuSerPoRws9UCq3EyHEjAbit6m3s3LGT\n2dNnE9M1hsUrLs1PadikIUlJSfy27Tfq1KvDxv/bSN8BfVmzYg0AG37cQHBIMOGVwwGYOG4ir459\nlRatWgAQGh7Knzv/5KMPPiI6JjrHdjVu2pgN27N9P6gQQviEBCpC+JnIBpFex1H/imLWO7PQWnv1\nqJQMKkmtOrVY/9N6LFYLAQEBdOnehYmvTyQlOYWN6zfS6G735NeU5BQO7DvA4GcHM+S5IZ46nE4n\nQUFB+XNjQghxDSRQEaIQa9y0Met/WI/NaqPR3Y0IuiWIqrdX5ef1P7Pxp4306d8HgAsXLgDw1rS3\nqBdVz6sO2Q9HCOHPJFARws9s+WWL13H8pngiqkRkueKmcdPGfPLxJ1itVprd3wxwz11ZtmQZ+/bu\no3HTxgCULVeWChUrsH/ffh6NftTn9yCEEHlFJtMK4WcOJx5m1Muj2Lt7L8sWL2PurLk81fepLPM2\nbNKQ80nn+XrN156gpMk9Tfgs7jPKB5cnokqEJ+/glwbzzqR3+OC9D/hrz1/88fsffLLgE2ZPn51t\nW3JariyEEL4mPSpC+BGlFNGdoklNSaX1fa0xm8306tuLzt06Z5k/6JYgatxRg79P/U2ValUAd/Ci\ntaZJ0yZeeWO6xlCiRAlmTJnB2FfHUrxEcWrcUYNefXt5XT9ze4QQoiCpG+E3JqVUJBAfHx9PZGRk\njvmF8BXDMNiXuI+AoLx5KaG/MQyDtLNpRIRGFMn7E+JGlJCQQFRUFECU1johv68vQz9CCCGE8Fsy\n9CPyjdPpxOVyFXQzCpRhGBiGgckwYTKZZMWNEELkQAIVkS+cTieHjhzC7rAXdFMKlOEwSDyeiC3Z\nRomAEoRUDJFgRQghrkACFZEvXC4Xdocd801mLJYb94+dyTBhu2DDXMKM3W7H5XJJoCKEEFdw435j\niAJhsVhu+EmWFqsFs9mME2dBN0UIIfyeBCqi0CqMc14Mw8BhONBmjdNwYhhGntUtc16EEEWRBCqi\nUHI6nRw5egS7s3DNeXE4HBw9cRRt1ThTnBgOI896mKwWKyEVCnbOi8PhKLBrCyGKJglURKHkcrmw\nO+2YS5gxWwpPL4LFaaGEUYKU1BScJieKvNlQzel0Yk+yk2JLKfChNZvFhskkOx8IIfKGBCqiUDNb\nzAX+xXxVrFCpYiXsdjv2JDvhFcPzZHKxYRiknUsjolLBb7QmQ1BCiLwkgYoocNGtoqldtzYjx40s\n6KbkC7PZjNVixWVx5enkYpfFhdVqLfBARQgh8pL0zwq/F90qmsULFxfItR++72Fi58dedbneT/Rm\nxWcrfNAiIYS4sfi8R0Up1Q8YDAQD24DntNabr5C/GTARuAM4CIzVWs/LlKc9MAqoDOwCXtRar/ZF\n+4UoygrjyqnckOEnIYoOnwYqSqmOuIOO3sAmYCCwVilVXWt9Kov8lYEVwAygM3A/MEcpdURr/VV6\nnibAQmAosBLoAixTSt2ltf7dl/cjfMfpcDJ88HA+/eRTLBYLXXt2ZcjwIZflGz18NHt27WHOgjkA\nxM6PZcqEKUx/fzqNmjYC4NEHHsWcZqbpfU3572P/5Z1J7/Dnzj9xOBzcXuN2nn/peW6vdbunzplT\nZ/LFZ19w+tRpgm4Jovl/mzPk5SFEhURBcZj4+kTeGvsWSil++eMXALbGb+WdSe/w+2+/U6p0KZo1\nb8Zzzz9HseLFrvreG9VpRK++vej5TM9reXTXrCjvFmyz2AgLCZNgRYgiwNc9KgOBmVrr+QBKqT5A\nK6AHMCGL/M8Af2mtX0g//lMp1TS9nq/S0/oDq7XWk9KPX1VKPQA8C/T1zW0IX4tbGEdM1xhWfreS\n7Vu2M6T/EELDQ4npGoNSl1bGNLq7EYvmL+LiW7+3xm+lVOlSbN60mUZNG3Hi+AkOHzpM7aq1Abhw\n4QIPt3uYF+u8iEu7+PiDj3mu13OElw+nTec2FAssxsL5Cxn/9ngiqkbw98m/2fXHLgCUUpQsVZIu\nT3Whbfu2njYcOnCIZ596lmcHPsvIcSM5/fdpJoyZwITRE3j19Vc9ZfNSozqNePu9t2l0d6M8q7Oo\n7hbscDiwX5Bdf4UoKnz2r5NSygpEAa9fTNNaa6XU10DjbIo1Ar7OlLYWmJzhuDHuXprMeR69rgaL\nAlUptJJnMu1tVW9j546dzJ4+m5iuMSxecWl+SsMmDUlKSmLH9h2ULFeSLb9soXuv7nz39XcAxP8c\nT/ng8tiK2QBo0KiB13Veeu0lvlz1JUlJSQAcP3qcsuXK0qBxA8xmMxWCK1CrTi1PfrMyU6JECUqX\nKe1J+3D2hzz0yEN06toJgNDwUAa/NJhej/di2MhhWG1WZs6fmfcPyUeK4m7BsuuvEEWHLyfTlgXM\nwPFM6cdxz1fJSnA2+UsqpQJyyJNdnaIQiGwQ6XUc9a8o9u3d5+k5uahkUElq1anFhv/bwN49e7HZ\nbLTr2I5dO3eRmpJK/C/xnrocDgejnh9Fw8oNaRDWgEbVG3Fv1L2kJKdgT3MPd9zf8n5SU1NpeU9L\nHrzrQRre2pD2zdrz8w8/e103fkM89SvV53zSeXbv3M2Kz1bQpHYTokKiaFK3Cf169kPbNc3rNOfH\nr3+k3T3tuLvK3Qx9eiipKaksj1vOww0f5j+1/sObr7x52X0lJSXRr0c/qlWsRlSNKD6c/WEeP2Eh\nhCicZNWPKHQaN23Mhh83sO2XbdxV/y4CSwZSuUpltvyyhYRNCUT9KwqAFXEr2LRhExF3RNC1X1fM\nTjM9e/Yk6JYgT6BQIbgCS9csxeawYbFZCKwQiCqumDp26mXDNxePk1OSaduxLROmTMBkMjF9znQ+\n+eIT+g/ujz3NzicffML4meN5Z+E7/PJ/vzC452A2fLeBqR9PZfS00Sz9eCnfrvrWq+6Z02ZS+87a\nfPnTlzw78FlGDB3Bj+t+vOzaQghxo/HlwPQpwAlUyJReATiWTZlj2eQ/p7VOyyFPdnV6DBw4kKCg\nIK+0mJgYYmJicioqfGzLL1u8juM3xRNRJSLLL+jGTRuz6KNFGBjc/eDdgLtHZs2KNRzcf5D6Deqz\n+pPVBFcK5ujZo/R7uh8tH2kJLlj+yXLOnD5Dz5d70rp9awASNiRw6sQpVm5eyfnz52n333YMfXko\nE4ZNwGQx4XJ6r4qpUasG+/bs49E27tHG4JBgKoZWpFSZUjgdTl4a/xIhYSEANG/dnNWfruar7V9R\nrHgxIqpFUL9JfeLXx9P03qaeOus3rM8z/3sGgIgqEWzeuJnZ02dzT7N7ANiwfUNePGYhhLii2NhY\nYmO9t2Q4e/ZsAbXGzWc9KlprA4gHml9MU+5vnebA+myKbciYP92D6elXyvNApjxZmjx5Ml988YXX\nR4IU/3A48TCjXh7F3t17WbZ4GXNnzeWpvk9lmbdhk4acTzrP+h/Xe4Z56jesz+rlqylbvixhlcMA\nqB1Zm/DK4az8fCX7/9pPUOkgDv51kIBiAZ66li9dzvJPl1O6XGlSU1NZ+flKihUvxj3N3QFC6TKl\nSfglgX9O/+Mp061XN7Zt2cbcmXPRaI4mHmXdN+tY9fkqihUv5glSAMqULUPFsIpeq4FKlyvN6b9P\ne93TxV6gjMd7/txzLY8yT0S3imbksJG5zh8aFMqXq77M83ZMGjeJB5s+mOf1CiGyFhMTc9n35OTJ\nk3Mu6EO+nuo/CfhQKRXPpeXJJYAPAZRS44AQrXW39PzvAf2UUuOBD3AHJNHAQxnqnAKsU0oNwr08\nOQb3pN1ePr4X4SNKKaI7RZOakkrr+1pjNpvp1bcXnbt1zjJ/0C1B1LijBsdPHye8cjgAkfUj0VpT\n/1/1vfKOGDuCsa+MpUu7LgQWD8RkNnlNjA0MCmT7lu2cOHaCTo92omr1qkyZOYXAkoEAtGzdktVr\nVvPy4JdxuVxoral2ezXmfDyH8SPHo12a555+jvCIcMJCwrBYvf9KKaUuW1GjUGiX9xyVwia6VTTt\nO7enXYd2njSHw5Gnb4MGcLrck2Iv1vv2hLc5fOgwb057M9syhmFgOAxPGdlTRYjCzaeBitY6TilV\nFvfmbBWArUALrfXJ9CzBQFiG/PuVUq1wr/LpDyQCPbXWX2fIs0Ep1RkYm/7ZDTwqe6gUXhlX9bw+\n6fUr5Lxk1bpVHDxy0HNcMqgkm3d67yP425bfqF6zOvOWuPcLnPb6NH786kfivo3z5GnWvBnFLMUY\n0G0Ay9Yuo0y5MgCsX7cepRS3RtxK7LJY9u3eR/tm7Tl1/BSBJQOpWbsmj7R9hB0bd7Bk5RIqhlZk\nedxyNq/Ldi/DK0rYnHDZcdXbq15TXflCwz9n/+Hg0Uv/D06eOel1nBfOJp3Fbtg99Z5NOsuF5AtX\nvI7DcGBPsoMGq9Uqe6oIUcj5fPMErfUM3Bu4ZXXuySzSfsDdQ3KlOj8FPs2TBooi69jhY0x+bTLt\nHm/Hzu07iZsbx6CRgy7L1/DehoRHhPNq/1cZ8MoAzied593x73rlCYsIo0JIBd576z36DO7Dgb0H\nWPDeAsC9WZ3DcLjnsmj3F+VFLlcWadqFS7twOBw4HA601mzeuJnpk6fzQMsH+PG7H1n5+UrmLpqb\n6x6KzL0IV1PGZFw+Aqy1xrAbvDToJZbGLcVqtfL4k48zaNggzz04nA7MJcxYzO5/Rs6eO8vQZ4eS\nsDGBsuXL8uywZ7nvofs8dU5/Yzrr1q7jxNETlClXhhZtWvDUgKe8Aoh5M+ax6INFpKWm0fyh5txS\n5hZMJhMBge7hOkuABZP10nFWzIa7voCgAJRSsqeKEIVc0dnlSYgMlFK0jm5NWmoaXVt1xWw207lX\nZ9p2aZtl3okfTGTU86Po1robFUMrMmT0EJ7r8pxXnufHPs/0sdPp0qILVWtVpUPvDrw17C2O/X0M\np9XJmXNncGkXh08c9pQ7d+EcdofdKy05JZnUtFSOnjwK2j280b5rezZu2MjkNyZzU+BN9HuhH+G3\nh+e6hyJzL0JuGIZB4vFEbBdslw1ZpaWlEbcwjtaPtea92Pf4c8efvDniTYoFFqP1Y60xDAOtNRaz\nxVN21qRZ9H+5P0PHDmXF4hUMf244cXfEUblqZcA9zDZq6ijKli/Lnj/2MGbIGAKDAun6TFcAvvzi\nS96f8j4vjnuReg3qsWLJCha9v4jQW0M91zCZTJhMpsvam5FGe+0NI3uqCFG4qcz7ORRFSqlIID4+\nPp7IyMgc84u8ZxgG+xL3ERAUkCebixmGwcEjB7GVtOXLZmUOw8HhE4cxFTd5ehCutz77eTshFUKw\nWq6//Q7DQVpSGmEVw64qUDl09BABgQGXffE/0/EZ/jn9D7FfXZr9P338dH76+ic+WvMRaefSQMNN\npW/CYrVQv1J92ndrz9DXh3ryd3+4OzXr1PRKy+ij9z7iqy++Yv6q+QD0eKQHNerW4IUxL3jVYU+z\ns/DLhbl+FoZhYD9nJzzEPX8p7WwaEaERRW5TOyHyS0JCAlFRUQBRWuuEnPLnNelREeIqWMyWPNlu\n3vNbv8V6xd6Bq67Par2qL2SLxYLZar58ErBJUSeqjld6vQb1iJ0Ti9mU9RBKnag6Xsd1o+qy6/dd\nnuMvP/+SRR8sIvFAIikXUnA6ndwceLPn/L49+4juFn1ZHb+s/yXX9yOEKHpkwzchhM9t/2U7w58b\nzj3338OU+VNY+NVCevTv4ZlTs2j+IpyOrIdozpw+w47tO/KzuUIIPyI9KqJQcDqd7omp6QzDwOFw\nYDbMKHy/a6vD6Z74anLkTWzvNJzu5bwO9xe1Mim/m+z525bfvI63x28nPCI8211yf43/lYcec+8k\n0PuJ3uzfsZ//PPgfHIaDrZu2UjG0Ik/0ecKT//CBw56JxlWqV8HutLP5p808+MilfVO2/7KdgIAA\nXhzwIh/EfkCp0qVybLfn2aYHQYZxdZOMr5cshxYib0mgIvKVw+HIOVMmTqeTo8ePYnfYL9VjODh6\n8ijWQGu+vPnX6XBy/MxxrOetmC3X/yXkdDrRKRqny4nZZMZmsVG+XPlsh1VyrC/Tl3NuXCnY0y7N\nscPHmDhiIm07t2Xnr+5VUwNeGeBeXeQwQOMJtAC+WvEV1e+ozp0N7iRxbyKnj5+macumHDh8gICS\nARw7fIzYebFUqVmFMS+O4ezRs5jNZg4cPkDp4NI80PYBli9aTtjtYdSsW5Mf1vzA3j/3UiG0Ah2e\n6sALg19gxLgRXm19uuvT/G/I/7xeJOlwODCS3O1D4Z5krMiTuUC5IcuhhchbEqiIfGEymbBZbNgv\n2K96FYbhMDh/5rx7Ims+BCVZ0Whw4f7yc11+fuKrEwmPCKf9k+1zVZ9ZmTGXMJOUloThMtCpGrvD\nfs33l/HLOac5Lx1ad6BDlw60iW7jDvZSLg/27Iade/97L6fPnKbbw90wm8206tSK+vfV58jxI6Sc\nSwENxY3iWCwWlFJ06NmBFUtWMGH4BLTS1GxaE2sZK3+n/E1EVAT3PXYfcybNwWE4sNxsoe6/6/Lb\nT7/xd8rfADzc82GK3VSMj9/5+P/Zu/M4m+r/geOvz91mNWNssxnMoIiokeTrVxGhLG2kUdGCipAi\nlJCi7C34ZldCpbRI39ZvSiV9s2RNdsYw9mXWe8+9n98fZ+aaOzOWGbPh/exxHznnfM45n/OZO/e+\n57Picrr4V+t/cWfXO1nzyxqat2tO83bN8zyL8ldYg6zYy50JQpRLobXGEeLwBjV+IUXTift8DMOQ\n4dBCFDEJVESJsFqtxETF+DTfXCiXy/wCzjliyOVyYbPb8h2xUhxchgubzYYj2IHVnvcLyM/mR3Bg\nMNFR0fmeP/TJodze4XZua3dmXhG3282Ro0dw2B24be6zXvtC5Pxyzi6PZZ8tY/LYyXy/8nvftFlf\n7o4QB7Z0G/bgvB16NyZu5NmJz1I5vDJ9Xunjc8zisuB2u0HjzfPnmz4HoP0j5vpJQ/oPoXJ0ZT75\n+BN+++03rFYrLVu3ZOoPUwEYM2IMDVo0oM+rZ67tdru586E7efTZR31rP84MAsr73Chvp2TvPq3Q\nNo3NbkOhcNvdBe5kfDFkOLQQRUsCFVFirFZrof/KzP6iyfllY7PZQGXVdhQzj9ucoC2/ydHAbCrx\nuD24Xfl/SeV33HAZZGZkopRCuzSGu+DNYjnzp3L8Z7gNNBqlVL59eFQ+//mwnEmXz8lgMZ8JlX8a\njcNbgvAAACAASURBVOa3Fb9xa5tbefn1l9m1Yxez/z2bKhFVaNGqBRar2Y8jd02OJ7/qqnORRaWF\nuOxJoCIuSRaLBYfVgTOt4E1JBeXxeDiQdICkI0nYgmz5zqOSeTqTk0dOMnHYRP730/+wWq38X5v/\no8MDHbzHjyYfJXFXIkveW8Kh/Yfo8XwPjh47yqpfV/Htp9/y5HNPUr9hfQBGDBxB6/atada8GVpr\n/vPZf/h1+a+knE4hIiqCjvd3pF6DegAcPXKUEc+O4PE+j7Pyx5Vs/2c7vZ7pxZSxU1BK0aR+E5RS\ndOnehfu73Y/hMUg+nMyLg1/kt19/I7BiIB07d+SWlreceSANh48dxmPNGzgYLoOMlAzQ4O/yz7e5\nyulxEloplNva3YbVZqVW3Vrc3PJmvvriK1q0asGwUcOK4kfDux+9WyTXEUKUXRKoiEuS1WolKjKq\nUE1JBeVyuXBmODEcBv7l/fNtavK3+fPHd39w+323M+n9SWzbvI1pI6dR8+qa3H7v7QTYA6hQoQLR\nsdHc1Pwm3nzxTaKqRWELtHEo8RDBQcEk70+m7V1tOXL4CEcPHKV56+aER4XzxQdfsPw/y+kzpA+x\ntWP5fun3zJg4gykLpxAZHYnDzwEGfLn4S3oM6EHNq2qiLIpez/ViwfQFzPhkBlprAgIDcPg7UEqx\n9JOlJPRIoNW9rVi3bh3zZ8ynXqN6RERHmA+kwBZgwx6Ut7nE2/QDOIIcWOx5a5ksfhaqR1XHHmT2\nf3G73VSLrcby75ajtT7ryCEhhMhN5lERlyyr1Yrdbifh7gRGDx/t0zxU1C+b3YbD30FAQEC+L4vV\nQnh0OH2G92H7zu3MmDWDu7rdxbIFywgICGD8wvHc2eVOAgICaNSsEemp6STtTsLf359tm7bR5r42\nbPlrC/4B/mzduJVK4ZWoXrM6/gH+fP7h53R5rAst27Uk7qo4ej3Xi5p1avKfT/6Df4A/fgF+oODe\nh+/l5lY3E1UtisiqkQSHBKMsioqVK1KpSiWCgoO8HV9vvPlGOnTuQER0BB26dCAkJIStG7eaE9pZ\nbSz4egERURHe7Zwvi93ibcaz2q3YrfY8L2VRKKXMyeSsNp8mv1eff5X50+eX4jvnwiTuTaRqaFU2\nb5T1ToUoTRKoiMtep3adWLxw8fkTXqQ619Xx/luhqHt9XZJ2J/HNsm8Y1G+Q91hQuSDi6sSx4Y8N\nJO5MxGa3cesdt7Lzn51kpGewcc1GjFMG08dMJy01jaOHjlLvuno+96rXsB57d/quA1S7bu0Lzmv2\n+jvZQsNCOXXiVAGe9vz27vHN3+5du4mIjDhvv5KhTw7luy+/K9K8FFbOmp/swEWIy53b7fbOP5Rr\nHiKbUspeAi+fzozS9CNECcjd1NGgSQPWr1pPmiuNqxteTWBwIDE1Yti4biPrV68nODj4LFcyrfhi\nBYGhgT77/AP8Lzg/ufvZKKXw6KJpRvtkzifs/t9udLjmsw8/45YWt7B7126WL1hO2wfbsnv7bm9a\nt8dN9npjbsOcCwYPaLcu1Jw7gHdVarfLjVYaw2UU6Fo5J2zLvRaaNFmJy53b7WZf0j6feauSjiSB\nHShHJJU5XuyZOIVTKZWotXaDBCriMuE23AwbOIxPPvwEm81Gt8e7MWjYoHzTnjp5ileHv8q3y77l\n9KnTxNaMZejIobRs0xKAZZ8vY+KYiezeuZsqEVXo1qMbLdu19J6fkpLCtDem8cdvf+Byubj2umtx\nZjrZ+tdWn/tsWbuFqBpR+X65XXvjtXyz+BsMZXDtzeYaOfXj67P86+Uk7U3i6tirAQgMCqRilYps\nXLuR+tfX956fmpJKTO0Y73Z+97Db7N6+JOficXu8o5IM1/m/0A234Z3u3nAZeeaV8bg9aDTxN8Tj\nzHAy9pWxKIvCz8+PBg0bsOufXRhOg9lvzebXH37FYrPQtHlTWt7VEiPVINOZydETR0k8kAhAcmIy\n7779Lnt37KVyZGUe6PUAk1+czJMvPsl1Ta4DzFluP5rxETu27sDh5+C666/jyeeexD/AH1eqC41m\nwawFLP14KSePn6R6XHWeGPAENza70ZvvzRs2M3HURPbu3EudenXoN7BfnnK9EhZxFVc2j8eD03Bi\nDTozKs9RzgF+QAROWpBZrBk4jY1VODiABZBARVw+Plr4EQndElj24zLWr13PoH6DqFqtKgndEny+\nbLTWPHjvg6SnpTNl9hSq16jOjm07vMfXr13PU488xcAXB9Lhng78uepPhgwYgoFB/G3mytvjXx3P\nwaSDvDz2ZQICA3i+5/OkHUrD3+7PzNdnEhIRguu0i6XvL6XXC73y5LVtfFsGvzqYtNQ0/lr1F52f\nNieJq9+oPq8NeY0KlSvg8HfgNtxMGzWN0/tPM2/sPH75+hf2H99P+w7tyUjLACv0fqw3I0aPQGdo\nnun0DK5MFxUrV+T/2v4fG7ZvIP14Ot1bdOetT9/Cz9+PNSvWkLw1mc1/bqZDlw7YLDbG9h3LicQT\neK734EpxsfbXtSxdsJSkPUmUr1ieZq2a0e6hdlgsZkvxqZOn+HDqh2z8cyNut5saV9Wgy1NdiImL\n4ddvf+WzGZ+Bgt/n/w4KHuj3ANfdeh2v93odI9XAk+FhxbIVNLutGU8NfIqk/Ul8OvdTQkJCuK7R\ndeACZSgzePJ4mDpiKpUiKjF03FDS09P5ZPonZhq3mcaZ6eStoW8Rd00cL054kdMnTvPexPd4d8q7\n9H25LwCfffwZi+cvZujrQ6l9TW2WfriUoX2H8sH3H1C1elXS09IZ2ncoTW5uwkuvv4Q73c3LQ1/O\n87OTGhVxpche5DT73yjAhpvKFH4ehQsnTT/i8hNdNZqRr40EIK5WHFs2bWHm1JkkdEtg8Zdn+qf8\n/N+fWb92PT/9+RM14moAEFP9TM3EzKkzubn5zfQb2A+A2JqxbNm0hQ8XfEj8bfHsT9zPql9XMfmd\nydStVxeA6jWqs+ngJq5pfA2ZGZksnLSQzMxMuj7VlTvuv4NBvQb5rF8D4B/oT+xVsRw7eYzwmHC0\nU5s1JtpcMfjonqN8v+R72tzfhmlfTGPeW/P4ZdkvWAItrFm5htjasQQEBnDq1Cn+t/x/4IS4ZnEE\nhQXR/bHu7Px7Jw898xCTh0/m16W/0qVFFx566iFSj6RisVpI2p2E1WqlQoUK7NywkwpVKxAcEMzR\nvUeZ+9pcnnzpSeo1qseBPQeYMmIKwYHBJPROYOG8hXzyzifUb1CfF8e+SFhUGN998h0v93iZEe+M\noEPnDpw4cII1v6yh79i+WAOtlAstZ3as9VipEFoBP4sf4VXC6Te4HwePHKR+4/qkHk1l1beraHtn\nW8bPHe8dWbV6xWqO7TvGpPmTKF+hPADlA8sz7PFhVAyrSExUDF8v/hpPhoeXJr5kjoACQoNDefmp\nl3ls4GMEOAL4YPYHPPL0I7S5uw0A/V/qz+rfV/PR3I94fvTzfLf0O9AwbNwwPJkeqkVWI/lgMi88\n+4L3Z1a1WlX2ndhXVG9ZIcQFks604rIQ3zjeZ7vRjY3YtWNXnqr6zRs3Exkd6Q1Sctv2zzZuuOkG\nn303NLmBxL2JoGHfnn3Y7DbqXHOm46zFZiE0JpRrml3D0yOfps/rfQiuGUy3Z7qdM89vLXmLCYsm\neLfLhZTjqz+/YvDowQBUjqrMEy88QdXYqgydMJT7e91PTFQMUxdNZdoX04i7Ng4AZ4aTmLgYqtaq\nil+gH1ddexVtO7clODiYlya9RGhYKMNeH8ZDvR5i/ar1PPrsoxw/ZDYzb9u4DY/bw/RPp9PtqW58\n8M4HPPDkA7S+tzXR1aO54ZYb6PZMN75e/DU2u40jSUdwpbp4bsxz1KpXi5i4GHoN7YXVamXTn5sI\nDA4kqJw5uqh8pfKEVQkjIDAAm93G2A/H0rBpQ5RFUefaOtjsNmw2Gza7jasbXs2hxEPmJHB2m/d1\nYN8BKkdVplJ4Je++a+KvMQOfrLRJe5KoWbcmgcGB3jT1b6iP9mgSdyWSnprOkeQjNGzc0Kf8r2t8\nHbu27QJg9/bd1K5bG7vjzHDsRjc2OufPTwhRMqRGRVxR/P0vvMNpvs5S86+1ZtWPq/h49sd4PB7c\n6tx9Q44eOcrwZ4azft16ykeWp9NDnWjf3px+PvlgMuvXrqdh44YM6juIrVu20ve5vmS6M9mzfU+e\neUhubnszn837jF8+/oXQ8FB+++43mtzWhCEDhlCzdk3qN67P+lXrOZV+ip1/7yQxLRHnESfDnxtO\nXNU4rmpwFQ4/B+vXrWft72vZuHojcyfPRXs0Fou5vpJhGHz1+Vd8+8W34Iaut3QFwO6XNU+Ky83R\n5KMXV7ZCCJEPqVERl4W1f6712V79x2pia8bm6VNQt35dDuw/wK4du/K9Tu2ravPn73/67Pvf7/8z\nm4cUVKtWDcMw2LJpi/e42+Xm1JFTlA8rz9vvv81t7W/Dleri60+/BvLv1zD/3/O5ueXNvPzGyzS9\npSnT35jOvr2+zQr//P0Pd99/NzPfn8kNTW7Ic41slSMrM+vbWdRpWgeL1cLUl6cyqOsgb21SgyYN\nWP/Hevb+s5eqNasy/b3p1L+hPvu27eOrj7/i2sbXnrmYB8rHlGfIW0MY894YajWrRfXG1Zn97Wxu\na3sb9RvUx+qw8trc1xgzewxvffIW076YRqXYSjTv2Pysecxt60bfjsfbNm4jPCY8T1lVjavKkQNH\nOHH0xJlz1/ueG1Mzhp1/7yQz40wfv02rN2GxWqgaV5WAoAAqhVdi3R/rfM5b9791xF1t1krVqF2D\nbVu24XKeWQ169R+rL/h5hLgibGIcYxlR0reVQEVcFvYn7mfUi6PYsW0Hny3+jLkz5tKjd4886W5q\ndhM3/utGej7ck59//Jl9e/bx43c/svz75QA80fcJfvnpF94Y9wY7t+/kowUf8d7s9+jyYBcAoqpG\n0fT/mvLmuDfZtGETO7btYO+evdgcNoZPGE50tWjqNKiDPdDOpws/BWDc9HHc3v52n3zccvst3N7h\ndsKjwrn7obuJrRnLl0u+9EkTYAug2c3NCI8IJ6xCGEm7k7DYLfkGPg4/B5VjKhN7fSxj549ly9ot\nZJzOAMwRRnu37SXlSAr/avkvIiIjaNaqGeEVwjl1+BR1r6975kJ2qFGtBre2vpX4m+J5uOfDbNux\njUqRlfD38ye8ajhul5vQ0FBq1a1FTM0YIqtF8v6379PkliaAuXqz23PuGqVDBw8x641ZHEw8yO/f\n/853n3xHm05tvMcH9RrE9InTiW8WT0RMBBOen8CurbvYtHoT705+12cNo9s63obD4WDC8xPYvW03\nf/3+F/9+5d+0vLsloWGhADz0xEO8O+1dvv3iW/bs2MNbo99i2+ZtPPD4AwDccc8dKKUYPXg0u3fs\n5sfvfmT629PP+QxCiFxeZTFv0+msx8fzLK8xsaCXlUBFXPKUUnR6oBMZ6Rm0v609wwYNo2fvnnTt\n3jXf9LPen8V18dfx9ONP06JJC0aPGO2dir9+w/q88+47LF2ylFZNWzHp9UkMfGEgre880xl24AsD\nqX11bUYMHsGzvZ8FoOmtTX1mX7U6rOzfu/+sw1nrNKjjs13zqpp5alTSTqcx8/WZJO5K5MelP7Ju\nxTrs5fNOaf/dku/4ZvE3pBxPISMlgx8++wG/AD8cAWbH0rg6cQSHBpvX2LiOhzs/zJx5c1j36zrQ\nUKlqpTMXKwdrV6xlwdsL2LNtD840J6TDrHGzAIiMjcS/nD/jh4xn/R/rObT/EJvXbGbepHls27QN\ngPDocJITk9m3fR8pJ1LyDHlWSnF7+9txZjp5dcCrzH9zPm27tKV5u+Z5ns1isTDinRFkpGXQ/77+\nvDnsTRJ6J+DxeNiwbgMAfv5+vDr3VU6fPE3/+/ozpt8Y4pvF03t4b+91ujzahY4PdGToU0N5oNUD\n/P7T70yeN5mYGmZH6oDAACa/O5mdW3fS4/4eTHxtIsNeKZr1iIQQF0f6qIhLXs5RPWMmjTlv+tDy\noUyYMuGsx+/ocAd3dLjDu+1yudi5Z6d3Oyg4iIEvDvRuD+o1iIDAAO926ztbUy6wHKMHj77gZ8hP\nk5ZNyMzIpP99/bHarMQ3j2fTzk150gWHBPPh9A/Zvnk72qPJaJDByzNe5v0F73vT1L2+Lqt+XEVE\n9Qh63deLkJAQ+t/dn7TMNN9+N37wwugX+GT2JyyetRhlUZAJlSLOBDMR9SKoG1OXf4/5N6dOnCKs\nUhj1bqhHufLlMAyDm1rexIqvVzC+/3jSU9PpMbIHTds09U7CNmaq+TMyDIOO3TriCHJgtVu9E7MZ\nhnFmXhfDICImgtfnv+69/+Y15pT25cPKeydyqxpblVdnv5qnbLJXpFZKkdAjgYWLFrLy75X5lnf9\n6+vz3lfvkXk6k2qR1bDb7TLKR4i8bLzOq2RwHwqDQN5lEGf/QC2SGwohLlruPhdbNmwhulr0Wefd\n+Hv939x6+63e7R3/7KBu3RxNMBXh/t73U7NWTZ4e+TQA3371LZum5A1UmrZqStNWTZk4ZiKpqakM\nHz0cwCdQefjZh1m1ZRU9+/SkUmUz6Ojzeh/Gjx6f53oNb2rIv1r9y8zXth30ebwPt7Y382pz2NBo\nHnv2MXoN7IVH+c725kwxZ7N8ZuQzHDp2CFugOQrHmebESDNwpjjx2M1zDJeBkWYGEja72WHXSDNw\npjrBDa4MF1PHTOWHpT9gtVlpcUcLGjdpzJxJcwgMCSQwKNB7v3Ox28y1h0AmbBPioqVzPwEs5Dru\nJJEGHGECb5JIfz4AiuUXTAIVIQogv5letUeTfCCZd8a/Q9t727J9y3Y+X/Q5PQf0zH+mVw0/f/cz\n1WtWp0pMFVb9voqd23fy7OBncRkunE4neMDpdJKRnuE9zeVy4dEe7z7DMPC4fbcNt+Hd1m6N23CT\nkZ5BSGgINruNjxd9zB0d72DPzj0smLcANKSnpZOakmqe54HUtFTArIXIdGaCBzIyM8hIz6BChQok\nH0hm5/ad1L2qLuXLlfcZ0pvNcBm4tRtHoFlb4na5cdqchFcM986RkjuN4TJwWV2EVwjHbrfz47c/\ncte9d9GrTy8WzVzEV/O/4pelv3Br61vZe2AvYSFhRFeJPu/PTFmUt1lOJmwT4iJZ2M9gsmdD3MU4\nruEkvYAPGMb95zx3EJMKc0sJVIS4AMqisFvtuNPcuMkVrGRAixYtSD+RzoCEAVgtVjrc04HbWt6G\n83Q+f/G7oMuDXfh56c9s2ryJ0PBQHuv5GFXCqpB5MpND+w+BYU4db7ecCQKOHT6GJ8ND4h5zavmT\nx07iTHV6t1NPppKelu7dzkjJ4PTx097thx5+iC8+/oIvP/qSmBoxtO/YnhlvzuCfrf9w8vRJ9mzf\nAwZs3LARP38/rFYraSfSwICD+w7iTHMSUy2GOlfXYfjzw8k4nUHfwX1p0aZFnkc0XAZHjx/Flmk7\nU1uSYnjnPjlrmtMGVrsVl8tFpSqVeLDXg4RXCufRPo/y9sS3WfHfFYx8Y2ShfoaR0ZH8+fef508o\nhDg7O2t8tkNYTRq9cGGuB1QM1JVQFaqUigdWr169mvj4+POmF2WLy+ViV+Iu/EL9vFM6l/T99ybt\nxRpk9ekwe7EMl8H+5P1Y/a14Mj1EhUcBkJSchDWgaO+V7/3dBslHkrH4WfLcy+1248n0EF4pPM8C\nhuBbS5HvtQ2Dg4cPevufuF1unClOIipHnAlUcqUxXAau0y4iqkQwrO8wIqIj6D2gN9FVorHZbfz0\nw08M7j+YlRtW+tSMjBkxhq+++MrMl1KsWLOCfj37seZP8/M0KjqKj7786MLKxGX49FEpKJfLRebJ\nTGKrxpbKe1WIi5Xf5+2GDRto27YtpPEXNjYzlDOd9N6hNQeZzovEFkmgchgb3+PHVnZrrV0gNSpC\nXDCr1VrkXz52mx2L3YLH48Fuy1pXw27D4e/wLghWXAzDwM/PD1uwDbvV97kMt4FLufD39y90Pmw2\nGxa7xQx0tLlttVt9rueTxgPaprHZbCiLwqIubFBi72d60+1x31mAh48eTkaG2QSWHRgJIYqAi+t9\ntk/RCAu7iqs2BYoxUFFKhQFTgPaY66t+AvTXWqee57xRQA+gPPAr8JTWenuO48uBW3KcooHpWuve\nCCEuG1s3+XZQXr9uPdWqV8vTz6R8WHnKh5X32VepSiWEEMXAQzRjeYmrWMB+riWNRwk7yyRwr/EG\nVg7yPK/ne/wCFeefGguBcKAl4ADmAdOBh852glJqMPA00A3YDbwKfKOUqqu1zm7s18AM4CXODKxM\nK/rsi7ImeyhqSXO5XBiGgdVl9U4yVhTcLjeGYWBxW7BcplMaeQwPLly4DTcuw4XbdaZ/T/bzK5dC\naWX2UXGZr+wOynOmzqHrw13ZvnU7H8z/gAFDBuByuc5xx4vjdrnNJqhC3qO03qNClBh/Pkbjz18s\nQ+EmiBn0Z1G+ad1EQe5OfQVXLIGKUqoO0AZopLVem7WvL7BMKTVQa33wLKf2B17RWn+ZdU43IBm4\nG8jZyJymtT5cHHkXZY/FYsFhc+BMdebtyFoCXC6Xt1Os21509zdcBq4UFxaPBX8/f9xuN9qjzeDF\nZSmmgX457p81ZwkuUB7fAMxwG97AoTD5cLvdoM8MV3YbboxUgzR7mrfpxzAMMk9mYrgMbDYbHrcH\nT5rHPCcDbr31VtJPpNOjUw9sVhtdOnfhjtvvwHnq/EOSC8swDJynnWQGZeKxec5/Qj4cNgcWy+UZ\neIorXD2e5142Zm29eN705xsFdIGKq0alKXA8O0jJ8j3mR14T4PPcJyilYoEI4IfsfVrrU0qpVVnX\nyxmoPKiUehg4CCzFDG7Si/wpRJlgtVqJiYrxzh5b0lwuFyjwC7m4zrxut9tnHg+3240FC07DiVKK\njJMZGIZB+rF03E43VlvezqojhowgtmYsj/R8pND58N7fcJNxIgO7047L5luDMPGliUTHRPNknyfz\nzceFCLIEef9taAOnxUmYIww/Pz/ALFeXw4XD34HFZn6x20PshFcJZ9ZHs3C73GSeziQmMqbEOqa6\nXC4yAzOJjS58Z1iLJW/nZCFE4RVX2B8BHMq5Q2vtBo5lHTvbORqzBiWn5FznLMBsPmoOjAEeBuZf\ndI5FmZbdkbXUXraLO99isXD42GGSDid5X8nHktEWs/OozxebBbTSZsNmrpdSKt/92a+RQ0ey/Ifl\nPvvub38//1v1P+/25g2b6f1Yb7QyV2HO/n/OF5h5OFs+su9z+NBh7m9/f55ro8yf2c/Lf+aVl17x\ndqL9cO6HTB07lbhqcTx232McP3icmOgYqletTvWq1akaXRV/f3/sdjs2uzlsuVR/7oV4SZAiRNEq\nUI2KUuo1YPA5kmig7jmOXzSt9awcm5uUUgeAH5RSsVrr/JfEFaIIXEz/A5fLRVp6GtZAq89wXyu+\nX2oWw4I93e4dspub8lNY/a3Yg/P/a1/5KWxBNt/jdrAFnNlnDbKi/BWOIAe2DBu2IFueIcjKX2EN\nsJ4zH977OMhz7extW5DNvFc5B0ortEVjsViw2+0opbDarN5gRAgh8lPQpp8JwNzzpNmJ2SRTJedO\npZQVqJB1LD8HMf8uC8e3ViUcWJvvGaY/ss6rBZwzUBkwYAChoaE++xISEkhISDjXaeIKVxR9ZFyG\n2c/FgQNlO3uHXI/hQadrPFYPFiOfCs9McKe5mTFhBst/WI7NaqNN+zZnFmDMBJ2u0ek5OpY4QWfm\n2JdhpvNkZN3L4kHbfDuiaKfGneJm5uSZ/Pzjz2e9jyfd4/13zmtnb+t0jcpUWJwWs8lHJoYVokz7\n4IMPmPv+XCyBFm9fq1OnTpVqngoUqGitjwJHz5dOKbUSKK+Uuj5HP5WWmB9Tq85y7V1KqYNZ6dZn\nXScEs0/L1HPc7nrMmpwD58vX5MmTZcI3UWBF0UfG5XKB5ryT1mWPNvEr55dvTYaf1Y8fv/6Ru+67\ni/cWvceWjVsYPWI0V9W6io73dcTP5kdYSBhREVFnTjKgYmhF776D+w5iw0Z0eDQoszYkew4X730s\nfqz6adV57xNROQLc5Ll29nZYSBgOq4OoiCizT06OTqYynb0QZc8DDzxAk/9rkv+Eb6WkWDrTaq3/\nVkp9A8xUSj2FOTz5bWBRzhE/Sqm/gcFa6+zOtW8Aw5RS2zGHJ78CJJLV+VYpFQd0Bb7CDJgaApOA\nn7TW2T2RhShyVuvFzxSbsx/DuWT3zchvojJlUURFRTFo2CAA4mrFsXPHTj6c/yH3PXAfM9+fmfeC\nCqy2M5PVNflXE77875fe0TY2uy3PpG7KoqgSUYXnhj6HzW47531Wb13t/Xf2tbPd3flu7u58t3f7\n0ScepVpUNQBWrFnB3qS95ywLIYQoznlUumJO+PY95oRvH2MOP86pNuBti9Faj1NKBWLOt1IeWAHc\nkWMOFSfQKus6QcA+YDEwuvgeQ4ii5Xa7z1o743K5MFzmnC06n3HB2qO5psE1PvN81Ktfj/fnvI/L\n6cq/aUWbI3xyzw2Sex6T3PepdXUtc06TrHyc9z7nkT1HSnY+sp/V4sq/T//FzmmSHxmRI8SFydkn\nzzCypikwsHK4mGe0P533+sV2Q631Cc4xuVtWmjyfGFrrkcDIs6RPxBztI8Qlye12czD5IE6371wg\n/Xr1o/bVtXmq/1McOHQAe5o936nrM92ZpGamknQwybvv6MmjaKtm/8H9+Ten2M00Oc8B88Pn0JFD\nWNOseZp+Mj2ZpLvS2Z+835uP897nPFyGCyPFAGVOnW8YBgcOH8By2uIdnpyTxzDnVHG5XEXW2dZu\nsxMVHnXWYEUmbBNXuvz65DlPOyETOIiD7/Er9kycwolZwQHIWj9ClCiPx4PT7cQaaPWZn8Tib8Ea\naMWvnB/2DHPUT85AZchTQ2jVoRUqQLF913Yc5RzeY9v3bCe6ZjR+Ifl/fnz565f57rcYFnPlL+7K\nlgAAH1JJREFU4qC8a/1Y/C3s2b8HR7kz+ci+zz/b/2Hok0P58L8fEhR8Zq6UyaMmEx4VTtceXfO9\nnzLMoc/Z/W9sbhuBmYG4PPnXmGilwVJ0fVncbjfO007SHennDHxkwjZxJcuvT97xQ8fBBRzjAMfY\nXQLZ8GRNaQJIoCJEqcjZZwTMPiEWq8XbXyR3v5Hs1YqVRXH4yGHmTJ/DnR3vZNs/2/jysy95ot8T\n+fZpOXr4KM8/8zx9n+tL/Yb1fYchZ9Vs2Oz5DE+2KI4fPc68WfNof1d7n/tYrBYzP3arzz0tFov3\nGfKjlTYXHczuf2OH6OhotCf/qW9dhgvnKSfVoqoVyQKNLpeLzFPnn8xNmofElS53n7wcvy9G9orG\nJUkCFSHKCI/hYcLoCSz7Yhk2u412ndrRvXd3nzRKKW5vczvHDh+jZ+eeaJcmtEIoURFRtI1vy4hJ\nI2javCnJB5Lp3q47vYf0Zv/W/Qx7ahj9X+rP7e1vZ+PajcybMo9/Nv9DUEgQN9xyAwlPJeDnb9bI\n/PLNL+zZsAdnppP/LPoPSxctxT/Un3vuv4f4RvF0b9cdpRSdbulk5qfD7Tw78tlCPbPVaoVzxAQe\nuwebrejmWfHYPBfUoVkIUXZIoCJEGfHFp1/Q8b6OjPv3OHbv2c2UMVMIjwyn7T1tvc0f494ch8fj\noed9PWnUuBGPP/M4aalpzJg4I98mkiXzl/DS+JeoVacWDoeDpMQkhj09jEeffpRnhj/Djp07WDR9\nEfMmz+OJoU8AZhNJn5F9qFypMnbszH5jNkEhQXTv0R2Px8OwicMYPXA0cz6fQ0BQgHdKfBluLIQo\nDhKoCFFGREZGMuD5Aew/tJ8adWqwd+dePl34KW3vacu46eO86db8vobkpGQmzplI+bDyADzS5xGG\nPjU0zzXvefAemrVo5t2eN2oet7W7jbsS7jI7jjqga9+uvNb/NR4b+Bh2u51b77wVw23gSnURUSmC\nJwc9Sf+H+5ORnoF/gD8hISEAhIaF+vRRKWytihBCnIsEKkKUEScPnOSNUW/Q+cnOANS9ti5L3l+C\n1tqntiJxdyKVwyt7gxSAq+tfne81a9et7bO9a9sudm/fzX+X/RfAXIlYA/5w+MBhoqpFsWvrLj6e\n/TF7tu0hPTXdu3ryoYOHqBZb7aKfc9nHy5g8ajLLtyw/a5oZE2fw439+ZNZHs86aRghxZZBARYhL\nxPMPPU/r+1oX6Bz/AH+f7fS0dO649w7u7no3bsPNvv37sAXYCAoOolJEJTIzMhn77FiubXItvQb3\nIq56HEcOH2FYn2EYrqIbunshzUQFbUqqGlqVVRtXER0TXdhsCSHKIAlUhCgjUlJSfLa3bNhCdLXo\nPF/YVWOrcjj5MCeOn/DWqmzduDXP9fL7oq9VpxZ7d+4lMjrS2/RjCz4zPHnvjr2knEqhc6/OlAss\nR0SlCLZv3e5zjewROBezpEBxkD4yQlyeJFARooxwOp2s/t9q9iftZ9WPq8jMzOTG227Mky6+STwR\n0RG8PuR1Am2B/PX7X2YTDZCceGY9T52hGdN3DAm9E3jvjfdIOZlC3fi6bN68mWljp9GqQyvmTZrH\nyeMnqRlfk+4DuhMaFopyKgbfPRiP4SGyeiQZOsN7zcdaPUaLji1QSvH7z7/TuFljknYn8VyX55jz\n/Rwiq0WyZO4Svv3kWw7uO0i50HI0ua0JPZ7vgdXhO7xn+dfLefPVN0lOSib+pnhemvAS4VHhZy2f\nhe8uZMaUGezbs4+Y6jE8+sSjdO9xZlSU1vkPcxZCXNpkViMhygClFJUqVWLnlp389tNv2EJtNG3b\nlPUr1/P14q+9acCc52Pk5JFsW7ONlctXUi68HH1H9wULzJ80n5RTWTUzCo4cOMLK71fyysxXGDVz\nFDu37KR5i+bs37ufwU8M5q9Vf5G0J4mwymEAfDnnSwL9AwmoGABB5qqpKQdTvEFA606t+fWbX3n4\nyYeZ89Ycut7elTdeeoNrb7yWyGqR3vz1fqk3M/4zg4HjB7L+9/XMHj/b53nT09KZ8/YcXnn7FeZ8\nPofTp07zQu8Xzlo+Sz5cwsTXJjJ05FB++vMnhowYwoTRE/h40cc+ZSiEuPxIjYoQZcD096bTq1Mv\ngoODmbBgApYACzarjXmT5rFkzhJa3dOK0XPNJa0Mw+BY8jHwwKd/forNbmPzus2oAEVQuSB+WvYT\nbTq3oWvPrnw651OeGfOMt69Kiw4t2LR6ExMWTcAwDMYOHEtGRgZtOrXh1IlT/LD4B3qM6EGj2xrh\nznBTOawyvdr2IqF7AtXiqhEcHMz7b73PdTdcR+dHOuM23HS7tRudH+vsnX6+/YPtvc9VoUoFHuz3\nINNensbjzz+OYRi43W7chptBowZRp34dAIZPGE6Xll3YsHoDdRvUxePx+Kx1NPG1iQwfPZw27doA\nULVaVbZu2cr8OfPplNAJgH0n9hX/D0oIUeIkUBGiDKkfXx+7zY4r3YUTJ7WursVn8z4j83SmT43B\nd0u+Iy0ljS5NuqA9GpfLhUJxNPUo+3fsx5nixO10UzmyMha3BWeKubZQSEgIJ46cwJnixOP2gBs8\nTg9ph9PYv3s/breb6OhoMo9lYrVa0U5Nrbq12LN1D84UJ8GBwcT/K56vP/ia6kOq8/t/f8fldNG4\nWWPvPf5a9RefvvspiXsSSU9Nx22YiwumHE1BGQoj3cBqtRIXG0fm6UwAIqpEEFwumG0bthEXG4eR\naaA9GofNQWZGJnt27WHg0wMZ1HeQtwzcbjehoaEIIS5vEqgIUYYopQivFO6dVr5i+YoopYiu4tup\nVhsai9WCDtCUCylHw+sb8kjPRwguF0xwSDCh5UMJCQohwD+A6CpnRsGUDymPRVm8+8LKh2G32Ln+\nmusJsYSgUNSrXY/wqHCUUlgtVgL8AgjyD/Ke06V7F14e8DLDXhvGyu9W0rpja2JjYgE4kHiA1597\nnfu63cczw54hpHwI6/5Yx5jBY6gcZk4gtz/MXNSwWqTvUGeLxULF8hWpFlmN0HKhOOwOosKjOH7s\nOAAT3p7AdY2u8zlHproX4vIngYoQZcjGtRt9ppXftG4T1WKrYXf4Tvl+b8K9LP9qOUu+XkJk1ch8\nr2WxWMz1fHKux2O1oJTy7rNazDU9/P39ibsqDpvdxt/r/6Z6XHXAbGbasn4LDz7xoPecW1rfQkBg\nAJ8u/JSVP61k1qezvMe2bdmGRvPcy8957/nDsh+ArHWFsGGxWDAMg80bNtMwviEA27dt59TJU9St\nVxe73Y7VYjUDJauVSpUrER4Zzu5du7mr010XW8RCiEuMBCpClCEH9x9k8suTufehe9myfgsfzf0o\n3xlfm9zShAaNGvDcY8/R78V+VK9ZnUMHDvHrf3+lxR0tqNugboHvHRAYQKdunXjzlTcpV74cEVER\nvDvtXTIyMuj4QEdvOovFQvv72zPltSlUj6vOtfHXeo/F1IjBcBksmrWIW1rfwro/1rFk/pI897LZ\nbAwbNIxRY0dhtVoZNmgYNzS5gQbXN8g3bwNfGMjwwcMpV64czVs1x+l08tfavzh14hQ9+/Qs8LMK\nIS4dEqgIUUYopWjfqT2ZGZl0a9cNq9VK155duefBe/JN/9b7bzH19amMenYUx48dp2LlisTfFE/F\nyhULnYe+L/RFa82IfiNITUnlmobXMHXRVMqFlPNJd1fCXcx5a45PAANQ+5raDBg5gPf+/R5TX59K\n/E3xPP3i04zoN8InXWBgIH0G9OHpx5/m4MGD3PSvmxg/ZfxZ85XQLYHAwECmvTmN0cNHExAYQJ16\ndejZW4IUIS536kqYe0ApFQ+sXr16NfHx8aWdHXEFcrlc7ErchSXQwoHDB3CEOMrECr4v9H4Bm83G\nqLdGFei8tavW0vuB3nz151eEVQy7oHMMl0Hm6UyqRVYrlWd3uVxknswktmpsmSh7IS4Va9asoVGj\nRgCNtNZrSvr+UqMiRAkyDAPDMLC6rChKb94Pt9vNnh17WP/neu596N4Lnh7f5XRx7Ogxpk+YTst2\nLSkXUu6CzzXcRTcFvxDiyiGBihAlwGKx4LA5SE1NxXnaHMbrtrtLLT/b/95O74d7E98knnZ3tfMO\nEz6f/3z+H8YNH0fturUZ+urQCz4vm8PmMDv5CiHEBZJARYgSYLVaiYmKITMzExT4hfiVavNDtchq\n/J34d4HPe+LJJ3jiyScKfV+LxSJDioUQBSKBihAlxGq1Yrfbsdvs5v+ln8Qly+12l7lFGc9HgkRx\nqZJARQghCsDtdrMvaR9Ow1naWSkQh81BTFSMBCvikiOBihBCFIDH48FpOLEGWbHZLo2PUMMwcKY6\n8Xg8EqiIS86l8VsmxGUmewE/UXKKusxtNtsl1XznpvQ6bwtxMSRQEaIEZY/+caY65YujFMioIyEu\nPRKoCFGCskf/XGodMS8X0qFUiEuPBCpClDCr1SpflkIIcYGkDlQIIYpIp3adGDl0ZGlnQ4jLSrEF\nKkqpMKXUAqXUSaXUcaXULKVU0HnOuUcp9Y1S6ohSyqOUyrOUqlLKTyk1NSvNaaXUx0qpKsX1HEII\nUVQ6tevE4oWLi/Sak16bxLO9866wLcTlojhrVBYCdYGWQDvgFmD6ec4JAlYAzwNnWy3xjazr3Zd1\nzSjgkyLIrxBCCCHKmGIJVJRSdYA2wONa6z+11r8BfYEHlFIRZztPa/2+1vpV4AfIu2KbUioEeAwY\noLX+SWu9FngUaKaUurE4nkUIIQrCbbgZNnAYdWPqcm3stYx/dfxZ086YMoNWTVtRO7I2ja9pzAvP\nvkBaapr3+EcLPuKaatfw0w8/0bxxc66KuoqH7n2Iw4cOl8SjCFEmFFeNSlPgeFYgke17zFqSJhdx\n3UaYHYB/yN6htd4K7M26pxBClKqPFn6EzW5j2Y/LeGXcK8yYOoNF7y0CQCnfv7+sViuvjH+F5X8s\n583pb/Lbit8YPXy0T5r0tHSmvz2dKbOmsOTrJexP3M+oF0eV2PMIUdqKa9RPBHAo5w6ttVspdSzr\n2MVc16m1PpVrf/JFXlcIIYpEdNVoRr42EoC4WnFs2bSFmVNnktAtgcVf+vZPefypx8+cFxPNoGGD\nGDpgKKMnnglWDMNg7JtjiakeA8AjvR7hzXFveo8/O1T6p4jLW4ECFaXUa8DgcyTRmP1ShBDiihTf\nON5nu9GNjZgxZQZa6zw1Kj//+DNTJ09l+z/bSTmdYk51n+kkIyMDf39/AAICA7xBCkB4RDhHDh8p\n/gcRoowoaI3KBGDuedLsBA4CPiNxlFJWoELWscI6CDiUUiG5alXCL+S6AwYMIDQ01GdfQkICCQkJ\nF5ElIYQouMS9iTza5VG69+zOkOFDKB9Wnj9W/sHApwficrq8gYrd5jtNv1IKrc821kCIi7No0SIW\nLVrks+/kyZOllBtTgQIVrfVR4Oj50imlVgLllVLX5+in0hKzg+yqC71dPvtWA0bWtT7NutfVQDVg\n5fkuOHnyZOLj48+XTAghCm3tn2t9tlf/sZrYmrF5alPWr1uP1prho4d7933+yeclkkchzia/P97X\nrFlDo0aNSilHxdSZVmv9N/ANMFMp1Vgp1Qx4G1iktfbWfCil/lZK3ZVjO0wp1RCohxnU1FFKNVRK\nhWdd9xQwG5iklGqulGoEzAF+1Vr/URzPIoQQBZHd2XXHth18tvgz5s6YS4/ePfKkqxFXA5fLxex/\nz2bv7r18vOhj3p/7finkWIiyrTjnUekK/I052udL4GfgiVxpagM522I6AmuBpZg1KouANbnOG5B1\nvY+B5UAS5pwqQghRqpRSdHqgExnpGbS/rT3DBg2jZ++edO3eNU/aa+pfw4gxI5j25jRaNm3J5x9/\nzgsjXyiFXAtRtqkroa1TKRUPrF69erU0/QghLorL5WJX4i78Qv2w2+3nP6EMcLlcZJ7MJLZq7CWT\nZ1F25Gj6aaS1XlPS95e1foQQQghRZkmgIoQQQogySwIVIYQQQpRZEqgIIYQQosySQEUIIYQQZZYE\nKkIIIYQos4prUUIhhLisGYZR2lm4YJdSXoXITQIVIYQoAIvFgsPmwJnqxI27tLNzwRw2BxaLVKKL\nS48EKkIIUQBWq5WYqBg8Hk9pZ6VALBYLVqu1tLMhRIFJoCKEEAVktVrlS1+IEiL1gEIIIYQosyRQ\nEUIIIUSZJYGKEEIIIcosCVSEEEIIUWZJoCKEEEKIMksCFSGEEEKUWRKoCCGEEKLMkkBFCCGEEGWW\nBCpCCCGEKLMkUBFCCCFEmSWBihBCCCHKLAlUhBBCCFFmSaAihBBCiDJLAhUhhBBClFkSqAghhBCi\nzJJARQghhBBllgQqQgghhCizJFARQgghRJlVbIGKUipMKbVAKXVSKXVcKTVLKRV0nnPuUUp9o5Q6\nopTyKKUa5JNmedax7JdbKTWtuJ5DCCGEEKWnOGtUFgJ1gZZAO+AWYPp5zgkCVgDPA/osaTQwAwgH\nIoDIrPRCCCGEuMzYiuOiSqk6QBugkdZ6bda+vsAypdRArfXB/M7TWr+flbY6oM5xizSt9eEizrYQ\nooi53W48Hk9pZwMAi8WC1Wot7WwIIQqoWAIVoClwPDtIyfI9Zm1IE+Dzi7z+g0qph4GDwFLgFa11\n+kVeUwhRhNxuN/uS9uE0nKWdFQAcNgcxUTESrAhxiSmuQCUCOJRzh9barZQ6lnXsYiwA9gBJQANg\nHHAV0OkiryuEKEIejwen4cQaZMVmK66PmgtjGAbOVCcej0cCFSEuMQX69FBKvQYMPkcSjdkvpdho\nrWfl2NyklDoA/KCUitVa7zrXuQMGDCA0NNRnX0JCAgkJCcWQUyEEgM1mw263l3Y2cOMu7SwIUeYt\nWrSIRYsW+ew7efJkKeXGVNA/cyYAc8+TZidmk0yVnDuVUlagQtaxovQHZn+WWsA5A5XJkycTHx9f\nxLcXQgghLg/5/fG+Zs0aGjVqVEo5KmCgorU+Chw9Xzql1EqgvFLq+hz9VFpiBhSrLvR2F5ju+qy0\nBy4wvRBCCCEuEcUyPFlr/TfwDTBTKdVYKdUMeBtYlHPEj1Lqb6XUXTm2w5RSDYF6mEFNHaVUQ6VU\neNbxOKXUMKVUvFKqulKqI/Au8JPWemNxPIsQQgghSk9xzqPSFfgbc7TPl8DPwBO50tQGcnYa6Qis\nxRzJo4FFwJoc5zmBVphB0BZgPLA46zwhxCWkU7tOjBw6srSzIYQo44otUNFan9BaP6S1DtVah2mt\ne2qt03KlsWqt38ux/a7W2pK1P+drVNbxRK11c611Za11oNb6aq31UK11SnE9hxCidHRq14nFCxcD\ncPzYcV587kWa1G9C7cja3HX7XWz860wl6oCnBjD59cmllVUhRDGStX6EEGXezu07sVqtzHhvBt+s\n+IZKlSvR8+GepZ0tIUQJkEBFCFFq3IabYQOHUTemLtfGXsv4V8fnm67RjY0YNW4UDeMbElcrjk4J\nnTiUfKjMzHorhCg+Eqjkknv8+JVIykDKAEqmDD5a+BE2u41lPy7jlXGvMGPqDBa9Z95XqfxX0Th5\n4iSTx06mc9fOWCyWc6a9GPIekDIAKQOAr7/+ulTvL4FKLvKmlDIAKQMomTKIrhrNyNdGElcrjrs7\n381jTzzGzKkzAVj85WI6d+3skz7ldAr3d7ifGnE1GD1htHf/pGmTGDBkQJHmTd4DUgYgZQDwzTff\nlOr9JVARQpSa+Ma+EzA2urERu3bsQuv8p1GaP2c+J0+cZNqcaTIVvhBXCAlUhBCXjOSDycRUjyn1\ntYOEECVHftuFEKVm7Z9rfbZX/7Ga2JqxZ+1z0uvpXmSkZ5RE1oQQZcSVEqj4A2zZsuW8CU+ePMma\nNWuKPUNlmZSBlAFcfBm4XC6SjiThKOfItwYkNSWVvXv20u+JftzR8Q62bd3G7H/Pple/XmzYsCHf\na859Zy5HDx9l4EsDC5QXwzBwnnZy/NDxC14gUd4DUgYgZQBw+vTp7H/6l8b91dnagi8nSqmuwILS\nzocQVxw74Ie5IEZuaZxpfHZlpclOfzbpmHNWBxYwHxrIzLqPEKKwHtRaLyzpm14pgUpFoA2wG5B6\nYyFKho0KRBKBExvuUs2JgZWDODjGAcAo1bwIcenxB2oA32QtTlyirohARQhR8pRSdq6mBq3IpHIp\nBweHsfE9fmxlt9Za6lWEuITIqB8hhBBClFkSqAghhBCizJJARQghhBBllgQqQgghhCizJFARQggh\nRJl1xQcqSqkwpdQCpdRJpdRxpdQspVTQBZxXVyn1uVLqhFIqRSm1SilVtSTyXJQK+/w5zn9HKeVR\nSvUrznwWp4KWgVLKppQaq5Ran/Wz36+UelcpFVmS+b4YSqk+SqldSql0pdTvSqnG50nfXCm1WimV\noZT6RynV/YJvdhobh0v5dTrv5JYFKQOl1D1KqW+VUoey3ie/KaVaX3AZlFEFfR/kOK+ZUsqllLrk\nZ0IrxO+CQyk1Wim1O+v3YadS6pESym6xKEQZPKiUWqeUSlVKJSmlZiulKhRX/q6UmWnPZSEQDrQE\nHMA8YDrw0NlOUErVBFYAM4GXgNNAPS7NOVoK/PzZlFL3AE2A/cWYv5JQ0DIIBK4DXgbWA2HAW8Dn\nwI3FnNeLppTqAkwEegF/AAOAb5RSV2mtj+STvgbwJTAN6Aq0AmYppZK01t+d41YeTuFkFQ6g9FcQ\nPIUT8EDBywC4BfgWGAqcAB4DliqlbtRa/1US2S9qhSiD7PNCgXeB7zF/by5ZhSyDxUBl4FFgBxDJ\nJfxHfyE+D5ph/vz7Y34uRGN+Xs4AOhVLJrXWV+wLqIP5wXV9jn1tMCeEijjHeYuAd0s7/6X1/Fnp\nooG9QF1gF9CvtJ+npMsg13VuANxA1dJ+pgvI6+/Amzm2FZAIPH+W9GOB9bn2LQK+uoB7WTHnmy0L\nL2thy+Asz7YRGFbaP8+Seh/k+tm/DIwA1pT2c5RkGQBtgWNA+dLOeymWwXPAtlz7ngb2FlceL9ko\nsIg0BY5rrXOujPY95oTbTfI7QZmrpbUDtimlvlZKJWdVld1V/NktcgV+fvCWwXvAOK31+RdQKtsK\nVQb5KJ91zokizFuRU0rZgUbAD9n7tPlJ8z1mWeTnpqzjOX1zjvReWmu31tpVRl7uiyiD3OWogHKY\nX1qXnMKWgVLqUSAWM1C5pBWyDDoAfwKDlVKJSqmtSqnxSqlSWQPnYhWyDFYCMUqpO7KuEQ50BpYV\nVz6v9EAlAjiUc0fWh9mxrGP5qQIEA4OBr4DbgU+BJUqpm4svq8WiMM8PMARwaq2nFGPeSkphy8BL\nKeUHvA4s1FqnFHkOi1YlzFqO5Fz7kzn780acJX1I1rNfagpTBrkNAoKAj4owXyWpwGWglKoNjMFc\n78VTvNkrEYV5H8QBN2M29d+N2fzRCZhaTHksbgUuA631b5jN4h8qpZzAAeA4Zq1KsbgsAxWl1GtZ\nHTzP9nIrpa4q5OWzy+wzrfVbWuv1WuuxmG11TxbNE1yc4nx+pVQjoB9m+2yZVczvgZz3sWG2WWug\n90VnXJR5ylzk9CWgsz5HX47LiVLKgrmw6wit9Y7s3aWYpdJiwWwq7qq1/lNr/TXwLND9Eg3aC0wp\ndQ3wJjASiMdsKo/F7KdSLC7XzrQTgLnnSbMTOIhZQ+KllLICFbKO5ecIZv+F3E0eW4BmBc5p8SjO\n5/8/zI5k+8zab8CMyCcppZ7RWscVNtNFrDjLIDtddpASA9x2CdSmgPn+dZO3E2Q4Z3/eg2dJf0pr\nnVm02SsRhSkDAJRSD5DVaVBr/WPxZK9EFLQMymH2w7pOKZVde2DBbAVzAq211suLKa/FpTDvgwPA\n/ly/61swg7aqmJ1rLyWFKYMhwK9a60lZ2xuVUr2BFUqpF7XWuWtnLtplGahoc3XH867wqJRaCZRX\nSl2fo49CS8w33aqzXNullPofcHWuQ1cBewqf66JTnM+P2Tcl90iPb7P2ny8wKDHFXAY5g5Q4oIXW\n+vjF57r4Zb1/V2M+4xfg7W/REnPkUn5WAnfk2tc6a/8lp5BlgFIqAZgFdMn6S/qSVYgyOAXUz7Wv\nD9ACuA9zZfpLSiHfB78CnZRSgVrrtKx9V2PWsiQWc5aLXCHLIBBw5trnwaxVLp5atuLqpXupvDD7\nmfwJNMasEdkK/9/e3bM0DEUBGH4dRFBwc3AScXByc3Hp5Cjo5CD4MTiI/hDBX6AIOuji6Oiim4Og\ni4soOPkHxC8UKg7nCiV0UMHc2/I+EEqapM05acJJ0tNyUJnnBphtGZ8jWpFXgTHi3twHMJU7njri\nb/MaHdv185ccEAX+MVGYThBnH99Db+54fhDvPPAKLBFdTztEUTeUpm/S0tVG/L37E9H9M07c4voA\npnPHUmMOFlLMa5XtPZg7lrpy0Gb5buj6+e3nYCDt90dEx2MjHS+2c8dSYw6Wgfe0L4ymY+YFcP5v\n65g7SbkHolvjEHgkvhC0C/RX5mkCS5XnVoBb4AW4AmZyx1Jn/JXp93R2ofKrHAAjabx1+EyPjdzx\n/DDmdeIs+I24MjLZMm0fOK3M3wAu0/x3wGLuGOrMAXDWZps3gb3ccdT5Oags2/GFyl9yQFw9PwGe\niaJlC+jLHUfNOdgArlMOHojfVRn+r/XrSW8qSZJUnK7s+pEkSd3BQkWSJBXLQkWSJBXLQkWSJBXL\nQkWSJBXLQkWSJBXLQkWSJBXLQkWSJBXLQkWSJBXLQkWSJBXLQkWSJBXrC1wxAr3q6sORAAAAAElF\nTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1f92b742320>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import random\n",
    "import numpy as np\n",
    "from cs224d.data_utils import *\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "from q3_word2vec import *\n",
    "from q3_sgd import *\n",
    "\n",
    "# Reset the random seed to make sure that everyone gets the same results\n",
    "random.seed(314)\n",
    "dataset = StanfordSentiment()\n",
    "tokens = dataset.tokens()\n",
    "nWords = len(tokens)\n",
    "\n",
    "# We are going to train 10-dimensional vectors for this assignment\n",
    "dimVectors = 10\n",
    "\n",
    "# Context size\n",
    "C = 5\n",
    "\n",
    "# Reset the random seed to make sure that everyone gets the same results\n",
    "random.seed(31415)\n",
    "np.random.seed(9265)\n",
    "wordVectors = np.concatenate(((np.random.rand(nWords, dimVectors) - .5) / \\\n",
    "    dimVectors, np.zeros((nWords, dimVectors))), axis=0)\n",
    "wordVectors0 = sgd(\n",
    "    lambda vec: word2vec_sgd_wrapper(skipgram, tokens, vec, dataset, C, \n",
    "        negSamplingCostAndGradient), \n",
    "    wordVectors, 0.3, 40000, None, True, PRINT_EVERY=10)\n",
    "print(\"sanity check: cost at convergence should be around or below 10\")\n",
    "\n",
    "# sum the input and output word vectors\n",
    "wordVectors = (wordVectors0[:nWords,:] + wordVectors0[nWords:,:])\n",
    "\n",
    "# Visualize the word vectors you trained\n",
    "_, wordVectors0, _ = load_saved_params()\n",
    "wordVectors = (wordVectors0[:nWords,:] + wordVectors0[nWords:,:])\n",
    "visualizeWords = [b\"the\", b\"a\", b\"an\", b\",\", b\".\", b\"?\", b\"!\", b\"``\", b\"''\", b\"--\", \n",
    "    b\"good\", b\"great\", b\"cool\", b\"brilliant\", b\"wonderful\", b\"well\", b\"amazing\",\n",
    "    b\"worth\", b\"sweet\", b\"enjoyable\", b\"boring\", b\"bad\", b\"waste\", b\"dumb\", \n",
    "    b\"annoying\"]\n",
    "visualizeIdx = [tokens[word] for word in visualizeWords]\n",
    "visualizeVecs = wordVectors[visualizeIdx, :]\n",
    "temp = (visualizeVecs - np.mean(visualizeVecs, axis=0))\n",
    "covariance = 1.0 / len(visualizeIdx) * temp.T.dot(temp)\n",
    "U,S,V = np.linalg.svd(covariance)\n",
    "coord = temp.dot(U[:,0:2]) \n",
    "\n",
    "for i in range(len(visualizeWords)):\n",
    "    plt.text(coord[i,0], coord[i,1], visualizeWords[i], \n",
    "        bbox=dict(facecolor='green', alpha=0.1))\n",
    "    \n",
    "plt.xlim((np.min(coord[:,0]), np.max(coord[:,0])))\n",
    "plt.ylim((np.min(coord[:,1]), np.max(coord[:,1])))\n",
    "\n",
    "plt.savefig('q3_word_vectors.png')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAkgAAAFoCAYAAACsWgQEAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAAPYQAAD2EBqD+naQAAIABJREFUeJzt3XucXdP9//HXZ2YyQSSDhEQkSFyDKBnq2mta6k6bqkEp\n6l780lLfb5tWKuouKUpp+Lm0jKI0frRNRato0DZR16RucYlI3CKJIHNbvz/OyTizzWQyMmfGzLye\nj8c8zFn7s9ZZe3Zi3tl77X0ipYQkSZI+UtLZE5AkSfq0MSBJkiRllHX2BCRJ3V9ElNI9/1HekFKq\n7+xJqP0ZkCRJRRURpazPEPpR3tlzaXeLqYmIuYak7seAJEkqthL6Uc5O1NOXus6eTLtZQhmPUs7r\nlAAGpG7GgCRJ6hh9qWPdbhSQcko7ewIqju54PViSJGmVGJAkSZIyDEiSJEkZBiRJUuc6h9u4gLNW\nun48c7mKPdp9HhfxfSYwtd3HVZdkQJIkfbqdw21czpgOf9+L+D7ncUmHv68+FQxIkiRJGd7mL0n6\nNCjjfM7hQ75BUMca3MAZXNxi9YcM5Of8hlp2pYQFDOAcTuKPjdsv5Ed8yNdoYDAlvMHq3MHxTKQf\nDY01l3Ay73EsidXozd2U8nZR91BdimeQJEmd7wMOJqhlO/amP+NYyvFcyiH5relj9Ys4g77czfaM\nZnXu4A1+xR1s0ri9hCVszGnsxOcZyE94n0OZzHGN269gP5bwfdbmXLZjL8pYwPscWezdVNfhGSRJ\nUucr4TXO5Gf5V3O4kK1YxHHALYzj4I/V9+YuTuV3+VcXM4HP8yxHAeMAOJ3LC6rnMZGrWcr+wFUA\nvMMxrM7NnMqt+ZqLmMDngN6Nvc5gYvvtoLoaA5IkqfP1YmaT1/2YwfscRy3Qq5n6Ppn6cmZQy1aN\nr69gf97haBrYiEQfoJRgSeP2ejZjDW782Bg17Lqqu6LuwUtskqTu5QYqeZPLWYN72ZRvM4qvsiaX\nkZqNWlKzDEiSpM5Xy/ZNXi+mkhLmtBhpljKqyesaRlHGcwC8TSUlvMoPuILDeIr9eZlahjapL+W5\nZseQ8nrEJbaI6A/sCbwEfNi5s5GkHqeMN1mfv1FDWTOfep/oQwMbcj6XMoA/soTNeJ9j6MtV3ME2\nzY64jIOYyDv04yneZjT1bM8gfsUdbEMp9TQwhEmcTD/+y0J2ooZ9gGgcry9/4V1OZyJv04+n82Ns\nRQnzWnzPrDpKeZNyYO2I6G4fwltoNWBjYGpKqcfc6RcpffzmgO4mIg4FburseUhSj9WL3PLnaGbb\n+3x0PaM2X7O8vjlL8tvqgPp8fW+arlValh8rkTsVUJpv67uCmpL8mH1Wcp9SwRg9w2EppZs7exId\npUecQSJ35ojf/va3jBgxopOn0rWMHTuWSZMmdfY01AYes66nux+z2tpa5r01j/K+5ZSVdY9fO2ed\ndRY/+clPqFlSw+ABg+nVq/sub5o1axaHH3445H+X9hTd409q6z4EGDFiBKNGeYm5LSoqKvyZdTEe\ns66nux+z2tpa1p67Nr0renebINGvXz+22WYbli1axrAhw7rNfrWiRy1RcZG2JElShgFJkiQpw4Ak\nSZKU0VPWIOkTqqqq6uwpqI08Zl2Px6zrOfDAAzt7CkVTX19PQ8NHn+lbW9t4m15ZRHTnxVYNKaXG\nx1AYkLRC/o+76/GYdT0es67nwAMPLAwO3UZ9fT2vznuVmrqaxrZ5b83LPUKhL+uzLgs7b3ZFtpia\niJi7PCQZkCRJHaKurns9S7G77Q9AQ0MDNXU1lPYpbXwkQ3nf8txzpgZRw5dY1rkzLJIllPEo5bxO\nCbmnaxmQJEnFVVJSQnlZOTVLa6hv5kHaXVl5WTklJd1vOW9ZWVnjowvKyspyD+Mso5516X6p8COl\nhS8MSJKkoiotLWXo4KFN1rV0FyUlJZSWlrZeqC7HgCRJKrrS0lKDhLqU7ndeUJIkaRUZkCRJUuue\n5kIu4KzOnkar/sgGjGcut7BKH75qQJIkSavuHG7jcsZ09jTyUuN3ywNTGxmQJElSdxOZ16nZqhVw\nkbYkSVpZZZzPOXzINwjqWIMbOIOLm628g014mkuoZyQlvMQGjONVbmMQR3MCfwHgVrbgWc6mjh0I\nPqA393Ag49mSDwCoBS5lLEs5lER/SnmOgZzLcfy98X1+w3a8xAXUsymlzGYAl/HxQJQNTK3yDJIk\nSVo5H3AwQS3bsTf9GcdSjudSDslv/SiUvE/wFNcRvMdW7M2GnMk8ftSk5jlWYxY3U8JCtmFPhnAs\nNXyOOzmnseZyjuU9jmUAP2M7RtObvzOP67mLjQCYzeq8yA2UMZtt2JP1uIQ3+WkzM/cMkiRJKpIS\nXuNMfpZ/NYcL2YpFHAfcwjgObqz7LV+ggaF8mQPZnXcAuIYLmMstjTV/5OtAbw7jNDZiGfA8V/Nj\nXucGHuLn7M47LOF4+vJLTubufK9zmcCuPMOx7M84pvJ1IPgOp7M+tcDz/ILBvMt5je+zN6+xN0Pb\nvquSJEkroxczm7zuxwwaGEb2Y+mWMpwS5jWGI4CdeKxJzYdsSilP58NRzs78CyjhOTbhSfqQGMQ6\n/LtJv3L+RR2bFozxTD4c5QxixirsYSMDkiRJUoYBSZIkrZxatm/yejGVlDCHXpm6PrxAA4N5iHUa\n2x7N9F2N56hna15ktca2R/gsUM/mPM9IlhLM5x12bNKvhh3pxbMFY2zF6wUzmE/lKuxho6IHpIg4\nOSLmRMQHEfFIROzYSv0XI2JGRHwYEc9GxJErqD0kIhoi4o72n7kkSWqigQ24gJ9wJ8P5JQfwPkdR\nweSP1R3OA5TwCvdzGb9jS65nR17nh+QWS+cWTO/HncCH3MIvuJXNuZZdWcAEenM7u7EQgL78iiWc\nzBXsx50M50J+RD1bMYJrAdiTO4HE9VzM7WzK1XyZxRzfHrta1IAUEd8CLgHOArYHHgemRsSAFuo3\nBu4G7gM+A1wKXBMRX22h9iLggfafuSRJ+pjVuJ3EajzOPbzNz+nDrzmN6o/VrUFiJEcBazCLe3iF\nC1mPS4GglA8BGM6HjOBQGliLZ/gjc7mach5gDD9uHOcUrmVNfs1b/JTHmcYyvsAGHMn+vAzAlnzA\ncI6kji15iqm8wQ9Zt+AuuFUQKbX5zreVHzziEeDRlNJp+dcBvApcllK6sJn6C4C9UkrbFrRVAxUp\npb0L2krIBaNrgc/nt399BfMYBcyYMWMGo0aNap+dkySpm6mtrWXO3Dn0ruhNr165q1ZPPvkkX/va\n12AYe/J1nvrEg1/PjrzEHezAruzLq+0153bxJmVMozf/5aWUUi0U8Tb/iOgFVALnLm9LKaWImAbs\n0kK3nYFpmbapwKRM21nAgpTSdRHx+XaasiRJai+/Yk968T6DeZF5DGceP6OMf37qwlELivkcpAFA\nKbAg074A2KKFPoNaqO8XEb1TSssiYnfgKHKX4CRJ0qdRHWvyJj9mLhsQvEM5D/AVzu7saa2sLvWg\nyIhYE7gRODaltLCt/ceOHUtFRUWTtqqqKqqqqtpphpIkdV233HIL1/32OkrWKKGkJLdMefHixZ9s\nsFP4PfD79ptdxypmQHoLqAcGZtoHAvNb6DO/hfrF+bNHWwIbAf8vv54J8gvNI6IG2CKlNKelCU2a\nNMk1SJIkteCQQw5hp913an4NUg9TtLvY8oucZgCjl7flQ81oYHoL3R4urM/bI98OMBsYCWxH7hLb\nZ4C7gL/mv+8S1zUlSdKnW7EvsU0Ero+IGcA/gbHAGsD1ABFxHjA4pbT8WUdXASfn72b7v+TC0hhg\nb4CU0jLgmcI3iIh3c5vSrCLviyRJ6iGKGpBSSrfmn3l0NrlLZf8B9kwpvZkvGQQffYBcSumliNiH\n3F1rpwJzgWNSStk72yRJUpHU1dU1/T4BdZTyZtdau7zSlnx8v4q+oymlK4ErW9h2VDNtD8DKPya8\nuTEkSVLblZSUUF5WTs3SGuqpB6BmSQ0sA+ZTzjR6d+4Mi2gxNUDD8pfdMwlKkqQ2Ky0tZejgoTQ0\nNOYEFr6xEGqBd3idd3ip0yZXfA0ppfrlLwxIkiSpUWlpKaWlpY2vl9/NBtQtf8p0T1D0D6uVJEnq\nagxIkiRJGQYkSZKkDAOSJElShgFJkiQpw4AkSZKUYUCSJEnKMCBJkiRlGJAkSZIyDEiSJEkZBiRJ\nkqQMA5IkSVKGAUmSJCnDgCRJkpRhQJIkScowIEmSJGUYkCRJkjIMSJIkSRkGJEmSpAwDkiRJUoYB\nSZIkKcOAJEmSlGFAkiRJyjAgSZIkZRiQJEmSMooekCLi5IiYExEfRMQjEbFjK/VfjIgZEfFhRDwb\nEUdmtn83Ih6IiHfyX/e2NqYkSVJbFDUgRcS3gEuAs4DtgceBqRExoIX6jYG7gfuAzwCXAtdExFcL\nyr4A3Ax8EdgZeBX4S0SsX5SdkCRJPU6xzyCNBa5OKd2YUpoNnAC8DxzdQv2JwIsppR+mlP6bUroC\nuD0/DgAppW+nlK5KKT2RUnoW+C65/Rhd1D2RJEk9RtECUkT0AirJnQ0CIKWUgGnALi102zm/vdDU\nFdQD9AF6Ae984slKkiQVKCvi2AOAUmBBpn0BsEULfQa1UN8vInqnlJY10+cC4DUKglVElNI0/JUB\n1NbWUltbu9I70NWVlJRQWlra2dOQJKnLKWZAKrqI+B/gYOALKaWafFsp6zOEfpQ3Fr7J+iyBeW/N\nY+25a3fSbDteeVk5QwcPNSRJktRGxQxIbwH1wMBM+0Bgfgt95rdQvzh79igiTgd+CIxOKT1dsKmE\nfpSzE/X0pQ6Av1FDDZw38Tz69evXZPAD9j+A/Q/Yvw271TXU1dVRs7SGhoYGA5IkaaVUV1dTXV3d\npG3RokWdNJvOVbSAlFKqjYgZ5BZP3wUQEZF/fVkL3R4G9sq07ZFvbxQRPwT+F9gjpfRYsyP1pY51\n8wGpjHoCJkyYwMiRIz/ZDnVB9dR39hQkSV1IVVUVVVVVTdpmzpxJZWVlJ82o8xT7LraJwLERcURE\nbAlcBawBXA8QEedFxA0F9VcBwyPigojYIiJOAsbkxyHf50zgbHJ3wr0SEQPzX32KvC+SJKmHKOoa\npJTSrflnHp1N7lLZf4A9U0pv5ksGAUML6l+KiH2AScCpwFzgmJRS4Z1tJ5C7a+32zNv9LP8+kiRJ\nq6Toi7RTSlcCV7aw7ahm2h4g93iAlsYb1n6zkyRJ+rge9VlsP/zeDxn/v+M7exqSJOlTrkcFpNaM\n2WcMt918W2dPo9HDDz3MkIohLFm8pEn72BPHMun8SZ00K0mSuj8DUieqq6tb4faUEhFB7gHkkiSp\no/S4gFRfV8+408cxYugIRg4byUXnXNRs3YRxEzjy4CMbX0++YjJDKobw9/v+3ti223a7cctvbgFy\nYWbS+ZPYYcQODF93OHvsvgf3T7u/sXbuK3MZUjGEu+64izF7j2GTgZtw56138tqrr3HkwUey9YZb\ns9n6mzF659H87d6/MfeVuRy878EAbLXhVgxdayjfP+n7RfiJSJKkrC79JO1P4tabb6XqiCru+ds9\nPPHYE5xx6hkM2XAIVUdUkXtMU87Ou+3MLTfe0ngW59Hpj9J/QH+mPzSdL4z+Aq/Pe51XXnqFXT+3\nK5ALUJOvnMwFl17A1iO35pbf3MJRhxzF3/75NzYevnHjuOePP5+fnvtTttl2G8p7l3PGKWdQV1vH\nnX+5k9VXX51nZz9LnzX7sMHQDZj828kc9+3jeOixh1iz75qsttpqAE3mKUmS2l+PC0gbDNmA8eeN\nB2D4psOZ9fQsJl8xmaojqrjt7o/WH+20604sWbKEpx5/ipHbjeSRfzzCSf/nJP58958BePjBhxk0\neBAbbrwhAFf/8mpOHnsy+x20HwA/+tmPmP7AdK658hrOuficxnGPPflYvrbv1xpfvzb3NfY5YB82\n33JzAIZu1PjUA9Zaey0A+g/oT99+fRvbJ17Z+FgoSZJUBD3uEtuoHUc1eV352UrmvDDnY+t8+lX0\nY6uRWzH9oenMenoWvXv35rDvHMbTTzzNB+9/wCPTH2Hn3XYG4L0l77Hg9QXssNMOTcbYYecdeO7Z\n55q0jdyu6ZO8jznhGC698FIO3ONALjn3EmY9Pau9dlWSJH1CPS4gtcUuu+/C9Aem88hDuTBUsVYF\nm26xKY9Of5RHHnqEXXbfpc1jrtFnjSavq46o4uEnH2ZM1Rhmz5rN3l/Ym+t/fX077YEkSfokelxA\neuzfTT+6bcY/ZzBsk2HNruvZZfdd+Ncj/+IfD/yDXT6XC0M777Yzf7j9D8x5YU5jQFqz75oMXH8g\n/3rkX036//uRf7P5Fps3vm5p7dD6g9fn8KMOZ/JvJnPcKcdx0w03AdCrVy8A6uv9TDVJkjpSjwtI\nr819jbN/fDYvPPcCf7jtD1z36+v47knfbbZ2p1134r0l7zHtz9Maw9Cun9uVO2+9k/UGrcewTT56\nqPeJp57Ilb+4krvuuIsXnnuBc886l2eeeoZjTjymsaa52/XP+p+z+Pt9f+fVl1/lyf88yfQHpjeG\nqiEbDiEiuPdP9/LO2+/w/tL32/NHIUmSWtCjFmlHBGMOGcOHH3zIvl/el9LSUo496VgOPfLQZusr\n1qpgy6235O233maTzTYBcqEppcSuu+/apPaYE49hyZIlTBg3gbfffJvNttyM6393fZM72Jo7g9RQ\n38CPT/8xr897nb59+/Klr36Js849C4BB6w/iBz/6AeeNP48fnPwDxlSNcYG2JEkdILrbQwgjohdb\nsDFfYRnrknsS4x1swxym/vnPf2bkyJGtjNA91NbWsmzRMoYNGdZ4qU6SpLaaOXMmlZWVAJUppZmd\nPZ+O0uMusUmSJLXGgCRJkpRhQJIkScowIEmSJGUYkCRJkjIMSJIkSRnd9zlISwr2rY5SEtTV1VFb\nW9uJk+o4dXV1nT0FSZK6rO4YkBpYTA2PUg6UAvAm5SyDmiU1LFu0rHNn14HKy8opKfEkoSRJbdXt\nAlJKqT4i5vJ6k8uHawMMHjCYYUOGtdCz+ykpKaG0tLSzpyFJUpfT7QIS5EIS0PgJrxFRB7kPf/Wp\n0pIkqTVef5EkScowIEmSJGUYkCRJkjIMSJIkSRkGJEmSpAwDkiRJUkbRA1JEnBwRcyLig4h4JCJ2\nbKX+ixExIyI+jIhnI+LIZmq+GRGz8mM+HhF7FW8PJElST1PUgBQR3wIuAc4CtgceB6ZGxIAW6jcG\n7gbuAz4DXApcExFfLajZFbgZmAxsB0wB/hARWxVtRyRJUo9S7DNIY4GrU0o3ppRmAycA7wNHt1B/\nIvBiSumHKaX/ppSuAG7Pj7PcqcCfUkoT8zU/BWYC3yvebkiSpJ6kaAEpInoBleTOBgGQUkrANGCX\nFrrtnN9eaGqmfpeVqJEkSfrEinkGaQC5D4tdkGlfAAxqoc+gFur7RUTvVmpaGlOSJKlNuuVnsbVk\n7NixVFRUNGmrqqqiqqqqk2YkSdKnR3V1NdXV1U3aFi1a1Emz6VzFDEhvkfvA2IGZ9oHA/Bb6zG+h\nfnFKaVkrNS2N2WjSpEmMGjWqtTJJknqk5k4azJw5k8rKyk6aUecp2iW2lFItMAMYvbwtIiL/enoL\n3R4urM/bI9++opqvZmokSZI+sWLfxTYRODYijoiILYGrgDWA6wEi4ryIuKGg/ipgeERcEBFbRMRJ\nwJj8OMtdCnwtIr6frxlPbjH4L4u8L5IkqYco6hqklNKt+WcenU3uMth/gD1TSm/mSwYBQwvqX4qI\nfYBJ5G7nnwsck1KaVlDzcEQcCvw8//UccEBK6Zli7oskSeo5ir5IO6V0JXBlC9uOaqbtAXJnhFY0\n5u+B37fLBCVJkjL8LDZJkqQMA5IkSVKGAUmSJCnDgCRJkpRhQJIkScowIEmSJGUYkCRJkjIMSJIk\nSRkGJEmSpAwDkiRJUoYBSZIkKcOAJEmSlGFAkiRJyjAgSZIkZRiQJEmSMgxIkiRJGQYkSZKkDAOS\nJElShgFJkiQpw4AkSZKUYUCSJEnKMCBJkiRlGJAkSZIyDEiSJEkZBiRJkqQMA5IkSVKGAUmSJCmj\naAEpItaOiJsiYlFELIyIayKiz0r0Ozsi5kXE+xFxb0RsmhnzsoiYnd/+ckRcGhH9irUfkiSp5ynm\nGaSbgRHAaGAf4PPA1SvqEBFnAt8DjgM+CywFpkZEeb5kMLA+8H1ga+BI4GvANUWYvyRJ6qHKijFo\nRGwJ7AlUppQey7edAtwTEaenlOa30PU0YEJK6e58nyOABcCBwK0ppaeBbxbUz4mIHwO/iYiSlFJD\nMfZHkiT1LMU6g7QLsHB5OMqbBiRgp+Y6RMQwYBBw3/K2lNJi4NH8eC1ZC1hsOJIkSe2lWAFpEPBG\nYUNKqR54J7+tpT6J3BmjQgta6hMRA4BxtHLpTpIkqS3adIktIs4DzlxBSSK37qjoIqIvcA/wFPCz\nlekzduxYKioqmrRVVVVRVVXV/hOUJKmLqa6uprq6uknbokWLOmk2nStSSitfHNEf6N9K2YvAt4GL\nU0qNtRFRCnwIjEkpTWlm7GHAC8B2KaUnCtrvBx5LKY0taFsT+AuwBNgvpVTTyrxHATNmzJjBqFGj\nWpm+JElabubMmVRWVkJuXfHMzp5PR2nTGaSU0tvA263VRcTDwFoRsX3BOqTRQJBbU9Tc2HMiYn6+\n7on8OP3IrVm6omDsvsBU4ANg/9bCkSRJUlsVZQ1SSmk2uRAzOSJ2jIjdgMuB6sI72PLPMzqgoOsv\ngHERsV9EjARuBOYCU/L1fYF7gTWA75ILYQPzXz70UpIktYui3OafdyjwS3J3rzUAt5O7jb/QZkDj\noqCU0oURsQa5RddrAQ8CexWcJRoF7Jj//vn8f4Pc2qdhwCvtvxuSJKmnKVpASim9CxzeSk1pM23j\ngfEt1P8d+FgfSZKk9uRlKUmSpAwDkiRJUoYBSZIkKcOAJEmSlGFAkiRJyjAgSZIkZRiQJEmSMgxI\nkiRJGQYkSZKkDAOSJElShgFJkiQpw4AkSZKUYUCSJEnKMCBJkiRlGJAkSZIyDEiSJEkZBiRJkqQM\nA5IkSVKGAUmSJCnDgCRJkpRhQJIkScowIEmSJGUYkCRJkjIMSJIkSRkGJEmSpAwDkiRJUoYBSZIk\nKaNoASki1o6ImyJiUUQsjIhrIqLPSvQ7OyLmRcT7EXFvRGy6gto/RURDROzfvrOXJEk9WTHPIN0M\njABGA/sAnweuXlGHiDgT+B5wHPBZYCkwNSLKm6kdC9QDqX2nLUmSerqiBKSI2BLYEzgmpfTvlNJ0\n4BTgkIgYtIKupwETUkp3p5SeAo4ABgMHZsbfDhgLHA1EMfZBkiT1XMU6g7QLsDCl9FhB2zRyZ3t2\naq5DRAwDBgH3LW9LKS0GHs2Pt7xudeAm4KSU0hvtP3VJktTTFSsgDQKahJeUUj3wTn5bS30SsCDT\nviDTZxLwUErp7vaZqiRJUlNtCkgRcV5+UXRLX/URsXmxJptfjP1lcpfXJEmSiqKsjfUXA9e1UvMi\nMB9Yr7AxIkqBdfLbmjOf3HqigTQ9izQQWH6p7kvAcGBRRJOlR3dExAMppS+vaGJjx46loqKiSVtV\nVRVVVVUr6iZJUo9QXV1NdXV1k7ZFixZ10mw6V6TU/jeB5RdpPw3ssHwdUkTsAfwRGJJSajYkRcQ8\n4KKU0qT8637kwtIRKaXbImI9YECm21PkFoDfnVJ6uYVxRwEzZsyYwahRo1Z9ByVJ6iFmzpxJZWUl\nQGVKaWZnz6ejtPUM0kpJKc2OiKnA5Ig4ESgHLgeqC8NRRMwGzkwpTck3/QIYFxHPAy8BE4C5wJT8\nuG+QWduUP5P0akvhSJIkqa2KEpDyDgV+Se7utQbgdnK38RfaDGi85pVSujAi1iD3vKS1gAeBvVJK\nNSt4H5+DJEmS2lXRAlJK6V3g8FZqSptpGw+Mb8P7fGwMSZKkVeFnsUmSJGUYkCRJkjIMSJIkSRkG\nJEmSpAwDkiRJUoYBSZIkKcOAJEmSlGFAkiRJyjAgSZIkZRiQJEmSMgxIkiRJGQYkSZKkDAOSJElS\nhgFJkiQpw4AkSZKUYUCSJEnKMCBJkiRlGJAkSZIyDEiSJEkZBiRJkqQMA5IkSVKGAUmSJCnDgCRJ\nkpRhQJIkScowIEmSJGUYkCRJkjIMSJIkSRkGJEmSpIyiBaSIWDsiboqIRRGxMCKuiYg+K9Hv7IiY\nFxHvR8S9EbFpMzW7RMR9EfFefvz7I6J3cfZEkiT1NMU8g3QzMAIYDewDfB64ekUdIuJM4HvAccBn\ngaXA1IgoL6jZBfgT8Gdgh/zXL4GG9t8FSZLUE5UVY9CI2BLYE6hMKT2WbzsFuCciTk8pzW+h62nA\nhJTS3fk+RwALgAOBW/M1E4FfpJQuKuj3XBF2Q5Ik9VDFOoO0C7BweTjKmwYkYKfmOkTEMGAQcN/y\ntpTSYuDR/HhExLr5/m9FxD8iYn7+8tpuxdkNSZLUExUrIA0C3ihsSCnVA+/kt7XUJ5E7Y1RoQUGf\n4fn/nkXuct2ewEzgvojYZNWnLUmS1MZLbBFxHnDmCkoSuXVHxbI80F2VUrox//33I2I0cDTw4xV1\nHjt2LBUVFU3aqqqqqKqqaveJSpLU1VRXV1NdXd2kbdGiRZ00m87V1jVIFwPXtVLzIjAfWK+wMSJK\ngXXy25ozHwhgIE3PIg0Ell+qez3/31mZvrOADVuZF5MmTWLUqFGtlUmS1CM1d9Jg5syZVFZWdtKM\nOk+bAlJK6W3g7dbqIuJhYK2I2L5gHdJocgHo0RbGnhMR8/N1T+TH6UduzdEV+ZqXImIesEWm++bA\nH9uyL5IkSS0pyhqklNJsYCowOSJ2zC+ivhyoLryDLSJmR8QBBV1/AYyLiP0iYiRwIzAXmFJQcxFw\nakR8IyI2iYgJ5ALTtcXYF0mS1PMU5Tb/vEPJPZ9oGrlnFN1O7jb+QpsBjYuCUkoXRsQa5BZgrwU8\nCOyVUqopqLk0/1DIieQu2T0OfCWlNKeI+yJJknqQogWklNK7wOGt1JQ20zYeGN9KvwuBC1dhepIk\nSS3ys9jZ6A7JAAAPoklEQVQkSZIyDEiSJEkZBiRJkqQMA5IkSVKGAUmSJCnDgCRJkpRhQJIkScow\nIEmSJGUYkCRJkjIMSJIkSRkGJEmSpAwDkiRJUoYBSZIkKcOAJEmSlGFAkiRJyjAgSZIkZRiQJEmS\nMgxIkiRJGQYkSZKkDAOSJElShgFJkiQpw4AkSZKUYUCSJEnKMCBJkiRlGJAkSZIyDEiSJEkZBiRJ\nkqSMogWkiFg7Im6KiEURsTAiromIPivR7+yImBcR70fEvRGxaWb7wIj4TUS8HhHvRcSMiPh6sfZD\nkiT1PMU8g3QzMAIYDewDfB64ekUdIuJM4HvAccBngaXA1IgoLyj7DbAZsC+wDXAHcGtEfKa9d0CS\nJPVMRQlIEbElsCdwTErp3yml6cApwCERMWgFXU8DJqSU7k4pPQUcAQwGDiyo2QW4PKU0I6X0Ukrp\n58C7QGUx9kWSJPU8xTqDtAuwMKX0WEHbNCABOzXXISKGAYOA+5a3pZQWA4/mx1vuH8C38pfwIiIO\nAXoD97frHkiSpB6rrEjjDgLeKGxIKdVHxDv5bS31ScCCTPuCTJ9vAb8D3gbqyF2GOyil9GI7zFuS\nJKltASkizgPOXEFJIrfuqJjOASqAL5MLSQcCt0XE7imlp1fUcezYsVRUVDRpq6qqoqqqqlhzlSSp\ny6iurqa6urpJ26JFizppNp0rUkorXxzRH+jfStmLwLeBi1NKjbURUQp8CIxJKU1pZuxhwAvAdiml\nJwra7wceSymNjYjhwPPA1imlWQU19wLPpZROamHeo4AZM2bMYNSoUSu3s5IkiZkzZ1JZWQlQmVKa\n2dnz6ShtOoOUUnqb3FmbFYqIh4G1ImL7gnVIo4Egt6aoubHnRMT8fN0T+XH6kVuzdEW+bA1yZ6nq\nM93r8ZlOkiSpnRQlVKSUZgNTgckRsWNE7AZcDlSnlOYvr4uI2RFxQEHXXwDjImK/iBgJ3AjMBZaf\ncZpN7izTr/PjDo+IHwBfAe4sxr5IkqSep1iLtAEOBX5J7u61BuB2crfxF9qM3HoiAFJKF0bEGuSe\nl7QW8CCwV0qpJr+9LiL2As4H7gLWJHfJ7YiU0tQi7oskSepBihaQUkrvAoe3UlPaTNt4YPwK+rwA\nfHMVpydJktQi1+1IkiRlGJAkSZIyDEiSJEkZBiRJkqQMA5IkSVKGAUmSJCnDgCRJkpRhQJIkScow\nIEmSJGUYkCRJkjIMSJIkSRkGJEmSpAwDkiRJUoYBSZIkKcOAJEmSlGFAkiRJyjAgSZIkZRiQJEmS\nMgxIkiRJGQYkSZKkDAOSJElShgFJkiQpw4AkSZKUYUCSJEnKMCBJkiRlGJAkSZIyDEiSJEkZRQtI\nEbF2RNwUEYsiYmFEXBMRfVrpc1BETI2ItyKiISK2baamd0Rcka9ZEhG3R8R6xdoPSZLU8xTzDNLN\nwAhgNLAP8Hng6lb69AEeBH4IpBZqfpEf7xv5MQcDv2+H+UqSJAFQVoxBI2JLYE+gMqX0WL7tFOCe\niDg9pTS/uX4ppd/mazcCoplx+wFHA4eklP6ebzsKmBURn00p/bMY+yNJknqWogQkYBdg4fJwlDeN\n3FmhnYApn3DcSnJzvm95Q0rpvxHxSv49DUiS9ClWX19PQ0NDZ0+j6EpKSigtLe3saWgVFCsgDQLe\nKGxIKdVHxDv5basybk1KaXGmfcEqjitJKrL6+npenfcqNXU1nT2VoisvK2fo4KGGpC6sTQEpIs4D\nzlxBSSK37kiSpCYaGhqoqauhtE8pZWXF+vd556urq6NmaQ0NDQ0GpC6srX9CLwaua6XmRWA+0OTO\nsogoBdbJb/uk5gPlEdEvcxZp4MqMO3bsWCoqKpq0VVVVUVVVtQpTkiS1RVlZGb169ersaRRVPfWd\nPYVPpLq6murq6iZtixYt6qTZdK42BaSU0tvA263VRcTDwFoRsX3BOqTR5BZeP7qyb9dM2wygLj/W\nnfn32gLYEHi4tQEnTZrEqFGjVvLtJUnqWZo7aTBz5kwqKys7aUadpyjnOFNKsyNiKjA5Ik4EyoHL\ngerCO9giYjZwZkppSv712uTCzgbkwtSWERHA/JTSgpTS4oi4FpgYEQuBJcBlwD+8g02SJLWXYj4H\n6VBgNrm71+4GHgCOz9RsBhRe89ofeAz4f+TOIFUDMzP9xubHux24H5hH7plIkiRJ7aJoASml9G5K\n6fCUUkVKae2U0rEppfczNaUppRsLXt+QUirJtxd+nV1QsyyldEpKaUBKqW9K6ZsppSZ3zEmSup4x\n+4xh/P+OX+n6IRVD+Msf/1K8CalH87PYJEldwph9xnDbzbd12PtNPG8ie+y+x8fadx65M4/845EO\nm4c6hwFJkqQW5JbBqicyIEmSPjXq6+oZd/o4RgwdwchhI7nonItWWD//9fl8+xvfZpOBm7Drtrty\nz5R7mmyf99o8TvjOCWy14VZsvdHWHF11NHNfmdu4ffqD09n3S/uy2fqbsdWGW3HQngfx2tzXuPWm\nW5l4/kSeefIZhlQMYehaQzv07JU6nwFJK5R9HoY+/TxmXY/H7CO33nwrZb3KuOdv9zDhwgn8+opf\nU31j7ufT3Nmci39+MfsetC/Tpk/joIMP4qSjTuL5554Hcg9sPOygw+jXrx9/+MsfmDJtCn3W7MNh\nXz+Muro66uvr+e5h32XXz+/KXx/5K3fddxeHfecwIoIDxhzA8acczxYjtuDxFx7nseceY/9v7N84\njwcffLDjfijqFAYkrZD/4+56PGZdj8fsIxsM2YDx541n+KbDOfCbB3L08Ucz+YrJANx2921889Bv\nNqnf76D9+Nbh32LYJsM4Y9wZbLv9tlx3Ve55xlNun0JKiQsvu5DNt9ycTTfblEuuuITX5r7G9Aen\ns2TxEpYsXsLoPUczdKOhbLrZpoypGsPgDQbTu3dv+vTpQ2lZKf0H9GfAugPo3bs3AA8/8TBPz366\nY38w6nAGJEnSp8aoHZs+zLfys5XMeWEOKTX37ODm65979jkAZj09izkvzGHzwZs3fm2z8TbULKvh\n5Tkvs9baa/HNQ7/JoQceyne+9R2u/dW1vLHAm6KV030/DEeS1KMtfW8p226/LVdce8XHAlb/Af0B\nmHjlRI458Rjun3Y/d91xFxeecyG3TLmF7XfYvjOmrE8RzyBJkj41Hvv3Y01ez/jnDIZtMqzFu8lm\n/mvmx15vtvlmAIzcbiRzXphD/wH92WjYRk2+1uy7ZmOfrUduzcljT2bKvVPYYsQW3HnbnQD0Ku9F\nfX3X/Ew1rbqecgZpNYBZs2Z19jy6nEWLFjFz5szWC/Wp4THrenrKMautrWXeW/Mo71tOWdnHf/0s\nfW8pr7z8Cqcefyp77b8Xz/33Oa791bUcd+pxPPnkk82O+Yfb/8A6667D1ttuzV+n/pX/zPgPJ/yf\nE3jyySfZfOvN6bNmHw7e72AOP+ZwBqw3gAWvL2D6A9M5+LCDqa2r5U9T/sTOu+9M/wH9efXlV3n+\n2efZ7Yu78eSTT5Ii8fKLLzPljikMWG8Aq6+xeuOH7C5evLjFOdXV1VGzpIaFbyzsFh/KW/C7c7XO\nnEdHi5au63YnEXEocFNnz0OSerxeQG9yn7aZ9T4fXdeozdcsr2/Okvy2OqA+X98732e5BmBZfnvK\n15Tl61Iz2wrfLwEf5seHXDxYmbyzfNzalajtWg5LKd3c2ZPoKD0lIPUH9gReIvfHXZLU8cpYh/UZ\nRA1ldN9rV3WUMp9y3uF1PopXXdlqwMbA1JTS2508lw7TIwKSJKnzRUQvtmBjvsIy1u0WwaF5b1LG\nNHrzX15KKXW/80g9hIu0JUmSMgxIkiRJGQYkSZKkDAOSJElShgFJkiQpo6c8KFIrKSLWBn4J7Evu\nCSK/B05LKS1tob4M+DmwFzAcWARMA/4npfR6h0y6h2vrMcv3OQg4AagE1gG2Syk90QHT7ZEi4mTg\ndGAQ8DhwSkrpXyuo/yJwCbA18Arw85TSDR0w1Y6xpAv87rmDI3iP40isSynPsAFn8SWa/zvyJOvy\nFOOoYyQNDKM31zGACzp4xmpnn/4/pOpoNwMDgdFAOXA9cDVweAv1awDbAT8DngDWBi4DpgCfLfJc\nldPWYwbQB3gQ+B0wucjz69Ei4lvkws5xwD+BscDUiNg8pfRWM/UbA3cDVwKHAl8BromIeSmleztq\n3kXSwGJqeJRyoLSzJ9OieezNh4yjNz9hNR7nPY7iOX7Du+xBX979WP27rMkHLKSUX1HLd6inhMXU\nkPsHi7oon4OkRhGxJfAMUJlSeizftidwDzAkpTR/JcfZAXgU2CilNLdY89WqH7OI2AiYg2eQiiYi\nHgEeTSmdln8dwKvAZSmlC5upvwDYK6W0bUFbNVCRUtq7g6ZdNBFRyqd/ecdD5MLs9wva5pA7U3tJ\nK33vBf4D/CCl1H0fhtkDeAZJhXYBFi7/RZs3jdyD83cid1ZoZayV7/Pxf2mpvbXXMVMRREQvcpcx\nz13ellJKETGN3LFrzs7kjmGhqcCkokyyg+VDw6c2OOSP2ShylzVrC9qnATu19uDHiEhAg+Go6/u0\np3h1rEHAG4UN+b/k7+S3tSoiegPnAzenlN5r9xkqa5WPmYpqALlLSQsy7Qto+fgMaqG+X/7vl4rr\nkxwzdUMGpB4gIs6LiIYVfNVHxObt8D5lwG3kzl6ctMoT78E66phJkprnJbae4WLgulZqXgTmA+sV\nNubXC6yT39aignA0FPiyZ49WWdGPmTrEW+QuJw3MtA+k5eMzv4X6xSmlZe07PTXjkxwzdUMGpB4g\n/+nLrX4Cc0Q8DKwVEdsXrGkZDQS5Rdct9VsejoYDX0opLVz1WfdsxT5m2bf7ZLNUa1JKtRExg9wx\nuQsaF2mPJne3Z3MeJvfYjEJ75NtVZJ/wmKkb8hKbGqWUZpNbDDo5InaMiN2Ay4HqwruhImJ2RByQ\n/76M3HN3RpG7rbxXRAzMf/Xq+L3oWT7JMcu/XjsiPkPuOTsBbBkRn4mI7L+ateomAsdGxBH5uw6v\nIvd4jOuh8XJq4TOOrgKGR8QFEbFFRJwEjMmPo47R1mNG/u/PdsCawLr51yM6eN5qR55BUtah5G5l\nnUbuGR63A6dlajYDKvLfb0DuAYWQu7UVcr9wE/Al4IFiTlZA248ZwP7kLuGl/Fd1vv1nwNnFnGxP\nk1K6NSIGkPu5DiT392TPlNKb+ZJB5C5NL69/KSL2IXfX2qnAXOCYlFL2zjYVSVuPWd5jfHQ2dhS5\nv5cvkzuzri7I5yBJkiRleIlNkiQpw4AkSZKUYUCSJEnKMCBJkiRlGJAkSZIyDEiSJEkZBiRJkqQM\nA5IkSVKGAUmSJCnDgCRJkpRhQJIkScr4//LwjXqkTmPTAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1f92cff8470>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "visualizeWords = [b\"good\", b\"best\", b\"bad\", b\"worst\"]\n",
    "visualizeIdx = [tokens[word] for word in visualizeWords]\n",
    "visualizeVecs = wordVectors[visualizeIdx, :]\n",
    "temp = (visualizeVecs - np.mean(visualizeVecs, axis=0))\n",
    "covariance = 1.0 / len(visualizeIdx) * temp.T.dot(temp)\n",
    "U,S,V = np.linalg.svd(covariance)\n",
    "coord = temp.dot(U[:,0:2]) \n",
    "\n",
    "for i in range(len(visualizeWords)):\n",
    "    plt.text(coord[i,0], coord[i,1], visualizeWords[i], \n",
    "        bbox=dict(facecolor='green', alpha=0.1))\n",
    "    \n",
    "plt.xlim((np.min(coord[:,0]), np.max(coord[:,0])))\n",
    "plt.ylim((np.min(coord[:,1]), np.max(coord[:,1])))\n",
    "\n",
    "plt.savefig('q3_word_vectors.png')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## 4 - Sentiment Analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### a - Implement a sentence featurizer and softmax regression. Fill in the implementation in q4_softmaxreg.py. Sanity check your implementation with python q4_softmaxreg.py."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import random\n",
    "\n",
    "from cs224d.data_utils import *\n",
    "\n",
    "from q1_softmax import softmax\n",
    "from q2_gradcheck import gradcheck_naive\n",
    "from q3_sgd import load_saved_params\n",
    "\n",
    "def getSentenceFeature(tokens, wordVectors, sentence):\n",
    "    \"\"\" Obtain the sentence feature for sentiment analysis by averaging its word vectors \"\"\"\n",
    "    # Implement computation for the sentence features given a sentence.                                                       \n",
    "    \n",
    "    # Inputs:                                                         \n",
    "    # - tokens: a dictionary that maps words to their indices in    \n",
    "    #          the word vector list                                \n",
    "    # - wordVectors: word vectors (each row) for all tokens                \n",
    "    # - sentence: a list of words in the sentence of interest \n",
    "\n",
    "    # Output:                                                         \n",
    "    # - sentVector: feature vector for the sentence    \n",
    "    \n",
    "    sentVector = np.zeros((wordVectors.shape[1],))\n",
    "    \n",
    "    ### YOUR CODE HERE\n",
    "    for word in sentence:\n",
    "        ## We add the wordvectors\n",
    "        ## Here I need an encode ascii to reprsent the word as binary\n",
    "        ## Since I used it before\n",
    "        sentVector += wordVectors[tokens[word.encode('ascii')]]\n",
    "    sentVector /= len(sentence)\n",
    "    ### END YOUR CODE\n",
    "    \n",
    "    return sentVector"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([-0.37392192, -0.19534872, -0.17744234,  0.17258671,  0.58775367,\n",
       "        0.1249855 , -0.14014352,  0.22952412, -0.52053499, -0.12615378])"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sentence = ['good', 'world']\n",
    "getSentenceFeature(tokens, wordVectors, sentence)\n",
    "## We will certainly have problem with this binary iossue"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def softmaxRegression(features, labels, weights, regularization = 0.0, nopredictions = False):\n",
    "    \"\"\" Softmax Regression \"\"\"\n",
    "    # Implement softmax regression with weight regularization.        \n",
    "    \n",
    "    # Inputs:                                                         \n",
    "    # - features: feature vectors, each row is a feature vector     \n",
    "    # - labels: labels corresponding to the feature vectors         \n",
    "    # - weights: weights of the regressor                           \n",
    "    # - regularization: L2 regularization constant                  \n",
    "    \n",
    "    # Output:                                                         \n",
    "    # - cost: cost of the regressor                                 \n",
    "    # - grad: gradient of the regressor cost with respect to its    \n",
    "    #        weights                                               \n",
    "    # - pred: label predictions of the regressor (you might find    \n",
    "    #        np.argmax helpful)  \n",
    "    \n",
    "    prob = softmax(features.dot(weights))\n",
    "    if len(features.shape) > 1:\n",
    "        N = features.shape[0]\n",
    "    else:\n",
    "        N = 1\n",
    "    # A vectorized implementation of    1/N * sum(cross_entropy(x_i, y_i)) + 1/2*|w|^2\n",
    "    cost = np.sum(-np.log(prob[range(N), labels])) / N \n",
    "    cost += 0.5 * regularization * np.sum(weights ** 2)\n",
    "    \n",
    "    ### YOUR CODE HERE: compute the gradients and predictions\n",
    "    grad_reg = regularization * weights ## regularizer part\n",
    "    \n",
    "    grad_fit = prob\n",
    "    grad_fit[range(N), labels] -= 1 ## The gradient wrt to argument of softmax is yhat - y\n",
    "    grad_fit = features.T.dot(grad_fit) / N\n",
    "    \n",
    "    grad = grad_reg + grad_fit\n",
    "    ### END YOUR CODE\n",
    "    \n",
    "    if nopredictions:\n",
    "        return cost, grad\n",
    "    else:\n",
    "        return cost, grad, pred\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "ename": "TypeError",
     "evalue": "sequence item 0: expected str instance, bytes found",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mTypeError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-68-12ba30c3064f>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m     37\u001b[0m     \u001b[0mprint\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0msoftmaxRegression\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mdummy_features\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mdummy_labels\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mdummy_weights\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;36m1.0\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m     38\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m---> 39\u001b[0;31m \u001b[0msanity_check\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m<ipython-input-68-12ba30c3064f>\u001b[0m in \u001b[0;36msanity_check\u001b[0;34m()\u001b[0m\n\u001b[1;32m     28\u001b[0m     \u001b[0mdummy_labels\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mnp\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mzeros\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;36m10\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mdtype\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mnp\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mint32\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m     29\u001b[0m     \u001b[1;32mfor\u001b[0m \u001b[0mi\u001b[0m \u001b[1;32min\u001b[0m \u001b[0mrange\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;36m10\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m---> 30\u001b[0;31m         \u001b[0mwords\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mdummy_labels\u001b[0m\u001b[1;33m[\u001b[0m\u001b[0mi\u001b[0m\u001b[1;33m]\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mdataset\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mgetRandomTrainSentence\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     31\u001b[0m         \u001b[0mdummy_features\u001b[0m\u001b[1;33m[\u001b[0m\u001b[0mi\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;33m:\u001b[0m\u001b[1;33m]\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mgetSentenceFeature\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mtokens\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mwordVectors\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mwords\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m     32\u001b[0m     \u001b[0mprint\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m\"==== Gradient check for softmax regression ====\"\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[0;32mC:\\Users\\Peter martigny\\Desktop\\Cours\\Additional Courses\\Deep Learning for NLP\\pset1-Peter\\cs224d\\data_utils.py\u001b[0m in \u001b[0;36mgetRandomTrainSentence\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    169\u001b[0m         \u001b[0msplit\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mdataset_split\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m    170\u001b[0m         \u001b[0msentId\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0msplit\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;36m0\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m[\u001b[0m\u001b[0mrandom\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mrandint\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;36m0\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mlen\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0msplit\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;36m0\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m)\u001b[0m \u001b[1;33m-\u001b[0m \u001b[1;36m1\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m--> 171\u001b[0;31m         \u001b[1;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0msentences\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m[\u001b[0m\u001b[0msentId\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mcategorify\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0msent_labels\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m[\u001b[0m\u001b[0msentId\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    172\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m    173\u001b[0m     \u001b[1;32mdef\u001b[0m \u001b[0mcategorify\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mlabel\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[0;32mC:\\Users\\Peter martigny\\Desktop\\Cours\\Additional Courses\\Deep Learning for NLP\\pset1-Peter\\cs224d\\data_utils.py\u001b[0m in \u001b[0;36msent_labels\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    142\u001b[0m         \u001b[1;32mfor\u001b[0m \u001b[0mi\u001b[0m \u001b[1;32min\u001b[0m \u001b[0mrange\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mnumSentences\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m    143\u001b[0m             \u001b[0msentence\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0msentences\u001b[0m\u001b[1;33m[\u001b[0m\u001b[0mi\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m--> 144\u001b[0;31m             \u001b[0mfull_sent\u001b[0m \u001b[1;33m=\u001b[0m \u001b[1;34mb' '\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mjoin\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0msentence\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mreplace\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m'-lrb-'\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;34m'('\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mreplace\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m'-rrb-'\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;34m')'\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    145\u001b[0m             \u001b[0msent_labels\u001b[0m\u001b[1;33m[\u001b[0m\u001b[0mi\u001b[0m\u001b[1;33m]\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mlabels\u001b[0m\u001b[1;33m[\u001b[0m\u001b[0mdictionary\u001b[0m\u001b[1;33m[\u001b[0m\u001b[0mfull_sent\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m    146\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mTypeError\u001b[0m: sequence item 0: expected str instance, bytes found"
     ]
    }
   ],
   "source": [
    "def accuracy(y, yhat):\n",
    "    \"\"\" Precision for classifier \"\"\"\n",
    "    assert(y.shape == yhat.shape)\n",
    "    return np.sum(y == yhat) * 100.0 / y.size\n",
    "\n",
    "def softmax_wrapper(features, labels, weights, regularization = 0.0):\n",
    "    cost, grad, _ = softmaxRegression(features, labels, weights, \n",
    "        regularization)\n",
    "    return cost, grad\n",
    "\n",
    "def sanity_check():\n",
    "    \"\"\"\n",
    "    Run python q4_softmaxreg.py.\n",
    "    \"\"\"\n",
    "    random.seed(314159)\n",
    "    np.random.seed(265)\n",
    "\n",
    "    dataset = StanfordSentiment()\n",
    "    tokens = dataset.tokens()\n",
    "    nWords = len(tokens)\n",
    "\n",
    "    _, wordVectors0, _ = load_saved_params()\n",
    "    wordVectors = (wordVectors0[:nWords,:] + wordVectors0[nWords:,:])\n",
    "    dimVectors = wordVectors.shape[1]\n",
    "\n",
    "    dummy_weights = 0.1 * np.random.randn(dimVectors, 5)\n",
    "    dummy_features = np.zeros((10, dimVectors))\n",
    "    dummy_labels = np.zeros((10,), dtype=np.int32)    \n",
    "    for i in range(10):\n",
    "        words, dummy_labels[i] = dataset.getRandomTrainSentence()\n",
    "        dummy_features[i, :] = getSentenceFeature(tokens, wordVectors, words)\n",
    "    print(\"==== Gradient check for softmax regression ====\")\n",
    "    gradcheck_naive(lambda weights: softmaxRegression(dummy_features,\n",
    "        dummy_labels, weights, 1.0, nopredictions = True), dummy_weights)\n",
    "\n",
    "    print(\"\\n=== Results ===\")\n",
    "    print(softmaxRegression(dummy_features, dummy_labels, dummy_weights, 1.0))\n",
    "\n",
    "sanity_check()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [default]",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}