{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Problem Set 1: Text Classification\n",
    "=============\n",
    "\n",
    "In this problem set, you will build a system for automatically classifying song lyrics comments by era. You will:\n",
    "\n",
    "- Do some basic text processing, tokenizing your input and converting it into a bag-of-words representation\n",
    "- Build a machine learning classifier based on the generative model, using Naive Bayes\n",
    "- Evaluate your classifiers and examine what they have learned\n",
    "- Build a machine learning classifier based on the discriminative model, using Perceptron\n",
    "- Build a logistic regression classifier using PyTorch\n",
    "- Implement techniques to improve your classifier, and compete on Kaggle"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 0. Setup\n",
    "\n",
    "In order to develop this assignment, you will need [python 3.6](https://www.python.org/downloads/) and the following libraries. Most if not all of these are part of [anaconda](https://www.continuum.io/downloads), so a good starting point would be to install that.\n",
    "\n",
    "- [jupyter](http://jupyter.readthedocs.org/en/latest/install.html)\n",
    "- numpy (This will come if you install scipy like above, but if not install separately)\n",
    "- [matplotlib](http://matplotlib.org/users/installing.html)\n",
    "- [nosetests](https://nose.readthedocs.org/en/latest/)\n",
    "- [pandas](http://pandas.pydata.org/) Dataframes\n",
    "\n",
    "Here is some help on installing packages in python: https://packaging.python.org/installing/. You can use ```pip --user``` to install locally without sudo."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## About this assignment\n",
    "\n",
    "- This is a Jupyter notebook. You can execute cell blocks by pressing control-enter.\n",
    "- Most of your coding will be in the python source files in the directory ```gtnlplib```.\n",
    "- The directory ```tests``` contains unit tests that will be used to grade your assignment, using ```nosetests```. You should run them as you work on the assignment to see that you're on the right track. You are free to look at their source code, if that helps -- though most of the relevant code is also here in this notebook. Learn more about running unit tests at http://pythontesting.net/framework/nose/nose-introduction/\n",
    "- You may want to add more tests, but that is completely optional. \n",
    "- **To submit this assignment, run the script ```make-submission.sh```, and submit the tarball ```pset1-submission.tgz``` on Canvas.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import sys\n",
    "from importlib import reload"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "My Python version\n",
      "python: 3.6.4 (default, Jan 13 2018, 17:11:19) \n",
      "[GCC 4.8.4]\n"
     ]
    }
   ],
   "source": [
    "print('My Python version')\n",
    "\n",
    "print('python: {}'.format(sys.version))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import nose\n",
    "\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import scipy as sp\n",
    "import matplotlib\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "import torch\n",
    "from torch.autograd import Variable\n",
    "from torch import optim\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 130,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "My library versions\n",
      "pandas: 0.22.0\n",
      "numpy: 1.14.0\n",
      "scipy: 1.0.0\n",
      "matplotlib: 2.1.1\n",
      "nose: 1.3.7\n",
      "torch: 0.3.0.post4\n"
     ]
    }
   ],
   "source": [
    "print('My library versions')\n",
    "\n",
    "print('pandas: {}'.format(pd.__version__))\n",
    "print('numpy: {}'.format(np.__version__))\n",
    "print('scipy: {}'.format(sp.__version__))\n",
    "print('matplotlib: {}'.format(matplotlib.__version__))\n",
    "print('nose: {}'.format(nose.__version__))\n",
    "print('torch: {}'.format(torch.__version__))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To test whether your libraries are the right version, run:\n",
    "\n",
    "`nosetests tests/test_environment.py`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 131,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      ".\r\n",
      "----------------------------------------------------------------------\r\n",
      "Ran 1 test in 0.001s\r\n",
      "\r\n",
      "OK\r\n"
     ]
    }
   ],
   "source": [
    "# use ! to run shell commands in notebook\n",
    "! nosetests tests/test_environment.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1. Preprocessing\n",
    "\n",
    "Total: 1 point"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Read the data into a dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df_train = pd.read_csv('lyrics-train.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A dataframe is a structured representation of your data. You can preview a dataframe using `head()`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Era</th>\n",
       "      <th>Lyrics</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>pre-1980</td>\n",
       "      <td>come on come on let me show you where its at a...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1980s</td>\n",
       "      <td>welcome to the big time youre bound to be a s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>pre-1980</td>\n",
       "      <td>once i believed that when love came to me it ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2000s</td>\n",
       "      <td>i took my love and i took it down climbed a m...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>pre-1980</td>\n",
       "      <td>do do do do do do do do do do do do do do do ...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        Era                                             Lyrics\n",
       "0  pre-1980  come on come on let me show you where its at a...\n",
       "1     1980s   welcome to the big time youre bound to be a s...\n",
       "2  pre-1980   once i believed that when love came to me it ...\n",
       "3     2000s   i took my love and i took it down climbed a m...\n",
       "4  pre-1980   do do do do do do do do do do do do do do do ..."
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_train.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Bags of words\n",
    "\n",
    "Your first task is to convert the text to a bag-of-words representation. For this data, a lot of the preprocessing is already done: the text is lower-cased, and punctuation is removed. You need only create a `counter` for each instance.\n",
    "\n",
    "- **Deliverable 1.1**: Complete the function `gtnlplib.preproc.bag_of_words`. (0.25 points)\n",
    "- **Test**: `nose tests/test_preproc.py:test_d1_1_bow`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from gtnlplib import preproc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# run this block to update the notebook as you change the preproc library\n",
    "reload(preproc);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "y_tr,x_tr = preproc.read_data('lyrics-train.csv',preprocessor=preproc.bag_of_words)\n",
    "y_dv,x_dv = preproc.read_data('lyrics-dev.csv',preprocessor=preproc.bag_of_words)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "y_te,x_te = preproc.read_data('lyrics-test-hidden.csv',preprocessor=preproc.bag_of_words)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Unseen words\n",
    "\n",
    "One challenge for classification is that words will appear in the test data that do not appear in the training data. Compute the number of words that appear in `lyrics-dev.csv`, but not in `lyrics-train.csv`. To do this, implement the following deliverables:\n",
    "\n",
    "- **Deliverable 1.2**: implement `gtnlplib.preproc.aggregate_counts`, a counter of all words in a list of bags-of-words.  (0.25 points)\n",
    "- **Deliverable 1.3**: implement `gtnlplib.preproc.compute_oov`, returning a list of words that appear in one list of bags-of-words, but not another.  (0.25 points)\n",
    "- **Tests**: `tests/test_preproc.py:test_d1_2_agg`, `tests/test_preproc.py:test_d1_3a_oov`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from collections import Counter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "reload(preproc);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To write fast code, you can find bottlenecks using the %%timeit cell magic. \n",
    "\n",
    "Here I'm evaluating two different implementations of `aggregate_counts`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "156 ms ± 1.19 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)\n"
     ]
    }
   ],
   "source": [
    "%%timeit\n",
    "preproc.aggregate_counts(x_tr)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "6.3 s ± 183 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)\n"
     ]
    }
   ],
   "source": [
    "%%timeit\n",
    "preproc.aggregate_counts_slow(x_tr)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "counts_dv = preproc.aggregate_counts(x_dv)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can see the most common items in a counter by calling `counts.most_common()`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('you', 5542), ('i', 5535), ('the', 5061), ('to', 3203), ('and', 2953)]"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "counts_dv.most_common(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "counts_tr = preproc.aggregate_counts(x_tr)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2677"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(preproc.compute_oov(counts_dv,counts_tr))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "30459"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(preproc.compute_oov(counts_tr,counts_dv))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.297246280257606"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "preproc.oov_rate(counts_dv,counts_tr)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "30% of the words in the dev set do not appear in the training set."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Power laws\n",
    "\n",
    "Word count distributions are said to follow [power law](https://en.wikipedia.org/wiki/Power_law) distributions. \n",
    "\n",
    "In practice, this means that a log-log plot of frequency against rank is nearly linear. Let's see if this holds for our data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEOCAYAAACTqoDjAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvAOZPmwAAIABJREFUeJzt3Xd4FVX+x/H3994khISQhIQioXdCh1CUIiAKKEVRQRQVRbDhWhZcdO0/XfuKuCjSlsVGEykqgqg0ASVBkN5bQDoECIS08/tjAgRI4KbczC3f1/Pch8zcyeSTeUK+OefMOSPGGJRSSqlLOewOoJRSyjNpgVBKKZUjLRBKKaVypAVCKaVUjrRAKKWUypEWCKWUUjnSAqGUUipHWiCUUkrlSAuEUkqpHGmBUEoplaMAuwPkh4h0B7qHhYUNrFWrlt1xlFLKqyQkJBw2xpS+2nHizWsxxcXFmfj4eLtjKKWUVxGRBGNM3NWO88ouJhHpLiKjk5KS7I6ilFI+yysLhDFmtjFmUHh4uN1RlFLKZ3llgdAWhFJKuZ9XDlIbY2YDs+Pi4gbanUUp5Zq0tDQSExNJSUmxO4rfCA4OpkKFCgQGBubr872yQCilvE9iYiJhYWFUqVIFEbE7js8zxnDkyBESExOpWrVqvs6hXUxKqSKRkpJCVFSUFociIiJERUUVqMXmlQVCB6mV8k5aHIpWQa+3VxYIpZTKq+PHj/Pxxx/n63Nvvvlmjh8/fsVjXnrpJebPn5+v8xfEjBkzWL9+vVvOrQVCKeUXrlQg0tPTr/i533//PREREVc85rXXXqNTp075zpdf7iwQXjlIfW6pjfCYGgyYsCJf53A4hLBiAYQFB1CyeCBhwQGEBV/8b8lsHxcPdGrzWCkvNmzYMLZt20bjxo258cYbueWWW3jxxReJjIxk48aNbN68mVtvvZU9e/aQkpLCk08+yaBBgwCoUqUK8fHxnDp1iq5du9KmTRuWLl1KTEwMM2fOpHjx4vTv359u3bpxxx13UKVKFe6//35mz55NWloaU6dOpU6dOhw6dIi7776bffv2ce211/Ljjz+SkJBAdHT0+ZwZGRkMGDCA+Ph4RIQHH3yQp59+mm3btvH4449z6NAhQkJCGDNmDEePHmXWrFksXLiQ119/na+//prq1asX2jXzygJx7jbXiEp1Bh44mb8BmPQMw8mUdE6mpHHqbDqZV1lxJMAhlAjOKiiXFZJAYiKK07dlJUoU88pLqlSRenX2OtbvO1Go54wtX5KXu9fL9f233nqLtWvXsmrVKgAWLFjAypUrWbt27fm7fMaPH0+pUqU4c+YMzZs35/bbbycqKuqi82zZsoWvvvqKMWPG0Lt3b77++mv69et32deLjo5m5cqVfPzxx7z33nuMHTuWV199lY4dO/Lcc8/xww8/MG7cuMs+b9WqVezdu5e1a9cCnO/aGjRoEKNGjaJmzZr89ttvPPbYY/z888/06NHjfGEqbF7926xGmRJ8+0TbAp/HGENyagYnU9I4mZLOiTNZ/2Ztnysk2f89kZLGnqOnLzpu7JLt/POWWLo3vEZbG0p5gRYtWlx0C+iIESP45ptvANizZw9btmy5rEBUrVqVxo0bA9CsWTN27tyZ47l79ep1/pjp06cDsGTJkvPn79KlC5GRkZd9XrVq1di+fTtPPPEEt9xyCzfddBOnTp1i6dKl3HnnneePO3v2bD6/a9d5dYEoLCJCiWIBlCgWwDX5vDEqYdcxXpq5lr999Qdf/bab13rWo2bZsMINqpSPuNJf+kUpNDT0/McLFixg/vz5LFu2jJCQENq3b5/jLaLFihU7/7HT6eTMmTM5nvvccU6n86pjHNlFRkayevVq5s6dy6hRo5gyZQrDhw8nIiLifOunqHjlILUnzoNoVjmSWYPb8H+31mfdviS6friYN7/fwKmzrv9gKKXcJywsjJMnT+b6flJSEpGRkYSEhLBx40aWL19e6Blat27NlClTAJg3bx7Hjh277JjDhw+TmZnJ7bffzuuvv87KlSspWbIkVatWZerUqYDV67F69WqXvq+C8MoC4anzIJwO4d5WlfllSHt6NY3h00Xb6fT+Qmav3oc3L6uulC+IioqidevW1K9fn6FDh172fpcuXUhPT6du3boMGzaMVq1aFXqGl19+mXnz5lG/fn2mTp1KuXLlCAu7uKdh7969tG/fnsaNG9OvXz/efPNNAL744gvGjRtHo0aNqFevHjNnzgTgrrvu4t1336VJkyZs27atUPPq8yDc6Fy307p9J7iuehSv9axHjTLa7aT804YNG6hbt67dMWx19uxZnE4nAQEBLFu2jEcffdTt3UY5XXdXnwehYxBudK7b6cvfdvHu3E10Gb6YAW2q8rcbahKqdzsp5Xd2795N7969yczMJCgoiDFjxtgd6Yr0t5SbOR3CvddWoWuDa3jnh418umg7M1ft47EO1YmrXIpaZUsQ4PTKnj6lVB7VrFmTP/74w+4YLvPKAnFuolyNGjXsjuKy6BLFeOeORvRpXomXZq7lpZnrAAgOdNAgJpxGFSJoVDGCxhUjqBBZXG+TVUrZTscgbGCMYffR06zac5zVe5JYnXictXuTOJueCUCp0CDqx4RTMbI4MZHFiYkoToXI4sREhFAmrBgOhxYP5X10DMIeOgbhZUSEylGhVI4KpWfjGADSMjLZtP8kqxOPs3rPcdb/dYI/E49z/HTaRZ8b6BSqly5xvsXRsEI4tcuFEajdVEqpQqYFwkMEOh3Ujwmnfkw497SsfH5/8tl09h4/w95jZ0g8fobEY6fZ+NdJ5q7fz+T4PQAUC3DQqloUT3aqSdNKl8/MVEqp/NAC4eFCiwVQq2wYtS6ZlW2MYc/RM+dbHDNW7aXXx0vpVLcsQzrXok65kjYlVso7vPLKK5QoUYIhQ4YU2dfcuXMnS5cu5e677y6yr1kQ2i/hpUSESlEhdG9Unhe6xbJwaAeGdq7NbzuO0PXDxTw56Q92Hk62O6ZSKpudO3fy5Zdf2h3DZVogfERosQAe71CDJc925NHrqzNv3QFu+PdCnpu+hr+Scl4rRil/88Ybb1CrVi3atGnDpk2bzu/ftm0bXbp0oVmzZrRt25aNGzeSlJRE5cqVycy0bh5JTk6mYsWKpKVdPC44depU6tevT6NGjWjXrh1gLdk9dOhQmjdvTsOGDfn0008Ba8nxxYsX07hxYz744IMi+q7zzyu7mLzxNteiEh4SyLNd6tC/dRVG/ryVL3/fzdcrE7n/2soM6VybYgFOuyMqBXOGwf41hXvOcg2g61u5vp2QkMCkSZNYtWoV6enpNG3alGbNmgG5L6XduHFjFi5cSIcOHfj222/p3LkzgYGBF533tddeY+7cucTExJxfmnvcuHGEh4ezYsUKzp49S+vWrbnpppt46623eO+99/j2228L93t3E68sEOeeBxEXFzfQ7iyeqkxYMK/2rM9DbasxfP4WxizeQdKZNN6+vaHOsVB+afHixdx2222EhIQA0KNHD4ArLqXdp08fJk+eTIcOHZg0aRKPPfbYZedt3bo1/fv3p3fv3ueX+J43bx5//vkn06ZNA6yFALds2UJQUJBbv8fC5pUFQrmuYqkQ3u/diPIRwXz081bqXlOSB1pXvfonKuVOV/hLv6hlZmbmupR2jx49eP755zl69CgJCQl07NjxsmNGjRrFb7/9xnfffUezZs1ISEjAGMNHH31E586dLzp2wYIF7vo23ELHIPzE051qcWNsWV7/bgO/bj1sdxylily7du2YMWMGZ86c4eTJk8yePRvgiktplyhRgubNm/Pkk0/SrVs3nM7Lu2i3bdtGy5Ytee211yhdujR79uyhc+fOfPLJJ+fHKzZv3kxycrJbl+Z2By0QfsLhED7o05jqpUN57IuVeoeT8jtNmzalT58+NGrUiK5du9K8efPz7+W2lDZY3Uyff/45ffr0yfG8Q4cOpUGDBtSvX5/rrruORo0a8dBDDxEbG0vTpk2pX78+Dz/8MOnp6TRs2BCn00mjRo28YpBal9rwM7uOJNNz5K8EOBx80q8pzauUsjuS8hO61IY9CrLUhrYg/EzlqFCmPHwtJYo56Tt6OZ8t26kPM1JK5UgHqf1QrbJhzBzchqcnr+LFmev479KdRIcWIzI0kHrlw2lZtRSx5UsSFOAg0OHQxQGV8lNaIPxUePFAxt4Xx/hfdxC/8xjHTqey5eAp5q0/QPYGRZDTwcPXV+OJjjUJCtAGp1L+xKMKhIiEAguBV4wx3jGTxIs5HMJDbavxUNsL+46fTmXFzmNsO3SKjEzD+r9O8NHPW/lx/QE+6tuEmmX1kakq/4wxOg+nCBW0+9itfxKKyHgROSgiay/Z30VENonIVhEZlu2tfwBT3JlJXVlESBA3xpblkeur83iHGoy8uylj74vj8KlUbvt4KfPXH7A7ovJSwcHBHDlyRMe8iogxhiNHjhAcHJzvc7j1LiYRaQecAiYaY+pn7XMCm4EbgURgBdAXiAGigGDgsCstCL2Lqej8lXSGQRMTWLsviXIlgykTVoznb65Ly2pRdkdTXiItLY3ExERSUlLsjuI3goODqVChwmXLg7h6F5Pbb3MVkSrAt9kKxLVYXUids7afyzq0BBAKxAJngNuMMZlXOrcWiKKVkpbB6EXb2X30NL/vOMr+pBRe7B5L00oRVC9dguBAXedJKW/gyU+UiwH2ZNtOBFoaYwYDiEh/rBZEjsVBRAYBgwAqVark3qTqIsGBTv52Q03AGqsY9FkCL86weg/LlQzmhW51ubZaFKVCg7SfWSkf4FGD1ADGmAlXeX80MBqsFkRRZFKXiwgJ4ouHWrJ6z3H2JaXwyYJtDP7yDwCqRIXQr1Vl2tSMplKpEEKCPO7HTCnlAjv+5+4FKmbbrpC1z2W63LdnCHQ6iMuaiX1z/XIs3HyIHYeTmbN2P69/t+H8ccGBDqpFl2BY1zq0q1XarrhKqTyyYwwiAGuQ+gaswrACuNsYsy6v59YxCM+1+8hpVu4+xl9JKRxNPsv8DQfZcTiZWxpew02xZWlXszSRod619LFSvsIjBqlF5CugPRANHABeNsaME5GbgeGAExhvjHkjj+c914IYuGXLlkJOrdwhJS2DUQu38fGCbaSmZ1IswMGtjWN4sE1VapfTuRVKFSWPKBDupi0I75OSlsGm/SeZtGIP3/yRSEpaJoFO4YY6ZXm0fXUaVgjXAW6l3MynC4S2IHzDseRUpv+xl91Hkpn+x15OpqQTFOCgUqkQejWN4ZYG11A5KtTumEr5HJ8uEOdoC8J3nExJ4/s1f7HtUDJLthxm/V8nCA508PmAlucHwpVShUMLhPJqe4+f4a7Rywh0OJj7dDsCnbpQoFKFxZMnyhXYuS6m2lVjYOev+TuJIwBCoiA0CoIjQPu9PUpMRHGe61qXx75YybJtR/T2WKVs4N0tiPJOEz+oRMFP5AiA4qUgNDqraGT9GxKd7eNz+6MhpBQ4A69+XlUgKWkZxL0+nyrRITzevgZ1rylJlWgdk1CqoPyji6lBbRM/c3T+PjkjDU4fgdOHIfmw9e/poxc+Tj4MKcdz//zg8GwFJBpKVYW2f7eKhyo0787dyMhftgHWsynevqMBtzWpYHMqpbybTxeIIruLKSMdzmQrGqePZH185JJ9R+DwJggtA7eNgmrXuy+THzqanMq+42d447sNLNt+hKmPXKvP0laqAHy6QJzjUYPU+/6ArwfCka1w3WDo+CIEFLM7lU85k5rBtW/9RLXoUJ67uS61yoYRXly7+pTKK1cLhN4aUljKN4GHF0LcA7D0Ixh7AxzaZHcqn1I8yMkzN9Zi1Z7j3DlqGU1em8fz36xh7d4ku6Mp5ZO0BeEOm+bAzMchNRlueh2aP6R3SRWiAydSWL/vBPM3HOCL33bjdAgvdYulT/OK+kwKpVzg011MXjGT+uQBmPkYbJ0PNW+CniOhRBm7U/mcHYeTefizeDYfOEVwoINW1aJ44Za61Cij6zsplRufLhDneGwL4hxj4Pcx8OOLEFTCKhK1u9idyucYY1i05TDz1u3n65WJpGcY7mpRkWdurE0pXTFWqctogfAkBzfA1w/BgbUQN8DqdgoKsTuVTzpwIoX35m5ixqq9NKwQwRu31adOuZJ2x1LKo+ggtScpUxcG/gzXDob4cTD6eqvr6cwxu5P5nLIlg3n3zkb867YGbPzrBHeOWsby7UfsjqWUV9IWRFHb9gvMeBRO/mVtl4yBsvWgTKz1b9l6EFUTArRrpKC2HzrF3WN+Y/+JFAa0qcq9rSrrTGyl0C4mz5ZyAvb8ZnU5HVgPB9dbt8RmplnvOwKsIhFV3XqVyvZvWDm9IyoPth48yds/bOLH9QcQgZ6NyvNKj3pEhGgBVv7LpwuEV9zFlFcZaXB4i1UsDqyzxi2OboNjOyEj9cJxgaEQURFKlrdaH+EVrI9j4qBsrG3xPd3Wgyf58KetfPvnPtrUiOazAS3tjqSUbXy6QJzjtS2IvMjMgKQ9cHQ7HNlm/Xt8N5zYCyf2wakD1nHigDZPw/XDtHvqCkb+spV3527i2S61eax9DbvjKGULn17u2684nBBZxXpV73j5++mpVrFY/B4sft8a/O41BkrXLuqkXqFfy8r8vPEg7/xgzXLXIqFU7vQuJm8XEGStJNtzJPT5HJIS4dN28Nun1jwMdZHwkEC+eKglzSpH8t7cTTwzZRWHTp61O5ZSHkkLhC+p2x0eXQZV2sKcZ62XFonLBAc6+d+DLejTvCLTV+6l75jlrN93wu5YSnkcLRC+Jqws3DMVWj0Ov4+GX96wO5FHKlEsgDd7NWRY1zpsP3SKm0cs5s05GzhwIsXuaEp5DC0QvkgEOr8BTe6FRe/CryPsTuSxHrm+Osueu4Eb6pTh04XbuWXEYnYeTrY7llIewSsLhIh0F5HRSUm6zHOuRKD7hxB7q7UWVPx/tbspF2VLBjOuf3O+GtiKlLRMbvpgEa/OXoc33+GnVGHQ21x9XXoqTOpr3d10bg5FeEVo2Nt6qYvsT0ph6LTVLN5ymGaVIxnQpipd6pXD4dDJicp36DwIdUHaGVj1hTWP4vhuOLTRevJd435w87u6cOAlMjINX/6+m1ELtrH3+Bnuv7Yyr/Soh+gMduUjdB6EuiCwuPXQonMyM2Dh27DwHdi3Eq5/FkKioHgklK4LTv/+sXA6hHtbVebuFpV4aeZa/rdsF06Hg7/dUEOX6FB+xb9/E/grhxM6PA8VW8L0gTC1/4X3witBy4eh6b0QHG5bRE/gdAgvdovlaHIq/126g7V7k5j8cCttSSi/oV1M/u7sSWv5jpQkSNoLf3wGu36F4Ajo9DI0vd8qKH7u8+W7eGHGWgZ3qMGQzjpLXXk3HYNQ+bfvD5j3IuxcDOWbWrO0/XwhwIxMw1OTVzF7tbXYX4c6ZbihThldPlx5JS0QqmCMgTXTYO5zVuui4wtQq6s1nlEyBhxeeYd0gZxNz2DUgu1MWrGbv5KsCXUTH2xBu1qlbU6mVN5ogVCFI/kwzH4SNn57YV9wBFRtC53ftG6b9UNbD57knrG/kZEJ3zx2HRVL6Z1gynt43SNHRaSuiIwSkWki8qjdeVSW0GhrEcD7v4Xbx0G34daaT9sWwOe3++1jU2uUCWN4nyacSEnj3bmb7I6jlFu4tUCIyHgROSgiay/Z30VENonIVhEZBmCM2WCMeQToDbR2Zy6VRyJWi6HBHRD3APT8D/T9Co7tgP/1sJbyOLrd7pRF7trqUfS/rgqzVu9jxE9bSE3PtDuSUoXK3S2ICUCX7DtExAmMBLoCsUBfEYnNeq8H8B3wvZtzqYKq2tZqUaQct5by+KQ1/DnF7lRF7qlONbmhThn+/eNmbvxgITP+2EtahhYK5RvcWiCMMYuAo5fsbgFsNcZsN8akApOAnlnHzzLGdAXucWcuVUhie8BTa6zXNY2tORVT7oejO+xOVmRCggIYe38cH/RpRIBDeGryKh7+LEHXcVI+wY4xiBhgT7btRCBGRNqLyAgR+ZQrtCBEZJCIxItI/KFDh9ydVbkiohLcP9u602nTHBjRGD5qBruX252sSIgItzWpwI9PX8/fOtbg540HefzLlRxNTr36JyvlwTxmJrUxZgGwwIXjRgOjwbqLyb2plMucAdBuKDToDZt/gN9GwYRuULebVUCCSkBE5awlPSKs22UDgq19PrK0h8MhPNWpFsmpGXy2bBfLti1g5N1Nua5GtN3RlMoXt9/mKiJVgG+NMfWztq8FXjHGdM7afg7AGPNmHs7ZHeheo0aNgVu2bCn0zKoQnDkGc1+AnYvg5H7IyOWv6WLhENcfGtwJZetbA+I+YPWe49w3/ncyMw3D72pMxzpldIkO5TEKbR6EiEQZY44UIEgVLi4QAcBm4AZgL7ACuNsYsy6v59Z5EF4k7Qwc32MVjpQkSD8DZ0/Btp9g7XTAWEXixtegZHm70xaK3UdOM+izeDbuP0nbmtGMvKcpJYMD7Y6lVKEWiC3AKuC/wByThyaHiHwFtAeigQPAy8aYcSJyMzAccALjjTF5ei6mtiB8zLGdsOpLa3XZgGLQ/3uo0MzuVIUiJS2D9+ZuYuySHVSLDuXTe5tRs2yY3bGUnyvMAiFAJ+BBoDkwBZhgjNlcGEELQlsQPubINph4K6SnQNtnoPHdPrOi7LSERF6auZb0DMPbdzTgtiYV7I6k/JhbltoQkQ7A50AosBoYZoxZlu+U+aQtCB92cANMGwAH10FsT+g90e5EhebAiRQe/TyBlbuP81CbqgzuqM+XUPYo1DEIoB9wL1Y30ThgFtAYmGqMqVrwuPmjLQgflZkJ3w+B+HHQ/nlo9YjPtCSOJqfy7LQ/mb/hABEhgbzaox7dGpbHqY80VUWoMNdiWgaUBG41xtxijJlujEk3xsQDowoaVKnLOBzQ+kkoXgoW/AuGN4QNs+1OVShKhQYx9v44pj92HeHFA3ly0io6D1/EtkOn7I6m1GVcGoPIy8B0UdAuJj+ybxV8+zQcWAd9v4TqN/jMrbAZmYbZq/fxz2/WkJZhePrGWgxsW5UAp8esoal8VGG2IOaJSES2E0eKyNwCpSsgY8xsY8yg8HDf6HZQV1C+MfT7GsIrWKvHjmoLib7Rreh0CLc2iWHWE21oWa0Ub/+wkVs//pUVOy9dnUYpe7hSIEobY46f2zDGHAPKuC+SUpcIKQUPzoUub8GZozD2BvjheWtuhQ+oXroEnw1oyci7m7Lz8GnuHLWMoVNXs/nASbujKT/nShdTAnCbMWZ31nZl4BtjTNMiyJdbJu1i8ldnjsHcf8KqL6BMLDTpB6XrQExT60FGXt79dCw5lQ9/2sKkFbtJScvk6U61eLJTTbtjKR9TmHcxdcFa+2ghIEBbYJAxxtZuJtC7mPyWMbDuG/ju71aL4pzyTeGeaRAaZV+2QnI0OZUhU1fz88aDvNw9lgda23azoPJBhToPQkSigVZZm8uNMYcLmK9QaIHwc8ZYLYrdy+HAWlj4trWvajvrwUZ1uoHDaXfKfDudmk73j5aw7VAyEx5ozvW1Sut6TqpQFHaBiAEqk23116xnPdhKC4S6SGICrJtutS5O7LUW/3vwByjmvUtb7DycTPePlnDybDq9msTwr14NCA703qKnPENhdjG9DfQB1gHnHpVljDE9Cpwyn3QMQl1RRjos/9h60l2Xt62Jdl7sTGoGL85cy7SERLrUK8fwuxprkVAFUpgFYhPQ0BhztrDCFRZtQagr+qQNHNoAd30JtTrbnabARvy0hX//uJnmVSL5amArnS+h8q0w50FsB3SNYuV9+nwGZevBlPsg5YTdaQrsbzfUZFjXOqzYeYw352wkI9Oj5q8qH+RKgTgNrBKRT7MeCTpCREa4O5hSBVaqKtz0hrU67I8vwf41dicqsIfbVeOu5hUZt2QH94xdTvLZdLsjKR/mSoGYBfwfsBRIyPZSyvNVamXdzZQwAUa1gakPwK6ldqfKNxHhzV4NGNyhBsu3H+Wh/8VzMiXN7ljKR7l6F1NxoJIxZpP7I12dDlKrPDt1CH553XowUWa6NTO7Ygu7UxXIxwu28s4Pm6gSFcLo++KopQ8iUi4qtDGIrF/Gq4AfsrYbi8isgkfMP12LSeVZidLQ/UMYsgVCy8CU+2HHYrtTFchj7WvwUd8m/JWUwi0jFjPyl612R1I+xpUupleAFsBxAGPMKqCaGzMp5T7FI+DWkdZzsf/XDcZ0tJ6V7aW6NyrPwqEduL5Wad6du4lZq/fZHUn5EFcKRJoxJumSfZk5HqmUN6jRCZ5cDR1fhL/+hM97wcbv4OQBu5PlS7nwYEb1a0adcmG8PWcjR5NT7Y6kfIQrBWKdiNwNOEWkpoh8hDVgrZT3KlEa2g2BO/8LJ/fDpLvhg1j4+Q1ruQ4vE+B08NbtDTl06iwDJ+rAtSocrhSIJ4B6wFngK+AE8JQ7QylVZOp2h6fXwX2zoPbNsOgd+LQd/DbampHtRRpXjOD9Oxuxes9xHv18JekZ2tBXBePSXUyeSmdSq0KVmQkJ4+H3sdYM7Pq3Q68xXrfg3+QVu/nH12vo1TSG9+5ohEOfd60u4epdTAFXO0BEfgEuqyLGmI75zKaUZ3I4oPlDEDcAvh8CK8ZCdG1o8zQEBNmdzmV9mldi4/6T/PfXnZQuUYx/dKmjRULliytrMTXLthkM3A6kG2OedWewK9F5EMrtjIEJ3WDXEgiJhlveh3q32p3KZcYYHpywgl82HaJF1VKM6teMUqHeU+SUexXqct85nPx3Y4zts4y0i0m5VfpZWD8TvhsCZ5OgZAW4/llocq/V2vBwqemZ/G/pTt6cs4FqpUvw8T1NdTKdAgp3olypbK9oEekM6Aw15fsCikHD3vDYUuj6jrU9+2/WHU8pl9757XmCAhwMbFeN8f2bcyApha4fLmb4/M06eK1c5koX0w6sMQgB0oEdwGvGmCXuj3dl2oJQRSozE5a8Dz+/DmHloeXDcO1gcF51KM92h06eZcjU1SzcfIiGFcKZ8EAL7XLyY27tYvIUWiCULXYsgvmvwN4E6PgCtBtqdyKXGGMY+ctW3pu3mYiQQL58qBWx5UvaHUtsdoyUAAAYF0lEQVTZoDAfGNTrSu8bY6bnMVuh0QKhbGMMfHYbbP8FBidAdA27E7nsl40HGfRZPIIwc3Br6l6jRcLfFOYDgwYA44B7sl5jgQeB7kC3goRUymuJQKeXITAURrWGtV/bnchlHeqU4dsn2lIswEH3j5aQeOy03ZGUh3KlQAQCscaY240xt2PNqg40xjxgjHnQvfGU8mDlm8DDC60HEk17EJZ9bHcil9UuF8bY++NIzzR0+vdCNh84aXck5YFcKRAVjTF/Zds+AFRyRxgRuVVExojIZBG5yR1fQ6lCFV0TBsyHiMow9znYv9buRC5rWS2K//ZvTkpaJl0/XMy8dfvtjqQ8jCsF4icRmSsi/UWkP/AdMN/VLyAi40XkoIisvWR/FxHZJCJbRWQYgDFmhjFmIPAI0Mf1b0MpG1VsDn0nWR+P6Wgt+HfWO/4i71CnDF8/eh2ZxjDoswR+WKtFQl1w1QJhjBkMjAIaZb1GG2OeyMPXmAB0yb5DRJzASKArEAv0FZHYbIe8kPW+Ut6hbCwM/AVKVbMW/Pt3Pfjp/yA12e5kV9WsciTTH72OsGIBPPJ5Av/6foPdkZSHcHU66ErgO2PM08BcEXF5OqYxZhFw9JLdLYCtxpjtxphUYBLQUyxvA3OMMStd/RpKeYSYpvD4cmtl2JLlYfF78GZFWPCW3cmuqkmlSH59riOtqpVi9KLt/H3Karz5FnhVOFyZST0QmAZ8mrUrBphRwK8bA2R/jFdi1r4ngE7AHSLySC55BolIvIjEHzp0qIAxlHKDatdbheLeGVA8Eha8aT3i9Mwxu5NdUcngQCY80IKq0aF8vTKRl2etszuSspkrLYjHgdZYz4HAGLMFKOOOMMaYEcaYZsaYR4wxo3I5ZrQxJs4YE1e6dGl3xFCqcFTvAM9sgFaPw/oZ8GFjWPkZZGbYnSxXwYFO5jzZltAgJxOX7eJvX/2hS3P4MVcKxNmsbiAARCSAHJb/zqO9QMVs2xWy9rlERLqLyOikJM9fD0f5uYAg6PIv6DsZAkNg1mBrKXEPFhzoJOHFG+lavxyzVu+j4/sLtUj4KVcKxEIReR4oLiI3AlOB2QX8uiuAmiJSVUSCgLuAWa5+sjFmtjFmUHi4rhmovETtLvD0Wmh4F8SPh+mDIOWE3alyFRzo5JN+zejboiK7j56m1Zs/cfjUWbtjqSLmSoEYBhwC1gAPA99j3WXkEhH5ClgG1BaRRBEZYIxJBwYDc4ENwBRjjMsdntqCUF7J4YSe/4HYW+HPyfBhI+vRph7sX7c14Pmb63D4VCpxr8/X22D9zBXXYsq6HXWiMeaeoovkOl2LSXmtjd9bs6/Tz0CNTtDhn9ZdUB7qmz8SeXryagD+eXNdBrarZnMiVRCFshaTMSYDqJzVDaSUKix1boYhm6BJP9g6H8Z0gA0F7bl1n9uaVGDyoFYAvPH9Bj76SZ/k6A9cWc11IlAXa4zg/KwfY8y/3Rvtipn0kaPKd+z8Fb64E9KSofVTcOOrdifK1a4jyVz/7gIAnupUk6c61bI3kMqXArcgROSzrA97AN9mHRuW7WUbHaRWPqVKa3hmHUTVhF+HW0t1eOgktcpRocx7uh0Aw+dv4d25G21OpNwp1xaEiKzHmrT2A9D+0veNMZfOji5yOgahfMqpQzCxBxxcD3W7Q5/P7U6Uqz1HT9P2nV8AiKscyZcDWxEU4PnP6VaWwhiDGAX8BNQC4rO9ErL+tY3exaR8UonS8MgSuKaxNR6x9D8eO6muYqkQlvyjAwDxu45R64U5JOzy7JniKu9cGYP4xBjzaBHlyRNtQSifdGwXfNjQ+jioBPQaYw1qe6DMTMNjX6zkh6ylwt+5oyG94ype5bOU3QrtiXKeWhyU8lmRleHvm6HBnZB6Cib1he/+bneqHDkcwqh7mzH63mYAPDvtT0Yv2mZzKlVYrtqC8ER6F5PyG4e3wH+y/tCLqgH3fgMRbnleV4Gt3H2MXh8vBaBn4/J80LsxDofYnErlpDCfSe1x9C4m5Teia1qtiejacGQrDG8ASz6wO1WOmlaKZO5T1h1OM1fto8P7C8jM9L4/QNUFXlkglPIrYWVh8O/QLaswzH8FRnfwyFtha5cLY/1rnQl0CruOnKb12z/rcyW8mBYIpbxF3IPw1FooVhL2rYRPrvPIIhESFMDG/+tKREggfyWlcMP7C+2OpPLJKwuE3uaq/FZERRiyBQKKW/MlPqgHZ0/ZneoyTofw+/OdCApwsP1wMn+fstruSCofvLJA6BiE8muBwfDsdihRFk7shTdj4NRBu1NdJijAwYp/dgLg65WJvD9vk82JVF55ZYFQyu8FhcDfN0GF5tb2ezWtRf88THjxQOY/cz0AH/28lX/P26RjEl5EC4RS3koEBvwILbOmKn1+u0euCFujTAmmP3YdACN+3sp943/XIuEltEAo5c1EoOtb1mxrgMn94PtnPW7wummlSBY/ay3NsXjLYa5/dwEpaZ65jIi6wCsLhA5SK3WJhr1hoLV4Hr9/ClP7Q0a6rZEuVbFUCGtf7UxIkJPdR0/T8l8/cTrVszKqi3llgdBBaqVyENMU/rHL+nj9DJjY0948OShRLIA/X76Ja8KDSTqTRuxLc9m433Ofze3vvLJAKKVyUTzCmnkNsGuJVSQ8rLspwOnglyHtaV0jCoAuwxfz2/YjNqdSOdECoZSvCSsLz2ywJtRtXwCT7vG4ZcODA5188VArXu1RD4A+o5fz4fwtOnjtYbRAKOWLSpaHZ9ZbH2/6Dj69Ho7vsTdTDu6/rgrv3mEtbf7B/M288d0GmxOp7LRAKOWrioXB0G3W6q8H1sDw+nBos92pLnNnXEV+GdIegLFLdjBqoS4X7im0QCjly0Kj4ak1EDfA2h7ZHDIz7c2Ug6rRoUx75FoA3pqzkTGLtmt3kwfQAqGUP+j27wuzrifcDMmeNygcV6UU3z7RBoA3vt/AfeN/17kSNvPKAqHzIJTKh3umQWAo7F4G71aDv/60O9Fl6seE8/Wj1yFiTai78YOFnE3XImEXrywQOg9CqXwoHgGDV0CTe63tT9vC3gSPuw22WeVIlvyjIxUii7Pn6Bn6jf2NtAzP6xbzB15ZIJRS+RQeA91HQPvnre0xHSFhgscViZiI4swe3IaSwQGs2HmMx75YqWMSNtACoZS/cTjg+meh90QoFg7fPgUzHrU71WUiQ4P47m9tKR7o5Mf1B7hv/O92R/I7WiCU8kciENsTbh8L5RrA6q/gj8/tTnWZiqVCmDW4NUEBDpZsPczAifHakihCWiCU8me1boIub1sfL3oXVoy1N08OapYNY/qj19EgJpwf1x/g5VnrOHAixe5YfkELhFL+rkpraPO09VS674d67N1Nr/SoR9mSxZi4bBdjF2/nTKre3eRuWiCUUtDpFej6DphM6+6mA+vtTnSZppUi+fUfHSke6GTM4h36CNMi4DEFQkSqicg4EZlmdxal/FKju6w7nACm3Gc9oe7sKXszXSLA6eDbv7WhQmRxpq1M5P7xv5N8Vp8p4S5uLRAiMl5EDorI2kv2dxGRTSKyVUSGARhjthtjBrgzj1LqCpyB1oOH6t9hreO0dT5smOVxs66rly7Bw+2qUalUCAs3H2Luuv0knU6zO5ZPcncLYgLQJfsOEXECI4GuQCzQV0Ri3ZxDKeWKwOJwxzjo+R9re8ajMLGHvZlycO+1VfjXbQ0AeGbKagb8b4XNiXyTWwuEMWYRcPSS3S2ArVkthlRgEuB5j75Syp+ViYUH5kDtW+DINlj8PqyebHeqi9QrX5LJg1rRpkY0Ww6eYuQvW/l162G7Y/kUO8YgYoDsC9MnAjEiEiUio4AmIvJcbp8sIoNEJF5E4g8dOuTurEr5JxGofB3UuQXSz8BPr8E3g+D0pX/v2UdEaFktig51ypB0Jo13527iuelr7I7lUzxmkNoYc8QY84gxprox5s0rHDfaGBNnjIkrXbp0UUZUyv80uQdeOAQ9R1rb816En1+H1NP25spmQJuqbH69K/1aVeLAiRRem72emav22h3LJ9hRIPYCFbNtV8ja5zJdzVWpIhQQBOWbQGgZWPeNNaFu91K7U10kKMBB8yqlKBbg4LPlO3ll1jq7I/kEOwrECqCmiFQVkSDgLmBWXk6gq7kqVcTK1oOhW+Ch+db2+pmwYhwke06ff8/GMfz5SmcealuNEynpTFy2k+krE8nM1KU58ivAnScXka+A9kC0iCQCLxtjxonIYGAu4ATGG2PyVO5FpDvQvUaNGoUdWSl1JWHlIDAEVk4EJsLpI9bCfx6kWnQoGZmGl2Zav1aqlS5B44oRNqfyTuLNC1/FxcWZ+Ph4u2Mo5V9SkyHtDAxvCM36Q5d/2Z3oMsdPp7Jy9zEenBDPxAdb0K6WjldmJyIJxpi4qx3n1haEUsoHBYVar+BwWD4Sln9s7W96H/QYYW+2LBEhQZSPKA7AfeN/RwQebledYV3r2JzMu3hlgdAuJqU8QLcPrCfSAaz9GvZ71iJ/tcqE8WK3WJJOpzI1IZF1+/SmlrzyygJhjJkNzI6Lixtodxal/FbtLtYL4PAm+Gu19QIoUQ7CytqXDXA4hAFtqgIQv+sYR06lsnavVSTKlCxGmbBgO+N5Ba8sEEopD1O8FBzbCZ+2s7aDI+AfO60Jdx4gMiSIpduO0O2jJQCEFQtg9cs34XB4Rj5P5ZUFQruYlPIwN7wENW+0Pl4/E/6cDBlp1hwKD/Byj1h6Ni4PwA/r9jN95V5SMzIJdjhtTubZPGYmdV7oPAilPExIKWtZjjq3QLmG1r50z3nqW5mwYG6qV46b6pWjfnnr98bZtEybU3k+r2xBKKU8WGBW3/7IlnDuL/RGd0HHF+zLlE1woJWp8/BFOB1CgFP4d+/GNKscaXMyz+OVLQhdakMpD1arizU/onoHqNoOMjNg2892pzqvQ53S9G1RiTY1o2lWOZJdR06zJvG43bE8kle2IPQuJqU8WHgF6P7hhe2v+sLxPbkfX8SuCS/Om72sZ0mcOpvOrNX7SM3Q7qaceGULQinlRZxBkHHW7hQ5CnJavwJT07VA5MQrWxBKKS8SEAxHd8DH117Y53DCze9BpVb25QICnYII/PfXncxe/ddF711bPYpXetSzKZln8MoWhI5BKOVFGve1JtRFVbdekVVh/xrYvczuZIgIT3SsSfMqpagaHXr+lZyazpy1f139BD7OK1sQOgahlBep1t56nZORDv8XZc2T8ADP3Fjrsn0vzFjD92v225DGs3hlC0Ip5cXO3frqIQUiJwEOB2k6cK0FQilVxESyBq5T7U6Sq6AALRDgpV1MSikv5wyCLT9CyiXzD8QJrR6F6Jr25MoS6BTOpmfy3PQ1l73nEHigdRVqlAmzIVnR8soCoWsxKeXlqrWHxBWwac7F+08dgNDS0OE5O1Kd16hCBGXDgpm/4cBl7x06eZaIkECGdvb9Z0t4ZYHQQWqlvNxdX+S8/9VSkJletFlycG7dppzUfmEO6X7ynGsdg1BKeQ5HAGR67uA1gNMhZGRogVBKqaLlCLDWbvJgTodoC0IppYqcM8AjupiuJMAhZGiBUEqpIubw/ALhdDj8pgXhlYPUSikf5QiAE/tg929XP7ZMXQgu6f5Ml3A64OCJFBJ2HXXp+DrlShJazDt/1XpnaqWUbwoOh03fW6+rqX8H3DHO/ZkuUTI4kJ82HuSnjQddOv7OZhV4985Gbk7lHl5ZIHQehFI+6p5pcGTr1Y+b8yyk2LNY538faM72Q8kuHfv8N2s4keLZd2VdiVcWCJ0HoZSPiqxsva4mOAKMPUthVIgMoUJkiEvHhgUH4s3DFTpIrZTyPuIA49m3w4K1LIcx3lshtEAopbyPw2lbCyIvnF5+S6wWCKWU9xEHeMFf5iKiXUxKKVWkxOHxM67B6mLK9IJClhstEEop7yMO7+hiEtECoZRSRcprBqmFTM+vY7nymNtcRSQU+BhIBRYYY3JZD1gp5fe8pAUh2sWUOxEZLyIHRWTtJfu7iMgmEdkqIsOydvcCphljBgI93JlLKeXlvKRAOES8YSw9V+7uYpoAdMm+Q0ScwEigKxAL9BWRWKACsCfrMM9vOyql7ONwesUgtdMhZHhxhXBrF5MxZpGIVLlkdwtgqzFmO4CITAJ6AolYRWIVOjailLoSccDxXTD9Yfecv1JLiHuwwKcRgR2Hk3lm8qpCCHWxPs0r0rJaVKGfNzs7xiBiuNBSAKswtARGAP8RkVuA2bl9sogMAgYBVKpUyY0xlVIeq+r1cGAd7F7mnvOHRhfKadrUiGbnkWRWuLjya17cULdsoZ/zUuLuaeBZLYhvjTH1s7bvALoYYx7K2r4XaGmMGZzXc8fFxZn4+PhCTKuUUr5PRBKMMXFXO86Orpy9QMVs2xWy9rlMRLqLyOikJHtWc1RKKX9gR4FYAdQUkaoiEgTcBczKywmMMbONMYPCw8PdElAppZT7b3P9ClgG1BaRRBEZYIxJBwYDc4ENwBRjzLo8nldbEEop5WZuH4NwJx2DUEqpvPPkMQillFJewCsLhHYxKaWU+3llgdBBaqWUcj+vLBDaglBKKffz6kFqEUkCtlyyOxxIymU7+8fRwOFCjHPp1y3o8Vd6P6f3rrbvStfF169FXrbtvBauHJvbMfn5mbh021OugyvH6/8P197P7fuOMMaUvupXNsZ47QsYfbV92bcv+Tje3VkKcvyV3nfl+77S9+5v1yIv23ZeC1eOze2Y/PxMXOlnxNd/Jq70vfvjtcjt5ZVdTNnktGbTpftmX+E9d2cpyPFXet+V7/vSfVe6LoXN065FXrcLU17O7cqxuR2Tn5+JS7c95Tq4crz+/3DtfVevRY68uoupIEQk3rhwH7A/0GtxgV4Li16HC/z5Wnh7C6IgRtsdwIPotbhAr4VFr8MFfnst/LYFoZRS6sr8uQWhlFLqCrRAKKWUypEWCKWUUjnSApFFREJF5H8iMkZE7rE7j51EpJqIjBORaXZnsZOI3Jr18zBZRG6yO4+dRKSuiIwSkWki8qjdeeyU9bsiXkS62Z3F3Xy6QIjIeBE5KCJrL9nfRUQ2ichWERmWtbsXMM0YMxDoUeRh3Swv18IYs90YM8CepO6Vx+swI+vn4RGgjx153SmP12KDMeYRoDfQ2o687pLH3xMA/wCmFG1Ke/h0gQAmAF2y7xARJzAS6ArEAn1FJBbr0ad7sg7LKMKMRWUCrl8LXzaBvF+HF7Le9zUTyMO1EJEewHfA90Ub0+0m4OJ1EJEbgfXAwaIOaQefLhDGmEXA0Ut2twC2Zv2VnApMAnoCiVhFAnzwuuTxWvisvFwHsbwNzDHGrCzqrO6W158JY8wsY0xXwKe6YPN4HdoDrYC7gYEi4nO/K7ILsDuADWK40FIAqzC0BEYA/xGRW3DvNHtPkuO1EJEo4A2giYg8Z4x505Z0RSe3n4kngE5AuIjUMMaMsiNcEcvtZ6I9VjdsMXyvBZGTHK+DMWYwgIj0Bw4bYzJtyFZk/LFA5MgYkww8YHcOT2CMOYLV7+7XjDEjsP5w8HvGmAXAAptjeAxjzAS7MxQFn24e5WIvUDHbdoWsff5Ir4VFr8MFei0seh3wzwKxAqgpIlVFJAi4C5hlcya76LWw6HW4QK+FRa8DPl4gROQrYBlQW0QSRWSAMSYdGAzMBTYAU4wx6+zMWRT0Wlj0Olyg18Ki1yF3ulifUkqpHPl0C0IppVT+aYFQSimVIy0QSimlcqQFQimlVI60QCillMqRFgillFI50gKhVBESkVdEZIjdOZRyhRYIpfIpa7VX/T+kfJb+cCuVByJSJeshMhOBtcC4rKeLrRORV7Mdt1NEXhWRlSKyRkTq5HCugSIyR0SKF+X3oJSrdDVXpfKuJnC/MWa5iJQyxhzNesDMTyLS0BjzZ9Zxh40xTUXkMWAI8NC5E4jIYOBG4FZjzNki/w6UcoG2IJTKu13GmOVZH/cWkZXAH0A9rKePnTM9698EoEq2/fdhPansDi0OypNpgVAq75IBRKQqVsvgBmNMQ6zHcQZnO+7cL/8MLm6tr8EqGBVQyoNpgVAq/0piFYskESmL1SpwxR/Aw8AsESnvrnBKFZQWCKXyyRizGuuX/UbgS+DXPHzuEqzWx3ciEu2ehEoVjC73rZRSKkfaglBKKZUjLRBKKaVypAVCKaVUjrRAKKWUypEWCKWUUjnSAqGUUipHWiCUUkrlSAuEUkqpHP0/AR417RL2e8sAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fd778fc7a20>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.loglog([val for word,val in counts_tr.most_common()])\n",
    "plt.loglog([val for word,val in counts_dv.most_common()])\n",
    "plt.xlabel('rank')\n",
    "plt.ylabel('frequency')\n",
    "plt.legend(['training set','dev set']);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Reflect**: The dataset we are working with does not include capitalization. How do you think this figure would change if capitalization distinctions were included?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pruning the vocabulary\n",
    "\n",
    "Let's prune the vocabulary to include only words that appear at least ten times in the training data.\n",
    "\n",
    "- **Deliverable 1.4:** Implement `preproc.prune_vocabulary` (0.25 points)\n",
    "- **Test**: `tests/test_preproc.py:test_d1_4_prune`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "reload(preproc);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "x_tr_pruned, vocab = preproc.prune_vocabulary(counts_tr,x_tr,10)\n",
    "x_dv_pruned, _ = preproc.prune_vocabulary(counts_tr,x_dv,10)\n",
    "x_te_pruned, _ = preproc.prune_vocabulary(counts_tr,x_te,10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4875"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(vocab)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "88 79\n",
      "187 176\n"
     ]
    }
   ],
   "source": [
    "i = 94\n",
    "print(len(x_dv[i]),len(x_dv_pruned[i]))\n",
    "print(sum(x_dv[i].values()),sum(x_dv_pruned[i].values()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. Linear classification\n",
    "\n",
    "Now you'll implement the linear classification rule, $\\hat{y} = \\text{argmax}_y \\theta^{\\top} f(x,y)$.\n",
    "\n",
    "You will use these functions in all classifiers in this assignment.\n",
    "\n",
    "Total: 2 points for 4650, 1 point for 7650."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from gtnlplib import clf_base\n",
    "reload(clf_base)\n",
    "\n",
    "from gtnlplib import constants\n",
    "reload(constants);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Recall from class and the reading that the feature function vector $f(x,y)$ can be viewed as a dict, in which the values are counts, and the keys are tuples $(y,x_j)$, where $y$ is a label and $x_j$ is a base feature.\n",
    "\n",
    "- **Deliverable 2.1**: Implement the function ```make_feature_vector``` in ```clf_base.py```. (1 point for 4650, 0.5 points for 7650)\n",
    "- **Test**: `tests/test_classifier.py:test_d2_1_featvec`\n",
    "\n",
    "Note that you must also include the offset feature, ```gtnlplib.constants.OFFSET```.\n",
    "\n",
    "Desired output is shown below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "fv = clf_base.make_feature_vector({'test':1,'case':2},'1980s')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{('1980s', 'test'): 1, ('1980s', 'case'): 2, ('1980s', '**OFFSET**'): 1}\n"
     ]
    }
   ],
   "source": [
    "print(fv)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's compute the entire set of labels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'1980s', 'pre-1980', '1990s', '2000s'}\n"
     ]
    }
   ],
   "source": [
    "labels = set(y_tr) #figure out all possible labels\n",
    "print(labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now implement the prediction rule, $\\hat{y} = \\text{argmax}_y \\theta^{\\top} f(x,y)$.\n",
    "\n",
    "- **Deliverable 2.2**: Implement the function ```predict``` in ```clf_base.py```. (1 point for 4650, 0.5 points for 7650)\n",
    "- **Test**: `tests/test_classifier.py:test_d2_2_predict`\n",
    "\n",
    "The output should be:\n",
    "\n",
    "- A predicted label\n",
    "- The scores of all labels\n",
    "\n",
    "This function will be called **a lot**, so try to make it fast. You don't need to do anything crazy, but avoid making your code do silly extra work. It's worth trying out a couple different versions using %%timeit.\n",
    "\n",
    "You can test this function using these simple hand-crafted weights."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from collections import defaultdict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# weight vectors must be defaultdicts\n",
    "theta_hand = defaultdict(float,\n",
    "                         {('2000s','money'):0.1,\n",
    "                          ('2000s','name'):0.2,\n",
    "                          ('1980s','tonight'):0.1,\n",
    "                          ('2000s','man'):0.1,\n",
    "                          ('1990s','fly'):0.1,\n",
    "                          ('pre-1980',constants.OFFSET):0.1\n",
    "                         })"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('2000s',\n",
       " {'1980s': 0.0, '1990s': 0.0, '2000s': 1.3000000000000003, 'pre-1980': 0.1})"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf_base.predict(x_tr_pruned[0],theta_hand,labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's see how good these weights are, by evaluating on the dev set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from gtnlplib import evaluation\n",
    "reload(evaluation);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.3422222222222222\n"
     ]
    }
   ],
   "source": [
    "# this applies your predict function to all the instances in ```x_dv```\n",
    "y_hat = clf_base.predict_all(x_dv_pruned,theta_hand,labels)\n",
    "print(evaluation.acc(y_hat,y_dv))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3. Naive Bayes\n",
    "\n",
    "You'll now implement a Naive Bayes classifier, as described in chapter 1 of the notes.\n",
    "\n",
    "Total: 2 points"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from gtnlplib import naive_bayes\n",
    "reload(naive_bayes);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- **Deliverable 3.1**: (warmup) implement ```get_corpus_counts``` in ```naive_bayes.py```. (0.5 points)\n",
    "- **Test**: `tests/test_classifier.py:test_d3_1_corpus_counts`\n",
    "\n",
    "This function should compute the word counts for a given label."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "50.0\n",
      "14.0\n"
     ]
    }
   ],
   "source": [
    "eighties_counts = naive_bayes.get_corpus_counts(x_tr_pruned,y_tr,\"1980s\");\n",
    "print(eighties_counts['today'])\n",
    "print(eighties_counts['yesterday'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- **Deliverable 3.2**: Implement ```estimate_pxy``` in ```naive_bayes.py```. (0.5 points)\n",
    "- **Test**: `tests/test_classifier.py:test_d3_2_pxy`\n",
    "\n",
    "This function should compute the *smoothed* multinomial distribution $\\log P(x \\mid y)$ for a given label $y$.\n",
    "\n",
    "Hint: note that this function takes the vocabulary as an argument. You have to assign a probability even for words that do not appear in documents with label $y$, if they are in the vocabulary.\n",
    "\n",
    "You can use ```get_corpus_counts``` in this function if you want to, but you don't have to."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "log_pxy = naive_bayes.estimate_pxy(x_tr_pruned,y_tr,\"1980s\",0.1,vocab)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Probabilities must sum to one! (or very close)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9999999999998161"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sum(np.exp(list(log_pxy.values())))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's look at the log-probabilities of the words from the hand-tuned weights"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'money': -7.689562807416871, 'name': -7.5683247138169865, 'tonight': -6.2166375570076395, 'man': -6.631876946457978, 'fly': -8.636944126361056, '**OFFSET**': 0.0}\n"
     ]
    }
   ],
   "source": [
    "print({word:log_pxy[word] for (_,word),weight in theta_hand.items() if weight>0})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "log_pxy_more_smooth = naive_bayes.estimate_pxy(x_tr_pruned,y_tr,\"1980s\",10,vocab)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'money': -7.801363512554179, 'name': -7.691160372420565, 'tonight': -6.4054072405225515, 'man': -6.808471387093178, 'fly': -8.60745110429472, '**OFFSET**': 0.0}\n"
     ]
    }
   ],
   "source": [
    "print({word:log_pxy_more_smooth[word] for (_,word),weight in theta_hand.items() if weight>0})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- **Deliverable 3.3**: Now you are ready to implement ```estimate_nb``` in ```naive_bayes.py```. (0.5 points)\n",
    "- **Test**: `tests/test_classifier.py:test_d3_3a_nb`\n",
    "\n",
    "\n",
    "\n",
    "- The goal is that the score given by ```clf_base.predict``` is equal to the joint probability $P(x,y)$, as described in the notes.\n",
    "- Don't forget the offset feature, whose weights should be set to the prior $\\log P(y)$.\n",
    "- The log-probabilities for the offset feature should not be smoothed.\n",
    "- You can call the functions you have defined above, but you don't have to."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "theta_nb = naive_bayes.estimate_nb(x_tr_pruned,y_tr,0.1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('2000s',\n",
       " {'1980s': -2153.0199277981355,\n",
       "  '1990s': -2125.1966084804503,\n",
       "  '2000s': -2099.2474010561396,\n",
       "  'pre-1980': -2136.8348423968023})"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf_base.predict(x_tr_pruned[155],theta_nb,labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.46444444444444444\n"
     ]
    }
   ],
   "source": [
    "y_hat = clf_base.predict_all(x_dv_pruned,theta_nb,labels)\n",
    "print(evaluation.acc(y_hat,y_dv))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.46444444444444444"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# this block shows how we write and read predictions for evaluation\n",
    "evaluation.write_predictions(y_hat,'nb-dev.preds')\n",
    "y_hat_dv = evaluation.read_predictions('nb-dev.preds')\n",
    "evaluation.acc(y_hat_dv,y_dv)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# execute this block to write predictions for the test set\n",
    "y_hat = clf_base.predict_all(x_te_pruned,theta_nb,labels)\n",
    "evaluation.write_predictions(y_hat,'nb-test.preds')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.5577777777777778"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# you can't run this, but this how the grading code works\n",
    "y_hat_te = evaluation.read_predictions('nb-test.preds')\n",
    "evaluation.acc(y_hat_te,y_te)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- **Deliverable 3.4**: Write a function in ```naive_bayes.py``` called ```find_best_smoother```, which finds the smoothing value that gives best performance on the dev data.  (0.5 points)\n",
    "- **Test**: `tests/test_classifier.py:test_d3_4a_nb_best`\n",
    "\n",
    "Your function should trying at least the following values in `vals` below.\n",
    "\n",
    "Then, using this smoothing value, run your Naive Bayes classifier on the test set, and output the results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1.00000000e-03 3.16227766e-03 1.00000000e-02 3.16227766e-02\n",
      " 1.00000000e-01 3.16227766e-01 1.00000000e+00 3.16227766e+00\n",
      " 1.00000000e+01 3.16227766e+01 1.00000000e+02]\n"
     ]
    }
   ],
   "source": [
    "vals = np.logspace(-3,2,11)\n",
    "print(vals)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "best_smoother, scores = naive_bayes.find_best_smoother(x_tr_pruned,y_tr,x_dv_pruned,y_dv,vals)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZIAAAEOCAYAAACjJpHCAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvAOZPmwAAIABJREFUeJzt3Xt8VdWZ//HPkyuBXCAQxJyAICAKSQWNaGt1WlsV2yqM9oI6vUxnxjo/ae20tdUZ60ytM446tZ1Onc5gW2c6LVKr1TJFoZcp1bYqBEVIQOSqEG6BcL/k+vz+ODt4CCfJyWXnnOR836/XeeXstfde51kJycNea+21zd0RERHpqYxkByAiIgObEomIiPSKEomIiPSKEomIiPSKEomIiPSKEomIiPSKEomIiPSKEomIiPSKEomIiPRKqInEzGaZ2Xoz22hmd3Zy3A1m5mZWGWzfbGarYl6tZjY92LcsqLNt3+gw2yAiIp2zsJZIMbNM4A3gSmA7sAK40d3XtjuuAFgM5ADz3L2q3f4K4Bl3nxhsLwO+1P64zowaNcrHjx/f88aIiKShlStX7nX3kq6OywoxhpnARnffDGBmC4HZwNp2x30deAC4o4N6bgQW9iaQ8ePHU1WVcN4RERHAzN5M5Lgwu7YiwLaY7e1B2UlmdgEw1t0Xd1LPx4DH25U9FnRrfdXMrE+iFRGRHknaYLuZZQAPA1/s5JiLgWPuXh1TfLO7VwCXBa+Pd3DuLWZWZWZVdXV1fRi5iIjECjOR1AJjY7bLgrI2BUA5sMzMtgKXAIvaBtwDc2l3NeLutcHXw8ACol1op3H3+e5e6e6VJSVddvGJiEgPhZlIVgCTzWyCmeUQTQqL2na6+0F3H+Xu4919PPAScF3bIHpwxfJRYsZHzCzLzEYF77OBDwGxVysiItLPQhtsd/dmM5sHLAUygR+4e42Z3QtUufuizmvgcmBb22B9IBdYGiSRTODXwKMhhC8ifeyZV2t5aOl6dhw4TunwPO64egpzZkS6PlFSXmjTf1NJZWWla9aWSPI882otd/1sDcebWk6W5WVncv/1FUomKczMVrp7ZVfH6c52EQndQ0vXn5JEAI43tfDgkteTFJH0pTDvIxERAWDHgePxyw+eYNo9SyjOz6F4aA7Fw3IYMSyHkbFfh+YwMj/4OiyXgiFZZGQkPutfXWrhUyIRkVBVba0nM8Nobj29G71wSBYfvnAs+481su9oI3uPNPLG7iPUH2087QqmTWaGMWJoDsXDsikeFk0uI4ZlUzwsl+Kh2RTn555MSsu37uOfn3udE02tANQeOM5dP1sDoGTSh5RIJK3pf6vhOXyiiQeXrOd/XnqT4XnZHGtsobGl9eT+vOxM7p1d3uH3+3hjC/XHGqk/0si+ow3RZHOkkf3HGqk/+vbr9V2HqD/ayIHjTSQy5Hu8qYWHlq7Xz7kPKZFI2mo/AKz/rfadX6/dzd3PVLP78Ak+fekEvnjVOfxq7e5uJe28nEwiOXlEhucl9Jktrc6BY42nJJxbf/RK3GM76mqTnlEikbTV0QDw/c+t46ppZzA0R78e3VV3uIF/+N8aFq/eyZQzCvjun13AjHEjgGhyDjNBZ2YYI/NzGZmfy6RgTfDI8Dxq4ySN0gSTkyRGvymSdt7ad4wlNTvj/oEB2H2ogan3LGVIdka0r/3kQG/nA8HDh+aQmeAg8GDrUnN3nly5nfsWr+N4Ywtfuuocbrl8IjlZyZ0YesfVU+JOO77j6ilJjGrwUSKRQc/d2bDnCEuqd7Gkehdrdx4CIDvTaGo5vVN9eF42t75n4in98PuONvLmvmPUH23kSENz3M8xi55bPCzntFdswlmz/SDf+e1GGpoHxwDwW/uOcdfTq/nDxn1cNH4E91//DiaNzk92WMDb38+Hlq6n9sBxMgzduxIC3ZAog5K7s3r7QZbU7GJp9S427z2KGVw4bgSzysdw9bQxrHxzf49ukmtobmH/0aa3E82xRuqPNLz9/mjsq4n9xxppiTNjqb3I8Dz+cOcVfdL+/tDc0spjf9jKN361nqyMDO685lxumjmuW1Nz+9P3XtjMfYvXUXX3+xmVn5vscAaERG9I1BWJDBotrU7V1nqeq97FL2t2sePgCbIyjHdOHMmn3z2Bq6aewejCISePH1s8FKDbXUy5WZmMKcpkTNGQTo9r09rqHD7RfHLm0Q3ffTHucbUHjrNs/R4un1ySsn+M29TsOMidT61hTe1B3n/eGXx9zjTOLErtcYfySBEAa2oP8t4perBqX1IikQGtsbmVP27ay9KaXfyyZjf7jjaSm5XB5eeU8MWrpvC+80YzfGhOh+eHPQAMkJFhFA3NpmhoNtDxAHCGwaceW0HZiDzmXjSWj1aOPSXxpYITTS386282MP/5zYwYms0jN13AByrGMBAeCzSttBCAGiWSPqdEIicNlAHgY43NPP9GHUuqd/Gb1/dw+EQzw3IyueK8M5g1bQzvmVLCsNzU/afd0QDw12dPIzc7k8eXv8W//PINvvXrDbz/vDO48eJxXDZpVNKvUl7ctI+7fraarfuO8dHKMv72A+d1mqRTTcGQbM4eNYw1tQeTHcqgk7q/bdKvUv2eioPHm/jt63tYUr2LZW/s4URTKyOGZnNN+RhmlY/hXRNHMSQ7M9lhJiR2ADhe0r72/FI21x3hJyu28dOV21lSs4uyEXncOHMcH6ksY3RB/16lHDzexP3PrmPhim2MKx7Kj//yYi6dNKpfY+gr0yJFvPLm/mSHMehosD3N7Tl0guVb6/nKk6s52nj6khRZGcYlZ488Zbpr7LpIb89KyiYrs2dTPTu6Etp7pIFfrd3Nkupd/HHTXppanDMKc5k1bQxXl49h5vjiHn/mQNHQ3MLSmt08/vJbvLh5H1kZxpVTz+DGmeN4dz9cpSyp3slXf17DviMN/NVlZ/P5959DXs7ASNjxzH9+E//07Ou88tUrKR42cK6mkiXRwXYlkjTi7myrP87LW/axYms9y7fUs3XfsS7Pu2Dc8JOzkA6diD/1FaAoL/vkPRbFw3JO3oPRPum0vYbmZPLzVTtO6+bJzjTGjhjK1n1HaXUYVzyUa8qjyWN62fCkd/Eky+a6IyxcsY2fVm1j/7EmxhbnMfeicK5Sdh86wT0/r2ZpzW6mnlnIAze8g4qyoj79jGT446a93PToy/zw0zO5/Bw9ObUrSiQx0jWRtLZG759YvmUfy7fuZ/mWfew+1ADA8KHZXDS+mIsnFHPR+GL++scr2XHgxGl1tJ+S2tTSetpaR/uD+yxOfm23JlK8ezUAcrMyaGppJd7M2KwM47b3TmJW+RjOHVMwIAZz+0tDcwtLqnfx+PK3eGlzPVkZxlXTolcpl07s3VVKa6vzk6pt/NOz62hsbuXz7z+Hv7xsAtmD5Mrv4PEmzv/aL7nj6inc9t5JyQ4n5Wn6bxpqammlZschVmyp5+Ut9VS9Wc+BY00AjCkcwsUTRnLRhGjymFSSf8ofnC9ffW5CdwBnZ2YwumBIwv8DdneONDS3u7fi7dd/Pr857nktrc7fXHlOd78FaSE3K5PZ0yPMnh5hU90RHn/5LZ58ZTvPrtnFuOKhzJ05lo9cOJaSgu7dK7G57gh3/WwNL2+p55Kzi7n/+ncwYdSwkFqRHEV52Zw1cijVGnDvU0okA9iJphZWbTvA8i3RbqpX3trPsWCcY8KoYVw19QxmThjJzPHFjC3O6/R/9V0NAPeUmVEwJJuCIdmcNfL0P0q/WB1/qRKthZSYiSX53P2hqXzp6iksrdnFgpff4sEl6/nmr97gqqljuHHmON41cWSnVylNLa3Mf34z//qbDeRmZfDADRV8tHLsoL0KLI8UsXr7gWSHMaiEmkjMbBbwr0Sfr/49d//nDo67AXgSuMjdq8xsPLAOWB8c8pK73xoceyHwX0Ae8Cxwuw+i/rnOpuAeOtHEyjf3s3xLPSu21PPa9gM0tThmMOWMAj5yYRkXTShm5vjiHt1/0B/3VLSntZD6xpDst69SNu45wuPL3+KpV7azeM1Ozho59ORYyqj83FP+jY0qyCU7w9hx8ATXlI/ha9dNS7l7V/paeWkRi1fv5MCxxgE1fTmVhTZGYmaZwBvAlcB2YAVwo7uvbXdcAbAYyAHmxSSSX7h7eZx6lwOfA14mmki+7e7PdRbLQBkjifdc65xM4+KzR1J/tJF1Ow/R6tHxg4qyImYGSaPyrOKTN7sNRAPl/pWB5kRTdCxlwctvsXxrPdmZxrTSQtbuOHzKc0EA/vzS8fz9tdOSFGn/+v2GvfzZ91/mR39xMe+ePDCnMfeXVBgjmQlsdPfNQUALgdnA2nbHfR14ALijqwrN7Eyg0N1fCrZ/CMwBOk0kqe5EUwtb9h7la/9bc9qy5o0tzgsb9vKuiSP57BWTuXhCMdPHDR9US5wn40ooHQzJzjz5vd245zALXt7GY3/cEvfhT7+s2Z02iaQ8Er3DvXrHQSWSPhLmX6MIsC1meztwcewBZnYBMNbdF5tZ+0QywcxeBQ4Bd7v7C0Gd29vVOWD+Ah060cTGPUfYuOcIm4KvG+uOsK3+WNyZS20MWPBXl/RbnDL4TBpdwD3XTuWxP2yJuz+dHvQ0fGgOZSPydId7H0raf2vNLAN4GPhUnN07gXHuvi8YE3nGzLr13yUzuwW4BWDcuHG9jDZx7k7dkYbTksXGPUdOTr0FyMnMYMKoYZSXFjF7eoRJo/O57xdr2XO44bQ6NfAsfaVUD3oCoCJSpJlbfSjMRFILjI3ZLgvK2hQA5cCyYHbIGGCRmV3n7lVAA4C7rzSzTcA5wfllndR5krvPB+ZDdIyku8F31W/f2ups33+cjXWHT15ltL1ib9rLz81iYskwLp00ismjC5g0Op9Jo/MZOyLvtLuyW1tdA88SKk1uiCqPFPFc9S4OHm+iKG/gji+mijATyQpgsplNIPrHfi5wU9tOdz8InOygNLNlwJeCwfYSoN7dW8zsbGAysNnd683skJldQnSw/RPAv/V14PHWnfryk6+xtGYXWZkZbNxzhM11R04+mAhgVH4OE0vyufb80pPJYtLofMYUDkl4GmVYU3BF2ujfWFTbkvI1Ow7yrokaJ+mt0BKJuzeb2TxgKdHpvz9w9xozuxeocvdFnZx+OXCvmTUBrcCt7l4f7Pt/vD399zlCGGiP9yzvxhbnuero4nmTRudz6cSRpySMvppGqIFnCZv+jUW7tgCqa5VI+kKoYyTu/izRKbqxZfd0cOx7Yt4/BTzVwXFVRLvEQtPRwKMBv//KwHmCnYjEVzwsh8jwPKprDyU7lEFhcCyg08c6GnhMtwFJkcFsWmmhBtz7iBJJHHdcPYW8ds+2SMcBSZHBrCJSxOa9Rzl8oinZoQx4SiRxzJkR4f7rK4gMz8OIroB7//UVad+vLDKYlAfL4q/doe6t3ho8t0f3MQ1Iigxu5aXRRLKm9iAXnz0yydEMbLoiEZG0VFKQy5jCIRon6QNKJCKStsojRVSra6vXlEhEJG2VRwrZVHeEow0dP0JauqZEIiJpqyJShDus3amrkt5QIhGRtBV7h7v0nBKJiKSt0YVDKCnI1ZLyvaREIiJpTUvK954SiYiktfJIERv3HOF4Y0vXB0tcSiQiktbKSwtp1YB7ryiRiEhaqyjTgHtvKZGISFobUziEUfk5SiS9oEQiImnNzCiPFGnmVi8okYhI2isvLWLDniOcaNKAe08okYhI2iuPFNHS6ry+63CyQxmQlEhEJO21Dbire6tnQk0kZjbLzNab2UYzu7OT424wMzezymD7SjNbaWZrgq9XxBy7LKhzVfAaHWYbRGTwKy0awoih2VRvVyLpidAebGVmmcAjwJXAdmCFmS1y97XtjisAbgdejineC1zr7jvMrBxYCsQ+Zepmd68KK3YRSS8acO+dMK9IZgIb3X2zuzcCC4HZcY77OvAAcKKtwN1fdfcdwWYNkGdmuSHGKiJpriJSxBu7D9PQrAH37gozkUSAbTHb2zn1qgIzuwAY6+6LO6nnBuAVd2+IKXss6Nb6qplZn0UsImmrPFJEc6uzXgPu3Za0wXYzywAeBr7YyTHTiF6tfCam+GZ3rwAuC14f7+DcW8ysysyq6urq+i5wERmU2paUV/dW94WZSGqBsTHbZUFZmwKgHFhmZluBS4BFMQPuZcDTwCfcfVPbSe5eG3w9DCwg2oV2Gnef7+6V7l5ZUlLSZ40SkcGpbEQeRXnZVNdqza3uCjORrAAmm9kEM8sB5gKL2na6+0F3H+Xu4919PPAScJ27V5nZcGAxcKe7/6HtHDPLMrNRwfts4ENAdYhtEJE0ER1wL9RSKT0QWiJx92ZgHtEZV+uAJ9y9xszuNbPrujh9HjAJuKfdNN9cYKmZrQZWEb3CeTSsNohIeimPFLF+12Eam1uTHcqAEtr0XwB3fxZ4tl3ZPR0c+56Y9/cB93VQ7YV9FZ+ISKyKSBGNLa28sfsw5cGYiXRNd7aLiATKS7WkfE8okYiIBM4aOZSCIVmaudVNSiQiIgEzo7y0iOodmrnVHUokIiIxyiOFrNt5iKYWDbgnqstEYmbfCG4MFBEZ9MojRTQ2t7Jh95FkhzJgJHJFsg6Yb2Yvm9mtZqapDCIyaLXd4V69Q+Mkieoykbj799z9UuATwHhgtZktMLP3hh2ciEh/Gz9yGPm5WZq51Q0JjZEES8KfG7z2Aq8BXzCzhSHGJiLS7zIyjKmlhZq51Q2JjJF8E3gd+ADwT+5+obs/4O7XAjPCDlBEpL9VRIpYt/MQzRpwT0giVySrgenu/hl3X95uX9wFE0VEBrLySCEnmlrZVHc02aEMCIkkkgPELKViZsPNbA5EF14MKzARkWTRkvLdk0gi+fvYhOHuB4C/Dy8kEZHkmjAqn6E5mRpwT1AiiSTeMaEu9igikkyZGcbUM7WkfKISSSRVZvawmU0MXg8DK8MOTEQkmcojRdTsOERLqyc7lJSXSCL5LNAI/CR4NQC3hRmUiEiyVUSKON7Uwpa9usO9K112Ubn7UeDOfohFRCRllMcMuE8aXZDkaFJbl4nEzEqALwPTgCFt5e5+RYhxiYgk1cSSYQzJzmDN9kP8qe6Y61QiXVs/JnpD4gTga8BWos9jFxEZtLIyM6ID7lpzq0uJJJKR7v59oMndf+funwYSuhoxs1lmtt7MNppZh91jZnaDmbmZVcaU3RWct97Mru5unSIivVUeKWLtjkO0asC9U4kkkqbg604z+6CZzQCKuzopWJ/rEeAaYCpwo5lNjXNcAXA78HJM2VRgLtHutFnAv5tZZqJ1ioj0hfJIEUcamtmyT3e4dyaRRHJfsHT8F4EvAd8D/iaB82YCG919s7s3AguB2XGO+zrwAHAipmw2sNDdG9x9C7AxqC/ROkVEeu3kkvK6n6RTnSaS4ApgsrsfdPdqd39vsGjjogTqjgDbYra3B2Wx9V8AjHX3xQme22WdIiJ9ZfLofHKzMpRIutBpInH3FuDGMD7YzDKAh4le6YRR/y1mVmVmVXV1dWF8hIgMclmZGZx7ppaU70oiXVt/MLPvmNllZnZB2yuB82qBsTHbZUFZmwKgHFhmZluBS4BFwYB7R+d2VedJ7j7f3SvdvbKkpCSBcEVETlcRKaSmVgPunUlkzazpwdd7Y8qcrmdurQAmm9kEon/s5wI3nawguhDkqLZtM1sGfMndq8zsOLAgWI6lFJgMLAesszpFRPpaRaSIH730Fm/VH2P8qGHJDiclJXJne48eqevuzWY2D1gKZAI/cPcaM7sXqOpsnCU47glgLdAM3BZ0sxGvzp7EJyKSiGmlb9/hrkQSXyJ3tt8Tr9zd741X3u6YZ4Fn25V1VN972m3/I/CPidQpIhKWc84oICczOuB+7fmlyQ4nJSXStRU7gXoI8CFgXTjhiIiklpysDM49s0B3uHcika6tb8Rum9m/EO1aEhFJC9NKi3h2zU7cHTNLdjgpJ5FZW+0NJTpbSkQkLVREijh4vIlt9ceTHUpKSmSMZA3RWVoQHeAu4dQZXCIig9rJO9x3HGTcyKFJjib1JDJG8qGY983AbndvDikeEZGUc86YfLIzjTW1B/lAxZnJDiflJNK1dSZQ7+5vunstkGdmF4ccl4hIysjNyuScMwq0VEoHEkkk3wVinzV5NCgTEUkbFZEiqmsP4q473NtLJJGYx3zn3L2VxLrEREQGjWmRIvYfa6L2gAbc20skkWw2s8+ZWXbwuh3YHHZgIiKpREvKdyyRRHIr8C6ia1ttBy4GbgkzKBGRVHPumAIyM4zq2kPJDiXlJHJD4h6iiyOKiKStIdmZTB6dryXl4+jyisTM/tvMhsdsjzCzH4QblohI6tGAe3yJdG29w90PtG24+35gRnghiYikpoqyIvYdbWTXoRNdH5xGEkkkGWY2om3DzIrRrC0RSUMnl5Tfru6tWIkkkm8AL5rZ183sPuCPwIPhhiUiknqmnllIhmnmVnuJDLb/0MxWAm0PuLre3deGG5aISOrJy8lk8ugCqndo5lashLqogicW1hF9HglmNs7d3wo1MhGRFDQtUsgLG/YmO4yUksisrevMbAOwBfgdsBV4LuS4RERSUkWkiLrDDezWgPtJiYyRfB24BHjD3ScA7wNeCjUqEZEUpTvcT5dIImly931EZ29luPtvgcpEKjezWWa23sw2mtmdcfbfamZrzGyVmf3ezKYG5TcHZW2vVjObHuxbFtTZtm90N9orItIr551ZiBm6MTFGImMkB8wsH3ge+LGZ7eHU57jHZWaZwCPAlUSXVllhZovaDdQvcPf/CI6/DngYmOXuPwZ+HJRXAM+4+6qY825296oEYhcR6VPDcrOYWJKvK5IYiVyRzAaOAX8DLAE2AdcmcN5MYKO7b3b3RmBhUNdJ7h479WEYbz+JMdaNwbkiIikheoe7Zm61SWT6b9vVRyvw392oOwJsi9luW/DxFGZ2G/AFIAe4Ik49H6NdAgIeM7MW4CngPtd6BSLSj6aVFvL0q7XUHW6gpCA32eEkXSJXJKFy90fcfSLwFeDu2H3BkxiPuXt1TPHN7l4BXBa8Ph6vXjO7xcyqzKyqrq4upOhFJB1pwP1UYSaSWmBszHZZUNaRhcCcdmVzgcdjC4LH/eLuh4EFRLvQTuPu89290t0rS0pKuhm6iEjHpimRnCKR+0huT6QsjhXAZDObYGY5RJPConb1TI7Z/CCwIWZfBvBRYsZHzCzLzEYF77OBDwGxVysiIqHLz83i7FHDNHMrkMgVySfjlH2qq5PcvRmYBywF1gFPBHfI3xvM0AKYZ2Y1ZraK6DhJ7GddDmxz99inMeYCS81sNbCK6BXOowm0QUSkT5UHS8pLJ4PtZnYjcBMwwcxiryQKgPpEKnf3Z4Fn25XdE/O+wysbd19G9EbI2LKjwIWJfLaISJgqIkUsem0H+440MDI/vQfcO5u19UdgJzCK6ArAbQ4Dq8MMSkQk1ZW3jZPsOMSfnJPe47Addm25+5vuvszd30l0fa1sd/8d0W6qvH6KT0QkJU2LFAIacIfEBtv/CngS+M+gqAx4JsygRERSXeGQbMaPHKqHXJHYYPttwKXAIQB33wBofSsRSXvlkSKqdyiRJJJIGoIlToDoFFziL2UiIpJWyiNFbN9/nP1HG7s+eBBLJJH8zsz+FsgzsyuBnwL/G25YIiKp7+Qd7ml+VZJIIrkTqAPWAJ8hOp337k7PEBFJA+WlbXe4p/cCjoks2thK9Ka/R82sGCjTIokiIlA0NJuxxXlpP3MrkVlby8ysMEgiK4kmlG+GH5qISOqriBSl/VIpiXRtFQXPDbke+KG7X0z0cbsiImmvPFLEW/XHOHisKdmhJE0iiSTLzM4kuoDiL0KOR0RkQGkbJ6lJ4wH3RBLJvUQXXtzo7ivM7GxiVukVEUlnbTO30rl7K5HB9p8SnfLbtr0ZuCHMoEREBooRw3KIDM+jekf6ztxK+hMSRUQGuvJIYVrP3FIiERHppYpIEVv2HuXQifQccE9k+m9mfwQiIjJQtS0pvzZNu7cSuSLZYmbzzex9ZmahRyQiMsCUp/kz3BNJJOcCvya6CvAWM/uOmb073LBERAaOUfm5nFk0JG1nbnWZSNz9mLs/4e7XAzOAQuB3iVRuZrPMbL2ZbTSzO+Psv9XM1pjZKjP7vZlNDcrHm9nxoHyVmf1HzDkXBudsNLNv6ypJRFJBOj/DPaHBdjP7EzP7d6JLpAwhenNiV+dkAo8A1wBTgRvbEkWMBe5e4e7TgQeBh2P2bXL36cHr1pjy7wJ/BUwOXrMSaYOISJjKS4vYvPcoRxqakx1Kv0tksH0r8HngBaDC3T/q7k8lUPdMojcxbg6eZ7IQmB17QLD0SpthdPGck+AO+0J3fylYOPKHwJwEYhERCVVFWSHu6Tng3uUNicA72v3BT1QE2BazvR24uP1BZnYb8AUgB7giZtcEM3uV6JMZ73b3F4I6t7erM9KD2ERE+lTsgPvMCcVJjqZ/JdK1NcbMfmNm1QBm9g4z67Pnkbj7I+4+EfgKbz/nZCcwzt1nEE0yC8yssDv1mtktZlZlZlV1dXV9Fa6ISFyjC4YwuiA3LcdJEkkkjwJ3AU0A7r4amJvAebXA2JjtsqCsIwsJuqncvcHd9wXvVwKbgHOC88sSqdPd57t7pbtXlpSUJBCuiEjvpOuS8okkkqHuvrxdWSKjSSuAyWY2wcxyiCafRbEHmNnkmM0PEiwGaWYlbTdCBotETgY2u/tO4JCZXRLM1voE8PMEYhERCV15pIhNdUc41pheA+6JjJHsNbOJBAPhZvZhol1PnXL3ZjObR3Tl4EzgB+5eY2b3AlXuvgiYZ2bvJ3q1sx/4ZHD65cC9ZtYEtAK3unt9sO//Af8F5AHPBS8RkaQrjxTR6rBu5yEuPCt9xkkSSSS3AfOBc82sFtgC/Fkilbv7s0Sf8R5bdk/M+9s7OO8pIO7MMHevAsoT+XwRkf50ckn57QeVSGIFy8a/38yGARnufjj8sEREBp4zCnMZlZ+bdkvKd5hIzOwLHZQD4O4Px9svIpKuzCwtl5Tv7IqkIPg6BbiItwfKrwXaD76LiAjR7q0XNuzlRFMLQ7LTY/H0DhOJu38JL3zYAAAOJUlEQVQNwMyeBy5o69Iys38AFvdLdCIiA0x5pIiWVmfdzkPMGDci2eH0i0Sm/54BNMZsNwZlIiLSTjouKZ/IrK0fAsvN7Olgew7R6bciItJOadEQioflpNWNiYnM2vpHM3sOuCwo+nN3fzXcsEREBqbogHsR1bXpM3MrkSsS3P0V4JWQYxERGRQqIoX85+82p82Ae0LPIxERkcSVlxbR3Oqs35Uet90pkYiI9LG2Afd0GSdRIhER6WNlI/IYPjSbmh1KJCIi0gNmRnlp+iwpr0QiIhKC8kgR63cdpqG5JdmhhE6JREQkBBWRIppanA27jyQ7lNApkYiIhKA8En06eDp0bymRiIiEYFzxUAqHZCmRiIhIz7Td4V6jRCIiIj1VHili3a7DNLW0JjuUUCmRiIiE5HhTC43NrZzzd89x6T//H8+8WpvskEIRaiIxs1lmtt7MNprZnXH232pma8xslZn93symBuVXmtnKYN9KM7si5pxlQZ2rgtfoMNsgItITz7xayxMrtgHgQO2B49z1szWDMpmElkjMLBN4BLgGmArc2JYoYixw9wp3nw48CLQ9vncvcK27VwCfBP6n3Xk3u/v04LUnrDaIiPTUQ0vX09B8apfW8aYWHlq6PkkRhSfMK5KZwEZ33+zujcBCYHbsAe4eu87yMKKJG3d/1d13BOU1QJ6Z5YYYq4hIn9px4Hi3ygeyMBNJBNgWs709KDuFmd1mZpuIXpF8Lk49NwCvuHtDTNljQbfWV83M4n24md1iZlVmVlVXV9fzVoiI9EDp8LxulQ9kSR9sd/dH3H0i8BXg7th9ZjYNeAD4TEzxzUGX12XB6+Md1Dvf3SvdvbKkpCSc4EVEOnDH1VPIa/csEgNuf9+k5AQUojATSS0wNma7LCjryEKij/EFwMzKgKeBT7j7prZyd68Nvh4GFhDtQhMRSSlzZkS4//oKIsPzMGBUfg4OVL25P9mh9bmEnpDYQyuAyWY2gWgCmQvcFHuAmU129w3B5geBDUH5cGAxcKe7/yHm+CxguLvvNbNs4EPAr0Nsg4hIj82ZEWHOjLd79L/xy/X82/9t5NJJo5g9/bSe/gErtCsSd28G5gFLgXXAE+5eY2b3mtl1wWHzzKzGzFYBXyA6Q4vgvEnAPe2m+eYCS81sNbCKaIJ6NKw2iIj0pdvfN5nKs0bwd09X8+a+o8kOp8+Yuyc7htBVVlZ6VVVVssMQEaH2wHGu+dbzjB81jCdvfRc5WUkfqu6Qma1098qujkvdFoiIDEKR4Xk8+OHzWb39IA8tfT3Z4fQJJRIRkX42q3wMn3jnWTz6whZ++/rAv6daiUREJAn+9gPnce6YAr7409fYfehEssPpFSUSEZEkGJKdyXduuoDjjS18fuEqWloH7ni1EomISJJMGp3P12ZP48XN+/juso3JDqfHlEhERJLoIxeWMXt6Kd/89QZWbK1Pdjg9okQiIpJEZsZ9c8opG5HH7Y+/yoFjjckOqduUSEREkqxgSDb/duMM6o408OUnVzPQ7u9TIhERSQHvKBvOV2adyy/X7uZ/Xnoz2eF0ixKJiEiK+It3T+C9U0q4b/E61u441PUJKUKJREQkRZgZ//KR8xmel828x1/hWGNzskNKiBKJiEgKGZmfy7fmTmfL3qP8/c9rkh1OQpRIRERSzLsmjuKz753ET1du55lXO3uMU2pQIhERSUGfe99kLho/gr97eg1b96b2kvNKJCIiKSgrM4N/nTuDrMwM5j3+Cg3NLckOqUNKJCIiKap0eB4PfvgdVNce4sEl65MdToeUSEREUtjV08bwyXeexfd/v4X/e313ssOJS4lERCTF3fWB8zjvzEK+9NPV7DqYekvOh5pIzGyWma03s41mdmec/bea2Zrgmey/N7OpMfvuCs5bb2ZXJ1qniMhgE11yfgYnmlr4/E9eTbkl50NLJGaWCTwCXANMBW6MTRSBBe5e4e7TgQeBh4NzpwJzgWnALODfzSwzwTpFRAadiSX5fO26aby0uZ5HfptaS86HeUUyE9jo7pvdvRFYCMyOPcDdY9cAGAa0pdnZwEJ3b3D3LcDGoL4u6xQRGaw+fGEZc6aX8q1fv8HyLamz5HyYiSQCbIvZ3h6UncLMbjOzTUSvSD7XxbkJ1SkiMhiZGff9aQXjiody+8JX2X80NZacT/pgu7s/4u4Tga8Ad/dVvWZ2i5lVmVlVXV1dX1UrIpJU+blZ/NuNF7D3SANffio1lpwPM5HUAmNjtsuCso4sBOZ0cW7Cdbr7fHevdPfKkpKSboYuIpK6KsqKuPOa8/jV2t388MXkLzkfZiJZAUw2swlmlkN08HxR7AFmNjlm84PAhuD9ImCumeWa2QRgMrA8kTpFRNLBpy8dz/vOHc0/Ll5HzY6DSY0ltETi7s3APGApsA54wt1rzOxeM7suOGyemdWY2SrgC8Ang3NrgCeAtcAS4DZ3b+mozrDaICKSqsyMhz5yPiOGZfPZBa9ytCF5S85bKvSvha2ystKrqqqSHYaISJ97cdM+bvreS1w/o4xvfPT8Pq3bzFa6e2VXxyV9sF1ERHrunRNH8tkrJvPUK9t5+tXtSYlBiUREZID73BWTmDm+mLufrmZLEpacVyIRERngsjIz+Nbc6WRnZfDZJCw5r0QiIjIIlA7P48EbokvOP/Bc/y45n9WvnyYiIqG5atoYPvWu8fzgD1t4ZlUt+482Ujo8jzuunsKcGeEtAqJEIiIyiEwrLcSA+mD5lNoDx7nrZ2sAQksm6toSERlEvvXrDbS/qeN4UwsPLQ2vu0uJRERkENlx4Hi3yvuCEomIyCBSOjyvW+V9QYlERGQQuePqKeRlZ55SlpedyR1XTwntMzXYLiIyiLQNqD+0dD07DhzXrC0REem+OTMioSaO9tS1JSIivaJEIiIivaJEIiIivaJEIiIivaJEIiIivZIWT0g0szrgABD7YOOiTrZj348C9vZBGO0/rzfHxtufSNlAbXNH+9Tm+GVqc/w291V7O4qpJ8f1VZvD+hmf5e4lXR7l7mnxAuYnut3ufVUYn9+bY+PtT6RsoLa5o31qs9rcnTb3VXu70+ae/C73pM1h/4y7eqVT19b/dmO7/b4wPr83x8bbn0jZQG1zR/vU5vhlanPqtLknv8sdlSfaxjDa26m06NrqDTOrcvfKZMfRn9Tm9JBubU639kL/tTmdrkh6an6yA0gCtTk9pFub06290E9t1hWJiIj0iq5IRESkV5RIRESkV5RIRESkV5RIesHMzjOz/zCzJ83sr5MdT38wszlm9qiZ/cTMrkp2PGEzs7PN7Ptm9mSyYwmTmQ0zs/8OfrY3Jzue/pAuP9tYof3+9sfNKqn4An4A7AGq25XPAtYDG4E7E6wrA/hRstvUz20eAXw/2W3qx/Y+mez2hNl+4OPAtcH7nyQ79v78mQ/En20ftLlPf3+T/k1I4jf/cuCC2G8+kAlsAs4GcoDXgKlABfCLdq/RwTnXAc8BNyW7Tf3V5uC8bwAXJLtN/djeAffHppvtvwuYHhyzINmx90ebB/LPtg/a3Ke/v2n7hER3f97MxrcrnglsdPfNAGa2EJjt7vcDH+qgnkXAIjNbDCwIL+Le64s2m5kB/ww85+6vhBtx7/TVz3ig6k77ge1AGbCKAdzl3c02r+3f6MLRnTab2TpC+P0dsP9gQhIBtsVsbw/K4jKz95jZt83sP4Fnww4uJN1qM/BZ4P3Ah83s1jADC0l3f8Yjzew/gBlmdlfYwfWDjtr/M+AGM/suSVhiI2Rx2zwIf7axOvo5h/L7m7ZXJH3B3ZcBy5IcRr9y928D3052HP3F3fcBAzFhdou7HwX+PNlx9Kd0+dnGCuv3V1ckp6oFxsZslwVlg1m6tTnd2tteOrZfbQ65zUokp1oBTDazCWaWA8wFFiU5prClW5vTrb3tpWP71eaQ25y2icTMHgdeBKaY2XYz+wt3bwbmAUuBdcAT7l6TzDj7Urq1Od3a2146tl9tTk6btWijiIj0StpekYiISN9QIhERkV5RIhERkV5RIhERkV5RIhERkV5RIhERkV5RIhFJIjMbb2Y3xWx/ysy+08Gxz5rZ8P6LTiQxSiQiyTUeuKmrgwDc/QPufiDccES6T4lEJI7giYGLzew1M6s2s4+Z2VYzu9/MVplZlZldYGZLzWxT20qqFvVQcM4aM/tYZ+VEl/S+LKjzb4KyUjNbYmYbzOzBmJi2mtmo4CpmXfCkuxoz+6WZ5QXHXGRmq4P6HjKz6n78tkmaUiIRiW8WsMPdz3f3cmBJUP6Wu08HXgD+C/gwcAnwtWD/9cB04Hyiy3U/ZGZndlJ+J/CCu093928GdUwHPkb0YVsfM7PYxffaTAYecfdpwAHghqD8MeAzQYwtvf82iHRNiUQkvjXAlWb2gJld5u4Hg/JFMftfdvfD7l4HNATjF+8GHnf3FnffDfwOuKiT8nh+4+4H3f0E0YcvnRXnmC3uvip4vxIYH3x+gbu/GJSn9IPWZPDQ80hE4nD3N8zsAuADwH1m9ptgV0PwtTXmfdt2X/0+xdbb0kG97Y/J66PPFuk2XZGIxGFmpcAxd/8R8BDRZ2In4gWi3VGZZlZC9HnayzspPwwU9EXMwUD8YTO7OCia2xf1inRFVyQi8VUQHcdoBZqAvwaeTOC8p4F3Aq8BDnzZ3XeZWUfl+4AWM3uN6JjL/l7G/RfAo0HcvwMOdnG8SK9pGXmRQcTM8t39SPD+TuBMd789yWHJIKcrEpHB5YNmdhfR3+03gU8lNxxJB7oiERGRXtFgu4iI9IoSiYiI9IoSiYiI9IoSiYiI9IoSiYiI9IoSiYiI9Mr/Byfv1536lxz7AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fd70f0950f0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.semilogx(list(scores.keys()),list(scores.values()),'o-');\n",
    "plt.xlabel('smoothing')\n",
    "plt.ylabel('dev set accuracy');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Reflect:**\n",
    "\n",
    "- what might explain the dramatic drop in accuracy when the smoothing is increased from $10$ to $30$?\n",
    "- before you check, predict whether the accuracy will continue to significantly drop if you further increase the smoothing to $10000$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "theta_nb = naive_bayes.estimate_nb(x_tr_pruned,y_tr,best_smoother)\n",
    "y_hat = clf_base.predict_all(x_te_pruned,theta_nb,labels)\n",
    "evaluation.write_predictions(y_hat,'nb-best-test.preds')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.5555555555555556\n"
     ]
    }
   ],
   "source": [
    "# you can't run this\n",
    "y_hat = evaluation.read_predictions('nb-best-test.preds')\n",
    "print(evaluation.acc(y_hat,y_te))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 4. Perceptron \n",
    "\n",
    "Total: 1.5 points\n",
    "\n",
    "The perceptron update is,\n",
    "\n",
    "\\begin{align}\n",
    "\\hat{y} = & \\text{argmax}_y \\theta^\\top f(x,y)\\\\\n",
    "\\theta \\gets & \\theta + f(x,y) - f(x,\\hat{y})\n",
    "\\end{align}\n",
    "\n",
    "You will now implement this classifier, using the file ```gtnlplib/perceptron.py```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from gtnlplib import perceptron\n",
    "reload(perceptron);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- **Deliverable 4.1**: Implement the perceptron *update*, $f(x,y) - f(x,\\hat{y})$, in the function ```perceptron_update``` in ```perceptron.py```. (0.5 points)\n",
    "- **Test**: `tests/test_perceptron.py:test_d4_1_perc_update`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "theta_perc = defaultdict(float)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "defaultdict(<class 'float'>, {})\n"
     ]
    }
   ],
   "source": [
    "# no update when the prediction is correct\n",
    "i=20\n",
    "update = perceptron.perceptron_update(x_tr_pruned[i],y_tr[i],theta_perc,labels)\n",
    "print(update)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[(('2000s', 'with'), 1), (('2000s', 'her'), 1), (('2000s', 'lips'), 1), (('2000s', 'yeah'), 1), (('2000s', 'shes'), 2)]\n",
      "122\n",
      "2000s **OFFSET** 1\n",
      "1980s **OFFSET** -1.0\n"
     ]
    }
   ],
   "source": [
    "# update when the prediction is incorrect\n",
    "i=110\n",
    "y_hat,_ = clf_base.predict(x_tr_pruned[i],theta_perc,labels)\n",
    "update =perceptron.perceptron_update(x_tr_pruned[i],y_tr[i],theta_perc,labels)\n",
    "print(list(update.items())[:5])\n",
    "print(len(update))\n",
    "print(y_tr[i],constants.OFFSET,update[((y_tr[i],constants.OFFSET))])\n",
    "print(y_hat,constants.OFFSET,update[((y_hat,constants.OFFSET))])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-1.0"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "update[(('1980s','with'))]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "122"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(update)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now implement the perceptron algorithm. Your implementation should take as inputs:\n",
    "\n",
    "- The training instances $x$\n",
    "- The training labels $y$\n",
    "- The number of iterations to train\n",
    "\n",
    "It should use your ```update``` function, and it should return:\n",
    "\n",
    "- weights $\\theta$\n",
    "- a list of the weights at each iteration\n",
    "\n",
    "\n",
    "- **Deliverable 4.2**: Implement ```estimate_perceptron``` in ```perceptron.py``` (1 point)\n",
    "- **Test**: `tests/test_perceptron.py:test_d4_2a_perc_estimate`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "reload(perceptron);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "theta_perc,theta_perc_history = perceptron.estimate_perceptron(x_tr_pruned[:10],y_tr[:10],3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-15.0\n",
      "4.0\n"
     ]
    }
   ],
   "source": [
    "print(theta_perc[('1980s','its')])\n",
    "print(theta_perc[('1980s','what')])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For reference, here is the running time on a relatively modern consumer-grade machine:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 274,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "11.3 s ± 390 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)\n"
     ]
    }
   ],
   "source": [
    "%%timeit\n",
    "theta_perc,theta_perc_history = perceptron.estimate_perceptron(x_tr_pruned,y_tr,20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "theta_perc,theta_perc_history = perceptron.estimate_perceptron(x_tr_pruned,y_tr,20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# run this to plot the accuracy over iterations\n",
    "def plot_accs(weight_history,x_tr=x_tr_pruned,y_tr=y_tr,x_dv=x_dv_pruned,y_dv=y_dv):\n",
    "    tr_accs = []\n",
    "    dv_accs = []\n",
    "    for theta in weight_history:\n",
    "        tr_accs.append(evaluation.acc(clf_base.predict_all(x_tr,theta,labels),y_tr))\n",
    "        dv_accs.append(evaluation.acc(clf_base.predict_all(x_dv,theta,labels),y_dv))\n",
    "    plt.plot(tr_accs,'--')\n",
    "    plt.plot(dv_accs)\n",
    "    plt.xlabel('iteration')\n",
    "    plt.ylabel('accuracy');\n",
    "    plt.legend(['training','dev'],loc='lower right');\n",
    "    return tr_accs,dv_accs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.4311111111111111\n"
     ]
    }
   ],
   "source": [
    "y_hat = clf_base.predict_all(x_dv_pruned,theta_perc,labels)\n",
    "print(evaluation.acc(y_hat,y_dv));"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEKCAYAAADjDHn2AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvAOZPmwAAIABJREFUeJzs3Xd4VGX2wPHvyaSHkIQk1EBIIGDokNCkCAgIFkQsgA0rrmV1i/pT18qua1lXV11cRcWu2BUVRYogRYQgIBBqEkpIgJCENNLn/f1xJzCElEkyd2YS3s/zzAO5c+/MSSBz7tvOK0opNE3TNK0uXu4OQNM0TfN8OllomqZp9dLJQtM0TauXThaapmlavXSy0DRN0+qlk4WmaZpWL50sNE3TtHrpZKFpmqbVSycLTdM0rV7e7g7AWSIiIlTXrl3dHYamaVqzsnHjxmNKqcj6zmsxyaJr164kJSW5OwxN07RmRUT2O3Ke7obSNE3T6qWThaZpmlYvnSw0TdO0epmaLERkkojsEpG9IvJADc+/ICKbbY/dInLc7rlKu+cWmhmnpmmaVjfTBrhFxALMBSYA6cAGEVmolEquOkcp9We78/8IDLR7iWKl1ACz4tM0TdMcZ2bLYgiwVymVqpQqAxYAl9Zx/kzgIxPj0TRN0xrJzGTRCTho93W67dgZRCQaiAGW2x32F5EkEVknIlPNC1PTNE2rj6ess5gBfKaUqrQ7Fq2UOiQiscByEdmqlEqxv0hEZgOzAbp06eK6aDVN084yZrYsDgGd7b6Osh2ryQyqdUEppQ7Z/kwFVnD6eEbVOfOUUolKqcTIyHoXIGqaxykoKedfi3dyMOeEu0PRtDqZmSw2AHEiEiMivhgJ4YxZTSJyDhAG/GJ3LExE/Gx/jwBGAMnVr9W05m7ZjqPM/SmFI/kl7g5F0+pkWrJQSlUAdwGLgR3AJ0qp7SIyR0Sm2J06A1iglFJ2x+KBJBHZAvwEPG0/i0rTWorvtmYC8K/Fu7BaVT1na5r7mDpmoZRaBCyqduzRal8/XsN1a4G+Zsamae5WUFLOyt1Z+Fq8+DUth8P5JXQMDXB3WJpWI72CW9PcZNmOo5RVWPnjuO4ApGYVuTkizUwl5ZXsOlzA4u2HT7Yi5/2cwrlPLeNPCzZxeueK5/GU2VCadtY5WlBCbEQQVyRG8e8lu0k9VsjIuAh3h6U1QXFZJQdyThAdHoi/j4WlyUd4c3Ua+7KLyMw7NS619oFxdAwNIDLYj+jwIL7anMGkPu2Z1KeDG6Ovm04WmuYms0d345aRsYhAkK/lrG1Z/J5+nGB/H2IigtwdikOsVoVVKbwtXuw6XMDba/eRdqyQfcdOcNg2UeHz288lITqMCquipKKS4bHhRIcH0TUikJiIICJa+QFw2cAoLunXkUv+u4YnvklmVFwkQX6e+bHsmVFpWgtXUl6Jv48FLy8BYHSPSEICfNwclettO5THla/+Qt9OIXx2+7nuDucMBSXl/Lz7GClZhew9WkhKViGpWUU8d2V/LurXgbzicr7flklsRBDndg8nJjyI6Iigk4nPaC20r/M9vC1e/GNqHy7/31peW5nCXyb2dMW31mDi6f1kjkpMTFR68yOtubj13SQqrYr5Nwx2dyhuk1tUxiX/XU16bjEikPS38YTb7rhdqaS8ku0ZeaQcLTqZFCb37cAVCVEczDnBqGd/AiAqLIBuka3o3rYVUwd0om9UCEopRMQpcSzcksGYnpG09nftTYOIbFRKJdZ3nm5ZaJqLVc2Cumbo2V114F8/7uJofikvzRxIsJ83rfzd83E0a/56fk3LAcDX24vYiCBKK4xiEp1CA/ju7pHERrQiwNdyxrXOShQAU/p3BKC80opF5GSr01PoZKFpLlY1C+rifqcGM9emHOPeT7Yw/8bBnNO+tRujc50HJp/DBb3bc14P91VfUEoxukcko3tEcnG/DkSFBWKx+5D28hJ6dwxxWTxZBaVc+8av3DSyK9MHe9bNhJ46q2ku9t3WTNq39mdg57CTx1r7+5CRV3JWDHJvOXickvJKWvv7nEwU+7OLeGHJ7pN39K4iItw5tjt3ju1OdHjQaYnCHcKDfGkd4M1T3+8kp6jMrbFUp5OFprlQVRfU5L7tT+tmiI00BkRTswrdFZpL7D1ayDVv/MpjX28/7XhKViEvLtvDutQcl8azPi2HE2UVLn3Punh5Cf+Y2pfCkgqe/n6Hu8M5jU4WmuZC3l5ePDm1D9MHdz7teKCvNx1C/Ft0y6KwtILb3kvCz9uLe8bHnfbcud0i8PfxYtmOIy6LJ6+4nKtfX8eLy/a47D0d0bN9MDePiuGTpHQ27HNt8qyLThaa5kIBvhauTOxc47hEbGQQqcdaZrJQSnHvJ1vYl32Cl68eeEZZE38fCyO7R7Jsx1GXrWResesoFVbFxF51T211h3vOj6NTaADvrN3n7lBO0slC01ykoKSct9akkV1YWuPzE3u1Z1hsuIujco3XV6Xyw/bDPDj5HM7tVvMq9fHxbTl0vJidhwtcEtPi7YeJDPZjYOdQl7xfQwT6evP+LUP5z3TP2Vlaz4bSNBdZtuMoT3yTTL+okBrXE8w6t6vrg3KRCb3ak1dczs0jY2o9Z1x8WwJ9Lew9Wkh8B3NnhJWUV7JiVxZTB3byuCmqVaoW9uUVl1NWYSUy2PVrUOzpZKFpLlLTLKjqKiqtVFgV/j5nzulvjvKKy2nt701MRBD3XXBOnee2DfZn86MT8fU2v8NjXWo2J8oquaC353VB2SuvtHLJy6s5p30w866vd92cqXQ3lKa5QNUsqAv7dqj1TjYzr5hzHvmBLzfVtqFk81JSXsnVr6/j4a+2OXxNVaIwe9zivB6RLLp7FMM9vNvPx+LFjCGd+TH5iEsH/2uik4XW7GUXllJeaXV3GHWqWoh3Ub/a72TbBvvj5SUtYvqsUoqHvthKcmY+4+PbOXxdTlEZF720ik+T0k2Mzlhf0atja5e0YprqlpGxxLVtxWMLt1Nc5tp1KPY8/yelaXXYmp5Hwj+W8uR3njUnvbrdRwroGFJ3F5TFS4gJDyLNxTOi/rcihbs+/I1fU7Oddkf/3rr9fLHpEH86vwdjz2nr8HVhgT7kFpWxxMS76N8O5HLfp1uazVa2vt5GocH03GJeXu6+ab6mJgsRmSQiu0Rkr4g8UMPzL4jIZttjt4gct3tulojssT1mmRmn1nx1b9sKgLfX7vPo1sX9k85h2V/H1DuYGhsZ5PK1FjMGd2bPkUKmz1vHpP+s4v11+ykqbfxCtQ37cpjzTTLj49ue3NjJUSLC+F7tWLUni5Jyc+6iv92SyddbMmjloaXAazI0NpzLB0Wx92ih2zZJMi1ZiIgFmAtMBnoBM0Wkl/05Sqk/K6UGKKUGAC8DX9iubQM8BgwFhgCPiUjtt2TaWSeroJSCknICfC3Muy4BgDV7j7k5qppV/XLXVIiuupiIIA7knHBp4gsL8uWrO0fw7OX98LYID3+1jWFPLSOroOYpvvUpq7DSNyqE56cPaNRMo/Pj21FSbmVtivP/PZVSLN5+mFHdIzx234ja/HNaH167LsGpxQsbwsyWxRBgr1IqVSlVBiwALq3j/JnAR7a/XwAsUUrlKKVygSXAJBNj1ZqRgpJyZs1fz01vb0ApxXk9Iwn292bhlgx3h1ajOz74jfs/2+LQuePOacufJ/RwWbJYknyEP7y3kZLySq4a3Jlv/ziSz28/l1tGxp6cqvnayhR+3H6YSmvdd7RVSXFE9wi+uP3cRpfaHhbbhiBfC0t3HG3U9XVJzszn0PFiJvZ2fBzFU/h5WxARDmSfcMuNkZmptRNw0O7rdIyWwhlEJBqIAZbXcW2nGq6bDcwG6NLFsyo0auYoq7By+/u/setIAW/OSkRE8PO2MG1gJ8oqPW9vloKScpbtPMq1Q6MdOj+xaxsSu7YxOapTfk3NZvmuowTbyoOLCAnRYSREGw358korH64/wP7sE3QKDeCaYV2Ynti5xnUiT3yTTLvW/tw+pluT7n79vC388fw4OocFNvo1arN4+xG8hAYNunua//v8d3YfKWD5X8cQEui6vS88ZYB7BvCZUqpBnZRKqXlKqUSlVGJkpPvKHGuuYbUq7v9sC6v3HuPpaX0Z0/PUwOkTl/bhqWl93RhdzRyZBVXd4bwSMo4XmxjVKcmZ+ZzTPhhvS80fBT4WL5b95TxevTaB6PBAnv1hF8OfWn5GK+7TpIO8vXYfx084p1LqH87rxkX9nL8fdSs/Cxf36+iWTZac5eGL48k9UcYzi3e69H3NbFkcAuyrpUXZjtVkBnBntWvHVLt2hRNj05qhV1bs5avNGdw7sQdXJnau8Zyj+SW0be3v4shq9+3vmXSoZxZUdRe/vIrx8e14+vJ+JkZmdBttz8jnwr71b/tZtT3oniMFvLduPwOijBIZG/fnsmFfDs8v2c253cK57wLnbQmamVdMVkEp/aKcV45j9uhuTnstd+ndMYQbzo3hrbVpXJkQxcAurhnONbNlsQGIE5EYEfHFSAgLq58kIucAYcAvdocXAxNFJMw2sD3Rdkw7i102KIp7J/bgzrE1z7CZ+9NeRj37EwUl5S6OrGbG/s1ZTO5T+0K8msREuGZGVEZeCXnF5fRqQGmNuHbBzLm0D13CjS6irzYd4unvdxIR5MvLMwfW2kJpjLs+3MRDX2512utlFZRirWfcpbn4y8QetA32429fbqPCReNbpiULpVQFcBfGh/wO4BOl1HYRmSMiU+xOnQEsUHbzwZRSOcDfMRLOBmCO7Zh2FtqekYfVqugUGsBd4+Jq7Q8fGtOG0gorS9280rWKVcE94+O4POGM4bY6xUa0IvWY+Qvz8k6U079zKH06NX4nuDmX9uazPwzn49uGO71rZ3x8O7Ydyiczzzldcre/v5FZb613ymu5Wys/bx6/pDeJXcMod9FYnaljFkqpRUqpHkqpbkqpJ23HHlVKLbQ753Gl1BlrMJRS85VS3W2Pt8yMU/NcG/blcNkra3nJgcVIg7qE0Sk0gIWbPWNWVEiAD3eO7d7gbTljI4M4VlhGXrG5LaReHVvz9Z0jmtSNISIkdm1D5zbOH4weH2+MSS1zwqyorIJSNh7IJTHadZMHzDa5bwfmXNrHoSnZzuApA9yadoY9Rwq45Z0kokIDuH5413rP9/ISLu7XgVV7jpHr5i0pC0rK+fb3jEbtwhYbaSw0NLvsh7sWdzmqe9tWdGkT6JSaSEt3HEEpuKBP850FVZuN+3NdMpVWJwvNIx3OK2HW/PX4WLx456YhtAnydei6S/p3pMKq+H7bYZMjrNvSHUe468NN7Mhs+N4MAzqH8sL0/nQx4W7d3qT/rOLfP+4y9T2aQkQYH9+OX1Kzm7yae/H2w3RpE0jPdsFOis5zdAoNYET3mvcIcSadLDSPo5TiD+9vJK+4nLdvHNygLo7eHVvz36sHcnF/50+7bIjvfj9smwXV8Jk8kcF+XDYwytTpnXknytl1pMBlXRiNNXt0LD/fN7ZJJdsLSspZuzebC3q3c9vqZzO1D3HN7L/mtd5dOyuICA9dGE9pRWWDB19FhIv7dTQpMsfk22ZBXTc8utEb6yRn5JNfUm7aznnJmfkADZoJ5Q7O+CD097Hw+qxEOocF1H+yVivdstA8htWqWJeaDcCQmDaMimvcQkurVfHm6jS+/d09A93LdhyhrNLKhX0b37r594+7eOzr7U6M6nRVyaKhg+/usHrPMe7+aFOjp736WLw4r0fkybEgrXF0stA8xjM/7GTGvHVs3N+0WdJeXsIXv6Xz+qo0J0XWMOvTchvdBVUlNjKItOwi09YFJGfkExns5/atOh1xrLCUhVsy2JJ+vP6TqymtqOTfP+5in4vLvrdEOlloHmH+6jRe+zmV64dHM8gJK1Iv6d+RLQePsz/b9R8S/7ysD1/fNaJJezvHRrairMLKIZPKfiREh3HdMMfqVbnbmJ6RWLykUVNo1+7N5uXle0lzw/+DlkYnC83tvvs9k79/l8wFvdvx2CW9nTIIeUl/Y9zi298zm/xaDSUitA1uWl97bEQQAKkm3RFfPbQLd58fZ8prO1tooC8J0WGNWmz5Y/JhWvl5c243z94+tTnQyUJzq+Mnyrj30y0kdAnjxRkDsTThbtxep9AAEqPDXL5A7y+fbOaZH5pe4M3MtRaFpRXknfCMkiiOmhDfjp2HC0jPPeHwNZVWxZLkI4zpGYmft2fP+moOdLLQ3CrY34cXpg/gmSv6NWl6ZE0uHdCRsCAfCpuw61tD5JeU8+2WTMoqml6rJ6KVLwtmD+OygQ0rFeKIRb9n0n/OjxzIdvyD193Oj29L/6gQsgsdX2z524FcjhWWcUFvxyv+arXTU2c1t7J4CZP6mPPLfO2waK5zYOW3szhjFlQVETF12myQr4WoZjSVNDayFV/fNbJB1+zPPkFooA9jeurtC5xBtyw0t1m24wjP/7jLtL2Wq8Y+corKXFLa4rvfM+nYxFlQ9jYfPM781c6f0ZWckU98h9ZNGoB3lxNlFZRWOPb/5YqEKJL+Np7gRu7Yp51OJwvNbV5ZkcLXWzLwdWJZ6+pW7DrK4CeXsiU9z7T3gKqFeMeY3Ldh5cjrsmp3FnO+TW5UfanaWK2K5Mx8enX07MV4Ndl1uIABc5aw3IFZUVVbwDqzZPrZTv8kNbfYcvA4G/fnMmt4V1PvcAd2CcMiwjcm789dWm7lmmFdmDrAeWMMpwa5nTcj6mDuCQpLKzx+5XZNukUGEeBjYYkDs6JeXr6HC19c5XArRKufThaaW7y1Jo1Wft5cmRhl6vuEBPhwXs9Ivv094+Tdphkig/147JLe9I1y3oro2EjnT59t7e/DPy/r65LCc87mbfFibM9IVuzKqvffcvH2IwT5WfQsKCfSyUJzuaP5JXy3NZMrE6Nc0p98Sf+OHMkvZcM+c/bPKigp59fUbKcno5iIIEQgzYkti7AgX64e2sWU/SdcYXyvduQUlbHpQG6t5xzMOcGOzHwm9tKzoJzJ1GQhIpNEZJeI7BWRMzY4sp1zlYgki8h2EfnQ7niliGy2Pc7YjlVrvorLKzn/nHbMctFMpfHxbQnwsbDQpK6oJclHmD5vXaPKUdTF38dCx5AA0py4a9661Gy3rGp3ltE9IvH2kjq7ohZvN8rTT+zd8vaucCfTps6KiAWYC0wA0oENIrJQKZVsd04c8CAwQimVKyJt7V6iWCk1wKz4NPeJDg/i1esSXPZ+gb7evDhjAPEm9dM7exaUvS/vONfhvTwccc+CTYzoFsHz05vnr1Zrfx+eubxfnd19P24/wjntg4kOD3JhZC2fmesshgB7lVKpACKyALgUSLY751ZgrlIqF0Ap1fT9EzWPtnF/LmGBPi6vADrRpIVZX25KZ9nOo9w5tpspeyW0be28vQqOFZZyJL+0Wc6Esnd5Qt3jXDOGdNazoExg5k+0E3DQ7ut02zF7PYAeIrJGRNaJyCS75/xFJMl2fKqJcWouopTi4a+2cccHv7llS88lyUd4a43z1i2sS83m/s9+Z3hsOPec38Npr2tvR2Y+D3+1lWOFpU55LaDZJwuljDIeq/fUvJXotEFRTOnv3j1NWiJ3p19vIA4YA8wEXheRqrZ8tFIqEbga+I+IdKt+sYjMtiWUpKysLFfFrDXSr2k57MjMZ9a5Xd2yY9nS5CM8t9h5iwADfCwkRrfh1WsT8PU251cpu7CM99cdYPeRhm/PWt32jOax4VF9RIR/Ld7JKyv2nvHckuQjZJhUqfdsZ2ayOAR0tvs6ynbMXjqwUClVrpRKA3ZjJA+UUodsf6YCK4CB1d9AKTVPKZWolEqMjNRL+j3dW2vSCAv0MaXekSMu6d+RorJKftrZtN7OqmTTv3MoH80eRkigeTO6Tk6fdcKMqOSMfDqFBhAa6LwxEHc5P74d69NyyCs+VRCxsLSCOz/4jTdNWPWumZssNgBxIhIjIr7ADKD6rKavMFoViEgERrdUqoiEiYif3fERnD7WoTUzB3NOsCT5CDOHdHF6wUBHDe8WTkQrvybNiiopr2TGvHU8v2S3EyOrXfvW/vj7eDklWTx0YTxzrxnkhKjcb3x8WyqsipW7T/UorNyVRVmllYm99CwoM5iWLJRSFcBdwGJgB/CJUmq7iMwRkSm20xYD2SKSDPwE3KeUygbigSQR2WI7/rT9LCqt+dmSfpwAHwvXDXffhjsWL+Givu1ZtvMoBSUNL9FttSr+tGAzW9KPu6wrx8tLiIloRaoTps+2D/FngAkzttxhQOcw2gT5ssxuCu3i7YdpE+RLYtc2boys5TK16qxSahGwqNqxR+3+roC/2B7256wF+poZm+ZaF/fryLhz2hLo695Cx1MGdGTV3mOk5xYT36Fh3UdPfb+DH7Yf5pGLe5lWKbcm3du24kh+SZNeI+1YEUuTjzBtUCfCW3n+Vqr1sXgJ485py7ZDeSilKK9U/LTzKJP7tnfanija6XSJcs10x0+UERro6/ZEATCoSxjL/nJegwfY3/tlH6+vSmPW8GhuGtHVlNhq8+L0AU2un7Vm7zGeXLSDyX1bzqrmx6f0JsjXgoiw9dBxCssq9N4VJnL3bCithbNaFdNeWcuDX2x1dyiAMZNGRCgpr2zQrKjWAT5c2Lc9jzpp29eGcEahxeTMfEICfOgU2nz2sKhPKz/vk/8WCdFh/PrQ+YyMa341r5oLnSw0U63ck0XqsSKGxnhOP/LhvBIG/2Mpn/+WXu+5VVVLLx3QiblXD3JLF8fhvBJufntDresKHLE9I59eHVq7Zcqymd5ak8ZVr/0CQNtgf1040EQ6WWimmr86jbbBfk7ZPc5Z2rX2o21rv3r35844Xsz5/17JD9syAdz2QRvkZ2HZzqP8fqhxtacqKq3sbKZ7WNTH4iWsT8th5DPL2efE6rzamXSy0Eyz92gBq/Yc47ph0aYtWmsMEeGS/h1Zvy+Hw3k1DxwXlJRz09sbyDtRTkyEa0uTVBfs70NksF+jp88eOl5MhVXRuwUmi/PjjWmy6bnFhDmxhpZ2Js/5DdYazR2lMxzx/roD+Hp7cfXQLu4O5QxT+ndEKfj29zNbF+WVVu744Df2Hi3kf9cm0LN9sBsiPF1sRBCpWY2bPhsdHsT2Jy7wqNads3QKDSAkwIce7VoREqC3TzWTThbNnNWquH7+el7/OdXdoZzh/kk9efvGwR45VTM2shV9OrU+Ywc9pRSPfLWNVXuO8c9pfT1mwDQ2slWTNkHy97G4bTGk2dY8MI6v7xzp7jBaPPfPZdSaxFi0FcSTi3YwonuER/VLB/p6c243z/iwrclDk+Px9z39A1QpCA305Y/junNVYudarnS9flEh7DlSQHFZJQG+DfvQf/r7nbRv7ccNI2JMis69WvnpjzFXEE/twmioxMRElZSU5O4wXKqswopVKYrLKpnwws+0Dfbj67tG4OPm8swVlVZufHsD1w/vyoRmVHqhrMJ6cmxFKdUiZg4ppUj4x1ImxLfjmSv6uTsczQOJyEZb0dY66W6oZuyzjemMfGY5xeWV/POyPiRn5jP3pzMrcbraj8lHWLXnGNZmcCOy7VAeLyzZzfq0HMY+t4Kdh43KrC0hUQAcyS8lp6jMo1qcWvOkk0UzVWlVzPs5hY6hAXQI8Wdi7/ZMHdCRV1akcLSgaaUhmuqtNWl0bhPA+HjPb1X8diCXF5ft4ca31uPn40V7J2425ExKKabOXcN/ljasgOH2jDyAFjkTSnMtnSyaqR+3H2Zf9gluG31qh7bHp/Tmw1uG0jbYfR94W9Pz2LAvl1nDuzaLGj0X9u2AxUvw97Hw9g1DPLZ8t4hQWFpBsm1PCkclZ+QjAuc08z0sNPfTI0PNkFKKV1emEB0eeFpBu9DAUxU3M/OK6RDi+tIOb61JI8jXwlWDPWdwuC4RrfyYd10C0eGBdAkPdHc4dYqNCGrwjCirgoGdQ/UgsNZkumXRDG0+eJwt6XncOiq2xrv3rzcf4rxnV7DtUJ7LY5vYux33XdCT1v7NZ877+fHt6N7W/Wsp6hMb2Yr92UVUVFodvuae8XF8cccIE6PSzhY6WTRDAzqHsmD2MK6oZeP6MT3aEhrow72fbqGswvEPFmeY1KdDi52i6W6xkUGUVyrSc/W2oZrr6WTRDIkIw2LDa11kFRLow1PT+rLzcAEvL9/jkphKKyp5dWUKOUVlLnm/s1GfjiFc0r8jjs4x27AvhwnPr2zwOIem1cTUZCEik0Rkl4jsFZEHajnnKhFJFpHtIvKh3fFZIrLH9phlZpzNyYNfbOWZH3bWe9758e24fFAUr6xIYWu6+d1R327J5Onvd7ql6+ts0atja16eOZCYiCCHzt92KI89RwuJaOWZg/Za82JashARCzAXmAz0AmaKSK9q58QBDwIjlFK9gT/ZjrcBHgOGAkOAx0QkzKxYm4uDOSf4JOmgw11Lj17Si85hAaQ0sqaQo5RSzF+TRve2rRjlIeUxWjJH9+FIzsgnopUvkcGeV25Fa37MbFkMAfYqpVKVUmXAAuDSaufcCsxVSuUCKKWO2o5fACxRSuXYnlsCTDIx1mbhzdVpCHDzSMfGBEICfFjyl/OYOrCTqXFt2JfL9ox8bhzRtcUsZvNUs+avZ9b89Q6duz0jn14dQ/S/ieYUZiaLTsBBu6/Tbcfs9QB6iMgaEVknIpMacO1ZJbeojI83HGTKgI50bMBuZz4WL5RSLNySwe/pjdsPoT7zfk4lJMCHaQNrHnDXnCcy2M+h6bNlFVb2HC2gl15foTmJuwe4vYE4YAwwE3hdREIdvVhEZotIkogkZWVlmRSiZ3j3l/0Ul1dy2+huDb62uLySf363g3s/3XJy5zdnau3vzXXDohtc4E5ruNjIILIKSikoKa/zvKLSCqb078TwbuEuikxr6cxMFocA+5VZUbZj9tKBhUqpcqVUGrAbI3k4ci1KqXlKqUSlVGJkZKRTg/c0F/ZtzyMX92rU3gqBvt48dXlfdh8p5MWlTZ8dZbUq3vtlHzsyjVk2c6b24U/j45r8ulr9Ym0bMaXV07oIC/Ll31f157weLfv3QnMdM5PFBiBORGJExBeYASysds5XGK0KRCQCo1sqFVialVZBAAAgAElEQVQMTBSRMNvA9kTbsbNWXLtgh8cqajK2Z1uuTIji1ZUpbDnY+O6ovUcLuOq1X3jk6+18mmTsYd3KzxtvN1e6PVt0izRmQtW3a15BSbnHboqlNU+m/YYrpSqAuzA+5HcAnyiltovIHBGZYjttMZAtIsnAT8B9SqlspVQO8HeMhLMBmGM7dtapqLTyyFfb2HW4oMmv9fDFvWgb7M+9n26hvAGrgMHoA39x6R4ufHE1e44W8tyV/Xnk4vgmx6Q1TJfwQG4dFUNsZN3TZ29+J4nrHRwI1zRHOFQwRkS+AN4EvldKOfwpo5RaBCyqduxRu78r4C+2R/Vr5wPzHX2vlmrRtsO8t24/o+Iimry9Z0iAD89d2Z+84vIG73nx3rr9vLB0N5f078ijF/fS0zHdxM/bwt8u6lXnOUopdmTkc+nAji6KSjsbOFpd7BXgRuAlEfkUeEsptcu8sDSwFQxckUK3yCCnlfu23ya00qrqrAxbWFpBeu4JzmnfmmuGdiGubStG6z5wtysuqyQjr5huka1qfP5gTjEFpRX06hDi4si0lsyh20ul1FKl1DXAIGAfsFRE1orIjSLSfCrGNTOr9x4jOTOf20Z3w8vJ5b4/3nCAKf9dXesCr592HuWCF37mlneSKK+04u9j0YnCQzz34y4uemkVVmvNYxLJmXoPC835HO6LEJFw4AbgFmAT8CJG8lhiSmQar65MoV1rP1O6EzqEBLA9I58Xqm2mc6ywlLs/2sSNb28gwNfCizMGuH2bVu10sZFBlJRbycyveZOr5Ix8vIQmd1tqmj1Hxyy+BHoC7wGXKKUybU99LCJn18bXLlJpVfSLCmVS7/b4eTt//cLoHpHMHNKZ139O5YLe7RnUJYx9x4qY+soaikor+NP4OG4f082U99aapqo2VGpWIZ1qWKA5Mi6SVv7etRaa1LTGcHTM4iWl1E81PeHIRt9aw1m8hP+bdI6p7/HQhfH8vPsYd3+0iaV/OY/o8ECmDYxixpDO9Gin70o9VdVYRWpWEaPizuwaHBLThiExbVwdltbCOdq/0Mt+ZbVt/cMdJsV01juYc4LlO4+YPk8+2N+Hpy/vS3puMem5xYgIj17SSycKD9c22I8gXwupNRSILCqtYNOBXIeLDWqaoxxNFrcqpU6u5LIV97vVnJC0135O4Q/v/UZWYanp7zUqLpIPbx1Ka3+97WZzISL8c1pfpg06sxbXpgPHueyVtSTty3VDZFpL5ugnhEVExLYuoqr8uC6Sb4JjhaV8mpTOtEGdaBvs75L3PLebLive3Fw6oOa6mlUzoXrpmVCakznasvgBYzD7fBE5H/jIdkxzsnfW7qOs0sqto2PdHYrmwbILS1m248gZ3U3bM/LpEOJPmyB9L6c5l6PJ4v8wynHcbnssA+43K6izVVFpBe/+sp+JvdrVuuBK0wDWpGRz8ztJZxQUTM7I12XJNVM41A1lK/HxP9tDM0nasSKCfC384byGlyHXzi6xEacKCsbbkkNJeSUpWYVM7tPenaFpLZSj6yzigKcwtkc92ZGulNJ9JU7Up1MIP98/Vldw1epVVUgw7dipGVEWL+GjW4fRrrVrxrq0s4ujn0pvYbQqKoCxwLvA+2YFdTban11EaUWlThSaQwJ9vekQ4n9aqXIfixdDY8PpGlF3RVpNawxHP5kClFLLAFFK7VdKPQ5cZF5YZxelFLPf3cgt7+jF8JrjYiODSLEbs1i24wg/7TpaxxWa1niOTp0tFREvYI+I3IWxa50egXWSFbuy2HWkgNvO0716muMeujAeP+9T93v//WkvPhYvxvZs68aotJbK0ZbFPUAgcDeQAFwLzDIrqLPNqytT6BjizyX99f4DmuN6dwyhe1tjtX2lVbEzs0DPhNJMU2/LwrYAb7pS6l6gEGNfC81Jthw8zq9pOTx8Ubyu7qo1yPETZXz7eyYjukdgVYri8kpdllwzTb2fTkqpSmBkY15cRCaJyC4R2SsiD9Tw/A0ikiUim22PW+yeq7Q7Xn3v7hZj0bZMWvl5M31wZ3eHojUzBSUVPPzVNtalZpOckQ/olduaeRwds9hk+8D+FDg5oqaU+qK2C2wtkrnABCAd2CAiC5VSydVO/VgpdVcNL1GslBrgYHzN1gOTzmHm4C4E++s9pLSG6RQagK+3F6lZhfh6e+FjEeLa6iKQmjkcTRb+QDYwzu6YAmpNFsAQYK9SKhVARBYAlwLVk8VZq2pbUz3VUWsMLy8hJjyI1KwiXr8+kZlDuuDrrbsyNXM4uoK7MeMUnYCDdl+nA0NrOO9yERkN7Ab+rJSqusbftrFSBfC0Uuqr6heKyGxgNkCXLl0aEaL7FJdVMv75lfx5Qg+uSDizeqimOSI2Moidhwvw8hKiwgLdHY7Wgjm6gvstjJbEaZRSNzXx/b8BPlJKlYrIbcA7nGq9RCulDolILLBcRLYqpVKqvf88YB5AYmKiuZs/ONnnv6Vz6Hgx0eH6F1xrvNjIIL7fdpi/frKFW0bFnCz9oWnO5mib9VvgO9tjGdAaY2ZUXQ4B9qO2UbZjJymlspVSVZs2vIExLbfquUO2P1OBFcBAB2P1eFarYv6aNPpFhZAYHebucLRm7JaRsbw4YwCf/5ZOfnG5u8PRWjBHu6E+t/9aRD4CVtdz2QYgTkRiMJLEDODqaq/TwW4/7ynADtvxMOCErcURAYwAnnUk1uZg5e4sUrOKeHHGAETE3eFozVhYkC/pucUAxOuZUJqJGrs9WhxQ5zJRpVSFbbX3YsACzFdKbReROUCSUmohcLeITMEYl8gBbrBdHg+8JiJWjNbP0zXMomq25q9Jo31rfy7s28HdoWjNnNWq+NfiXQC01jPqNBM5OmZRwOljFocx9riok1JqEbCo2rFH7f7+IPBgDdetBfo6Eltz9MDkcziSX6IX4WlN5uVltEy9dANVM5mj3VB68rYT9e4YQu+OIe4OQ2shvr5zBOGt9M54mrkcurUVkctEJMTu61ARmWpeWC3TscJS/vrJFvZnF9V/sqY5qH/nUD1tVjOdo/0gjyml8qq+UEodBx4zJ6SW64N1B/j8t3TKK5vVLF9N0zSHk0VN5zV2cPysVFpRyXvr9jOmZyTd2+rq7pqmNS+OJoskEXleRLrZHs8DG80MrKX5ZksmxwpLuXlkjLtD0TRNazBHk8UfgTLgY2ABUALcaVZQLY1SijdXp9GjXStGdo9wdziapmkN5uhsqCLgjBLjmmNKK6yM6BZOn04hehGepmnNkqOzoZaISKjd12Eisti8sFoWfx8LD1/ci6kDO7k7FE3TtEZxtBsqwjYDCgClVC71rODWDAdzTrBqTxZK6RlQmqY1X44mC6uInKwBLiJdqaEKrXamN1alcvPbSRwrLHN3KJqmaY3m6PTXvwGrRWQlIMAobPtIaLXLKy7n043pXNK/I5HBfu4OR9M0rdEcHeD+QUQSMRLEJuAroNjMwFqCBesPcKKskptGdnV3KJqmaU3iaCHBW4B7MPak2AwMA37h9G1WNTsVlVbeWbuPYbFtdB0oTdOaPUfHLO4BBgP7lVJjMTYiOl73JWe3fdknqLAqbh4Z6+5QNE3TmszRMYsSpVSJiCAifkqpnSLS09TImrnubVux+v/G4a1rR2ua1gI4mizSbessvgKWiEgusN+8sJq3nKIygv298fXW+1VomtYyOPRpppS6TCl1XCn1OPAI8CZQb4lyEZkkIrtEZK+InLECXERuEJEsEdlse9xi99wsEdlje8xy/Ftyv0e/3sbFL63GatWzizVNaxkaXDlWKbXSkfNExALMBSYA6cAGEVlYw/aoHyul7qp2bRuMEuiJGOs5NtquzW1ovK526Hgx3287zM0jY07uYqZpmtbcmdlPMgTYq5RKVUqVYRQgvNTBay8AliilcmwJYgkwyaQ4nerdtftQSnH98Gh3h6JpmuY0ZiaLTsBBu6/Tbcequ1xEfheRz0SkcwOv9ShFpRV8tP4Ak/t00DuXaZrWorh7BPYboKtSqh9G6+GdhlwsIrNFJElEkrKyskwJsCG+33aY/JIKbtJ7Vmia1sKYmSwOAZ3tvo6yHTtJKZWtlCq1ffkGkODotbbr5ymlEpVSiZGRkU4LvLGmDezEJ7cNJyE6zN2haJqmOZWZyWIDECciMSLiC8wAFtqfICId7L6cAuyw/X0xMNFWCj0MmGg75tG8vIQhMW3cHYamaZrTmbaPtlKqQkTuwviQtwDzlVLbRWQOkKSUWgjcLSJTgAogB7jBdm2OiPwdI+EAzFFK5ZgVqzP84b2NDIoOZfbobu4ORdM0zelMSxYASqlFwKJqxx61+/uDwIO1XDsfmG9mfM6SnJHPD9sPM6BLaP0na5qmNUPuHuBuEeavSSPAx8LMwV3qP1nTNK0ZMrVl0dIdKyzl+SW7+XLTIa4e0oWQQB93h6RpmmYK3bJohNKKSgB8vLz4YdthZgzuzL0X6LqKmqa1XLpl0QDbM/J45acUDuScYOFdIwgJ9GHtA+Pw97G4OzRN0zRT6WThgN8O5DJ3+V6W7TxKsJ83158bTVmlFT9vi04UmqadFXSyqMeyHUe4+Z0kwgJ9+OuEHlx/bldCAvTYhKZpZxedLKpRSrFidxYlZZVM7tuBUXGRPDGlN1ckRBHkp39cmqadnfSnn43Vqvgx+TD//Wkv2w7lkxAdxuS+HfD19mLWuV3dHZ6maZpb6WQBrNqTxZxvktlztJCYiCCevaIflw30+CK3mqZpLqOTBVBhVXiJ8NLMgVzUtwMWvWmRpmnaaXSyAMb0iOS8uEi9s52maVotdLIARATReULTNK1WegW3pmmaVi+dLDRN07R66WShaZqm1UsnC03TNK1eOllomqY1xeGtkLHZ3VGYztRkISKTRGSXiOwVkQfqOO9yEVEikmj7uquIFIvIZtvjVTPj1DRNa5TNH8K8sTBvDHz3VyjJc3dEpjFt6qyIWIC5wAQgHdggIguVUsnVzgsG7gF+rfYSKUqpAWbFp2keo/g4+AaB5SwtUFmSB97+4O3n7kgcZ7XC8jmw+gWIOQ/a9oL1r8HO7+DCf0H8Je6O0OnMXGcxBNirlEoFEJEFwKVAcrXz/g48A9xnYiya5jkqK+BQEuxdajwyNsOAq2HqK+6OzDWslcb3XPX9H0qCjgNh1rfgG+ju6OpXVgRf3gY7voGEG43kYPGBflfCwnvg42vhnIth8rMQ0nLKBpmZLDoBB+2+TgeG2p8gIoOAzkqp70SkerKIEZFNQD7wsFJqVfU3EJHZwGyALl30/teaB8s7BCnLjA/HlBVQmgfiBVFDIGYUbPkIRv0Vwru5O1JzFB6FvVXf/3IozgHESBIJN0DSW/D1HXDFW3j0Ctn8DPhoBmT+Dhc8BcNuPxVvpwSY/ROsewV+egrmDoXxj0HiTeDV/Pe9cdsKbhHxAp4Hbqjh6Uygi1IqW0QSgK9EpLdSKt/+JKXUPGAeQGJiojI5ZE1zXEUpHPjFdve8DI7aGtTBHaHXFOg+HmLPg4Aw44P0P/1g1b9d37pQCj6/BbJ2QpsYaBN7+iO4I3g1YmizshzSNxjf/54lcPh343hQJMRNNL7/bmMhKMI4HhoNSx+DyHNgTK3Dm+6VsdlIFKUFMHMB9Jx05jkWHxhxD8RPgW//DIvuhd8/hktehHa9XR+zE5mZLA4Bne2+jrIdqxIM9AFWiJGZ2wMLRWSKUioJKAVQSm0UkRSgB5BkYrya1jQ5qafuntN+hvITYPGFLsNhwt+ND8i28WfeObdqC4k3wq+vwXn3Q1hX18W8Zwls+8y4K87aBbsXQ2XZqectftWSiN3fQzqffsd8/OCp1lPqSijNB7FA56Ew7hHj+2/fr+bkM+Ie4/1XPAURcdDncvO/94bY8S18cSsEhsNNi6F9n7rPbxMD130JWz+FHx6A10Yb3+Po+8AnwDUxO5koZc4NuYh4A7uB8zGSxAbgaqXU9lrOXwHcq5RKEpFIIEcpVSkiscAqoK9SKqe290tMTFRJSTqXaC52IgfWvAg7FhrJAiAsxvhg7D4euo4Ev1b1v05+JrzYH/rPgCkvmRtzFavV+BArK4A7N4C3rzGekH/I+F5OPtJO/b2i5NT1Xj4QFm18v3kHjdYJQOso6H7+qdaTf4hj8VSUwjtTIHMz3LjISGDuphSs+Q8sfcKIZ8aHENyuYa9RlA0/PgxbPjSS7MX/MX4uHkJENiqlEus7z7SWhVKqQkTuAhYDFmC+Umq7iMwBkpRSC+u4fDQwR0TKASvwh7oShaa5XEUZJL0JK5427qC7j4ehtxsfko0Zd2jdAQZdDxvfNu4+QzvXe0mTbf8CjmyFaa8biQKMlkJoF+MRO+b0861WKDxcLZHYHsEdYOB1xs8hsmfjxh28/WDGB/D6WPjoarh1uXsHiCvKjK6kze8bLZ1L5zauVRAUDpf9D/pPh2/+BO9OgQHXwMR/QGAb58dtEtNaFq6mWxaaSygFuxbBj49ATgrEjoULnnROf3ReOrw4ABJmwUX/bvrr1aWyHOYOAe8A+MPqxo1LmOVIMrw50ejKuekHY1qxqxVlwyfXwf41cN4DxjiKMwbey4th5bOw9iWjxTXpaeh7pVsH9R1tWXjQ/xBN83AZm+GdS2DB1eDlDdd8ZvRLO2vgMiQKBl4Dv71rzLox06b3jRbB+Y94VqIAaNcLrpgPR7YZU1StVte+f9ZueON8SE+CaW/A2Aed92HuE2DMkJq90ui+++JWeH8aHNxgrLfxYLploblfUTb4+LvnDtIR+Rmw/B/Gat3ANjD2IRh0A1hM6MXN3Q8vD4LBt8Lkp53/+mDc3b40yEhON//ouVNVf5kLix+CUfcaSc0VUlfAJ9cbExNmfAidh5j3XtZK2PAmLJtjjBuBMYBefUZa1SMgzJR/K7ePWWhajUoLjDv0Qxttj98gP914rlX7mmfdtIlxfJDUmcqKYO3LxgC2tQJG3G2shTAzlrBo6DcDNr4FI//c8MFUR2x4AwoyYNo8z00UAMPuMAbNVz1njIP0u8rc90uaD9/da7zXzAXGv4WZvCwwdDb0ngoHfz19DGj/Wvj9E8DuZt4/pPZEEhRp+r+lbllo5qkshyPbTyWFQxttM2Zs/+fCuhozTDoONKZr2s+8Kcg8/bUCI2r5RYlx/iCh1Qq/LzDu+AoyofdlMP5x101pzU6B/ybC8DuNQVBnKsk3Zl11HGB0oXm6ijJ47zJjzcYN35pzp2+tNGYrrXsFuk8wusD8Wzv/fRqqvASO7695QsHxg6AqT53bKcGYENAIumWhuZZSxn/iqqRwaKOxEKtqqmVguPEfuvdUW4IYZMwSqU1ZEeTuO/OXZP8aY5HTaXdcoXXccUU07I4rbRX8+DfI3GLEeeU70GVo/dc5U3g3Y9Bzw5sw4k+nFq45wy//NVZPn/+o817TTN6+MP09eH2cMVZ063Jjppaz7FtjdDEeWGvMZpv4D3O6FxvDx99o5UT2PPO5ynI4fuDUzZUL6mrploXWNKUFxvTCvUuhONc45h1g3Ll2SoBOg4w/Q6Od10yufseVnQK5tl+a4wdA2Q2I+gbXvDK5TSwEtz8VU3YKLHkUdn5rLDYb/zj0nua+wd+s3cZspZF/NgZEnaHomNGq6H4+XPWuc17TVbJ2wRvjjURx02LH1q7URilj0eTKZ2H/aghqayTPQdc5L95mRLcsNPOV5MMHVxizRvrPhKhEIzG07WXu3Vldd1wVZcYCsaokUpVQDm81EoG1wu51Ao0ZKcHtjQ8Pbz/jQ2PYHe5fZRvZw+j+Wj8Pzv2jc7raVv3bWFU+9uGmv5arRfaEK9+CD640ZhBNf7/h9ZaUMupSrXwWDq4z1oZMesaYquzuf+9mQCcLrXFK8uC9acZq2yvfgl6Xujsig7ev0Y0T3g3iJpz+XGXFqURiPz5yfD8MvNaY5dSqrXvirsno+4yFc+v+B+P+1rTXOn7QGNgecLWRiJqj7uOND/fv74NlT8CEOY5dp5RR1mTlM0aF29ad4MLnjEWEPv7mxtyC6GShNVxxrpEoDm81+vTjL3Z3RI6xeNu6pGIwqtB4uHa9jIJ0v75mDHYHhDb+tVY+Y/x5nocW6XPUkFuNSRJrXoSInsa6lNooBbu+N773zM0Q0sUotTHg6ua1d4aH8LDVOGchpeDwNqPLpLLc3dHU70QOvHupsWBq+nvNJ1E0V6PvM8qZr5/X+Nc4tgc2fwCJN7umjIiZRGDyM8aGQ9/cA/t/OfMcqxWSF8Jro2DBTCg5DlP+C3f/ZhRs1ImiUXTLwl2O7jAqUm79zOgGAaNCZ2jnmgdjQ6Pd32Quyob3LjUGX6d/AD0mujees0GHftDzQmOB2rDbwS+44a+x/B/GpINRf3V+fO5g8YGr3jEGvD++xpghFdbVmAKb/DX8/C+jJHybbjD1VWNmmafMcGrG9E/QlY4fgG2fGwniyDYjOcSOMe4evbyNWkNV/em/f2rcUZ4kxorbmmb2hHU1f/Vz0TGjImj2Xpj5odF/rLnG6PuM4nrrX4dRf2nYtRmbIfkrGH0/tIo0Jz53CAiDmR/DG+PgwxlG+e/VL8CxXRDRwyjT0Wdai9h0yFPoqbNmK8qG5C+NBHHA1mSOGmLc7fSeWvuAqlLG2EBNC3JyUuFE9unnx0+BCU8YycPZCo8aiSI3zVjZ2m2s899Dq9v7V0DGb/CnrQ27MXhvmnHdPVvcswrebKkrjO9RVRqz8EbfZ0y20EnCYXrqrDuVFhqVSbd+akzVs1YYO4CNe8Qoddwmpv7XEDGmSwa2MaakVld8/NTagozNxkyX3T/A0NuMWjpNGQy1V3DYKJ6Xlw5Xf+JRdfjPKufdD29OMEpSnPtHx67Zt9rYjGjC31tmogCIHUP5zM9ILw2kxNv2Pe7a7daQPJW/vz9RUVH4+Pg06nrdsnCWijLjF3Prp7BzEVQUG5vA9L3CaEW0621u7Zb8TFuxuw+MJvqYB43BPEvj/mMYr5lhJIr8TLjmU+g6wnnxag337qVG+e4//V7/ugCljDLfeQfh7k0teh1BWloawcHBhIeHI55c68qNlFJkZ2dTUFBATMzpN6u6RLmrWCvhh4fguThjf96Un4ypeTf+YHQZTHjC2ILR7P/ErTvA1Llw20ojMX1/H7wyHHb9YHxwNFTeIXj7IqNlcd0XOlF4gtH3Q9FR2PhO/efuXgzp640WSQtOFAAlJSU6UdRDRAgPD6ekpKT+k2tharIQkUkisktE9opIrRO8ReRyEVEikmh37EHbdbtE5AIz42ySdf+DdXONgeqrP4F7d8PFz0P0cPeUiujQH2Z9AzM+AhR8NN24Iz281fHXOH4Q3r7QGNS+7kvoMsy0cLUG6DoCokca23yW1/FLb7XC8r8b41cDz44SFjpR1K+pPyPTPs1ExALMBSYDvYCZItKrhvOCgXuAX+2O9QJmAL2BScArttfzLDlpRtdPj0lw5dvQ44Kmdfs4iwiccyHcsQ4mP2sU9Ht1FHx9l9FSqEvufiNRnMiF674yt56/1nDn3W9Uwt38fu3nbPvcmG039m+e8f+xhTt+/DivvPJKg6+78MILOX687g2PHn30UZYuXdrY0JzKzFvfIcBepVSqUqoMWADUVBPi78AzgP2t0qXAAqVUqVIqDdhrez3PoRR8c7cx5fWi5z1zXwCLjzHgffcmYwXwlgXGpjcr/wVlJ848PyfN6HoqyYdZX0NUgutj1uoWMxo6D4VVLxjjZNVVlsNPT0K7vkYhRM10tSWLioqKGs4+ZdGiRYSG1j0RZc6cOYwf7xnT1M1MFp2Ag3Zfp9uOnSQig4DOSqnvGnqt22163yg+N+EJ924q74iAMGOf6Dt/he7j4Kd/GPslbPn41JaV2SlGoigrhFkLjT0mNM8jYrQu8tNhy0dnPv/bu8YsOU/cLrWFeuCBB0hJSWHAgAEMHjyYUaNGMWXKFHr1MjpSpk6dSkJCAr1792bevFMr8bt27cqxY8fYt28f8fHx3HrrrfTu3ZuJEydSXFwMwA033MBnn3128vzHHnuMQYMG0bdvX3bu3AlAVlYWEyZMoHfv3txyyy1ER0dz7Ngxp3+fbps6KyJewPPADU14jdnAbIAuXZxY474+BYdh8d8gegQk3Oi6922q8G5Gtc59a4ztKr+cDb/+D4bfZWz+UllmjHe07+vuSLW6dDvf2A9k1b+NyRRVXU1lJ4yKqp2HQdzZu7p++mtnlgC5uF8HrhveleKySm54a/0Zz1+REMWViZ3JKSrj9vc3nvbcx7cNr/P9nn76abZt28bmzZtZsWIFF110Edu2bTs562j+/Pm0adOG4uJiBg8ezOWXX054+Ol7uezZs4ePPvqI119/nauuuorPP/+ca6+99oz3ioiI4LfffuOVV17hueee44033uCJJ55g3LhxPPjgg/zwww+8+eab9f6MGsPMW49DgH0hmijbsSrBQB9ghYjsA4YBC22D3PVdC4BSap5SKlEplRgZ6cLVqYvuNTb1ueSl5nn31nUE3PoTXPYaFByBz282ui9mfasTRXMgAuf9n1EmZuunp46vnweFh439LzyxW/QsMWTIkNOmp7700kv079+fYcOGcfDgQfbs2XPGNTExMQwYMACAhIQE9u3bV+NrT5s27YxzVq9ezYwZMwCYNGkSYWFhTvxuTjGzZbEBiBORGIwP+hnA1VVPKqXygJNbgInICuBepVSSiBQDH4rI80BHIA4483bAHZK/hh3fGJvjRHR3dzSN5+UF/WcYK7+3fAgxY5r393O26XEBtO8HPz8H/aYb3YerXzDKsESf6+7o3KqulkCAr6XO59sE+dbbkqhPUNCpFfYrVqxg6dKl/PLLLwQGBjJmzJgap6/6+Z0qbmixWE52Q9V2nsViqXdMxNlMuy1WSlUAdwGLgR3AJ0qp7SIyR0Sm1HPtduATIBn4AbhTKfsNZ92kOBcW3Wf8kt9trUwAAAy2SURBVA53cBWtp/MNhMG36ETR3FSNXeSkwLYvYO3LRnXV5rJdagsSHBxMQUFBjc/l5eURFhZGYGAgO3fuZN26dU5//xEjRvDJJ58A8OOPP5Kbm+v09wCTxyyUUouARdWO1fi/WSk1ptrXTwJPmhZcY/z4sLH24JpPdRVLzf16XmTUQ1rxT6M7sfdlxjobzaXCw8MZMWIEffr0ISAggHbt2p18btKkSbz66qvEx8fTs2dPhg1z/pqlxx57jJkzZ/Lee+8xfPhw2rdvT3BwI6oT10OX+3BUyk/w3lTbnsiPm/c+mtYQ276Az240Khjfuf6sbCHu2LGD+Ph4d4fhNqWlpVgsFry9vfnll1+4/fbb2bx5c43n1vSz0oUEnamsyNhoJby7MbCoaZ6i16XQKdFYPHkWJgoNDhw4wFVXXYXVasXX15fXX3/dlPfRycIRy580Zp7csKjF19nRmhkvC9y6zN1RaG4UFxfHpk2bTH+fZjjv08XSk4y1CIk362J6mqadtXSyqEtFGSz8IwR30OMUmqad1XQ3VF1Wv2Ds5Xv1J+Df2t3RaJqmuY1uWdTm6A5j4/c+VxgLoDRN085iOlnUxFppdD/5BcPkZ9wdjaZpzczjjz/Oc8895+4wnEp3Q9Vk/TxI3wDTXoegiPrP1zRNa+F0y6K63P2wbI5RtbPvle6ORtO0ZuLJJ5+kR48ejBw5kl27dgGQkpLCpEmTSEhIYNSoUezcuZO8vDyio6Ox2rYHKCoqonPnzpSXl7sz/HrploU9peDbP4F4ee6GRpqm1e77Bxq2hbAj2veFyU/XecrGjRtZsGABmzdvpqKigkGDBpGQkMDs2bN59dVXiYuL49dff+WOO+5g+fLlDBgwgJUrVzJ27Fi+/fZbLrjgAnx8PHtXQ50s7G35CFKWw4XPQWjn+s/XNE0DVq1axWWXXUZgYCAAU6ZMoaSkhLVr13Lllad6KEpLSwGYPn06H3/8MWPHjmXBggXccccdbom7IXSyqFJ4FH54ELoMNxbgaZrW/NTTAnAlq9VKaGhojXWapkyZwkMPPUROTg4bN25k3LhxboiwYfSYRZVF90F5MUx5uXluaKRpmtuMHj2ar776iuLiYgoKCvjmm28IDAwkJiaGTz81NqhSSrFlyxYAWrVqxeDBg7nnnnu4+OKLsVgs7gzfIfpTEWDHt5D8lbE/QEScu6PRNK2ZGTRoENOnT6d///5MnjyZwYMHA/DBBx/w5ptv0r9/f3r37s3XX3998prp06fz/vvvM336dHeF3SC6RHnxcZg7FIIiYfZPp/Yz1jStWTjbS5Q3RFNKlJvashCRSSKyS0T2isgDNTz/BxHZKiKbRWS1iPSyHe8qIsW245tF5FXTgqwsg04JcOnLOlFomqbVwrQBbhGxAHOBCUA6sEFEFiqlku1O+1Ap9art/CnA88Ak23MpSqkBZsV3Uqu2MPND099G0zStOTOzZTEE2KuUSlVKlQELgEvtT1BK5dt9GQS0jD4xTdO0FsbMZNEJOGj3dbrt2GlE5E4RSQGeBe62eypGRDaJyEoRGWVinJqmNXMtZezVTE39Gbl9NpRSaq5Sqhvwf8DDtsOZQBel1EDgL8CHInJGjXARmS0iSSKSlJWV5bqgNU3zGP7+/mRnZ+uEUQelFNnZ2fj7+zf6NcxclHcIsF8GHWU7VpsFwP/+v737j72qruM4/nxNUJw6BGFG+7qAhrVsU5k2KXRsMkpyZq4l6RZFW1FR2dYcy805/tKcbdVajYppjTmWhrGmAf0wzcYPZUAgJlS0fU3AgqHkxgDf/fH5XDhc7i+4955z4ft6bHf33HM+557393PP+b7v+ZxzPx+AiDgMHM7TL+UzjyuBk253ioilwFJId0P1LHIzO2sMDQ0xPDyMvzC2NmbMGIaGhs54/X4mi43ANElTSEliHnBXsYCkaRGxM7/8OLAzz58I7I+IY5KmAtOAf/QxVjM7S40ePZopU6ZUHcY5r2/JIiKOSloErAbOA5ZFxHZJS4AXI2IVsEjSbOAIcACYn1e/CVgi6QjwDrAwIvb3K1YzM2vNP8ozMxvBBuJHeWZmdm44Z84sJL0B/KuLt5gA/KdH4fSD4+uO4+uO4+vOIMf3noiY2K7QOZMsuiXpxU5Oxari+Lrj+Lrj+Loz6PF1ws1QZmbWlpOFmZm15WRxwtKqA2jD8XXH8XXH8XVn0ONry9cszMysLZ9ZmJlZWyMqWXQwGNMFklbk5eslTS4xtisk/VHSy5K2S/pGgzKzJB0sDAp1f1nxFWLYXRiw6pRfQSr5fq7DrZKmlxjb+wp1s1nSm5LuqStTah1KWiZpn6RthXnjJa2VtDM/j2uy7vxcZqek+Y3K9Cm+hyW9kj+/lZIubbJuy32hj/E9IOm1wmc4t8m6LY/3Psa3ohDbbkmbm6zb9/rrqYgYEQ9SlyN/B6YC5wNbgA/UlfkK8OM8PQ9YUWJ8k4DpefoS4NUG8c0CflNxPe4GJrRYPhd4BhBwA7C+ws97D+ke8srqkNR1zXRgW2Hed4DFeXox8FCD9caT+kMbD4zL0+NKim8OMCpPP9Qovk72hT7G9wDwrQ4+/5bHe7/iq1v+CHB/VfXXy8dIOrNoOxhTfv1Ynn4CuFmSygguIl6PiE15+i1gBw3G/zgLfAL4eSTrgEslTaogjptJoy1280PNrkXEc0B9v2bF/ewx4PYGq34UWBsR+yPiALCWE6NI9jW+iFgTEUfzy3WkHqMr0aT+OtHJ8d61VvHl/x2fBh7v9XarMJKSRSeDMR0vkw+Wg8BlpURXkJu/rgXWN1g8Q9IWSc9IuqrUwJIA1kh6SdIXGyzvaNCrEsyj+UFadR1eHhGv5+k9wOUNygxKPS4gnSk20m5f6KdFuZlsWZNmvEGovxuBvXGiZ+16VdbfaRtJyeKsIOli4Engnjh52FmATaRmlauBHwBPlR0fMDMipgO3AF+VdFMFMbQk6XzgNuCXDRYPQh0eF6k9YiBvSZR0H3AUWN6kSFX7wo+A9wLXkAZKe6Sk7Z6uz9D6rGLgj6WikZQsOhmM6XgZSaOAscB/S4kubXM0KVEsj4hf1S+PiDcj4lCefhoYLWlCWfHl7b6Wn/cBK0mn+0WnO+hVP9wCbIqIvfULBqEOgb21prn8vK9BmUrrUdLngFuBu3NCO0UH+0JfRMTeiDgWEe8AP2my3arrbxRwB7CiWZmq6u9MjaRkcXwwpvzNcx6wqq7MKk6MqfEp4A/NDpRey+2bPwN2RMR3m5R5V+0aiqQPkT6/MpPZRZIuqU2TLoRuqyu2CvhsvivqBuBgocmlLE2/0VVdh1lxP5sP/LpBmdXAHEnjcjPLnDyv7yR9DLgXuC0i3m5SppN9oV/xFa+BfbLJdjs53vtpNvBKRAw3Wlhl/Z2xqq+wl/kg3anzKukuifvyvCWkgwJgDKnpYhewAZhaYmwzSc0RW4HN+TEXWEga/AlgEbCddGfHOuDDJdff1LztLTmOWh0WYxTww1zHfwWuKznGi0j//McW5lVWh6Sk9TppgK9h4Auk62C/J40M+TtgfC57HfDTwroL8r64C/h8ifHtIrX31/bD2h2C7waebrUvlBTfL/K+tZWUACbVx5dfn3K8lxFfnv9obZ8rlC29/nr58C+4zcysrZHUDGVmZmfIycLMzNpysjAzs7acLMzMrC0nCzMza8vJwqwBSX/Jz5Ml3dXj9/52o22ZDTLfOmvWgqRZpB5Obz2NdUbFiY74Gi0/FBEX9yI+s7L4zMKsAUmH8uSDwI15zIFvSjovj/ewMXdk96Vcfpak5yWtAl7O857KncRtr3UUJ+lB4ML8fsuL28q/en9Y0rY8zsGdhfd+VtITSuNMLC+rN2SzmlFVB2A24BZTOLPI//QPRsT1ki4AXpC0JpedDnwwIv6ZXy+IiP2SLgQ2SnoyIhZLWhQR1zTY1h2kzvGuBibkdZ7Ly64FrgL+DbwAfAT4c+//XLPGfGZhdnrmkPq+2kzqQv4yYFpetqGQKAC+LqnWrcgVhXLNzAQej9RJ3l7gT8D1hfcejtR53mZgck/+GrMO+czC7PQI+FpEnNSpX7628b+617OBGRHxtqRnSX2PnanDhelj+Ni1kvnMwqy1t0jD3NasBr6cu5NH0pW519B6Y4EDOVG8nzTEbM2R2vp1ngfuzNdFJpKG7NzQk7/CrEv+dmLW2lbgWG5OehT4HqkJaFO+yPwGjYdF/S2wUNIO4G+kpqiapcBWSZsi4u7C/JXADFJPpAHcGxF7crIxq5RvnTUzs7bcDGVmZm05WZiZWVtOFmZm1paThZmZteVkYWZmbTlZmJlZW04WZmbWlpOFmZm19X+9y8Un9J+lNQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fd70ec849e8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_accs(theta_perc_history);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# execute this code to write the predictions on the dev and training data\n",
    "y_hat_dv = clf_base.predict_all(x_dv_pruned,theta_perc,labels)\n",
    "evaluation.write_predictions(y_hat_dv,'perc-dev.preds')\n",
    "y_hat_te = clf_base.predict_all(x_te_pruned,theta_perc,labels)\n",
    "evaluation.write_predictions(y_hat_te,'perc-test.preds')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.4311111111111111\n"
     ]
    }
   ],
   "source": [
    "y_hat = evaluation.read_predictions('perc-dev.preds')\n",
    "print(evaluation.acc(y_hat,y_dv))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 5. Logistic regression\n",
    "\n",
    "Total: 1.75 points\n",
    "\n",
    "You will implement logistic regression in PyTorch."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5.1 Converting data to numpy\n",
    "\n",
    "Numpy is a package for numerical computing in python.\n",
    "\n",
    "You will need to convert your bag-of-words list of counters to a numpy array. \n",
    "\n",
    "- **Deliverable 5.1**: Implement `preproc.py:make_numpy()` (0.5 points)\n",
    "- **Test**: `test_pytorch/test_d5_1_numpy`\n",
    "- **Hint**: one approach is to start with `numpy.zeros((height,width))`, and then fill in the cells by iterating through the bag-of-words list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[0. 0.]\n",
      " [0. 0.]\n",
      " [0. 0.]\n",
      " [0. 0.]]\n"
     ]
    }
   ],
   "source": [
    "X = np.zeros((4,2))\n",
    "print(X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 0.   0. ]\n",
      " [ 0.  -1. ]\n",
      " [ 1.5  0. ]\n",
      " [ 0.   0. ]]\n"
     ]
    }
   ],
   "source": [
    "X[1,1] = -1\n",
    "X[2,0] = 1.5\n",
    "print(X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "reload(preproc);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X_tr = preproc.make_numpy(x_tr_pruned,vocab)\n",
    "X_dv = preproc.make_numpy(x_dv_pruned,vocab)\n",
    "X_te = preproc.make_numpy(x_te_pruned,vocab)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['1980s', '1990s', '2000s', 'pre-1980']\n"
     ]
    }
   ],
   "source": [
    "label_set = sorted(list(set(y_tr)))\n",
    "print(label_set)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "Y_tr = np.array([label_set.index(y_i) for y_i in y_tr])\n",
    "Y_dv = np.array([label_set.index(y_i) for y_i in y_dv])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4"
      ]
     },
     "execution_count": 75,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(set(Y_tr))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5.2 Building a logistic regression model\n",
    "\n",
    "- **Deliverable 5.2**: Complete `logreg.build_linear` (0.25 points)\n",
    "- **Test**: `tests/test_pytorch.py:test_d5_2_logreg`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from gtnlplib import logreg\n",
    "reload(logreg);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "torch.manual_seed(765);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X_tr_var = Variable(torch.from_numpy(X_tr.astype(np.float32)))\n",
    "X_dv_var = Variable(torch.from_numpy(X_dv.astype(np.float32)))\n",
    "X_te_var = Variable(torch.from_numpy(X_te.astype(np.float32)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It's always a good idea to check the dimensions of your data first."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([4000, 4875])\n",
      "torch.Size([450, 4875])\n"
     ]
    }
   ],
   "source": [
    "print(X_tr_var.size())\n",
    "print(X_dv_var.size())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model = logreg.build_linear(X_tr,Y_tr)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Variable containing:\n",
      "-0.7673 -0.7583 -0.3767  0.0325\n",
      " 0.1110 -0.3801 -0.3979  0.1642\n",
      "-1.8015 -0.5595 -3.5092  0.9633\n",
      "               ⋮                \n",
      "-0.3965  0.0962 -0.4972 -0.1959\n",
      "-0.1721 -0.5167 -0.2574  0.1571\n",
      "-0.3643  0.0312 -0.4181  0.4564\n",
      "[torch.FloatTensor of size 450x4]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "scores = model.forward(X_dv_var)\n",
    "print(scores)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5.3 Adding a log-softmax layer\n",
    "\n",
    "We're done it for you in PyTorch. Write your own log-softmax function in numpy and verify the results.\n",
    "\n",
    "- **Deliverable 5.3**: Complete `logreg.log_softmax` (0.25 points)\n",
    "- **Test**: `tests/test_pytorch.py:test_d5_3_log_softmax`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model.add_module('softmax',torch.nn.LogSoftmax(dim=1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Variable containing:\n",
       "-1.7426 -1.7336 -1.3520 -0.9427\n",
       "-1.1841 -1.6752 -1.6930 -1.1309\n",
       "-3.0214 -1.7793 -4.7291 -0.2566\n",
       "[torch.FloatTensor of size 3x4]"
      ]
     },
     "execution_count": 83,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.forward(X_dv_var)[:3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Variable containing:\n",
       " 0.1751  0.1767  0.2587  0.3896\n",
       " 0.3060  0.1873  0.1840  0.3227\n",
       " 0.0487  0.1688  0.0088  0.7737\n",
       "[torch.FloatTensor of size 3x4]"
      ]
     },
     "execution_count": 84,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.forward(X_dv_var)[:3].exp()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that each row sums up to one."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Variable containing:\n",
       " 1\n",
       " 1\n",
       " 1\n",
       "[torch.FloatTensor of size 3]"
      ]
     },
     "execution_count": 85,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.forward(X_dv_var)[:3].exp().sum(dim=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "reload(logreg);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[-1.742636  , -1.7335784 , -1.3519608 , -0.9427422 ],\n",
       "       [-1.1840885 , -1.6752369 , -1.692963  , -1.1309159 ],\n",
       "       [-3.0213723 , -1.7793286 , -4.729114  , -0.25659686]],\n",
       "      dtype=float32)"
      ]
     },
     "execution_count": 87,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "logreg.log_softmax(scores[:3].data.numpy()) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These should be very close to the PyTorch results."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5.4 Negative Log-Likelihood Loss\n",
    "\n",
    "A loss function tells you how well your model is doing. It produces gradients that allows the optimizer to tune the model weights. We've done the Pytorch call for you, try implementing this yourself in numpy!\n",
    "\n",
    "- **Deliverable 5.4**: Complete `logreg.nll_loss` (0.25 points)\n",
    "- **Test**: `tests/test_pytorch.py:test_d5_4_nll_loss`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "loss = torch.nn.NLLLoss()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "Y_tr_var = Variable(torch.from_numpy(Y_tr))\n",
    "Y_dv_var = Variable(torch.from_numpy(Y_dv))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Variable containing:\n",
      " 1.5013\n",
      "[torch.FloatTensor of size 1]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "logP = model.forward(X_tr_var)\n",
    "print(loss.forward(logP,Y_tr_var))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "reload(logreg);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.5013313"
      ]
     },
     "execution_count": 92,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "logreg.nll_loss(logP.data.numpy(), Y_tr)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again, This should be very similar to the PyTorch result above."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5.5 Putting everything together\n",
    "\n",
    "An optimizer can be used to actually learn the weights. We provide the complete code below that you can train on in `logreg.train_model`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "reload(logreg);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# build a new model with a fixed seed\n",
    "torch.manual_seed(765)\n",
    "model = logreg.build_linear(X_tr,Y_tr)\n",
    "model.add_module('softmax',torch.nn.LogSoftmax(dim=1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1: Dev Accuracy: 0.2822222222222222\n",
      "Epoch 11: Dev Accuracy: 0.29555555555555557\n",
      "Epoch 21: Dev Accuracy: 0.29555555555555557\n",
      "Epoch 31: Dev Accuracy: 0.31333333333333335\n",
      "Epoch 41: Dev Accuracy: 0.30666666666666664\n",
      "Epoch 51: Dev Accuracy: 0.35777777777777775\n",
      "Epoch 61: Dev Accuracy: 0.32222222222222224\n",
      "Epoch 71: Dev Accuracy: 0.34444444444444444\n",
      "Epoch 81: Dev Accuracy: 0.3333333333333333\n",
      "Epoch 91: Dev Accuracy: 0.33111111111111113\n"
     ]
    }
   ],
   "source": [
    "model_trained, losses, accuracies = logreg.train_model(loss,model,\n",
    "                                                       X_tr_var,\n",
    "                                                       Y_tr_var,\n",
    "                                                       X_dv_var=X_dv_var,\n",
    "                                                       Y_dv_var = Y_dv_var,\n",
    "                                                       num_its=100,\n",
    "                                                       optim_args={'lr':0.02})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAsoAAACdCAYAAACtgl4mAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvAOZPmwAAIABJREFUeJzsnXecY1d5/p9XvUvT++5sm+3e9Rav6+KCG8aFFgyGxIEESHDwjxpqqIYEYlowJAZCDMF2bOq647pre922t9nu2el91Lt0fn/ce66upCtpRlN39nw/H33s0ehKZ2a0V899zvO+LzHGIBAIBAKBQCAQCLLRzfYCBAKBQCAQCASCuYgQygKBQCAQCAQCgQZCKAsEAoFAIBAIBBoIoSwQCAQCgUAgEGgghLJAIBAIBAKBQKCBEMoCgUAgEAgEAoEGQigLBAKBQCAQCAQaCKEsEAgEAoFAIBBoIISyQCAQCAQCgUCggWG2F6Cmurqatba2zvYyBAKBYMLs3r17mDFWM9vrmEnEOVsgEJytjPecPaeEcmtrK3bt2jXbyxAIBIIJQ0RnZnsNM404ZwsEgrOV8Z6zRfRCIBAIBAKBQCDQQAhlgUAgEAgEAoFAAyGUSxBNpPCFPxzAcDA220sRCAQCgWDaeOJgH/731XMuQSQQFEUI5RIc7vXhgde78PLJ4dleikAgEAgE08aDb3Thx8+emO1lCARzCiGUSzAWSgAAvOHELK9EIBAIBILpI5JIYTAQw1goPttLEQjmDEIol2A0LJ0wxsLixCEQCASC+Us0kQIAHBsIzPJKBIK5gxDKJfDKAlk4ygKBQCCYz0TiklA+LoSyQKAghHIJxmSB7IsIoSwQCASC+UtEdpSP9guhLBBwhFAuAc9qieiFQCAQCOYzPHpxXAhlgUBBCOUSjInohUAgEAjOAXj04thAAIyxWV6NQDA3EEK5BDx64RWOskAgEAjmKYwxRBIpeGxGBKJJ9Pmis72kcfGLF0/jcK9vtpchmMcIoVwCpZhPZJQFAoHgnGZv5xgefL1ztpcxLcRTaaQZsK7ZA+Ds6HwRjifxrcfa8fCu7tleyjnJfTs7cLB7/l+kCKFcgtFQppgvlRZbUQKBYPohoruJaPVsr0OQzf+90YVvP94+28uYFqLxNABgXYsklM+GnHLXaAQAMBg4O9zvucTzxwbxxMG+so9njOGbjx7B/+zsmLpFafDwri482z4wra9RCiGUi8AYgzcch9WoB2NAICpcZYFAMCO0A7iXiF4joo8RkXu2FySQukIEYkmk56Fpwjte1LssqHdZzgpHuWs0DAAY8MdmeSVnHz974RT+9cmjZR8fiCWRTDOcHJy+98lrp0fwud8fmPVpkUIoFyEovxFaq+0AMnllgUAgmE4YY79gjF0C4K8BtAI4QET3E9EVxY4jouuI6BgRnSSizxd53LuIiBHRJtV9X5CPO0ZE107VzzKfiMRTYAwIxpOzvZQphwtlq0mHtnonjp0FjnKnLJSFozxxRkNxdI6GES7zveyVd9tPDAanpfAzEE3gUw/tB2NSu8JkKj3lrzFeplUoE1EHER0kon1EtGs6X2s64OOrF8tCWRT0CQSCmYKI9ABWyLdhAPsBfIqIHizy+HsAXA9gFYD3EdEqjcc5AdwJ4DXVfasA3ApgNYDrAPxUfj6BimhS+rD2z8OaFd7xwmrUY0W9EycGg3M+bqgIZX9MdOmYIKOhOBgDTg4GyzqedwQLx1PonYbCz288cgR9vghu27IAsWQaHSOhKX+N8TITjvIVjLH1jLFNpR86t+BvhNZqGwDRIk4gEMwMRPQDAEcBvA3AtxljGxlj/8YYuxHA+QUOuwDAScbYacZYHMCDAG7WeNw3AfwbAPWn280AHmSMxRhjbwI4KT+fQEVUFpP+yPx1lC1GPdrqnIgn0zhTRJycGgri/tdmt7CRRy9iyfS8/JtMF6k0U/RNucNl1LMlxiO2nzs6gBeODY7ruZ85MoCHd3fjHy9fitu2LAQAHOmbvR0OEb0oAn8jLKp2AAC8EeEoCwSCGeEAgPWMsY8yxl7P+V4hAdsEoEv1dbd8nwIRbQDQwhh7bKLHqp7jI0S0i4h2DQ0Nlfgx5hfRpCQm5+OkVj5sxGrUY3mdEwCKxi8eeqMLX/zjQcSTs7cl3jkahl5HAET8YiJ4w5KbDJRftKkWyifGkWe/+y/H8dVth8f13I8f7EO1w4RPXLUMS2sdMOoJ7X1+5fvRRAo/fvbEjNWNTbdQZgD+QkS7iegjWg+Yyydd7iAv4hnl0Pw7OQoEgjmJF4CBf0FEHiK6BQAYY2X1YyIiHYDvA/j0ZBbGGLuXMbaJMbappqZmMk911sHjCf5JfEA/2z6A3WfGpmpJU4YSvTDpsazOAaLiLeL45+NsTa1ljKFrLIyVDZKoHwycOwV9HcMh/OiZEzjU4ysrcjIayvzNyi3a5HrIbNCNy1EeCsRwZiRcdJeCMxiIoaXSBpNBB5NBh6W1ThzpzQjlZ9sH8f2nj+OR/eV37ZgI0y2UL2WMbYCUmfs4EW3NfcBcPunyN9PCKhuIRC9lgUAwY3xVLYgZY14AXy1xTA+AFtXXzfJ9HCeANQBeIKIOABcC2CYX9JU6VoCMozyZjPK3H2/HT58/OVVLmjIiKkfZYtSjyWPFmZFwwcdzV10tumaSoWAM0UQamxZWAji3HOUH3+jCD545jrf/x0u45gc78OSh/gkdPyL/zRrcloK7Br5wAj994SQSBYrovOE4iIC1TW6cKCGU02mmvE92HC9tiA4Goqh1mpWvVzY4sxxl/hy7zoyWfK6pYFqFMmOsR/7vIIA/4izLvPE3QoXNBJfFKIr5BALBTKF1bjZo3KfmDQDLiGgREZkgFedt499kjPkYY9WMsVbGWCuAVwHcxBjbJT/uViIyE9EiAMsA5EY+znmiCbmYL1p+HjYYS85J00WdUQaAOpelqPicCqH8yqkR7Dw5XNaxPJ+8qbUCwLnVIq7XG0Gj24K73rEGyTTDl/90aELO8pj8N7tocRUGAzHlazV/2NuN7z55DM8c0e5hPBZOwG01Ynm9EydLdL7wRRJIyoWhO06U/nsP+GOodVqUr1c1uDAYiGE4KBVt7jghCeWZ2pmZNqFMRHa5uhpEZAdwDYBD0/V60wF/I+h1BI/NKIr5BALBTLGLiL5PREvk2/cB7C52AGMsCeAOAE9B6sP8EGPsMBF9g4huKnHsYQAPATgC4EkAH2eMpabkJ5lHZIr5yv8sCEaTc9J0UTLKJkko1zrNRcUnj5+MTEIof/epo/j2E+UNcOEdL1bUu2Az6TF4lgjlWDKFn+84Palsd58vggVVNty2ZSE+dOkiDAdjyu9jPPC/2YVLqgBoxy92dUgidNv+Xs3nGAvHUWkzYVmtA75IAkPBwr//Yfl7lXYTXjk1UtClBqT3oS+SQJ1L7Si7AADtfX6cHAyizxfF0loHzoyEMTQDkZvpdJTrALxERPshOROPMcaenMbXm3L4GwEAPDbTrGWxBALBOcc/AYgD+D/5FgPw8VIHMcYeZ4y1McaWMMbuku/7F8bYNo3HXi67yfzru+TjljPGnpiyn2QeoUQvyswop9MMoXhqThYDqtvDAZJQHvSPw1EuIpBKMeiPoXssUtaxnSPScc0V1pLu91zipRPDuOvxdrx4ovyarF5vFI0eKwBg00LJUefCdjyMqhxlADieI5QZY0qs4dmjg5pFc2PhODw2I5bWShnxkwOF4xdcRN+0rhHBWBJ7ijjBXPiqHWW1UN4uxy7uvGoZAGD3DMQvpk0oyy2K1sm31fykfTbB3wgAUGEzzsmTm0AgmH8wxkKMsc/z+g3G2BcYY7PXSFSAZCqNREraPi63FVlYdm294cSc6/ubG72odVngjyYVpzmXyUYvGGMYCsTgDSfK6l7QNRZGvcsCi1GPGqc5y1EeC8Xxm1c6JvU7Ptrvx9MFYgeToU/uOVws/12MVJqh3x9Fo1sSym11TjgtBuyaQAxhNBSH02JAc4UVbqsxL6fcPRbBgD+GW9Y3Ip5Ma/4exkIJVNhMWFYndQUrllMeDkrvkZvWN0KvIyU6oQW/4KlROcqVdhPqXRYc6fVjx4lhLKmx45rVdTAZdBO6QCiXedkebjAQnRI7nr8RAMBjNQpHWSAQzAhEVENE3yOix4noOX6b7XWdy0RVW+XlmiZBOducTDOE43Mr2RJJpGAy6JR2a7yYSivSkEozBOSfpdzohS+SQFzegi/HVe4cDWNBpU1Zq9pRfnh3F77y58M40F1WgxgAwE+fP4VPPLC3aEygHAZkl34iUQk1g4EoUmmmOMp6HWHDgooJOasjoTiq7CYQEZbX5U9h5Nnfv7tsMZo8Vs34hTcch8dmQq3TDKfFULTzxbCsxxZV2bFhgQc7jhfOKfP3W53KUQakgr59XV68dnoEW9tqYDbosa7ZPaELhHKZl0L5sw8fwGce3j/p5/GG46iwZ6IXIqMsEAhmiN9CGjiyCMDXAXRAKtYTzBIRlbAtN3oRjGWc6LlW0BeNp5TYBSA5yoB2Nwm1A1yuo6w2s8oRyl2jYTRXSmJRil5kpvPxVmKTEVH9/igiiVRWt4WpoF92lHMnzR3s9uH5o6UHcvR6pd9VgycjJDctrMDxgSB849Qoo6EYKmVts7zeiWMDgSz3fdeZUTjMBqxscOHGdY146cRw3t95LJxAhc0IIsKyWgdODBZuMzccjMGgI7itRmxdVoNDvT6MFIjs8DZ/tSpHGZDiFx0jYcSSaWxtkzqkbVxYicO9voK7HlPFvBTKfb4Ierzl5Z7UjIbjqJCjFx6bEYFoclbnjQsEgnOGKsbYLwEkGGPbGWMfAnDlbC/qXEb9YVxuMV+WUJ5jO5SRRLZQ5sVUWv2J1Y56uUJ5MEsoT8xdjSVT6PdHsxzlcDyl/H7b5Sluk8mvciE/1Vv7/dxRzolefPepo/iH3+4uKCA5vV7p+CbZUQaAjXLnjz2d41vrSDCOSrv0922rdyIQTSrrAqSf+fwFHuh1hBvXNSCZZnj8YKZncTSRQiSRUozEpbWO4o5yMIYqhwk6HWFrWw0YA14q0O1kwB+FQUdKfRhnVaOUUzYZdLhwkZSt3rSwAokUw/4u77h+7nKZl0LZG05otjuZCNFECtFEGh75j8UjGCKnLBAIZgB+oukjohuI6HwAlbO5oHOdqJLh1Smxg4kSUgnl8bp/M0UkkVY6XgCZYqoBjYI+/jloMuimxFHuGs0YW+k0w/2vdRbNLfeMRcAYMkJZJeqjiRRODkmibVfHWNk5ZV7IONUtyLij3DUWRiqdWdvR/gCiiTR+/cqZoscrjrI74yivb5FE7Xj7Co+G4qi0SyYgn8LIR1n7IgkcGwgo/alXNbiwpMaeFb/gu+tcFy2rdWI4GC+ou4aDcVTJwnxNkxvVDhP+clg7/z0YiKHGaYZOjgBxeEHfBa2Vyvt0Iy9knOb4xbwTyowxeCMJjIXjSKfLD/LzPLKSUZad5bE5dnITCATzkm8RkRvSFL3PAPgFgE/O7pLObXgP5VqnZUoc5blmukTiKaWQD5AK2I16Kuoot1bZJi2UG9yWLEd5f7cXX/zjQdy3s6PgsTzfy4Uyz7MO+mM4ORhEKs1w4eJKDAbK66oRjCURkqM2u86MTmnhZb8/CrtJj0SKKaJ3NBTHUCAGo57w61c6EI4XvhDr80XhtBjgtBiV+2wmA1Y3usblfjPGpI5esnBVhLLswu/tHANjmf7URISrV9VjV8eoktfmf3O+475ULujjFyi5DAdjqJYz73od4fo1DXimfSDr3wNnMBDLGjbCaa2yY22TG+/a2KTcV2E3YUmNfdr7Kc87oRxNpBFPppFmkzsR8fGM/KrLozjKc2u7TCAQzC+ISA9gmTwg5BBj7ArG2EatFm+CmYN3hahzmRGIJbPcwPESjM7hjHIiBasxIwmICLVOi2YxX0Yo28s2pQYDUZgNOqyod2aJ2SNyJrhQ/14gM2wk31GOKsd/8MJWAOVNb+Nu8vkLPBiYRAu7XEKxJALRJDa2Sm4t73zBi+n+4fKlGAsn8PCu7oLP0eONKB0v1GxcWIH93V4kUmlEEyk8tKtL05UPxJJIpBiq5NiE22bEinon/mvHKZwcDGD3mTHodYT1LR7lmMXVdqRZxs3msSGui9oUsa2d5x4JxlHtyEQpblrfiFgyrTnMZNAfRU1OIR8gCexH/ulSvOP85qz7Ny2sxO4zY5MyRksx74SyujPF6CQyYGM5bwSPVXaUQ3Pr5CYQCOYX8qCP9832OgTZ8OgFjyQEy4hfhOLqjPLc+iyJJFJZ0QsAUts1jWI+3h5vUY0koMoR/UOBGGpdZrRU2rIcZV48d3wgWHC8cudoGGaDDjWy81ijcpSP9PphM+lxzeo6OM2Ggi7rEwf70OfTFsDcRb9hbQMAbbGdTKXx5309mq5oIXgOeMsiWSiPSgV9vI/xbVsWYOPCCvz8xdMF66H6fBE0evKF5ObWSkQTafxpbw9uuedlfO53BzRjHKNyqzZezAcA935wE4x6HT7wi9fx9JEBrGxwwm7ODAJtkS9IeESG76xXyEZio9uCCpsRh3ryhTJjDEPBGGocGZd444IKNLotmhdDg4FY1rCRUmxqrUA0kZqSurRCzDuhrD75TGa0Zm70gv9X64Twm1fP4E97e8p+LYFAIMjhZSL6CRFdRkQb+G22F3Uuowhl+UO8nB1LLqp0BHhnaHfy1dMjOF1gS1xNJKfrBcCHjhR2lBdX2wFIXRQmymBAEk8tFTb4o0nlOY/0+rGs1gG9jrBtv/bnaudoGC2VNhBJOVaXxQCLUYfBQBTtfX4sr3fCqNdh/QKP5rZ850gY//DbPfjqnw9rPj/PZV+2rKag2H78UD/ufHAfPvLrXYglM4WeB7q9BcdyD8j55PUtHpgMuoyjPBCAx2ZErdOMj25djO6xCB5TFc+p6fVG0eDJd5T54JHP/u4AhuSc766OfIHP2/lVqhzeBVU2/PpDFyAcT+JofyafrP4+kIm85OojIsKaJjcO9uS34wvEkogn06hWCWWdjnDjukbsOD6UlWuOJ9MYDcWzho2U4sZ1jTj4tWsVMT8dzD+hrDr5TE4o87C6dMXklv+rVan8q5fexN1PH5tzDeQFAsFZy3oAqwF8A8Dd8u3fZ3VF5ziRHEe5nBZxwWgSeh2hymGe1BjsifDph/bja48cKfm4aCI7owyg4MQ7XyQBo57Q5JHEyUhw4p+1Q4EYap0WNFdIoq97LIx0muFofwCXLK3GxUuq8Mj+vrzP1bFQHDtPjSjFXUAmJjLgj6G9z698b9PCShwbCORd1DxyQHIyn24f0OzWwPPT9S5LQbH99JEBWIw67Dw1gjsf2IdYMoUfPH0ct9zzMm7/1RtKPEQNd5Qb3BYsqLThjNwi7lh/AG11ThAR3rqyDour7ZoZ7Ug8hdFQPKvjBafWZcFFi6tw5YpaPPH/LsNVK2o1IwlcF1XZs7tKrGxw4b9v34wquwlXrazN+l69ywKjnhShnIleZHLSa5rcOD4QyGvVxnsoVzuzX+/GdY1IphmeONSv3Mcn+OW2hiuGxaiHyTC9UnbeCWXfVDnKoezohctigF5Hmttlw8EYukYjeHNYDM4SCASTR84l595Ee7hZJCYX8/Ft4XKEbiiWhMNsgMdqnLHohTccx2unR0r2ms1tDwdIjvJYOJHlmAKSUHZbjcr2fTnDuHh3g+YKSWx3j0XQORpGOJ7CKrl/b+doGPtzhob8+LkTCMWS+Kcrl2bdX+cyY2/XGPzRZEYot1aAsfy2aY/s78WKeidMeh1+vuO05trMBh1cVoOm2I4n03jh6CBuWteIr7x9FZ483I9L/vV5/OjZE3jb2gbodMD3njqW97x8Kl+924KFlTacGQmDMYbj/QGlqE6nI7x/ywLs6fTmRU94VETd8ULNAx+5EP99+2bUOi3YuLAC/mgyr8COu/+VOUJZ+n1VYteX34rLltVk3a/XEZo8VkX8j4YSsJv0MBsy75e1TW4k0yxvHDafyqd2lAFgdaMLi6vteEQVv+DZ8IlEL2aCeSeUvVPQ3xGQ/uE7zAblSoVIapade0KIJ9Pwy1k1PoNcIBAIJgMR/YvWbbbXdS4zFY5ygAtl28wI5VSaIRRPIZZM4/U3ixe1aWWUubOXO+nWH0nApRLKE53OF0um4IskUOs0qxzliFKIt7LBhWtX18Ok12HbvoyQ6hgO4TevnMF7Ny9QCsiUtTotSoZ2lSyUedu03aroxPGBAI72B/D+LQvwnk3N+OPenrwWeAP+KGpdZhCRIrb3qsT2a2+OIBBL4upV9fjwpYtw51XLkEyn8cP3rsdP3r8Bf3fpYmzb35vX33fAH4XLYoDNZMDCKjs6R8Po9UURiCWxvD7z87xzQzNMeh0eeL0z63jeQ7lRw1HOZZNcMJgbGxlRHGVtMcrjLLm0VNrQNZZxlD05fY7XNrkBIC9+MSy7xLmvRyTFL159c0T5/SvDRiYQvZgJ5p9Qlk8+Bh1NSih7w4msbQVA2mbIzSiPqLJZQigLBIIpIqS6pQBcD6B1Nhd0rhNVdb0AMgVtE4E7ym6raUa6XqgLzXaU+HzSzCgr0/myhTJ3lHkx16hG9CItD6nQKnbjwrvGaYbHZoTDbEDXaBjtfX7odYRldQ64rUZcvrwGjx7oVXoPf/epozAZdPjk1cvynpMX9hEBK2TRaTcbsKrBhTdUWd1t+3qhI+D6NQ34+8sWI5lO479ffjPruQb9MUWsKT2KVYLzGTl2cenSagDAJ69uw96vXI1bzpdal33s8iWodphw1+PtWdGRfl8U9bIbvLDKhnA8hZdPSHlmtVCutJtw7Zp6/GFPd9ZOQK/sKGtFL3JprbKhym7KK0QcDcZhMeryLopKsaDSlpVR5n97TnOFFW6rEYcKCOXc6AUAvP28BjAmxViAjKOs1R5uNpl/QjkSh0mvQ53LMqmhI2PhuBJU50jbZdnPybNZTR4rXh3H9pZAIBCUgjF2t+p2F4DLASwudRwRXUdEx4joJBF9XuP7HyOig0S0j4heIqJV8v2tRBSR799HRP855T/UWc5UOMqhWAp2sx4em3FGMsrq9mA7ThQWyuk0QyyZzssoc8EymOO4cqFsNujhNBs0HeX/2dmBf/ztHjyq0dlgSDWmmIjQXGFF91gE7X1+LK62K+u47cKFGArGcNG/Pov3/OdOPH6wHx97yxJNx7FOFvULK21ZHRu2tlXjtTdH8fvd3WCMYdv+XlyytBo1TjMWVtlx/doG3P9qZ9bfczAQVS6I7GYDNi6owP/t6sJoKA7GGJ4+MoBLl9ZkiU21E+swG/D/3tqG198cxV9ULdAG/NHMOuUCub8ckTK6bbXZDvn7NrfAH03iiUOZor5ebwREmZ+1GESEjQsr8vLVo+F4QTe5GAsqbfCGE/BFEvL46mx9JBX0ufI6XwwHYiBC3qQ9QJro1+SxKhdxg4EYdARUOYRQnlZ84QTcNiOqHKaC20HpNFNC9IUYC8WV8YycCpspb7uMXy29a0MToonS21sCgUBQBjYAzcUeIPdfvgeS+7wKwPu4EFZxP2NsLWNsPYDvAvi+6nunGGPr5dvHpnDt84JoIg2TXgenxQAdlZdRDsSSsJsNcGuYLtMBnyB4/gIPjg8EC7ZDi8oZ5LzohbO4owxI3RNyd29PDgbwb08eBZDplKBGcZQd0vNLQjmMI71+ZVQxALylrQbPf/pyfOLKZRgMxNBSacXfXbZI82fgol5d5AcAn7hqGS5ZWoXP/f4AfvD0cXSOhnHjeY3K9z90SSsCsSS2H8tcSKgdZQD42k2r4Q3H8aU/HsThXj96fVFcs6pOcx2cWze3YEGlDf/7aqZFW58vquSLF1ZJHUNePDGMepdFaRjAuXBxFVqrbHjg9a7M8d4oahzmcRevbWqtwJmRcFZ0RprKly9aS7FAaREX1oxeAFJB37H+AOLJTGu7oWAclTYTDPr8NRNJI613nhpBIpXGoD+GaocZep12/GO2mHahTER6ItpLRI9O92sBcmTCakSFzVSwwOCpw/248u7tynaOFtIVU/Yb162RK+OO8tvOa4DJoJtQ/GJflxenxtG2RyAQnFvIru8B+XYYwDEAPyxx2AUATjLGTjPG4gAeBHCz+gGMMbXdYwcgWvWME6krhA46HcFpMZbVHi4US8JpkYr5QvFUlqCYDrhQ5v2AXzyu3bYsIk+hy41eVNlN0Osor0VcllC2ZwvlRCqNT/7ffthMelTZTcqQCjWDgezuBs0VNpweDqHXF80Tuq3Vdnzy6ja88JnLsf0zV8BmMuQ9n/q5VuUcbzbo8V8f3IQ1jS78+LmTMOl1uHZNvfL9tU0eGPWk9G+OxFMIxJJKlAMAVjW68Kmrl+OJQ/34wh8Oggi4MqczRC4GvQ5XLK/Bro4xxJNpJFNpDAdjqJfd4CaPFXodIZZMZ8UuODod4b2bF+D1N0eVzhy9vohma7hCbJTbvO1WxS/KFcq8/Vr3WBhj4QQqc/QRAKxpdCOeSmcV9I0EY3mFfGre0laNYCyJvZ1eDASiE+p4MVPMhKN8J4D2GXgdAFL0wmMzospuKtiyho+4LNagWit6ITnKOdELOaPcXGHDlkWVExLKn/y/ffjWo6Xb9ggEgnOOtwO4Ub5dA6CRMfaTEsc0AehSfd0t35cFEX2ciE5BcpQ/ofrWItnU2E5El01q9fMQdfs0l9WgFHFPhGA0CbvJoNS/TPcY62BMev6NCytQ6zRje4H4BY+V5AplnY5Q4zBnFbul0wz+aAIueYRylT179/Y/nj2Bgz0+fOeda7Gk1qH5OTskb8fzFmXNFVbloiFXKHOICLoiTuOyWidsJj0uXlqV9z2H2YBf/e0FWNXgws3rGxWRDwAmgw5LahxKISFvh5cbb/jI1sXY3FqBgz0+bFxQUVT8cS5aUoVIIoWDPV4MBWNIM6BOdpRNBp0yOERLKAPAuzc2w2zQ4dty1rnXG0GTxrCRQqxpcsFk0GXlq0eC8bzWcOOBC+XTwyH4IglNR5kX9KlzytL46sKvd9GSauh1hB3Hh/Kc/LnCtAplImoGcAOAX0zn66jxhhNwW02osBdjbDdbAAAgAElEQVR2lPvkf/QjQe0m6fFkGoFoUjOjnOsCDAfjMBt0sJv0eEtbDU4OBrOmDBUilWboHgsXnI0uEAjOaRoAjDLGzjDGegBYiWjLVDwxY+wextgSAP8M4Mvy3X0AFjDGzgfwKQD3E5GmYiGijxDRLiLaNTR07hQwq7tCuCzlZYxDsSQcFgPc8mfLdAtl7ii7rEZctqwGL50Y1hy9zWtrLBoFXrUuc1b0IhhPgjFkOcq8HiiWTOHeF0/jxnWNuG5NA5o9VvRojH8eDMRQZc9sx/MWcQCwskFbNJai3m3B4a9fq7iouVTaTXjsE5fi3951Xt73VjW4FEd5wM87L2QLYb2O8P2/Wg+PzYib1zfmPYcWFyySRPsrp0aUHex6lQBvleMXuR08ODVOM/75uhV47uggHni9Sxo2ojG+uhBmgx7rmz3Ypcopl+sou63GrGK93B13QMpdOy2GrM4Xw8F40YsKt9WI9S0e7DgxNOGpfDPFdDvKPwTwOQAF95em+qQrXelIbWvC8ZRmcV2ffIVbKMM8XKDptUd+c6ld5WF5W4GIcPlyaSsm11VOptJ5PZYHA1EkUgzdYxFRACgQCHL5GQD1VXRIvq8YPQBaVF83y/cV4kEAtwAAYyzGGBuR/383gFMA2rQOYozdyxjbxBjbVFNTo/WQWWd/l3dSXY+0iCZSsBhUQnmCxXyMMQTjvOsFd5SnN6fMXW+nxYCtbdXwRRKa09MicekjOtdRBqScsloo81kFGaFsVorc9nV6EU2kcdM6SUg2VVjR748ikTOOeSiQvR3PW8RVO8yTchQLtTZTf1/LlV7Z4MKAP4aRYExxlLUiAC2VNrzxpbfigxe1jms9lXYTVtQ78erp0YxQVvVA5rnfFQUcZQC4/eJWXLK0Cl9/5DAiidS4WsOp2dhagUM9PkTiKemWSGVN5ZsICyptOCD3tc6t4QKk3+/qRle+o1zCfd+6rAYHe3wYCcWUceRziWkTykT0dgCD8km3IFN90uUZZX7FpHWy5E2/CznKmUKD7D9ug3wl2KvKNo8E46iW33RLauyotJtwMKdB+oNvdOGaH2zPer1u+SqbMShjLAUCgUCGmKqvFGMsDUA7nJnhDQDLiGgREZkA3ApgW9aTEqn7at0A4IR8f41cDAgiWgxgGYD8SQxnCR/85Wu45/mTU/qc0URacVxdVsOE28OF4ykwJnVR8Fj5pNfiYrvfF8XRfn/RxxSDd71wWYy4eInUykxrrHGh6AUgO8qq6AV3wV2KUDYinkojGEvildMjIAIukHv4NnmsSDPk1QMNBaJK6zkAaJEd5XLd5MnCCwjb+wJKHruugGAzahSlFePCxVXYdWZU6UGsdpQ3LKhAld2EpbWOgsfrdITvvXudUsDXWGDYSCEuWlyFZJrhkf29SlS0nOgFALRUWhXtohW9AKT4RXt/AIlUGuF4EuF4ClUlhPnWtmowJumhudYaDpheR/kSADcRUQck5+JKIvrfaXw9RBPS1RJ3lIHiQnm4QIZZ3eNRTZN81aveShoJxZRWJkSE5XVOHM2ZprOvy4tEiuHUUMZVVsczTo8jfuELJ4oWHwoEgnnFaSL6BBEZ5dudKCFcGWNJAHcAeApSXchDjLHDRPQNIrpJftgdRHSYiPZBilj8jXz/VgAH5Pt/B+BjjLGzsoVPKs3gjybRMcWTUiOJFCyyWCnHUQ7J/YT5wBGgtFD+xqOH8eH/2VXGaiUC0SSMeoLZoEO1wwSn2aAIHTWKUDblS4JapxkjobjiCvPIidpRBqTP2ldPj2B1o0vp4MA/M3ML+oYCsSwjym0zorXKpoj5mYbnotv7/BgIRGHS6/LmKJTLRUuqEE2k8dThAZj0uqzYwzs3NOG1L16V15Yvl0aPFXe9Yy0MOiqYZy7EZcuqsb7Fg7ufPqZol9xY6XjhOWVAu90bIOXh48k0nj4ygOGA9lS+XM5r9ii/7/G0vptppk0oM8a+wBhrZoy1QnI2nmOMfWC6Xg/I/AP22EwFhXI4nlSuiAtFL/i88UJCWS1yhwPZwfjl9U4cHwhkzVfn2Sf1ibt7NHPiOD2OE/pXtx3C7b96veTjBALBvOBjAC6GFJ3oBrAFwEdKHcQYe5wx1sYYWyL3XwZj7F8YY9vk/7+TMbZabgF3BWPssHz/71X3b2CMPTJtP9k0E4pLglRLEE6GmCqj7LZOPKMcVAtl6/gyygd7fOjxRrL6IU/oNaNS1IOIQERoqsiMIVbDu15oCTYuXHgk0ZcjlPnnX683ij2dXly4KFNMxwdjqAv6GGMYCsbyPl+f+dRb8NGtJVuFTwuVdhPqXGa09/kx5JfWVirGMV62LKoEEbD7zBjq3NnPS0SabdO0uGldIw59/VosrinsPmtBRPjSDSsx4I8pY7VLObyFWKASyoUuJN66sg5tdQ7825NHlQEpubvzueh1hEvk4S3nmqM843gVoVx4Bn2fypUdDmhHL/j9uW8ml0UKs2diEyzLUQakraNwPKVssyRSaZwYkBzjN0fUjnIE1Q4z6l2WcbWI2905htPDoawpPwKBYH7CGBtkjN3KGKtljNUxxt7PGBuc7XWdDYRjkujrHgtP6fkyos4oy4XdydT427txoWw3G+C0GECEotP5/NGEMpKZtwfj7OkcU8RtMQLRBJyWjKBprrBpXkBEi0UvZOHCi9wUoWzLFPMBwLPtA4gn07hoSUYo8zytehfWG04gkWJ5gsig1xXtajHdrGpw4UifH4OB2JS2KPPYTFhZLznW9ZN0S0s5z4XY3FqJ61bXK0V9lWUMHAGyhbJWRhmQ/o5feNtKnBkJ40fPnABQ2lEGpBaGFqMu6zXmCuMSykR0JxG5SOKXRLSHiK4Z74swxl5gjL29/GWOD76N5bGalG2B3BZxPL7gsRmzxk+rGQrGlKlDuTRXWJWrY380iUSKKRllAFgu/4Pg8YvTQyHE5ZOp2lHu8UbQXGHF4ho7Tg8Vd5R9YemEGU+mS27VCQSCsx8iuo+IPKqvK4jov2dzTWcLXJCG4qkpPV9GE2lV1wspLj6RFnFqR1mnI7gsRviKDB052peJ8J1QCeWhQAzv/tlO/Of2UyVfMxCV+jZz+GCP3AuITPSisKPMc8q5jjIXyk8c6oeOgM2LMl0nLEY9qh2mLEe50I7tbLOywaV0rZpqV/PCxdLFw2zGCv75+hUwyBci5XS9ADJC2agn2IuMwL68rQaXLq3GK6dHAGiPr87lbWsbsPcr1xQU4LPJeB3lD8mN6q8BUAHggwD+ddpWVSa8G4XHJjm/Osp3lHlWak2ju2Cf5aFA/rYQp8ljVaIXvDhP7Ty31TlABByThTKPXTRXWLM6X3SPhVVCOVjU+TjUmykO7BM5ZYHgXOA8xpiXf8EYGwNw/iyu56whHM+I165xtOocLxF54AiQKWSbSPwiGM0IZUD6nCrmKPPPDiLglEooH+zxIs2AF8bRsz9XKLdU2jQvIAoNHAGgTJLjE/Z8kQT0uoxQ4p9/Pd4I1jS5lf7KnCaPNUsoDxZovzbbrGxwIZlm6BgJT3kvX+6yT9ZRngyLqu340KWLUO0wKRd6E6XRY4WOJJe8WDSFiPCFt60Af8h4R2ZrXajNBcYrlPlv5G0AfiPn2ubWjEFktrHcViN0OkKFLX+MNXeUVze6MBqOa/aUzC00UMO3rqTYRX5Q3WYyYGGlTalUbu/zw6TX4aoVtegYCSGdZkjLw06aK2xYXO2AP5osmJcGspt39/unNncnEAjmJDoiquBfEFElSne9ECDj3AJTm1POGjgii8GJFPTx7LRDFilua/Hpfu19flTYjFhe58xylA/1SJ8tB7q9Sv9ivr5DOa3fArEkHGZ19EKKQuReQHBHWWtrv8phRnOFFXs6pW17XyQBl8WgCCWbyaBcQHDnVE1TRXYv5aGg9Bk8Fx1lzlSL+AsWVcJpNky4EG+q+cL1K7D9s1eUnb826nVocFsLFvKpWd3oxq2bW9BSaR33yO25ynhXv5uI/gJJKD9FRE4U6Y08W/D+jjxkXqFqhM7p9UVRZTeh0WMFY/mOMwDNQgNOc4UVYfmKXMky51wtLa/PdL440ufH0loHltY5EU2kMRCIYjAQQyLFFEcZQNH4xaFev3Kl3+/TjosIBIJ5xd0AXiGibxLRtwDshDRJb14SjCVxcjBQ9DEHu32axkYuPKMMYFzDn8ZL9mQ+7ihPJHohrctuzhQEFouGHOnzY2WDC8vqnDih+t0c7PHBpNeBMeClk5mR1P/+1DG846cvZznqgWgiyz1sVgrSsy8gookUiABzAUGzaWEFdnWMgTGWNb6aw4XTRVpCWXaU+a6p4ijPse4Gi6rtiuCf6oiE22rEK1+8Cu/a0DylzztRiAh28+Sut89rdmNRtX1cj/3WLWvxxJ1bJ/V6c4HxCuUPA/g8gM2MsTAAI4C/nbZVlYk3EodeR8rWVqVdy1GOoMFjUbaLtOIXxaIX6hPNsOIoZ19dLa93oWM4hGgihfa+AFY2uLBInsDz5nBIOXk3V1ixRK5gLdYi7lCPD5csrYaOpPULBIL5DWPs1wDeBWAAQD+AdzLGfjO7q5o+/uflN3Hjf7ycNfVUTZ8vgpvueQl/2NNd8rlC8al3lJOpNBIppirm4xnlyUQvTAUd5WQqjWP9AaxqcGFZrQPdYxElHnGox4erV9fBZTHgRXkkdTSRwu/2dCORYkp7U0AroyxlTHMvICLxFKxGfUGncWNrJQYDMXSNRuCPJvOFssMEvY6wqbUi79gmjxWxZFr5LD49FILTYiiacZ0N9DpSaoxqpmE6HM+mn+384L3r8cNb14/rsWo9djYzXqF8EYBjjDEvEX0A0tjT/PE+swwfNsL/sVfa8h3lPl8U9S6r4gLnDh0JxaQG2QUzyqoWcfzY3PD5ynon0kwaWzkcjGFlgxOt1dIJqmM4rJy8mytsaPRI2xKFWsQFogm8ORzC+hY3apxmkVEWCM4R5IjbQ5CGhgSJaMEsL2naGArEEEmk0DmqfR4cCsTAGLCn06v5fTUh2bmtdpg0W6GVQ1QW8LzPsBK9mEBGORRLQkeZHLCnSPSiYySEWDKNlQ0uLK11gDHg1FAQw8EY+nxRrG/24NJl1dhxfBiMMTx1uF9xp3kbN8YYgrFkVtcLt9UIlyW/l3IkkdLMJ3M2LZQE8K4zo1L0IkcoL6524ILWyqzX4jTJ4rxnLIJ0muG5Y4PYuqxmytqvTSWr5IEncy0/PZewGPVld984WxmvUP4ZgDARrQPwaUjjTX89basqE28kobSsAaSr3Nw+yn2+KBrcFsUFHs75fqGpfJzMFXkEI8E4PDZj3qQenkP60z5peuyqBhca3ZIg7hjJOMpNHiv0OsKiKntBR/lIr5RHW93kRr1bGgcqEAjmN0R0ExGdAPAmgO0AOgA8MauLmkYCcq74VIEIGheUh3tL+zM8etBW55wyRzmak+HljuqEHOVYEnZzJtsrRS/iWT33OYfl8/5K2VEGpBZxPIO8usmFrctq0O+P4sRgEA+83qnkQIfkIQ/heAqpNFMy0RytFnERVaxEi7Y6J5wWA3adGYNfI3rx3Xefh1/evknz2EaPFGPo8Uawv9uLoUAMb11VW/C1ZpPNrZWwGHVo9sy9FmWC2WO8Qjkpj1O9GcBPGGP3AJjdVLoGPtlR5lTaTBhTnYj4sBEpeqHtKJdqXeO2GuG0GNDjjWA4GNMcBbmwSso6PXW4H4B0stPpCK1VNjl6EUG1w6RUeBZrEXdQPjGuaXSjwWUR0/kEgnODbwK4EMBxxtgiAFcBeHV2lzR98FhCoZ7yPAt8tC9QMJ6hPFcsWyhPRS/lXKFsM+mh11HJgSG563KqtqE9NiPSDAjKwv7kYKb7UXtfAEY9YWmtAwur7DDoKEsor2lyY2tbDQDgvp0dePX0KG7d3AIg4yjz34MzTyjnDx2JqoapaKHXETYsqMDujjHNjLLFqIfNpL3FzkVnz1gETx8ZgF5HuGL53BTK7zi/CTs/f1WW4SYQjFcoB4joC5Dawj1GRDpIOeU5hTcSz5o/Xmk3Ic0ybgSPLTS4LfDI7eNyM8rDBcZXq+Et4kaCcc1G2nodYVmtVLxX77Io0YzWKjs6ZKHMt6MASSh3joaVEaFqDvf6Ue+yoMZpRr177gjlUCyJX738puaaBQLBpEkwxkYgdb/QMcaeB6Bt2c0DuKgrZBjwc3g8lcbxgeJFf+G41MattcqGSCKVt6tYDrlCmYjgshgmVMwXkh1lDhebvnACf9zbjbd+fzv+a4c0pby9z4+ltU6YDDqYDDq0VttxYjCAQz1+tFbZ4LIY0eixYmmtA799rRN6HeGjb1kCICOU+TS/3DiEunMTh2eUi7FpYQWODwbgDcfzhHIxXFYDHGbJXHr6yAAuaK3M+pyeSxBR2T2GBfOX8Qrl9wKIQeqn3A+gGcD3pm1VZeLNdZT5GGu5s0WflwtlK3Q6QqXdnDd0ZDzN0PmJZjgUKzhxZoUcv1jZkDHeF1XbcWYkjM7RsFIUCEj5rmSaKX0q1Rzq8WFNkzzVx21BIJYse5zpVPLYwT58/ZEjeGR/72wvRSCYj3iJyAFgB4DfEtGPAJSedX+WkhHKBRxl1TmvVPwiGJPGNvOYXFeJ+MVwMFZSTEcTckZZJSZdVmNZ0QsOF4sD/ii+++QxEAHf/8txtPf55Y4Xmc+OpTUOnBgM4mCPD6ub3Mr9W5dJrvJVK2rR5LHCYzMqQpkPQ8l1lFsqrXkXEKUyygCwsbUCjAFphgkJZSJCk8eKnaeGcWIwiKtX1Y37WIFgLjAuoSyL498CcBPR2wFE5arsOYUvnJNR5mOs5RNCn9wxgjdQr3aYMBzMzyjr5R7MhWiW+0IOB2IFZ6YvV4Rypjdja7Ud8VQ6XygXaBEXjidxaiiI1Y3urHUPzIGcMm+Gf/9rnbO8EoFgXnIzgDCATwJ4ElJdyI2zuqJpJBO9CGlGJXyRBAw6gtNiUOJohQjHkrCZDGiuzBReF+POB/fizgf3Fn1Mps9w5iOzzmXBK6dGsoZpFEMqrMt3lL//9HH0+aL42W0b4LIa8Y+/3YOhQAyrVJ8dy+oc6BgOoccbwVqVUOZZ3w9cuBCA1NN/WM4oB7hQNudnlIHsjiCRRBqWEl0o1rd4oNdl8tUToanCiuMD0kWQEMqCs43xjrD+KwCvA3gPgL8C8BoRvXs6FzZREqk0ArEkPNbs6AUApS0Nj17wHolVDlN+RjkQQ6XdpJwQtGiusCIQS8IfTRacOMNPcqsaVUK5KtN7sDkreiEVa+Tm89r7/EgzKCdGPtVnLvRS5kJ515mxkluhAoFgYjDGQoyxNGMsyRi7jzH2YzmKMS8JxJIgkgSxlrvLC8hWN7pwUB64UYhgLAWbSa8pCLXoGo1gV8cYkkViZDx6oXZdv3rjKkTiKXzwl6/lfY5oEYolYTdlZ5QBYOepEVy9qg7XrWnAd9+9VpngqhbKS2sd4DV/axozQvniJdV48XNXKHllyfyRM8qKo5wbvcgfOhKNp2A1FpcDNpMBq+XPs9yuF6Vo8kivuaLeiZZKUSgnOLsYb/TiS5B6KP8NY+yvAVwA4CvTt6yJw9v0eFSOckWeoywNG+E5s2qHOa/PcrGpfBy1G1zIUb5wcRV+dOt6XLu6XrmPO8e5z+G2GlHnMucJTj6BabUqeiH9HLPbS5kxhva+AK5dXQeTXlfQVU6k0lNSSCMQCOY3wWhS6e6g1fmCtyRb2+RGe5+/aG1EOC5FLxxmAypsxpKO8khQak2nnn6XC+9hrO4MsbrRjV/evhk9YxHc/qs3SkbigtFkVgcKHhPU6wifv34FAODKFXV4/5YFMOl1WSbLUvl3A0CJ4nHUwlP9mZbJKOcX8wG5jnLp6AUAbJTbxE3UUW6UhfI1wk0WnIWMVyjrGGODqq9HJnDsjODVEMp8WtCIKnrR4MlM3Kmym/OK+YpN5eOo3eDcYSMcnY5w8/qmrNZxtU4zbPL2VotKKANShXauUG7v86PSblKc5DrFUZ7d6EW/PwpfJIFLllbjujX1+P2ebuWDhHNyMIhL/vU5fOPRI7O0SoHg3IOIriOiY0R0kog+r/H9jxHRQSLaR0QvEdEq1fe+IB93jIiunak1p9IMkUQK5zV7AGjnlP3RJFxWI9Y0uRFPpnGyiKgNxZKwyXGD5gobukYLGwvRRAoh+dy1v6twj2beRzm3hdoFiyrxsw9swJE+P+56rD3v5zozkhH9PDvNcduMsBr1+MCWBcrgKQD45s1r8Myn3pJV8LakxgEiKV9crBBOil7wYj7tjLLTYoQn5wIiUqLrBefSpdUAMjHA8dJWJ63/ujUNEzpOIJgLjFfsPklETxHR7UR0O4DHADw+fcuaOLzZuvpK12rSo9phwmMH+hCKJdEvDxvhVDlMCMaSyrYaUHwqH0ftBhcq5tOCiLBQjl805fRpXF7nxMnBYNaI1vY+P1bUO5W+mxajHpV2U9m9lOPJtGbPzonCYxcrG1x4/5YFCESTePRApqivazSMD/ziNQwGYrhvZwdOiGiGQDAhiOjO8dyX8309gHsAXA9gFYD3qYWwzP2MsbWMsfWQRmJ/Xz52FYBbAawGcB2An8rPN+3wQr7ldc6Cw5d8EWkU8xo5hlYspxyKp+CQx0Q3V1iLOsrqHcV9xYRyPD+jzLlyRR1uv7gVD+3qwrH+zLnum48ewVV3b8eAP6oM/+DjqwHAbNDjL5/ciq+8PftPpNcRFlRlfz5YjHosr3Ni08LKgmsEJOMmIH+mBaIJECEr7sGRfi+ZC4hovHgf5czPWovtn71ciQuOlytX1OKFz1ye5ZILBGcL4y3m+yyAewGcJ9/uZYz9c7FjiMhCRK8T0X4iOkxEX5/8cgvji0gnvNyr7e+9ex2ODQTw8fv3oMcbyboS5m4wP1mm0wzD43CU3Vaj4gxUTUAoA1L8otphzrt6b6uT2snx/papNMOxgUBWMSAg5ZTLcZTD8SSu+cF2fP2RwwUf8/qbo/j0Q/vz3OFc2vukD4Pl9U5sWVSJJTV2/Of2U/jfV8/ghWODuO0XryGaTOGBv78QdrMB33ni6ITXKxCc4/yNxn23lzjmAgAnGWOnGWNxAA9CKgpUYIypA752APzK+WYADzLGYoyxNwGclJ9v2uFC2W01YnG1Hac03OKAnFFeVGWH3aRX+glrwYv5gIwgLBQBG5V3FI16Ki6Uk/kZZTX/dOVSOMwGfOcJyVV+8cQQ/mdnB5JphtffHEU0kUaaAQ5zdmShpdIGg358ftVvPrwF37h5ddHHcONmOBiDP5qEw6Q9NrnZYysreqE2eyZCuccJBHOBcccnGGO/Z4x9Sr79cRyHxABcyRhbB2A9gOuI6MJyF1oK7ih7crJTV6yoxbduWYMXjg0hEE3mRS+AzNARXySBRIqVzCjzdjdA4YxyIT59dRt+/L78OeltcpeMY7L7+uZwCNFEOl8ouy1ZY6x3HB8aV4uinzx3Eh0jYfx+T09BIfzU4X78fk83PvO7/UWzxe19fjRXWOGySOPCP3PNcvT7ovjynw7h9l+9gZFgDPf97QW4aEkV7rhiKZ47OoiXTw6XXKNAcK5DRO8jokcALCKibarb8wBGSxzeBKBL9XW3fF/ua3yciE5BcpQ/MZFjpwNedOawGKThS4UcZasROh1hdaO7qFBWRxxaKm2IJdN53Y04w3J70C2LqnB8IIBQTLsvslZGWY3HZsI/XbkMLxwbwmMH+vDZhw9gaa0DVqMeu8+MKRcDDnP5Jn2N06w5IlpNRijH87psqOFOO2MMiVQayTQbl1AWCM5FigplIgoQkV/jFiCioqXHTIJbA0b5Nm2VXYpQ1pio874LFuCOK5YCAFpU+WIucnlOeTw9lDnNFVaY9Lq81julWFzjwMVLqvPu54Usx+Wtu6P9PN6QPQCx3m1RoheHe3346/9+Hb96qaPoa54cDOLnL57GygYXgrEknm4f0Hxcvz8KHQGPHejDj549UfD52vv8WQL++rUNOPT1a/Hy56/EfR+6AE/+v61Y1yLlDf/m4lY0eay467H2KYl9CATznJ0A7gZwVP4vv30awJTkhhlj9zDGlgD4ZwBfnujxRPQRItpFRLuGhoYmvZ5gTDp3280GLKlxoHM0nDV9jzEGfzQzDW5NkxtH+vyaXSoYYwjHU0otiFaHBzXcUb5yRS3SDFkCnEcmgEwf5WLxhL++eCGaK6y444E9GA7G8MP3rse6Fjd2nRnNCOUCwnWqqJY/u4YDMQSiiYLCuqXShmhCuoDgre/Gk1EWCM5FigplxpiTMebSuDkZYyXDRkSkJ6J9AAYBPM0Ye22qFp6LNyLlsQqdGD59TRv+98NbcM3qTNWtepsKGN9UPs5FS6qwZXGlkh+eLHazAc0VVhyXtx3b+/ww6Cir2hkAGlwWjIbiiCZSSreJ3Z1jBZ+XMYavbjsEi1GP+z60GY1uC/6wp1vzsQO+KC5YVIl3bWjGD585kZU75kQTKbw5HMLK+mwBz132t7TVZFVhW4x6fO665TjS58ezRwdzn04gEKhgjJ1hjL3AGLsIQAcAI2NsO4B2ANaiBwM9AFpUXzfL9xXiQQC3TPRYxti9jLFNjLFNNTU1JZZUGl505jBLjnIqzdA5mnGVI4kUEikGl3xuX9fiRjSRxgENVzmWlNxRPtiDt+T8+/t24WvbDuc50Xzg1JUrpH7EPH7x8slhXPidZ/GSvBMWSaRg0uuKtg01G/T4/PUrwBhw51XLsKbJjU0LK9HeF8CgbG5o5YWnEh4nHA7GEIgWdpQXyOfoU0NBVf5aCGWBQItp7VzBGEvJRSPNAC4gojW5j5kqd2IsFIfLYix4IiMiXLqsGmZD5mRQlZNRnoij/HeXLcZvPryl7PVqsbzOqTjK7X0BLKlxZK0XyLSIe3M4hD/vk4Tsvs6xgm7tYwf78PLJEXz22uWodVrwjsxndPMAACAASURBVA1N2HF8CIOB/Jxzny+KBrcV337nGmxaWIHPPLw/bwrW8YEA0gx5kZBiXLu6HkSlJ2qVwzNHBvDiicm7WgLBXIKI/h7A7wD8l3xXM4A/lTjsDQDLiGgREZkgFedty3neZaovbwDAt462AbiViMxEtAjAMki986cd7rY6LQYsrs5vEcfHRHNH+YoVtbAa9Xh4VxdyCcuizy67o4trHPjV327GhYurcP/rnbjxJy+hVzUgZCQUh1FPWFhlw4JKG/Z3e5FKM3zz0SNgDMqQjGgiBXOJPsMA8PbzGvHMp96CO66UdjA3tVYglWZK9MwxwR3IiaI2fwI57ejUbFhYAb2OsPPkcMZRFkJZINBkRlq8Mca8AJ6HVE2d+70pcSdGw3FUTXBGu81kgNWoVzLKQxNwlKeDtnonTg0FEU+m5XiDM+8xXCjfu+M0grEkbt3cAn80qZnrA4A/7OnBgkobbtsiTW56x/nNSDNg275stzidZhgMRFHnssBs0ONnH9gIj9WEj/5md9YAgKNyId9EhLLFqEej24qOAmssl2Qqjc/9/gC++MeDol+zYL7xcQCXAPADAGPsBIDaYgcwxpIA7gDwFCQH+iHG2GEi+gYR3SQ/7A65uHofgE9BLhpkjB0G8BCAI5AmAX6cMVa8qneKCOY4ykD28CWf3PrTZZVEn8tixA3nNWDbvt68TDH/Wj0q+orltbjntg345d9sAmNAh6pl20gwjiq7GUSEdS0e7Ov04g97unFUNix65IK3WHJ8xW6A1POY7zRuWFgBImD7celifrqjFxajHk6zQZVR1t5hdVuNWN/iwfYTwyJ6IRCUYNqEMhHVEJFH/n8rgKsh5e6mhdFgXJnENxGk6XyyoxyIwWyYeO54qmircyCZZtjf7UWfL6opRnnXjj/v68GKeic+fOkiAMDeAvGLXm8EbXUOxWlfWuvAumY3/rAne1d1NBxHIsVQ75IuEmqcZvzXBzdiMBDDHffvUfKAR/r8sJn0ytbdeFlUbVcmTk0Vr3eMYjQUR9doBEf6ik/rEpw9vHhiCJu+9Qy8Ye0CrHOEmNy5AgBARAaMo8aDMfY4Y6yNMbaEMXaXfN+/MMa2yf9/J2NsNWNsPWPsClkg82Pvko9bzhh7Yhp+Jk3U+V2nxYhapxmn1Y5yNL/15/suWIBQPIVH9mdf8Ifi+UKZw+tTer2Z3bTRUOZzY32LB72+KL7zxFGsb/Fgaa0DPV4p2xwZZ/u0XFwWI5bXOZWYiNa6pppqp1l2lBMFoxcAsHVZDQ50e9En/z6EoywQaDOdjnIDgOeJ6ACkLcGnGWOPTteLqU94E6HKYcZwKCOUqx3mKcsdT5S2OslB/vM+ScSu0BDK9W4ppphm0ofFkhoHnBYD9hZobdTvl+IUat65oRlH+vxKP2QgM8SkXtU+b12LB3fdsgY7T43gH367B4P+KNr7/Fhe79RsOVSMRdVSNftUOr9PHOyH2aCDjoCnDvVP2fMKZpfdZ8YwHIxh95nC2ftzgO1E9EUAViK6GsDDAB6Z5TVNCzyjzPO7i2vsWUNH+NRVl8od3bDAg7Y6Bx54PXsqKHeUbRruqDLZVB29CMaUCN76FqlH82goji/dsBJNHit65MdGE+myheTGhRXgp72ZMGH4GGt/kYwyAFzWVg3GoBR3i4yyQKDNtAllxtgBxtj5jLHzGGNrGGPfmK7XAiRHtByhXG03YdAfxaunR3CgxzdrsQtAmr7Eu04A+R0vAGl70mk2wGLU4Zbzm6DTEda3eLBHQ1RE4il4w4ks8QsAN5wnTUfacTyT7R2Qi0349D/Oeza14Ms3rMT240O46u7t2N/txYr6iTeNX1RtRyCazIpxFOL5Y4NZOUIt0mmGJw/348oVtdjcWomnDmt38hCcffDt7mKT0s4BPg9gCMBBAB+FNOBpwh0q5hrJVBqnh4JZuwXBWBJ2k17Z9Wry2LJaYPLohdpRJiLcunkB9nf7cKQ3c8EfikkxAq0ssMUoDaDq9WVnlHlkb3WjG2aDDteursPm1ko0VViV92IkkdIcNjIeNrVWKP8/E45yld2MXm8U8WS6qDBf1+yB22rE00ekc6eIXggE2sypMdTlwhjDWNmOsglH+wO49d5XcWYkhLeuLBoDnFYsRj1aq+0YCydQ7TCh1qk9JvSCRZX44IULlQ+O8xdU4PhAQNnC5PTJHwi540arHWZU2U1ZWT3eci5XVANS4eJf5JZv0URacV4mwqJqKXtYKn7xzJEB/O2v3sAPnzle9HG7O8cwFIjhujX1uG5NPY4NBKY82iGYHbiLt697fMWf6sma8wXGWJox9nPG2HsAfATAa2weBPH7/VFcefd2PKnaAQrmFJ01uC0YDMSUuJfiKOf0yH/nhiaYDDo8+EbGVQ7HuaOsLRAb3Na86AUfGmUx6vGHf7wY//6edQCAJo8VY+EEwnFp0l25jiufpkek7XRPNdVOkzKNsFjfZb2OcOnSaqU2R0QvBAJt5oVQ9keSSKZZWUL5rza14LYtC/Cz2zZg779cgzuuXFb6oGmkrVZykYsVy/3y9s340g2ZsafnL/AgzYAD3dkOHHdlcqMXANBabc/KAQ74pB7KhYattFbb8ZsPX4BH7rgU79rQPP4fSIYL5UJFh4CUp/7M7/YDAHZ1FN92f+JgP0wGHa5cUYtrVtcDkAamzHUYY7h3xyllAqMgHy6U93d5S0Z1jg8EsPZrT827mAYRvUBELiKqBLAbwM+J6Aezva7JktuSE8geEAIADR4LUmmmdCHyyV0vXDkxAo/NhOtW12ObKqccLOIoA0Cjx6LsVkXiKYTjqazPjdWNbkVc8h7MPWORSQnl5gorap1mOEyGGYn1VTvM4E2QikUvAGBrW6anvxDKAoE280Ioj8rbeOUI5U2tlbjrHWtx/dqGaW/dMx74hL4V9fmxi0KcLw/32NtZSCjnu8SLqu15jnKN01x0nCoRYW2ze9wjV9U0V1hh0FHBzhfJVBqfeGAvEsk0bt3cgtPDIaUbSS7pNMMTh/qwdVk1nBYjmjxWnNfsznKp5iqnh0P49uNHce+O0zPyer5IAhd/59msmM1cJp1m6PNGUWU3wRdJoGOk+AXFs+2DSKQYXj09MkMrnDHc8rjpdwL4NWNsC4CrZnlNk0bdlYETjCXhUDmf/HzFz1/+aAJ2k17zvLOq0QVvOKHspimOcoEJeI0eK3q90khr3kO5usB0VT59tdsbmVRGmYiwZXEVKic4xbVcqlVmR6lJflvbMp2mLKZ5IQcEgilnXvzLGJVPeOUI5bnG8rrSjnIuHpsJi2vseZ0v+uXohVacYlG1HQP+mFL80ueLot6lHfWYCgx6HRZU2grGI/7juZPYdWYM337nWrxnk+RY71K5hMf6A/jAL17D9546igfe6ESfL4rr1jQo3792dT32dXmVosS5Cnf9n20fmJGWdod7fOj1RZXc+1xnKBhDPJXGdWukXYJ9XcWd4p2npP60xUYan6UYiKgBwF8BmLYi6Nmg2mlW3GJAEsrqLG29SxKo/N+yL5LIyierqZO79PAai8yo6AKOstuKUDwFfzSpdDuqtGvvojWpHOXJZJQB4Ks3rsIv/npT2cdPBLVQLmX+NLitylRY4SgLBNrMC6GcOeGd/UL50mXVePfGZlyxfGJZ6fNbKrC3M3urutcXRaXdpLllyKMQ3FUe8EfzCvmmmkIt4hKpNO57pQPXra7HzeubsKbJDZNBh10do8pj7nulA6+cHsF/bj+NL/3xEAw6wtUrM1MWubCa7vjF3s4xnBwMln5gAfZ3SYKu1xdFu9yTuhjecHxSGdxjA9Jr7Dw9XPZzzCTdcvHU5ctrYTPpld+XFrFkCm/I75H/396Zxzd2Vnf/e2RZ8iZb3nePZ/MsmTWZLJNJSEgmkGUgbAUCaYE3lKVAgLYvBUp5W96+nxdKgaa0BdKFsCSBBgLNnmYjCTPJTCaz7zOe1dt47Bnviyzr6R/3Xlm2JXkfW5rz/Xz0sXV1de/z6EpHR+f5nXP2Jp+j/A2sesjHjDFviMgChpqDJDQFWZ5wF1SwNMqZERHgMr9lhxyJREfvwCh9soOTx9HcYR2vp38Ql4DXHf2rrcyOEje09YYTi/NjRHqLfGm4XUJDmyW9mEqyW0GWl8XF418lnAqFvqH5jCW9AKuBiyfFpY6yosQgKRzlC1OQXsw1ctJT+fs/WE3uBOeytspPa3eA0xHa16Y4UWKntavjuDa190WNPE8njtxjZBfBLbWttPUM8N4rrEiy153C6ooc3rB1ysHBEM/sa+L2laXs/Pot/PDuK/jXP1pHTsbQl+fCwixqirN4cm/syOnAYGhUg4KJcu8vdvLXj+0fe8cY7K5rY6HdVOGFg2NX6viDH77GN544MOnzHbYbJ5w535sQumhHnzwvP4MV5TnhlsLR2HGqjb6BEFfMy6XuQm9S1V02xjxiVwz6E/v+cWPMe2d7XNNBQZY3ikZ56LOck55KWqprWEQ5tqNsRU+dTqNd/UEyvbG1wKV+R9bRGx5DrEZVKS6h1J9GfZsVUR7ZJXWuEhlRzh5DegFw782L+cUnr5mUpE5RLgWS4pPhtKDOj7GEdimwusLSKe+PKJVktaSO4SgXWMX3T5zrptdeipzpiHJ1QSZ9A6FwhQ2HJ/c04PO6uX7xUGLJuuo89je00xsY5PXjVmORO1aWkp2Wyq0rSnjr0tER902rynjj5PnwMuxIvv3sYd72vVcYsLPpwdI03vDtl3h0R92Y4+8JBDlzvpfdZ9pitgyPx8BgiP0NHbx1SRGrK/08f6h5zPMdbe6aUovuQ02dYWfitYuk4z3Q0MHgJF4fGCoNV+5PZ22lnwMNHfQHo0fUX6ttwSXwcbvpTuR7X5m7FGR5wzYbGNUYQ0Qoy0mP0CgHYzp8RdkjIsqBYLgeczQc3XF9W19ERDn290ZZjlUirn8glDDl04ZrlMeOKGd53VxelTvmfopyqZIUjvKF7gBpqa6EMWQzweJiqwZzZBORxvbecARlJBkeNyXZaZxo7R4qDTfDjvKCKCXiBgZDPLv/LBuXFw+TiFxZncvAoNWl8Mm9DWR6UrhxSfwW53esKsUYYupxNx9rob6td1gk9/HdDZxq7RmVCBkNp0pIZ39wWIvd8XK4qZNAMMTqSj8blxax+0xbuDTT3rp2vvfckWHSGed8Z873Tkp7HQoZjp7t5LYVJeRneni9duYd5ddqW7n9H1/liT0NY+8chfq2HvwZqWR63ayu9BMYDIXbpo9kc20rqyr8XLMgH0hK+UVSUpDlpa1ngIHBEMaYUVUvwMqrcMpbdsTRKGenufG6XeGIcnf/4DAZR7Rzu11CY1svrd0BPG4XmXG+N8pz0zlzoYfAYIi0BIkoZ3rdYRnFTLfMVpRLgaRwlK2i8ZduNBmsbPIFhVlh3avTbCRaaTgHRzMcrSvfTDC/cLSjvPlYC+29A9yxsnTYvk6E47XaVp7e1zTKkY7GwsIslpVmR3XS+gYGOWTLEB7edia8/aGtVg3Wxvb4DU6AYdrkWJ0Q47HbTuRbXeHnZltf/dKhZuou9PDRH2/jvheO0hDhEEc6429E6LXHS31bL92BQZaUZLN+YT5balvDjvgTexr4kwffnPaEwh++XDvp8YIVUXaifqvtai7R5Bdd/UF2n2ljw6J8cjM9lPvTkyqhT0QSwyubBAW2hra1K0DvwCAhM9qhK8lJC9slS6Mc3eETEYqyvTTbPzi7A8G4TT1SXEJJjlUirrUrQEGmJ27Jtgp/OmftaHV6AlWFKPB5SEt1kapyCkWZMknxKZps++pkY2mJj0NNVkQ5VrORSKoLMjnZ0h2WKsy0o1zsSyMt1TXMUX5yT6Mlu4io5wlWJY+a4ix+8tpJ2noG2LSqbFzn2LSqlB2n28JaV4f9De0MhgwryrN55eg5zpzvYV99O7vr2nEJwzqBxeJYcxcpLsHndQ9z3gLBED97/RSBYCjOs626wLkZqVTmpbOs1EdZThqP72ng4z/ZTpvdVOFQxIpA7bluXHaTgkjH0xgzLqfQ+WGwpMTH+oX5NHX0caKlmwvdAb722308tbcpHNGeDg41dfDykXOIjC5VOF7q24Yc5bKcNAp93qgd+t44cZ5gyHDtQut9s6I8O6kcZeCEiNwvIjfLxSi+exGJrKXc1Re9SkVZTjpnO/sJBEN09gdjRpTBsiuODevpHxyzqUdZTjoN7X20dvePWbLNqXwBidXiuSDLO2ZpOEVRxkdSOMoXugMTTn5LRpaVZlN3oZeOvoFxRYkX2F0AHYdqpqUXLpdQnT9U+SIQDPHs/iZuuaw4aqLMuuo82noGRumX47HJbs/91Aj5hVM94f/euQKAR7af4aFtp/G6Xdy+snTcjvK8/AxWV/qHOW//tauev/rtPl48NDw570cv1/LHP90e1jPvqWtnVYUfEUFEuGlZEa8ebeFocxf/dNdaYLh0pvZcF1V5GVxelRtObAT47a56Nn3/96PKAT66o45/e3WoPvMRu+JFTXFW2KF87Xgr33nuMG09lmPuVMVw+ObTh3jpcHztdCzuf+U4GZ4UPnx1FYeaOukNTKxahzHGiijbzomIsLrCz++PtXB6RD3lzcda8LhdXDHPWnlYWZ7DydYeOvoG4h7/O/99OJzgOMdZCjwPfAbLaf4nEblulsc0LTh1i8919dNpJ9eO1NKW5FhNR463WKsq8ZLSIiPK0WQcI3Gajpwfx0pkuT8j/H+iSC/AsuV5GfqdqCjTQVI4ypb0Qo3CslKr/NDhps64Xfkcqm3N8OvHW/F53XGXLKeLBYWZ4aYjLx1upqMvOEp24XBlteUE3TIO2YXDvPxMVpbnjJJf7K5rozQnjbVVudxQU8gv3jjDf+2s5x2ry1ha4uN899hl2I6d62JRYRZrKv3DHMHHbad8ZLm3p/Y28tyBs/xqRx09gSBHznayumKo/femVWWIwNc3Lee2laVU5WVwMMKJq23uYmFhFldW53GoqYN2O+r8s9dOAbBjRNT2Ry8f5++ePRze71BTJ+X+dHxpqVTnZ1CSncaDr5/moa2necdqK0If6TR29g3ww5drw8efCI3tvTy2q4H3r6vkxpoiBkNmwprhtp4BugOD4YgywCfesoDegUHu+P6rPLOvCWMMtee6+N2Rc6yblxt+X1xWbr2uB+Ik9J1s7eH7Lx7jke1nYu4zVzDG9Bhj/tMY8x5gLZANvDzLw5oWwhHlzjgRZTu3wnl/xosoF/nSOBeRzBerffXQsdM529HHuc7+Mb83hkWUEygH5iu3LeO7H1g928NQlKQgKRxllV5YLC2xmpQcauwYl/TCqaW8t76d4hmWXThU52dy+nwPn3loB5/++ZsU+bxcFyNavGFhASXZaXzwqqoJnWPTqlJ217UPi0LuPtMWrgxy11VVNHf20x0Y5ENXV1GSM7zBQTQGBkOcbOlmUVEWqyv9DIYM+xraae3qZ/Mxq0axI3sBq6SdE6n/1tOHeK22lZAZ0t0CXLMgn51/dQsfubYasKQzTkR5MGQ40dLNgsJMrqzOxRjYcfoCBxs7wg7y3oiW5VaFDCtZ8Jl9luN+pKkz3OFRRLh2YT4HGjvwZ3j42ztXUJDlCUedYahqxK4RraNPt/bwhV/sDCdMRePHm09igHuum8+aKqdT5MTaSjtymYoI5+Sq+Xk8de/1LCjI5FM/f5PVf/Pf3PydlznW3MXGiDraK8osR9mRX3T3B0fJSpzmJSOj6HMVEblBRP4Fq4V1GlbzkbGec6uIHBaRYyLy5SiP/6mIHBCRPSLygojMi3hsUER22bfHpnUyEQxJLwIxG4Q4TUecz1Cs8nBgRZQ7+4P0BIJ09Q+O+YO/1J/OwKChsb0vZg3l8L4RdjEtRm3muUhVfgaXleWMvaOiKGOSOJ/8GPQNDNITGFRHGcuo56SnctCOKOdmpMaNxFblZeASyymbadmFw6KiLIIhwyuHz/HHb1nA45+7LmZ90qLsNF7/6s1cNT9vQufYtNqK1D7yphU5bOsJcLK1J+yk3rS0iCKfl2Wl2ayt9FM2omVuNE61dhMMGRYVWRFlgF2n23hqXxOD9vZDEdHZk63d9AdDfGxDNed7AvzFr/cCsKrCP+y4/ojl0WWl2Zxsscr1NbT10h8MsbAwizVVftwu4Y0T53lo62k8bhfr5uUOi9juq+8gZKxkpd/srCcQDFF7rivcEh3g2kXWD5IvvX0JORmp1BT7OHy2K+IY1vHOj6jH/Zud9fx2VwP3PLA93CI4kvaeAR7aeprbV5ZSmZdBQZaXqryMCeuU68Kl4TKGba/My+CRT13LFzYu5vaVpXzrvSt57otv4WMbqsP7FPq8lGSnsa++nUNNHbzte6/w7n/ZPMzh32WPJxGkFyJyEvgC8Cqw0hjzfmPMr8d4Tgrwz8BtwHLgLhFZPmK3ncA6Y8wq4FfA30U81muMWWPf3jlNUxmFU5WhtaufTieinDa1iDJYJeKs8nDxI7/lEZWAYnXlc0hLTQk79pdyVSVFuZRJ+NoxTi1MdZStqOHSEh+H7KhhSRzZBYDH7aIiN4PT53tmvIayw6ZVZWR4UtiwqGDGkk3K/encvLSIh7ed4XM3LWZ3neUArq60IiypKS5+/vGr8bpdiEhYxx2v8oVT8WJRURaFPi/l/nR22eXdFhdlsWlVGd97/gjddsMDJzr7/nWVBIIhHtx6OpycFotlpdmEjKUtPm83z1hYlEWGx81l5Tm8cvQcp1p6uGNlKfPyM7jvhaNhTaajmb776ip+8topNh9rIRgy4YgywJ1rysjLTOXGGqsGdU2xj//cfoZQyOByCfvq23G7hGDIsPN0G/PspjRbalsoyPKwv6Gdzz60k/v/8IphzQke2HKSrv4gf3LjwvC2tVV+th4fSkC8/5Vafnf4HA/98TUx5+9ElCOXux08bhdf2FgT87mAnajZwnMHztIfDBEMGY42d1Fjd0RzEjCbO/sTIa9hlTFmooWhr8Lq5HccQER+AdwJhDvWGGNeitj/deDuqQ50MhT4PFYyn6NR9g63BU7TkcPhiHLsryqnjXVTRx89gXFElCPs4lgRZbDejy1d/QmVzKcoyvQxYxFlEakUkZfsZb79IvL5mTiPOsrDWVaazaGmThraesOR0ng4OuWSnItTXs/jdnHritIZz8i++5p5tHT18/S+RnafaUPESvhyqCn2hR1B54szXkTZcZQXFmYBsKbKSjLbdvI871xdFtaHO1KGg42dpKYICwuz+PO3LSE3I5UrquNHxp1jHGzsoHbE+a6qzmVffQed/UE+fHUVqypyMGZIk7u7ro1yfzr3XLcAsJqrgFXxwiE1xcVNS4txuST8WE9gMOyg7mvo4PrFBWR4UsJOZW9gkJ2n23jv5RV8484VvHioma8/tj8cqe3sG+A/Np/gluXFLCvNDp9rTaWfpo4+GtutpKl/eP4oW2pbae+JnWxXf6GX9NQUcjMm995YUZ7D+e4AC4uy+OUnLYfckcX0DQxyoLEjPMYjc19+UWJLI/YBiMgqEfnaGM8pByIF2HX2tljcAzwdcT9NRLaLyOsi8q5YTxKRT9j7bT93bnLNcKzufIFwp8yRtY9FhNKc9PB7czwRZSf3IV4dZRhqYw2xu/JFUmHvry2eFeXSZCalF0Hgz4wxy4FrgM9EWQacMuooD2ep7fwcOds5rnJvC8KOcvzoc6LxlsWFVOdn8PPXT7H7TBsLC7NiOufpnhT8GaljRpTLctLC0aq1lf5w0tym1WVhB8yRXxxo7GBxkQ+P20VupofHP3cd33jnZXHHXJmbQaYnhUNNndSe6yY3IzX8vl5nO9k1xVlcMS+XFbbTv8fWKVsVNXKoys9g3bxcDjR24HYJCwqyYp7PibQebuqk226isqrCz6qKnLC+ePup8wQGQ6xfmM/d18zj0zcu5KGtp/mnF48B8PPXT9PeO8Bn37po2LHX2nWwd55u40ev1NJjJz4eaY7toNa39VCemx63rm087r5mHl+7Yxm//MR6rpiXx7z8jLCjfKCxg4FBwwfWWW3SYznKgWAo7HDNMv8KfAUYADDG7AE+OF0HF5G7gXXAtyM2zzPGrAM+BPyDiCyM9lxjzP3GmHXGmHWFhfGbAMXCaWMd1ihHaYwRqQ+OW/XCXqU50eo4yvEjytlp7rA8I15XPgdnhSMtNeGVioqiTIIZ++QbYxqNMTvs/zuBg8SPbkwKdZSH4zhsIRM/kc+hOt/Sg14sjfLFwuUS7r5mHm+cvMDm2pZwIl8sSnPS4ybzHTvXxcKiIafT0SmvLM9hfkEm5f50srzucB3kAw0dwyKsFbkZYy71u1zCkhIfBxo7qD3XFY4mA1xVnUemJ4V7rptvNVnwpVGc7WVffXtYU+xosN+11vqYLSjMxBMnAamm2Dr+4bOdHGzswBhrPmurctnf0EHfwCBbaltxu4QrbUf9S29fwnvWlvOd547wwOYT/Nurx3lLTeGwJEWA5aXZeNwunj9wlp9uOcXVts48nj44sobyZCjI8vLx6xeEtaTXLizg9ePnCQ6Gwvrkt68oITvNPSqhbzBk+M3OOjZ+92U++uNtw9qczxIZxphtI7aNFogPpx6ojLhfYW8bhohsBP4SeKcxJpzxaIypt/8eB36HVW1jRnAc5c6+IJ4UV9Q8BeeHvtslcWsj+zNS8aS4OGF3sozXwhrsFtn2+2w8EeVK21G+GFWBFEWZe1yUn8giUo1ldLdGeWxKy3ittqOs5eEsaop9OAG5eKXhHC6fl4snxcWSYt+Y+yYa77uiAq/bRd9AiDWV8TPAS3PSaGiL7iiHQoZjzV0sLhp6jVaU51CQ5eEuuyKH4+QebOqkubOPlq5+lpdlRz1ePJaVZnPIll5EOsq5mR62f+0WPnDlUAWQleV+9ta3h6PKq+zSc3esLCU1RVhSEv/8vrRUyv3pHDnbxTR15wAAETBJREFUGU7kW1Gew5pKP8GQYX9DO1tqW1lT6Q87CSLCt963iusXF/DXjx+gtTvAvTctGnVsj9vFirJsHt1ZT39wkP//npVked1xJQ/1F3qHVbyYKtctKrA6+NW1s7uujeJsL6U56Swp8Q1z2I+f6+L2+17li7/cTZbXzf9552W4XbPe46PFjugaABF5HxC9N/sQbwCLRWS+iHiwItDDqleIyFrgR1hOcnPE9lwR8dr/FwAbiNA2TzeFWR7Odwdo7x2I2Wa5zLZf2empcVcZRIRCnzdcn308Dm3YUR6HRvk9l1fw/bvWjsueKoqSfMy4oywiWcCvgS9ES06Z6jLehe4AKS6JuzR3KZHuSWF+WHs7dpR4VYWfA994O1X5GWPum2j4MzzcucaqFzwy4jmSkpw0mjqGHOXG9l6++9wRuvqD1Lf10jcQYlFERDktNYVtX93IXVcNBfCW2ImUTj3l5aUTd5SXlmbT0RektTvAwqLMYY+NzLpfWZ7D8ZZuXqttHabBzs308MO7r+DzNy8e83w1xVkcOdvF3voOCrK8FGd7WWu/Vq8caWFvXRvXLswf9pzUFBc/uPsK1s3LZeOy4rAsZCSO/OLdaytYUJhFTXFWzIhyTyDIhZ6BqIl8k2W9Pe4tx1rYdaYtvApQU2w5yo7O+ge/q6XuQg/fv2stT3zuOt66pGjS8o9p5DNYDu1SEanHqoDx6XhPMMYEgc8Cz2Kt4P2nMWa/iHxDRJwqFt8GsoBHRpSBWwZsF5HdwEvAN40xM+YoF/i8hAycOd8Ts0GIE1GOp092KMr2csqu1DJW1Quw5BQZnpQxay6D5Xg7dccVRbn0mNG1JBFJxXKSHzTGPDoT52jtDpCbkRpOUFKsqOTxlm5Kx7mMHVnBINn4wsYaSnPSx6wpWpaTFm46kpaawk+2nOKHL9fy8uFmPmqXIYt0lIFR77llJT4e2hrkxYNWh77JOMrLS4ei1pER5WisrMjGGPjVm3WjNNg3R9QYjkdNiY/Nx1oZGAyxojzbknVkp1HuT+enr50kZGD9wtF1rrO8bh751Hoiqq+N4qalRfxmZz333mxFnJeU+MJNQyIdUWMMD2+zctAqcqfvB1tepofLyrJ5cm8jp1p7+KAdjV9a4uPBrUHOdvSTm5nKM/ubuHVF6Zxyhmz5w0YRyQRctnxtPM97CnhqxLavR/y/McbztgArJz/iieF0xDvZ2h0zyOH80M+OEXGOpNiXxs6gtbKSMY6I8qdvWMjblo/vM6IoyqXNjDnKYn0T/jtw0Bjz3Zk6z/nuftUnj2BtlZ+XDjePK6Kc7JT50/niLfHLigHDmo5UF2TyxsnzlGSncaipk//9yB5gtKM8kqW2Y/zY7gbK/enkTKJ6Q6RcYixH2Unoa+0OcMOSySVVLSn2ERgMcay5i9tWlIS3r6ny8+SeRrxuF2urokfjrVbcsY+9YVEBb35tY9gprin28fC2M5zr6g9XKmjvHeArj+7hqb1NXL+4gFvG6eCPlw2LCrj/Fautd2REGSxtdiAYorMvyDtWR+8OebERkT+NsR2AmbSlFxOnjXV9Wy9l1dF/0JdGSC/Goih7KCkva4yqF2DV5q7MS75VNEVRpp+ZDCVuAP4QuCmi29Pt032SC90D5GpP+2H80fpqnvvTG7Tu5wRwSuk1tPfSNzDInro27lxTxs/uuZp0u+nAWD/InFJsF3oGhiXyTYQsr5uqvAxSU2RMvW6RLy2chLlmDGlJLGqKh+uuHRz5xZXVeVN6H0VGjh0d/JEmq/RdcDDE+36whWf3n+Uvbl3KTz521bQ3ddhgN1lxyZCGuyY8jk4e391AbkZqeL85gM++rcOSWpTbt08Bl8/iuKaVArtShTHgixEBDkeUx+MoR9QnH4+cQlEUZbzMmEUxxvwemHE9RGt3/7BasYqVSDWV6gGXIo4esqm9j11n2hgYNFxZncdV8/N48t7r6eiLXf/XIdtOjqtv6x0moZgo6+blUtviGZckZkV5Dk0dfaM6/o2XRUVZuMSqkjLMUbb1xetH6JOngtMl8PDZTq5bXMC2E+c52tzFd/5gNe+9omLazhPJldW54XrWTpJXbqaHIp+XXWfaePFQM+++vJzUOSI/Msb8DYCIvAJc7kguROSvgSdncWjTSkFEWbZYyXz+jFTSU1PwjyuiPLR6ptUpFEWZThLeopzvDmhEWZkykU1H6u1WyuuqLWdxIomOy0p9lqM8iYoXDn/77hUMDMYR/0awYVE+O05fCDcrmShpqSlU52fS1jswrEHN2ko/X7tjGe+9fPoc2IIsL/mZHo7YCX2P72kkw5PC7StnTvaQ4XHzsQ3zR0Xnl5T4eHpfIyED71g1d7TJERQDgYj7AXtbUpCd5saT4iIwGIqZzCci3PfBNWNKnmB4RHk8yXyKoijjJaEd5cGQoa13QEvDKVMmsunIqdYelhT78E/iB9jSkmyeP9g8aekFTGzp+CPrq3n/usqodWjHy7vWltMdCA6TSbhcwsevXzDpY8aiptjH4bOdDAyGeGZfIzcvK552ucVIvnr7sqjjePVoC0U+L1fNj98xcZb4KbBNRH5j338X8MDsDWd6EREKsjw0tPfFdJQB3nZZSczHInE07163K6mTkxVFufgktKPc1hPAGG02okwPpTnp1F3oZcepC7z78sn1xvnQ1VXkZ3moukiJQi6XTHmp+d5xlJGbLpaU+Hhk+xk2H2vhQs8Am1bNThKdo5e+Y1UpKXOwYo4x5v+JyNPA9famjxljds7mmKabAp93TEd5vBTbyXwqu1AUZbpJaKvidOUbq+OZooyH0pw0ttS20DcQCneimyhl/nQ+tmH+NI8seagp9tEdGOT+V47j87q5oWZy1TqmytUL8ijO9vL+dZVj7zxL2J1Nd8z2OGYKR6ccS6M8EXIzPLhdQuY4Kl4oiqJMhKRwlJ2anIoyFUpy0ugbsFoXz9Hl+IRnSYmlN91S28p71pbPWmWWefmZbP1q1JLCykXCKRE3HRFll8vqzjdW+2pFUZSJktBWxXGUVXqhTAdOMltFbrq2q50hFkeUo7tjlmQXytzAiSj7piGiDFbliznQelxRlCQjoR3lefmZfPKGBdpYQ5kWnKYjV01SdqGMTXZaKqU5aXT3B7l+8ezILpS5Qb4jvfBOvDFPNP7i7UsIja9YjKIoyrhJaEd5eVn2lMpwKUokTkT5SpVdzCgfvbaa1BQXHrdWJ7iUKfRNn0YZ4Nq50zRGUZQkIqEdZUWZTtZV5/HFjTWzVonhUuGTNyyc7SEoc4CblhbxZ7fUsEKDHYqizGHUUVYUG4/bxec3XrxSaYpyKZPldfO5i1iaUFEUZTLo2qeiKIqiKIqiREEdZUVRFEVRFEWJgjrKiqIoiqIoihIFMWbu1NMRkXPAqQk+rQBomYHhzBV0fomNzi+xmcj85hljLqmad5O02aDvm0QmmecGOr9EZ9pt9pxylCeDiGw3xqyb7XHMFDq/xEbnl9gk+/xmi2R/XZN5fsk8N9D5JTozMT+VXiiKoiiKoihKFNRRVhRFURRFUZQoJIOjfP9sD2CG0fklNjq/xCbZ5zdbJPvrmszzS+a5gc4v0Zn2+SW8RllRFEVRFEVRZoJkiCgriqIoiqIoyrST0I6yiNwqIodF5JiIfHm2xzNVRKRSRF4SkQMisl9EPm9vzxOR50TkqP03d7bHOllEJEVEdorIE/b9+SKy1b6GvxQRz2yPcSqIiF9EfiUih0TkoIisT5brJyJftN+X+0TkYRFJS/TrJyL/ISLNIrIvYlvU6yUW/2jPdY+IXD57I09M1GYnJslst5PZZkPy2e3ZsNkJ6yiLSArwz8BtwHLgLhFZPrujmjJB4M+MMcuBa4DP2HP6MvCCMWYx8IJ9P1H5PHAw4v63gO8ZYxYBF4B7ZmVU08d9wDPGmKXAaqy5Jvz1E5Fy4F5gnTFmBZACfJDEv34PALeO2Bbret0GLLZvnwB+cJHGmBSozU5oktluJ6XNhqS12w9wsW22MSYhb8B64NmI+18BvjLb45rmOf4XcAtwGCi1t5UCh2d7bJOcT4X9Jr4JeAIQrMLg7mjXNNFuQA5wAlv7H7E94a8fUA6cAfIAt3393p4M1w+oBvaNdb2AHwF3RdtPb+N6ndVmJ+Atme12Mttse+xJabcvts1O2IgyQ28Ahzp7W1IgItXAWmArUGyMabQfagKKZ2lYU+UfgC8BIft+PtBmjAna9xP9Gs4HzgE/tpcp/01EMkmC62eMqQf+HjgNNALtwJsk1/VziHW9ktrmXASS+vVLUpsNyW23k9ZmwyVlt2fUZieyo5y0iEgW8GvgC8aYjsjHjPWzKOFKlYjIJqDZGPPmbI9lBnEDlwM/MMasBboZsWSXwNcvF7gT64ulDMhk9PJX0pGo10u5uCSjzYZLwm4nrc2GS9Nuz8T1SmRHuR6ojLhfYW9LaEQkFcvgPmiMedTefFZESu3HS4Hm2RrfFNgAvFNETgK/wFrGuw/wi4jb3ifRr2EdUGeM2Wrf/xWWEU6G67cROGGMOWeMGQAexbqmyXT9HGJdr6S0OReRpHz9kthmQ/Lb7WS22XDp2O0ZtdmJ7Ci/ASy2szc9WAL1x2Z5TFNCRAT4d+CgMea7EQ89BnzE/v8jWDq4hMIY8xVjTIUxphrrWr1ojPkw8BLwPnu3hJybgzGmCTgjIkvsTTcDB0iC64e1dHeNiGTY71Nnbklz/SKIdb0eA/7IzqS+BmiPWO5TxkZtdoKR7HY7yW02XDp2e2Zt9myLsqco6L4dOALUAn852+OZhvlch7VksAfYZd9ux9KEvQAcBZ4H8mZ7rFOc543AE/b/C4BtwDHgEcA72+Ob4tzWANvta/hbIDdZrh/wN8AhYB/wM8Cb6NcPeBhLuzeAFV26J9b1wkpi+mfb3uzFyiSf9Tkk0k1tduLektVuJ7PNtueXVHZ7Nmy2duZTFEVRFEVRlCgksvRCURRFURRFUWYMdZQVRVEURVEUJQrqKCuKoiiKoihKFNRRVhRFURRFUZQoqKOsKIqiKIqiKFFQR1mZs4jIFvtvtYh8aJqP/dVo51IURVEmh9psJRnR8nDKnEdEbgT+3BizaQLPcZuhXvbRHu8yxmRNx/gURVGUIdRmK8mERpSVOYuIdNn/fhO4XkR2icgXRSRFRL4tIm+IyB4R+aS9/40i8qqIPIbVfQgR+a2IvCki+0XkE/a2bwLp9vEejDyX3cHn2yKyT0T2isgHIo79OxH5lYgcEpEH7U5HiqIoCmqzleTEPfYuijLrfJmI6IRtPNuNMVeKiBfYLCL/be97ObDCGHPCvv+/jDHnRSQdeENEfm2M+bKIfNYYsybKud6D1alpNVBgP+cV+7G1wGVAA7AZ2AD8fvqnqyiKktCozVaSBo0oK4nI27D6t+8CtmK1r1xsP7YtwuAC3Csiu4HXgcqI/WJxHfCwMWbQGHMWeBm4MuLYdcaYEFar2uppmY2iKEpyozZbSVg0oqwkIgJ8zhjz7LCNli6ue8T9jcB6Y0yPiPwOSJvCefsj/h9EPz+KoijjQW22krBoRFlJBDoBX8T9Z4FPi0gqgIjUiEhmlOflABdsg7sUuCbisQHn+SN4FfiArakrBN4CbJuWWSiKolwaqM1Wkgb9daUkAnuAQXs57gHgPqwltB12csY54F1RnvcM8CkROQgcxlrKc7gf2CMiO4wxH47Y/htgPbAbMMCXjDFNttFWFEVRxkZttpI0aHk4RVEURVEURYmCSi8URVEURVEUJQrqKCuKoiiKoihKFNRRVhRFURRFUZQoqKOsKIqiKIqiKFFQR1lRFEVRFEVRoqCOsqIoiqIoiqJEQR1lRVEURVEURYmCOsqKoiiKoiiKEoX/ARs0AnJnp2dJAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fd70ebef320>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "logreg.plot_results(losses,accuracies)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Deliverable 5.5**\n",
    "The noisy progress of the loss and dev set accuracy suggests that something is wrong with our hyperparameters. Tune the inputs to `train_model` until you can get to a dev set accuracy of at least 0.5. (0.5 points)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# build a new model with a fixed seed\n",
    "torch.manual_seed(765)\n",
    "model = logreg.build_linear(X_tr,Y_tr)\n",
    "model.add_module('softmax',torch.nn.LogSoftmax(dim=1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model_trained, losses, accuracies = logreg.train_model(loss,model,\n",
    "                                                       X_tr_var,\n",
    "                                                       Y_tr_var,\n",
    "                                                       X_dv_var=X_dv_var,\n",
    "                                                       Y_dv_var = Y_dv_var,\n",
    "                                                       num_its=100,\n",
    "                                                       optim_args={'lr':0.02})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "_, Y_hat_dv = model_trained.forward(X_dv_var).max(dim=1)\n",
    "np.save('logreg-es-dev.preds.npy', Y_hat_dv.data.numpy())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.5044444444444445\n"
     ]
    }
   ],
   "source": [
    "acc = evaluation.acc(np.load('logreg-es-dev.preds.npy'),Y_dv_var.data.numpy())\n",
    "print(acc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "_, Y_hat_te = model.forward(X_te_var).max(dim=1)\n",
    "np.save('logreg-es-test.preds.npy', Y_hat_te.data.numpy())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.48\n"
     ]
    }
   ],
   "source": [
    "# You can't run this\n",
    "acc = evaluation.acc(np.load('logreg-es-test.preds.npy'),Y_te_var.data.numpy())\n",
    "print(acc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 6. Feature analysis\n",
    "\n",
    "Total: 1 point\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6.1 Top Features for Naive Bayes and Perceptron\n",
    "\n",
    "- **Deliverable 6.1**: Implement ```get_top_features_for_label_numpy``` in ```features.py```. (0.5 points)\n",
    "- **Test**: `tests/test_features.py:test_d6_1_topfeat_numpy`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from gtnlplib import features\n",
    "reload(features);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(('pre-1980', '**OFFSET**'), 949.0),\n",
       " (('pre-1980', 'lord'), 326.0),\n",
       " (('pre-1980', 'very'), 293.0),\n",
       " (('pre-1980', 'feelin'), 263.0),\n",
       " (('pre-1980', 'satisfied'), 251.0),\n",
       " (('pre-1980', 'darling'), 247.0),\n",
       " (('pre-1980', 'yes'), 235.0)]"
      ]
     },
     "execution_count": 104,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "features.get_top_features_for_label_numpy(theta_perc,'pre-1980',7)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(('1990s', 'cuz'), 452.0),\n",
       " (('1990s', 'hit'), 368.0),\n",
       " (('1990s', 'sweat'), 342.0),\n",
       " (('1990s', 'prove'), 322.0),\n",
       " (('1990s', 'yo'), 285.0),\n",
       " (('1990s', 'saw'), 271.0),\n",
       " (('1990s', 'jam'), 268.0)]"
      ]
     },
     "execution_count": 105,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "features.get_top_features_for_label_numpy(theta_perc,'1990s',7)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6.2 Top Features for Logistic Regression\n",
    "\n",
    "- **Deliverable 6.2**: Implement ```get_top_features_for_label_torch``` in ```features.py```. (0.5 points)\n",
    "- **Test**: `tests/test_features.py:test_d6_2_topfeat_torch`\n",
    "\n",
    "**Hint**: Extract linear layer weights from the PyTorch model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "reload(features);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load a fixed model so we have reproducible results. Feel free to change it to your own model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model_test = torch.load('tests/test_weights.torch')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['love', 'lovin', 'and', 'baby', 'on']"
      ]
     },
     "execution_count": 108,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "features.get_top_features_for_label_torch(model_test, vocab, label_set,'pre-1980',5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['here', 'power', 'jam', 'saw', 'yeah', 'want', 'yall']"
      ]
     },
     "execution_count": 109,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "features.get_top_features_for_label_torch(model_test, vocab, label_set,'1990s',7)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 7. Feature Engineering\n",
    "\n",
    "Total: 0.75 points"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 7.1 Example Feature - Token-Type Ratio\n",
    "\n",
    "You can have features other than bag of words in your model. For example, we can consider the token-type ratio for each song. The token-type ratio is:\n",
    "\\begin{equation}\n",
    "\\frac{\\text{length of song in tokens}}{\\text{number of distinct types}} = \\frac{\\sum_m w_m}{\\sum_m \\delta(w_m > 0)}\n",
    "\\end{equation}\n",
    "\n",
    "- **Deliverable 7.1**: Implement ```get_token_type_ratio``` in ```features.py```. (0.25 points)\n",
    "- **Test**: `tests/test_features.py:test_d7_1_token_type_ratio`\n",
    "\n",
    "Return zero if the length of the song is zero."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "reload(features);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Token-type ratios for the first five songs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[5.083333333333333,\n",
       " 2.6,\n",
       " 1.9113924050632911,\n",
       " 2.318840579710145,\n",
       " 6.188679245283019]"
      ]
     },
     "execution_count": 111,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[features.get_token_type_ratio(X_tr[i]) for i in range(5)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 7.2 Binning Your Features\n",
    "\n",
    "Discretize your token-type ratio feature into bins: \n",
    "\n",
    "\n",
    "$\\{ [0,1), [1,2), [2,3), [3,4), [4,5), [5,6), [6, \\infty) \\}$.\n",
    "\n",
    "For each instance, there will be seven new features (one per bin). Exactly one of these features will have the value one; all others will have the value zero.\n",
    "\n",
    "Use `np.concatenate` or `np.hstack` to concatenate your result to the variable X_tr.\n",
    "\n",
    "- **Deliverable 7.2**: Implement ```concat_ttr_binned_features``` in ```features.py```. (0.5 points)\n",
    "- **Test**: `tests/test_features.py:test_d7_2_discretize`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "reload(features);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(4000, 4875)"
      ]
     },
     "execution_count": 113,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_tr.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[0. 0. 0. ... 0. 1. 0.]\n",
      " [0. 0. 0. ... 0. 0. 0.]\n",
      " [0. 0. 0. ... 0. 0. 0.]\n",
      " ...\n",
      " [0. 0. 0. ... 0. 0. 0.]\n",
      " [0. 0. 0. ... 0. 0. 0.]\n",
      " [0. 0. 0. ... 0. 0. 0.]]\n",
      "(4000, 4882)\n"
     ]
    }
   ],
   "source": [
    "X_tr_new = features.concat_ttr_binned_features(X_tr)\n",
    "print(X_tr_new)\n",
    "print(X_tr_new.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X_dv_new = features.concat_ttr_binned_features(X_dv)\n",
    "X_te_new = features.concat_ttr_binned_features(X_te)\n",
    "X_tr_var = Variable(torch.from_numpy(X_tr_new.astype(np.float32)))\n",
    "X_dv_var = Variable(torch.from_numpy(X_dv_new.astype(np.float32)))\n",
    "X_te_var = Variable(torch.from_numpy(X_te_new.astype(np.float32)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model = logreg.build_linear(X_tr_var,Y_tr)\n",
    "model.add_module('softmax',torch.nn.LogSoftmax(dim=1))\n",
    "loss = torch.nn.NLLLoss()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "See if these features help!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1: Dev Accuracy: 0.31333333333333335\n",
      "Epoch 101: Dev Accuracy: 0.4\n",
      "Epoch 201: Dev Accuracy: 0.42444444444444446\n",
      "Epoch 301: Dev Accuracy: 0.4444444444444444\n",
      "Epoch 401: Dev Accuracy: 0.4577777777777778\n"
     ]
    }
   ],
   "source": [
    "model,losses,accuracies = logreg.train_model(loss,model,X_tr_var,Y_tr_var,\n",
    "                                             Y_dv_var=Y_dv_var,X_dv_var = X_dv_var,\n",
    "                                             num_its=500,status_frequency=100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAtQAAACdCAYAAACU5e99AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvAOZPmwAAIABJREFUeJzt3Xl8XVW5//HPk3ls0swdkqZ0pKUFSgfKWAQEpIIiMiqKKOBwVVS8jqjo/V24KoiAIJMFRZBBsCAyU2bapvNcOjdt07TN0LRpMz6/P85uSNu0DZlOcvJ9v17nlbPXHs6zYHflyTprr2XujoiIiIiItE1UuAMQEREREenJlFCLiIiIiLSDEmoRERERkXZQQi0iIiIi0g5KqEVERERE2kEJtYiIiIhIOyihFhERERFpByXUIiIiIiLtoIRaRERERKQdYsIdwMeVlZXlhYWF4Q5DRKRN5syZs93ds8MdR1dRmy0iPVlr2+wel1AXFhZSVFQU7jBERNrEzNaHO4aupDZbRHqy1rbZGvIhIiIiItIOSqhFRERERNqhxw35aIv/e3E5u2vq+dWFx4Q7FBEREZFe68XFW3h05oam7aS4aG793FjSk+LCGFX79YqEetmWnWzfVRvuMERERER6jdKde3lv9Y79yn738gr21jVQkJGEA29/uJ2+Scs58ajMQ14nOzWek4dmHVTu7ry6rJTdNfVHjCUnNZ6TWrhGR+kVCXVcTBS19Y3hDkNERESk1/jh0wuZsWLbQeX3XDmO88b0A+DKBz7g8dkbeXz2xsNe64Vvn8qo/n32K3t56Vau++ucVsVy+vBsJdTtFRcTTW2DEmoRERGRjjJjRSnX/XUO9Y3e4v6GRuerpwzmyhMHNZXFxUTRPy2hafvBL01gS+XeQ37GntoGPvundzn/zreJMjvo+v3SEnj0q5OwA/YdKDE2ujVVarPekVBHq4daREREpD127q3jtpdXsqe2AYDZ68pIS4zlkvH5LR4fGx3Fl08qJC0p9pDXTIiNZnBW8mE/94+XH8+i4soW9506LIujslNaWYPO0zsS6hijRgm1iIiISJsUl1cz7d11THtvHbl94jEMM7jh7OFcPrGgUz/7nNF5nDM6r1M/o716R0IdHUWdhnyIiIiIfGx7ahuYeuc7VFTXMWlwBv+4bnK4Q+p2Oi2hNrOHgKlAqbsfcr46M5sAvA9c5u5PdUYseihRREREpHUWbKzge0/Mp64hNDa6tr6Riuo6/uezx3DeMf3CHF331Jk91NOAu4BHDnWAmUUDtwIvd2IcoYRaPdQiIiIih1VSuZev/20OVXvrOWtUblP5gPRErphYcMSH/3qrTkuo3f0tMys8wmH/BTwNTOisOADioqNpaHQaGp3oKN0IIiIi0nvVNzSytaqmxX3/9+JyNlfu5btnDeO7Zw3v4sh6rrCNoTazAcBngTPo5IQ6NiaURNfWN5IY17nTpoiIdAUz+z3wkLsvCXcsItKz3PDEAp5bsPmQ+y+fmK9k+mMK50OJfwD+290bj/T1gZldC1wLUFDw8Z8kjYuOAqC2oZFElFCLSERYBtxnZjHAX4DH3L3leaVERIBb/rOc+RvLmb2unE+NyWPK8JyDjjGDs5sN9ZDWCWdCPR54PEims4BPmVm9uz974IHufh9wH8D48eNbnj38MOJjgoRaDyaKSIRw9weAB8xsBHA1sNDM3gXud/c3DneumZ0L3AFEAw+4+y2HOO5zwFPABHcvCobxLQNWBId84O7Xd0R9RKRjvLG8lO27Dh7OsbumnnvfXM2wnBROHprFz6eOol9aYhgijExhS6jdffC+92Y2DXi+pWS6I8TFfNRDLSISKYIHu0cGr+3AAuB7Znadu192mHPuBs4GioHZZjbd3ZcecFwq8B1g5gGXWO3ux3VsTUSkIyzYWMHV02Yfcn9ibDR//9qJZKfGd2FUvUNnTpv3GDAFyDKzYuAXQCyAu9/bWZ/bkjj1UItIhDGz2wlNTfo68P/cfVaw61YzW3HoM5kIrHL3NcF1HgcuBJYecNyvCc3CdGOHBi4iHeahd9Zy8/P7/9NNiotm+rdOJj7m4CGufRJiD7tqobRdZ87ycfnHOPbLnRUHhJa+BCXUIhJRFgI/c/fdLeybeJjzBgAbm20XA5OaH2Bm44B8d/+3mR2YUA82s3nAzuDz3z7wA9r73ItITzJ7XRlPFm3k2tOOYmhOapuv89ScYnL7xHPqsOxWHf/2h9u4+fmlHN2vD59sNub5uPz0dsUhbdNrVkoEtFqiiESSCpq14WaWDkxx92fb83CimUUBtwFfbmH3FqDA3XeY2QnAs2Y22t13Nj+ovc+9iHR3G8uq2VheDcCvn1/Gsi07SYmP5aZPjzroWHdn3sYK9tY1HPJ6NfWN/ODJBQD8/WuTDnlccz9/dnHo59SjOWlI1setgnSw3pFQB0M+atRDLSKR4xfu/sy+DXevMLNfAEd6FmUTkN9se2BQtk8qcAwwI3hoPA+YbmYXuHsRUBN83hwzWw0MB4raWxmRnmJPbQOfviu0DHdzizZVtHj8S0tKuP5vc1t9/SvuP/CxhUO747LjlEx3E70qodaQDxGJIFEtlLWmTZ8NDDOzwYQS6cuAK/btDHq3m35Dm9kM4AfBLB/ZQJm7N5jZUcAwYE3bqyDSc1RU13LF/TMpraqhorqOWy4aQ2FWMrHRUfxr/iYeeX89j83awOUTC/jbB+v50xurAKjcU8eA9ER+f8mxh71+36Q4qmvrW935FxcTxfH56e2ul3SMXpFQx2uWDxGJPEVmdhuhGTsAvgnMOdJJ7l5vZt8CXiI0bd5D7r7EzG4Gitx9+mFOPw242czqgEbgencva1ctRLrQwuIKXlm6tU3nrtxaxdItO/ns8QMoyEji0gn5TctwpyXG8NScYm5/ZSWbK/bw2KwN9EmM5YSCvgCcP7YfJx6V2WH1kO6nVyTUeihRRCLQfwE/B/4RbL9CKKk+Ind/AXjhgLKbDnHslGbvnwaebkOsIl2urqGRzRV79iu78cmFrNhaRdTh15M7pDNH5nD7pQfPGjk0J5V7vnACX3ukiLvfWEVsdBR/uPR4Thmm4Ri9Ra9IqDXkQ0QiTTC7x4/CHYdId3Xjkwt4dv7By2vfctEYLpvY8bPPnD48m5W/Oa/Drys9Q69IqDOS4wDYVrU3zJGIiHSMYDzzD4HRQMK+cnf/RNiCEukm7pmxmmfnb+a8Y/L2W0Y7Piaac0ZrWW3peL0ioc5OiSchNoqN5XuOfLCISM/wKKHhHlOB64EvAdvCGpFIN1BZXccdr60kNT6Gn00dxYB0La8tna+lp8QjjplRkJHEhrLqcIciItJRMt39QaDO3d90968A6p2WXqOx0alraDzo9dTcYvbWNfLYtScqmZYu0yt6qAHy+yaxYYcSahGJGPsmwd1iZucDm4GMMMYj0mVq6xv55O1vsu4Qv9ePHZjGMQPSujgq6c16TUJ9zIA0Xl9RyqaKPfqLVUQiwW/MLA34PnAn0Ae4IbwhiXSe7btq+L8Xl1NT30h5dR3rdlRz1eRB5KTGH3Ts2aPywhCh9Ga9JqG++ISB3Pn6h9z/1hp+ecHocIcjItJmZhYNDHP354FK4IwwhyTSIZZt2cm2qhoAkuOjGVfQFzOjrqGRn/xzEa8s28qgjCQAzhiRzS8/PZqots6BJ9KBek1CnZ+RxGUTC/jrB+u5clIBw3JTwx2SiEibBCsVXg7cHu5YRDrKhh3VnP/Ht2n0j8qmXT2BKSNyePCdtby8dCtnHZ3DA1+aEL4gRQ6h1yTUAN8/ezjPLdjMr/+9jIevntC0wpGISA/0rpndRWimj937Ct19bvhCku7o5ueW8tKSEu676gTKd9dx2ysreOSaSaTEd04K8OLiEn45fQkNHsqMzxyZwy2fGwvAqtJdXD1tFnvrDl4XYm9dA2bGw1dPIDkumuv/NpdvPDqX5PgYKqvrGJyV3OKiKiLdQa9KqDNT4rnhrOHc/PxSpi/YzIXHDQh3SCIibbUvs7i5WZmjmT4k8GTRRlZt28VD764F4OfPLmbuhgoAfvjUAgZlJnNcfjrnjO6Y8cYNjc6D76zhsVkbiTI4Y2Qua7bt4h9FG0lNiCEmOoo568sp3VnDReMGtniNMQPSOH14NgC3fm4Mry4rBcAMrphYQGpCbIfEKtLRelVCDfClkwp5fuFmbvrXEk48KpPcPglHPklEpJtxd42bloNsLKumpr6R0qq93PjUQmKijIzkOE4ZmsWLi0uIi4mitr6RV5eW0uBObLTx1PUnkRAb3XSNmChjUGbSYb/F3VSxh7TEWFKC3uNtu2ooWlfG/3thOfExUfzvRWO4aNxAtu7cy9Q73+Hh99Y3nfvFyYP4+dRRR6zLmUfncubRWoRFegZz9yMf1ZYLmz1EaMGBUnc/poX9FwK/BhqBeuC77v7Oka47fvx4Lyoqaldsa7fv5rw73mJCYQbTrp5ItB5oEJEuYmZz3H18B1znppbK3f3mlsrDpSPabGmd15Zt5ZqHP/pvHRttvPvfnyDnEB1HizdVMvXOln/t/u9FY7j8EMtzz1yzg0vv+4DhuSk891+ncPr/zaBkZ2gl4rw+Cbzz32cQE90rlrmQXqC1bXZn9lBPA+4CHjnE/teA6e7uZjYWeAIY2YnxNBmclcwvPz2aH/1zEbe9soIbz+mSjxUR6Ui7m71PINSBsSxMsUg3MO29deT1SeAn5x8NQH7fxEMm0xCaTvbvX53E9t21+5Xf/foqbn1xOf+YvZGrTy7kwuMG0NDo3PCP+Wwoq6akMpQ8r9y6i0/d8TYlO/fyvbOHU5iVzKh+qUqmpVfqtITa3d8ys8LD7N/VbDOZ0Ni/LnPZxALmb6zg7jdWM2ZAOuceozkrRaTncPffN982s98BL4UpHAmzDTuqefvD7dxw1nAuOLZ/q887aWjWQWUZSXHc//YaVm6t4pb/LGfn3no2le9h+oLNjCtIZ3heKt/8xFDmrCujrLqOiYMz+OYZQ/Vtr/RqYR1DbWafBf4XyAHOP8xx1wLXAhQUtPwVVFv86sLRLCup4vtPzKd/+omMHZjeYdcWEeliSUDLT3pJxHvk/XVERxmXTshv97VOGZbFKcOyeHXpVr76SBE/f3YxAP3TEnj82snExYR6oL944qB2f5ZIpAhrQu3uzwDPmNlphMZTn3WI4+4D7oPQeLyO+vz4mGju/+IJXHTPe3z5L7N54rrJDM1J6ajLi4h0GjNbxEff7EUD2ew/44dEuLXbd3PBne8wcXAGry0v5exRueSlddyD9meNymXBTZ+ktiE0xV1qQkxTMi0i++sWs3wEw0OOMrMsd9/elZ+d0yeBv10ziYvvfY+rHpzJ49dOpiAzqStDEBFpi6nN3tcDW929PlzBSNdZu303t7+ykhUlVVTV1PPa8lIyk+O4qRUzZ3xcaUmapk6kNcL2p6aZDbVgTh4zGwfEAzvCEUthVjIPf2Ui1XUNfP7P7/Hh1qpwhCEi8nH0A8rcfb27bwISzWxSuIOSjvXh1ipeX751v9f//HsZLy4uocGdrJQ4RualctcV48jPUGeQSLh0Wg+1mT0GTAGyzKwY+AUQC+Du9wKfA64yszpgD3Cpd9Ycfq0wun8a/7h2Ml94cCaX/Pl9HvnKJMYMTAtXOCIiR3IPMK7Z9u4WyqQHq9xTx4V3v0t1bcNB+66aPIibLzxoRloRCZPOnOXj8iPsvxW4tbM+vy1G5KXy5HWTufKBmVx23/vccdnxnDVKk8qLSLdkzTsh3L3RzLrFMD5pvz21DUz57RtU1zZw9xXjGNg3sWmfWej3lYh0H3q64ACFWck8/fWTOCo7ha/9tYh7ZqwmjB3nIiKHssbMvm1mscHrO8CacAclHeO5hZspr67jykkFnD+2H8fmpze9xg5MJz4m+sgXEZEuo4S6BXlpCTxx3WTOH9OPW19czrcfn0/V3rpwhyUi0tz1wEnAJqAYmEQwvaj0bOt37OZvH6xnaE4Kv/mMhnWI9ARKqA8hMS6aOy8/nhvPGcELi7Zw/h/fYf7GinCHJSICgLuXuvtl7p7j7rnufoW7l7bmXDM718xWmNkqM/vRYY77nJm5mY1vVvbj4LwVZnZOR9RFPjJjRSmn/3YGC4sruXJSAcGz+yLSzbUqoTaz75hZHwt50MzmmtknOzu4cDMzvnnGUP5x7Yk0NDoX3/Me98xYTUOjhoCISHiZ2cNmlt5su6+ZPdSK86KBu4HzgFHA5WZ20HxrZpYKfAeY2axsFHAZMBo4F/hTcD1po8o9dXzhgZlMvfNtpt75Nj94cgHZqfHcf9V4vqCFU0R6jNb2UH/F3XcCnwT6Al8Ebum0qLqZ8YUZvPDtU/nk6FxufXE5F93zHitKNLWeiITVWHdv+trM3cuB41tx3kRglbuvcfda4HHgwhaO+zWhB8f3Niu7EHjc3WvcfS2wKriefAwvLSnhL++u5S/vruVXzy3hnVXbyUyOJzc1gWMHpnPT1FGcPSqX2Gh9iSzSU7T2ifB93zl9Cviruy+xXvY9VFpSLHdfMY7pCzbzq+eWMvXOt/nGlKF844whejhERMIhysz6Bok0ZpZB69r0AcDGZtv7xl83CdYGyHf3f5vZjQec+8EB5w448APM7FqC8dwFBQWtCCnyNTY6u2vrWbe9muv+Ome/fZMGZ/DwV/R3iUhP1tqEeo6ZvQwMBn4cfBXY2HlhdU9mxoXHDeDUYdnc/NwS7njtQ/41fxM/O38UZx6do7FuItKVfg+8b2ZPEur0uBj4n/Ze1MyigNuAL7f1Gu5+H3AfwPjx4zVGDrjm4dm8sWIbALHRxqvfO520xNAqhKkJWo1QpKdrbUJ9DXAcsMbdq4OekKs7L6zuLSM5jj9cdjwXjRvIzc8v5auPFHHa8GxumjqKoTkp4Q5PRHoBd3/EzOYAZwRFF7n70lacugnIb7Y9MCjbJxU4BpgRdBLkAdPN7IJWnCstqGto5L3VOzh1WBanD89mWG4qgzKTwx2WiHSg1ibUk4H57r7bzL5AaCWuOzovrJ7htOHZ/Oc7p/LX99dz+6srOfcPb3HJhHy+/Ylh5KUlhDs8EYlwwfC7bUACgJkVuPuGI5w2GxhmZoMJJcOXAVc0u2YlkLVv28xmAD9w9yIz2wP83cxuA/oDw4BZHViliFNdW88z8zZRU9/IxScM5MLjDhohIyIRoLVPPNwDVJvZscD3gdXAI50WVQ8SGx3FV04ZzIwfTOGKSQU8WbSR0377Br95fik7dtWEOzwRiVBmdoGZfQisBd4E1gH/OdJ57l4PfAt4CVgGPBEk5jcHvdCHO3cJ8ASwFHgR+Ka7H7wutjS5/ZWV/PSZxUQZjCvoG+5wRKSTWGtWATSzue4+zsxuAja5+4P7yjo/xP2NHz/ei4qKuvpjW21jWTV3vPYh/5xbTGJsNF8+uZCrTx5MVkp8uEMTkW7AzOa4+/gjH3nE6ywAPgG86u7Hm9kZwBfc/Zp2B9mBunub3Rm+98R8Xlm6FYDdNfWcMSKHX14wmvyMpDBHJiIfV2vb7NYO+agysx8Tmi7v1OChFT1F0YL8jCR+9/ljuf70Idz+6kr+NGM1D7y9lksn5PO1U49SgyoiHaXO3XeYWZSZRbn7G2b2h3AH1ZtNe3ctq7ft5p9zN3HqsCyG5qQQZcYXThyktl8kwrU2ob6U0Bi7r7h7iZkVAL/tvLB6vqE5Kdx9xThWb9vFfW+u4bFZG3h05gY+PbYfXzvtKEb3Twt3iCLSs1WYWQrwFvComZUCu8McU69VWrWXXz63lMTYaPIzEvn9548lp4+epRHpLVqVUAdJ9KPABDObCsxyd42hboUh2SncevFYbjh7OA++s4ZHZ27g2fmbmViYwVUnDeKc0XmavF9E2uJCYA9wA3AlkAbcHNaIerHFmyoBePgrE5k4OCPM0YhIV2tVQm1mlxDqkZ5BaL7TO83sRnd/qhNjiyh5aQn89PxRfOuMYTxRtJFHPljHt/4+j9w+8Vw5aRCXTywgO1XjrEWkddx9X290I/BwOGPpDdZs28UN/5jPOcfk8Y0pQwFYXrKTG59cSF1DI+XVtZjB6P59whypiIRDa4d8/BSY4O6lAGaWDbwKHDKhNrOHgKlAqbsf08L+K4H/JpSgVwFfd/cFHy/8nictKZavnXZUaGaQFaVMe28dt72ykjtf/5CzR+Xy+fH5nDYsm+goLRIjItJdvLtqOwuKK1lQXMnR/fowdkAaX//bXLbu3Mupw7IYlJnE6P5pJMe39teqiESS1v7Lj9qXTAd2cOQp96YBd3Ho6fXWAqe7e7mZnUdoVa1Jhzg24kRHGWcencuZR+eyetsuHv1gA8/MK+aFRSXk9UngcycM4PMn5FOYpcn/RUTCbUNZddP7bz82j7NH5bJ2+26+dupgfnr+qDBGJiLdQWsH775oZi+Z2ZfN7MvAv4EXDneCu78FlB1m/3vuXh5sfkBoxa1eaUh2Cjd9ehQzf3IW91w5jqP7pXLPjNVM+d0MLvnz+zxRtJGde+vCHaaIdCNm9p3WlEnHWL+jmmE5KTzylYlU7a3nn3M3cf6YfvzkU0eHOzQR6QZa+1DijWb2OeDkoOg+d3+mA+O4hsMsSGBm1wLXAhQUFHTgx3YvcTFRnDemH+eN6UdJ5V7+Oa+YJ4uK+eFTC/nZM4uZMiKbC47rz5kjc0mMiw53uCISXl/i4BVrv9xCmXSADWXVDMpM4tRhWfzx8uOpqK7l3GPyCJZnF5FertWDvdz9aeDpjg4gWIzgGuCUw3z2fYSGhDB+/Pgjr0QTAfLSEvjGlKF8/fQhzN9YwfQFm/n3wi28vHQrSXHRnD0ql0+P7c9pw7OJi9EsISK9hZldTmga08FmNr3ZrlQO862gtN0Li7awvKSK88f0w8y44Nj+4Q5JRLqZwybUZlYFtJTAGuDu3q7Hmc1sLPAAcJ6772jPtSKVmXF8QV+OL+jLz84fxcy1O3huwRb+s3gL/5q/mT4JMZx5dC7njM7ltOHZJMXpgRiRCPcesAXIAn7frLwKWBiWiCJYSeVevvn3uQB8fnx+mKMRke7qsNmXu6d21gcHi8P8E/iiu6/srM+JJNFRxklDsjhpSBa/umA0767azvMLt/Da8q08M28T8TFRnDosm3NG53LW0bn0TY4Ld8gi0sHcfT2wHphsZoOAYe7+qpklAomEEmtpg7qGRj5/7/tcMbGASybkU767lpNueQ13eOK6yeSlaaEWEWlZp3VnmtljwBQgy8yKgV8QLFfu7vcCNwGZwJ+CMWj1rVkrXULiYqI4Y2QOZ4zMob6hkVnrynh5yVZeXlLCq8u2Eh1lTCjsy9mj8vjEyBwGa7YQkYhiZl8j9GxJBjCE0IPd9wJnhjOunmrO+jIefGct8zdWMH9jBctLqviwtIpGh++dPVyLtYjIYZl7zxqSPH78eC8qKgp3GN2Wu7N4005eWlLCS0tK+LB0FwCDMpM4Y0QOp4/IZvJRmSTE6qFGkXAwszkd0XlgZvOBicBMdz8+KFvk7mPae+2O1FPa7GsfKeK15aU0NH70OzEjOY6ThmRy1xXjwhiZiIRTa9tsDbiNMGbGmIFpjBmYxg/OGcGGHdXMWFnKjBXbeHz2Bqa9t474mCgmD8nkjBE5TBmRzaBM9V6L9EA17l67b5YJM4uh5WdepBW2VtVw0pBMHvjSeEb87EWG56bw8g2nhzssEekhlFBHuILMJK6aXMhVkwvZW9fAB2t2MGPFNmasKOUXK5YAUJiZxElDszh5SBaTh2SSobHXIj3Bm2b2EyDRzM4GvgE8F+aYepxVpbv4xfTFLN1cyYXHDSA+Jpp3/vsMUuNjwx2aiPQgSqh7kYTYaKaMyGHKiBxgNGu372bGilLe+XA70+dv5u8zN2AGo/r14eShWZw0JJOJgzM0c4hI9/QjQlOOLgKuI7TY1gNhjagHuvXF5by7KjTJVG6feAAG9k0KZ0gi0gMpU+rFBmclMzhrMFefPJi6hkYWFlfy7qrtvLtqO9PeXcd9b60hNjo0bd++3uuxA9M0/lqkG3D3RuB+4H4zywAGek97KKYbWLCxoul9Vkp8GCMRkZ5MCbUAEBsdxQmD+nLCoL58+8xh7KltYPa6Mt5dvZ33Vu3gD6+t5PZXQ7OLHJefzsTCDCYOzmDcoL6kxOs2EulqZjYDuIBQOz4HKDWz99z9hrAG1oPsqW2gtKqG3D7xbN1ZQ2qChnmISNsoE5IWJcZFc9rwbE4bng1ARXUts9eVM2vtDmatLeOeN1dz1xuriI4yRvfvw8TCDCYMzmBCYYbGYIt0jTR332lmXwUecfdfmJkWdvkYNpRVA/CTTx1NakIMpw3LDnNEItJTKaGWVklPiuPsUbmcPSoXgN019czdUM7stWXMXFvGIx+s54F31gIwLCeFEwb15fiCdMYV9GVIdgpRURbO8EUiUYyZ9QMuAX4a7mC6u3kbypsS6H2WbQmtgVOYmcyx+enhCEtEIoQSammT5PgYTh2WzalBj05NfQOLiiuZubaM2evK+M/iEh6fvRGA1IQYjstP5/iCvowrSOf4/L6kJemrVZF2uhl4CXjH3Web2VHAh2GOqVvavquGS//8AbUNjQfti4+JolALX4lIOymhlg4RHxPN+MIMxheGVhNzd9Zs3828DRXM3VDOvA0V3PX6h+xbM+Go7GTGFXzUiz0sJ4WY6Kgw1kCkZ3H3J4Enm22vAT4Xvoi6ryeLiqltaOTvX51E7gHLh6cnxpKWqD/wRaR9lFBLpzAzhmSnMCQ7hYtPGAjArpp6FhZXMG9DBfM2lPP68lKemlMMQGJsNKP792HMwDTGDkxjzIB0jspK1lARkU5gZucCdwDRwAPufssB+68Hvgk0ALuAa919qZkVAsuAFcGhH7j79V0Vd1sUrSvj1heXM2lwBicNzQp3OCISoZRQS5dJiY/hpCFZnDQk9EvN3dlQVs3cDeUsLK5kUXElj8/ayF/eXQdAclw0xwwIEuyB6YwdkMagzCT2rQwnIh+fmUUDdwNnA8XAbDOb7u5Lmx32d3e/Nzj+AuA24Nxg32p3P64rY26Pu95YBcDXpwwJcyQiEsmUUEvYmBmDMpMZlJnW42ZHAAASN0lEQVTMZ48P9WLXNzSyettuFhZXsGhTJQuLK3n4/fXU1oceeExNiGHMgNDS6mMGpDGqXx8KM9WTLb2PmUW7e0MbTp0IrAqGiGBmjwMXAk0JtbvvbHZ8Mj10SXN3Z1FxJZeMHxgsaCUi0jmUUEu3EhMdxYi8VEbkpfL58fkA1DU0snJrFYuKK1m0KfR66J211DWEfscnxUUzMi+VUf37MKpfGkf3S2VkXh8S47QAjUS0tWb2IvAP4PWPsajLAGBjs+1iYNKBB5nZN4HvAXHAJ5rtGmxm84CdwM/c/e0Wzr0WuBagoKCglWF1vOLyPezYXcuYAWlhi0FEegcl1NLtxUZHMbp/GqP7p3FZUFZT38CHW3exdMtOlm7eybItO/nX/M387YMNAERZaCXIUf1DvdihZLsP2alaCU0ixkhgKqGxzg+a2fPA4+7+Tkdc3N3vBu42syuAnwFfArYABe6+w8xOAJ41s9EH9Gjj7vcB9wGMHz8+bL3bz8zbBNA0G5GISGdRQi09UnxMaHz1Mc16ntyd4vI9TUn20i07mbu+nOcWbG46Jjs1nqP79WFkXirDc1MZmZfK0JwULacuPY67VwNPAE+YWV9CDxm+SehBw8PZBOQ32x4YlB3K48A9wWfWADXB+zlmthoYDhS1pQ6dwd0praohMS6aV5ZuZWJhhqbFE5FO12kJtZk9RKj3pNTdj2lh/0jgL8A44Kfu/rvOikV6BzMjPyOJ/Iwkzhmd11ReWV0XSrK3hHqyl27eybQ1O6itD81JG2WhhR2G56YyPC+1KdkuzEzSVH7SrZnZ6cClhB4YLCK0yMuRzAaGmdlgQon0ZcAVB1x3mLvvm9P6fIL5rc0sGyhz94Zg3uthwJqOqEtHefCdtfzm38uIMmh0uP50PYwoIp2vM3uopwF3AY8cYn8Z8G3gM50YgwhpSbFMHpLJ5CGZTWX1DY2s21HNyq1VrCgJvVZureLlpSVNc2XHxUQxNDulaUz3iCDh7p+WoJlGJOzMbB0wj1Av9Y3uvrs157l7vZl9i9CiMNHAQ+6+xMxuBorcfTrwLTM7C6gDygkN9wA4DbjZzOqARuB6dy/ryHq1xQNvryG3TwKfPrY/76/eQVZKPNt31QAwdqDGT4tI5+u0hNrd3wrmLD3U/lKg1MzO76wYRA4lJjqKoTkpDM1J4VNj+jWV761rYFXprqYEe8XWKj5Ys6NpLCaEHoIckp3SdP6+94Myk4hVj7Z0nbEHjl1uLXd/AXjhgLKbmr3/ziHOexp4ui2f2ZnufXM1ZsbxBem8tryUi8YNYEB6Ios2VXLyEM09LSKdT2OoRZpJiD14bDZA5Z46PgwS7FWlu1hVuuugRDs2OjQN4NBmyfa+hFszjkgnyDOzZ4Bcdz/GzMYCF7j7b8IdWFfaVVPP9l21AJxy6xsATBqcwaUTwje7iIj0Pj0ioe4uUzBJ75WWGLvf0ur77KqpZ3WQYK/aFvq5cmsVryzbSkPjR5MbDEhP3C/J3pdo902K1fARaav7gRuBPwO4+0Iz+zvQqxLqjWXV+21fNXkQl4zPP8TRIiKdo0ck1N1lCiaRA6XEx3BsfjrH5qfvV15T38D6HdVNvdnNe7VrgochIZSoD85K5qisZAZnJTM4O/iZlUxSXI/45ynhk+Tusw74g6w+XMGEy/odoYT6pqmj2F1Tz5dOLtQfqSLS5fQbW6QTxMdEh2YNyU3dr7yh0dlUvodV26pYu72atdt3sXb7bj5Ys4N/ztt/5rK8PglNSXZTwp2VTH6GxmoLANvNbAjBKoZmdjGheaJ7jVWlVTy3cDMJsVFcOiGf5Hj9ShOR8OjMafMeA6YAWWZWDPwCiAVw93vNLI/QNE99gEYz+y4wqq0P2Yj0BNFRRkFmEgWZSQft21PbwLodu1m7PfRas203a7fv4j+LtlBeXbf/NTKSmhLswqxkCjOTGJSRTP/0BE3113t8k9A3dyPNbBOwFvhCeEPqOsXl1Zz7h7epb3QuUzItImHWmbN8XH6E/SWEFhQQESAxLpqj+/Xh6H59DtpXvruWtTt2s3Zbs4R7+27eW72dvXUfDSGJiTIG9E1kUGYygzKSGJSZREFGEoMykynISNLDkRHE3dcAZ5lZMhDl7lXhjqmrVFTXctZtb1Lf6Dxw1XhOGpp55JNERDqR/qQX6QH6JsfRNzmOcQV99ytvbHS2Vu1l/Y5qNuyoZt2O3awvC72fv6GcnXv3H1KbkxrPoMykpoS7oNn7dD0g2SOY2fcOUQ6Au9/WpQGFway1Zeyta+TyifmcNSo33OGIiCihFunJoqKMfmmJ9EtL5MSjDu6lq6iuZX2QaG/YUd2UbL/94Tae2lmz37GpCTGhZDsjOUi6kyjISKYgM4m8PglERynZ7ib2DcwfAUwApgfbnwZmhSWiLrZ4UyVRBjdNHR3uUEREACXUIhEtPSmO9KS4g2YhgdCY7Y3l1azbvpsNZdWsDxLuJZsreWlJCfXNpv2LiTL6pycysG8i+X2TQj8zPvqZnRJPlBLuLuHuvwIws7eAcfuGepjZL4F/hzG0LjNrXRnDc1M1hElEug0l1CK9VGJcyzORQGhp9i2Ve1m3Yzcby/ZQXF7NxvLQz9eWlzYt67xPXEwUA9MTGbgvyT4g6c5MjtNwko6XC9Q2264NyiLOy0tKmPbeOr7/yeFkpyTwwZoyvn/28HCHJSLSRAm1iBwkJjqK/Iwk8jMOno0EQr3bmyqCJLusmuLyPWwsD/1cVFyx36wkAImx0fv3agcJ98C+SfRPTyBDCXdbPALMClZLBPgMMC184XSeR2du4L3VO1jz6DyunBRa3Ov8sf3CHJWIyEeUUIvIx5YYF83QnFSG5hzcuw2hFSSLy6spLvso0d4YJN6z15VRdcDDkgmxUfRPT2RAeiL90xLpn55I//SE0HZ6InlpCSTE6uv95tz9f8zsP8CpQdHV7j4vnDF1Bndn0aZKAEp27uX3r6wkNSGGwVnJYY5MROQjSqhFpMOlxMcwMq8PI/MOngIQoHJPXVOCvaVyD5sr9rC5Yi+bKvbwRkkppVU1B52TlRLPgPSEjxLv4DUgSL57Yy+3u88F5oY7js60bkc1Zbtr+fVnjmFRcQVPFBUzZkBar/t/LSLdmxJqEelyaYmxpA1I45gBaS3ur6lvoKQylGBvrtgbJNx72FSxh5Vbq3hjRel+828DxMdENUu0E5oS7oHq5e7RnpqzkSiDM0fmMHVMPwqzkjl9eHa4wxIR2Y8SahHpduJjokPzY2e2/LW+u1NeXdeUZG+u2L+Xe8aKbYfo5Y4jLy2BvD6J9EtLoF96Av2C7f7pCeT2UdLdXRSXV9M3KY43V25j0uBM+qcnAvCNKUPDHJmIyMGUUItIj2NmZCTHkZEcd9he7q2VNRRXVIcS7fI9lOzcw5bKvRSXVzNr7Y6DFr4ByEiOI69PAv3TE8hLS6BfWiJ5ffYl36H3mq6tc+2qqeeUW99g/KC+rCip4qunHhXukEREDksJtYhEpPiYaAoyQ6tBHsrumnpKdu6lpDI0rKSkci9bgu3i8j0UrS+n4oAZSwDSk2L5zpnDuPrkwZ1ZhV5r+vzNABStLwdg7CH+aBIR6S6UUItIr5UcH8OQ7BSGZKcc8pg9tQ2U7NzLlopQ73bJzr1sqdxDoWaZ6DT90xM4ZWgWgzKTiIuJ4jSNmRaRbk4JtYjIYSTGRTM4K1nTtHWhKSNymDIiJ9xhiIi0WlS4AxARERER6cmUUIuIiIiItIMSahERERGRdjB3D3cMH4uZbQPWt+HULGB7B4fTXURy3SCy6xfJdYPIrl9b6zbI3XvNU3Zqsw8pkusXyXWDyK5fJNcN2la/VrXZPS6hbiszK3L38eGOozNEct0gsusXyXWDyK5fJNetO4j0/76RXL9IrhtEdv0iuW7QufXTkA8RERERkXZQQi0iIiIi0g69KaG+L9wBdKJIrhtEdv0iuW4Q2fWL5Lp1B5H+3zeS6xfJdYPIrl8k1w06sX69Zgy1iIiIiEhn6E091CIiIiIiHa5XJNRmdq6ZrTCzVWb2o3DH83GZ2UNmVmpmi5uVZZjZK2b2YfCzb1BuZvbHoK4LzWxc+CI/MjPLN7M3zGypmS0xs+8E5ZFSvwQzm2VmC4L6/SooH2xmM4N6/MPM4oLy+GB7VbC/MJzxt4aZRZvZPDN7PtiOpLqtM7NFZjbfzIqCsoi4N7sztdndl9rsiGjX1GbT8fdmxCfUZhYN3A2cB4wCLjezUeGN6mObBpx7QNmPgNfcfRjwWrANoXoOC17XAvd0UYxtVQ98391HAScC3wz+/0RK/WqAT7j7scBxwLlmdiJwK3C7uw8FyoFrguOvAcqD8tuD47q77wDLmm1HUt0AznD345pNtRQp92a3pDa72983arN7frumNjukY+9Nd4/oFzAZeKnZ9o+BH4c7rjbUoxBY3Gx7BdAveN8PWBG8/zNweUvH9YQX8C/g7EisH5AEzAUmEZpYPiYob7pHgZeAycH7mOA4C3fsh6nTwKCB+gTwPGCRUrcgznVA1gFlEXdvdqeX2uyedd+oze5Z7Zra7M67NyO+hxoYAGxstl0clPV0ue6+JXhfAuQG73tsfYOvk44HZhJB9Qu+XpsPlAKvAKuBCnevDw5pXoem+gX7K4HMro34Y/kD8EOgMdjOJHLqBuDAy2Y2x8yuDcoi5t7spiL1v2PE3Tdqs4Ge166pze6kezOmrSdK9+HubmY9eroWM0sBnga+6+47zaxpX0+vn7s3AMeZWTrwDDAyzCF1CDObCpS6+xwzmxLueDrJKe6+ycxygFfMbHnznT393pTwiIT7Rm12z6M2u3Pvzd7QQ70JyG+2PTAo6+m2mlk/gOBnaVDe4+prZrGEGuZH3f2fQXHE1G8fd68A3iD0lVq6me37g7Z5HZrqF+xPA3Z0caitdTJwgZmtAx4n9BXiHURG3QBw903Bz1JCv1gnEoH3ZjcTqf8dI+a+UZvdY9s1tdmdeG/2hoR6NjAseIo1DrgMmB7mmDrCdOBLwfsvERrHtq/8quDp1ROBymZfdXQ7FurWeBBY5u63NdsVKfXLDno5MLNEQmMNlxFqpC8ODjuwfvvqfTHwugeDu7obd/+xuw9090JC/65ed/criYC6AZhZspml7nsPfBJYTITcm92Y2uxufN+ozQZ6aLumNhvozHsz3APIu+IFfApYSWgc1E/DHU8b4n8M2ALUERrjcw2hcUyvAR8CrwIZwbFG6An51cAiYHy44z9C3U4hNOZpITA/eH0qguo3FpgX1G8xcFNQfhQwC1gFPAnEB+UJwfaqYP9R4a5DK+s5BXg+kuoW1GNB8Fqyr+2IlHuzO7/UZoe/Doepm9rsHtyuNaun2uwOvje1UqKIiIiISDv0hiEfIiIiIiKdRgm1iIiIiEg7KKEWEREREWkHJdQiIiIiIu2ghFpEREREpB2UUEuPZmbvBT8LzeyKDr72T1r6LBERaRu12RKpNG2eRIRgGdUfuPvUj3FOjLvXH2b/LndP6Yj4RETkI2qzJdKoh1p6NDPbFby9BTjVzOab2Q1mFm1mvzWz2Wa20MyuC46fYmZvm9l0YGlQ9qyZzTGzJWZ2bVB2C5AYXO/R5p8VrKr0WzNbbGaLzOzSZteeYWZPmdlyM3s0WFVMRERQmy2RK+bIh4j0CD+iWW9H0MhWuvsEM4sH3jWzl4NjxwHHuPvaYPsr7l4WLDM728yedvcfmdm33P24Fj7rIuA44FggKzjnrWDf8cBoYDPwLnAy8E7HV1dEpEdTmy0RRT3UEqk+CVxlZvOBmYSWHh0W7JvVrGEG+LaZLQA+APKbHXcopwCPuXuDu28F3gQmNLt2sbs3ElqSt7BDaiMiEtnUZkuPph5qiVQG/Je7v7RfYWjc3u4Dts8CJrt7tZnNABLa8bk1zd43oH9jIiKtoTZbejT1UEukqAJSm22/BHzdzGIBzGy4mSW3cF4aUB40zCOBE5vtq9t3/gHeBi4NxvxlA6cBszqkFiIivYPabIko+ktMIsVCoCH4GnAacAehr+7mBg+ZbAM+08J5LwLXm9kyYAWhrxD3uQ9YaGZz3f3KZuXPAJOBBYADP3T3kqBxFxGRI1ObLRFF0+aJiIiIiLSDhnyIiIiIiLSDEmoRERERkXZQQi0iIiIi0g5KqEVERERE2kEJtYiIiIhIOyihFhERERFpByXUIiIiIiLtoIRaRERERKQd/j/ebaI/GgVJygAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fd6f067a7f0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "logreg.plot_results(losses,accuracies)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 7.3 Bakeoff\n",
    "\n",
    "**Deliverable 7.3**: Try to get the best accuracy possible. \n",
    "\n",
    "Some ideas:\n",
    "\n",
    "- Better features\n",
    "- Better optimization\n",
    "- Better classifier, e.g. multilayer neural networks\n",
    "- Better loss function\n",
    "- Better preprocessing\n",
    "- Dropout or other regularization scheme\n",
    "\n",
    "The current best accuracies from the staff are 55.5% dev, 59.3% test.\n",
    "\n",
    "### Rubric\n",
    "\n",
    "Dev set\n",
    "- $\\geq 55\\%$: 1 point\n",
    "- $\\geq 54\\%$: 0.75 points\n",
    "- $\\geq 53\\%$: 0.5 points\n",
    "- $\\geq 51.5\\%$: 0.25 points\n",
    "\n",
    "Test set\n",
    "- $\\geq 58\\%$: 1 point\n",
    "- $\\geq 55\\%$: 0.75 points\n",
    "- $\\geq 52\\%$: 0.5 points\n",
    "- $\\geq 50\\%$: 0.25 points\n",
    "    \n",
    "### Extra credit\n",
    "- We will run a Kaggle competition for this bakeoff. More details are coming soon.\n",
    "- Extra credit will be given to the top three submissions (combined across 4650/7650), by **test set** performance: 1 point, 0.75 points, 0.5 points. \n",
    "- Another 1 point of extra credit will be awarded to submissions that are better than the best staff system, on the test set.\n",
    "- Staff will continue to try to improve their results until the deadline, but we will not tune on test set accuracy.\n",
    "- **Extra credit will be based on Kaggle submissions.** You don't have to participate in the Kaggle part of the bakeoff, but only Kaggle submissions will be eligible for extra credit."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.47333333333333333"
      ]
     },
     "execution_count": 119,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "_, Y_hat_dv = model.forward(X_dv_var).max(dim=1)\n",
    "np.save('bakeoff-dev.preds.npy', Y_hat_dv.data.numpy())\n",
    "evaluation.acc(np.load('bakeoff-dev.preds.npy'), Y_dv_var.data.numpy())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "_, Y_hat_te = model.forward(X_te_var).max(dim=1)\n",
    "np.save('bakeoff-test.preds.npy', Y_hat_te.data.numpy())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.46444444444444444"
      ]
     },
     "execution_count": 121,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# You can't run this\n",
    "evaluation.acc(np.load('bakeoff-test.preds.npy'), Y_te_var.data.numpy())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 8. 7650 Research Question\n",
    "\n",
    "(1 point.) CS 4650 students may do this component if they want; if they do, then the assignment will be graded as if they are in 7650. This is optional for CS4650 students, but if you submit something for this part, that is how you will be scored -- we're not taking the max over the two possible scoring options. CS 7650 students must do this part.\n",
    "\n",
    "You will select a recent research paper that performs *document* classification, using text. Summarize the paper, answering the following questions:\n",
    "\n",
    "- What are the labels, and how were they obtained?\n",
    "- Why is it interesting/useful to predict these labels?  \n",
    "- What classifier(s) do they use, and the reasons behind their choice? Do they use linear classifiers like the ones in this problem set?\n",
    "- What features do they use? Explain any features outside the bag-of-words model, and why they used them.\n",
    "- What is the conclusion of the paper? Do they compare between classifiers, between feature sets, or on some other dimension? \n",
    "- Give a one-sentence summary of the message that they are trying to leave for the reader.\n",
    "\n",
    "Your selection of papers is determined by the last digit of your GTID.\n",
    "\n",
    "- Digits 0-4: choose from ACL 2017, AAAI 2017, EACL 2017\n",
    "- Digits 5-9: choose from NAACL 2017, KDD 2017, EMNLP 2017\n",
    "\n",
    "You must choose a paper in the main conference (not workshops). The paper must be at least four pages long. All papers from these conferences are available for free online."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}