{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "Problem Set 2: Sequence Labeling\n", "=======================\n", "\n", "- This problem set focuses on sequence labeling with Hidden Markov models and Deep Learning Models.\n", "- The target domain is part-of-speech tagging on English and Norwegian from the Universal Dependencies dataset\n", "\n", "You will:\n", "- Do some basic preprocessing of the data\n", "- Build a naive classifier that tags each word with its most common tag\n", "- Implement a `Viterbi` Tagger using `Hidden Markov Model` in PyTorch\n", "- Build a `Bi-LSTM` deep learning model using PyTorch\n", "- Build a `Bi-LSTM_CRF` model using the above components (`Viterbi` and `Bi-LSTM`) \n", "- then implement techniques to improve your classifier and compete on Kaggle.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# 0. Setup\n", "\n", "In order to develop this assignment, you will need [python 3.6](https://www.python.org/downloads/) and the following libraries. Most if not all of these are part of [anaconda](https://www.continuum.io/downloads), so a good starting point would be to install that.\n", "\n", "- [jupyter](http://jupyter.readthedocs.org/en/latest/install.html)\n", "- [numpy](https://docs.scipy.org/doc/numpy/user/install.html)\n", "- [matplotlib](http://matplotlib.org/users/installing.html)\n", "- [nosetests](https://nose.readthedocs.org/en/latest/)\n", "- [pandas](http://pandas.pydata.org/) Dataframes\n", "\n", "Here is some help on installing packages in python: https://packaging.python.org/installing/. You can use ```pip --user``` to install locally without sudo." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## About this assignment\n", "\n", "- This is a Jupyter notebook. You can execute cell blocks by pressing control-enter.\n", "- Most of your coding will be in the python source files in the directory ```gtnlplib```.\n", "- The directory ```tests``` contains unit tests that will be used to grade your assignment, using ```nosetests```. You should run them as you work on the assignment to see that you're on the right track. You are free to look at their source code, if that helps -- though most of the relevant code is also here in this notebook. Learn more about running unit tests at http://pythontesting.net/framework/nose/nose-introduction/\n", "- You may want to add more tests, but that is completely optional. \n", "- **To submit this assignment, run the script ```make-submission.sh```, and submit the tarball ```pset2-submission.tgz``` on Canvas.**" ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "collapsed": true }, "outputs": [], "source": [ "import sys\n", "from importlib import reload" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "My Python version\n", "python: 3.6.3 |Anaconda custom (64-bit)| (default, Oct 13 2017, 12:02:49) \n", "[GCC 7.2.0]\n" ] } ], "source": [ "print('My Python version')\n", "\n", "print('python: {}'.format(sys.version))" ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "collapsed": true }, "outputs": [], "source": [ "import nose\n", "\n", "import pandas as pd\n", "import numpy as np\n", "import matplotlib\n", "import matplotlib.pyplot as plt\n", "\n", "import torch\n", "import torch.nn as nn\n", "from torch.autograd import Variable\n", "from torch import optim\n", "\n", "%matplotlib inline" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "My library versions\n", "pandas: 0.22.0\n", "numpy: 1.14.0\n", "matplotlib: 2.1.0\n", "nose: 1.3.7\n", "torch: 0.3.0.post4\n" ] } ], "source": [ "print('My library versions')\n", "\n", "print('pandas: {}'.format(pd.__version__))\n", "print('numpy: {}'.format(np.__version__))\n", "print('matplotlib: {}'.format(matplotlib.__version__))\n", "print('nose: {}'.format(nose.__version__))\n", "print('torch: {}'.format(torch.__version__))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "To test whether your libraries are the right version, run:\n", "\n", "`nosetests tests/test_environment.py`" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ ".\r\n", "----------------------------------------------------------------------\r\n", "Ran 1 test in 0.001s\r\n", "\r\n", "OK\r\n" ] } ], "source": [ "# use ! to run shell commands in notebook\n", "! nosetests tests/test_environment.py" ] }, { "cell_type": "code", "execution_count": 10, "metadata": { "collapsed": true }, "outputs": [], "source": [ "from collections import defaultdict, Counter\n", "\n", "# importing all the necessary files from gtnlplib\n", "from gtnlplib import constants, preproc, most_common, clf_base, evaluation\n", "from gtnlplib import scorer, tagger_base, naive_bayes, hmm, viterbi, bilstm" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "1. Data Processing\n", "===================\n", "The part-of-speech tags are defined on the [universal dependencies website](http://universaldependencies.org/en/pos/index.html)." ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "collapsed": true }, "outputs": [], "source": [ "reload(constants);\n", "## Define the file names\n", "TRAIN_FILE = constants.TRAIN_FILE\n", "DEV_FILE = constants.DEV_FILE\n", "TEST_FILE = constants.TEST_FILE \n", "TEST_FILE_HIDDEN = constants.TEST_FILE_UNLABELED\n", "NR_TRAIN_FILE = constants.NR_TRAIN_FILE\n", "NR_DEV_FILE = constants.NR_DEV_FILE\n", "NR_TEST_FILE = constants.NR_TEST_FILE\n", "NR_TEST_FILE_HIDDEN = constants.NR_TEST_FILE_UNLABELED\n", "# change the constant to test_file_unlabeled for release" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- Here is demo code for using the function `conll_seq_generator(...)`. \n", "- The default value for max_insts is `1000000` indicating the num. of instances: and this should be enough for our dataset. " ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "['ADJ', 'ADP', 'ADV', 'AUX', 'CONJ', 'DET', 'INTJ', 'NOUN', 'NUM', 'PART', 'PRON', 'PROPN', 'PUNCT', 'SCONJ', 'SYM', 'VERB', 'X']\n" ] } ], "source": [ "## Demo\n", "all_tags = set([])\n", "for (words, tags) in preproc.conll_seq_generator(TRAIN_FILE):\n", " for tag in tags:\n", " all_tags.add(tag)\n", "all_tags = sorted(all_tags)\n", "print (all_tags)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Deliverable 1.1**: Counting words per tag. (* 1 point *)\n", "\n", "Implement the function `get_tag_word_counts` in `most_common.py`: The function should calculate the number of occurrences of all words for each tag.\n", "\n", "- **Input**: filename for data file, to be passed as argument to `preproc.conll_seq_generation`\n", "- **Output**: dict of counters, where keys are tags\n", "- **Tests**: ```test_most_common.py: test_get_top_noun_tags(),test_get_top_verb_tags()```" ] }, { "cell_type": "code", "execution_count": 9, "metadata": { "collapsed": true }, "outputs": [], "source": [ "reload(most_common);" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "PROPN [('Bush', 100), ('US', 81), ('Al', 78)]\n", "PUNCT [(',', 1616), ('.', 1588), ('\"', 383)]\n", "ADJ [('other', 46), ('many', 41), ('Indian', 35)]\n", "NOUN [('people', 53), ('time', 48), ('world', 46)]\n", "VERB [('is', 335), ('was', 128), ('have', 110)]\n", "DET [('the', 1926), ('a', 650), ('The', 216)]\n", "ADP [('of', 887), ('in', 738), ('to', 380)]\n", "AUX [('have', 139), ('was', 126), ('has', 124)]\n", "PRON [('I', 251), ('it', 208), ('he', 131)]\n", "PART [('to', 542), (\"'s\", 218), ('not', 172)]\n", "SCONJ [('that', 304), ('if', 56), ('as', 47)]\n", "NUM [('one', 52), ('two', 28), ('2001', 17)]\n", "ADV [('also', 63), ('now', 54), ('when', 53)]\n", "CONJ [('and', 932), ('or', 127), ('but', 88)]\n", "X [('1', 3), ('2', 3), ('3', 3)]\n", "INTJ [('Please', 15), ('please', 3), ('Well', 3)]\n", "SYM [('$', 20), ('-', 13), ('/', 7)]\n" ] } ], "source": [ "# this block uses your code to find the three most common words per tag\n", "counters = most_common.get_tag_word_counts(TRAIN_FILE)\n", "for tag,tag_ctr in counters.items():\n", " print (tag,tag_ctr.most_common(3))" ] }, { "cell_type": "markdown", "metadata": { "collapsed": true }, "source": [ "# 2. Tagging as classification \n", "\n", "Now you will implement part-of-speech tagging via classification.\n", "\n", "Tagging quality is evaluated using evalTagger, which takes three arguments:\n", "- a tagger, which is a **function** taking a list of words and a tagset as arguments and returns the predicted tags for the words\n", "- an output filename\n", "- a test file\n", "\n", "You will want to use lambda expressions to create the first argument for the `eval_tagger(..)` function, as shown below.\n", "Here's how it works. I provide a tagger that labels everything as a noun." ] }, { "cell_type": "code", "execution_count": 11, "metadata": { "collapsed": true }, "outputs": [], "source": [ "reload(tagger_base);" ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "0.1527613022274944\n" ] } ], "source": [ "# here is a tagger that just tags everything as a noun\n", "noun_tagger = lambda words, alltags : ['NOUN' for word in words]\n", "\n", "confusion = tagger_base.eval_tagger(noun_tagger,'nouns.preds',all_tags=all_tags)\n", "print (scorer.accuracy(confusion))" ] }, { "cell_type": "markdown", "metadata": { "collapsed": true }, "source": [ "** Deliverable 2.1 ** Classification-based tagging. (* 0.5 points *)\n", "\n", "Now do the same thing as above, but building your tagger *as a classifier.* To do this, implement `make_classifier_tagger()` in `tagger_base.py`. \n", "\n", "- **Input**: defaultdict of weights\n", "- **Output**: return a function that takes in (list of word tokens, list of all possible tags) $\\rightarrow$ tags for each word\n", "\n", "The function that you output should create the base-features for each token (**use the OFFSET and the TOKEN itself as base-features**) and then use your `clf_base.predict()` function from pset 1. You are free to edit the `clf_base.predict()` function if you don't think you got it right in pset 1.\n", "- **Tests**: ```test_classifier_tagger.py:test_classifier()```\n" ] }, { "cell_type": "code", "execution_count": 13, "metadata": { "collapsed": true }, "outputs": [], "source": [ "reload(tagger_base);\n", "reload(clf_base);\n", "from gtnlplib.constants import OFFSET #OFFSET token is for each tag" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- Now create a tagger with weights that predict every token to be a NOUN. \n", "The function `get_noun_weights` is already implemented for you." ] }, { "cell_type": "code", "execution_count": 14, "metadata": { "collapsed": true }, "outputs": [], "source": [ "classifier_noun_tagger = tagger_base.make_classifier_tagger(most_common.get_noun_weights())" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "confusion = tagger_base.eval_tagger(classifier_noun_tagger,'all-nouns.preds',all_tags=all_tags)\n", "print (scorer.accuracy(confusion))" ] }, { "cell_type": "markdown", "metadata": { "collapsed": true }, "source": [ "**Deliverable 2.2** Tagging words by their most common tag. (* 0.5 points *)\n", "\n", "Now build a classifier tagger that tags each word with its most common tag in the training set. To do this, implement `get_most_common_word_weights` in `most_common.py`. \n", "\n", "- **Input**: training file\n", "\n", "- **Output**: defaultdict of weights\n", "\n", "This function should return a set weights such that each word should get the tag that is most frequently associated with it in the training data. If the word does not appear in the training data, the weights should be set so that the tagger outputs the **most common tag** in the training data. For the out of vocabulary words, you need to think on how to set the weights so that you tag them by the most common tag.\n", "\n", "- **Tests**: ```test_classifier.py:test_mcc_tagger_output(), test_mcc_tagger_accuracy()```" ] }, { "cell_type": "code", "execution_count": 16, "metadata": { "collapsed": true }, "outputs": [], "source": [ "reload(most_common);\n", "theta_mc = most_common.get_most_common_word_weights(TRAIN_FILE)" ] }, { "cell_type": "code", "execution_count": 17, "metadata": { "collapsed": true }, "outputs": [], "source": [ "tagger_mc = tagger_base.make_classifier_tagger(theta_mc)" ] }, { "cell_type": "code", "execution_count": 18, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "['PRON', 'AUX', 'AUX', 'NOUN']\n" ] } ], "source": [ "tags = tagger_mc(['They','can','can','fish'],all_tags)\n", "print (tags)" ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "['DET', 'ADJ', 'NOUN', 'DET', 'PROPN', 'PUNCT']\n" ] } ], "source": [ "tags = tagger_mc(['The','old','man','the','boat','.'],all_tags)\n", "print (tags)" ] }, { "cell_type": "markdown", "metadata": { "collapsed": true }, "source": [ "- Now let's run your tagger on the dev data." ] }, { "cell_type": "code", "execution_count": 20, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "0.8111242915513378\n" ] } ], "source": [ "confusion = tagger_base.eval_tagger(tagger_mc,'most-common.preds',all_tags=all_tags)\n", "print (scorer.accuracy(confusion))" ] }, { "cell_type": "markdown", "metadata": { "collapsed": true }, "source": [ "### Naive Bayes as a tagger.\n", " \n", "- You can use your Naive Bayes classifier form pset1 to set the weights for the classifier tagger. We have added in a helper function `naive_bayes.get_nb_weights(..)`. Make sure to retain this function, when you copy your code.\n", "- If you don't think you got it right in pset1, you are free to change it now. If you got it right, then just examine the performance of naive bayes as tagger in the following blocks. There is no test or deliverable for this part.\n", "- Note that, for text classification, we had a bag of words feature vector and label for each document. For POS tagging, in order to estimate the weights for the classifier tagger, we will consider each token to be its own document. The following helper code converts the dataset to token level bag-of-words feature vector and labels.\n" ] }, { "cell_type": "code", "execution_count": 21, "metadata": { "collapsed": true }, "outputs": [], "source": [ "reload(naive_bayes);\n", "nb_weights = naive_bayes.get_nb_weights(TRAIN_FILE, .01)" ] }, { "cell_type": "markdown", "metadata": { "collapsed": true }, "source": [ "- This gives weights for each tag-word pair that represent $\\log P(word \\mid tag)$" ] }, { "cell_type": "code", "execution_count": 22, "metadata": { "collapsed": true }, "outputs": [], "source": [ "# obtaining vocab of words\n", "vocab = set([word for tag,word in nb_weights.keys() if word is not constants.OFFSET])" ] }, { "cell_type": "code", "execution_count": 23, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "6912\n" ] } ], "source": [ "print (len(vocab))" ] }, { "cell_type": "code", "execution_count": 24, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "0.9999999999999061\n", "1.0000000000000797\n", "0.9999999999999127\n" ] } ], "source": [ "print (sum(np.exp(nb_weights[('ADJ',word)]) for word in vocab))\n", "print (sum(np.exp(nb_weights[('NOUN',word)]) for word in vocab))\n", "print (sum(np.exp(nb_weights[('PUNCT',word)]) for word in vocab))" ] }, { "cell_type": "markdown", "metadata": { "collapsed": true }, "source": [ "- We have zero weights for OOV terms -- think about how this affects the classification here!" ] }, { "cell_type": "code", "execution_count": 25, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "0.0\n" ] } ], "source": [ "print (nb_weights[('ADJ','baaaaaaaaad')])" ] }, { "cell_type": "code", "execution_count": 26, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "-2.1314729713701404\n", "-3.097798554482699\n", "-2.8092821056787987\n" ] } ], "source": [ "print (nb_weights[('VERB',constants.OFFSET)])\n", "print (nb_weights[('ADV',constants.OFFSET)])\n", "print (nb_weights[('PRON',constants.OFFSET)])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- Offsets should correspond to log-probabilities $\\log P(y)$ such that $\\sum_y P(y) = 1$." ] }, { "cell_type": "code", "execution_count": 27, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "1.0000000000000002" ] }, "execution_count": 27, "metadata": {}, "output_type": "execute_result" } ], "source": [ "sum(np.exp(nb_weights[(tag,constants.OFFSET)]) for tag in all_tags)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- Now let us look at the accuracy of our naive_bayes tagger." ] }, { "cell_type": "code", "execution_count": 28, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "0.8104652695400026\n" ] } ], "source": [ "confusion = tagger_base.eval_tagger(tagger_base.make_classifier_tagger(nb_weights),'nb-simple.preds')\n", "dev_acc = scorer.accuracy(confusion)\n", "print (dev_acc)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- Just as good as the heuristic tagger from above." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# 3. Viterbi Algorithm\n", "\n", "In this section you will implement the Viterbi algorithm in **PyTorch**. To get warmed up, let's work out an example by hand. For simplicity, there are only two tags, **N**OUN and **V**ERB. Here are the parameters:\n", "\n", "| | Value |\n", "| ------------- |:-------------:|\n", "| $\\log P_E(\\cdot|N)$ | they: -1, can: -3, fish: -3 |\n", "| $\\log P_E(\\cdot|V)$ | they: -11, can: -2, fish: -4 |\n", "| $\\log P_T(\\cdot|N)$ | N: -5, V: -2, END: -2 |\n", "| $\\log P_T(\\cdot|V)$ | N: -1, V: -3, END: -3 |\n", "| $\\log P_T(\\cdot|\\text{START})$ | N :-1, V :-2 |\n", "\n", "where $P_E(\\cdot|\\cdot)$ is the emission probability and $P_T(\\cdot|\\cdot)$ is the transition probability.\n", " \n", "- In class we discussed the sentence *They can fish*. \n", "- Now work out a more complicated example: \"*They can can fish*\", where the second \"*can*\" refers to the verb of putting things into cans.\n", " \n", "** Deliverable 3.1 ** Work out the trellis by hand, and fill in the table in the file **```text-answers.md```** (*0.5 points*)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Implementing Viterbi ##\n", "\n", "Here are some predefined weights, corresponding to the weights from the problem 3.1." ] }, { "cell_type": "code", "execution_count": 29, "metadata": { "collapsed": true }, "outputs": [], "source": [ "START_TAG = constants.START_TAG\n", "END_TAG = constants.END_TAG\n", "UNK = constants.UNK\n", "\n", "nb_weights={('NOUN','they'):-1,\\\n", " ('NOUN','can'):-3,\\\n", " ('NOUN','fish'):-3,\\\n", " ('VERB','they'):-11,\\\n", " ('VERB','can'):-2,\\\n", " ('VERB','fish'):-4,}\n", "hmm_trans_weights={('NOUN','NOUN'):-5,\\\n", " ('VERB','NOUN'):-2,\\\n", " (END_TAG,'NOUN'):-2,\\\n", " ('NOUN','VERB'):-1,\\\n", " ('VERB','VERB'):-3,\\\n", " (END_TAG,'VERB'):-3,\\\n", " ('NOUN',START_TAG):-1,\\\n", " ('VERB',START_TAG):-2}" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "** Deliverable 3.2 ** Complete the ```hmm.compute_weights_variables(...)``` function in `hmm.py` file (*0.5 points*) The function should basically convert the weights to respective pytorch variables .\n", "\n", "- **Inputs** :\n", " - `nb_weights`: emission_weights: dictionary of weights\n", " - `hmm_trans_weights`: dictionary of weights\n", " - `vocab`: list of all the words\n", " - `word_to_ix`: a dictionary that maps each word in the vocab to a unique index. **Does not have the OFFSET_TOKEN.**\n", " - `tag_to_ix`: a dictionary that maps each tag (including the `START_TAG` and the `END_TAG`) to a unique index. \n", "\n", "- **Outputs** : returns two torch Variables\n", " - `emission_probs`: torch Variable of a matrix of size `Vocab x Tagset_size`: \n", " such that for a specific weight say `(word1, tag1):value` would result in\n", " `emission_probs[word_to_ix[word1]][tag_to_ix[tag1]]=value`, else a zero. Also, make sure to set weights such that `START_TAG` and `END_TAG` cannot generate any word. **Make sure to ignore the OFFSET weights that might be present in the nb_weights. Consider the words only in your word_to_ix.**\n", " \n", " - `tag_transition_probs`: torch Variable of a matrix of size `Tagset_size x Tagset_size`: \n", " such that for a specific feature say `(tag1, tag2):value` \n", " where tag1 is my succeeding tag, tag2 is my current tag. \n", " This would result in `tag_transition_probs[tag_to_ix[tag1]][tag_to_ix[tag2]]=value`. \n", " Also ensure to set the other weights such that there are no illegal transitions \n", " (like from some tag to START_TAG -or- from END_TAG to some other tag) \n", "\n", "- **Tests**: ```test_viterbi.py: test_compute_hmm_weights_variables()```\n", "\n", "Hint: Use `-np.inf` as weights for illegal transitions." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- Below, observe that we are calculating `tag_to_ix, ix_to_tag, word_to_ix`. These are useful to access a particular emission score for a particular token and a tag. Look through the variables: tag_transition_probs and emission_probs below and it should be clear." ] }, { "cell_type": "code", "execution_count": 30, "metadata": { "collapsed": true }, "outputs": [], "source": [ "reload(hmm);\n", "word_to_ix={'they':0, 'can':1, 'fish':2, UNK:3}\n", "tag_to_ix = {START_TAG:0, 'NOUN':1, 'VERB':2, END_TAG:3}\n", "ix_to_tag = {v:k for k,v in tag_to_ix.items()}\n", "all_tags = [START_TAG, 'NOUN', 'VERB', END_TAG]\n", "words = ['they', 'can', 'fish']\n", "vocab = ['they','can','fish',UNK]\n", "# note that we are also including an UNK token: this will be helpful later" ] }, { "cell_type": "code", "execution_count": 31, "metadata": { "collapsed": true }, "outputs": [], "source": [ "emission_probs, tag_transition_probs = hmm.compute_weights_variables(nb_weights, hmm_trans_weights, vocab, \n", " word_to_ix, tag_to_ix)" ] }, { "cell_type": "code", "execution_count": 32, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Variable containing:\n", "-inf -inf -inf -inf\n", " -1 -5 -1 -inf\n", " -2 -2 -3 -inf\n", "-inf -2 -3 -inf\n", "[torch.FloatTensor of size 4x4]\n", "\n" ] } ], "source": [ "print (tag_transition_probs)\n", "# tag_transition_probs[0] corresponds to scores for START_TAG from START_TAG, NOUN, VERB, END_TAG\n", "# tag_transition_probs[1] corresponds to scores for NOUN from START_TAG, NOUN, VERB, END_TAG\n", "# tag_transition_probs[2] corresponds to scores for VERB from START_TAG, NOUN, VERB, END_TAG\n", "# tag_transition_probs[3] corresponds to scores for END_TAG from START_TAG, NOUN, VERB, END_TAG" ] }, { "cell_type": "code", "execution_count": 33, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Variable containing:\n", "-inf -1 -11 -inf\n", "-inf -3 -2 -inf\n", "-inf -3 -4 -inf\n", "-inf 0 0 -inf\n", "[torch.FloatTensor of size 4x4]\n", "\n" ] } ], "source": [ "print (emission_probs)\n", "# emission_probs[0] corresponds to scores for the token 'they' for START_TAG, NOUN, VERB, END_TAG\n", "# emission_probs[1] corresponds to scores for the token 'can' for START_TAG, NOUN, VERB, END_TAG\n", "# emission_probs[2] corresponds to scores for the token 'fish' for START_TAG, NOUN, VERB, END_TAG\n", "# emission_probs[2] corresponds to scores for the token 'UNK' for START_TAG, NOUN, VERB, END_TAG" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- we will be using these emission scores as inputs for each token in the input in the following function: ```viterbi_step()```." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Deliverable 3.3** The Viterbi recurrence. (*0.5 point*)\n", "\n", "Implement `viterbi_step` in `gtnlplib/viterbi.py`. This is the method that will compute the best path score and corresponding back pointer for a particular token in the sentence for all possible tags, which you will later call from the main viterbi routine. You will also be using this later for the `Bi-LSTM_CRF` model.\n", "\n", "### Inputs\n", "- `all_tags`: list of all tags: includes both the `START_TAG` and the `END_TAG`\n", "- `tag_to_ix`: a dictionary that maps each tag (including the `START_TAG` and the `END_TAG`) to a unique index: this is useful to access the respective tag transition scores from the tag_transition_probs variable.\n", "- `cur_tag_scores`: pytorch Variable that contains the local emission score for each tag for the current token in the sentence.\n", " - `cur_tag_scores` size is : `[ len(all_tags) ] `\n", "- `transition_scores`: pytorch Variable that contains the `tag_transition_scores`. \n", " - `transition_scores` size is : `[ len(all_tags) x len(all_tags) ]` \n", "- `prev_scores`: pytorch Variable that contains the scores for each tag for the previous token in the sentence.\n", " - `prev_scores` size is : `[ 1 x len(all_tags) ] `\n", "\n", "### Outputs\n", "- `viterbivars`: a pytorch Variable that contains the global scores for each tag for the current token in the sentence\n", "- `bptrs`: a list of idx that contains the best_previous_tag for each tag for the current token in the sentence\n", "\n", "### Tests\n", "- ```test_viterbi.py: test_viterbi_step_init()``` \n", "\n", "There are a lot of inputs, but the code itself will not be very complex. Make sure you understand what each input represents before starting to write a solution.\n", "\n", "**Do not convert the pytorch variables into numpy. You will be using this function in BiLSTM-CRF model later on and you need the computation graph to be intact in order to backpropagate.**" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- Consider the sentence: `'they can can fish'`\n", "- Let us observe the viterbi scores at each of the tokens 'they', 'can', 'can', 'fish'.\n", "- We will walk through this example and all along: these scores should match with the scores you obtained when you worked it out by hand.\n", "- **Please note the dimensions of the tensors below**" ] }, { "cell_type": "code", "execution_count": 34, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Variable containing:\n", " 0 -inf -inf -inf\n", "[torch.FloatTensor of size 1x4]\n", "\n" ] } ], "source": [ "reload(viterbi);\n", "initial_vec = np.full((1, len(all_tags)),-np.inf) \n", "initial_vec[tag_to_ix[START_TAG]][0] = 0 #setting all the score to START_TAG\n", "prev_scores = viterbi.get_torch_variable(initial_vec)\n", "# these are the previous scores for each_tag: START_TAG, NOUN, VERB, END_TAG\n", "print (prev_scores)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- The block above says that the only possible previous tag at $m=1$ is `START_TAG`\n", "- Now let us look at the tag scores for the first token 'they'" ] }, { "cell_type": "code", "execution_count": 35, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "['--START--', 'NOUN', 'VERB', '--END--']\n" ] } ], "source": [ "# make sure both START_TAG and END_TAG is included in all_tags\n", "print (all_tags)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- Carefully observe all the inputs to the `viterbi_step(..)` function here.\n", " - `all_tags`: is the list of all possible tags here\n", " - `tag_to_ix`: a mapping from tags to unique ids: this is useful to access the respective tag transition scores from the `tag_transition_probs` variable.\n", " - `cur_tag_scores`: observe that from previous section: `emission_probs` indicates the emission scores for each tag for each word they: since we will be tagging the word 'they' right now in our example: we will be using ```emission_probs[0]```: note that '0' is the id for our word 'they'.Thus, we send in `emission_probs[0]` as our cur_tag_scores\n", " - `tag_transition_probs`: tag transition probabilities\n", " - `prev_scores`: prev_scores obtained: we have initially calculated these scores above such that the `START_TAG` has all the score." ] }, { "cell_type": "code", "execution_count": 36, "metadata": { "collapsed": true }, "outputs": [], "source": [ "reload(viterbi);\n", "viterbivars, bptrs = viterbi.viterbi_step(all_tags, tag_to_ix, \n", " emission_probs[0], \n", " tag_transition_probs,\n", " prev_scores)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- The following are the scores obtained for each tag for the word token 'they' and the backpointer refers to that particular previous tag which resulted in that score." ] }, { "cell_type": "code", "execution_count": 37, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "tag: --START-- score: [-inf] back-pointer-tag: --START--\n", "tag: NOUN score: [-2.] back-pointer-tag: --START--\n", "tag: VERB score: [-13.] back-pointer-tag: --START--\n", "tag: --END-- score: [-inf] back-pointer-tag: --START--\n" ] } ], "source": [ "scores = viterbivars\n", "for k,v in tag_to_ix.items():\n", " print ('tag: ',k, ' score: ',scores[v].data.numpy(), ' back-pointer-tag: ', \n", " ix_to_tag[bptrs[v]])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- `NOUN` has the highest score for the current tag, and its backpointer is to `START_TAG`\n", "- Now, let us look at the scores for the tags for the second token 'can'. Send in `emission_probs[1]` as our `current_tag_scores`, and update `prev_scores` to be the scores obtained for $m=1$." ] }, { "cell_type": "code", "execution_count": 38, "metadata": { "collapsed": true, "scrolled": true }, "outputs": [], "source": [ "prev_scores = viterbi.get_torch_variable([-np.inf, -2, -13, -np.inf])\n", "viterbivars, bptrs = viterbi.viterbi_step(all_tags, tag_to_ix,\n", " emission_probs[1],\n", " tag_transition_probs,\n", " prev_scores)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- The following are the scores obtained for each tag for the word token 'can' and its respective back_pointer tag." ] }, { "cell_type": "code", "execution_count": 39, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "tag: --START-- score: [-inf] back-pointer-tag: --START--\n", "tag: NOUN score: [-10.] back-pointer-tag: NOUN\n", "tag: VERB score: [-6.] back-pointer-tag: NOUN\n", "tag: --END-- score: [-inf] back-pointer-tag: --START--\n" ] } ], "source": [ "scores = viterbivars\n", "for k,v in tag_to_ix.items():\n", " print ('tag: ',k, ' score: ',scores[v].data.numpy(), ' back-pointer-tag: ',\n", " ix_to_tag[bptrs[(all_tags).index(k)]])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- Now, Below, let us look at the scores for the tags for the third token 'can'. So, now we send in `emission_probs[1]` as our `current_tag_scores` and we update `prev_scores` to be the scores obtained for the previous token 'can'." ] }, { "cell_type": "code", "execution_count": 40, "metadata": { "collapsed": true }, "outputs": [], "source": [ "prev_scores = viterbi.get_torch_variable([-np.inf, -10, -6, -4]) \n", "viterbivars, bptrs = viterbi.viterbi_step(all_tags, tag_to_ix,\n", " emission_probs[1],\n", " tag_transition_probs,\n", " prev_scores)" ] }, { "cell_type": "code", "execution_count": 41, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "tag: --START-- score: [-inf] back-pointer-tag: --START--\n", "tag: NOUN score: [-10.] back-pointer-tag: VERB\n", "tag: VERB score: [-11.] back-pointer-tag: VERB\n", "tag: --END-- score: [-inf] back-pointer-tag: --START--\n" ] } ], "source": [ "scores = viterbivars\n", "for k,v in tag_to_ix.items():\n", " print ('tag: ',k, ' score: ',scores[v].data.numpy(), ' back-pointer-tag: ',\n", " ix_to_tag[bptrs[(all_tags).index(k)]])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ " - Now, let us look at the scores for the tags for the last token 'fish', So, now we send in `emission_probs[2]` as our `current_tag_scores` and we update `prev_scores` to be the scores obtained above for the previous token 'can'" ] }, { "cell_type": "code", "execution_count": 42, "metadata": { "collapsed": true }, "outputs": [], "source": [ "prev_scores = viterbi.get_torch_variable([-np.inf, -10, -11, -np.inf])\n", "viterbivars, bptrs = viterbi.viterbi_step(all_tags, tag_to_ix,\n", " emission_probs[2],\n", " tag_transition_probs,\n", " prev_scores)" ] }, { "cell_type": "code", "execution_count": 43, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "tag: --START-- score: [-inf] back-pointer-tag: --START--\n", "tag: NOUN score: [-15.] back-pointer-tag: VERB\n", "tag: VERB score: [-16.] back-pointer-tag: NOUN\n", "tag: --END-- score: [-inf] back-pointer-tag: --START--\n" ] } ], "source": [ "scores = viterbivars\n", "for k,v in tag_to_ix.items():\n", " print ('tag: ',k, ' score: ',scores[v].data.numpy(), ' back-pointer-tag: ',\n", " ix_to_tag[bptrs[(all_tags).index(k)]])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Deliverable 3.4** Build the Viterbi trellis. (*0.5 points*)\n", "\n", "This function should compute the `best_path` and the `path_score`. This function takes in the `emission_scores` for each particular token in the sentence, the `tag_transitions_weight` and returns the best set of tags for that particular sequence. Use `viterbi_step` to implement `build_trellis` in `viterbi.py` in Pytorch. \n", "\n", "This function should take:\n", "- **Inputs**:\n", " - `all_tags`: a list of all tags: includes START_TAG and END_TAG\n", " - `tag_to_ix`: a dictionary that maps each tag to a unique id.\n", " - `cur_tag_scores`: a list of pytorch Variables where each contains the local emission score for each tag for that particular token in the sentence, len(cur_tag_scores) will be equal to len(words): each pytorch variables size would be equal to len(all_tags) indicating the score for each_tag.\n", " - `transition_scores`: pytorch Variable (a matrix) that contains the tag_transition_scores\n", "\n", "- **Outputs**:\n", " - `path_score`: the score for the best_path\n", " - `best_path`: the actual best_path, which is the list of tags for each token: exclude the `START_TAG` and `END_TAG` here. \n", " \n", "- **Tests**: ```test_viterbi.py: test_trellis_score(), test_build_trellis()```\n", "First, make sure to pass the ```test_trellis_score()``` test and then move on to the ```test_build_trellis()``` test.\n", "\n", "** Note that for the input cur_tag_scores: we are sending in a list of pytorch variables: one for each token in the sentence to be tagged **" ] }, { "cell_type": "code", "execution_count": 44, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "['--START--', 'NOUN', 'VERB', '--END--']\n" ] } ], "source": [ "# make sure START_TAG and END_TAG are in all_tags\n", "print (all_tags)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- consider the same sentence as above: 'they can can fish'" ] }, { "cell_type": "code", "execution_count": 45, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "['they', 'can', 'can', 'fish']\n", "{'they': 0, 'can': 1, 'fish': 2, '': 3}\n" ] } ], "source": [ "words = 'they can can fish'.split()\n", "print (words)\n", "print (word_to_ix)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- Below, we create `cur_tag_scores` using the `emission_probs` for each word in the sentence `'they can can fish'`" ] }, { "cell_type": "code", "execution_count": 46, "metadata": { "collapsed": true }, "outputs": [], "source": [ "#preparing cur_tag_scores for the above sentence 'they can can fish'\n", "cur_tag_scores=[emission_probs[0],emission_probs[1],emission_probs[1],emission_probs[2]];" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- Complete the code to obtain the correct path_score initially and then use the backpointers to obtain the best_path. \n", "- Observe the inputs we are sending in for our example: 'they can can fish'\n", " - `all_tags`: list of all tags including the `START_TAG` and `END_TAG`\n", " - `tag_to_ix`: a mapping from tags to their unique ids\n", " - `cur_tag_scores`: a list of pytorch variables: where each one is the score for each tag for a particular token. We send in these scores for each token in the sentence.\n", " - `tag_transition_probs`: tag transition probabilities" ] }, { "cell_type": "code", "execution_count": 47, "metadata": { "collapsed": true }, "outputs": [], "source": [ "reload(viterbi);" ] }, { "cell_type": "code", "execution_count": 48, "metadata": { "collapsed": true }, "outputs": [], "source": [ "path_score, best_path = viterbi.build_trellis(all_tags, \n", " tag_to_ix, \n", " cur_tag_scores, \n", " tag_transition_probs)" ] }, { "cell_type": "code", "execution_count": 49, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "['NOUN', 'VERB', 'VERB', 'NOUN']\n", "-17.0\n" ] } ], "source": [ "print (best_path)\n", "print (path_score.data[0])" ] }, { "cell_type": "code", "execution_count": 50, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "['NOUN', 'VERB', 'VERB', 'NOUN', 'VERB', 'NOUN'] [-25.]\n" ] } ], "source": [ "sentence = ['they','can','can','can','can','fish']\n", "cur_tag_scores = [emission_probs[word_to_ix[w]] for w in sentence]\n", "\n", "path_score, best_path = viterbi.build_trellis(all_tags, \n", " tag_to_ix, \n", " cur_tag_scores, \n", " tag_transition_probs)\n", "print (best_path, path_score.data.numpy())" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# 4. Hidden Markov Model: Estimation\n", "\n", "You will now implement the estimatation for a hidden Markov model.\n", "\n", "We'll start with the tag transitions.\n", "\n", "**Deliverable 4.1** (*0.5 points* for 4650, *0.25 points* for 7650) Complete the function `most_common.get_tag_trans_counts()`. This function should get the tag transition counts from the each tag to all possible tags. Don't forget to add the transitions from the `START_TAG` and the transitions from the `END_TAG`.\n", "\n", "You should use the `preproc.conll_seq_generator()` function. \n", "\n", "- **Inputs**: `trainfile`, name of file containing training data\n", "- **Outputs**: a dictionary where keys are current tags and values are counters of succeeding tags.\n", "- **Tests**: ```test_hmm_trans_counts.py: test_tag_trans_counts()``` \n" ] }, { "cell_type": "code", "execution_count": 51, "metadata": { "collapsed": true }, "outputs": [], "source": [ "reload(most_common);\n", "tag_trans_counts = most_common.get_tag_trans_counts(TRAIN_FILE)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This function returns a dict of counters, where the keys are tags.\n", "\n", "Each counter is the frequency of tags following a given tag, e.g.:" ] }, { "cell_type": "code", "execution_count": 52, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Counter({'NOUN': 1753, 'ADJ': 866, 'PROPN': 562, 'VERB': 65, 'PUNCT': 61, 'ADV': 59, 'NUM': 55, 'DET': 29, 'ADP': 29, 'PRON': 10, 'AUX': 7, 'X': 7, 'SYM': 2})\n", "Counter({'PRON': 422, 'PROPN': 327, 'DET': 284, 'ADV': 172, 'ADP': 142, 'NOUN': 108, 'PUNCT': 80, 'CONJ': 79, 'SCONJ': 75, 'ADJ': 67, 'VERB': 64, 'X': 51, 'NUM': 42, 'AUX': 42, 'INTJ': 31, 'SYM': 9, 'PART': 5})\n" ] } ], "source": [ "print (tag_trans_counts['DET'])\n", "print (tag_trans_counts[START_TAG])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Deliverable 4.2** Estimate transition log-probabilities for an HMM. (*0.5 points* for 4650, *0.25 points* for 7650)\n", "\n", "Implement `compute_transition_weights` in `hmm.py`. This function should return a dictionary of weights such that ```weights[(tag2,tag1)]``` = indicates the weights for transitions from `tag1` $\\rightarrow$ `tag2`. These weights will be used later for the Viterbi Tagger.\n", "\n", "### Inputs\n", "- Transition counts (generated from `get_tag_trans_counts`)\n", "- Smoothing\n", "\n", "### Outputs\n", "- Defaultdict with weights for transition features, in the form $[(y_m,y_{m-1})]$\n", "\n", "### Tests\n", "```test_hmm_trans.py: test_hmm_trans_weights_sum_to_one(), test_hmm_trans_weights_exact_vals() ``` \n", "\n", "Hints: \n", "\n", "- Don't forget to assign smoothed probabilities to transitions which do not appear in the counts. \n", "- Do not assign probabilities for transitions to the `START_TAG`, which can only come first. This will also affect your computation of the denominator, since you are not smoothing the probability of transitions to the `START_TAG`.\n", "- Don't forget to assign probabilities to transitions to the `END_TAG`; this too will affect your denominator.\n", "- As always, probabilities should sum to one (this time conditioned on the previous tag)" ] }, { "cell_type": "code", "execution_count": 53, "metadata": { "collapsed": true }, "outputs": [], "source": [ "reload(hmm);" ] }, { "cell_type": "code", "execution_count": 54, "metadata": { "collapsed": true }, "outputs": [], "source": [ "hmm_trans_weights = hmm.compute_transition_weights(tag_trans_counts,.001)" ] }, { "cell_type": "code", "execution_count": 55, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "108 -2.91877097316097\n", "64 -3.44201275126398\n", "65 -3.987548280583508\n", "0 -15.069706213958325\n", "1753 -0.6928664796044606\n", "0 -inf\n", "-inf\n" ] } ], "source": [ "print (tag_trans_counts[START_TAG]['NOUN'], hmm_trans_weights[('NOUN',START_TAG)])\n", "print (tag_trans_counts[START_TAG]['VERB'], hmm_trans_weights[('VERB',START_TAG)])\n", "print (tag_trans_counts['DET']['VERB'], hmm_trans_weights[('VERB','DET')])\n", "print (tag_trans_counts['DET']['INTJ'], hmm_trans_weights[('INTJ','DET')])\n", "print (tag_trans_counts['DET']['NOUN'], hmm_trans_weights[('NOUN','DET')])\n", "print (tag_trans_counts['VERB'][START_TAG], hmm_trans_weights[(START_TAG,'VERB')])\n", "#print (tag_trans_counts[END_TAG]['VERB']) # will throw key error\n", "print (hmm_trans_weights[('VERB',END_TAG)])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "These log-probabilities should normalize to when summing over $y_m$" ] }, { "cell_type": "code", "execution_count": 56, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "0.9999999999999999\n", "1.0000000000000007\n", "0.9999999999999997\n" ] } ], "source": [ "#calculating all tags here, we also add END_TAG here.\n", "all_tags = sorted(list(tag_trans_counts.keys()) + [END_TAG])\n", "print (sum(np.exp(hmm_trans_weights[(tag,'NOUN')]) for tag in all_tags))\n", "print (sum(np.exp(hmm_trans_weights[(tag,'SYM')]) for tag in all_tags))\n", "print (sum(np.exp(hmm_trans_weights[(tag,'ADJ')]) for tag in all_tags))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- **Now let us compute the weight variables for the whole dataset**\n", "- So, we recalculate them below." ] }, { "cell_type": "code", "execution_count": 57, "metadata": { "collapsed": true }, "outputs": [], "source": [ "#recalculating nb_weights for the whole dataset\n", "nb_weights = naive_bayes.get_nb_weights(TRAIN_FILE, .01)" ] }, { "cell_type": "code", "execution_count": 58, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "{'--END--': 0, '--START--': 1, 'ADJ': 2, 'ADP': 3, 'ADV': 4, 'AUX': 5, 'CONJ': 6, 'DET': 7, 'INTJ': 8, 'NOUN': 9, 'NUM': 10, 'PART': 11, 'PRON': 12, 'PROPN': 13, 'PUNCT': 14, 'SCONJ': 15, 'SYM': 16, 'VERB': 17, 'X': 18}\n" ] } ], "source": [ "# recalculating tag_to_ix={}\n", "tag_to_ix={}\n", "for tag in list(all_tags):\n", " if tag not in tag_to_ix:\n", " tag_to_ix[tag]=len(tag_to_ix)\n", "print (tag_to_ix)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- **Note about OOV's**: We provide a helper function to calculate the `vocab` as shown below. We add an `UNK` token to the `vocab`. This is useful because, when we don't find a token's emission weight, we choose the `UNK` tokens weight and proceed with our tagging." ] }, { "cell_type": "code", "execution_count": 59, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "6913\n" ] } ], "source": [ "# recalculating vocab for the whole dataset now. # we also add an UNK token to the vocab here\n", "reload(most_common);\n", "vocab, word_to_ix = most_common.get_word_to_ix(TRAIN_FILE) #obtains all the words in the file\n", "print (len(vocab))" ] }, { "cell_type": "code", "execution_count": 60, "metadata": { "collapsed": true }, "outputs": [], "source": [ "emission_probs, tag_transition_probs = hmm.compute_weights_variables(nb_weights, hmm_trans_weights, \n", " vocab, word_to_ix, tag_to_ix)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Deliverable 4.3** (*0.5 points*)\n", "\n", "We can now combine `Viterbi` and the `HMM` weights to compute the tag sequence for the example sentence. Make sure your implementation passes the test for this deliverable, and **explain (in `text-answers.md`) whether you think these predicted tags are correct**, based on your understanding of the universal part-of-speech tag set.\n", "\n", "- **Tests**: ```test_hmm.py: test_hmm_on_example_sentence()```" ] }, { "cell_type": "code", "execution_count": 61, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "['--END--', '--START--', 'ADJ', 'ADP', 'ADV', 'AUX', 'CONJ', 'DET', 'INTJ', 'NOUN', 'NUM', 'PART', 'PRON', 'PROPN', 'PUNCT', 'SCONJ', 'SYM', 'VERB', 'X']\n" ] } ], "source": [ "# make sure all_tags has END_TAG\n", "print (all_tags)" ] }, { "cell_type": "code", "execution_count": 62, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "(Variable containing:\n", " -32.4456\n", " [torch.FloatTensor of size 1], ['PRON', 'AUX', 'AUX', 'NOUN', 'PUNCT'])" ] }, "execution_count": 62, "metadata": {}, "output_type": "execute_result" } ], "source": [ "reload(viterbi);\n", "viterbi.build_trellis(all_tags,\n", " tag_to_ix,\n", " [emission_probs[word_to_ix[w]] for w in ['they', 'can', 'can', 'fish','.']], \n", " tag_transition_probs)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Deliverable 4.4** (*0.5 points*)\n", "\n", "- Run your HMM tagger on the dev data and test data, using the code blocks below.\n", "- **Tests**: ```test_hmm.py: test_hmm_dev_accuracy(), test_hmm_test_accuracy()```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- Observe that, based on our definition of the viterbi function, we need to send in two sets of important scores to the `build_trellis()` function\n", " - cur_tag_scores: a list of emission scores for each tag for each token in the sentence\n", " - tag_transition_probs: tag transition scores\n", "- When using the `HMM` with `Viterbi` Tagger: we have calculated the `cur_tag_scores` using the `naive_bayes_weights` and `tag_transition_probs` in 4.1.\n", "- As I have already mentioned above, for `cur_tag_scores`, we are sending in a list of pytorch variables: one for each token in the sentence to be tagged\n", "- Below, in the tagger that we create: we first calculate the set of `cur_tag_scores` for the words in the sentence and then send them in. \n" ] }, { "cell_type": "code", "execution_count": 63, "metadata": { "collapsed": true }, "outputs": [], "source": [ "reload(viterbi);" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- Observe the way `cur_tag_scores` is computed in the loop below: \n", " - For each particular word in a sentence: we assign the respective emission scores if it is present in our `vocab`, else we assign the emission_scores of an `UNK` token.\n", " - This is repeated everywhere from now on." ] }, { "cell_type": "code", "execution_count": 64, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "From/ADP the/DET AP/NOUN comes/VERB this/DET story/NOUN :/PUNCT \n", "\n", "President/PROPN Bush/PROPN on/ADP Tuesday/PROPN nominated/PROPN two/NUM individuals/NOUN to/PART replace/VERB retiring/DET jurists/NOUN on/ADP federal/ADJ courts/NOUN in/ADP the/DET Washington/PROPN area/NOUN ./PUNCT \n", "\n", "Bush/PROPN nominated/PROPN Jennifer/PROPN M./PART Anderson/VERB for/ADP a/DET 15/NUM -/PUNCT year/NOUN term/NOUN as/ADP associate/NOUN judge/NOUN of/ADP the/DET Superior/ADJ Court/NOUN of/ADP the/DET District/PROPN of/ADP Columbia/PROPN ,/PUNCT replacing/PROPN Steffen/PROPN W./PROPN Graae/PROPN ./PUNCT \n", "\n" ] } ], "source": [ "# this is just for fun\n", "for i,(words,_) in enumerate(preproc.conll_seq_generator(DEV_FILE)):\n", " cur_tag_scores = [emission_probs[word_to_ix[w]] \n", " if w in word_to_ix else emission_probs[word_to_ix[UNK]] for w in words]\n", " \n", " pred_tags = viterbi.build_trellis(all_tags,\n", " tag_to_ix,\n", " cur_tag_scores,\n", " tag_transition_probs)[1]\n", " for word,pred_tag in zip(words,pred_tags):\n", " print (\"%s/%s\"%(word,pred_tag),end=\" \")\n", " print ('\\n')\n", " if i >= 2: break" ] }, { "cell_type": "code", "execution_count": 65, "metadata": { "collapsed": true }, "outputs": [], "source": [ "tagger = lambda words, all_tags : viterbi.build_trellis(all_tags, \n", " tag_to_ix,\n", " [emission_probs[word_to_ix[w]] \n", " if w in word_to_ix \n", " else emission_probs[word_to_ix[UNK]] \n", " for w in words],\n", " tag_transition_probs)[1]\n", "confusion = tagger_base.eval_tagger(tagger,'hmm-dev-en.preds', all_tags)" ] }, { "cell_type": "code", "execution_count": 66, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "0.8409120864636879\n" ] } ], "source": [ "print (scorer.accuracy(confusion))" ] }, { "cell_type": "code", "execution_count": 67, "metadata": { "collapsed": true }, "outputs": [], "source": [ "tagger_base.apply_tagger(tagger,'hmm-te-en.preds',all_tags, testfile=TEST_FILE_HIDDEN)" ] }, { "cell_type": "code", "execution_count": 68, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "0.8402749140893471\n" ] } ], "source": [ "# you don't have en-ud-test.conllu, so you can't run this\n", "te_confusion = scorer.get_confusion(TEST_FILE,'hmm-te-en.preds')\n", "print (scorer.accuracy(te_confusion))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Part-of-Speech Tagging in Norwegian\n", "**Deliverable 4.5** (*0.5 points*)\n", "- Now, let us do part of speech tagging for data in Norwegian language using the Viterbi Tagger.\n", "- **Tests**: ```test_hmm.py: test_nr_hmm_dev_accuracy(), test_nr_hmm_test_accuracy()```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- First, we calculate the `nb_weights`/emission weights for the norwegian language in a similar way as we did for the english language." ] }, { "cell_type": "code", "execution_count": 69, "metadata": { "collapsed": true }, "outputs": [], "source": [ "#recalculating nb_weights for the whole dataset\n", "nb_weights_nr = naive_bayes.get_nb_weights(NR_TRAIN_FILE, .01)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- Now, we calculate the `tag_transition_weights` for the norwegian language" ] }, { "cell_type": "code", "execution_count": 70, "metadata": { "collapsed": true }, "outputs": [], "source": [ "tag_trans_counts_nr = most_common.get_tag_trans_counts(NR_TRAIN_FILE)\n", "hmm_trans_weights_nr = hmm.compute_transition_weights(tag_trans_counts_nr,.001)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- Now, we obtain the `vocab`, `word_to_ix` and `tag_to_ix` below for the norwegian language." ] }, { "cell_type": "code", "execution_count": 71, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "{'--END--': 0, '--START--': 1, 'ADJ': 2, 'ADP': 3, 'ADV': 4, 'AUX': 5, 'CCONJ': 6, 'DET': 7, 'INTJ': 8, 'NOUN': 9, 'NUM': 10, 'PART': 11, 'PRON': 12, 'PROPN': 13, 'PUNCT': 14, 'SCONJ': 15, 'SYM': 16, 'VERB': 17, 'X': 18}\n" ] } ], "source": [ "#Using helper functions to obtain vocab, word_to_ix, tag_to_ix\n", "all_tags_nr = sorted(list(tag_trans_counts_nr.keys()) + [END_TAG])\n", "vocab_nr, word_to_ix_nr = most_common.get_word_to_ix(NR_TRAIN_FILE) #obtains all the words in the vocab\n", "tag_to_ix_nr={}\n", "for tag in list(all_tags_nr):\n", " tag_to_ix_nr[tag]=len(tag_to_ix_nr)\n", "print (tag_to_ix_nr)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- Now, we convert these weights into pytorch variables." ] }, { "cell_type": "code", "execution_count": 72, "metadata": { "collapsed": true }, "outputs": [], "source": [ "emission_probs_nr, tag_transition_probs_nr = hmm.compute_weights_variables(nb_weights_nr, hmm_trans_weights_nr, \n", " vocab_nr, word_to_ix_nr, tag_to_ix_nr)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- Now, we construct a viterbi tagger for the norwegian language using these weights." ] }, { "cell_type": "code", "execution_count": 73, "metadata": { "collapsed": true }, "outputs": [], "source": [ "tagger = lambda words, all_tags : viterbi.build_trellis(all_tags_nr, \n", " tag_to_ix_nr,\n", " [emission_probs_nr[word_to_ix_nr[w]] \n", " if w in word_to_ix_nr \n", " else emission_probs_nr[word_to_ix_nr[UNK]] \n", " for w in words],\n", " tag_transition_probs_nr)[1]" ] }, { "cell_type": "code", "execution_count": 74, "metadata": { "collapsed": true }, "outputs": [], "source": [ "confusion = tagger_base.eval_tagger(tagger,'hmm-dev-nr.preds', all_tags_nr,\n", " trainfile=NR_TRAIN_FILE,\n", " testfile=NR_DEV_FILE)" ] }, { "cell_type": "code", "execution_count": 75, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "0.8615031554790591\n" ] } ], "source": [ "print (scorer.accuracy(confusion))" ] }, { "cell_type": "code", "execution_count": 76, "metadata": { "collapsed": true }, "outputs": [], "source": [ "tagger_base.apply_tagger(tagger,'hmm-te-nr.preds',all_tags_nr, \n", " trainfile=NR_TRAIN_FILE, testfile=NR_TEST_FILE_HIDDEN)" ] }, { "cell_type": "code", "execution_count": 77, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "0.8535171347590828\n" ] } ], "source": [ "# you don't have no_bokmaal-ud-test.conllu, so you can't run this\n", "te_confusion = scorer.get_confusion(NR_TEST_FILE,'hmm-te-nr.preds')\n", "print (scorer.accuracy(te_confusion))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "5. BiLSTM model for Part of Speech Tagging\n", "=========================\n", "\n", "A `BiLSTM` model for part-of-speech tagging takes as input the word embeddings of the tokens in the sentence, and passes them through an `LSTM`. For each token, the hidden state is used as input to a network that computes a score for each tag. A softmax layer then converts these scores to probabilities. This model should be trained end-to-end with the cross-entropy loss function.\n", "\n", "We will be building this `BiLSTM` model as a class using pytorch. Your implementation will include three functions:\n", "\n", "- `BiLSTM.__init__()`: define all the necessary model parameters\n", " 1. The word-embedding matrix, which maps the words to vectors\n", " 2. A BiLSTM Neural Network, which takes the word embeddings for the words as inputs and produces a hidden state for each token.\n", " 3. A one layer feedforward Neural Network, which projects the hidden state to a vector of scores for each tag\n", "- `forward()`: pass the input through the model, obtaining probability distributions over tags\n", " 1. Convert all the words to their word-vectors from the word-embedding matrix\n", " 2. Pass these word-vectors through a BiLSTM to obtain hidden states for the tokens\n", " 3. Pass these hidden states through the feedforward neural network to obtain the probability distributions of tags for each token.\n", "- `cross_entopy_loss()`, the training objective\n", "\n", "The description below provides additional help for each of these functions." ] }, { "cell_type": "code", "execution_count": 78, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "words in the vocabulary: 6900\n", "6899\n" ] } ], "source": [ "# recalculating vocab: obtains the most common 6900 words from the file\n", "vocab, word_to_ix = most_common.get_word_to_ix(TRAIN_FILE, 6900)\n", "print ('words in the vocabulary: ',len(word_to_ix))\n", "print (word_to_ix[UNK])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- updating `tag_to_ix` and `all_tags` to remove `START_TAG` and `END_TAG`: these labels are not necessary in tagging with `BiLSTM`" ] }, { "cell_type": "code", "execution_count": 79, "metadata": { "collapsed": true }, "outputs": [], "source": [ "if START_TAG in all_tags:\n", " all_tags.remove(START_TAG)\n", "if END_TAG in all_tags:\n", " all_tags.remove(END_TAG)\n", "tag_to_ix={}\n", "for tag in all_tags:\n", " if tag not in tag_to_ix:\n", " tag_to_ix[tag] = len(tag_to_ix)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- Take a look at the helper functions `preproc.load_data(...)` and `bilstm.prepare_sequence(...)` \n", " - `preproc.load_data(...)`: loads the data into a list of lists\n", " - `bilstm.prepare_sequence(...)` given a sequence of words/tags and the `to_ix` dictionary that maps them to its unique indices: it returns a sequence of its unique indices.\n", "- The function `prepare_sequence()` will be used a lot from now on to convert the input to a `torch.LongTensor` and then send it to the model. " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- Loading Train data for english:" ] }, { "cell_type": "code", "execution_count": 80, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[ 17 759 14 114 451 20 0 278 5 33 901 2020 2]\n", "[ 5 0 3 3 15 1 5 7 1 5 7 7 12]\n" ] } ], "source": [ "reload(preproc);\n", "X_tr, Y_tr = preproc.load_data(TRAIN_FILE)\n", "\n", "print (bilstm.prepare_sequence(X_tr[5],word_to_ix).data.numpy())\n", "print (bilstm.prepare_sequence(Y_tr[5],tag_to_ix).data.numpy())" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- Loading Dev data for english:" ] }, { "cell_type": "code", "execution_count": 81, "metadata": { "collapsed": true }, "outputs": [], "source": [ "X_dv, Y_dv = preproc.load_data(DEV_FILE)\n", "# loading dev data" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- Loading Test data for english:" ] }, { "cell_type": "code", "execution_count": 82, "metadata": { "collapsed": true }, "outputs": [], "source": [ "X_te, Y_te = preproc.load_data(TEST_FILE_HIDDEN)\n", "# loading test data" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Deliverable 5.1**: (*0.5 points*)\n", "- Complete the `__init__()` function in the Class `bilstm.BiLSTM` that defines the model parameters:\n", " - **Inputs**:\n", " - `vocab_size`: vocab size of the model\n", " - `tag_to_ix`: tag_to_ix: a dictionary that maps the tags to its unique id\n", " - `embedding_dim`: embedding dimension for the words\n", " - `hidden_dim`: hidden dimension for the `Bi-LSTM` model\n", " - `embeddings`: embedding matrix of size: vocab_size x embedding_dim (this is to initialize embeddings with pretrained embeddings).\n", " - The function does the following:\n", " - Create an embedding matrix using torch.nn.Embedding of the size vocab \n", " - [check this pytorch_doc_for_embedding](http://pytorch.org/docs/0.3.0/nn.html?highlight=embedding#torch.nn.Embedding)\n", " - Create a Bi-LSTM model with just one layer, and hidden dimension = hidden_dim \n", " - [check this pytorch_doc_for_LSTM](http://pytorch.org/docs/0.3.0/nn.html?highlight=lstm#torch.nn.LSTM)\n", " - Also add a FullyConnected Layer, that would project the hidden state onto the tag space. \n", " - [check this pytorch_doc_for_Linear](http://pytorch.org/docs/0.3.0/nn.html?highlight=linear#torch.nn.Linear)\n", " - Make sure to name the parameters as follows: The unit tests will check for these variables\n", " - `self.word_embeds`\n", " - `self.lstm`\n", " - `self.hidden2tag`\n", " - **Tests**: ```test_bilstm.py: test_dlmodel_init()```\n", " - All you need to do here is to define the model parameters here." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- Below, you can find brief example of how these components are created in Torch. In these examples, the dimensions are arbitrary; you will need to determine the correct dimensions for your model." ] }, { "cell_type": "code", "execution_count": 16, "metadata": { "collapsed": true }, "outputs": [], "source": [ "# this basically defines a matrix of embeddings where the vocab_size=10 and the embedding_dim=10 \n", "word_embeds = nn.Embedding(num_embeddings=10, embedding_dim=10)" ] }, { "cell_type": "code", "execution_count": 17, "metadata": { "collapsed": true }, "outputs": [], "source": [ "# the following line is how you define an LSTM such that the input_size of the vector=10, \n", "# hidden state of each LSTM (forward and backward) =20, num of layers=1, \n", "# bidirectional=True indicating both forward and backward LSTM will be included.\n", "lstm = nn.LSTM(input_size=10, hidden_size = 20, num_layers=1, bidirectional=True)" ] }, { "cell_type": "code", "execution_count": 85, "metadata": { "collapsed": true }, "outputs": [], "source": [ "# the following line is the way to define a Fully Connected Layer with input_dim=40, output_dim=10 and bias=True.\n", "hidden2tag = nn.Linear(in_features=40, out_features=10, bias=True)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- Once, you have defined the parameters of the model: check if you have done it right using the unit test." ] }, { "cell_type": "code", "execution_count": 86, "metadata": { "collapsed": true }, "outputs": [], "source": [ "reload(bilstm);\n", "torch.manual_seed(765);\n", "embedding_dim=30\n", "hidden_dim=30\n", "model = bilstm.BiLSTM(len(word_to_ix),tag_to_ix,embedding_dim, hidden_dim)" ] }, { "cell_type": "code", "execution_count": 87, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "BiLSTM(\n", " (word_embeds): Embedding(6900, 30)\n", " (lstm): LSTM(30, 15, bidirectional=True)\n", " (hidden2tag): Linear(in_features=30, out_features=17)\n", ")" ] }, "execution_count": 87, "metadata": {}, "output_type": "execute_result" } ], "source": [ "model" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Deliverable 5.2** (*0.5 points*)\n", "- Complete the `bilstm.BiLSTM.forward()` function in the `bilstm.BiLSTM` class to obtain the scores for each tag for each of the words in a sentence\n", "- **Input**:\n", " - sentence: a sequence of ids for each word in the sentence\n", "- The function does the following:\n", " - Obtains the embeddings for the input sequence\n", " - passes them through an `LSTM` to get the respective hidden states; use the hidden state initialized in the function.\n", " - projects them onto the tag-space using the FC layer\n", "- Make sure to reshape the embeddings of the words before sending them to the `BiLSTM`. The axes semantics are: `seq_len, mini_batch, embedding_dim`.\n", "- You can use the .view() method to reshape a tensor. You might need to use this, as the neural network components expect their inputs to have a certain shape. [check the pytorch doc on view](http://pytorch.org/docs/0.3.0/tensors.html?highlight=view#torch.Tensor.view)\n", "- **Tests**: ```test_bilstm.py: test_dlmodel_forward()```" ] }, { "cell_type": "code", "execution_count": 88, "metadata": { "collapsed": true }, "outputs": [], "source": [ "# setting the random seed\n", "reload(bilstm);" ] }, { "cell_type": "code", "execution_count": 89, "metadata": { "collapsed": true }, "outputs": [], "source": [ "#creating your model\n", "torch.manual_seed(765);\n", "model = bilstm.BiLSTM(len(word_to_ix),tag_to_ix,embedding_dim, hidden_dim)" ] }, { "cell_type": "code", "execution_count": 90, "metadata": { "collapsed": true }, "outputs": [], "source": [ "# preparing your first sentence to be an input\n", "words= X_tr[5]\n", "tags = Y_tr[5]\n", "sentence = bilstm.prepare_sequence(words, word_to_ix)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- this calls the `forward()` function on the model, which returns the tag_scores for each tag for each particular token in the sentence\n" ] }, { "cell_type": "code", "execution_count": 91, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Variable containing:\n", " 0.1702\n", "-0.1568\n", " 0.2289\n", " 0.0965\n", "-0.0905\n", "[torch.FloatTensor of size 5]\n", "\n" ] } ], "source": [ "lstm_feats = model(sentence)\n", "print (lstm_feats[0][0:5])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- we provide the `predict()` function that returns the set of tags obtained for the specific input by the model." ] }, { "cell_type": "code", "execution_count": 92, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "['ADV', 'NUM', 'ADJ']\n" ] } ], "source": [ "tags = model.predict(sentence)\n", "print (tags[0:3])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- Train the model for now\n" ] }, { "cell_type": "code", "execution_count": 93, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Epoch 1: Dev Accuracy: 0.5463292473968631\n", "Epoch 3: Dev Accuracy: 0.6546724660603664\n", "Epoch 5: Dev Accuracy: 0.6737841043890866\n", "Epoch 7: Dev Accuracy: 0.6985633320152893\n", "Epoch 9: Dev Accuracy: 0.7249242124686964\n" ] } ], "source": [ "reload(bilstm);\n", "torch.manual_seed(765);\n", "loss = torch.nn.CrossEntropyLoss()\n", "model, losses, accuracies = bilstm.train_model(loss, model, X_tr,Y_tr, word_to_ix, tag_to_ix, \n", " X_dv, Y_dv, num_its=10, status_frequency=2, \n", " optim_args = {'lr':0.1,'momentum':0}, param_file = 'best.params')" ] }, { "cell_type": "code", "execution_count": 94, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAtQAAACdCAYAAACU5e99AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xd8lvX1//HXySJ7L0gCCRC2bMWFoyjiqLZ1d9ihdbS2\njla7v939tbXW2mpr1dpqa2vrrFbqHjjqYCjIkikEEsJMSEIg4/z+uG9CUJBAxpXc9/v5eOSRXOvm\nXAIfD5/rXOdj7o6IiIiIiByamKADEBERERHpy5RQi4iIiIh0ghJqEREREZFOUEItIiIiItIJSqhF\nRERERDpBCbWIiIiISCcooRYRERER6QQl1CIiIiIinaCEWkRERESkE+KCDuBg5ebmemlpadBhiIgc\nkjlz5mxy97yg4+gpGrNFpC/r6Jjd5xLq0tJSZs+eHXQYIiKHxMzeCzqGnqQxW0T6so6O2Sr5EBER\nERHphKhJqN096BBEREREJAL1uZKPg+Xu3PjUu2xvbOKHZ40JOhwRERER6UY7m1tYvamBZdXbWbah\njuXVdZTmJnPdKSO67deM+ITazGhsauHu/73HtJEFHDcsat4FEhEREYlYjU0trNxY35Y4L6vezrLq\nOt7b3EBLa6gywQwGZSfTPyOxW2OJ+IQa4OunDOf5pdV848H5PHH1cWQkxQcdkoiIiIh0QMOuZlZU\n17clzKFZ5+2s2dJAOG8mNsYYlJNMeX4qp43pT3lBKkPzUxmSl0pifGy3xxgVCXVifCw3njees//w\nKj96bBE3njcu6JBEREREpJ26nc0sr65j2YbdiXPoe8XWHW3nxMcaZbkpjB6QwVnjiygvSKU8P43S\n3GT6xXV/4rw/UZFQA4wvyeRLJwzhd88t55TRBUwfXRh0SCIiIiJRp2ZHE8vbyjRCX8s3bGd9TWPb\nOQlxMQzOTWHiwCzOn1wSnnFOY1BOMvGxva+nRtQk1ABf+Ug5zyyu5tsPL2ByaTbZKQlBhyQiIiIS\nUZpbWqmqbWT9tkbWbWtg/bZGKrbuYM2WepZtqKN6+862cxPjYxian8qUwTkMzU+lPD+V8oI0SrKS\niOuFifP+RFVCnRAXw6/PG8eZt7zM9x55h1s/NTHokERERET6lPqdzazbtiP0tXUH68M/rw9vV9U2\nttU275aTkkBxdjLHDcsLJ82hUo2izCRiYiyYG+lCUZVQA4zsn87VJw3jhieXMuPt9Xx03ICgQxIR\nERHpFVpbnU31O0Ozy+2S5Yp2P9fsaNrrmrgYo39mIgMykjhySA5FmUkUZSYxIDOJoqwkBmQkkZQQ\nXH1zT4i6hBrgsuMG8/SiDXzv3+8wpSyb/PTubaUiIiIi0hvsbG6hclsj67ftoKLdrPL6mt3fG9nV\n3LrXNan94kJJclYSkwZltSXKRZmJFGUmk5fWj9gImGXujKhMqONiY7jxvHGcdvNLfOuhBdz52cmY\nRfcfBBEREYlMm+t28tSiDcxcUMmrKza39WiGUJ/m/LR+DMhMYkxRBqeMLmybVS7KCs0yq93wgUVl\nQg0wJC+V62eM4Mf/WcT9cyo4b3JJ0CGJiHSYmd0I3OXuC4OORUR6n011O3lyYRUzF1Ty2sottLQ6\npTnJXHJsGUPzUynKSqI4M5mCjH6BtpuLFFGbUAN8/uhSnlpYxY8eW8TRQ3IozkoOOiQRkY5aDNxu\nZnHAn4F/uHtNwDGJSIA2bt/JEwurmDm/ktdXbabVYXBuClccP4TTDuvPyP5peiLfTaI6oY6JMX51\n7jhO+c0srn9gPn+7eEpEvGkqIpHP3e8E7jSz4cDngflm9gpwh7s/H2x0ItJTqrc38sQ7VTw+v5I3\nVm/BHYbkpXDliUM5bWx/hhcoie4JUZ1QA5RkJ/Pd00fx7YcX8LfX3+Oio0qDDklEpEPMLBYYEf7a\nBLwNXGtml7n7BYEGJyLdZkNtI/9dUMnMd6p4M5xEl+en8tWPlHP62P6U56cqie5hUZ9QA1x4RAlP\nLKzi/81cwtTyPMpyU4IOSUTkQ5nZTcAZwHPAz9z9jfChX5jZ0uAiE5HuUFXTyMwFlfz3nUpmv7cV\ndxhekMbV04Zx2mGFlBekBR1iVFNCDZgZvzx7LNNvepGv3/82/7rsqKhv/yIivd584LvuXr+PY0f0\ndDAi0vXWb9vBf98JvVg4572tAIwoTOOak4Zx2mH9GZqfGnCEspsS6rDCjER+eNZorvnn29z50kou\nO35I0CGJiHyYbbQbw80sEzjB3R850MuJZjYDuBmIBe5095+/7/h1wKfCm3HASCDP3bcc6FoR6Zx1\n23bw3wWVPL6gknlrtgGhRem+Pn0Ypx7WnyF5SqJ7IyXU7XxsfBFPvFPFjU+9y4kj8hmmxyci0nt9\n390f3r3h7tvM7PvAIx92Ubju+lbgZKACeNPMHnX3Re0+6wbghvD5HwWuCSfTB7xWRA7e2i0N/Ped\nSh5fUMXba0NJ9OgB6Vx3ynBOO6y/SlH7ACXU7ZgZP/34YUy/aRbX/ustHv7SMcTHxgQdlojIvuxr\ncOrImH4EsNzdVwKY2X3AWcD+kuILgX8c4rUish9rtzTw+IJKZi6oZH5F6KHSYUUZfGPGCE4dU0ip\nkug+RQn1++Sm9uNnHx/D5X+by63PL+fqk4YFHZKIyL7MNrNfE5oxBvgyMKcD1xUBa9ttVwBT9nWi\nmSUDM4ArD+ZaM7sUuBRg4MCBHQhJJHI1NrWwenM9y6vrWFFdz4qNdSyt2s7SDdsBGFecwbdOHcGp\nY/ozMEfrYfRVSqj3YcaY/nxs/ABueW4500YUcFhxRtAhiYi831eA7wH/DG8/TSip7kofBV5x9y0H\nc5G73w7cDjB58mQ/wOkiEWFr/S5WbKwLJc4b61ixMZREr93agIf/FphBcVYSQ/JSOXtSEaeO6U9J\ntpLoSKCEej9+eOYY/rdyM1+7/y0evfJYEuO1LKeI9B7h7h7fPIRL1wEl7baLw/v25QL2lHsc7LUi\nEae11Vm3bQfLN9axYnfiXF3P8o11bKnf1XZev7gYBuelMrY4g49PKGJofipD8lIZnJeifCJCKaHe\nj4zkeH5+9lg+/+c3uemZd/nWqSODDklEpI2Z5QHXA6OBxN373f0jB7j0TaDczMoIJcMXAJ/cx+dn\nAMcDnz7Ya0X6usamFlZurP/AjPPKjXXsbG5tOy87JYGheamcMrqAIXmpDMlPZWheKgMyk9R+N8oo\nof4QJw7P58IjSrh91kqmjypg0qDsoEMSEdntXkLlHmcAlwOfBTYe6CJ3bzazK4EnCbW+u8vdF5rZ\n5eHjt4VP/TjwVPs+1/u7tgvvSaTHuDtb6ne1lWaEkubQV8XWHXuVaZRkJTM0P5Vjh+YwJC+Vofmp\nDM5LJTslIdibkF7D3LunvM3M7iI00Fe7+5h9HD8B+DewKrzrIXf/0YE+d/LkyT579uyuDPVD1e1s\nZsZvZhEXY8y8airJCfo3iIgcOjOb4+6Tu+hzJpnZfHcfG973prsf3vkou05Pj9kiH6Z6eyOvLt/M\ny8s38cryTVTWNLYdS4yPYXBualt5xpD8FIbmp1KaozKNaNbRMbs7s8O/ALcA93zIOS+5+xndGEOn\npfaL45fnjOWTd7zOL59Yyg/OHB10SCIiAE3h75VmdjqwHtBjNJF26nY288aqzby8bDOvLN/U1lkj\nMzmeY4bkMmFgZlsCXZSZRIzKNOQQdVtC7e6zzKy0uz6/Jx09JJfPHV3KX15dzfRRBRw9NDfokERE\nfhKuc/4a8DsgHbgm2JBEgtXU0srba7e1zUDPW7ON5lanX1wMR5Rl8/GJRRw7NJdR/dOVPEuXCrp+\n4Wgzm0/o5Zav768Wrzf0NP3GjBG8+O5GrntgPk9cPZW0xPhA4hARCa9YWO7u/wFqgBMDDkkkEO7O\nsuo6Xl4WSqBfW7mZ+l0tmMHYogwuPW4wxw7NZeKgLJVtSLcKMqGeCwx09zozO43Qcrnl+zqxN/Q0\nTUqI5VfnjuPc217lJ/9ZzC/OGRtEGCIiuHuLmV0I3BR0LCI9rbJmB68sD5VwvLx8Exu37wSgLDel\nbQb6yME5ZCbrhUHpOYEl1O5e2+7nmWb2ezPLdfdNQcV0IJMGZXHZ8UP4wwsrOGVMAR8ZURB0SCIS\nvV4xs1sIdfpo34ljbnAhiXS92sYmXluxJ4FesTH0xz0nJYFjhuZy7NBcjh6aQ3GWFkiR4ASWUJtZ\nIbDB3d3MjgBigM1BxdNRV59UznOLq/nGgwt46uosstQyR0SCMT78vX13JAcO1IdapFfb2dzCvDXb\n2hLot9duo9UhKT6WKYOzufCIgRwzNJfhBWmqg5Zeo9sSajP7B3ACkGtmFcD3gXho63N6DnCFmTUD\nO4ALvLt6+HWhfnGx3HjeOD526yt8/9GF/PbCCUGHJCJRyN1VNy0RobXVWVK1vS2BfmPVFnY0tRAb\nY4wrzuDKE4dyzNBcJgzMIiEuJuhwRfapO7t8XHiA47cQaqvX54wpyuCr08r59dPvcsroQk4f2z/o\nkEQkypjZ/+1rf0f6+YsEobmllYqtO1i1uZ5VG+tZvbmeVZvqWbS+ls3hZbuH5qdy/uElHDM0lymD\ns0lXAwDpI4Lu8tFnXXHCEJ5ZvIHvPrKAI8qyyUvrF3RIIhJd6tv9nEhoIa3FAcUiAoRmm6tqG1m1\nqb7ta3X4+5otDTS37nkQndYvjtLcFI4fnscxQ3I5ZmguhRmJAUYvcuiUUB+i+NgYbjx3HKf/7mW+\n9dAC7rhoEmaq5RKRnuHuN7bfNrNfEVoSXKRbuTub6naFZpg31n9gxnlnc2vbuYnxMZTmpDC8MI1T\nxhRSlpNCWV4KpTkp5KYm6P+bEjGUUHdCeUEa100fzk9nLuahues4e1Jx0CGJSPRKBjQISZepaWgK\nJcub6li1qWGv2ea6nc1t58XFGANzkinLSeHYobmU5qYwODeF0twUCtMT9eKgRAUl1J30hWPLeGpR\nFT94bCFHDclhQGZS0CGJSBQwswWEunoAxAJ57N3xQ+SAmltaWb25gaVV29slznWs3tzAlnBdM4AZ\nFGclUZqTwtkTiyjNTaEs/FWUmURcrF4WlOimhLqTYmOMX507jhm/eYlvPDife75whB5hiUhPOKPd\nz82E2pA27+9kkS31u1hSVcviyu0sqaxlSdV23t2wfa8SjcL0REpzkzlldAFluaHSjMF5KZRkJ9Mv\nTisNiuyPEuouMCgnhW+fPpLvPfIO976+hk8fOSjokEQk8vUHFrr7dgAzSzOzUe7+esBxScB2Nbey\nclMdSyq3s7iqliWV21lSVcuG2p1t5+SkJDCyfzoXHTWIEYXpDC9MY3BeCskJSgtEDoX+5nSRT08Z\nyFMLq/jZzMVMLc9lUE5K0CGJSGT7AzCx3Xb9PvZJBHN3NtbtbEuYF1duZ3FlLSs21tHUEqoGSoiN\nYWh+KscMzWVkYToj+qcxojBdnalEupgS6i5iZvzi7LGcctMsrrt/PvddeqRexBCR7mTtF8Ny91Yz\n05geoRqbWlheXcficKnGkvDM8+Z2dc6F6YmM6J/GCcPzGdk/jZH90ynLTSFe9c0i3U6DbxcakJnE\n/310FNc9MJ+7XlnFJVMHBx2SiESulWb2VUKz0gBfAlYGGI90AXensqZxT61zVWjWedWmelrCPZwT\n42MYXpDGSSML2macRxSmkZWSEHD0ItFLCXUXO2dSMU8urOKXTy7lhOF5DM1PCzokEYlMlwO/Bb5L\nqNvHs8ClHbnQzGYANxPqDnKnu/98H+ecAPwGiAc2ufvx4f2rge1AC9Ds7pM7eyMCqzbV89PHF/HG\nqi3UNu55t7Q4K4kRhemcOqYwlDj3T6M0J4VYPQEV6VWUUHcxM+NnnziM6TfN4mv/epsHrzha7YRE\npMu5ezVwwcFeZ2axwK3AyUAF8KaZPerui9qdkwn8Hpjh7mvMLP99H3Oiu2869Ohlt5ZW566XV/Gr\np5bSLy6GM8YNYGT/dEYWpjGsME1Lb4v0ER3K9MzsKjNLt5A/mdlcM5ve3cH1VflpifzkY2N4u6KG\n215cEXQ4IhKBzOzucOK7ezvLzO7qwKVHAMvdfaW77wLuA8563zmfBB5y9zXQlrxLF1u2YTtn/+FV\nfjpzMVPL83j62uP52ccP4zNHDmJyabaSaZE+pKNTp19w91pgOpAFfAb4wCNC2eOMsQM4Y2x/bn52\nGa+t3Bx0OCISeca6+7bdG+6+FZjQgeuKgLXttivC+9obBmSZ2QtmNsfMLmp3zIFnwvv3WWJiZpea\n2Wwzm71x48YO3Uw0aWpp5dbnl3P6b1/mvc31/PbCCdxx0SQK0hODDk1EDlFHSz52F2udBvzV3Rea\nVi85oB+fNYZ5a7Zxwe2vMX1UAV+bPpzhhaqpFpEuEWNmWeFEGjPLpuvK+OKAScA0IAn4n5m95u7v\nAse6+7pwGcjTZrbE3We1v9jdbwduB5g8ebIjbRaur+H6B+azcH0tp4/tzw/PHE1uqlrYifR1HR18\n55jZU0AZ8C0zSwNaD3BN1MtKSeDJa47jrpdXcceslcy4eRZnjRvA1ScNozRXfapFpFNuJJTo3k9o\n0uMc4KcduG4dUNJuuzi8r70KYLO71wP1ZjYLGAe86+7rIFQGYmYPEyohmYV8qF3Nrdzy3DJ+/8IK\nMpMTuO3TE5kxpn/QYYlIF+loQn0xMB5Y6e4N4ZmQz3dfWJEjtV8cX51WzkVHDeK2F1fyl1dX8dj8\nSs6bXMxXPlLOgMykoEMUkT7I3e8xsznAieFdn2j/YuGHeBMoN7MyQon0BYRqptv7N3BLuK91AjAF\nuMnMUoAYd98e/nk68KMuuJ2I9vbabVz/wHyWbtjOJyYU8b0zRqnFnUiE6WhCfRTwlrvXm9mnCa3E\ndXP3hRV5MpMT+OapI/jCsaX8/vkV3Pv6ezw4dx2fnjKIL504RI/8ROSghcvvNgKJAGY2cPeLhB9y\nTbOZXQk8Saht3l3hz7k8fPw2d19sZk8A8wk9jbzT3d8xs8HAw+GKvzjg7+7+RLfdYB/X2NTCTc+8\nyx2zVpKflshdn5vMR0YUBB2WiHQDa7fQ1v5PMptP6HHfWOAvwJ3Aebv7kvakyZMn++zZs3v6l+1y\nFVsb+O2zy3hgTgWJ8bF8/phSLp06hIxkvdUtEsnMbE5X9G42szMJlX0MAKqBQcBidx/d2c/uSpEy\nZh+s2au3cP2D81m5sZ4LjyjhW6eNVNcOkT6oo2N2R7t8NIeXuD0LuMXdbwX0dl0nFGcl88tzxvHM\ntcczbWQBtz6/gqm/fI5bn19O/c7mA3+AiES7HwNHEqprLiP0AuFrwYYkDbua+eFjCzn3j/9jZ1Mr\nf7t4Cv/vE2OVTItEuI4m1NvN7FuE2uU9bmYxhFbPkk4anJfK7y6cwMyvTuWIsmxueHIpx9/wPH96\neRWNTS1BhycivVeTu28m1O0jxt2fB7RqYYBeXbGJGb95iT+/sprPHDmIp645jmPLc4MOS0R6QEdr\nqM8n9NLKF9y9yswGAjd0X1jRZ9SAdO787OHMXbOVXz25lB//ZxF3vrSSr04r55xJxcRrtUUR2ds2\nM0sl1GHjXjOrBuoDjikqbW9s4uf/XcK9r6+hNCeZf156JFMG5wQdloj0oA7VUAOYWQFweHjzjaBW\nzoqWerxXl2/ihqeWMm/NNgblJHPtycP46NgBxMSo/bdIX9aFNdQpwA5CTxo/BWQA94ZnrXuNSB+z\nX1hazbcfWkBlbSMXH1PG16YPJykhNuiwRKSLdHTM7tAMtZmdR2hG+gVC/U5/Z2bXufsDnYpS9uvo\nobk8NCSH55ZUc8OTS7nqvrf4/fMruHb6MKaPKkDr6ohEt3CPaAh14bg7yFiiUU1DEz9+fBEPzKlg\naH4qD15xNBMHZgUdlogEpKMlH98BDt89K21mecAzgBLqbmRmTBtZwInD83l8QSU3Pf0ul/11DuOK\nM/ja9OFMLc9VYi0i0sOeXrSB7zy8gM31u/jyiUP4ykfKSYzXrLRINOtoQh3zvhKPzXT8hUbppJgY\n46PjBnDqmEIemreOm59ZxkV3vcGUsmyuO2U4k0uzgw5RRCTibanfxQ8eXcijb69nRGEaf/rs4RxW\nnBF0WCLSC3Q0oX7CzJ4E/hHePh+Y2T0hyf7ExcZw3uQSzho/gPveWMvvnlvOObf9jxOG5/H16cMZ\nU6SBXSRamNlV7n7zgfZJ57k7MxdU8X//fofaxiauOWkYV5wwhIQ4zSuJSMjBvJR4NnBMePMld3+4\n26L6EJH+gsvBaNjVzN2vvsdtL66gZkcTpx1WyLUnD2NovlqEi/RWXfhS4lx3n/i+ffPcfUJnP7sr\n9fUxu3p7I//3yEKeWFjFYUUZ3HDuWEYUpgcdloj0kC59KRHA3R8EHuxUVNKlkhPiuOKEIXzqyIHc\n+dIq/vTSSp54p4qPTSjimpOGUZKdHHSIItLFzOxCQm1My8zs0XaH0oAtwUQVedydR95axw8fW0TD\nrha+MWMEX5xaRpxamIrIPnxoQm1m24F9TWEb4O6uf6b3AumJ8Vx78jA+d3Qpt724grtfXc2jb63n\n/MNLOHtSMeOKM4lVuz2RSPEqUAnkElp6fLftwPxAIoowlTU7+M7D7/DckmomDszkl+eMY2h+atBh\niUgv9qEJtburdqAPyU5J4NunjeQLx5Rxy/PL+Oeba7n39TXkpCRw4oh8ThqZz7HleaT26/CDCRHp\nZdz9PeA94CgzGwSUu/szZpYEJBFKrOUQ/Wf+er714AKaWlv53hmj+NzRpZqQEJEDUmYVgQozEvnJ\nxw7j69OH8+K7G3l2cTVPLazigTkVJMTGMGVwNieNLGDayHyKs1QWItIXmdkXgUuBbGAIUAzcBkwL\nMq6+7NXlm7j6vrcYW5zBr88bT2luStAhiUgf0W0JtZndBZwBVLv7mH0cN+Bm4DSgAficu8/trnii\nUWZyAmeNL+Ks8UU0tbQy572tPLt4A88urub7jy7k+48uZHhBGtNG5jNtZAHjS1QaItKHfBk4Angd\nwN2XmVl+sCH1XSs31nHFvXMpy03h7i8cQVpifNAhiUgf0p0z1H8BbgHu2c/xU4Hy8NcU4A/h79IN\n4mNjOHJwDkcOzuE7p49i5cY6nltSzTOLN/DHWSv5/QsryElJ4IThodKQqcNUGiLSy+109127F3cy\nszj2/c6LHEBNQxOX3D2b2Bjjrs8drmRaRA5at2VM7j7LzEo/5JSzgHs81LfvNTPLNLP+7l7ZXTHJ\nHoPzUhmcl8olUwdT09DEi8s28uziDTyzeAMPzq0gPtY4cnAO00aEZq/VMUSk13nRzL4NJJnZycCX\ngMcCjqnPaWpp5Ut/n0PF1h3c+8UpGutE5JAEOQVZBKxtt10R3veBhNrMLiVUK8jAgQN7JLhokpEc\nz5njBnDmuAE0t7Qy+72tbbPXP3hsET94bBHDClKZNrKAk0bmM74kS6UhIsH7JnAxsAC4jNBiW3cG\nGlEf4+784NGFvLJ8M786dxyHa9VZETlEfeKZvrvfDtwOoUUCAg4nosW1Kw359mkjWbWpvq3u+o5Z\nK/nDCyvITknghOF5nDSygKnluXo8KhIAd28F7gDuMLNsoNg7uFKXmc0g9A5LLHCnu/98H+ecAPwG\niAc2ufvxHb22r7j71dXc+/oaLj9+COdMKg46HBHpw4JMqNcBJe22i8P7pBcpy03hkqmDQ6UhO5qY\n9e7GtgT7obnriI81ppTlMG1kPiepNESkx5jZC8CZhMbxOUC1mb3q7tcc4LpY4FbgZEJPBt80s0fd\nfVG7czKB3wMz3H3N7pcdO3JtX/HC0mp+9J9FTB9VwPWnDA86HBHp44JMqB8FrjSz+wi9jFij+une\nLSMpno+OG8BHw6Uhc9qVhvzwsUX88LFFlOeHSkNOHJ7HuJJMEuNjgw5bJFJluHutmV1C6H2U75tZ\nRxZ2OQJY7u4rAcJj8FlA+6T4k8BD7r4GwN2rD+LaXm/Zhu185e/zGF6Yzk3njydGJWwi0knd2Tbv\nH8AJQK6ZVQDfJ/ToEHe/jVC932nAckJt8z7fXbFI14uLjWHK4BymDM7hW6eNZPWmep5dUs2zizdw\n50srue3FFcTFGCP7pzNxYCYTBmYxYWAmA7OT2d2VQEQ6Jc7M+gPnAd85iOv29f7K+zssDQPiw7Pg\nacDN7n5PB6/t1bbU7+Liu2fTLz6WP312MinqZiQiXaA7u3xceIDjTqiPqkSA0twULj62jIuPLaNm\nRxNvrNrCW2u3Mve9bdw/p4K7//ceEFrNcUJJJhPCSfbY4gzVYIscmh8BTwIvu/ubZjYYWNZFnx0H\nTCK0SEwS8D8ze62jF/fWF8l3Nbdy+V/nUFXbyD8vPZIBmUlBhyQiEUL/NJcul5EUz8mjCjh5VAEA\nLa3Ouxu2M2/NNuat2cq8tdt4dknoCbIZDMtPY+KgTCaUhGaxh+Sl6hGsyAG4+/3A/e22VwJnd+DS\njry/UgFsdvd6oN7MZgHjwvsP+O5Lb3yR3N35zsMLeGP1Fn574QQmDMwKOiQRiSBKqKXbxYZLP0b2\nT+eTU0KzVTU7mnh77TbmrdnG3DVbmbmgin+8EXqSnNYvjvEDM8Mz2VmML8kkKyUhyFsQiSRvAuVm\nVkYoGb6AUM10e/8GbgkvFpNAqKzjJmBJB67tlW6ftZL751Rw1bRyzhw3IOhwRCTCKKGWQGQkxXPc\nsDyOG5YHQGurs2pz/Z5Z7DXbuOX55bSG57bKclPaykQmlGQyojCNuNiYAO9ApG9y92Yzu5JQuUgs\ncJe7LzSzy8PHb3P3xWb2BDAfaCXUHu8dgH1dG8iNHISnF23g508s4fSx/blqWnnQ4YhIBLIOti3t\nNSZPnuyzZ88OOgzpAfU7m1mwrqYtyZ67Zhub6nYCkBQfy2HFGaEkuySLiQMzyU9PDDhikQMzsznu\nPrkLPifW3Vu6IqbuFPSYvWh9Lefc9irl+an887Kj1HlIRA5KR8dszVBLr5XSL65tkRkI1UBWbN3B\nvLV7ZrHvenkVTS0rASjKTGL8wEzGF2cyekA6owakk5msUhGJWKvCs8j/BJ7r6KIu0aR6eyOX3P0m\n6Ynx3HGLzyIbAAAP40lEQVTRZCXTItJtlFBLn2FmlGQnU5Kd3FYD2djUwqLK2r1KRR6fv6edeVFm\nEqMHpDN6QEboe1E6hemJat0nkWAEcAahbkl/MrP/APe5+8vBhtU7NDa1cNlf57C1oYn7Lz9KT7BE\npFspoZY+LTE+lokDs5g4MAsoA2Bz3U4WVdaycP3urxqeXryB3fN32SkJbTPYuxPtspwUdRaRPsXd\nG4B/Af8ysyxCy4G/SKi2Oaq5O994cD7z1mzjtk9PZExRRtAhiUiEU0ItEScntR9Ty/OYWp7Xtq9+\nZzNLqsIJ9rpaFlbW8OeXV7OrpRWA5IRYRvZPD89mhxLt8oJU+sVFfW4ivZiZHQ+cD8wAZhNa5CXq\n3fLccv791nquO2U4M8b0DzocEYkCSqglKqT0i2PSoGwmDcpu27eruZXl1XUsXF/DwvW1LFpfy0Nz\n13FPeBGauBijvCBtryR7ZP80LUQjvYKZrQbmEZqlvi7cMzrqzVxQyY1Pv8snJhTxpROGBB2OiEQJ\nJdQStRLiYhgVLv04N7yvtdVZs6WhrVRk4fpaXli6kQfmVLRdV5qTzOgBGeGSkVCinZfWL5ibkGg2\n1t1rgw6iN5lfsY1r//UWkwZl8f/OPkzvSohIj1FCLdJOTIxRmptCaW4Kp4/d86i4urZxryR7wboa\nHl+w5+XH/LR+jB6QTnlBGmW5KW1f+Wn99D916S6FZvYwUODuY8xsLHCmu/8k6MCCUFXTyBfvmU1O\nSj/++JlJKtcSkR6lhFqkA/LTE8lPT+TEEflt+2obm1jU7sXHRetreWXFZnY1t7adk5IQS2k4uR6c\nm0JZXgqlOSkMzk0lI1mlI9IpdwDXAX8EcPf5ZvZ3IOoS6h27Wrjknjepa2zmwS8dTW6qnhiJSM9S\nQi1yiNIT4/fqkw2hkpHK2kZWbaxn1aY6Vm6qZ9Wmehasq2Hmgsq2lR8h1G2k/Wz27q/SnBSSEjS7\nJgeU7O5vvO8JSHNQwQSltdW59l9vsXB9LX/67GRGFKYHHZKIRCEl1CJdKCbGKMpMoigziWPLc/c6\ntqu5lTVbGlgdTrJDyXYdLy3bu0YbYEBGYttsdlluCoPzUijLTaU4K4l4LbkuIZvMbAjgAGZ2DlD5\n4ZdEnpueeZf/vlPFd08fyUdGFAQdjohEKSXUIj0kIS6GofmpDM1P/cCx+p3NrAon2u0T7sfeXk9t\n455Jx7gYY2B2clsZye5SktLcFArTE9VLO7p8GbgdGGFm64BVwKeDDalnPTJvHb97bjkXHF7CxceW\nBR2OiEQxJdQivUBKvzjGFGV8YAEKd2drQ1OofGRjPas3h5PtjfW8umITjU176rXjY0Oz4yXZyRRn\nJVGcldz2c0lWMrmpCXpBMoK4+0rgJDNLAWLcfXvQMfWkOe9t5foH53Pk4Gx+dNYY/dkWkUApoRbp\nxcyM7JQEslP27qENodrRqtrGtpnttVsbqNi6g4otDTy5vpYt9bv2Oj8xPiaUZLcl2+Hv4Z8zkuKV\nlPQBZnbtfvYD4O6/7tGAAlCxtYHL/jqbARmJ/OFTk0iIUxmUiARLCbVIHxUTYwzITGJAZhLHDM39\nwPH6nc2hBHtrA2u3hJLttVsbWLtlB3Pe27pXKQlAar+40Gx2u1nt3dsl2cmk9tNw0Uukhb8PBw4H\nHg1vfxR4I5CIelDdzmYuuXs2O5tbue/Sw8lKSQg6JBERJdQikSqlXxzDC9MYXpi2z+M1O5rCyXYo\n6a7YuoO1WxpYs7mBV5ZvomFXy17nZybH751kh2e6i7OSKMxI1AqSPcTdfwhgZrOAibtLPczsB8Dj\nAYbW7Vpanav+MY9l1XX85fOH7/N9BBGRICihFolSGUnxZCRlMHpAxgeO7a7dXruloa2UZPcs99IN\n23l2SfVe/bYhNMNdmJFI/4xECtMTKcxIbLedRP+MRDKTVVbShQqA9nU9u8L7ItYvnljCs0uq+fFZ\no5lanhd0OCIibZRQi8gH7KndTmBcSeYHjre2OpvqdrYl2xtqG6msaaSqJvR92YZNVG9v3KvvNkC/\nuJhQop0eTrQzksLf9+zLSe1HrLqVdMQ9wBvh1RIBPgb8Jbhwute/3lzL7bNWctFRg/jMUaVBhyMi\nshcl1CJy0GJirG31yEmD9n1Oc0srm+p2UVmzoy3RrqoNJd1VNY3MWbOVDTVV7GrZe6Y7LsYoaDfD\nvSf53pOE56f1i/p+3O7+UzP7LzA1vOvz7j4vyJi6y2srN/OdRxYwtTyX/ztjVNDhiIh8gBJqEekW\ncbExbUnx/rS2OlsadrUl2ZW1jVTV7Gib7V68vpbnFlezo2nvem4zyE3tx4DMJEraarpD3UpKspIZ\nkJkUFZ0f3H0uMPdgrzOzGcDNQCxwp7v//H3HTwD+Tai3NcBD7v6j8LHVwHagBWh298mHGn9HvLe5\nniv+NoeB2cnc8smJxEX5P6REpHdSQi0igYmJMXJT+5Gb2u8DPbh3c3dqG5vDs9ztZrtrGlm3bQcL\n1tXw5MIqmlr21JeYQWF6YuglynCSvftFypLsZArSE6O2rMTMYoFbgZOBCuBNM3vU3Re979SX3P2M\n/XzMie6+qTvjBKhtbOLiu2fjwJ8+ezgZSXrxVUR6JyXUItKrmVn4Bcr4/XYsaQn35K7Y0sDa8AuU\na7c2ULFlB/9bsZmHa9fh7eq5dy+Cs1c/7nYJd05KRC+CcwSwPLwwDGZ2H3AW8P6EOlDNLa1c+fd5\nrN5Uz98umUJpbkrQIYmI7JcSahHp82JjQglyUWYSU/ZxfGdzC+u3NbYl2rtbBa7duoOnFm5g8/sW\nwUmKj20rH9nTmztcUpKdTHrfbhFYBKxtt10B+/zPdrSZzQfWAV9394Xh/Q48Y2YtwB/d/fb3X2hm\nlwKXAgwcOPCQgvzJ44uZ9e5GfnH2YRw5OOeQPkNEpKcooRaRiNcvLpay3BTK9jPLuXsRnPYJ99rw\ngjivr9pC3c69F8HJSIrnqmnlfOHYsp4IPwhzgYHuXmdmpwGPAOXhY8e6+zozyweeNrMl7j6r/cXh\nJPt2gMmTJ7+v18uB/e219/jLq6v54tQyzj/80BJyEZGepIRaRKLehy2C4+7U7GjaK8leu7Vhv8l5\nH7AOKGm3XRze18bda9v9PNPMfm9mue6+yd3XhfdXh1v2HQHslVB31sj+6Zw7qZhvnjqyKz9WRKTb\nKKEWEfkQZkZmcgKZyQkcVrzvFyf7mDeBcjMrI5RIXwB8sv0JZlYIbHB3N7MjgBhgs5mlADHuvj38\n83TgR10d4KRBWUwalNXVHysi0m2UUIuIRBF3bzazK4EnCbXNu8vdF5rZ5eHjtwHnAFeYWTOwA7gg\nnFwXAA+HX9iMA/7u7k8EciMiIr2IEmoRkSjj7jOBme/bd1u7n28BbtnHdSuBcd0eoIhIH6MO+SIi\nIiIinWDuB/0CdqDMbCPw3iFcmgt0+0IEvVA03rfuOXr0xfse5O55QQfRUzRmH7RovO9ovGeIzvvu\ni/fcoTG7zyXUh8rMZnf3Erm9UTTet+45ekTrfUeDaP29jcb7jsZ7hui870i+Z5V8iIiIiIh0ghJq\nEREREZFOiKaE+gPL40aJaLxv3XP0iNb7jgbR+nsbjfcdjfcM0XnfEXvPUVNDLSIiIiLSHaJphlpE\nREREpMtFfEJtZjPMbKmZLTezbwYdT08wsxIze97MFpnZQjO7KuiYeoqZxZrZPDP7T9Cx9BQzyzSz\nB8xsiZktNrOjgo6pu5nZNeE/2++Y2T/MLDHomKTrRNu4rTFbY3bQMfWESB+3IzqhNrNY4FbgVGAU\ncKGZjQo2qh7RDHzN3UcBRwJfjpL7BrgKWBx0ED3sZuAJdx9BaBW7iL5/MysCvgpMdvcxhJbPviDY\nqKSrROm4rTE7ukTVmA3RMW5HdEINHAEsd/eV7r4LuA84K+CYup27V7r73PDP2wn9ZS0KNqruZ2bF\nwOnAnUHH0lPMLAM4DvgTgLvvcvdtwUbVI+KAJDOLA5KB9QHHI10n6sZtjdkas4ONqsdE9Lgd6Ql1\nEbC23XYFUTBItWdmpcAE4PVgI+kRvwGuB1qDDqQHlQEbgT+HH5veaWYpQQfVndx9HfArYA1QCdS4\n+1PBRiVdKKrHbY3ZES/qxmyIjnE70hPqqGZmqcCDwNXuXht0PN3JzM4Aqt19TtCx9LA4YCLwB3ef\nANQDEV1zamZZhGYsy4ABQIqZfTrYqEQ6T2N2VIi6MRuiY9yO9IR6HVDSbrs4vC/imVk8oYH5Xnd/\nKOh4esAxwJlmtprQI+KPmNnfgg2pR1QAFe6+ezbrAUKDdSQ7CVjl7hvdvQl4CDg64Jik60TluK0x\nW2N2hIv4cTvSE+o3gXIzKzOzBEIF8I8GHFO3MzMjVJ+12N1/HXQ8PcHdv+Xuxe5eSuj3+Tl3j6h/\n/e6Lu1cBa81seHjXNGBRgCH1hDXAkWaWHP6zPo0oeKknikTduK0xW2N2gCH1lIgft+OCDqA7uXuz\nmV0JPEnojdK73H1hwGH1hGOAzwALzOyt8L5vu/vMAGOS7vMV4N5w8rES+HzA8XQrd3/dzB4A5hLq\njjCPCF59K9pE6bitMTu6RNWYDdExbmulRBERERGRToj0kg8RERERkW6lhFpEREREpBOUUIuIiIiI\ndIISahERERGRTlBCLSIiIiLSCUqopU8zs1fD30vN7JNd/Nnf3tevJSIih0ZjtkQqtc2TiGBmJwBf\nd/czDuKaOHdv/pDjde6e2hXxiYjIHhqzJdJohlr6NDOrC//4c2Cqmb1lZteYWayZ3WBmb5rZfDO7\nLHz+CWb2kpk9Snh1KjN7xMzmmNlCM7s0vO/nQFL48+5t/2tZyA1m9o6ZLTCz89t99gtm9oCZLTGz\ne8MrQomICBqzJXJF9EqJElW+SbvZjvAgW+Puh5tZP+AVM3sqfO5EYIy7rwpvf8Hdt5hZEvCmmT3o\n7t80syvdffw+fq1PAOOBcUBu+JpZ4WMTgNHAeuAVQiugvdz1tysi0qdpzJaIohlqiVTTgYvCy/i+\nDuQA5eFjb7QbmAG+amZvA68BJe3O259jgX+4e4u7bwBeBA5v99kV7t4KvAWUdsndiIhENo3Z0qdp\nhloilQFfcfcn99oZqturf9/2ScBR7t5gZi8AiZ34dXe2+7kF/R0TEekIjdnSp2mGWiLFdiCt3faT\nwBVmFg9gZsPMLGUf12UAW8MD8wjgyHbHmnZf/z4vAeeHa/7ygOOAN7rkLkREooPGbIko+peYRIr5\nQEv4MeBfgJsJPbqbG37JZCPwsX1c9wRwuZktBpYSeoS42+3AfDOb6+6farf/YeAo4G3AgevdvSo8\nuIuIyIFpzJaIorZ5IiIiIiKdoJIPEREREZFOUEItIiIiItIJSqhFRERERDpBCbWIiIiISCcooRYR\nERER6QQl1CIiIiIinaCEWkRERESkE5RQi4iIiIh0wv8H/2RJrFR7PGoAAAAASUVORK5CYII=\n", "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "reload(bilstm);\n", "bilstm.plot_results(losses, accuracies)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Deliverable 5.3**: (*0.5 points*)\n", "- As you can see from above, BiLSTM model performs worse than the Viterbi tagger.\n", "- Use pretrained embeddings like the following to improve your performance.\n", "- Check the following sources: [fastText](https://github.com/facebookresearch/fastText), [Polyglot](polyglot.readthedocs.io/en/latest/Embeddings.html), [word2vec](https://code.google.com/archive/p/word2vec/), [Glove](https://nlp.stanford.edu/projects/glove/)\n", "- Tune the hyperparameters such that you obtain atleast `85.00%` accuracy on the dev set.\n", "- You can try changing the no. of iterations, optimizers, no. of LSTM layers, the hidden dimension units, the FC layer dimensions, or use pretrained word embeddings like word2vec, fastText, polyglot, character features etc..\n", "- **Tests**: ```test_bilstm.py: test_bilstm_dev_accuracy(), test_bilstm_test_accuracy()```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- **Remember: After training a model once, everytime when you rerun the train_model(...) function, the model is retrained on top of the previous parameters. If you want to start afresh, make sure the model is reinitialized and then trained.**" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Example: pretrained embeddings\n", "\n", "- *Polyglot* provides pre-trained word embeddings for many languages\n", "- You can download the english polyglot embeddings [here](https://sites.google.com/site/rmyeid/projects/polyglot).\n", "- Use the following helper function to load in the polyglot embeddings, which you can use to initialize your parameter embeddings in your model. You need to make sure your embedding_dim matches your embeddings size." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "reload(bilstm);\n", "filename = 'data/polyglot-en.pkl'\n", "word_embeddings = bilstm.obtain_polyglot_embeddings(filename, word_to_ix)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- Let us have a look at these word embeddings. We can observe the cosine similarity between two word vectors." ] }, { "cell_type": "code", "execution_count": 96, "metadata": { "collapsed": true }, "outputs": [], "source": [ "def cosine(emb1, emb2): #function to return the cosine similarity between the embeddings\n", " return emb1.dot(emb2) / (np.linalg.norm(emb1) * np.linalg.norm(emb2))" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true, "scrolled": true }, "outputs": [], "source": [ "diremb = word_embeddings[word_to_ix['dire']]\n", "catemb = word_embeddings[word_to_ix['catastrophic']]\n", "amazemb = word_embeddings[word_to_ix['success']]\n", "print(cosine(diremb,catemb)) # dire and catastrophic are similar\n", "print(cosine(diremb,amazemb))\n", "print(cosine(catemb,amazemb))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- Now, all we need to do is send in the word_embeddings when initializing your model" ] }, { "cell_type": "code", "execution_count": 101, "metadata": { "collapsed": true }, "outputs": [], "source": [ "torch.manual_seed(765);\n", "embedding_dim=64\n", "hidden_dim=30\n", "model = bilstm.BiLSTM(len(word_to_ix),tag_to_ix,embedding_dim, hidden_dim, word_embeddings)" ] }, { "cell_type": "code", "execution_count": 102, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Epoch 1: Dev Accuracy: 0.2638724133386055\n", "Epoch 2: Dev Accuracy: 0.5257677606432055\n", "Epoch 3: Dev Accuracy: 0.627125345986556\n", "Epoch 4: Dev Accuracy: 0.7076578357717148\n", "Epoch 5: Dev Accuracy: 0.7502306577039674\n" ] } ], "source": [ "loss = torch.nn.CrossEntropyLoss()\n", "model, losses, accuracies = bilstm.train_model(loss, model, X_tr,Y_tr, word_to_ix, tag_to_ix, \n", " X_dv, Y_dv, num_its=5, status_frequency=1, \n", " optim_args = {'lr':0.01,'momentum':0}, param_file = 'best.params')" ] }, { "cell_type": "code", "execution_count": 103, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAtQAAACdCAYAAACU5e99AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xl8lOW5//HPlYUEkpCFLCxJSAhbBGRHFoEA7rVaW+tW\nrdpShGqP3Wt7ek5Pt/Nrf/3V1moLUrVqpbWntbYetdWyo4JsIgphDWEnAQn7muT6/TEjRgwQSGZJ\n8n2/XvPKzPM8M8+Vh5mbK/fc932ZuyMiIiIiIhcmJtIBiIiIiIg0Z0qoRUREREQaQQm1iIiIiEgj\nKKEWEREREWkEJdQiIiIiIo2ghFpEREREpBGUUIuIiIiINIISahERERGRRlBCLSIiIiLSCHGRDuB8\nZWZmekFBQaTDEBG5IMuWLdvj7lmRjiNc1GaLSHPW0Da72SXUBQUFLF26NNJhiIhcEDPbHOkYwklt\ntog0Zw1ts0M25MPM8sxsjpmtNrNVZnZ/PceUmNl+M1sRvP1nqOIREREREQmFUPZQVwNfc/flZpYC\nLDOzf7n76tOOW+Du14YwDhERERGRkAlZQu3uO4GdwfsHzawU6AKcnlCH3M9eWcO+IyeZNKYbXTsk\nhfv0IiIiIhIGB4+dZFvV0eDtyKmfRVnJfPOq3iE7b1jGUJtZATAQeLOe3SPNbCWwHfi6u6+q5/mT\ngEkA+fn5533+4ydr+fPSbfxx8RY+dnFnJo/tRp/Oqef9OiIiIiISOWdKmN/ftv/oyQ8d3zY+ltz0\ntuSltwtpXObuoT2BWTIwD/ixu//1tH3tgVp3P2Rm1wAPuXuPs73ekCFD/EImuFQeOMbjr29ixqIt\nHDpezdieWUwpKeKSwgzM7LxfT0TkQpjZMncfEuk4wuVC22wRaZ0OHjvJ9n1H2bb3w4nytn2B+/uO\n1J8wB27tTvvZloykNo3K8xraZoe0h9rM4oHngBmnJ9MA7n6gzv2Xzew3Zpbp7nuaOpbs9ol8++pi\nvljSnWcWbeaJ1zZxy/RFDMpPY0pJdyb0ziYmRom1iIiISKgcOl4dSJQbmDAnxsecSpAH5KWRm96O\nvCZMmJtKyBJqC/x2jwOl7v7gGY7pCFS4u5vZMAKrjrwXqpgAUtvGc++47nz+0kL+vHQrj84v4wtP\nL6VHdjKTxxZx3YDOxMeq3o2IiIjI+WqKhLluL3OHKEmYzyWUPdSjgDuAd8xsRXDbd4B8AHefBtwI\nTDGzauAocIuHegxKUGJ8LHeMKODWYfm89M5Ops7dyNf+/DYP/msdE0cXcvPQPNq1aXbLdIuIiIiE\nTGtNmM8l5GOom1qoxuO5O3PWVjJ17kaWlFeR3i6eu0YWcufIrqS1a9Pk5xOR1kljqEUkmjUmYa5v\nHHNzT5ijYgx1c2JmjO+dw/jeOSwp38u0uRv5xcx1PDp/I7cNy+fzowvplNo20mGKiIiIXLBDx6vZ\nfoYVMrZVHaGqlfYwN5YS6noMLchg6F0ZrNl1gEfnlfG7N8p5amE5Nwzswj1jiyjKSo50iCIiIiIf\ncfREDVv2HmlwwpwQF3MqQb44N/Ujvc2ZyUqYG0IJ9Vn07tieX9w8gK9e3pPHFpTx7JKt/HnZNq68\nqCNTSoron5cW6RBFRESkFdtz6DhLy/eyeFMVSzfvZdWOA9TUfjCcVwlzeCihboC8jHZ8//q+fGlC\nD558vZynF5bzz1W7GFnUgS+WdGdU9w56M4pIVDCznwNP1FckS0SaN3dny94jLN60l6XlVSwp30vZ\nnsNAIHEekJfGlLFF9OqYooQ5zJRQn4fM5AS+fmUv7hnbjT8u3sJjCzZx++Nv0q9LKlNKiriyT0di\ntZa1iERWKTDdzOKA3wF/dPf9EY5JRC5ATa1TuvMAS8o/SKArDx4HAssADy1I56aheQwtyKBfl1Ta\nxGnZ30hRQn0BUhLjmTSmiDtHFvD88u08Or+ML85YTmFmEveM6cYNg7qQEBcb6TBFpBVy98eAx8ys\nF3A3sNLMXgd+6+5zIhudiJzNsZM1rNi6LzCEo7yK5ZurOHS8GoAuaW0ZWdSBIQUZDCvMoHtWsgrS\nRREl1I2QEBfLLcPy+fSQPF5ZtYvfzN3AA399h1/MXMfES7tx6yX5JCfoEotIeJlZLNA7eNsDvA18\n1czucfdbIhqciJyy78iJQM/z5r0s2bSXd7bv52RNYPxzr5wUrh/QmWGFGQwpyKBLmlYai2bK9ppA\nbIxxTb9OXN23I69t2MPUuRv58culPDx7PXeOLOCukQV0SE6IdJgi0gqY2S+Aa4HZwH+7++Lgrp+a\n2drIRSYi2/cdDU4g3MuS8r2sqzgEQHys0a9LKp+7tJBhBRkM7pquGhjNjBLqJmRmjO6RxegeWazY\nuo9pczfyyJwN/HZBGTcPyWPi6G7kZbSLdJgi0rKtBL7r7ofr2Tcs3MGItFa1tc76ykMsKd97agz0\n9n1HAUhOiGNQ13Su69+ZIQUZDMhLIzFeQ0WbMyXUITIgL41pdwxmQ+Uhps/fyB8Wb+GZN7dwXf/O\nTA7OwBURCYF91GnbzSwNKHH3v2lyokjonKiu5Z3t+4PJ816Wbq46VVUwKyWBYQUZTBxdyNCCDIo7\ntdciBi2MSo+Hyc79R3l8wSb+sHgLR07UcFlxNlNKihjcNSPSoYlIGIW69LiZrXD3Aadte8vdB4bq\nnGfTXNtskXM5eOwky7fsY0lw+MaKrfs4Xl0LQLfMJIYWZDCkIJ1hhRnkZ7TT0nXNlEqPR5lOqW35\n7rUXce+47jy9cDNPvrGJT01dyLCCDKaUFFHSK0sfNhFpCvWtm6W2XqSRKg8cY0lw6bol5Xsp3XmA\nWg/Mo+rTuT2fuaQrwwrTGdw1g6wUzZtqbdTIhll6Uhvuv6wHXxhTyJ+WbOW388u4+8kl9O6YwpSS\nIj7WrxNxsVpHUkQu2FIzexD4dfDxvcCyCMYj0uy4O5v2HA4mz4EkevN7RwBIjI9hYF46943vwbCC\nDAbkp2lFL9GQj0g7WVPLCyt2MG3eRtZXHiIvoy2TxhTx6cG5mqAg0gKFYchHEvAfwGXBTf8CfnSG\nSYoh19LabGmZqmtqWb3zQCB53rSXpZv3sufQCQDS28UH1n4ODuHo2yWVeHV8tRoa8tFMxMfG8KnB\nudwwsAszSyv4zdyN/Mff3uWhmeu4e1Qhtw/vSmrb+EiHKSLNRDBxfiDScYhEsyMnqlmxZd+p3ufl\nW6o4cqIGgNz0tozpkRUsoJJOUVayhmTKOSmhjhIxMcYVfTpy+UU5vLlpL1PnbuRnr6xl6tyNfGZ4\nPp8fVUh2+8RIhykiUc7MsoBvAn2AU42Gu4+PWFAiEbb38IlTq28sLq9i1fb9VNc6ZoECKjcOzmVI\nQQZDC9LplKoCKnL+lFBHGTNjeLcODO/WgVU79jNtXhm/nV/G714v58bBuUwa3Y2CzKRIhyki0WsG\n8CcCxV0mA3cCuyMakUgYuTvbqo6emjy4pLyKDZWBAiptYmPon5fKF8Z0Y1hBBoO6putbYGkSSqij\nWJ/OqTx860C+fkVPps8v48/LtvHs4i1c068Tk8cW0bdLaqRDFJHo08HdHzez+919HjDPzJZEOiiR\nUKmpddZVHGRJsALh0vIqdh04BkBKYhxDuqZzw8AuDCvMoF+XVM1PkpBQQt0MdO2QxI9v6Mf9l/Xg\nd6+X88zCzby4cidjemYxZWwRw7tlaHyXiLzvZPDnTjP7GLAD0IL30qJsqzrCnDWVzFm7myXlezl4\nrBqAnPYJDC3IYFhhBkO6ZtCrY4oKqEhYhCyhNrM84GkgB3Bgurs/dNoxBjwEXAMcAe5y9+Whiqm5\ny05J5FtX9WZKSRHPLNrME69t4tbfLmJAXhpTSoq4vDiHGDUcIq3dj8wsFfga8DDQHvhKZEMSaZya\nWmfF1n3MKq1g9ppK1uw6CEDXDu249uJODOkaSKJz09uqg0kiIpQ91NXA19x9uZmlAMvM7F/uvrrO\nMVcDPYK3S4CpwZ9yFu0T4/liSXc+N6qQvyzbxvT5Zdzz+2V0z05m8tgiruvfmTZxWtJHpLUxs1ig\nh7u/COwHxkU4JJELdvDYSeav28OsNRXMXbubvYdPEBtjDOmazneu6c2E4hy6ZSYpgZaoELKE2t13\nAjuD9w+aWSnQBaibUF8PPO2BxbAXmVmamXUKPlfOITE+ltuHd+WWoXm8/O4ups7dyNf//DYPvrqW\niaO7ccuwPNq10agekdbC3WvM7FbgFxfyfDO7isC3hrHAY+7+k3qOKQF+CcQDe9x97IVHLPJh5XsO\nM2tNJbPXVPBm2V6qa53UtvGU9MpiQnEOY3tkkdpOkwgl+oQl2zKzAmAg8OZpu7oAW+s83hbcpoT6\nPMTFxnBd/858/OJOzF23m6lzN/KDF1fz8Oz13DmygDtHFJCe1CbSYYpIeLxuZo8QWOnjVDGXcw2n\nC/Zu/xq4nEBbvMTMXqj7raKZpQG/Aa5y9y1mlh2KX0Baj+qaWpZurmL2mkpmllZQtjvwlu2enczn\nRxcyoXcOg/LTVEFYol7IE2ozSwaeA77s7gcu8DUmAZMA8vPzmzC6lsXMGNcrm3G9slm2eS9T55bx\ny5nrmT6/jFuH5TNxdKHW1xRp+QYEf/6gzjYHzrUO9TBgg7uXAZjZswS+Raz7reJtwF/dfQuAu1c2\nScTSquw7coJ563Yzq7SSuWsrOXCsmvjYwJKxdwzvyvje2XTtoOVhpXkJaUJtZvEEkukZ7v7Xeg7Z\nDuTVeZwb3PYh7j4dmA6BMrYhCLXFGdw1g8fuzGBdxUGmzdvIk2+U8/TCcj4xoAv3jC2ie3ZypEMU\nkRBw9wsdN13fN4anz2npCcSb2VwgBXjI3Z++wPNJK+HubNx9iJmllcwurWTp5r3UOnRIasMVfToy\noXc2o3tmkZygIYrSfIVylQ8DHgdK3f3BMxz2AnBfsCfkEmC/xk83rZ45KTx40wC+enlPHluwiWeX\nbOEvy7dxxUU5TCnpzoC8tEiHKCJNyMz+s77t7v6D+rafpzhgMDABaAssNLNF7r7utBj0rWIrd6K6\nljc3vces0kpmr6lky94jABR3as+947ozvnc2/XPTtDKVtBih/HNwFHAH8I6ZrQhu+w6QD+Du04CX\nCSyZt4HAsnl3hzCeVi03vR3/dV0fvjS+O0+9Uc5TCzfzyqrXGdGtA1NKihjdI1MzpUVahsN17icS\nqJhY2oDnNeQbw23Ae+5+GDhsZvOB/sCHEmp9q9g67Tl0nDlrAgn0gvV7OHS8mjZxMYwq6sCkMd0Y\n3zubzmkadigtkwUW2Gg+hgwZ4kuXLo10GM3eoePVPLt4C48t2MSuA8fo07k9U0qKuLpvJy2CLxJC\nZrbM3YeE8XwJwCvuXnKO4+IIJMYTCCTSS4Db3H1VnWOKgUeAK4E2wGLgFnd/90yvqza75XJ3Snce\nZPaaCmatqWTF1n24B4qrjO+dw4Te2Yzs3kGrTUmz1tA2W+/yVio5IY6Jo7vx2REF/G3FdqbN28h9\nf3iLrh3WMmlMNz41KFflWUVahnYEepvPyt2rzew+4BUCy+Y94e6rzGxycP80dy81s38CK4FaAkvr\nnTGZlpbn2MkaFm58j5nBAis79wdKfPfPTeXLE3oyoTibPp3b6xtPaXXUQy0A1NY6r66uYOq8jby9\ndR+ZyQl8/tJCPjM8n/aJWvNTpKmEuofazN4hsKoHBBLjLOAH7v5IqM55Nmqzm7+KA8eCY6EreG3D\nHo6drKVdm1gu7Z7JZcU5lPTOIjslMdJhioSEeqjlvMTEGFf17ciVfXJYWPYe0+aV8dN/ruE3czZw\n+4iu3D2qQA2mSPNwbZ371UCFu1dHKhhpfmprnXe27z9VYOXd7YEVb7ukteXmIXmML87hksIMfYsp\nUocSavkQM2NkUSYjizJ5d/t+ps3byKPzNvL4a5u4cXAuk0Z3oyBT64OKRLFOwCp3PwhgZilmdpG7\nn15YS+SUw8ereW3DHmaXVjJ7bSW7Dx4nxmBQfjrfvKoXE3rn0DMnWUM5RM5ACbWcUd8uqTxy2yDK\n9xxm+oIy/rJ0G88u3sI1/ToxeWwRfbukRjpEEfmoqcCgOo8P17NNhG1VR5i9ppJZpZUsLHuPE9W1\npCTEMaZXFhN6Z1PSK5sMVdkVaRAl1HJOBZlJ/PcN/fjyhB488Xo5MxZt5sWVOxndI5MpJUWM6NZB\nvRYi0cO8zuQYd68NruAhrVxNrbNia9WpAitrKw4CUNChHXcM78qE4myGFmQQrzLfIudNjaw0WHb7\nRB64ujdfHFfEjEVbePy1Tdz22zfpn5fGlLFFXHFRjhbpF4m8MjP7NwK90gBfBMoiGI9E0IFjJ5m/\nbjezSyuZs7aSqiMniY0xhhVk8N2PFTO+dzbdslQ5V6SxlFDLeWufGM+UkiLuHlXAc8u38ei8MiY/\ns4xuWUlMHlvEJwZ0oU2cejhEImQy8CvguwRW+5hFsGqhtA6b9hxmVnBZu8Wb9lJd66S1i2dcr2zG\n985mTM8sUttq9SaRpqRl86TRqmtq+ce7u5g2byOrdhygY/tEJo4u5NZh+SQl6G82kbrCXdgl0tRm\nh97JmlqWlledKrBStjtQLLNnTnKgwEpxNgPz0ojTUA6R89aky+aZ2f3A74CDwGPAQOABd3+1UVFK\nixAXG8PH+3fm2os7sWD9HqbO3ciPXirl4dkbuHNEV+4cWUCH5IRIhynSKpjZU8D97r4v+Dgd+Lm7\nfy6ykUlTqjp8gnnrdjOztIJ563Zz8Fg1bWJjuKRbBp8d3pUJxTnkZbSLdJgirUZDuw8/5+4PmdmV\nQDpwB/B7QAm1nGJmjOmZxZieWby1pYpp8zby8JwNTF9Qxi1D85k4upDcdDXwIiF28fvJNIC7V5nZ\nwEgGJE1jQ+VBZpZWMqu0gmWbq6h1yExO4Oq+HRnfO4dLe2SSrG8FRSKioZ+892eaXQP8PliOVrPP\n5IwG5qfz6B1D2FB5iOnzNzLjzc38ftFmruvfmclji+jVMSXSIYq0VDFmlu7uVQBmloHmyzRrx6tr\n+OGLq3lm0RYA+nRuz33jujO+OIeLu6RqMrhIFGhoI7vMzF4FCoFvm1kKUBu6sKSl6J6dzP+9sT9f\nubwnjy/YxB8Wb+H5t7YzoXc2U0qKGFKQEekQRVqanwMLzezPBDpDbgR+HNmQ5EJtqzrCvTOW8/a2\n/Uy8tJDPjy6kU2rbSIclIqdp0KREM4sBBgBl7r4v2OOR6+4rQx3g6TTBpXnbd+QETy/czO9e30TV\nkZMMLUhnSkkR43play1raRXCMSnRzPoA44IPZ7v76lCe72zUZl+4OWsr+cqfVlBT4/y/m/pzZZ+O\nkQ5JpNVp0kmJwAhghbsfNrPbCVTceqgxAUrrlNauDf82oQcTRxfyP0u28tsFm/jck0vplZPC5JJu\nXHtxZxUVEGmk4LC83UAigJnlu/uWCIclDVRT6zw0cx0Pz9lAr5wUpt0+mILMpEiHJSJn0dDMZSpw\nxMz6A18DNgJPhywqafHatYnjrlGFzP1GCb+4uT+O85U/vU3Jz+by1BvlHD1RE+kQRZolM7vOzNYD\nm4B5QDnwj4gGJQ323qHj3PW7xfxq9gZuHJTL3+4dpWRapBloaEJdHSxlez3wiLv/GtCsMmm0+NgY\nbhiYyz/vH8Pjdw6hU2oi33thFaN+OptfzVrPviMnIh2iSHPzQ2A4sM7dC4EJwKLIhiQNsWxzFR/7\n1Wu8uWkvP/1UP3726f4kxsdGOiwRaYCGDvk4aGbfJrBc3ujgmGqVWZImExNjTCjOYUJxDkvK9zJt\n7kYe/Nc6ps3byG3D8jURR6ThTrr7e2YWY2Yx7j7HzH4Z6aDkzNydJ98o58cvldI5rS1/nTKSvl1S\nIx2WiJyHhibUNwO3EViPepeZ5QM/C11Y0poNLchg6F0ZrNl1gEfnlfG7N8p5amE5NwzswqQxRXTP\nTo50iCLRbJ+ZJQPzgRlmVgkcjnBMcgaHjlfzredW8tLKnVxWnMPPb+qvsuAizVCDS4+bWQ4wNPhw\nsbtXhiyqs9CM8dZn694jPLagjD8t3crx6lquuCiHKSXdGZCXFunQRM5bqFf5MLMk4CiBIX2fAVKB\nGe7+XqjOeTZqs89sXcVBJj+zjPI9h/nGlb25Z0w3rSktEmUa2mY3aAy1md0ELAY+DdwEvGlmN57j\nOU+YWaWZvXuG/SVmtt/MVgRv/9mQWKT1yctox/ev78vr3xrPl8Z1Z1HZXj7x69e5dfoi5q/bTUP/\nKBRpDdz9sLvXunu1uz/l7r+KVDItZ/a3t7Zz/SOvc+BoNTMmDmdKSZGSaZFmrKFDPv4dGPp+r7SZ\nZQEzgb+c5TlPAo9w9tVAFrj7tQ2MQVq5DskJfPWKXkwaW8Szi7fw2IJNfPaJxfTp3J7JY4u4pl8n\nYvUfkohEsbpVD4cVZPDIbQPJbp8Y6bBEpJEauspHzGlDPN4713PdfT6w90IDEzmT5IQ4Jo7uxvxv\njuP/fupijp6s4Ut/fIvxP5/LjDc3c+ykltwTkeizreoIN01byDOLtjBpTDdmfOESJdMiLURDE+p/\nmtkrZnaXmd0FvAS83ATnH2lmK83sH8HKXvUys0lmttTMlu7evbsJTistQZu4GG4amsfMr4xl2u2D\nSWsbz78//y6X/nQOU+du5MCxk5EOUSTszOz+hmyT8JqztpJrH36Nst2HmXb7YL5zTbGKWIm0IOcz\nKfFTwKjgwwXu/nwDnlMAvOjufevZ1x6odfdDZnYN8JC79zjXa2qCi5yJu7Ow7D2mzt3IgvV7SEmI\n4/YRXbl7VAHZKeoFkugQhkmJy9190Gnb3nL3gaE659m09jb79KqHU28fTKEKtYg0G01dehx3fw54\nrlFRffj1DtS5/7KZ/cbMMt19T1OdQ1oXM2NkUSYjizJ5d/t+ps7byKPzNvL4a5u4cXAuk0Z3U8Ux\nabHM7FYCy5sWmtkLdXaloOF3EfHeoeN8+U8rWLB+D58enMsPP9FXhVpEWqizJtRmdhCorwvbAHf3\n9hd6YjPrCFS4u5vZMALDTzQTXZpE3y6p/Pq2QZTvOcz0BWX8Zek2nl28hWv6dWLy2CIVTZCW6A1g\nJ5AJ/LzO9oPAyohE1Iot21zFfX9YznuHT/DTT/Xj5qH5kQ5JRELorAm1u19weXEz+yNQAmSa2Tbg\newSrK7r7NOBGYIqZVRNYM/UW1/pn0sQKMpP47xv68eUJPXji9XJmLNrMiyt3MrpHJlNKihjRrQNm\nWhlEmj933wxsBkaYWVegh7vPNLO2QFsCibWEWN2qh53SElX1UKSVaPAY6mjR2sfjSeMcOHaSGYu2\n8Phrm9hz6Dj989KYMraIKy7K0RqwEhZhGEP9BWASkOHuRWbWA5jm7hNCdc6zaU1t9keqHn66P6nt\nVPVQpDlr8jHUIi1B+8R4ppQUcfeoAp5bvo1H55Ux+ZlldMtKYvLYIj4xoAtt4jTzXpq1e4FhwJsA\n7r7ezLIjG1LLt67iIFOeWcamPYf51lWqeijS2iihllYpMT6Wz1zSlZuH5PGPd3cxbd5GvvmXlfzs\nlbVc2SeHCcU5jOjWQROIpDk67u4n3h/KZGZx1D8XRprI31ds54Hn3iEpIY4ZE4czoqhDpEMSkTBT\nQi2tWlxsDB/v35lrL+7EgvV7mPHmZv66fDvPLNpCuzaxXNo9k8suymF872wykxMiHa5IQ8wzs+8A\nbc3scuCLwP9GOKYW6Xh1DT96sZTfL9rMsIIMHr5tIDkq1CLSKimhFiGw5N6YnlmM6ZnFsZM1LCp7\nj5mlFcwqreTV1RWYwYC8NC4rzmFCcTa9clI0mVGi1QPA54F3gHsIFOF6LKIRtUDbqo5w74zlvL1t\nP5PGdOMbV/ZSoRaRVkyTEkXOwt1ZvfMAs0ormVlawcpt+wHITW97Krm+pLCDxl1Lg4V6UuJp58oA\nct09YsvmtcQ2e+7aSr78pxXU1Dg/+3R/rurbMdIhiUiIaFKiSBMwM/p0TqVP51T+bUIPKg4cY/aa\nSmaVVvDski08+UY5yQlxjOmZyWXFOYzrlU16UptIhy2tmJnNBa4j0L4vAyrN7A13/0oDnnsV8BAQ\nCzzm7j85w3FDgYUEljv9S1PFHu1qap2HZq3n4dnrVfVQRD5ECbXIechpn8itw/K5dVg+R0/U8MbG\nPaeGhrz8zi5iDAZ3TWdCcQ6XFWdTlJWsoSESbqnufsDMJgJPu/v3zOycPdRmFgv8Grgc2AYsMbMX\n3H11Pcf9FHg1BLFHrbpVD28cnMsPr+9L2zaatCwiAUqoRS5Q2zaxTCgOrAhSW+u8u2M/M0sDvdc/\n+ccafvKPNXTt0O7U0JChBRkaYynhEGdmnYCbgH8/j+cNAza4exmAmT0LXA+sPu24LwHPAUObINZm\nYfmWKu6dEah6+JNP9uPmoXn6Q1lEPkQJtUgTiIkxLs5N4+LcNL56eU927DvKrODQkN8v2szjr22i\nfWIcY3tlc1lxNiU9s1XwQULlB8ArwGvuvsTMugHrG/C8LsDWOo+3AZfUPcDMugA3AOM4S0JtZpMI\nFJchP7/5ltx2d556o5wfv1xKx1RVPRSRM1NCLRICndPacsfwrtwxvCuHj1ezYP0eZpVWMGdtJf/7\n9g5iY4yhBenB3uscjcOUJuPufwb+XOdxGfCpJnr5XwLfcvfas/XQuvt0YDoEJiU20bnD6tDxah54\nbiUvrtzJZcXZ/PzTA/RHsIickRJqkRBLSojjqr4duapvR2prnRXb9jGrtIKZqyv50Uul/OilUrpl\nJXF5MLkelJ9GnIaGSPhtB/LqPM4NbqtrCPBsMJnOBK4xs2p3/1t4QgyP9RUHmRysevjNq3oxeUyR\nqh6KyFkpoRYJo5gYY1B+OoPy0/nGlb3ZuvcIs0ormLWmkide38Sj88tIaxfPuF7ZTCjOZkzPLNon\nqldMwmIJ0MPMCgkk0rcAt9U9wN0L379vZk8CL7a0ZPrvK7bz7b++Q7s2sTwz8RJGFmVGOiQRaQaU\nUItEUF5GO+4aVchdowo5eOwk89d9MDTk+be2Ex9rXFLYgQnF2VxWnENeRrtIhyxRzsxi3b3mfJ/n\n7tVmdh8FfPe0AAAPoUlEQVSB8dexwBPuvsrMJgf3T2viUKOKqh6KSGOosItIFKqpdZZvqWJmaQUz\nV1ewcfdhAHrmJJ8adz0gL41YfQ3d7IS6sIuZbQH+CfwJmO0RbuSbQ5u9reoI9/7hLd7euk9VD0Xk\nQ1TYRaQZC0xazGBoQQbfvrqY8j2HT613/ej8Mn4zdyMdktowrndg1ZDRPbJIStDHWQDoDVwL3As8\nbmYvAs+6+2uRDSs61a16OO32QVzVt1OkQxKRZkj/A4s0AwWZSUwc3Y2Jo7ux/8hJ5q6rZFZpJa+u\n2sVflm2jTWwMI4o6cFlxNhOKc+ic1jbSIUuEuPsR4H+A/zGzdAKVD+cRGMYhQap6KCJNSQm1SDOT\n2i6e6wd04foBXThZU8vS8qpg73UF//H3VfzH31dR3Kk9lweT635dUrVCQStjZmOBm4GrgKUEirxI\n0N7DJ7j/2bdU9VBEmowSapFmLD7YMz2iqAPf/VgxG3cfDqwaUlrJI3M28KvZG8hKSWBC78CkxlHd\nM5U4tHBmVg68RaCX+hvufjiyEUUXVT0UkVBQQi3SQpgZ3bOT6Z6dzD1ji6g6fII5awNDQ15cuZNn\nl2wlIS6GS7tnBkumZ2sVg5bpYnc/EOkgoo2qHopIKCmhFmmh0pPa8MlBuXxyUC4nqmtZvGlvYNWQ\n4LrXPA/9uqQGVw3Jpk/n9uqpaxk6mtnzQI679zWzi4Hr3P1HkQ4sUg4fr+aBv77D/769Q1UPRSQk\nQrZsnpk9QWCmeaW7961nvxGYLHMNcAS4y92Xn+t1m8MSTCLRzN1ZV3HoVHK9Yus+3KFTaiLjg0ND\nRhR1IDFeQ0NCIQzL5s0DvgE86u4Dg9vera8dDodIt9l1qx5+/UpVPRSR8xMNy+Y9CTwCPH2G/VcD\nPYK3S4CpwZ8iEkJmRq+OKfTqmMK947qz59BxZq+pZFZpBc+/tZ0Zb26hXZtYLu2eyWXFOYzrnU1W\nSkKkw5aGa+fui0/7tqE6UsFEkqoeiki4hCyhdvf5ZlZwlkOuB54OFh1YZGZpZtbJ3XeGKiYR+ajM\n5ARuGpLHTUPyOHayhoVl752a2Pjq6goA8jPacVGn9vTp3J6LOrenT+dUctonaIhIdNpjZkWAA5jZ\njUCralePV9fw45dKeXrhZoYWpPPIbYM0X0BEQiqSY6i7AFvrPN4W3PaRht/MJgGTAPLz88MSnEhr\nlBgfy7he2Yzrlc0Pr3dW7zzAvHW7eXf7flbvOMA/V+06dWyHpDZcFEywA8l2KoWZSareGHn3AtOB\n3ma2HdgE3B7ZkMJn+76jfHHGct7euo8vjC7km1f1VtVDEQm5ZjEp0d2nE/gPgiFDhjSvWukizZSZ\n0adzKn06f7ASwsFjJyndeZDVO/azascBVu88wBOvbeJkTeBj2TY+lt6dUugT7MW+qFN7enVM0Xjs\nMHL3MuAyM0sCYtz9YKRjCpd563bz5Wff4qSqHopImEUyod4O5NV5nBvcJiJRKiUxnmGFGQwrzDi1\n7UR1LRsqD7GqTpL997d28MyiLUCgjHpRVlIwOQ/0Zl/UuT1p7dpE6tdokczsq2fYDoC7PxjWgMKo\nptb51az1/EpVD0UkQiKZUL8A3GdmzxKYjLhf46dFmp82cTGnhn58OritttbZWnWE1TsOnEqy39i4\nh+ff+uBv5i5pbYPjsYNDRrqk0jk1UeOyL1xK8GcvYCiBNhbg48DiiEQUBnWrHn5qUC4/+oSqHopI\n+IUsoTazPwIlQKaZbQO+B8QDuPs04GUCS+ZtILBs3t2hikVEwismxujaIYmuHZK4ut8HX7vvOXT8\nQ0n2qh37mVlawfurd6a1i//I5MdumUnEaQzsObn79wHMbD4w6P2hHmb2X8BLEQwtZN4KVj3cc/gE\n/+eT/bhFVQ9FJEJCucrHrefY7wQmz4hIK5GZnMCYnlmM6Zl1atvh49Ws2RUYlx1Isg/w1MLNnKiu\nBSAhLobeHVO46P0hI53bU9yxvXohzywHOFHn8YngthajbtXDnPaqeigikdcsJiWKSMuVlBDH4K7p\nDO6afmrbyZpaynYfZtWO/ad6tF9auYM/Lg6My44xKMysMy472JudkaRx2QTW/l8crJYI8AkCdQFa\nhLpVDyf0zubBm1T1UEQiTwm1iESd+NiYU8VnPjkosM3d2b7vaGC4SDDJXlq+lxfe3nHqeR3bJwZX\nGPkgyc5Nb9uqhgG4+4/N7B/A6OCmu939rUjG1FQ2VB5k8jPLKdt9iG9c2YspY1X1UESigxJqEWkW\nzIzc9Hbkprfjyj4dT22vOnyC1TvfT7IDK43MWVtJbXBcdkpi3Kl1st+fBNk9O7lFr03s7suB5ZGO\noymp6qGIRDMl1CLSrKUntWFU90xGdf8gwTp2soY1uw5+aMjIHxZv5tjJwLjsNrEx9OyYTJ9OHyTZ\nxZ3ak5SgJjHa1K16OKRrOr/+jKoeikj00f8eItLiJMbHMiAvjQF5aae21dQ6m/Yc+tCQkVdX7+JP\nSwMFW82goEPSh5fy65xKVkpCpH6NVq9u1cOJlxbyratV9VBEopMSahFpFWJjjO7ZKXTPTuH6AV2A\nwLjsXQeOsWr7B8v4rdy2j5dWfrAkflZKwgfjsjsFJkHmZ7TT2N0QU9VDEWlOlFCLSKtlZnRKbUun\n1LZcdtEHK8vtP3qS1XXWyl694wCvrd9DdXBgdnJCHF+7oid3jyqMVOgt2pOvb+L7L65W1UMRaTaU\nUIuInCa1bTwjijowoqjDqW3HTtawvuIQq3cGJj4qyQudAfnp3DQ4j/+6ro/WGxeRZkEJtYhIAyTG\nx9IvN5V+uSogEmqnj38XEYl2mt0hIiIiItIISqhFRERERBpBCbWIiIiISCOYu0c6hvNiZruBzRfw\n1ExgTxOHc6GiJZZoiQOiJ5ZoiQMUS32iJQ648Fi6untWUwcTrdRmN6loiQMUS32iJQ6InliiJQ4I\ncZvd7BLqC2VmS919SKTjgOiJJVrigOiJJVriAMUSzXFAdMXSEkXT9Y2WWKIlDlAs0RwHRE8s0RIH\nhD4WDfkQEREREWkEJdQiIiIiIo3QmhLq6ZEOoI5oiSVa4oDoiSVa4gDFUp9oiQOiK5aWKJqub7TE\nEi1xgGKpT7TEAdETS7TEASGOpdWMoRYRERERCYXW1EMtIiIiItLkWlxCbWZXmdlaM9tgZg/Us9/M\n7FfB/SvNbFCE4igxs/1mtiJ4+88QxfGEmVWa2btn2B+W69HAWMJ1TfLMbI6ZrTazVWZ2fz3HhOt9\n0pBYQn5dzCzRzBab2dvBOL5fzzHhuiYNiSUs75XguWLN7C0ze7GefWH7/LRUarPrjSUq2m212Rcc\nS7iuS1S022qzg9y9xdyAWGAj0A1oA7wNXHTaMdcA/wAMGA68GaE4SoAXw3BNxgCDgHfPsD/k1+M8\nYgnXNekEDAreTwHWReJ9ch6xhPy6BH/P5OD9eOBNYHiErklDYgnLeyV4rq8Cf6jvfOH8/LTEm9rs\nM8YTFe222uwLjiVc1yUq2m212YFbS+uhHgZscPcydz8BPAtcf9ox1wNPe8AiIM3MOkUgjrBw9/nA\n3rMcEo7r0dBYwsLdd7r78uD9g0Ap0OW0w8JyXRoYS8gFf89DwYfxwdvpEyzCdU0aEktYmFku8DHg\nsTMcErbPTwulNrse0dJuq82+4FjCIlrabbXZAS0toe4CbK3zeBsffaM35JhwxAEwMviVwz/MrE8T\nx9BQ4bge5yOs18TMCoCBBP6irivs1+UssUAYrkvwa7IVQCXwL3eP2DVpQCwQnvfKL4FvArVn2B9t\nn5/mRm32hYmm953a7Ai12cEYoqLdVpvd8hLq5mQ5kO/uFwMPA3+LcDzRIKzXxMySgeeAL7v7gVCe\nq5GxhOW6uHuNuw8AcoFhZtY3FOdpolhCfk3M7Fqg0t2XNfVrS7OkNvuj1GZHsM2G6Gm31Wa3vIR6\nO5BX53FucNv5HhPyONz9wPtfkbj7y0C8mWU2cRwNEY7r0SDhvCZmFk+gMZzh7n+t55CwXZdzxRLu\n94q77wPmAFedtivs75UzxRKmazIKuM7MygkMARhvZs+cdkzUfH6aKbXZFyYq3ndqs6OjzQ6eJyra\n7dbcZre0hHoJ0MPMCs2sDXAL8MJpx7wAfDY403M4sN/dd4Y7DjPraGYWvD+MwL/Fe00cR0OE43o0\nSLiuSfAcjwOl7v7gGQ4Ly3VpSCzhuC5mlmVmacH7bYHLgTWnHRaua3LOWMJxTdz92+6e6+4FBD7D\ns9399tMOi5rPTzOlNvvCRMX7Tm125Nrs4GtHRbutNjsgrileJFq4e7WZ3Qe8QmDW9hPuvsrMJgf3\nTwNeJjDLcwNwBLg7QnHcCEwxs2rgKHCLuzf5IH4z+yOB2bWZZrYN+B6BCQNhux7nEUtYrgmBv2Lv\nAN6xwJgvgO8A+XViCdd1aUgs4bgunYCnzCyWQEP3P+7+Yrg/O+cRS7jeKx8RoWvSIqnNrl+0tNtq\nsy84lnBdl2hpt9Vmo0qJIiIiIiKN0tKGfIiIiIiIhJUSahERERGRRlBCLSIiIiLSCEqoRUREREQa\nQQm1iIiIiEgjKKGWZs3M3gj+LDCz25r4tb9T37lEROTCqM2WlkrL5kmLYGYlwNfd/drzeE6cu1ef\nZf8hd09uivhEROQDarOlpVEPtTRrZnYoePcnwGgzW2FmXzGzWDP7mZktMbOVZnZP8PgSM1tgZi8A\nq4Pb/mZmy8xslZlNCm77CdA2+Hoz6p4rWGHpZ2b2rpm9Y2Y313ntuWb2FzNbY2Yz3q8MJSIiarOl\n5WpRlRKlVXuAOr0dwUZ2v7sPNbME4HUzezV47CCgr7tvCj7+nLvvtUDJ1CVm9py7P2Bm97n7gHrO\n9UlgANAfyAw+Z35w30CgD7ADeJ1AVa3Xmv7XFRFp1tRmS4uiHmppqa4APhssDfsm0AHoEdy3uE7D\nDPBvZvY2sAjIq3PcmVwK/NHda9y9ApgHDK3z2tvcvRZYARQ0yW8jItKyqc2WZk091NJSGfAld3/l\nQxsD4/YOn/b4MmCEux8xs7lAYiPOe7zO/Rr0GRMRaQi12dKsqYdaWoqDQEqdx68AU8wsHsDMeppZ\nUj3PSwWqgg1zb2B4nX0n33/+aRYANwfH/GUBY4DFTfJbiIi0DmqzpUXRX2LSUqwEaoJfAz4JPETg\nq7vlwUkmu4FP1PO8fwKTzawUWEvgK8T3TQdWmtlyd/9Mne3PAyOAtwEHvunuu4KNu4iInJvabGlR\ntGyeiIiIiEgjaMiHiIiIiEgjKKEWEREREWkEJdQiIiIiIo2ghFpEREREpBGUUIuIiIiINIISahER\nERGRRlBCLSIiIiLSCEqoRUREREQa4f8D54xnT1x9ocoAAAAASUVORK5CYII=\n", "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "reload(bilstm);\n", "bilstm.plot_results(losses, accuracies)" ] }, { "cell_type": "code", "execution_count": 104, "metadata": { "collapsed": true }, "outputs": [], "source": [ "reload(tagger_base);\n", "reload(bilstm);\n", "confusion = tagger_base.eval_model(model,'bilstm-dev-en.preds', word_to_ix)" ] }, { "cell_type": "code", "execution_count": 105, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "0.7506260709107684\n" ] } ], "source": [ "print (scorer.accuracy(confusion))" ] }, { "cell_type": "code", "execution_count": 106, "metadata": { "collapsed": true }, "outputs": [], "source": [ "tagger_base.apply_model(model,'bilstm-te-en.preds',word_to_ix, all_tags, testfile=TEST_FILE_HIDDEN)" ] }, { "cell_type": "code", "execution_count": 107, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "0.7503780068728523\n" ] } ], "source": [ "# you don't have en-ud-test.conllu, so you can't run this\n", "te_confusion = scorer.get_confusion(TEST_FILE,'bilstm-te-en.preds')\n", "print (scorer.accuracy(te_confusion))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- You may want to try other ideas like: learning rate, hidden layer sizes, optimizers, other pretrained embeddings for english etc. to improve the performance of the model." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Part of Speech Tagging for Norwegian\n", "Make sure your code runs on the Norwegian dataset as well. You may want to use the pretrained embeddings for norwegian to improve the performance of the model. However, there is no deliverable or test for this part.\n", "- Run the `BiLSTM` model on the norwegian dataset using the `BiLSTM` tagger. " ] }, { "cell_type": "code", "execution_count": 106, "metadata": { "collapsed": true }, "outputs": [], "source": [ "#recalculating all_tags and tag_to_ix for the norwegian language\n", "if START_TAG in all_tags_nr:\n", " all_tags_nr.remove(START_TAG)\n", "if END_TAG in all_tags_nr:\n", " all_tags_nr.remove(END_TAG)\n", "\n", "tag_to_ix_nr={}\n", "for tag in all_tags_nr:\n", " if tag not in tag_to_ix_nr:\n", " tag_to_ix_nr[tag] = len(tag_to_ix_nr)" ] }, { "cell_type": "code", "execution_count": 107, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "words in the vocabulary: 7600\n", "7599\n" ] } ], "source": [ "#recalculating the vocab for the norwegian language: obtains the most common 7600 words from the file\n", "vocab_nr, word_to_ix_nr = most_common.get_word_to_ix(NR_TRAIN_FILE, 7600)\n", "print ('words in the vocabulary: ',len(word_to_ix_nr))\n", "print (word_to_ix_nr[UNK])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- Loading training data for norwegian:" ] }, { "cell_type": "code", "execution_count": 108, "metadata": { "collapsed": true }, "outputs": [], "source": [ "reload(preproc);\n", "X_tr_nr, Y_tr_nr = preproc.load_data(NR_TRAIN_FILE)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- Loading dev data for norwegian:" ] }, { "cell_type": "code", "execution_count": 109, "metadata": { "collapsed": true }, "outputs": [], "source": [ "X_dv_nr, Y_dv_nr = preproc.load_data(NR_DEV_FILE)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- Loading test data for norwegian:" ] }, { "cell_type": "code", "execution_count": 110, "metadata": { "collapsed": true }, "outputs": [], "source": [ "X_te_nr, Y_te_nr = preproc.load_data(NR_TEST_FILE_HIDDEN)" ] }, { "cell_type": "code", "execution_count": 111, "metadata": { "collapsed": true }, "outputs": [], "source": [ "#setting up the random seed\n", "torch.manual_seed(765);\n", "\n", "# initializing your model\n", "embedding_dim=30\n", "hidden_dim=30\n", "model = bilstm.BiLSTM(len(word_to_ix_nr),tag_to_ix_nr,embedding_dim, hidden_dim)" ] }, { "cell_type": "code", "execution_count": 112, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Epoch 1: Dev Accuracy: 0.6328169822145726\n", "Epoch 3: Dev Accuracy: 0.7308089500860585\n", "Epoch 5: Dev Accuracy: 0.7577739529546759\n", "Epoch 7: Dev Accuracy: 0.7927710843373494\n", "Epoch 9: Dev Accuracy: 0.8055077452667814\n" ] } ], "source": [ "# training your model for norwegian data\n", "loss = torch.nn.CrossEntropyLoss()\n", "model, losses, accuracies = bilstm.train_model(loss, model, X_tr_nr,Y_tr_nr, word_to_ix_nr, tag_to_ix_nr, \n", " X_dv_nr, Y_dv_nr, num_its=10, status_frequency=2, \n", " optim_args = {'lr':0.2,'momentum':0}, param_file = 'best.params')" ] }, { "cell_type": "code", "execution_count": 113, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAtoAAACdCAYAAACKLN/OAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xl4leXV9/3vyjyTmYSQEOYZBAOodcIJtFK11TrVtlqr\n7aOda2vbu8PT6fF+1ba22lq1amtRa1WsWAWH1lkRBJkRkCEQCEPCmAAhyXr/2JsYMEgg2bmS7N/n\nOHIk13Duva4DOLNY+xzM3RERERERkfYVE3QAIiIiIiLdkRJtEREREZEIUKItIiIiIhIBSrRFRERE\nRCJAibaIiIiISAQo0RYRERERiQAl2iIiIiIiEaBEW0REREQkAiKWaJvZ/Wa22cwWHeb66Wa2w8ze\nC3/9pNm1yWb2vpmtNLObIxWjiIiIiEikWKR2hjSzU4HdwN/cfUQL108Hvuvu5x9yPhZYDpwNrAdm\nA5e7+5IjvWdubq6Xlpa2PXgRkQ727rvvbnX3vKDj6Ejqs0Wkq2ptnx0XqQDc/VUzKz2GpuOBle6+\nCsDMHgUuAI6YaJeWljJnzpxjeEsRkWCZ2dqgY+ho6rNFpKtqbZ8d9Bjtk8xsgZk9Z2bDw+eKgHXN\n7lkfPtciM7vOzOaY2ZwtW7ZEMlYRERERkVYLMtGeC5S4+yjgD8BTx/Ii7n6Pu5e5e1le3tF/6hqp\noTMiIiIiEt0iNnTkSNx9Z7OfnzWzP5pZLlABFDe7tXf4XLubtaqKXz+3jL9dPZ4eKfGReAsRERER\n6QT21TewrrqWD7bUsGpLDau37mb11hoe+tIEkuJjI/KegSXaZlYAbHJ3N7PxhKrrVcB2YKCZ9SWU\nYF8GXBGJGFIT41hcsYOfTV/Mby89LhJvISIiIiIdxN3ZtHMfq7bs5oOtNazeUsOqcEK9rrqWxmYD\nGXLTEumXm8rOPfu7XqJtZo8ApwO5ZrYe+CkQD+DudwMXA181s3pgD3CZh8Zx1JvZjcBMIBa4390X\nRyLGEUU9uGHiAO54aQWTRxQwaXhBJN5GRERERNrRrr37Wb01VJletbWGVVtCyfTqrTXU1jU03Zcc\nH0vf3FRGFPXggtG96JuXSr/cNEpzU+mRHPnRDJFcdeTyI1y/E7jzMNeeBZ6NRFyHuvGMAbywZBM/\nmraQcaXZZKcmdMTbioiIiMjH2N/QyLrq2vAwj1Bl+kBivWXXvqb7Ygx6Z6XQNzeV8X2z6ZebSr+8\nNPrmplKQkURMjAX2DIENHeks4mNj+M2lo5nyh9f58VOLuOvKsUGHJCIiItLtNTQ6tXX17N5XT3lV\n7UGV6VVbaiivrqW+2ViP7NQE+uamcvqgvKZEun9eKiU5KSTGRWboR1tFfaINMKQgg2+eNYhbZ77P\n5PkbmDK6V9AhiYh0ODObDNxBaNjefe5+yyHXewB/B0oI/f64zd0faE1bEel63J26hkb21DVQG/4K\n/VxP7f6GpvN76uqbru/d3+ze/fUfabenroHa8D119Y0fec/EuBj65qYyuCCdc0cW0Dc3jX55qfTL\nTSUzpeuNOlCiHXb9qf14fskmfvyvRUzol01+elLQIYmIdJjwrrx30WxXXjN7+pBdeW8Alrj7FDPL\nA943s6lAQyvaikgnUlffyMrNu1m6cSdLN+5kWeUuqmrqmpLmAwlxQ+PRLYOcGBdDSkIsKQlxJCfE\nkpIQS3J8LLlpCaQkpHx4LiGWlPi40L2JsRSHh34UZSYHOtSjvSnRDouLjeH2S0Zx3u9f54dPLuLe\nzx+PWff5gxYROYLW7MrrQLqFOsc0oBqoBya0oq2IBGTr7n2hZHrjLpZu3MmSjTv5YMtu9jeEkujE\nuBgG9UynKDM5nCTHNiXEKQlxJMc3Pxd38PX4OJISYprui+1GSXJ7UKLdzID8dG46ZzC/enYpT86t\n4DPH9w46JBGRjtLSrrwTDrnnTuBpYAOQDlzq7o1m1pq2IhJh9Q2NrNpa05RMLw0n1s0nDvbMSGRo\nYQYTh+QztDCDYYXplOakEhcb9Gbh3ZMS7UNcc3JfZi6u5GfTF3PSgBwKeyQHHZKISGcxCXgPOAPo\nD7xgZq8dzQuY2XXAdQAlJSXtHqBItNheW3dQMr2scifLN+1uGvccH2sMyE/nlIG5DCvMYGj4S6ur\ndSwl2oeIjTFuu2Q0597xGt9/YiF/vXqchpCISJdhZrdzbPsPtGZX3quBW8J7Hqw0s9XAkFa2BcDd\n7wHuASgrKzu6wZ8iUaih0VlTVdM0lvpAYr1xx96me3LTEhhamMEXTyplaGE6Qwsz6J+XRryq1IFT\not2C0txUbj53CD99ejGPzl7H5eNVdRGRLmMpcI+ZxQEPAI+4+45WtJvNkXflLQfOBF4zs57AYGAV\nHbijr0h3VltXz6KKnc2S6p28v2kXe/eHqtSxMUb/vNBa0UObqtTpWsChE1OifRhXndCHGYsq+eUz\nSzh5QC7F2SlBhyQickTufh9wn5kNJlSBXmBmbwD3uvt/P6Zdi7vymtlXwtfvBn4BPGhmCwEDvu/u\nWwE6akdfke5k7/4G5q7dxlurqnjzgyrmr9vetG50Zko8QwsyuGJ8n6Yq9YD8tIhtFS6RYaFPALuH\nsrIynzNnTru93rrqWib/7lVG9c5k6rUTutVyMyLSuZjZu+5e1k6vFQucTyjRLgYeA04Gatz9svZ4\nj/bQ3n22SGe3r76B98q389aqKt76oIp55dupa2gkxmBk70xO7JfDuNIshvXKoCAjSUNXO7HW9tmq\naH+M4uwU/uf8YfzgyYU89PZavnBSadAhiYh8LDP7LaEk+z/Ar939nfCl/zWz94OLTCT67G9oZMH6\nHbwdTqznrK1m7/5GzGBYYQafP7EPJw3IYVxpNulJ8UGHKxGgRPsILhtXzIxFldzy3DJOG5RHaW5q\n0CGJiHycBcD/uHtNC9fGd3QwItGkodFZvGEHb30QGgoyZ001NXUNAAwpSOeycSWc2D+HCX2zu+Qu\nh3L0lGgfgZlxy2dGcs5vX+W7/5zPP64/UYuxi0hntp1mfbuZZQKnu/tTrZwUKSKt1NjoLK3cyVsf\nVPH2qipmra5m1956APrnpXLR2CJO7JfLCf2yyUlLDDhaCULEEm0zu5/Qx5eb3X1EC9evBL5PaELN\nLuCr7j4/fG1N+FwDUN9e4xaPVWGPZH42ZTjf+ed87n99NV8+tV+Q4YiIfJyfuvu0Awfuvt3Mfgo8\nFWBMIt2Cu7Ni827e+iA0FOTt1VVsr90PQJ+cFM4fVcgJ/XI4sV8O+RlaCUQiW9F+kNAuYn87zPXV\nwGnuvs3MziW0rmrzncQmHpjN3hl8emwRzy2q5Nbn32fikDwG5KcHHZKISEtaWjhXn16KHAN3Z/XW\nGt78oIq3VlUxa1UVW3fXAVCUmcxZQ3tyYr8cTuyfQ69MbXAnHxWxztfdXzWz0o+5/mazw7cJbXDQ\naZkZv/70CM757at857H5PPHVk7RdqYh0RnPM7DfAXeHjG4B3A4xHpEup2r2Pl5Zu5s0PtvLWqio2\n7QxtX94zI5GTB+RyUv9cTuyfo2V/pVU6S5XjS8BzzY4deNHMGoA/h3cSa1FHbuebn57ELy4Ywdce\nmcefX13FDRMHRPT9RESOwdeAHwP/CB+/QCjZFpHD2FffwH+WbuaJuRW8/P5m6hud3LSE0DCQ/qGh\nIH1zU7Xcnhy1wBNtM5tIKNE+udnpk929wszygRfMbJm7v9pS+47eznfK6F7MWFTJ715czhlD8hla\nmBHptxQRabXwaiM3Bx2HSGfn7swt38YTcyt4Zv4Gdu6tJz89kS+d3JdPHdeLYYUZSqylzQJNtM1s\nFHAfcK67Vx047+4V4e+bzWwaoSWpWky0g/CLC0cwa3UV33lsPk/d8AkS4jSEREQ6BzPLA74HDAea\nZmO5+xmBBSXSiZRX1TJtXgVPzlvP2qpakuNjmTS8J58e25tPDMjVymLSrgJLtM2sBHgSuMrdlzc7\nnwrEuPuu8M/nAD8PKMwWZacm8KuLRnL9Q+9y539X8u2zBwUdkojIAVMJDRs5H/gK8AVgS6ARiQRs\nx579PLtwI0/OXc/sNdswgxP75fC1MwYyeUQBaYmBf8Av3VQkl/d7BDgdyDWz9cBPgXgAd78b+AmQ\nA/wx/NHMgWX8egLTwufigIfdfUak4jxWk4YXcNGYIu7670rOHtqTkb17BB2SiAhAjrv/xcy+4e6v\nAK+Y2eyggxLpaPsbGnltxRaemFvBC0s2UVffSP+8VG6aNJgLxxRRpFVCpANEctWRy49w/Vrg2hbO\nrwJGRyqu9vSzKcN584OtfOef7zH9ayeTGBcbdEgiIvvD3zea2SeBDUB2gPGIdBh3Z/GGnTwxdz3T\n529g6+46slLiuWJ8CZ8eW8TIoh4ady0dSp+VtEGPlHhu+cworn5gNr99YQU3nzsk6JBERH5pZj2A\n7wB/ADKAbwUbkkhkVe7Yy1PvVfDk3PUs37SbhNgYzhyaz6fH9ua0QXmaSyWBUaLdRhMH53NpWTH3\nvPoB5wzvydiSrKBDEpEoZWaxwEB3fwbYAUwMOCSRiKnZV8/MxZU8ObeCNz7Yijsc3yeLX100gvNH\n9qJHSnzQIYoo0W4P/3P+UF5fuZXvPjaff3/9FJITNIRERDqeuzeY2eXAb4OORSQSGhqdt1dV8cTc\n9cxYVEltXQPF2cl87YyBXDSmiL65qUGHKHIQJdrtID0pnv/v4lFced8sbp35Pj+ZMizokEQker1h\nZncSWnmk5sBJd58bXEgibbNi0y6emFvBv96rYOOOvaQnxnHBcb349NjelPXJ0rhr6bSUaLeTTwzI\n5aoT+vDAm6uZNLwnE/rlBB2SiESn48Lfmy+L6oDW0ZYuZevufUyfv4En51awsGIHsTHGaYPy+NEn\nh3LW0J4kxevTY+n8lGi3o5vPHcIry7fw3cfnM+Mbp5KqdTlFpIO5u8ZlS5fQ2Ohs2rWX8qpa1lbX\nsq66lrVVtZSHf66qqQNgRFEGPzl/GFNG9yIvPTHgqEWOjjLBdpSaGMetF4/isnvf5v89t5RfXjgy\n6JBEJMqY2U9aOu/unWrjL4kOe+oaWLft4AR6bVVN6Odte6irb2y6N8agV2YyfXJSOGd4T/rkpDJx\ncD6DC9IDfAKRtlGi3c4m9Mvh6pP6cv8bq5k8vJCTB+YGHZKIRJeaZj8nEdohcmlrGprZZOAOIBa4\nz91vOeT6TcCV4cM4YCiQ5+7VZrYG2AU08OEGZNLNuTtbdu87qBpdfuB7dS2bd+076P60xDhKslMY\nmJ/OWUN7UpydQkl2Cn1yUuiVmUx8rJbhk+5FiXYEfG/yYF5+fzPff2IBM755CulJWmJIRDqGu9/e\n/NjMbgNmHqldeGnAu4CzgfXAbDN72t2XNHvtW4Fbw/dPAb7l7tXNXmaiu29t+1NIZ7KvvoH12/Yc\nlESvrQpVp8ura9mzv6HpXjMozEiiODuF0wbl0ScnheLsFPrkpFKSnUJWSrwmLkpUUaIdAUnxsdz2\n2dFc/Kc3+eUzS/nfi0cFHZKIRK8UoHcr7hsPrAzvzouZPQpcACw5zP2XA4+0S4TS6azcvJvp8zfw\n7MKNrNyyG/cPryXHx1KSHUqgTx6YS0m4Kl2Sk0JRZrImKYo0o0Q7QsaWZHHdqf25+5UPmDyigIlD\n8oMOSUSigJktJLTKCISGgORx8Aokh1MErGt2vB6YcJj3SAEmAzc2O+3Ai2bWAPzZ3e85TNvrgOsA\nSkpKWhGWdJR11bVMX7CB6fM3snTjTsxgQt9szhs5kD45KU3V6by0RFWlRVpJiXYEfevsgfxn2SZu\nfnIBz3/zNO1SJSId4fxmP9cDm9y9vp3fYwrwxiHDRk529wozywdeMLNl7v7qoQ3DCfg9AGVlZX7o\ndelYm3bu5d8LNjJ9wQbmlW8HYExJJj85fxifHFVIz4ykgCMU6doilmib2f2EOvzN7j6ihetGaNLN\neUAt8MUDGyocaUJOV5EYF8vtlxzHhX98g59NX8xvLz3uyI1ERNqmEFjs7rsAzCzdzIa5+6wjtKsA\nipsd9w6fa8llHDJsxN0rwt83m9k0QkNRPpJoS/Cqa+p4btFGps/fwKzV1bjDsMIMvj95COePKqQ4\nOyXoEEW6jUhWtB8E7gT+dpjr5wIDw18TgD8BE1ozIacrGdm7BzdMHMDvX1rB5BEFTBpeEHRIItK9\n/QkY2+y4poVzLZkNDDSzvoQS7MuAKw69ycx6AKcBn2t2LhWIcfdd4Z/PoXXDVaSD7Ny7nxcWb+Lp\n+Rt4feVWGhqdfnmpfP2MgUwZ3YsB+WlBhyjSLUUs0Xb3V82s9GNuuQD4m7s78LaZZZpZIVDK0U3I\n6fRunDiAF5ds4kfTFjKuNJvs1ISgQxKR7svC/SoA7t5oZkfs69293sxuJLRCSSxwv7svNrOvhK/f\nHb71IuB5d2++jGBPYFp43G4c8LC7z2ifx5FjtaeugZeWbWL6/A389/0t1NU3UpSZzJdP6ceU0YUM\nK8zQWGuRCAtyjHZLE2+KDnO+xQk5XUVCXAy/uXQ0U/7wOj9+ahF3XXmkwpKIyDFbZWZfJ1TFBvg/\nwKrWNHT3Z4FnDzl39yHHDxL6xLL5uVXA6GMLV9rTvvoGXl2+lenzN/Di0k3U1jWQn57IlRNKmDK6\nF2OKM5Vci3SgLj8ZsqvMYB9SkME3zxrErTPfZ/L8DUwZ3SvokESke/oK8HvgfwitBPIS4T5Suqf6\nhkbeWlXF9PkbmLGokp1768lKiefCMUVMGdWL8X2ziY1Rci0ShCAT7cNNvIk/zPkWdaUZ7Nef2o/n\nl2zix/9axIR+2eSnaza3iLQvd99MaHy1dGONjc6ctdua1rquqqkjLTGOc4b35FOje/GJAbnaZVGk\nE2hVom1m3wAeILS97n3AGOBmd3++De/9NHBjeAz2BGCHu280sy20YkJOVxQXG8Ptl4zivN+/zg+f\nXMS9nz9eH+GJSLsys78C33D37eHjLOB2d78m2MikrdydBet3MH3+Bp5ZsJHKnXtJio/hzKE9mTKq\nF6cPztNmMSKdTGsr2te4+x1mNgnIAq4CHgIOm2ib2SPA6UCuma0HfkqoWn1gzN+zhJb2W0loeb+r\nw9danJBz9I/WOQ3IT+emcwbzq2eX8vi767mkrPjIjUREWm/UgSQbwN23mdmYIAOStqnYvodHZpUz\nfcEG1lbVEh9rnDYonx+cN4SzhvYkNbHLjwIV6bZa+6/zQNn1POCh8Ez0jy3FuvvlR7juwA2HufaR\nCTndyTUn9+X5JZXc9PgCnpxbwRUTSpg0vICEOH3MJyJtFmNmWe6+DcDMsukG83GikbvzxNwKfvb0\nYvbsb+Ck/jncMHEAk4YVaAM0kS6itZ3vu2b2PNAX+IGZpQONkQure4uNMe77wjj+/vZaHnmnnK89\nMo/ctAQuKSvm8nEllORoswAROWa3A2+Z2T8JFUkuBn4VbEhytLbV1PHDaQt5blEl4/tmc/slo7WR\njEgXZM2WWz38TWYxwHHAKnffHq6Q9Hb3BZEO8GiUlZX5nDlzgg7jqDQ2Oq+u2MLUWeW8tHQTjQ6n\nDsrjygklnDkknzhNZhGJCmb2rruXtdNrDQcmhg//01k3/OqKfXZHeGX5Fm7653y21dbxnXMG8+VT\n+mnVEJFOprV9dmsr2icC77l7jZl9jtAOY3e0JUAJiYkxTh+cz+mD89m4Yw//mL2OR99Zx/UPvUvP\njEQuHVfCZeOK6ZWZHHSoItJFhIf3bQGSAMysxN3LAw5LjmDv/gZueW4ZD765hkE903jg6nEM79Uj\n6LBEpA1aW9FeQGgzglGENiq4D/isu58W0eiOUnepjtQ3NPLf97cwddZaXlm+BQPOGJLPlRP6cOqg\nPFU2RLqh9qpom9mnCA0f6QVsBvoAS919eFtfu711lz67PSyq2ME3//EeKzfv5upPlPL9yUO0gohI\nJ9beFe16d3czuwC4093/YmZfaluIcjhxsTGcPawnZw/rybrqWh6dXc4/Zq/nxaWzKcpM5vLxxXy2\nrJj8DK3DLSIf8QvgBOBFdx9jZhOBzwUckxxGQ6Nz9ysf8LsXl5OdmsBDXxrPKQPzgg5LRNpJaxPt\nXWb2A0LL+p0SHrOtKc8doDg7hZsmDeEbZw7ixaWbmDprLbc9v5zfvbiCs4f15MoJfTipfw4xqnKL\nSMh+d68ysxgzi3H3/5rZ74IOSj5qXXUt337sPWav2cYnRxbyq4tGkJmSEHRYItKOWptoX0po05hr\n3L3SzEqAWyMXlhwqIS6G80YWct7IQlZvreGRd8r555x1PLeokj45KVwxvoSLj+9NTlpi0KGKSLC2\nm1ka8Cow1cw2AzUBxyTNNF+2z4DffHY0F40p0gZmIt1Qq8ZoA5hZT2Bc+PCd8Da/nUq0jffbu7+B\nmYsrmfp2Oe+sqSYhNobJIwq4ckIJ4/tmq9MW6ULacYx2KrAHiAGuBHoAU929qq2v3d6irc+G0LJ9\nP3pqIc8uDC3b95vPjqZ3lpbtE+lq2nWMtpl9llAF+2VC67L+wcxucvfH2xSltElSfCwXHFfEBccV\nsXzTLh6eVc4Tc9fz9PwNDMhP44rxJXxmbG9tbCASRdz9QPW6EfhrkLHIwV5dvoXvhpft+/7kIVx3\nqpbtE+nuWrvqyHzg7ANVbDPLIzTRZnSE4zsq0VgdOdSeugaeWbCBqbPKeW/ddhLjYjh/VC+uPKGE\nMcWZqnKLdFLtuY52VxEtfXbzZfsG5qfx20uPY0SRlu0T6crae9WRmEOGilQR+lhSOpnkhFguKSvm\nkrJiFm/YwcOzynlqXgVPzF3P0MIMrphQwoXH9SI9SVVuEZFIW1Sxg2/94z1WbN7NF08q5eZztWyf\nSDRpbbI8w8xmmtkXzeyLwL+BZyMXlrSH4b168KuLRjLrR2fx64tGEmPw46cWMeHXL/Htx97j+cWV\n7N3fEHSYItKOzOwbrTknkdXQ6Pzp5Q+46I9vsHPvfh760nh+9qnhSrJFoszRTIb8DPCJ8OFr7j4t\nYlEdo2j5GPJYuTvz1+/gkVnlzFhcyY49+0mOj+X0wXlMHlHAxCH5ZKjSLRKIdpwMOdfdxx5ybp67\nj2nra7e37tpnr6uu5TuPzeedNdWcN7KAX104kqxULdsn0p2099AR3P0J4ImjDGIyoa3aY4H73P2W\nQ67fRGhW/IFYhgJ57l5tZmuAXUADoQ1zomrsYiSYGccVZ3JccSa/vGgEs1ZVM3NxJTMXV/Lcokri\nY42T+ucyeUQBZw/rSa6WChTpMszsckLLsPY1s6ebXUoHqlv5Gm3psz+2bTRwd6bNq+An/1oMwO2X\njObTY7Vsn0g0+9iKtpntAlq6wQB394yPaRsLLAfOBtYDs4HL3X3JYe6fAnzL3c8IH68Bytx9a+se\npftWRyKtsdGZt247z4cT7vLqWmIMykqzmTS8gEnDe2r5KZEIa2tF28z6AH2B/wfc3OzSLmCBu9cf\nof0x99lH2/aA7tRnb6+t40fTFvHvhRsZV5rFbz57HMXZ6jdFuqt2qWi7e3obYhgPrHT3VeGAHgUu\nAA7X8V4OPNKG95NjFBNjHN8ni+P7ZHHzuUNYVrmLGYtCle5fPLOEXzyzhJFFPZg0vCeTRxQwIL8t\nfy1EJBLcfS2wFjgxnHQPdPcXzSwZSCaUcH+ctvTZR9u2W3ltRWjZvuqaOr43eTDXn9pfy/aJCHAU\nQ0eOQRGwrtnxemBCSzeaWQowGbix2WkHXjSzBuDP7n5PpAKVD5kZQwszGFqYwbfOHsSarTXMXFzJ\njMWV3Pb8cm57fjn981KZPKKAScMLGFnUQx+LinQiZvZl4DogG+gP9AbuBs48QtO29NlH0/a6cHyU\nlJQcIaTObe/+Bv53xjIeeGMNA/LT+MsXxmnZPhE5SCQT7aMxBXjD3ZuPIzzZ3SvMLB94wcyWufur\nhzbsTp12Z1Sam8r1p/Xn+tP6U7ljLy8sCSXdd7+yirv++wFFmcmcM7wnk4YXMK40W1UckeDdQKjC\nPAvA3VeE+9H21FKf3Srhosk9EBo60s5xdZjFG3bwzUe1bJ+IfLxIJtoVQHGz497hcy25jEOGjbh7\nRfj7ZjObRugXx0cS7e7SaXcFBT2SuOrEUq46sZRtNXW8tGwzMxZVMnVWOQ+8sYac1ATOHtaTSSMK\nOKl/Dolx+qUjEoB97l534JMmM4uj5bk2h2pLn300bbu0hkbn3tdWcfvz75OVksBfrxnPaYPygg5L\nRDqpSCbas4GBZtaXUId7GaEZ8Qcxsx7AacDnmp1LJbRJzq7wz+cAP49grHKUslITuPj43lx8fG9q\n9tXz8vtbmLm4kmcWbOTR2etIS4zjjCH5TB5RwGmD8khN7Cwfnoh0e6+Y2Q+BZDM7G/g/wPRWtDvm\nPru1bbu69dtq+fZj83lndTXnjijg1xdp2T4R+XgRy37cvd7MbgRmElru6X53X2xmXwlfvzt860XA\n8+5e06x5T2BauCITBzzs7jMiFau0TWpiHJ8cVcgnRxWyr76BN1dWMXNxJc8v2cTT8zeQGBfDKQND\na3WfNTSfzBT9YhKJoJuBLwELgesJbS5235EataXPPlzbdnymwL2+Yitf/fu7OHDbJaP5jJbtE5FW\naPWGNV1Bd1oqqjuob2hkztptzFhUyfOLK9mwYy+xMcaIXhmUlWYzrjSL4/tkk5eu9bpF2mvDmkNe\nMxvo7e4L2vN120tX6bNnrariCw+8Q2lOKvd+vkzL9olI+29YI3K04mJjOKFfDif0y+GnU4axsGIH\nLyzZxDurq/n722v5y+urAeibm0pZnyzGlWZTVppF39xUVYpEjpGZvQx8ilD//i6w2czedPdvBRpY\nFzWvfBvXPDibosxk/n7tBG3kJSJHRYm2dAgzY1TvTEb1zgSgrr6RRRt2MGdNNbPXbOPFpZv457vr\nAchJTaCsNIuyPqHEe3ivHiTExQQZvkhX0sPdd5rZtcDf3P2nZtYpK9qd3aKKHXzh/nfITU/k4S+f\noCRbRI6aEm0JREJcDGNLshhbksV1p4a2Lv5gS01T4j1nbTUzF28CICk+huOKM8MV72zGlmSSnhQf\n8BOIdFo+LwjOAAASS0lEQVRxZlYIfBb4UdDBdFXLN+3iqr/MIj0pnqnXTqBnRlLQIYlIF6REWzoF\nM2NAfhoD8tO4bHxoPfTNO/cyZ+02Zq+pZs6abdz135U0OsQYDCnIYFxpVnisdzYFPfRLUCTs54Qm\nJb7u7rPNrB+wIuCYupRVW3Zzxb2ziI+NYeq1E+idpTHZInJslGhLp5WfkcR5Iws5b2QhALv31fNe\n+fZQ4r22msfmrOevb60FoHdWctMY73Gl2QzISyNGm+dIFHL3fwL/bHa8CvhMcBF1Leuqa7nyvlm4\nOw9fdwKlualBhyQiXZgSbeky0hLjOHlgLicPzAVgf0MjSzfuDA01WVPNayu2Mm1eaI+MHsnxlPXJ\n4vhw4j2yqId2bRORj7Vxxx6uuO9tausaeOTLJzAgPz3okESki1OiLV1WfGxM0wTLL53cF3dnbVVt\n01CT2WureWnZZgASYmMYUZTB8F49GFqYwZDCdIYUpJOSoH8CIgKbd+3lyntnsb1mP1O/PIFhvTKC\nDklEugFlGdJtmBmluamU5qZySVloN+iq3fuYszZU8Z5Xvp1p8yp46O214fuhT3YKQwszQsl3QTpD\nCzPonZWs5QWlyzKzWHdvCDqOrqS6po7P3TeLyp17eehL45tWRxIRaSsl2tKt5aQlMml4AZOGFwDQ\n2OhUbN/Dko07WbpxJ8s27mLJxp08t6iyqU16Yly44p0RTsLTGazqt3Qdq81sBvAP4D/enXYli4Ad\ne/Zz1V9msbaqlgeuHsfxfbKDDklEuhFlDhJVYmKM4uwUirNTmpJvgJp99Syr3MWyyg8T8EOr36U5\nqU1V7wMVcFW/pRMaApwP3AD8xcyeAR5199eDDavz2b2vni8+8A7LN+3i3s+XcVL/3KBDEpFuRom2\nCJCaGMfxfbI4vk9W07mjqX6HEm9VvyV47l4LPAY8ZmZZwB3AK4BmAzezp66Bax6czYL1O/jjlWM5\nfXB+0CGJSDekbEDkMI6m+v3k3Ap27zu4+j202fATVb+lI5nZacClwGRgDqHNayRs7/4GrntoDnPW\nVHPHZWMO+vctItKelGiLHKXWVr8Xb9jJswtV/ZaOZWZrgHmEqto3uXtNsBF1LnX1jdwwdS6vrdjK\nbZeMZsroXkGHJCLdWER/w5vZZEIfW8YC97n7LYdcPx34F7A6fOpJd/95a9qKdCZtqX4fWPnkQPKt\nlU+kjUa5+86gg+iM6hsa+eY/5vHSss388sIRXHx876BDEpFuLmKJtpnFAncBZwPrgdlm9rS7Lznk\n1tfc/fxjbCvSqbVU/XZ31m/bw9KNO1m6MZSEL6vcxYzFlRxYHyItMY4hBekHVcCHFKSTmqjqtxxR\ngZlNA3q6+wgzGwV8yt1/GXRgQWpodG56fAHPLqzkfz45lM+d0CfokEQkCkTyt/Z4YGV4+1/M7FHg\nAqA1yXJb2op0amYfVr/POaT6vXzTrg+T7427+Ne8Dfz97fKme/rkpDC0IKNp+cFh4eq3tpuXZu4F\nbgL+DODuC8zsYSBqE21350fTFjJtXgU3TRrMtaf0CzokEYkSkUy0i4B1zY7XAxNauO8kM1sAVADf\ndffFR9FWpNtITYxjTEkWY0oOrn5XbN8TSr43hirfSzfuZOaSD6vfqQmxDAlPuBxSmMGwwnQGF2SQ\npup3tEpx93cOGXpUH1QwQXN3/u/0JTw6ex03ThzADRMHBB2SiESRoH8TzwVK3H23mZ0HPAUMPJoX\nMLPrgOsASkpK2j9CkQCZGb2zUuidlcLZw3o2nd9T18D7m0LJ99KNO1lauYvp8zcwddaH1e+S7BSG\nFIQmXA7IT2NAfhr989JIitcqb93cVjPrDziAmV0MbAw2pGC4O7fMWMaDb67h2pP78p1zBgUdkohE\nmUgm2hVAcbPj3uFzTZpP2HH3Z83sj2aW25q2zdrdA9wDUFZWph3QJCokJ8RyXHEmxxV/uFW0u7Nh\nx96Dku+lG3fy0rLNNDSG/mmYQXFWCgPDifeA/DQG9kynf14q6UnxQT2OtK8bCPWJQ8ysgtBk88+1\npmFrJqGHJ7H/DogHtrr7aeHza4BdQANQ7+5lbX6SNrrjpRX8+ZVVfO6EEn70yaGaYCwiHS6SifZs\nYKCZ9SWUJF8GXNH8BjMrADa5u5vZeCAGqAK2H6mtiBzMzCjKTKYoM5kzh35Y/d5X38CarbWs3Lyb\nFZt3sXLzblZu3s1rK7ZS19DYdF9hj6Sm5HtAfhoD89MZmJ9GVmpCEI8jxyg8t+UsM0sFYtx9V2va\ntWYSupllAn8EJrt7uZkdusvLRHff2i4P0kZ3v/IBv3txBRcf35uff2qEkmwRCUTEEm13rzezG4GZ\nhKoj97v7YjP7Svj63cDFwFfNrB7YA1zm7g602DZSsYp0Z4lxsQwODyGBwqbz9Q2NrNu2hxWbdrFy\ny25WbtrNyi27+cfsddTWNTTdl5Oa0Cz5TmNAfjoDe6aRn56o5KUTMbNvH+Y8AO7+myO8RGsmoV9B\naBnW8vBrbm5j2BHx4BurueW5ZUwZ3Yv//cwoTRYWkcBEdIy2uz8LPHvIubub/XwncGdr24pI+4mL\njaFvbip9c1M5p9n5xkZnw449TZXvUCV8N9Pnb2Dn3g/n1KUnxTVLvkMV8AH5aRRlahWUgKSHvw8G\nxgFPh4+nAO+0on1rJqEPAuLN7OXw+93h7n8LX3PgRTNrAP4cHtb3EZGeV/PoO+X8bPoSzhnWk998\ndjSx+rsoIgEKejKkiHQyMTEfTsA8ffCHIwPcnS279zVVvldsCiXh/1m2hcfmrG+6Lyk+hv55oeS7\nNCeVkuwUSnJS6JOdQp6q4BHj7v8XwMxeBcYeGDJiZj8D/t1ObxMHHA+cCSQDb5nZ2+6+HDjZ3SvC\nw0leMLNl7v5qC3FGbF7NtHnr+cG0hZw2KI8/XDGG+NiY9nx5EZGjpkRbRFrFzMhPTyI/PYmTBuQe\ndG17bV1T5fvA9zlrtvH0/A1NyxBCKAkvzkppSr5Lsj/8Ks5O0Yoo7aMnUNfsuC587khaMwl9PVAV\n3ta9JpzUjwaWu3sFhIaThDfMGQ98JNGOlGcXbuQ7j83nxH45/Pmq40mM098lEQmeEm0RabPMlATK\nSrMpK80+6Py++gYqtu2hvLqWddW1rK2qpbw69PXWqqqDxoID9MxIbEq6+2SnUpKT3HScl6ZqeCv9\nDXgnnOwCXAg82Ip2R5zADvwLuNPM4oAEQkNLftt84mX453OAn7f5SVrppaWb+Poj8xhbksW9ny/T\nf9hEpNNQoi0iEZMYF0u/vDT65aV95Jq7U1VT15SEl1fVsvZAEv5BFdPmVRxUDU+Oj21KukNV8GT6\n5KRSnJ1C76xkJVdh7v4rM3sOOCV86mp3n9eKdkecwO7uS81sBrAAaCS0BOAiM+sHTAv/RygOeNjd\nZ7T/033Uayu28NW/z2VYrwzuv3ocqdqoSUQ6EXPvPktPl5WV+Zw5c4IOQ0Tawd79DVRs30N5syp4\neTghL6+uZc/+g6vhBRlJTcNRirNSKMpKpldmEkWZyRT0SOr0QwnM7N3OsPZ0R2prnz1rVRVfeOAd\nSnNSefS6E8hM0VKUItIxWttn67/+ItIpJcXH0j8vtJvlodydrbvrPjIkZV11La+v2Erlzr0faZOX\nnkivzGSKMpPo1SOZXpnJ4eNQQp6dmqChKV3I3PJtXPPgbHpnpfD3aycoyRaRTkmJtoh0OWZGXnoi\neemJHN8n6yPX9+5voHLHXjZs30PF9j1s2B76ecOOPSyr3MV/lm1m7/7Gg9okxsWEk+5Q4n1wIp5M\nYY8kDU/pJBZV7OAL979DbnoiU6+dQG5aYtAhiYi0SIm2iHQ7SfGxlOamUpqb2uJ1d2db7f5mifiB\nr71UbN/Dy+9vYfOufR9pl5uWEErAe3yYkH+YnCeTm6aqeKS9X7mLq/4yi4ykeKZeO4GeGUlBhyQi\nclhKtEUk6pgZ2akJZKcmMKKoR4v37KtvYNOOfQcn4jv2ULF9Lyu37OaV5Vs+Mk48IS6GUUU9ePyr\nJ3XEY0Sd8qparrxvFvGxMTz85Qn0zkoJOiQRkY+lRFtEpAWJcbGhyZU5LSdz7s722v2HJOJ7iY9V\nRTtSctMTOKl/Dl8/cwB9clr+tEJEpDNRoi0icgzMjKzUBLI+piou7SslIY7fXz4m6DBERFpN+9OK\niIiIiESAEm0RERERkQhQoi0iIiIiEgHdamdIM9sCrD3KZrnA1giE09lF43NH4zNDdD53V3zmPu6e\nF3QQHekY+2zomn++bRWNzwzR+dzR+MzQ9Z67VX12t0q0j4WZzYm2bY8hOp87Gp8ZovO5o/GZo0k0\n/vlG4zNDdD53ND4zdN/n1tAREREREZEIUKItIiIiIhIBSrThnqADCEg0Pnc0PjNE53NH4zNHk2j8\n843GZ4bofO5ofGbops8d9WO0RUREREQiQRVtEREREZEIiOpE28wmm9n7ZrbSzG4OOp5IM7NiM/uv\nmS0xs8Vm9o2gY+pIZhZrZvPM7JmgY+kIZpZpZo+b2TIzW2pmJwYdU0cws2+F/34vMrNHzCwp6Jik\nfURbnw3R3W9HW58N0dlvd/c+O2oTbTOLBe4CzgWGAZeb2bBgo4q4euA77j4MOAG4IQqeublvAEuD\nDqID3QHMcPchwGii4NnNrAj4OlDm7iOAWOCyYKOS9hClfTZEd78dbX02RFm/HQ19dtQm2sB4YKW7\nr3L3OuBR4IKAY4ood9/o7nPDP+8i9A+4KNioOoaZ9QY+CdwXdCwdwcx6AKcCfwFw9zp33x5sVB0m\nDkg2szggBdgQcDzSPqKuz4bo7bejrc+GqO63u3WfHc2JdhGwrtnxeqKg8zrAzEqBMcCsYCPpML8D\nvgc0Bh1IB+kLbAEeCH/0ep+ZpQYdVKS5ewVwG1AObAR2uPvzwUYl7SSq+2yIun472vpsiMJ+Oxr6\n7GhOtKOWmaUBTwDfdPedQccTaWZ2PrDZ3d8NOpYOFAeMBf7k7mOAGqDbj2k1syxCVc6+QC8g1cw+\nF2xUIm0XTf12lPbZEIX9djT02dGcaFcAxc2Oe4fPdWtmFk+os57q7k8GHU8H+QTwKTNbQ+jj5jPM\n7O/BhhRx64H17n6g8vU4oQ68uzsLWO3uW9x9P/AkcFLAMUn7iMo+G6Ky347GPhuis9/u9n12NCfa\ns4GBZtbXzBIIDb5/OuCYIsrMjNDYr6Xu/pug4+ko7v4Dd+/t7qWE/pz/4+7d6n/Mh3L3SmCdmQ0O\nnzoTWBJgSB2lHDjBzFLCf9/PpJtPJooiUddnQ3T229HYZ0PU9tvdvs+OCzqAoLh7vZndCMwkNMv1\nfndfHHBYkfYJ4CpgoZm9Fz73Q3d/NsCYJHK+BkwNJyWrgKsDjifi3H2WmT0OzCW0WsM8uuluY9Em\nSvtsUL8dbaKq346GPls7Q4qIiIiIREA0Dx0REREREYkYJdoiIiIiIhGgRFtEREREJAKUaIuIiIiI\nRIASbRERERGRCFCiLd2Wmb0Z/l5qZle082v/sKX3EhGRY6M+W7ojLe8n3Z6ZnQ58193PP4o2ce5e\n/zHXd7t7WnvEJyIiH1KfLd2JKtrSbZnZ7vCPtwCnmNl7ZvYtM4s1s1vNbLaZLTCz68P3n25mr5nZ\n04R34zKzp8zsXTNbbGbXhc/dAiSHX29q8/eykFvNbJGZLTSzS5u99stm9riZLTOzqeFdsEREBPXZ\n0j1F7c6QElVupll1JNz57nD3cWaWCLxhZs+H7x0LjHD31eHja9y92sySgdlm9oS732xmN7r7cS28\n16eB44DRQG64zavha2OA4cAG4A1CO7693v6PKyLSpanPlm5DFW2JRucAnw9vZzwLyAEGhq+906zD\nBvi6mc0H3gaKm913OCcDj7h7g7tvAl4BxjV77fXu3gi8B5S2y9OIiHRv6rOly1JFW6KRAV9z95kH\nnQyNC6w55Pgs4ER3rzWzl4GkNrzvvmY/N6B/fyIiraE+W7osVbQlGuwC0psdzwS+ambxAGY2yMxS\nW2jXA9gW7rCHACc0u7b/QPtDvAZcGh5TmAecCrzTLk8hIhId1GdLt6H/nUk0WAA0hD9OfBC4g9BH\ngHPDk1u2ABe20G4G8BUzWwq8T+ijyAPuARaY2Vx3v7LZ+WnAicB8wIHvuXtluNMXEZEjU58t3YaW\n9xMRERERiQANHRERERERiQAl2iIiIiIiEaBEW0REREQkApRoi4iIiIhEgBJtEREREZEIUKItIiIi\nIhIBSrRFRERERCJAibaIiIiISAT8/716SqhopurxAAAAAElFTkSuQmCC\n", "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "reload(bilstm);\n", "bilstm.plot_results(losses, accuracies)" ] }, { "cell_type": "code", "execution_count": 114, "metadata": { "collapsed": true }, "outputs": [], "source": [ "reload(tagger_base);\n", "reload(bilstm);\n", "confusion = tagger_base.eval_model(model,'bilstm-dev-nr.preds', word_to_ix_nr, \n", " trainfile=NR_TRAIN_FILE, testfile=NR_DEV_FILE)" ] }, { "cell_type": "code", "execution_count": 115, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "0.8139988525530695\n" ] } ], "source": [ "print (scorer.accuracy(confusion))" ] }, { "cell_type": "code", "execution_count": 116, "metadata": { "collapsed": true }, "outputs": [], "source": [ "tagger_base.apply_model(model,'bilstm-te-nr.preds',word_to_ix_nr, all_tags_nr, \n", " trainfile=NR_TRAIN_FILE, testfile=NR_TEST_FILE_HIDDEN)" ] }, { "cell_type": "code", "execution_count": 117, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "0.8049471785622262\n" ] } ], "source": [ "# you don't have no_bokmaal-ud-test.conllu, so you can't run this\n", "te_confusion = scorer.get_confusion(NR_TEST_FILE,'bilstm-te-nr.preds')\n", "print (scorer.accuracy(te_confusion))" ] }, { "cell_type": "markdown", "metadata": { "collapsed": true }, "source": [ "# Using cuda\n", "If you want to run the code on a GPU: \n", "- make sure you convert the input and target Tensors to be cuda Variables like below:\n", "\n", "`inputs, labels = Variable(inputs.cuda()), Variable(labels.cuda())`\n", "- and convert your model to cuda:\n", "\n", " `model.cuda()`\n", "\n", "The following links would be useful: when you want to run your model on a GPU: \n", " - [pytorch_doc](http://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html#training-on-gpu)\n", " - [mnist pytorch gpu example](https://github.com/pytorch/examples/blob/master/mnist/main.py)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "6. Adding a CRF on top of BiLSTM\n", "=================\n", "\n", "We will now be building a `CRF` on top of the `BiLSTM` for Part-of-Speech tagging. The LSTM tagger above is typically sufficient for part-of-speech tagging, but a sequence model like the CRF might be helpful for tasks like Named Entity Recognition where the transitions and the overall structure is very important.\n", "\n", "Recall that the `CRF` computes a conditional probability. Let Y be a tag sequence and X an input sequence of words. Then we compute \n", "\n", "$P (Y \\mid X) = \\frac{exp(Score(X,Y))}{\\sum_{Y_i}exp(Score(X,Y_i))}$ \n", "where the the score is determined by defining some log potentials $\\log\\Psi(X,Y)$ such that these potential only look at the local features. \n", "\n", "In the Bi-LSTM CRF, we define two kinds of potentials: emission and transition. \n", "- The emission potential for the word at index $i$ comes from the hidden state of the Bi-LSTM at timestep $i$. \n", "- The transition scores are stored in a `|T|x|T|` matrix `transitions`, where T is the tag set. In my implementation, `transitions[j][k]` is the score of transitioning to tag k from tag j.\n", "\n", "$ Score(X,Y) = \\sum_{i}\\log\\Psi_{emit}(y_i \\rightarrow x_i) + \\sum_{i}\\log\\Psi_{trans}(y_{i-1} \\rightarrow y_{i}) \\\\\n", " Score(X,Y) = \\sum_{i} h_{i}[y_{i}] + transitions_{y_i,y_{i-1}} $\n", " \n", "See the notes for more details on this." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- Since, we are now using a `CRF`, we need to add `START_TAG` and `END_TAG` to the tagset: so, we modify `tag_to_ix` and `ix_to_tag` below." ] }, { "cell_type": "code", "execution_count": 134, "metadata": { "collapsed": true }, "outputs": [], "source": [ "#adding START_TAG and END_TAG to all_tags, updating tag_to_ix\n", "if START_TAG not in all_tags:\n", " all_tags.append(START_TAG)\n", "if END_TAG not in all_tags:\n", " all_tags.append(END_TAG)\n", "all_tags = sorted(all_tags)\n", "if START_TAG not in tag_to_ix:\n", " tag_to_ix[START_TAG] = len(tag_to_ix)\n", "if END_TAG not in tag_to_ix:\n", " tag_to_ix[END_TAG] = len(tag_to_ix)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- We have already provided the transitions parameter to the `bilstm.BiLSTM_CRF` class. Take a look at it." ] }, { "cell_type": "code", "execution_count": 135, "metadata": { "collapsed": true }, "outputs": [], "source": [ "# setting random seed\n", "reload(bilstm);" ] }, { "cell_type": "code", "execution_count": 136, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "BiLSTM_CRF(\n", " (word_embeds): Embedding(6900, 30)\n", " (lstm): LSTM(30, 15, bidirectional=True)\n", " (hidden2tag): Linear(in_features=30, out_features=19)\n", ")\n" ] } ], "source": [ "# initializing model\n", "torch.manual_seed(765);\n", "embedding_dim=30\n", "hidden_dim=30\n", "model = bilstm.BiLSTM_CRF(len(word_to_ix),tag_to_ix,embedding_dim, hidden_dim)\n", "print (model)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- Observe that `BiLSTM_CRF` class is derived from the `BiLSTM` class\n", "- Note that we would be using the forward function from `bilstm.BiLSTM` class to obtain the lstm hidden states" ] }, { "cell_type": "code", "execution_count": 137, "metadata": { "collapsed": true }, "outputs": [], "source": [ "# let us prepare an input sequence\n", "sentence = bilstm.prepare_sequence(X_tr[5], word_to_ix)" ] }, { "cell_type": "code", "execution_count": 138, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Variable containing:\n", "-0.2339\n", " 0.1689\n", " 0.0046\n", " 0.0083\n", " 0.0977\n", "[torch.FloatTensor of size 5]\n", "\n" ] } ], "source": [ "lstm_feats = model.forward(sentence)\n", "print (lstm_feats[0][0:5])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Deliverable 6.1** (*0.5 points*) \n", "Complete the `bilstm.BiLSTM_CRF.forward_alg()` function to do the following: \n", "This function calculates the log likelihood score for a particular sentence. It works very similar to the `viterbi` algorithm: Instead of finding the maximum `prev_tag`, you need to calculate the probability to arrive at the `curr_tag` for the `curr_token`. The forward algorithm is described in chapter 6 of the class notes.\n", "\n", "- **Inputs**: `feats`, the hidden states for each token in the input_sequence. Consider this to be the emission potential of each token for each tag.\n", " - Make sure to use the `self.transitions` that is defined to capture the tag-transition probabilities\n", "- **Outputs**: `alpha`, a pytorch variable containing the scalar score for the entire input, $\\log \\sum_{y_{1:M}} P(w, y)$ \n", "- **Tests**: ```test_bilstm_crf.py: test_forward_alg()```" ] }, { "cell_type": "code", "execution_count": 139, "metadata": { "collapsed": true }, "outputs": [], "source": [ "torch.manual_seed(765);" ] }, { "cell_type": "code", "execution_count": 140, "metadata": { "collapsed": true }, "outputs": [], "source": [ "reload(bilstm);\n", "model = bilstm.BiLSTM_CRF(len(word_to_ix),tag_to_ix,embedding_dim, hidden_dim)" ] }, { "cell_type": "code", "execution_count": 141, "metadata": { "collapsed": true }, "outputs": [], "source": [ "# passing an input sentence to obtain lstm_feats\n", "lstm_feats = model.forward(sentence)" ] }, { "cell_type": "code", "execution_count": 142, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[42.802666]\n" ] } ], "source": [ "alpha = model.forward_alg(lstm_feats)\n", "print (alpha.data.numpy())" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Deliverable 6.2** (*0.5 points*)\n", "Complete the function `bilstm.BiLSTM_CRF.score_sentence()` to obtain the joint log-likelihood score of the particular sequence of tokens and their tags.\n", "- **Inputs**:\n", " - `feats`: the hidden state scores for each token in the input sentence. Consider this to be the emission potential of each token for each tag.\n", " - `gold_tags`: a pytorch Variable of the gold sequence of tags: obtain the joint-log-likelihood score of the sequence with the feats and gold_tags.\n", "- **Outputs**:\n", " - a pytorch variable of the score. \n", "- **Tests**: ```test_bilstm_crf.py: test_score_sentence()```" ] }, { "cell_type": "code", "execution_count": 143, "metadata": { "collapsed": true }, "outputs": [], "source": [ "torch.manual_seed(765);" ] }, { "cell_type": "code", "execution_count": 144, "metadata": { "collapsed": true }, "outputs": [], "source": [ "reload(bilstm);\n", "model = bilstm.BiLSTM_CRF(len(word_to_ix),tag_to_ix,embedding_dim, hidden_dim)" ] }, { "cell_type": "code", "execution_count": 145, "metadata": { "collapsed": true }, "outputs": [], "source": [ "# prepare both the tokens and tags to be passed as input\n", "sentence = bilstm.prepare_sequence(X_tr[5], word_to_ix)\n", "tags = bilstm.prepare_sequence(Y_tr[5], tag_to_ix)" ] }, { "cell_type": "code", "execution_count": 146, "metadata": { "collapsed": true }, "outputs": [], "source": [ "torch.manual_seed(765);\n", "lstm_feats = model.forward(sentence)" ] }, { "cell_type": "code", "execution_count": 147, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[-0.24207509]\n" ] } ], "source": [ "score = model.score_sentence(lstm_feats, tags)\n", "print (score.data.numpy())" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "** Deliverable 6.3 ** (*0.5 points* for 4650, *0.25 points* for 7650) Complete the `bilstm.BiLSTM_CRF.predict()` function to decode the tags using the viterbi algorithm. Make sure to use the viterbi functions defined previously.\n", "- **Inputs**:\n", " - `sentence`: a pytorch Variable of sequence of ids of the input_tokens\n", "- **Outputs**:\n", " - `best_path`: a list of tags for the sequence of tokens\n", "- **Tests**: ```test_bilstm_crf.py: test_predict()```" ] }, { "cell_type": "code", "execution_count": 148, "metadata": { "collapsed": true }, "outputs": [], "source": [ "reload(bilstm);\n", "torch.manual_seed(765);" ] }, { "cell_type": "code", "execution_count": 149, "metadata": { "collapsed": true }, "outputs": [], "source": [ "model = bilstm.BiLSTM_CRF(len(word_to_ix),tag_to_ix,embedding_dim, hidden_dim)" ] }, { "cell_type": "code", "execution_count": 150, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "['PROPN', 'X', 'PROPN', 'X', 'PROPN', 'X', 'PROPN', 'X', 'PROPN', 'X', 'PROPN', 'X', 'PROPN']\n" ] } ], "source": [ "best_tags = model.predict(sentence)\n", "print (best_tags)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Deliverable 6.4** (*0.5 points* for 4650, *0.25 points* for 7650)\n", "Complete the `bilstm.BiLSTM_CRF.neg_log_likelihood()` function to obtain the negative log likelihood loss. This function calculates the loss function for the `CRF`. Observe that, this can be easily calculated using the previously defined functions. \n", "You should use the `forward(), forward_alg(), score_sentence()` functions defined previously.\n", "\n", "- **Tests**: ```test_bilstm_crf.py: test_neg_log_likelihood()```" ] }, { "cell_type": "code", "execution_count": 151, "metadata": { "collapsed": true }, "outputs": [], "source": [ "#setting random seed\n", "reload(bilstm);" ] }, { "cell_type": "code", "execution_count": 152, "metadata": { "collapsed": true }, "outputs": [], "source": [ "torch.manual_seed(765);\n", "model = bilstm.BiLSTM_CRF(len(word_to_ix),tag_to_ix,embedding_dim, hidden_dim)" ] }, { "cell_type": "code", "execution_count": 153, "metadata": { "collapsed": true }, "outputs": [], "source": [ "lstm_feats = model.forward(sentence)" ] }, { "cell_type": "code", "execution_count": 154, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Variable containing:\n", " 43.0496\n", "[torch.FloatTensor of size 1]\n", "\n" ] } ], "source": [ "loss = model.neg_log_likelihood(lstm_feats, bilstm.prepare_sequence(Y_tr[5], tag_to_ix))\n", "print (loss)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- Putting all the above components together, let us train the `BiLSTM-CRF` model. **Don't worry about matching the scores for the following code blocks from now. Just Make sure that you pass the unit tests.** \n", "- You may want to use pretrained embeddings to improve the performance of the model." ] }, { "cell_type": "code", "execution_count": 155, "metadata": { "collapsed": true }, "outputs": [], "source": [ "reload(bilstm);\n", "torch.manual_seed(765);\n", "embedding_dim=30\n", "hidden_dim=30\n", "model = bilstm.BiLSTM_CRF(len(word_to_ix),tag_to_ix,embedding_dim, hidden_dim)" ] }, { "cell_type": "code", "execution_count": 157, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Epoch 1: Dev Accuracy: 0.39325842696629215\n", "Epoch 3: Dev Accuracy: 0.43258426966292135\n", "Epoch 5: Dev Accuracy: 0.48314606741573035\n", "Epoch 7: Dev Accuracy: 0.4606741573033708\n", "Epoch 9: Dev Accuracy: 0.4887640449438202\n" ] } ], "source": [ "# sending loss function so that we can calculate the loss for the BiLSTM-CRF\n", "loss = model.neg_log_likelihood\n", "model, losses, accuracies = bilstm.train_model(loss, model, X_tr,Y_tr, word_to_ix, tag_to_ix,\n", " X_dv, Y_dv, \n", " num_its=10, status_frequency=2,\n", " optim_args = {'lr':0.1,'momentum':0}, param_file = 'best.params')" ] }, { "cell_type": "code", "execution_count": 158, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAtEAAACdCAYAAAByzCQ5AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xd8VGX2+PHPyaQSICQkhBBKAlKlE7pYsKGgwtoQCyIu\nsnbXta676+66+8PdtX0VCzZQUcSOwIJdFBQTWiB0qYEAoffU8/tjLjiygUwgk5vMnPfrNa+Z+9z7\nzJyr5MnJM08RVcUYY4wxxhjjvzC3AzDGGGOMMaamsSTaGGOMMcaYCrIk2hhjjDHGmAqyJNoYY4wx\nxpgKsiTaGGOMMcaYCrIk2hhjjDHGmAqyJNoYY4wxxpgKsiTaGGOMMcaYCrIk2hhjjDHGmAoKdzsA\nfyQmJmpaWprbYRhjTIXNmzdvu6omuR1HVbI22xhTk/nbbteIJDotLY2srCy3wzDGmAoTkfVux1DV\nrM02xtRk/rbbNpzDGGOChIgMEJEVIrJaRB4s4/zZIrJHRBY6jz+XV1dEEkTkcxFZ5TzHV9X9GGNM\ndRa0SbSquh2CMcZUGRHxAGOBi4B2wDUi0q6MS79T1c7O429+1H0Q+FJVWwJfOsfGGBPygjKJPlxU\nwojxmUxZtNntUIwxpqr0AFar6hpVLQQmAZdVQt3LgAnO6wnA4EqM2RhjKtXew0V8OD+Xm8ZnMjNn\nS0A/q0aMia6o4lLlYGEJd01awKHCYq7u3tTtkIwxJtBSgY0+x7lAzzKu6yMi2cAm4A+qmlNO3WRV\nzXNebwGSy/pwERkFjAJo2tTaXGNM1TlQUMwXy7YyNTuPb1fmU1hcSqO4aA4UFAf0c4Myia4dFc6E\nET0Y/dY8HvhgMQcKSrjpjHS3wzLGGLfNB5qq6n4RuRj4GGjpb2VVVREpc6ycqo4DxgFkZGTYeDpj\nTEAdLCzm6+X5TM3ezFfLt1FQXEpy3Siu7dmUQR0b0aVJPcLCJKAxBGUSDRAT6WHcDd24850F/G3q\nUg4VlXDbOae5HZYxxgTKJqCJz3Fjp+woVd3r83q6iDwvIonl1N0qIimqmiciKcC2gERvjDHlOFxU\nwjcrvInzl8u2caiohMTaUVzdvQmDOjYio1l8wBNnX0GbRANEhXsYO6wr972fzb9nrmB/QTH3X9ga\nkar7D2yMMVUkE2gpIul4E+ChwDDfC0SkIbDV6VHugXdezA5g9wnqTgGGA2Oc50+q4F6MMQaAguIS\nvlu5nanZm/l86VYOFJaQEBvJkK6pDOqYQs/0+niqMHH2FdRJNEC4J4wnruxEdISHF775mUOFJfx5\nULsq/UvFGGMCTVWLReR2YCbgAV5T1RwRGe2cfxG4AvidiBQDh4Ch6l3KqMy6zluPASaLyEhgPXBV\nld6YMSbkFBaXMvvn7UxdlMdnS7ew73AxcTERDOrYiEGdUujdvD7hHvfXxgj6JBogLEz455D2xEZ6\neOX7tRwoKGbM5R1d+8vFGGMCQVWnA9OPKXvR5/VzwHP+1nXKdwDnVm6kxhjza8Ulpcz5eQfTsvOY\nkbOFPYeKqBMdzgXtGjKoUwpnnJZIRDVInH2FRBINICL8cWBbYqPCeebLVRwqKuGpqztXu/8hxhhj\njDGhoKRUmbtmB1MX5zFjyRZ2HiikdlQ457dLZmCHFPq1SiQq3ON2mMcVMkk0eBPpe85vRWyUh39O\nX87hohKeG9aV6Ijq+z/IGGOMMSZYlJYqmet2Mm1xHtMXb2H7/gJiIjyc5yTOZ7dOqjF5WUgl0UeM\nOrMFMZHh/OnjJYyckMm46zOIjQrJ/xTGGGOMMQFVWqos2LiLqdl5TF+cx9a9BURHhNG/TQMGdmhE\n/zYNiImsGYmzr5DNHK/v1YxaER7ue38RN7z2E6/d2J24mAi3wzLGhDgReYJfT+wzxpiTtudgET+s\n2YF3DnHVKlVYuHEX07Lz2LznMJHhYZzdKolBnRpxbpsGNb4Ds2ZHf4ou79aYWpEe7py0gGEv/8ib\nI3uSEBvpdljGmNC2DBgnIuHA68A7qrrH5ZiMMTXQqq37uGlCJht3HnIthgiPcGbLJO4b0Jrz2iZT\nJzp4OixDOokGuKhDCuMiPYx+cx5Xv/QDb93ck+S60W6HZYwJUar6CvCKiLQGRgDZIjIbeFlVv3Y3\nOmNMTTFrZT63TZxPVISH10d0JyXOndwmJS4maL/pD/kkGuCc1g0YP6IHIydkctVLPzDx5p40jq/l\ndljGmBAlIh6gjfPYDiwCfi8it6jqUFeDM8ZUe2/+uJ5Hp+TQskFtXhmeYTlNgNj6bo7eLerz1s09\n2XWgkCtf/IE1+fvdDskYE4JE5ClgOXAx8E9V7aaqj6vqJUAXd6MzxlRnJaXKXz/N4U8fL+GsVkm8\n/7s+lkAHkCXRPro2jeedUb0oLC7lqpd+ZPmWvW6HZIwJPdlAZ1W9RVV/OuZcDzcCMqYyFRSXUFRS\n6nYYQWd/QTE3T8jk9dnruKlvOi/fkEHtGj5xr7qzJPoYpzeK491beuMJg6HjfiQ7d7fbIRljQstu\nfIbaiUg9ERkMYBMMTU2373ARlz03m7P+9TXfr9rudjhBI3fXQa54YQ6zVm3nscHt+fMl7WxX5ioQ\nsCRaRKJF5CcRWSQiOSLyV6c8QUQ+F5FVznN8oGI4Wac1qM17t/ShTnQ4w16ey09rd7odkjEmdPzF\nN1lW1d3AX1yMx5hKUVxSyu1vL2D1tv1EhIdx3atzeeTjxRwoKHY7tBptwYZdDB47h027DvH6jd25\nrlczt0MKGYHsiS4A+qtqJ6AzMEBEegEPAl+qakvgS+e42mlavxaTb+lNg7pR3PDaXGatzHc7JGNM\naCirXbbvZE2N99i0ZXy7Mp/HBrdn5t1ncvMZ6Uycu4GLnvnOOqtO0tTszQwd9yMxkWF8eGsfzmyV\n5HZIISVgSbR6HZmdF+E8FLgMmOCUTwAGByqGU5USF8PkW3qTnlibmydkMTNni9shGWOCX5aIPCki\nLZzHk8A8t4My5lS88cM6xs9Zx2/7pTO0R1OiIzw8Mqgd747qDcDV437g71OXcrioxN1AawhV5f++\nXMXtby+gQ2ocH9/al5bJddwOK+QEdEy0iHhEZCGwDfhcVecCyaqa51yyBUg+Tt1RIpIlIln5+e71\nAifWjmLSb3vRrlFdbp04n08WbnItFmNMSLgDKATedR4FwG2uRmTMKfh2ZT5//XQp57VtwIMXtf3V\nuR7pCfz3rn5c17MZr36/lov/7zsWbNjlUqQ1w+GiEu55dyFPfr6SIV1SmfjbntSvHeV2WCEpoEm0\nqpaoamegMdBDRNofc17x9k6XVXecqmaoakZSkrtfT8TViuCtm3vSPS2eu99dyDs/bXA1HmNM8FLV\nA6r64JH2T1UfUtUD/tQVkQEiskJEVovIcYfKiUh3ESkWkSuc49YistDnsVdE7nbOPSoim3zOXVw5\nd2pCwaqt+7h94nxaJdfhmaFdypzsFhsVzt8Ht+etkT05XFjC5S/M4V8zllNQbL3Sx9qxv4DrXpnL\nxws3c+/5rXjyqk5EhXvcDitkVck4O1XdLSJfAwOArSKSoqp5IpKCt5e62qsdFc74ET0Y/dY8Hvpw\nMQcLSxh5RrrbYRljgoyIJAH3A6cDR7cYU9X+5dTzAGOB84FcIFNEpqjq0jKuexz4zOe9V+Cdu3Lk\n/CbgI59qT6nqf07htkwI2rG/gJsmZBId6eHV4RnElrPc2hktE5lxz5k8NnUpz3/zM18t38Z/ruxE\n+9S4Koq4ejuyhfe2vQU8N6wLgzo2cjukkBfI1TmSRKSe8zoGb8O+HJgCDHcuGw58EqgYKlt0hIeX\nru/GgNMb8vepS3n2y1V4O9ONMabSTMTbVqYDfwXWAZl+1OsBrFbVNapaCEzCOwflWHcAH3D8Doxz\ngZ9VdX0F4zbmqILiEm55cx7b9hbw8g0ZNKoX41e9utER/OuKTrx2YwY7DxQyeOxsnv5iZcivKz1r\nZT6/eX4OhwpLefeW3pZAVxOBHM6RAnwtItl4fwF8rqpTgTHA+SKyCjjPOa4xosI9PDesC7/pksoT\nn69kzIzllkgbYypTfVV9FShS1W9V9SbghL3QjlRgo89xrlN2lIikAkOAF07wPkOBd44pu0NEskXk\nteMtS1pd5rEY96kqD36wmKz1u3jyqs50blKvwu/Rv00yn91zJoM6pvD0F6sY8vxsVm7dF4Boq783\nf1zPiPGZpMbH8MntfU/qv6cJjECuzpGtql1UtaOqtlfVvznlO1T1XFVtqarnqWqNW9cm3BPGf67s\nxLU9m/LSt2v48yc5lJZaIm2MqRRFznOeiAwUkS5AQiW999PAA6paZreeiEQClwLv+RS/ADTHO9wj\nD3iirLrVaR6LcdfYr1fz0YJN/OGCVgzsmHLS71OvViRPD+3Ci9d1JW/3YQb93/e88M3PlITI79uy\ntvBO9bNH31QNW3v0JIWFCY8Nbk9sVDjjZq3hYGEJj1/egXCPbQJpjDklj4lIHHAv8CxQF7jHj3qb\ngCY+x42dMl8ZwCQRAUgELhaRYlX92Dl/ETBfVbceqeD7WkReBqZW7HZMKJmWncd/PvOuGnHbOadV\nynsOaJ9C97QEHvl4CY/PWM5nS7fwxJWdaJ5Uu1Levzrad7iIO99ZwNcr8rmpbzp/HNjWdiCshiyJ\nPgUiwkMXtSE2MpynvljJ4aISnrq6M5HhlkgbYyrOmdTX0hn6tgc4pwLVM4GWIpKON3keCgzzvUBV\nj86GFpHxwFSfBBrgGo4ZynFkIrhzOARYUoGYTAhZtHE3v5+8kIxm8Yy5vAPOH2uVon7tKJ6/titT\nFm3mz5/kcNEz3/HAgDbc2CeNsCBLLnN3HWTk+CxW5+/nscHtbQfCasyS6FMkItx1XktqRXr4x/Rl\nHCoq4flruxIdYUvOGGMqRlVLROQa4KmTqFssIrcDMwEP8Jqq5ojIaOf8iyeqLyKxeCeA33LMqX+J\nSGe8y5GuK+O8MWzefYib38iiQd0oXrq+W0CWXRMRLuucSq/m9Xnow8X8bepSZuZs4T9XdqJJQq1K\n/zw3LNiwi9++MY+C4hLGj+hOv5Y2NKo6k5owKS4jI0OzsrLcDqNcE+eu55GPl9ArvT4v3dCNutER\nbodkjHGZiMxT1YwKXP8U3h1e3wWOrg+tqvMDEF5A1JQ221SOAwXFXPHiD+TuPMgHt/ahVRXsnKeq\nvDcvl79/upQSVf44sC3DejSt1N7vqvbpos3c+94ikutG8fqN3Tmtge1A6BZ/223ria5E1/ZsRq1I\nD394L5tLn/2esdd25fRGtr6lMaZCOjvPf/MpU/xbocOYKlVSqtw1aSErtuzl9RE9qiSBBm+v9FUZ\nTeh7WiIPvJ/NHz9awowlW3j88o5+L6dXXagqz361mic/X0n3tHhevK6b7UBYQ9jg3Uo2pEtjJo3q\nxaGiEoY8P4dJP22wJfCMMX5T1XPKeFgCbaqlMf9dxhfLtvLopadzVquqH3qQWi+GN0f24O+D2zNv\n/S4ufGoW72VtrDG/d4/dwvutm20L75rEeqIDoHtaAtPu7Mfdkxby4IeLyVy3i8cGtycm0sZJG2NO\nTET+XFb5kWVCjaku3vlpAy9/t5bhvZtxQ+801+IQEa7v1YwzWyZy33vZ3Pd+NjNztvDP33SgQZ3o\n8t/AJTv2F3DLm/PIWr+LP1zQitvOOa1GD0cJRdYTHSCJtaOYcFMP7jq3JR8uyGXw2Nn8nL/f7bCM\nMdXfAZ9HCd5l59LcDMiYY81Zvf3o+sV/GtTO7XAAaFY/lkmjevHIwLZ8t2o7Fzw1i08XbXY7rDKt\n2rqPwc/PZvGmPYwd1pXb+7e0BLoGsiQ6gDxhwj3nt2LCiB7k7y/g0me/r7Y/0MaY6kFVn/B5/AM4\nG+9mJ8ZUC2vy9zP6rXk0T4rl2WFdqtX+CGFhws39mjP9rn6k1Y/ljncWcNvE+ezYX+B2aEcdu4X3\nqWxIY9xVff7lB7EzWyUx7c4zaJNSlzveWcCfP1lCQXGJ22EZY2qGWng3TjHGdbsOFHLT+EwiPGG8\nOrx7tV2FqkVSbd4f3Zv7B7Tms6VbuPDpWczM2eJ2WLaFd5CxMdFVJCUuhkmjevH4f5fzyvdrWbRx\nN88N6xo0a1saYyqHiCzGuxoHeNd7TuLXK3UY44rC4lJ+N3Eem3cf5p1RPav9769wTxi3nn0a/ds0\n4N7Ji7jlzXkM6ZLKhac3dCWe2au38+aP6zm3TQOeuaYLtaMsBavpbJ1oF8xYsoX73ltEWJjw1NWd\n6N8m2e2QjDEBchLrRPtuT1YMbFXV4sqPLHCCrc023mXYHvggm8lZuTx9dWcGd0l1O6QKKSop5bmv\nVjP269UUl7qX94w8I52HL7YtvKs7Wye6GhvQviFtU+pw68T53DQ+i9+d3YJ7z29VrcaVGWNckwLk\nqOo+ABGpIyLtVHWuy3GZEDZu1homZ+VyZ//TalwCDRDhCeOe81sxrGdTduwvdCWG2CgPzerHuvLZ\nJjAsiXZJs/qxfPC7Pvz106W88M3PzF+/i2ev6UKDutV3OR5jTJV4Aejqc3ygjDJjqszMnC2MmbGc\ngR1TuPu8Vm6Hc0qS60aTbL9nTSWxrk8XRUd4+H+/6cCTV3UiO3cPF//f98z5ebvbYRlj3CXqM85O\nVUuxDg/jkiWb9nD3pIV0bFyPJ67sRJgNQzDmKEuiq4HfdG3MJ7f3JS4mnOtemcvYr1dT6uKYLWOM\nq9aIyJ0iEuE87gLWuB2UCT1b9x7m5glZxNeK4OUbuhEdYRuGGePLkuhqolVyHabcfgaDOjbi3zNX\nMHJCJrsOuDNuyxjjqtFAH2ATkAv0BEa5GpEJOYcKS7h5Qhb7Dhfx6o3dq/XOf8a4xZLoaiQ2Kpxn\nhnbm74PbM3v1DgY9+z0LNuxyOyxjTBVS1W2qOlRVG6hqsqoOU9VtbsdlQkdpqfL7yQtZsnkP/3dN\nF9qm1HU7JGOqJb+SaBG5S0TqiterIjJfRC4op04TEflaRJaKSI7zlSQikiAin4vIKuc5vjJuJFiI\nCNf3asb7v+uNCFz10g+8PnstNWEpQmPMqRORCSJSz+c4XkRe87PuABFZISKrReTBE1zXXUSKReQK\nn7J1IrJYRBaKSJZPubXZIeaJz1fw3yVb+OPFbTm3rS3Baszx+NsTfZOq7gUuAOKB64Ex5dQpBu5V\n1XZAL+A2EWkHPAh8qaotgS+dY3OMjo3rMe2OfpzVKom/frqU299ewL7DRW6HZYwJvI6quvvIgaru\nArqUV0lEPMBY4CKgHXCN0+aWdd3jwGdlvM05qtr5mPVRrc0OIe/Py2Xs1z9zTY+mjDwj3e1wjKnW\n/E2ij0zHvRh4U1VzfMrKpKp5qjrfeb0PWAakApcBE5zLJgCDKxp0qIirFcG46zN48KI2zMjZwqXP\nzWZZ3l63wzLGBFaYb2+viCTg3+ocPYDVqrpGVQuBSXjb22PdAXwA+DtExNrsEPHT2p089GE2fU+r\nz98uOx0RW4nDmBPxN4meJyKf4U2iZ4pIHaDU3w8RkTS8PSlzgWRVzXNObQHsu6ITCAsTRp/Vgrdv\n7smBgmIGj53N5KyNbodljAmcJ4AfROTvIvIYMAf4lx/1UgHfxiHXKTtKRFKBIXjXnT6WAl+IyDwR\n8Z3I6FebLSKjRCRLRLLy8/P9CNdUJ+t3HOCWN7NoklCL54d1I8I2/zKmXP7+lIzE+xVed1U9CEQA\nI/ypKCK18fZ63O0MCTnKWQu1zMG+1iD/Ws/m9Zl2Zz+6NYvn/vezue+9RRwqLHE7LGNMJVPVN4DL\nga14k9bfqOqblfT2TwMPOGtPH+sMVe2MdzjIbSJyZhmxHbfNVtVxqpqhqhlJSUmVFK6pCnsOFXHT\n+EwUeG14d+JqRbgdkjE1gr9JdG9gharuFpHrgEeAPeVVEpEIvAn0RFX90CneKiIpzvkUjvOVojXI\n/yupThRvjuzJnf1P4715uQx5fjZr8ve7HZYxppI5Q+YmA1OA/SLS1I9qm4AmPseNnTJfGcAkEVkH\nXAE8LyKDnc/c5DxvAz7COzwE/GyzTc1UVFLK7W/PZ8POg7x4XTfSEm1bamP85W8S/QJwUEQ6AfcC\nPwNvnKiCeAdTvQosU9UnfU5NAYY7r4cDn1Qo4hDnCRN+f0Frxo/ozta9h7n0udlMy84rv6IxpkYQ\nkUtFZBWwFvgWWAf814+qmUBLEUkXkUhgKN729ihVTVfVNFVNA94HblXVj0Uk1hmmh4jE4p1EvsSp\nZm12kFJVHp2Sw3ertvOPIR3o1by+2yEZU6P4m0QXO1/jXQY8p6pjgTrl1OmLdxWP/s6SSQtF5GK8\nq3qc7/ySOI/yV/kwZTi7dQOm3dmPlsm1ue3t+Tw6JYfCYr+HqRtjqq+/413RaKWqpgPnAj+WV0lV\ni4HbgZl4J3JPVtUcERktIqPLqZ4MfC8ii4CfgGmqOsM5Z212kBo/Zx0T525g9FktuCqjSfkVjDG/\n4s+Mb4B9IvIQ3qS4n4iE4R0XfVyq+j3HX8HjXP9DNMfTqF4M747qzZj/Lue12WtZuHE3Y6/tSmq9\nGLdDM8acvCJV3SEiYSISpqpfi8jT/lRU1enA9GPKXjzOtTf6vF4DdDrOdTuwNjuorN62j08Wbmbs\n16u5oF0y91/Y2u2QjKmR/E2irwaG4V0veoszPu/fgQvL+CsyPIw/X9KO7mneCYcXPT2Ley9ozbU9\nmxJus6uNqYl2OxOyZwETRWQbcMDlmEwNt3b7AaYu2szU7DxWbN2HCJzbpgFPD+1MWJgtZWfMyRB/\nd8ITkWSgu3P4U1VuQ5uRkaFZWVnlXxji1m0/wCMfL+H71dtpnVyHv1zajj4tEt0Oy5iQJiLzjtm8\npLzrY4FDeIfbXQvE4Z2cvSNAIVY6a7Orhw07DjJ18WamLspjqbPHQPe0eAZ1bMRF7RvSoG60yxEa\nUz3522771RMtIlfh7Xn+Bu8QjWdF5D5Vff+UojSVKi0xljdH9mBmzlYem7aUYS/PZWCHFB4e2NaG\neBhTQ6jqkV7nUn7Z5MQYv+TuOsj0xXlMzc4jO9e7iFaXpvX406B2XNyhISlx9rvAmMri73COP+Jd\nI3obgIgkAV/gnd1tqhERYUD7hpzdOolxs9bw/Der+XL5Vm49+zRGndmc6AiP2yEaY4ypRHl7DjF9\n8RamZm9mwQbvjvEdG8fx8MVtuLhDCo3ja7kcoTHByd8kOuyY4Rs78H9lD+OC6AgPd57bksu7Neaf\n05fx5OcrmZy1kUcGtuPC05NtO1djjKnBtu09zPTFeUxbnEfmul0AtEupy/0DWjOoQyOa1rfE2ZhA\n8zeJniEiM4F3nOOrOWYGuKmeUuvFMHZYV67ruYO/fprD6LfmccZpifzlkna0TC5vlUJjTFUTkbtU\n9Znyykzo2b6/gBlLvD3Oc9fuRBXaNKzDvee3YmDHFJon1XY7RGNCSkUmFl6Od+1ngO9U9aOARXUM\nm6RSOYpLSnn7pw088dlK9hcUM7x3Gned15K4GNvi1ZhAOYmJhfNVtesxZQtUtUvlRxcY1mZXnl0H\nCpmRs4Vp2XnM+Xk7pQotkmIZ1LERgzqmWGeIMQFQqRMLAVT1A7xbeJsaKtwTxg290xjUsRH/+WwF\nr89ZyycLN3H/gNZc2a2JLXNkjItE5Bq8S4mmi4jvToN1gJ3uRGXcsOdgETOXehPn2au3U1yqpNWv\nxa1nn8agTim0Tq5jQ/KMqQZOmESLyD6grK5qAVRV6wYkKhNQCbGR/HNIB4b1aMqjU3J44IPFTJy7\ngb9ccjrdmsW7HZ4xoWoOkAckAk/4lO8Dsl2JyFSZfYeL+HzpVqZl5zFrVT5FJUqThBhu7tecQR1T\nOL1RXUucjalmTphEq6p9TxTE2qfG8d7o3kxZtJl/Tl/G5S/M4TddU3lwQBtbP9SYKqaq64H1QG8R\naQa0VNUvRCQGiMGbTJsgUlqqTFucx5RFm/l2ZT6FxaU0iovmxj7ebww7No6zxNmYaszv4RwmOIkI\nl3VO5by2yYz9ejWvfLeWz3K2ckf/0xjRN53IcFuExZiqJCK/BUYBCUALoDHwIrb1dlA5WFjM3ZMW\n8tnSrSTXjeLank0Z1LERXZrUs6F1xtQQlkQbAGKjwrl/QBuuymjCY9OW8v/+u5x3Mzfyp0vacU7r\nBm6HZ0wouQ3oAcwFUNVVImI/hEFky57DjJyQybK8vfxpUDtG9EmzxNmYGsi6Gc2vpCXG8srw7rw+\nwrvD+4jXMxk5PpN12w+UU9MYU0kKVLXwyIGIhFP23BRTAy3ZtIfLxn7Puu0HeGV4BiPPSLcE2pga\nypJoU6ZzWjdgxt1n8tBFbfhxzQ4ueGoW/5qxnAMFxW6HZkyw+1ZEHgZiROR84D3gU5djMpVgZs4W\nrnzxB8LDwvjg1j70b5PsdkjGmFNgSbQ5rsjwMG45qwVf/+FsBnVK4flvfqb/E9/wycJN+Lu+uDGm\nwh4E8oHFwC14N7Z6xJ+KIjJARFaIyGoRefAE13UXkWIRucI5biIiX4vIUhHJEZG7fK59VEQ2ichC\n53HxKd1dCFJVXvz2Z0a/NY/WDevw0W19aNPQFrcypqazJNqUq0HdaJ68qjMf/K4PDepEc9ekhVz5\n4g8s2bTH7dCMCTqqWqqqL6vqlXgnGM5VP/5qFREPMBa4CGgHXCMi7Y5z3ePAZz7FxcC9qtoO6AXc\ndkzdp1S1s/Ow3WoroLC4lAc+yGbMf5dzcYcUJo3qRYM6tvqRMcHAkmjjt27N4vnktr48fnkH1m4/\nwCXPfc/DHy1m54HC8isbY/wiIt+ISF0RSQDmAS+LyFN+VO0BrFbVNc6Y6knAZWVcdwfejbO2HSlQ\n1TxVne+83gcsA1JP8VZC3u6Dhdzw2lwmZ+VyZ//TeHZoF6IjPG6HZYypJJZEmwoJCxOu7t6Ur/5w\nNiP6pPNu5kbO/vfXTJizjuKSUrfDMyYYxKnqXuA3wBuq2hP/lrdLBTb6HOdyTCIsIqnAEOCF472J\niKQBXXBWB3HcISLZIvKaiJS5I5OIjBKRLBHJys/P9yPc4LYmfz9Dnp/D/PW7eerqTvz+gtY2gdCY\nIGNJtDkVqFd/AAAUDUlEQVQpcTER/PmSdsy4qx8dGsfxlyk5nPfkt4yfvZb9NvnQmFMRLiIpwFXA\n1Ep+76eBB1S1zL94RaQ23l7qu51EHrwJd3OgM94dFZ8oq66qjlPVDFXNSEpKquSwa5Y5P29nyPNz\n2HOoiLd/25MhXRq7HZIxJgAClkQ7PRbbRGSJT1mCiHwuIqucZ9tjuoZrmVyHt0b25KXruxEfG8mj\nny6l1z+/5K+f5rB+hy2LZ8xJ+BswE+/QjEwRaQ6s8qPeJqCJz3Fjp8xXBjBJRNYBVwDPi8hgABGJ\nwJtAT1TVD49UUNWtqlriJN4v4x02Yo5jcuZGbnj1JxrUieLjW/uSkZbgdkjGmACRQK2yICJnAvvx\nfh3Z3in7F7BTVcc4M8fjVfWB8t4rIyNDs7KyAhKnqVwLN+5m/Oy1TFucR3Gpcm6bBozom06fFvVt\n+1oTkkRknqpmVMHnhAMr8Q792ARkAsNUNec4148Hpqrq++L94ZyAt32++5jrUlQ1z3l9D9BTVYee\nKJZQbLNLS5XHZyznpVlr6NcykbHXdqVudITbYRljToK/7XbAdixU1VnO2DpflwFnO68nAN8A5SbR\npubo3KQeTw/twsMXt+WtH9czce4Gvlg2l1bJtbmxTzpDuqQSE2kTa4ypbKpaLCK34+3F9gCvqWqO\niIx2zr94gup9geuBxSKy0Cl72FmJ418i0hnvhi/r8C67Z3z4buF9Xa+mPHrJ6YR7bLSkMcEuYD3R\ncHSCylSfnujdqlrPeS3AriPHZdQdhXd5J5o2bdpt/fr1AYvTBM7hohKmZufx+uy15GzeS71aEQzt\n3pQbejejUb0Yt8MzJuCqqie6Ogmlnuhjt/C+sU+afetmTA3nek90eVRVReS4GbyqjgPGgbdBrrLA\nTKWKjvBwRbfGXN41lcx1u3h99lrGzfqZl79bw4WnJzOibzoZzeLtl44xDhHxqGqJ23GY8i3ZtIeR\nEzLZf7iYV4Zn2A6ExoSYqk6itx4ZX+fMPt9Wbg0TFESEHukJ9EhPYNPuQ7zxwzom/bSR6Yu30D61\nLjf2SeeSTilEhdtQDxPy1orIDOBd4Ct/NloxVW9mzhbunrSQhNhIPrjVdiA0JhRV9aCtKcBw5/Vw\n4JMq/nxTDaTWi+Ghi9ryw0P9+ceQ9hQUlfKH9xbRd8xXPPn5SrbtPex2iMa4qQ3wBXAb3oT6ORE5\nw+WYjMO28DbGHBHI1TnewTuJMBHYCvwF+BiYDDQF1gNXqerO8t4rlMbXhSJVZfbqHbw+ey1frdhG\neJgwsEMKI/qm06lJmUPmjakxTmVMtLMM6DPAtapaY76mCdY2u7C4lEc+XszkrFwGdUzhP1d2sh0I\njQlCro+JVtVrjnPKn523TAgREc5omcgZLRNZt/0AE35Yx3tZuXy8cDNdm9ZjRN90BrRvSITNdjch\nQkTOAq4GBgBZeDdeMS7afbCQ0W/N48c1O7mz/2ncfV4r24HQmBDn2sRCY8qSlhjLXy45nd+f34r3\n5+UyYc467nhnAQ3rRnN972Zc06MpCbGRbodpTMA4G6EswPut3X2qarsWuWxN/n5GTshi065DPH11\nZwZ3SS2/kjEm6FkSbaqlOtERjOibzvDeaXyzchuvz17Hv2eu4JkvVzG4cyNG9E2nbYqNQzRBqaPP\nltvGZXN+3s7v3pqPJ0x4+7c9bQdCY8xRlkSbai0sTOjfJpn+bZJZtXUfr89Zx4fzc5mclUuv5gmM\n6JvOeW2T8djXqiZ4NBSRj4BkVW0vIh2BS1X1MbcDCzWTMzfy8EeLSU+M5dXh3Wlav5bbIRljqpGA\nbrZSWYJ1koo5ObsPFvJu5kbe+GE9m3YfonF8DBe1b0hGWgIZzeKpXzvK7RCNOaqiEwtF5FvgPuAl\nVe3ilC05smlVTVDT22zbwtuY0Ob6xEJjAqVerUhuOasFI89I54tlW3nzx/VM+GE9L3+3FoDmibFk\npMWTkZZA97QE0urXss1cTE1SS1V/OubfbLFbwYQa28LbGOMvS6JNjRXuCWNA+xQGtE+hoLiEJZv2\nkLluF1nrdvLZ0q1MzsoFILF2JBnNEshIi6d7WgLtGtW1lT5MdbZdRFoACiAiVwB57oYUGny38H70\nknYMty28jTEnYEm0CQpR4R66NUugW7MEOKsFpaXKz/n7jybVmet3MiNnCwAxER66NK3n9FTH06Vp\nPLWj7EfBVBu3AeOANiKyCVgLXOduSMFvce4ebn7Du4X3q8O7c06bBm6HZIyp5ixzMEEpLExomVyH\nlsl1GNazKQBb9x4ma90uMtftJGv9Tp77ahWlCmECbVPq0j3tl97q5LrRLt+BCVWqugY4T0RigTBV\n3ed2TMGspFSZmr2ZBz9YbFt4G2MqxJJoEzKS60YzsGMKAzumALC/oJgFG3Yd7a1+N3Mj4+esA6BJ\nQgzdmyUc7a1ukVTbNlYwASUivz9OOQCq+mSVBhTESkuVrPW7mJa9melLtpC/r4DOTeox7oZuNKhj\nf0AbY/xjSbQJWbWjwunXMol+LZMAKCopZenmvWSu28m89buYtSqfDxdsAqBerQgymsUfTarbp8YR\nFW7b/ZpKVcd5bg10B6Y4x5cAP/nzBiIyAO824R7gFVUdc5zrugM/AENV9f0T1RWRBOBdIA1YB1yl\nqrsqeG+uU1Xmb9jNtOw8pi/OY8vew0SFh9G/TQMGdWzEee0a2M+0MaZCLIk2xhHhCaNTk3p0alKP\nm/t5f+mu33HQO/xj3S4y1+/ki2XbAIgMD6NT4zi6NouneWIsTeJr0SShFilx0TaT35wUVf0rgIjM\nAroeGcYhIo8C08qrLyIeYCxwPpALZIrIFFVdWsZ1jwOf+Vn3QeBLVR0jIg86xw+c4u1WCVUlO3cP\n0xbnMS07j027DxHpCeOs1kk81LEN57ZNtvkQxpiTZq2HMcchIqQlxpKWGMuVGU0A2L6/gHnrncmK\n63bx6ndrKS79Za11T5jQsG40TRJiaBJfi8bxtbyvE2rROD6G5DrRNizElCcZKPQ5LnTKytMDWO2M\nqUZEJgGXAUuPue4O4AO8vd3+1L0MONu5bgLwDdU4iVZVcjbvZWp2HtMWb2bjzkNEeIR+LZO494JW\nnNcu2dZ8NsZUCkuijamAxNpRXHh6Qy48vSHgHQKSt/swubsOsnHXQTbuPOS8PsS3K/PZtq/gV/Uj\nPWGkxsfQOD7maILdOL4WTeK9iXb92EhbUsu8Afzk7FoIMBgY70e9VGCjz3Eu0NP3AhFJBYYA5/Dr\nJPpEdZNV9cgSe1vwL6GvUqrKiq37mLooj2mL81i7/QCeMKHvaYnc0b8lF7ZrSFwtS5yNMZXLkmhj\nTkGEJ4ym9Wsddzvgw0UlbNp9iI07D5K76xAbd3mfc3ceZObmLew8UPir62MiPDR2Euom/5No17JE\nIASo6j9E5L9AP6dohKouqKS3fxp4QFVLT+aPNVVVESlzm1sRGQWMAmjatOkpBemv1dv28amTOK/e\ntp8wgT4tEhl1ZnMuPL0hCbGRVRKHMSY0WRJtTABFR3hokVSbFkm1yzx/oKDYm1zvPHi0B/tIwp25\nbif7Dv96o7o60eHOMJEYGtWLIS4mgroxEdSNDj/62resdlS49WzXQKo6H5hfwWqbgCY+x42dMl8Z\nwCTn30QicLGIFJdTd6uIpKhqnoikANuOE/M4vOtbk5GRUWaiXRnWbj/A1EWbmZqdx4qt+xCBHmkJ\nDB/cnovaNySxdlSgPtoYY37FkmhjXBQbFU7rhnVo3bBOmef3HCxyeq9/PVRk7fYD/LBmx/8k2ccK\nE5yE+khy7STb0T4Jd3S495pjrqsbHUF0hK1WUINkAi1FJB1vAjwUGOZ7gaqmH3ktIuOBqar6sYiE\nn6DuFGA4MMZ5/iTA9/E/Nuw4yNTFm5m6KI+leXsB6J4Wz6OXtOPiDik0sHXdjTEusCTamGosrlYE\ncbXiaJ8aV+b5klJl/+Fi9h4uYs+hIvYeKvJ5Xex9/tW5Yrbu3c/eQ96yguLSE35+VHjY/yTbcTER\n1Ir0EOkJIzI8jAjnOTI87GiZv+eiwo+5zhNGhEes9/wkqGqxiNwOzMS7TN1rqpojIqOd8y9WtK5z\negwwWURGAuuBqwJ5H0fk7jrI9MV5TM3OIzt3DwBdmtbjkYFtGdgxhZS4mKoIwxhjjsuSaGNqME+Y\nOIl2xK++i/fX4aIS9h0uO9ne67w+cm7voWJ27C9kTf4BDhWVUFhc6n2UlFJSWrnf3kf6JN8RHvFJ\nsMOIivAQFR529BEZHkZUuDepj4o4puzodb+8LqvsyHFURNjR94n0hNW45QpVdTow/ZiyMpNnVb2x\nvLpO+Q7g3MqL8vjy9hxi+uItTM3ezIINuwHo2DiOhy5qw8COKTSOL3vugTHGuMGVJNrfDQGMMYEV\nHeEhOsJDUp1TG0daUqoUlZRSUFxKUUnp0QT7SFlhyS/Hvsn3keci3+PiUgpL1HkuoahYj5775b1K\n2F9QzM4D3rKC4pJfzheXcriohMrI6z1h8qvkfNz1GXRqUu/U39j8j9vfns/UbO8iIO1S6nLfha0Z\n1DGFZvVjXY7MGGPKVuVJtL8bAhhjag5PmOAJ81SrMdTFJd6Eu6Dol+eC4hIn6T6SlP+SfJdd9uvk\nvJ6tjhIwnZvUo1VyHQZ1TKH5cSbiGmNMdeJGT7S/GwIYY8xJC3eGY9SyVc5qhJv7NXc7BGOMqRA3\nBvyVtah/6rEXicgoEckSkaz8/PwqC84YY4wxxpjyVNtZM6o6TlUzVDUjKSnJ7XCMMcYYY4w5yo0k\n2p8NAYwxxhhjjKm2RDVgG0uV/YHeRf1X4l0yaRPeDQKG+axJWladfLzrk1ZUIrD9ZOKswULxniE0\n7zsU7xlq3n03U9WQ+jrN2uwKC8X7tnsOHTXxvv1qt6t8YmE5i/ofr85J/QISkSxVzTiZujVVKN4z\nhOZ9h+I9Q+jed01ibXbFhOJ92z2HjmC+b1fWiT7eov7GGGOMMcbUBNV2YqExxhhjjDHVVbAn0ePc\nDsAFoXjPEJr3HYr3DKF736EgVP/fhuJ92z2HjqC97yqfWGiMMcYYY0xNF+w90cYYY4wxxlS6oEyi\nRWSAiKwQkdUi8qDb8VQFEWkiIl+LyFIRyRGRu9yOqaqIiEdEFojIVLdjqSoiUk9E3heR5SKyTER6\nux1ToInIPc6/7SUi8o6IRLsdk6kc1mZbmx3srM0OzjY76JJoEfEAY4GLgHbANSLSzt2oqkQxcK+q\ntgN6AbeFyH0D3AUsczuIKvYMMENV2wCdCPL7F5FU4E4gQ1Xb410ec6i7UZnKYG22tdkhwtrsIGyz\ngy6JBnoAq1V1jaoWApOAy1yOKeBUNU9V5zuv9+H9AU11N6rAE5HGwEDgFbdjqSoiEgecCbwKoKqF\nqrrb3aiqRDgQ42zYVAvY7HI8pnJYm21tdlCzNjt42+xgTKJTgY0+x7mEQMPkS0TSgC7AXHcjqRJP\nA/cDpW4HUoXSgXzgdecr0VdEJNbtoAJJVTcB/wE2AHnAHlX9zN2oTCWxNtva7GBnbXaQttnBmESH\nNBGpDXwA3K2qe92OJ5BEZBCwTVXnuR1LFQsHugIvqGoX4AAQ1ONIRSQeb+9kOtAIiBWR69yNyphT\nZ212SLA2O0jb7GBMojcBTXyOGztlQU9EIvA2xhNV9UO346kCfYFLRWQd3q+A+4vIW+6GVCVygVxV\nPdJr9T7eBjqYnQesVdV8VS0CPgT6uByTqRzWZlubHeyszQ7SNjsYk+hMoKWIpItIJN6B7FNcjing\nRETwjrdapqpPuh1PVVDVh1S1saqm4f3//JWqBt1fusdS1S3ARhFp7RSdCyx1MaSqsAHoJSK1nH/r\n5xLkE3NCiLXZ1mYHNWuzg7fNDnc7gMqmqsUicjswE+9s0NdUNcflsKpCX+B6YLGILHTKHlbV6S7G\nZALnDmCik3SsAUa4HE9AqepcEXkfmI93VYMFBPEuWKHE2mxrs0OEtdlB2GbbjoXGGGOMMcZUUDAO\n5zDGGGOMMSagLIk2xhhjjDGmgiyJNsYYY4wxpoIsiTbGGGOMMaaCLIk2xhhjjDGmgiyJNjWOiMxx\nntNEZFglv/fDZX2WMcaYk2NttglWtsSdqbFE5GzgD6o6qAJ1wlW1+ATn96tq7cqIzxhjzC+szTbB\nxnqiTY0jIvudl2OAfiKyUETuERGPiPxbRDJFJFtEbnGuP1tEvhORKTi7RInIxyIyT0RyRGSUUzYG\niHHeb6LvZ4nXv0VkiYgsFpGrfd77GxF5X0SWi8hEZ3cmY4wxWJttglfQ7VhoQsqD+PRqOA3rHlXt\nLiJRwGwR+cy5tivQXlXXOsc3qepOEYkBMkXkA1V9UERuV9XOZXzWb4DOQCcg0akzyznXBTgd2AzM\nxrsT2feVf7vGGFOjWZttgor1RJtgcgFwg7OF7lygPtDSOfeTT2MMcKeILAJ+BJr4XHc8ZwDvqGqJ\nqm4FvgW6+7x3rqqWAguBtEq5G2OMCW7WZpsazXqiTTAR4A5VnfmrQu84vAPHHJ8H9FbVgyLyDRB9\nCp9b4PO6BPu5MsYYf1ibbWo064k2Ndk+oI7P8UzgdyISASAirUQktox6ccAupzFuA/TyOVd0pP4x\nvgOudsbwJQFnAj9Vyl0YY0xosDbbBBX768vUZNlAifMV33jgGbxfy813JorkA4PLqDcDGC0iy4AV\neL8ePGIckC0i81X1Wp/yj4DewCJAgftVdYvToBtjjCmftdkmqNgSd8YYY4wxxlSQDecwxhhjjDGm\ngiyJNsYYY4wxpoIsiTbGGGOMMaaCLIk2xhhjjDGmgiyJNsYYY4wxpoIsiTbGGGOMMaaCLIk2xhhj\njDGmgiyJNsYYY4wxpoL+PxO8kiQiRLGsAAAAAElFTkSuQmCC\n", "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "reload(bilstm);\n", "bilstm.plot_results(losses, accuracies)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- Try and Tune the hyperparameters such that you improve the accuracy on the dev set.\n", "You can try changing the number of LSTM layers, the hidden dimension units, the FC layer dimensions, using pretrained embeddings etc.." ] }, { "cell_type": "code", "execution_count": 159, "metadata": { "collapsed": true }, "outputs": [], "source": [ "reload(bilstm);\n", "torch.manual_seed(765);\n", "embedding_dim=30\n", "hidden_dim=30\n", "model = bilstm.BiLSTM_CRF(len(word_to_ix),tag_to_ix,embedding_dim, hidden_dim)" ] }, { "cell_type": "code", "execution_count": 160, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Epoch 1: Dev Accuracy: 0.29213483146067415\n", "Epoch 2: Dev Accuracy: 0.4044943820224719\n", "Epoch 3: Dev Accuracy: 0.4157303370786517\n", "Epoch 4: Dev Accuracy: 0.42134831460674155\n", "Epoch 5: Dev Accuracy: 0.4606741573033708\n" ] } ], "source": [ "# sending loss function so that we can calculate the loss for the BiLSTM-CRF\n", "loss = model.neg_log_likelihood\n", "model, losses, accuracies = bilstm.train_model(loss, model, X_tr,Y_tr, word_to_ix, tag_to_ix,\n", " X_dv, Y_dv, \n", " num_its=5, status_frequency=1,\n", " optim_args = {'lr':0.2,'momentum':0}, param_file = 'best.params')" ] }, { "cell_type": "code", "execution_count": 161, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAtEAAACdCAYAAAByzCQ5AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xd8leX9//HXJ5uEMJJAGFksQRBkBAJutI6fC0UrKmpr\nVUprW7tstct+ba20/X7r6LKI1oVbQeuuEwd7yFYBmSIjIDJkJPn8/jgHPEKABHLOfc7J+/l45HHO\nvc555ya58uE6133d5u6IiIiIiEjdpQQdQEREREQk0aiIFhERERGpJxXRIiIiIiL1pCJaRERERKSe\nVESLiIiIiNSTimgRERERkXpSES0iIiIiUk8qokVERERE6klFtIiIiIhIPaUFHaAuCgoKvKysLOgY\nIiL1Nn369PXu3iroHLGkNltEElld2+2EKKLLysqYNm1a0DFEROrNzJYFnSHW1GaLSCKra7ut4Rwi\nIiIiIvWUtEW0uwcdQURERESSVFIW0TU1zpX3TeXBiUupqVExLSIiItIYfL59F0/PWMlV903l5Xmf\nRvW9EmJMdH1t3l5FVbXz62fm8fTMVdw6tCfd2jQLOpaIiIiINLDPt+/i1flreH72at7+aD07q2to\n1zyLrTuqovq+SVlEN89O58GrBjB+1ip+99wCzr7zHUac0JEfnNKFrPTUoOOJiIiIyGHYX+F8xaBS\nzuzVlj7FLTCzqGZIyiIawMw4v08RJx7Rmj+8sIB/vLmY5+es5g/n9+TYzgVBxxMRERGReoiHwjlS\n0hbRu+XlZPC/Xz+aoX3a84txcxg+ZjJD+7Tnl2cdSX7TzKDjiYhEnZmdAdwBpAJj3H3UfvbrD0wE\nLnb3J8PrlgKbgWqgyt3LYxJaRIT4K5wjJX0RvdsxnQt46Ycn8LfXF3HXW4t544O1/PKs7lzQt31g\nJ19EJNrMLBX4O3AqsBKYambPuvv8Wvb7I/BKLS8z2N3XRz2siAjxXThHajRFNEBWeio/Pb0r5/Zu\nx41Pz+GnT7zP0zNWcsv5PelQkBN0PBGRaBgALHL3JQBm9igwBJi/137fB54C+sc2nohI4hTOkRpV\nEb3bEYW5PPHtQTw8ZTl/fHEhp98+getO6cI1x3ckIy0pZ/0TkcarPbAiYnklUBG5g5m1B84HBrNv\nEe3Aq2ZWDfzL3UfX9iZmNgIYAVBSUtIwyUUkqSVi4RypURbRACkpxmUDSzm1eyH/8595/PnlD3hm\nVmg6vH6leUHHExGJpduBn7t7TS1/sI5z91Vm1hr4r5ktdPcJe+8ULq5HA5SXl2uCfhGp1cEK595F\nLUhJid/COVKjLaJ3K2yWxT+G9+PV+Wv4zTNzufCuiQyvKOFnZ3SjWVZ60PFERA7XKqA4YrkovC5S\nOfBouIAuAM40syp3H+/uqwDcfa2ZjSM0PGSfIlpEZH+SqXCO1OiL6N2+1r2QgZ3y+b9XPuD+95by\nyrw1/M+5PTjjqDZx/VGCiDQeZvZ/wL3uPq8eh00FuphZB0LF88XApZE7uHuHiPe4D3jO3cebWQ6Q\n4u6bw89PA24+zG9DRBqBZC2cI6mIjtA0M42bzunB+X3ac8NTc/jO2Bl87chCbh7Sg3YtmgQdT0Rk\nATDazNKAfwOPuPumAx3g7lVm9j3gZUJT3N3r7vPMbGR4+10HOLwQGBfuSEgDHnb3lxrg+xCRJNQY\nCudI5h7/Q9fKy8t92rRpMX3Pquoa7n33Y27770ekGPzktK5845gyUpPoH19Eos/Mpjf03Mpm1hW4\nErgEeBe4293faMj3OBxBtNkiEozdhfMLc1Yz4cMvC+cze7ZN2MK5ru22eqL3Iy01hREndOL/HdWW\nX42fy83PzeeZWav4w9Ce9GjXPOh4ItJIhedz7hb+Wg+8D/zYzL7t7hcHGk5EGoX9Fc7J2uO8Pyqi\nD6I4L5v7ruzPf2av5ub/zOPcv73L1cd14LqvdSE7Q6dPRGLHzG4DzgZeB/7g7lPCm/5oZh8El0xE\nkt3+CufLB5VyViMqnCOpCqwDM+Pco9txQpcCRr24kH9NWMLzc1bz+/OO4qSurYOOJyKNx2zgV+6+\ntZZtA2IdRkSSmwrnA1MRXQ8tsjMYdUEvzu/TnhvHzeGb/57KuUe349dnd6dVbmbQ8UQk+X1GRLtt\nZi2Ak8JT0R3wAkMRkbpQ4Vx3KqIPQUXHfF687nj++eZi/vHGYt76cB2/OLMbF5UXazo8EYmmm9x9\n3O4Fd//MzG4CxgeYSUQSnArnQ6Mi+hBlpqXyw68dwdm92vGLcXP4+VNzeGrGKv5wfk86t24adDwR\nSU4ptaxTOy4i9abC+fCp8T1MnVs35dFrBvLE9BXc8vwCzrzjbb47uBPfOakTmWmpQccTkeQyzcz+\nAvw9vHwtMD3APCKSQA5UOJ/Zsy19ilU410fUi+jwdEzTgFXufraZ5QGPAWXAUuAid98Y7RzRlJJi\nDOtfwsndCvndc/O5/dWP+M/7n3Dr0F4M6JAXdDwRSR7fB35NqA0F+C+hQlpEpFYqnKMnFj3R1xG6\ny1az8PINwGvuPsrMbggv/zwGOaKuVW4md17Sh6F92/Or8XO56F8Tubh/MTf+vyNpnp0edDwRSXDh\nWTluCDqHiMQ3Fc6xEdUi2syKgLOAW4Afh1cPAU4KP78feJMkKaJ3O6lra1750Qnc/upH3PPOx7y6\nYC2/Oac75/RqqwsPReSQmVkr4GdADyBr93p3PzmwUCISF1Q4x160e6JvJ9Tg50asK3T31eHnnwKF\ntR1oZiOAEQAlJSXRzBgV2Rlp/OLMIxnSux03Pj2HHzwyk6dnrOR3Q46iOC876HgikpjGEhrKcTYw\nEvgGsC7QRCLSYGpqnC92VbN1RxVbd4Yet+2sZuvOKrbt2L0+vC68bcuOKtZu3sGkxZUqnGMsakW0\nmZ0NrHX36WZ2Um37uLubme9n22hgNEB5eXmt+ySCHu2aM+67x3L/e0v531c+4LTbJvDjU4/gymPL\nSEut7UJ7EZH9ynf3e8zsOnd/C3jLzKYGHUqkMTpQwbt1R7joraXg3baziq07qvc8bo1Y3razus7v\nn5Zi5GSmkZORSrMm6SqcAxDNnuhjgXPN7ExCHzs2M7OHgDVm1tbdV5tZW2BtFDPEhdQU41vHdeD0\no9rwm/FzueWFBYyftYpRQ3vRs6h50PFEJHHsCj+uNrOzgE8AXb0schD1LXh37xdZ8G7Z8dXlQy14\nszPT9jxv1yKDnMxUsjPSaBp+zIl4zMkI7ZudkbrnsWlmGtkZaWSkqSMuaFErot39RuBGgHBP9E/d\n/TIz+zOhjyBHhR+fiVaGeNO+RRPGfKOcF+d+ym+fnceQv7/DN4/pwE9OO4KcTM02KCIH9Xszaw78\nBPgroQu2fxRsJJHo2lVdQ+WWnazbvIO1m7dTuXXnl0VwDAve9ip4ZS9BVG6jgMfN7CpgGXBRABkC\nY2ac2bMtx3Up4E8vLeTedz/m5XmfcvOQHpxyZK3Dw0VEdk8X2sXdnwM2AYMDjiRyyNydzTuqQoXx\n5ztYt2XHniJ53eYdX/nasG0nvp9BnSp4JUjm+/vJjCPl5eU+bdq0oGNExfRlG7jx6Tl8uGYLZ/Vs\ny03ndKd1s6yDHygiCcHMprt7eQO91hR3H9AQrxVNydxmy4Ht3WscevyyIF67efuegnn7rpp9js9I\nTaFVbuZXvlrvft40k9bNssjPySA3SwWvRE9d222NIQhYv9I8nvv+8YyesJg7X1/EhI/W8fMzunHp\ngBJdGCAie3vXzP5GaIaOrbtXuvuM4CJJsmuIXuMW2enhIjiTfiUtw8Vx1leL5NxMmjdJ11SwkjBU\nRMeBjLQUvndyF87q1Y5fPD2HX42fy7iZq7h1aE+OKMw9+AuISGPRO/x4c8Q6BzRPtNRbQ/YaF+dl\n07e05T69xq1yMylomkFmWmoA36FIdKmIjiMdCnJ4+JoKnpqxiluen89Zd77NyBM7ce3gzmSlqwES\naezcXeOg5YDUaywSOyqi44yZcWG/IgZ3bcUtzy/gr68v4rnZq7nl/KM4plNB0PFEJEBm9pva1rv7\nzbWtl+TRUL3GBeo1FmkwKqLjVH7TTP4yrDdD+xbxy/FzuPTuyVzYr4hfnnkkLXMygo4nIsHYGvE8\ni9CdCxcc7CAzOwO4A0gFxrj7qP3s1x+YCFzs7k/W51hpeI9NXc6/312qXmOROKUiOs4d16WAl394\nAne+9hGjJyzh9YVr+fXZR3Je7/ZqEEUaGXf/v8hlM/tf4OUDHROeGu/vwKnASmCqmT3r7vNr2e+P\nwCv1PVYa3l1vLWbUiws5uqg5px/VRr3GInFIRXQCyEpP5WdndOPc3u248ek5/Oix93l6xip+f95R\nlObnBB1PRIKTDRQdZJ8BwCJ3XwJgZo8CQ4C9C+HvA08B/Q/hWGkg7s5t//2QO19fxDlHt+MvFx1N\neqqmcROJR/rNTCDd2jTjyZHH8LshPZi5/DNOu20C/3hzEbuq9x3/JiLJx8zmmNns8Nc84APg9oMc\n1h5YEbG8Mrwu8nXbA+cD/6zvsdJw3J3fP7+AO19fxLDyYm4f1lsFtEgcU090gklNMS4fVMap3dvw\n22fn8aeXPuDZWZ9w69Ce9ClpGXQ8EYmusyOeVwFr3L2qAV73duDn7l5zqMPEzGwEMAKgpKSkASI1\nLtU1zq/Gz+WRKcv55jFl/Obs7rpXgEic039xE1Sb5lncdXk/Rl/ej8+27WLoP9/jpmfmsnn7rqCj\niUj0tAU2uPsyd18FNDGzioMcswoojlguCq+LVA48amZLgQuBf5jZeXU8FgB3H+3u5e5e3qpVqzp/\nQwJV1TX85PFZPDJlOdcO7sRN56iAFkkEKqIT3Gk92vDqT07kG4PKeGDSMk79ywRenvdp0LFEJDr+\nCWyJWN7KvkMw9jYV6GJmHcwsA7gYeDZyB3fv4O5l7l4GPAl8193H1+VYOTw7qqq59uEZjJ/1Cdef\n3pXrT++mi8ZFEoSK6CTQNDON357bg3HfPZaWORl8+8HpjHhgGqs3fRF0NBFpWOb+5URn7l7DQYbl\nhYd7fI/QLB4LgMfdfZ6ZjTSzkYdy7GF+DxL2xc5qrnlgOi/PW8Nvz+nOtYM7Bx1JROrBvLaJJ+NM\neXm5T5s2LegYCWFXdQ33vPMxt7/6IWkpKVx/elcuG1hKqj4aFAmEmU139/IGeq2ngTf5svf5u8Bg\ndz+vIV6/oajNPrjN23dx1f3TmLZ0A6OG9uKi/sUHP0hEYqKu7bZ6opNMemoKI0/sxCs/PJE+JS24\n6dl5XPDP91iw+vOgo4nI4RsJHENoXPJKoILwxXySOD7btpPLxkxmxrKN3HFxHxXQIglKRXSSKsnP\n5oFvDeD2Yb1ZsWEb5/z1HUa9uJAvdlYHHU1EDpG7r3X3i929tbsXuvul7r426FxSd+s27+Di0ZNY\nsHoz/7ysH+cc3S7oSCJyiOpURJvZdWbWzELuMbMZZnZatMPJ4TEzzuvTnld/fCJD+7bnrrcWc+wf\nX+fWFxawdP3Wg7+AiMQVM7vfzFpELLc0s3uDzCR1t3rTFwz710SWVW7j3m/259TuhUFHEpHDUNee\n6G+5++fAaUBL4HJgVNRSSYNqmZPBny48midGDqKiQx5j3vmYk/73TS6/ZzIvzV2tm7WIJI5e7v7Z\n7gV33wj0CTCP1NGyyq18/a6JrNu8gwevGsBxXQqCjiQih6muN1vZfVXamcCD4Su7daVagulflkf/\nsjzWfL6dx6eu4JEpyxn50Axa52Zycf9ihg0ooX2LJkHHFJH9SzGzluHiGTPLQzfNinuL1m5m+JjJ\n7Kiq4eFrBtKzqHnQkUSkAdS18Z1uZq8AHYAbzSwXUPdlgipslsX3T+nCdwd35s0P1jJ28nL++sYi\n/vbGIk7u1prhFaWccEQrzeghEn/+D5hoZk8Q6ty4ELgl2EhyIPM+2cTl90whNcV4bMQgurbJDTqS\niDSQuhbRVwG9gSXuvi3c+3Fl9GJJLKSmGKccWcgpRxaycuM2Hp2ygkenruDVBVNp36IJl1aU8PXy\nIlrnZgUdVUQAd3/AzKYDg8Orhrr7/CAzyf7NWL6Rb947haaZaYy9ZiAdCnKCjiQiDahO80Sb2bHA\nLHffamaXAX2BO9x9WbQDguYcjaVd1TX8d/4axk5exruLKklLMU7v0YbhFSUM6pSvO2mJ1FNDzhMd\n8ZqtgT3/u3X35Q35+odLbTa8t3g9V98/jda5mYy9ZqCGyokkkLq223Xtif4ncLSZHQ38BBgDPACc\neOgRJR6lp6ZwZs+2nNmzLUvWbeGRKct5YvpKnp+zmo4FOVwyoIQL+xXRMicj6KgijY6ZnUtoSEc7\nYC1QSuhOgj2CzCVf9cbCtYx8aDql+dk8dFUFrZvp0zyRZFTX2TmqwreaHQL8zd3/DmhgV5Lr2Kop\nvzyrO5NuPIXbhh1NXk4Gt7ywgIpbX+NHj81i2tINJMIdL0WSyO+AgcCH7t4BOAWYFGwkifTCnNWM\neHAaXQqb8uiIQSqgRZJYXXuiN5vZjYSmtjvezFKA9OjFkniSlZ7K+X2KOL9PEQs//ZyHJy9n3IxV\njJu5iq6FuQwfWMJ5fdrTLEs/EiJRtsvdK80sxcxS3P0NM7s96FAS8tT0lVz/5Pv0LWnJvVf2V5so\nkuTq2hM9DNhBaL7oT4Ei4M9RSyVxq1ubZtw85Cgm//IU/nhBTzLSUvjNM/OouOU1bnhqNnNWbgo6\nokgy+8zMmgITgLFmdgegOyfFgQcnLeMnT7zPMZ0KeOCqASqgRRqBOl1YCGBmhUD/8OKUWN5qVhep\nxLfZKz/j4cnLeWbWJ3yxq5peRc0ZXlHCOUe3IztDU9hK49aQFxaaWQ7wBaEOkOFAc2Csu1c2xOs3\nlMbWZv/rrcXc+uJCvnZka/52aV+y0lODjiQih6Gu7XZdZ+e4iFDP85uE5iY9Hrje3Z88zJx10tga\n5ET1+fZdjJ+5iocmLePDNVvIzUxjaN/2XFpRqrlRpdGKxuwc8a6xtNnuzm2vfsSdr33E2b3actuw\n3qSn1vUDXhGJVw09O8cvgf67e5/NrBXwKhCTIloSQ7OsdK4YVMblA0uZvmwjYycv55GpK7h/4jL6\nl7VkeEUpZxzVRr00IpLw3J1bnl/AmHc+5qLyIm4d2ks3qBJpZOpaRKfsNXyjkrqPp5ZGxswoL8uj\nvCyPX5/dnaemr2Ts5GX88LFZtPxPOl8vL+aSASW68YCIJKSaGudXz8zl4cnL+eYxZfzm7O6kqIAW\naXTqWkS/ZGYvA4+El4cBL0QnkiSTvJwMrjmhI1cd14GJSyoZO3kZ977zMaMnLOHYzvkMryjl1O6F\n+ghUpA7M7Dp3v+Ng6yR6qqpruP7J2YybuYrvntSJ60/vqptQiTRSdSqi3f16M7sAODa8arS7j4te\nLEk2KSnGsZ0LOLZzAWs/387j01bwyJQVfHfsDFrlZjKsvJiLBxRT1DI76Kgi8ewbwN4F8zdrWSdR\nsKOqmh88MpOX563h+tO7cu3gzkFHEpEA1Xl2jiA1lotUGpvqGuetD9cydtJyXv8gNFpocNfWDK8o\n4aSurTW+UJJCQ1xYaGaXAJcCxwFvR2zKBWrc/ZTDef2Gloxt9hc7qxn50HTe+nAdN53TnSuP7RB0\nJBGJkga5sNDMNgO1VdkGuLs3O8CxxYRuDV4Yfo3R7n6HmeUBjwFlwFLgInffeLCgknxSU4yTuxVy\ncrdCVm7cxmNTV/DY1BVcdf802jXP4pIBJQzrX6w7fonAe8BqoIDQbb932wzMDiRRI7JlRxVX3TeV\nKUs38KcLenFR/+KgI4lIHIhaT7SZtQXauvsMM8sFpgPnEfrocYO7jzKzG4CW7v7zA71WMvZqSO12\nVdfw2oI1jJ28nLc/Wk9ainFq90KGV5RyTKd8XbwjCaehp7gzs1Kgi7u/amZNgDR339xQr98QkqnN\n/mzbTr7x76nMXbWJ24b15tyj2wUdSUSirKGnuKs3d19NqOcEd99sZguA9sAQ4KTwbvcTmnv6gEW0\nNB7pqSmccVRbzjiqLUvXb+WRKct5fNoKXpz7KWX52VxaUcKF/YrJy8kIOqpIzJnZNcAIIA/oROju\nsXcBBxzOYWZnEBo3nQqMcfdRe20fAvwOqAGqgB+6+zvhbUsJ9XhXA1WNac7r9Vt2cNmYySxZt5V/\nDu/LaT3aBB1JROJITMZEm1kZodvUHgUsd/cW4fUGbNy9vNcxIwj9saCkpKTfsmXLop5T4tP2XdW8\nPO9Txk5azpSlG8hITeHMnm24tKKU/mUtdWW8xLUGvmPhLGAAMNnd+4TXzXH3ngc4JhX4EDgVWAlM\nBS5x9/kR+zQFtrq7m1kv4HF37xbethQod/f1dc2ZDD3Rqzd9wfAxk1n92XZGX9GP47u0CjqSiMRI\n4D3REUGaAk8R6tn4PLLgCTfYtVbx7j4aGA2hBjnaOSV+ZaWnMqR3e4b0bs+Hazbz8OTlPDVjJeNn\nfUKX1k0ZXlHC+X2LaN4kPeioItG2w9137m5HzSyN2q9biTQAWOTuS8LHPEroE8E9RbS7b4nYP6cO\nr5nUlldu49Ixk9i0bRcPXDWA/mV5QUcSkTgU1cl5zSydUAE91t2fDq9eEx4vvXvc9Nr9HS+ytyMK\nc/ntuT2Y/ItT+NMFvcjOSOW3/5lPxR9e5WdPvs/7Kz4jEWacETlEb5nZL4AmZnYq8ATwn4Mc0x5Y\nEbG8MrzuK8zsfDNbCDwPfCtikwOvmtn08CeEtTKzEWY2zcymrVu3ro7fTvxZtHYzX//Xe2zZUcXY\naypUQIvIfkWtJzo8VOMeYIG7/yVi07OE5jodFX58JloZJHllZ6RxUf9iLupfzJyVm3h4yjLGz/yE\nx6et5Kj2zRheUcq5R7cjJzPqH7aIxNINwFXAHODbhG56NaYhXjg89/84MzuB0Pjor4U3Hefuq8ys\nNfBfM1vo7hNqOT7hPz2c98kmrrhnCmbGYyMG0bVNbtCRRCSORbMn+ljgcuBkM5sV/jqTUPF8qpl9\nRKiRHnWgFxE5mJ5Fzbl1aC8m//IUfjekB1XVzo1Pz6HiD6/x6/FzWbD686AjijQId69x97vd/euE\nrhmZ7Af/6GUVEDknW1F43f7eYwLQ0cwKwsurwo9rgXGEhocknRnLN3LJ6ElkpqXwxEgV0CJycNGc\nneMdQvNJ1yaubgwgyaFZVjqXDyrjsoGlzFi+kbGTlvPYtBU8OGkZ/UpbMryihDN7tiUrPTXoqCKH\nxMzeBM4l1HZPB9aa2Xvu/qMDHDYV6GJmHQgVzxcTunFL5Ot2BhaHr1PpC2QClWaWA6SEZ1jKAU4D\nbm7o7ytoExdXctX9U2mdm8lDV1fozqkiUif6rFuSjpnRrzSPfqV5/Prs7jw1YyUPT17Ojx9/n5uf\nm8+FfYu4pKKETq2aBh1VpL6ahy/Qvhp4wN1vMrMD3mzF3avM7HvAy4SmuLvX3eeZ2cjw9ruAC4Ar\nzGwX8AUwLFxQFxIa4gGhvxcPu/tL0fv2Yu+NhWsZ+dB0SvKyGXt1hW7uJCJ1ptt+S6Pg7kxcUsnY\nyct5ee6nVNU4x3TKZ3hFKad2LyQjLarX2Eoj1sBT3M0h1Bt8P/BLd59qZrPdvVdDvH5DSZQ2+8U5\nq/nBozPp2iaXB75VofnnRQSIoynuROKBmXFMpwKO6VTA2s3beWJaqHf62odnkJuZxoAOeQzqlM/A\njvl0b9tMd0aUeHUzoR7ld8IFdEfgo4AzJaSnpq/k+iffp09JS/59ZX+aZWmKTBGpHxXR0ui0zs3i\n2sGdGXliJyZ8tI5X5n3KpCUbeG1haLbF5k3SqeiQx8CO+QzqlE/XwlwV1RIX3P0JQtPa7V5eQmgo\nhtTDQ5OW8avxczm2cz53X1FOdob+FIpI/anlkEYrNcUY3LU1g7u2BkJ3KJu0pJKJiyuZtGQDr8xf\nA0DL7HQGdszfU1R3ad1Ud0kUSVCjJyzmDy8s5JRurfn78L660FhEDpmKaJGwts2bcH6fIs7vUwTA\nyo3bmLRkQ7ioruTFuZ8CUNA0g4rdRXXHfDq1ylFRLRLn3J3bX/2IO177iLN6teX2Yb1JT9W1ECJy\n6FREi+xHUctsLuyXzYX9inB3Vm78gomLK5kY7q1+fvZqAFrnZn6lp7osP1tFtUSFmaW6e3XQORKN\nu/OHFxZw99sf8/V+RYy6oBepGqIlIodJRbRIHZgZxXnZFOdlc1H/YtydZZXb9hTUE5dU8uz7nwDQ\npllW+CLFPAZ1LKA4r4mKamkoH5vZS8BjwOt1uNFKo1dT4/z6mbmMnbycbwwq5aZzeugaBxFpECqi\nRQ6BmVFWkENZQQ6XDCjB3Vmyfuuegvrtj9YxbmbopnDtWzQJ91SHZgDRjRzkMHQDzgauBe4xs+eA\nR8M3t5K9VFXX8LMnZ/P0zFV856RO/Oz0rvoPrYg0GBXRIg3AzOjUqimdWjXlsoGluDuL1m7Z01P9\n+sI1PDVjJQDFeU0YFDH8o23zJgGnl0Th7tuAx4HHzawlcAfwFqGbqEiEHVXVXPfILF6a9ynXn96V\nawd3DjqSiCQZFdEiUWBmdCnMpUthLlcMKqOmxvlw7eZQT/XiSl6et4bHp4WK6rL87D1zVA/qmK87\npskBmdmJwDDgDGAacFGwieLP9l3VfPvB6bz14Tp+c3Z3vnVch6AjiUgSUhEtEgMpKUa3Ns3o1qYZ\nVx7bgZoaZ8Gnn++Z+eO52at5ZMoKADq2ytnTUz2wYz6tcjMDTi/xwsyWAjMJ9UZf7+5bg00Uf7bs\nqOKq+6YyZekGRg3tycUDSoKOJCJJSkW0SABSUowe7ZrTo11zrj6+I9U1zvxPPmfikvVMXFzJM7M+\nYezk5QB0ad10T0/1wI75ujVx49bL3T8POkS82rRtF9/49xTmrNrE7cN6M6R3+6AjiUgSUxEtEgdS\nU4yeRc3pWdScESd0oqq6hrmffL7nQsUnp6/kgYnLAOjWJndPQT2wYx4tslVUNyJtzGwcUOjuR5lZ\nL+Bcd/990MGCtn7LDi6/ZwqL127hn8P7clqPNkFHEpEkpyJaJA6lpabQu7gFvYtb8J2TOrGruobZ\nKzftuaMn9JfbAAAOeUlEQVTio1OXc997SzGDI9s023OR4oAOeTRvkh50fImeu4HrgX8BuPtsM3sY\naNRF9KebtnPpmEl88tkX3PPNco7v0iroSCLSCKiIFkkA6akp9CttSb/Sllw7uDM7qqqZvXLTngsV\nH5q8jHvf/ZgUgx7tmu+ZTq9/WR65WSqqk0i2u0/Za5q2qqDCxIMVG7Zx6ZhJbNy6iwe+VcGADnlB\nRxKRRkJFtEgCykxLpX9ZHv3L8vjBKV3YvquaWSs+2zP84/73lnH32x+TmmIc1b55+ELF0P45mfq1\nT2DrzawT4ABmdiGwOthIwVm0dguXjZnM9qpqxl5dwdHFLYKOJCKNiP6aiiSBrPTUPeOkf0Roiq8Z\nyzbumaf6nneWcNdbi0lLMXoVNd9zoWJ5aR5NMjTFcAK5FhgNdDOzVcDHwGXBRgrGvE82ccU9UzAz\nHh0xkG5tmgUdSUQaGRXRIkkoKz2VYzoXcEznAgC27axi+rKNe3qq73prCX9/YzHpqUbv4hZ7ptTr\nW9qSrHQV1fHK3ZcAXzOzHCDF3TcHnSkIM5dv5Bv3TqFpZhoPXV1Bx1ZNg44kIo2QimiRRiA7I43j\nu7Tac8HVlh1VTFu6gYlLKpm0uJK/vbGIO19fREZaCn2KW9C/LI+OrXIozc+mND+H/JwM3S45QGb2\n4/2sB8Dd/xLTQAGauLiSq++fSkFuJmOvrqCoZXbQkUSkkVIRLdIINc1M46SurTmpa2sANm/fxdSl\nG/b0VP/jzUXU+Ff3L8nLpqwgVFSX5oUeywqyKczNIiVFBXaU5YYfuwL9gWfDy+cAUwJJFIA3PljL\nyAenU5KXzUNXV1Cou3uKSIBURIsIuVnpnNytkJO7FQKwo6qalRu/YFnlVpZVbgt/bWXh6s38d/4a\ndlV/WWFnpqVQkpe9p9e6LD+bkvBj+xZNSEtNCerbShru/j8AZjYB6Lt7GIeZ/RZ4PsBoMfPinNX8\n4NGZHFGYy4NXVeimQyISOBXRIrKPzLRUOrVqSqdaxppW1ziffPYFyyq3sbRyK8s3bGPp+tDjO4vW\ns31XzZ5901KM9i2bRPReZ1OWHxomUpyXrfHX9VcI7IxY3hled0BmdgZwB5AKjHH3UXttHwL8Dqgh\nNGXeD939nbocGwtPz1jJT594nz4lLbn3m/01F7qIxAUV0SJSL6kpRnFeqAg+rkvBV7a5O2s379hT\nYEf2ZM9cvpHN27+c0tgM2jbLomRPYb17DHaoR7uppuKrzQPAlPBdCwHOA+470AFmlgr8HTgVWAlM\nNbNn3X1+xG6vAc+6u4fvgvg4oRlA6nJsVI2dvIxfjZ/LoI753H1FuaZoFJG4odZIRBqMmVHYLIvC\nZln73PTC3fls266I3uvQEJFlG7bx6oI1rN+y8yv7FzTN2Gf8dUleqOBukZ3eKC90dPdbzOxF4Pjw\nqivdfeZBDhsALArP7IGZPQoMAfYUwu6+JWL/HMLzUNfl2Gi6e8ISbnlhASd3a80/hvfVJxciEldU\nRItITJgZLXMyaJmTQZ+Slvts37Kjap8x2EsrtzJpSSVPz1z1lX2bZaXt03O9e5hI69zMpC6w3X0G\nMKMeh7QHVkQsrwQq9t7JzM4HbgVaA2fV59jw8SOAEQAlJSX1iLcvd+eO1z7i9lc/4qyebbltWG8y\n0jS2XkTii4poEYkLTTPT6NGuOT3aNd9n2/Zd1azYsO2r47ArtzFn1SZenPsp1RFTiTRJT6U0P9xr\nXRAutPNCj+1aNCFVM4nUyt3HAePM7ARC46O/Vs/jRxO6EQzl5eV+kN0P9Drc+uJCRk9YwoX9ivjj\nBb30byYicUlFtIjEvaz0VLoU5tKlMHefbbuqa/jksy9YWrmN5ZVbWRruxf54/Vbe/HAdO6u+vNAx\nPdUobvll7/XuCx1L8rMpbpmdrL2dq4DiiOWi8LpaufsEM+toZgX1PfZw1dQ4v35mLmMnL+eKQaX8\n9pwemj5RROKWimgRSWjpqSnhgjgHaPWVbTU1zprN278y/npZ5VaWrt/G1KUb2bLjywsdUwzatWjy\nZYG911js7IyEbS6nAl3MrAOhAvhi4NLIHcysM7A4fGFhXyATqAQ+O9ixDaWquoafPTmbp2euYuSJ\nnfj5GV2TeliOiCS+hP2rICJyMCkpRtvmTWjbvAmDOuV/ZZu7U7l1Z8T46y97sl+cs5qN23Z9Zf/W\nuZncdXk/+tYynjueuXuVmX0PeJnQNHX3uvs8MxsZ3n4XcAFwhZntAr4Ahrm7A7UeG4WMXPfoLJ6f\ns5qfnnYE1w7urAJaROKeimgRaZTMjIKmmRQ0zaRf6b6F8aYvdrG8chvLNoQudly6fmvC3iHP3V8A\nXthr3V0Rz/8I/LGuxzY0M2Ngp3z6lrbkquM6RPOtREQajIpoEZFaNG+STs+i5vQs2vdCR2l4lw8s\nDTqCiEi9JOVVNCIiIiIi0aQiWkRERESknlREi4iIiIjUk4UuwI5vZrYOWHYIhxYA6xs4zqGIlxyg\nLLWJlxwQP1niJQfET5ZDzVHq7q0OvlvySII2G+InS7zkgPjJEi85QFlqEy85IMrtdkIU0YfKzKa5\ne7lyfElZ4jcHxE+WeMkB8ZMlXnIks3g6x/GSJV5yQPxkiZccoCzxnAOin0XDOURERERE6klFtIiI\niIhIPSV7ET066ABh8ZIDlKU28ZID4idLvOSA+MkSLzmSWTyd43jJEi85IH6yxEsOUJbaxEsOiHKW\npB4TLSIiIiISDcneEy0iIiIi0uASvog2szPM7AMzW2RmN9Sy3czszvD22WbWN8AsJ5nZJjObFf76\nTZRy3Gtma81s7n62x/KcHCxLrM5JsZm9YWbzzWyemV1Xyz5RPy91zBGrc5JlZlPM7P1wlv+pZZ+Y\n/KzUMUtMzkv4vVLNbKaZPVfLtpj9/iSreGm31WYfUpZG1WbXI0vUz4va7APmCabNdveE/QJSgcVA\nRyADeB/ovtc+ZwIvAgYMBCYHmOUk4LkYnJcTgL7A3P1sj8k5qWOWWJ2TtkDf8PNc4MMgflbqmCNW\n58SApuHn6cBkYGAQPyt1zBKT8xJ+rx8DD9f2frH8/UnGr3hpt9VmH3KWRtVm1yNL1M+L2uwD5gmk\nzU70nugBwCJ3X+LuO4FHgSF77TMEeMBDJgEtzKxtQFliwt0nABsOsEuszkldssSEu6929xnh55uB\nBUD7vXaL+nmpY46YCH+fW8KL6eGvvS+SiMnPSh2zxISZFQFnAWP2s0vMfn+SVLy022qzDy1LTMRL\nm12PLFGnNrt2QbbZiV5EtwdWRCyvZN8f7LrsE6ssAMeEP0540cx6RCFHXcTqnNRVTM+JmZUBfQj9\nzzlSTM/LAXJAjM5J+COwWcBa4L/uHtg5qUMWiM15uR34GVCzn+3x9vuTaOKl3VabfegaZZt9kCwQ\ng/OiNrtWgbXZiV5EJ5oZQIm79wL+CowPOE88iOk5MbOmwFPAD93982i+12HkiNk5cfdqd+8NFAED\nzOyoaL1XA2SJ+nkxs7OBte4+vaFfWxKS2ux9Nco2uw5ZYnJe1GZ/VdBtdqIX0auA4ojlovC6+u4T\nkyzu/vnujz/c/QUg3cwKopDlYGJ1Tg4qlufEzNIJNYBj3f3pWnaJyXk5WI4gfk7c/TPgDeCMvTbF\n/Gdlf1lidF6OBc41s6WEPt4/2cwe2mufuPn9SVDx0m6rzT4EjbHNrkuWWP+sqM3eI9A2O9GL6KlA\nFzPrYGYZwMXAs3vt8yxwRfjqzIHAJndfHUQWM2tjZhZ+PoDQ+a+MQpaDidU5OahYnZPwe9wDLHD3\nv+xnt6ifl7rkiOE5aWVmLcLPmwCnAgv32i0mPyt1yRKL8+LuN7p7kbuXEfodft3dL9trt7j5/UlQ\n8dJuq80+BI2tza5rllicF7XZ+wq6zU5riBcJirtXmdn3gJcJXWl9r7vPM7OR4e13AS8QujJzEbAN\nuDLALBcC3zGzKuAL4GJ3b/CB+Gb2CKGrYgvMbCVwE6FB/zE9J3XMEpNzQuh/q5cDcyw0hgvgF0BJ\nRJZYnJe65IjVOWkL3G9mqYQat8fd/bkgfn/qmCVW52UfAZ2TpBQv7bba7EPO0tja7LpmicV5UZtd\nR7E6J7pjoYiIiIhIPSX6cA4RERERkZhTES0iIiIiUk8qokVERERE6klFtIiIiIhIPamIFhERERGp\nJxXRknDM7L3wY5mZXdrAr/2L2t5LREQOjdpsSVaa4k4SlpmdBPzU3c+uxzFp7l51gO1b3L1pQ+QT\nEZEvqc2WZKOeaEk4ZrYl/HQUcLyZzTKzH5lZqpn92cymmtlsM/t2eP+TzOxtM3sWmB9eN97MppvZ\nPDMbEV43CmgSfr2xke8VvtPRn81srpnNMbNhEa/9ppk9aWYLzWzs7js0iYiI2mxJXgl9x0Jp9G4g\nolcj3LBucvf+ZpYJvGtmr4T37Qsc5e4fh5e/5e4bLHS70qlm9pS732Bm33P33rW811CgN3A0UBA+\nZkJ4Wx+gB/AJ8C6hu1u90/DfrohIQlObLUlFPdGSTE4DrgjflnUykA90CW+bEtEYA/zAzN4HJgHF\nEfvtz3HAI+5e7e5rgLeA/hGvvdLda4BZQFmDfDciIslNbbYkNPVESzIx4Pvu/vJXVobG4W3da/lr\nwCB332ZmbwJZh/G+OyKeV6PfKxGRulCbLQlNPdGSyDYDuRHLLwPfMbN0ADM7wsxyajmuObAx3Bh3\nAwZGbNu1+/i9vA0MC4/hawWcAExpkO9CRKRxUJstSUX/+5JENhuoDn/Edx9wB6GP5WaELxRZB5xX\ny3EvASPNbAHwAaGPB3cbDcw2sxnuPjxi/ThgEPA+4MDP3P3TcIMuIiIHpzZbkoqmuBMRERERqScN\n5xARERERqScV0SIiIiIi9aQiWkRERESknlREi4iIiIjUk4poEREREZF6UhEtIiIiIlJPKqJFRERE\nROpJRbSIiIiISD39f78jwmT4RVv+AAAAAElFTkSuQmCC\n", "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "reload(bilstm);\n", "bilstm.plot_results(losses, accuracies)" ] }, { "cell_type": "code", "execution_count": 162, "metadata": { "collapsed": true }, "outputs": [], "source": [ "reload(tagger_base);\n", "reload(bilstm);\n", "confusion = tagger_base.eval_model(model,'bilstm_crf-dev-en.preds', word_to_ix)" ] }, { "cell_type": "code", "execution_count": 163, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "0.38961381310135756\n" ] } ], "source": [ "print (scorer.accuracy(confusion))" ] }, { "cell_type": "code", "execution_count": 164, "metadata": { "collapsed": true }, "outputs": [], "source": [ "tagger_base.apply_model(model,'bilstm_crf-te-en.preds',word_to_ix, all_tags, testfile=TEST_FILE_HIDDEN)" ] }, { "cell_type": "code", "execution_count": 165, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "0.39670103092783504\n" ] } ], "source": [ "# you don't have en-ud-test.conllu, so you can't run this\n", "te_confusion = scorer.get_confusion(TEST_FILE,'bilstm_crf-te-en.preds')\n", "print (scorer.accuracy(te_confusion))" ] }, { "cell_type": "markdown", "metadata": { "collapsed": true }, "source": [ "### Part of Speech Tagging for Norwegian Language.\n", "We will use the same `BiLSTM-CRF` part of speech tagger for Norwegian now." ] }, { "cell_type": "markdown", "metadata": { "collapsed": true }, "source": [ "- Running a `BiLSTM-CRF` model on the norwegian language. Make sure your model runs on the norwegian data. \n", "- You may want to use the pretrained embeddings for norwegian to improve the performance of the model." ] }, { "cell_type": "code", "execution_count": 166, "metadata": { "collapsed": true }, "outputs": [], "source": [ "if START_TAG not in all_tags_nr:\n", " all_tags_nr.append(START_TAG)\n", "if END_TAG not in all_tags_nr:\n", " all_tags_nr.append(END_TAG)\n", "all_tags_nr = sorted(all_tags_nr)\n", "if START_TAG not in tag_to_ix_nr:\n", " tag_to_ix_nr[START_TAG]=len(tag_to_ix_nr)\n", "if END_TAG not in tag_to_ix_nr:\n", " tag_to_ix_nr[END_TAG]=len(tag_to_ix_nr)\n", "# we will be using this tag_to_ix and ix_to_tag from now on.." ] }, { "cell_type": "code", "execution_count": 167, "metadata": { "collapsed": true }, "outputs": [], "source": [ "reload(bilstm);\n", "torch.manual_seed(765);\n", "embedding_dim=30\n", "hidden_dim=30\n", "model = bilstm.BiLSTM_CRF(len(word_to_ix_nr),tag_to_ix_nr,embedding_dim, hidden_dim)" ] }, { "cell_type": "code", "execution_count": 168, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Epoch 1: Dev Accuracy: 0.34065934065934067\n", "Epoch 3: Dev Accuracy: 0.37362637362637363\n", "Epoch 5: Dev Accuracy: 0.3626373626373626\n", "Epoch 7: Dev Accuracy: 0.3626373626373626\n", "Epoch 9: Dev Accuracy: 0.4010989010989011\n" ] } ], "source": [ "# sending loss function so that we can calculate the loss for the BiLSTM-CRF\n", "loss = model.neg_log_likelihood\n", "model, losses, accuracies = bilstm.train_model(loss, model, X_tr_nr, Y_tr_nr, word_to_ix_nr, tag_to_ix_nr,\n", " X_dv_nr, Y_dv_nr, \n", " num_its=10, status_frequency=2,\n", " optim_args = {'lr':0.1,'momentum':0}, param_file = 'best.params')" ] }, { "cell_type": "code", "execution_count": 169, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAtEAAACdCAYAAAByzCQ5AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xd8VVW2wPHfyk1vQCAQeuihSJFQFQH7WAYsA+KIoIJi\nG9vYpvicGecN4ziWNyqIyAhYULCh49ilKAgkCEjvnUCoKaRnvT/uSYwYIAlJTu696/v55JN7zj37\nnHWM7Kzsu89eoqoYY4wxxhhjKi7I7QCMMcYYY4zxNZZEG2OMMcYYU0mWRBtjjDHGGFNJlkQbY4wx\nxhhTSZZEG2OMMcYYU0mWRBtjjDHGGFNJlkQbY4wxxhhTSZZEG2OMMcYYU0mWRBtjjDHGGFNJwW4H\nUBGNGjXSxMREt8MwxphKS01NPaiq8W7HUZuszzbG+LKK9ts1nkSLiAdIAfao6hUiEge8BSQC24ER\nqnrkVOdITEwkJSWlpkM1xphqJyI73I6htlmfbYzxZRXtt2tjOsc9wLoy248AX6pqB+BLZ9sYY4wx\nxhifUaNJtIi0AC4HppbZPQyY7ryeDgyviWurak2c1hhjjDHGmBqfzvEs8BAQU2ZfE1Xd57xOA5pU\n90Wz8gq5+43lXNu7JZd3b1rdpzfGGGOM8QnzN6bz5pKdtG8cTVLTGJISYklsGEmwx7/WligoKmbb\nwWzW7ctgQ1om69MyubZ3Cy47q+bywBpLokXkCuCAqqaKyJDyjlFVFZFyh4xF5FbgVoBWrVpV6toh\nHiEjt5AHZq+gZVwE3VvUr1zwxhhjjDE+bn9GLvfM+p6iIuXzdfspKvamXGHBQXRoEk1SQixJCd7E\nOqlpDI2iw1yO+PRUlfTMPNanZbI+LYP1+zJZl5bJlgNZ5BcVAxAcJLRvHE1uQVGNxlKTI9HnAL8U\nkcuAcCBWRF4D9otIU1XdJyJNgQPlNVbVKcAUgOTk5ErNzQgL9vDS6N4Mf+Fbxk1P4YO7zqFpvYgz\nuxtjjDHGGB9RXKz8dvZK8gqK+c9vzqVZ/Qi2pGexfp+TfKZlMn9jOnNSd5e2aRQdRuemMXRqEkNS\nU2+C3b5xNOEhHlfuISe/iE0HMp1E+ccR5sPZ+aXHJMSG0ykhhvM6NqJzQiydEmJoFx9NaHDNj7TX\nWBKtqo8CjwI4I9G/VdUbROQfwBhgovP9g5q4fqPoMF4Z04drJi1i3PQUZk8YQGSoT6zoZ4wxxhhz\nRmYs3s7CTQd5Yng32sZHA9C1WT26Nqv3k+MOZeWxIc07mrt+nze5nvndDvIKvaO6niChTaMoZ8T6\nx1Hr5vUjEJFqibW4WNl9JKdMouwdYd52KJuSR9wiQjx0TIjh4i5N6FQSR0IMDaJCqyWGqnAjq5wI\nvC0itwA7gBE1daFOCTH8a1Qvbpm+jHtnrWDyDb0JCqqeH7gxxtQ1InIp8BzgAaaq6sSTHNcHWAxc\np6pzTtW2KsuSGmPctWl/Jn/773rOT2rMr/udekpsw+gwBrYPY2D7RqX7ioqV7YeyWb8vkw1pGaxL\ny2Tl7qN8tGpf6TExYcHeZLbpjwltp4QYYsJDTnm9YzkFpYnyOmdUfGNaJtn53qkXItA6LpJOCTFc\n2aMZnZ3zt4qLrHM5nPjCKhbJycl6JmuOTvtmG3/+aC23D2nHw5cmVWNkxhhzaiKSqqrJtXAdD7AR\nuAjYDSwDRqnq2nKO+xzIBaap6pxTtRWRJ4HDqjpRRB4BGqjqw6eK5Uz7bGNM1eUXFjP8hW9Jy8jl\nk3sH0TgmvNrOnZVX+JORYu8IdgaZuYWlx7RoEPGTEeuiYi2dhrF+XwZ7j+WWHlsvIoSkhBg6N/0x\nCe/YJIaoMHdnDlS03w6I+Q03nZPI5vQsJs3bQrv4aK7t3cLtkIwxprr1BTar6lYAEZmFd0nRtScc\ndzfwDtCngm2HAUOc46YD84BTJtHGGPc888VG1u7LYMro3tWaQANEhwXTu3UDerduULpPVdl7LLd0\nKkhJsvz1hvTSBxlLHvTr2yaOTk5y3TkhliaxYdU2JcQNAZFEiwh/+mVXdhzK5tF3V9EqLpK+beLc\nDssYY6pTc2BXme3dQL+yB4hIc+AqYCg/TaJP1bbGlyU1xlSPpdsOM3n+FkYmt+Tirgm1ck0RoXn9\nCJrXj+CCzj92D7kFRWxJzyJIpNYe9Ktt/ndHJxHiCeLF63vTskEkt81MYeeh426HZIwxte1Z4GFV\nLa5KY/XO/zvpsqQikiIiKenp6WcSozGmCjJzC7jvrRW0iovksSu7uB0O4SEeujarR+emsX6ZQEMA\nJdEA9SJDeGVsH4oVbp6+jIzcArdDMsaY6rIHaFlmu4Wzr6xkYJaIbAeuBV4UkeGnabvfWY6U0y1L\nqqrJqpocHx9/pvdijKmkx+euZd+xHJ4e0dP1OcWBIqCSaIA2jaKYdMPZbD+YzV1vfE9hUZUGZIwx\npkaIyD9FpGsVmi4DOohIGxEJBa4D5pY9QFXbqGqiqiYCc4A7VPX907Sdi3c5UqjBZUmNMVX33x/2\n8c7y3dw1tP1P5iubmhVwSTTAwHaNeGJ4NxZsTOeJ/6xzOxxjjClrHTBFRJaIyAQRqXfaFoCqFgJ3\nAZ8653hbVdc455hQlbbO2xOBi0RkE3Chs22MqSP2Z+Ty6Hs/0KNFPe6+oIPb4QSUgB3vv65vKzYf\nyGLqN9toFx/F6AGJbodkjDGo6lRgqoh0Am4CVonIt8DLqvr1adp+DHx8wr7JJzl27OnaOvsPARdU\n5h6MMbWjpCphbkERT4/sSYgnIMdGXRPQ/7UfvawzFyQ15vEP17Jwkz0IY4ypG5x1m5Ocr4PASuB+\nZ+k5Y4wBfqxK+PvLu9DOqUpoak9AJ9GeIOG5Ub3o0DiaO15fzuYDWW6HZIwJcCLyDLAeuAz4X1Xt\nrap/V9UrgV7uRmeMqStKqhIO7RTPDaepSmhqRkAn0eBdOHzqmGTCgoO4ZfoyDmfnux2SMSawrQJ6\nquptqrr0hPf6uhGQMaZuyS8s5t63VhAVFszfr+3u0wVLfFnAJ9EALRpE8tLoZPYdy2XCa6nkF9qK\nHcYY1xylzPMqIlLfWYYOVT3mWlTG+IEdh7LJLShyO4wz9uwXG1mzN4OJV59V7VUJTcVZEu3o3boB\n/7i2O0u3Heb37/2At6aAMcbUuv8pmyyr6lHgf1yMxxi/MPO7HQx9ah4jXlrMwaw8t8OpsmXba78q\noSmfJdFlDOvZnN+c357ZqbuZsmCr2+EYYwJTef1ywK6kZMyZUlWe+nQDf3x/Nb1bN2Dj/kyunbSI\nHYey3Q6t0kqqErZoEMkf60BVwkBnSfQJ7r2wI5ef1ZSJn6zn87X73Q7HGBN4UkTkaRFp53w9DaS6\nHZQxvqigqJgH56zi+a83c12flrw5vj+vj+vPsZwCrn5xEat2H3U7xEp5fO5a9h7N4ZmRPYm2qoSu\nsyT6BEFBwlO/6sFZzetxz6zvWbPXpiAaY2rV3UA+8JbzlQfc6WpExvig7LxCxs9IYU7qbu69sAN/\nu/osgj1B9G7dgDm3DyQ8xMN1U75j3oZyK9nXOVaVsO6xJLocEaEept6YTL2IEMZNT+FARq7bIRlj\nAoSqZqvqI6qa7Hw9qqq+97mzMS46mJXHqJe/Y8HGdP529Vnce2HHn6xg0S4+mvfuGEhiwyjGTfcm\n2nWZVSWsmyyJPonGseG8fGMyR48XMH5mql88zWuMqftEJF5E/iEiH4vIVyVfbsdljK/YfjCbayYt\nYuP+TKaMTmZU3/LXUG4cG85bt/Wnf9uG/Hb2Sl74enOdXFRAVXlwziqrSlgH1dhPQkTCRWSpiKwU\nkTUi8idnf5yIfC4im5zvdfYziW7N6/HsdT1Ztfsov529sk7+4zLG+J3X8RZbaQP8CdgOLHMzIGN8\nxardR7lm0iIycgp4Y3x/LuzS5JTHx4SHMG1sH4b1bMY/Pt3AYx+soai4bv2un7F4Bws2pltVwjqo\nJv+cyQPOV9UeQE/gUhHpDzwCfKmqHYAvne0665KuCTx0SRIfrdrHs19scjscY4z/a6iqrwAFqjpf\nVW8Gznc7KGPquq83HOC6Kd8REephzu0DObtVxcboQoODeGZET247ry0zv9vBHa/XnU+fNx/I5H8/\nXmdVCeuoGkui1aukjnaI86XAMGC6s386MLymYqguEwa35ZqzW/Dcl5v4YMUet8Mxxvi3Auf7PhG5\nXER6AXFuBmRMXTc7ZRfjpqfQplEU794xsNIjtkFBwqOXdeaxK7rw2dr93DB1CUePu1vBOL+wmHtm\nWVXCuqxGJ9aIiEdEVgAHgM9VdQnQRFX3OYekAeV+1iIit4pIioikpKen12SYpyUi/O/V3eibGMeD\nc1axfOcRV+Mxxvi1J0SkHvAA8FtgKnCfuyEZUzepKs9/tYkH56xiQNuGzLq1/xlV8Lv53Db8a1Qv\nVu0+xrWTF7PnaE41Rls5JVUJ/2ZVCeusGk2iVbVIVXsCLYC+ItLthPcV7+h0eW2nlDydHh8fX5Nh\nVkhYsIfJo3uTEBvOrTNSXf2HZYzxTyLiATqo6jFVXa2qQ1W1t6rOdTs24z+KipUt6VlsSMt0O5Qz\nUlSsPPbBGp76bCNX9WrOtLF9iAkPOePzXtG9GdNv7sv+jFyufvFb1u3LqIZoK6dsVcJLrCphnVUr\nj3g6ZWu/Bi4F9otIUwDnu28s0AjERYXyyphk8gqKuOXVZWTlFbodkjHGj6hqETDK7TiM/ziUlce3\nmw/yyjfbeHD2Sq781zd0eewTLvjnfC55dgE3/XupK0nimcotKOL211KZ+d0Obhvcln/+qgehwdWX\n0gxo15DZEwYgCCMmL2bRloPVdu7TsaqEvkNqasUJEYnH+2DMURGJAD4D/g4MBg6p6kQReQSIU9WH\nTnWu5ORkTUlJqZE4q2L+xnRu+vdSzk9qzEujk/EE2TwlY0z5RCRVVZMrcfwzeJ8heQsoXR9aVZdX\noO2lwHOAB5iqqhNPeH8Y8BegGCgE7lXVb0Skk3O9Em2Bx1T1WRF5HBgPlMyr+52qfnyqOOpanx0I\n8gqL2Hwgi/X7MlmflsH6tEzWp2WSnplXekyj6DCSEmK8X01jOZCZy+R5W8jMK2R4z+bcf1FHWsZF\nungXFXP0eD63TE9h+c4jPHZFF246p02NXWvv0RzGTFvKjkPH+eeIHlzZo1mNXavEb2ev5N3lu5k9\nYaAVVXFJRfvtmkyiu+N9cNCDd8T7bVX9s4g0BN4GWgE7gBGqevhU56qLHfL0Rdv5n7lrGD+oDb+/\n3P5SNMaUrwpJ9Nfl7FZVPeUKHc5UkI3ARcBuvMvijVLVtWWOiQayVVWdPvptVU0q5zx7gH6qusNJ\norNU9amK3kNd7LP9haqy91gu6/f9mCiv35fB1oPZpUuzhQYH0bFJNEkJsU7SHEunhBjiY8J+dr5j\nxwt4cf5mXv12O8Wq3NC/NXcNbU/D6J8fWxfsPnKcMdOWsuuwt/T15d2b1vg1jx7PZ/yMFJZtP8If\nr+jCLefWXNL+yep9THhtOXef354HLu5UY9cxp1bRfrvGCq+r6iqgVzn7DwEX1NR1a8uYgYlsSc/i\n5YXbaN84mpF9bOkZY8yZU9WhVWzaF9isqlsBRGQW3tWQSpPoMismAURR/jMpFwBbVHVHFeMw1SQr\nr5ANac7I8r5MNqRlsi4tg8zcH6cStmgQQVJCLJd0TSCpqTdhTmwYSXAFC3LUiwzh0V90ZuzARJ77\nYhPTF21ndspuxg9qy7hBbYgKq7E0odLW7ctg7L+Xcjy/iBm39KV/24a1ct36kaHMvKUf985awV8+\nWkvasRwe/UVngqr5U+j9Gbk88u4PdG9Rj99YVUKfUHf+dfigx67owraD2fz+vdW0iotiQLva+Qdt\njPFfIvJYeftV9c+nadoc2FVmezfQr5zzXwX8DWgMXF7Oea4D3jxh390iciOQAjygqrZEUTUqKla2\nH8p2EuUM1jmJ867DPz7AHhMWTKeEGIb1bFY6wtwxIYbYaniQDqBpvQgmXtOdcYPa8tSnG3jmi43M\n/G47d5/fgVF9W1XrfOOqWLT5ILfNTCUqLJg5EwbSKSGmVq8fHuLhhV+fzZ8+XMPLC7exPyOPp6px\nHnbZqoTPWFVCn1Fj0zmqU13+aPBYTgFXv/gth7Lzef+Oc0hsFOV2SMaYOqQK0zkeKLMZDlwBrHOK\nrpyq3bXApao6ztkejXdKxl0nOf48vPOeLyyzLxTYC3RV1f3OvibAQbyj1n8BmpYXi4jcCtwK0KpV\nq947dthA9unsz8jl7je/Z+Wuo+QVFgMQJNA2PpqkhBg6N42lU5MYkprG0Lx+RK2uE/z9ziNM/O96\nlmw7TKu4SB64uCNXdm9W7aOvFTF35V4eeHsFbRpF8epNfWlWP6LWYyihqkyav4UnP9nAOe0bMvmG\n3tWyIkjJFNG/DO/G6P6tqyFScyZcnxNdnepyEg2w41A2w1/4lgZRobx3+znUi6yekQFjjO+rbBJd\nTvsw4FNVHXKa4wYAj6vqJc72owCq+rdTtNkK9FXVg872MOBOVb34JMcnAh+parfy3i9R1/vsuuLx\nuWt4fckObhyQSOem3tHl9o2jCQ/xuB0a4E0Y521M58lPNrBuXwZdm8Xy0KVJnNehUa0l9FMXbuWJ\n/6yjb2IcL9+YXGd+v76TupuH31lFhyYxvHpTH5rEVn0d580HMrn8/75hYLuGTBvbx4qq1AEV7bft\n84Jq0LphFJNv6M2uw8e5441UCoqK3Q7JGOM/IvGutX86y4AOItLGGVG+DvjJ+tIi0l6c39AicjYQ\nBhwqc8goTpjKUbIkqeMqYHWl78D8zKGsPGYt28mwns354xVduLZ3C7o1r1dnEmjwFhob2qkx/7n7\nXJ4d2ZOM3ALGTFvK9S8vYcWuozV67eJi5YmP1vLEf9bxi24JzLilb51JoAGu6d2CV8b2YcehbK5+\ncRGbD2SdvlE58guLufctq0roqyyJrib92jbkr1edxbebD/H43DX4wgi/MabuEZEfRGSV87UG2AA8\ne7p2qloI3AV8CqzDu/LGGhGZICITnMOuAVY7lWRfAEY6Ra8QkSi8K3u8e8KpnyyJCRiKVU+sFtMX\n7yC3oJgJg9u6HcppBQUJw3s158v7h/D4lV3YuD+T4S98y+2vpbIlvWrJ46nkFRZx71srmPrNNsYM\naM3z159dp/64KDG4Yzxv3TqAvMIirp28iNQdp1xorFzPfbmR1XusKqGvsukc1exv/13HS/O3MnZg\nIn+4vHOFn5A2xvinKsyJLjshshDY7yTIPsOX+mw3ZOcVMnDiV/Rt452i4Guy8gqZunArLy/YSm5h\nMSOSW3DPBR1JqHfmSWBGbgETZqayaMshHr40iQmD29b50dmdh45z47Ql7DuWy79G9eLiClYYTNl+\nmBEvLeba3i148toeNRylqQybzuGShy9J4pZz2/Dqou2Mm5FCZm6B2yEZY3xLU+Cwqu5Q1T1AhIj8\nbJUN47tmLdvFsZwCbh/Szu1QqiQ6LJh7L+zI/IeGMrp/a+ak7mbIU1/z90/Wcyyn6r/z9mfkMmLy\nYpZuO8zTI3pw+5B2dT6BBmjVMJJ3bh9IUtNYJryWyutLTv9QbWZuAfe97a1K+NiVXWshSlMTLImu\nZkFBwh+v6MITw7uxcNNBrp20mN1HjrsdljHGd0wCyn5Gnu3sM34gv7CYqQu30rdNHGe38u1qdI2i\nw3j8l1358v4hXNo1gcnzt3Dek1/z0vwt5BYUVepcmw9kcvWLi9h1+DjTxvbh6rMr8hhA3dEwOow3\nx/djcMd4fv/eap7+bMMpp3X+6cO17DniLRgTXYfW4jaVY0l0Dbmhf2tevakPe4/lMPyFb1m+05ZV\nNcZUiGiZ376qWoyt6e835q7cy75juT47Cl2eVg0jefa6Xnx097n0bFmfv/13PUOfmsfby3ZRWIEH\n7VN3HOaaSYvJKyzirdsGcF7H+FqIuvpFhgbz8o3JjEhuwf99tZmH31lV7kIDn6zex5zU3dw5tL2V\n9fZxlkTXoEEd4nnvjoFEhgZz3ZTv+HDlXrdDMsbUfVtF5DciEuJ83QNsdTsoc+aKi5XJ87eQlBDD\nEB9NFE+la7N6TL+5L2+O70/j2HAeemcVlz63kE/XpJ10VPbTNWlc//IS4qJCeff2c+jWvF4tR129\ngj1B/P2a7vzmgg68nbKbW2ekcDz/x0caDmTk8qhVJfQblkTXsPaNY3j/znPo0aIed7/5Pc99sclW\n7jDGnMoEYCCwhx+rDt7qakSmWny5/gCbD2QxYbBvzPWtqgHtGvL+HQOZfMPZFKty28xUrpm0iCVb\nD/3kuNe+28Htr6WS1DSWORMG0KphpEsRVy8R4f6LOvK/V53F/I3pjJryHYey8kqrEuZYVUK/YR8R\n1oK4qFBeG9ePR975gWe+2Mi2g1lMvKZ7nVyyxxjjLlU9gHeNZ+NHVJVJ8zbTokEEV3RvevoGPk5E\nuLRbUy7s3IQ5qbt59otNjJzyHUM7xfPgJUl8/MM+nv96M+cnNeb563sRGep/6cj1/VoRHxPGXW8s\n55pJi7jsrKbM35jOX4Z1pV18tNvhmWpQoT+DROQeEYkVr1dEZLmIlFvRypQvLNjD0yN68NuLO/L+\nir3cMHUJh7Ly3A7LGFPHiMh0EalfZruBiExzMyZz5pZtP8LynUcZP6htQC19GuwJ4rq+rZj34BAe\n+UUSqTuOcNn/LeT5rzczMrklU0b39ssEusRFXZrwxvj+HM0p4MV5WxjSKZ4brKy336jo/7k3q+pz\nInIJ0AAYDcwEPquxyPyQiHDX+R1IbBTFA2+vZPiL3zJtTB86NIlxOzRjTN3RXVVLy8Gp6hER6eVm\nQObMTZ6/hbioUEYkt3Q7FFeEh3iYMLgdo/q0Yuo3W4kND2HcoDZ+Pa2lRO/WDZgzYSD//nYb917Y\nMSDuOVBU9M/hkp/4ZcBMVV1TZp+ppCu6N+Ot2waQk1/M1S8uYuGmdLdDMsbUHUEiUvrIvojEYVPv\nfNr6tAy+Wn+AsQMTiQgN7Gl89SJDeODiTow/r+4XUalO7RtH89erziI+JsztUEw1qmgSnSoin+FN\noj8VkRjg9OvWmJPq2bI+H9x1Ds0bRDD238uY+d3pF2c3xgSEfwKLReQvIvIEsAh40uWYzBl4af5W\nIkM93DjAPsY3xp9UNIm+BXgE6KOqx4EQ4KZTNRCRliLytYisFZE1zjJNiEiciHwuIpuc7wG7SGLz\n+hHMuX0ggzvG88f3V/OnD9dQVGwrdxgTyFR1BnANsB9IA65W1ZnuRmWqatfh48xduZdRfVtRPzLU\n7XCMMdWookn0AGCDqh4VkRuAPwDHTtOmEHhAVbsA/YE7RaQL3mT8S1XtAHzpbAes6DDv4uw3n9OG\nf3+7nfEzUsjKKzx9Q2OM33KmzL0NzAWyRKSVyyGZKnrlm20IcMu5bdwOxRhTzSqaRE8CjotID+AB\nYAsw41QNVHWfqi53XmcC64DmwDBgunPYdGB4FeL2K54g4bErvaXC529M59pJi9hzNMftsIwxLhCR\nX4rIJmAbMB/YDvzX1aBMlRzKymPWsp0M79WcZvUj3A7HGFPNKppEFzplaIcBz6vqC0CFl5QQkUSg\nF7AEaKKq+5y30oAmJ2lzq4ikiEhKenpgPHhXUip8z9Echj3/LSt2HT19I2OMv/kL3k/vNqpqG+AC\n4Dt3QzJVMX3xDnILipkwuK3boRhjakBFk+hMEXkU79J2/xGRILzzok9LRKKBd4B7VTWj7HtOYl7u\nJGBVnaKqyaqaHB/vf+VRT2ZQh3jevX0gEaFBjHxpMR+tslLhxgSYAlU9hHeVjiBV/RpIdjsoUznZ\neYXMWLydi7o0oX1jW8bUGH9U0SR6JJCHd73oNKAF8I/TNRKRELwJ9Ouq+q6ze7+INHXebwocqHTU\nfq5Dkxjev+Mczmpej7ve+J5/fWmlwo0JIEedwYcFwOsi8hyQ7XJMppJmLdvF0eMFTBjczu1QjDE1\npEJJtJM4vw7UE5ErgFznCfKTEu8CkK8A61T16TJvzQXGOK/HAB9UOuoA0DA6jNfH9+OqXs355+cb\nuf/tleQVFrkdljGm5g0DjgP3AZ/gfQblyoo0FJFLRWSDiGwWkZ89tC0iw0RklYiscKbLnVvmve0i\n8kPJe2X224pKlZRfWMwrC7fSt00cvVvbfy5j/FVFy36PAJYCvwJGAEtE5NrTNDsH7/SP851OeYWI\nXAZMBC5yHpy50Nk25SgpFf7ARR157/s9/PplKxVujL9T1WxVLVbVQlWdrqr/50zvOCUR8QAvAL8A\nugCjnBWRyvoS6KGqPYGbgaknvD9UVXuqatnpI7aiUiXNXbmXvcdyud1GoY3xaxWtgvV7vGtEHwAQ\nkXjgC2DOyRqo6jecvKrhBZUJMpCJCHdf0IE28d5S4Ve9uIhpY5Ntjp0x5kR9gc2quhVARGbhHdVe\nW3KAqmaVOT6KkzyTcoJhwBDn9XRgHvDwmYfrn4qLlZfmbyEpIYYhnQLneR5jAlFF50QHlSTQjkOV\naGuqwRXdmzHr1v4czy/iKisVboz5uebArjLbu519PyEiV4nIeuA/eEejSyjwhYikisitZfZXaEUl\n4/Xl+gNsOpDFhMHtAqqstTGBqKKJ8Cci8qmIjBWRsXg7349rLixTnl6tGvD+nQNpXt9bKvw1KxVu\njN8pqe56un1VparvqWoS3jX6/1LmrXOdaR6/wFsc67xy2p50RaVAXJa0PJPnb6F5/Qiu6N7U7VCM\nMTWsog8WPghMAbo7X1NU1T7Oc0GLBpHMnjCA8zo04g/vr+bPH661UuHG+Jcx5ewbW4F2e4CWZbZb\nOPvKpaoLgLYi0sjZ3uN8PwC8h3d6CFRwRaVAXZa0rGXbD5O64wi3nteWYI99WGuMv6vonGhU9R28\ny9UZl8WEhzB1TB+e+M9apn27jR2HsnluVC+iwyr84zTG1DEiMgq4HmgjInPLvBUDHK7AKZYBHUSk\nDd7k+TrnfGWv0R7YoqoqImcDYcAhEYnCO20v03l9MfBnp1nJikoTsRWVTmnSvC3ERYUyIrnl6Q82\nxvi8U2b3kQemAAAZBklEQVRdIpJJ+R/dCd5P9mJrJCpzWp4g4X+u7Erb+Ggen7uGayctYtrYPlZa\n1hjftQjYBzQC/llmfyaw6nSNVbVQRO4CPgU8wDRVXSMiE5z3JwPXADeKSAGQA4x0EuomwHvOHN5g\n4A1V/cQ59UTgbRG5BdiBd4Umc4L1aRl8tf4A91/UkYhQj9vhGGNqgfhCEY/k5GRNSUk5/YEBasHG\ndO58fTnhoR5evjGZni3rux2SMcYhIqknLBlXkTatgQ6q+oWIRADBqppZMxFWv0Dss+97awWfrklj\n0SPnUz8y1O1wjDFnoKL9tk3a8gPndYzn3TsGEh4SxK8mL+KxD1ZzIDPX7bCMMVUgIuPxLh/6krOr\nBfC+exGZ09l95DhzV+5lVN9WlkAbE0AsifYTHZrE8MGd5zIiuSWvL9nJ4Cfn8dSnG8jILXA7NGNM\n5dyJt1hVBoCqbgIauxqROaWpC7chwC3ntnE7FGNMLbIk2o/ERYXy16vO4ov7B3NhlyY8//Vmznvy\na15esJXcAisZboyPyFPV/JINEQmmYkVRfFpxsVJQVOx2GJV2ODufWct2MrxXc3smxZgAY0m0H2rT\nKIp/jerFR3efS/cW9fnrx+sY+tQ83lq2k0If/CVlTICZLyK/AyJE5CJgNvChyzHVKFXlwTmruO+t\nFT63ZOeri7aTW1DMhMFt3Q7FGFPLLIn2Y92a12PGzX15Y3w/msSG8/A7P3DJswv4ZPU+fOGBUmMC\n1CNAOvADcBvewlZ/cDWiGiYidGgSzUer9vGH91f7TP90PL+QGYu3c2HnJrRvHON2OMaYWmYLCweA\nge0a8d4dDfls7X7+8ekGJry2nB4t6/PwJZ0Y2L6R2+EZY8pQ1WLgZeBlEYkDWqivZJVnYMLgdhzL\nKWDSvC3UiwjhkV8kuR3Sac1auoujxwu4fUg7t0MxxrjAkugAISJc0jWBCzs34d3lu3nm841cP3UJ\ngzo04qFLkjirRT23QzTGACIyD/gl3v45FTggIotU9T5XA6sFD13SiYycAibP9ybSdTk5LSgqZurC\nrfRNjKN36wZuh2OMcYFN5wgwniDhV8kt+eq3Q/jD5Z1ZvecYVz7/DXe+sZyt6Vluh2eMgXqqmgFc\nDcxQ1X7ABS7HVCtEhD8P68aVPZrx90/W88aSnW6HdFJzV+xl77HcOp3oG2Nqlo1EB6jwEA/jBrVl\nZJ+WvLxwG1MXbuWT1WmMSG7JPRd0IKFeuNshGhOogkWkKd7KgL93O5ja5gkSnh7Rg6zcAn7//g/E\nhAdzZY9mbof1E8XFyuT5W0hKiGFIp3i3wzHGuMRGogNcTHgI91/UkfkPDmV0/9bMSd3F4H98zcT/\nrufYcVtj2hgX/Blv6e7NqrpMRNoCm1yOqVaFeIJ48de96dM6jvveWsHXGw64HdJPfLX+AJsOZDFh\ncDucUunGmABUY0m0iEwTkQMisrrMvjgR+VxENjnfbSJZHREfE8bjv+zKVw8M4fKzmvLSgi0MevIr\nXpy3mZx8W2PamNqiqrNVtbuq3uFsb1XVa9yOq7ZFhHqYOjaZTgkx3P5aKku3HXY7pFKT5m+hef0I\nruje1O1QjDEuqsmR6FeBS0/Y9wjwpap2AL50tk0d0jIukqdH9uTj3wyiT2IcT36ygcH/+JrXl+zw\nyUIIxhjfFRsewvSb+9KsXgS3vLqM1XuOuR0Sy7YfJnXHEcYPakOwxz7MNSaQ1VgPoKoLgBOHDoYB\n053X04HhNXV9c2Y6N43llbF9mD1hAK3iIvn9e6u56On5fLhyL8U+VgzBGOO7GkWHMXNcP2LCgxkz\nbSlbXH4AevK8LcRFhTKyTytX4zDGuK+2/4xuoqr7nNdpQJNavr6ppD6JccyeMIBXxiQTFuzh7je/\n58rnv2H+xnSfKYhgjC8REY/bMdQ1zetH8Nq4fgCMnrqEvUdzXIljQ1omX64/wJgBiUSE2o/JmEDn\n2mdRTvGAk2ZhInKriKSISEp6enotRmZOJCJc0LkJH98ziGdG9uBYTgFjpi1l1Mvf8f3OI26HZ4y/\n2SYiU0TkArGn1kq1jY9m+s19ycwt5IZXlnAwK6/WY3hp/hYiQz3cOKB1rV/bGFP31HYSvd9Zugnn\n+0kfuVbVKaqarKrJ8fG2hFBd4AkSrurVgi8fGMzjV3Zh0/4srnpxEbfNTGHzgUy3wzPGXyQBXwB3\n4k2onxeRc12OqU7o1rwe027qw96jOYyZtpSM3NpbQWj3keN8sHIv1/VpRYOo0Fq7rjGm7qrtJHou\nMMZ5PQb4oJavb6pBWLCHsee0Yf5DQ7n/oo58u/kQFz+zgNtmpjBr6U7XPmo1xh+o6nFVfVtVrwZ6\nAbHA/Iq0FZFLRWSDiGwWkZ89uC0iw0RklYiscD7pO9fZ31JEvhaRtSKyRkTuKdPmcRHZ47RZISKX\nVdOtVkmfxDgm3dCbDWmZjHs1pdZWD5q6cBsCjBvUplauZ4yp+2qs2IqIvAkMARqJyG7gf4CJwNsi\ncguwA28xAeOjosOC+c0FHbihf2smz9/C3BV7+XTNfgA6NI5mcMd4BneKp09iHOEhNn/QmIoSkcHA\nSLwrHKVQgb7SmUv9AnARsBtYJiJzVXVtmcO+BOaqqopId+BtvCPfhcADqrpcRGKAVBH5vEzbZ1T1\nqeq6vzM1tFNjnhnZk9/M+p47Xk/lpdHJhAbX3JjQ4ex8Zi3bybCezWlWP6LGrmOM8S01lkSr6qiT\nvBUQ5WsDSVxUKL+7rDOP/iKJjfuzWLAxnfkb05mxeAdTv9lGeEgQ/do0ZHDHeM7rGE+7+CgrUGDM\nSYjIduB7vAnug6qaXcGmffEWaNnqnGcW3hWRSpNoVS27tEUUznMpzgPf+5zXmSKyDmhetm1dc2WP\nZmTmFvK7937ggdkreXZkTzxBNdOvTF+0ndyCYiYMblsj5zfG+CYr+22qjYjQKSGGTgkxjD+vLTn5\nRXy37RDzN6SzYFM6f/7I+/u4ef0IzusYz+CO8Qxs35DY8BCXIzemTumuqhlVaNcc2FVmezfQ78SD\nROQq4G9AY+Dyct5PxDuNZEmZ3XeLyI14R8UfUNU68UTx9f1acSyngL9/sp6Y8GD+Orxbtf+Bfjy/\nkOmLt3Nh5yZ0aBJTrec2xvg2S6JNjYkI9TC0U2OGdmoMwK7Dx1mwKZ35G9L5cOVe3ly6E0+Q0LtV\nA87r2IjBHRvTtVksQTU0mmSMj0gQkffwLgnazZl28UtVfaI6Tq6q7wHvich5wF+AC0veE5Fo4B3g\n3jKJ/CTnOHW+/xO4+cTzisitwK0ArVrV3hrKtw9px7GcAibP30K9iBAevjSpWs8/a+kujh4v4PYh\nNgptjPkpS6JNrWkZF8mv+7Xm1/1aU1BUzPIdR7xJ9cZ0nvpsI099tpGGUaEM6tCI8zrGM6hDPPEx\nYW6HbUxtexl4EHgJQFVXicgbwOmS6D1AyzLbLZx95VLVBSLSVkQaqepBEQnBm0C/rqrvljluf8lr\nEXkZ+Ogk55sCTAFITk6u1UXkH760Exm5BUya502kJwxuVy3nLSgqZurCrfRNjKN367hqOacxxn9Y\nEm1cEeIJol/bhvRr25AHL0kiPTOPbzans2DjQRZsTOf9FXsB6Nos1vuAYsd4zm7dgBArs2v8X6Sq\nLj1hWkJhBdotAzqISBu8yfN1wPVlDxCR9sAW58HCs4Ew4JCzHvUrwDpVffqENk3LFMm6ClhdlZuq\nSSLCX4Z1IyOngIn/XU+9iBBG9T3z0fC5K/ay91guT1zVrRqiNMb4G0uiTZ0QHxPGVb1acFWvFhQX\nK2v3ZTDfeUBxyoKtvDhvC9FhwQxo17A0qW4ZF+l22MbUhIMi0g7noT8RuRbnob9TUdVCEbkL+BTw\nANNUdY2ITHDenwxcA9woIgVADjDSSajPBUYDP4jICueUv1PVj4EnRaSnE8924LZqvNdq4wkSnh7R\nk6w878OGMeHBXNG9WZXPV1ysvLRgC52axJROSTPGmLLEF0o3Jycna0pKitthGJdk5hawaMshb1K9\nIZ09zjrUbRtFlT6g2KtVfepHWgEEU/eISKqqJlfi+LZ4p0UMBI4A24AbVHV7zURY/dzss3Pyi7hx\n2hJW7DrKyzcmM6SKCfAXa/czbkYKz4zswVW9WlRzlMaYuqyi/bYl0canqCpbD2aXLqP33dZD5BYU\nA9A4Jsy7OkiTGDo63zs0iSYy1D5wMe6pbBJdpl0UEKSqPlcO1O0++1hOAaOmfMfWg1nMvKUffRIr\nP5/52kmL2Hcsl3kPDrFpZMYEmIr225ZdGJ8iIrSLj6ZdfDQ3ndOG3IIiUnccYe3eDDbsz2Tj/kxe\nW7KjNLEWgZYNIn+WXLdpFFWjxRmMqSwRuf8k+wE4ca6yObl6ESHMuKUvIyYv5uZXlzHr1v50bVav\nwu2XbT9Myo4jPH5lF0ugjTEnZUm08WnhIR7Oad+Ic9o3Kt1XVKzsOnzcm1SnZbJhfyYb0jL5av0B\nioq9n7yEeIS2jaKdpDqajk2861u3bBBpS+wZt5QsQtwJ6APMdbavBJa6EpEPaxQdxsxx/fjVpEXc\n+MpSZk8YQNv46Aq1nTxvC3FRoYzsU3tL9RljfI8l0cbveIKExEZRJDaK4pKuCaX78wqL2HYwmw1p\n3hHrDWmZrNh1hA9X7i09JiLEQ8cySXXHJjEkJcQQHxNmVRZNjVLVPwGIyALg7JJpHCLyOPAfF0Pz\nWc3rRzBzXD9GTF7MaCeRPl3Z7g1pmXy5/gD3XdiRiFBPLUVqjPFFlkSbgBEW7CEpIZakhNif7M/O\nK2TTgSw2pGWwIS2Ljfszmbcxndmpu0uPqR8Z4k2sneS6U0IMHRvHUC/Sqi2aatcEyC+zne/sM1XQ\nLj6a6Tf3ZdSU77jhlSXMvm0ADaNPvv78S/O3EBHi4cYBrWsxSmOML7Ik2gS8qLBgerasT8+W9X+y\n/3B2/o+j1s7UkPdX7CEz98clexvHhNE4Noy4qDAaRoUSFxVKw+hQ53WYdzsqlLjoUGLCgm0021TE\nDGCpU7UQYDjwqnvh+L5uzevxytg+jH5lCWP+vZQ3xvcnNvznfwDvPnKcuSv3cuOARBpE2Wo/xphT\nsyTamJOIiwplQLuGDGjXsHSfqrLvWG5pUr35QBYHs/I4nJ3PlgNZHM7OJ6egqNzzhXqCaBAVQlxU\nGI2ivQl3aZJdknBH/7gvNjzE5mcHIFX9q4j8Fxjk7LpJVb93MyZ/0LdNHJNv6M34GSmMm57CjJv7\nEh7y0+kaUxduA2DcoDZuhGiM8TGWRBtTCSJCs/oRNKsfcdICDDn5RRzK9ibWh7LzOZyV/+Pr7DwO\nZXlf7zh0nMPZ+WTllV+MzhMkZZLsExLu6FDiIkOJCQ8mOjyY2PBgYsJDiAkPJiLEYyPePk5VlwPL\n3Y7D3wxNaszTI3tyz6zvueP15bw0unfp6huHs/N5a9kuhvVsftp508YYA5ZEG1PtIkI9tAiNpEWD\nilVUzC0o4sjxfA45yXZJwn0oq0winp3Pmr0ZHMrKIyP31BWgPUFCdFiwN8EOCybWSa5LEu6SZDsm\nrMzr8JDSNrHhIUSFeQi2pb2MH/plj2Zk5hbw+/dW88DbK3lmZE88QcL0RdvJKShiwuC2bodojPER\nlkQb47LwEA9N60XQtF7FRr8Kioo5kp3PkeMFZOYWkJlbSGZeYenrrNwfX2fkFpKVV0BaRi6bDhSS\n5RxXUHT6IkuRoZ5yE+zosGCiwoIJDwkiIsRDRKiHsBAPESGe0n3hpV8/HhMe7BwbHGQj5cZVv+7X\nmmM5BTz5yQZiwoP53WWdmb54Oxd2bkKHJjGnbW+MMeBSEi0ilwLPAR5gqqpOdCMOY3xRiCeIxrHh\nNI4Nr1J7VSWvsNibfJck4U6ynVHyusx7WXmFZDiv9x3LJTO3gON5ReQUFFFYXLWKp+EhQYQ7iXdE\nSEkS/uO+kiQ8IjSoNPku2RfqETxBQQQHCcEewRMkhHiCnO/e90KCvPuDPd7jfn5Mme2gIDwe8Z7P\nOdaSfP93x5D2HMsp4KX5W1m5+yhHjxdw+xAbhTbGVFytJ9Ei4gFeAC4CdgPLRGSuqq6t7ViMCUQi\nUpqQxsecfKmviigoKia3oIjcgpLv3uQ6J7+I3MJicvKLyCv0buc4x+UUFJF3iuOO5RT85Jw5zpdW\nLV+vkpMl3lNGJ9PjhFVcjO965NIkMnIKeXPpTvomxtG7deXLgxtjApcbI9F9gc2quhVARGYBwwBL\noo3xMSGeIEI8QcRUbVC8wlSV/KJicvOLKSguprBIKSz9rhQVKwVFxRQV/3R/YbFSWFT882NK3y9z\nrmKlqEgpKFaKyp7DaV9YpNS3dcH9iojwxPButIuPYkineLfDMcb4GDeS6ObArjLbu4F+LsRhjPER\nIkJYsIewYKsgZ6qXJ0gYN8imcRhjKq/OPn4vIreKSIqIpKSnp7sdjjHGGGOMMaXcSKL3AC3LbLdw\n9v2Eqk5R1WRVTY6Pt4/ZjDHGGGNM3eFGEr0M6CAibUQkFLgOmOtCHMYYY4wxxlSJaG0+8l5yUZHL\ngGfxLnE3TVX/eprj04EdVbhUI+BgFdr5skC8ZwjM+w7Eewbfu+/WqhpQH6dZn11pgXjfds+Bwxfv\nu0L9titJdG0RkRRVTXY7jtoUiPcMgXnfgXjPELj3HQgC9WcbiPdt9xw4/Pm+6+yDhcYYY4wxxtRV\nlkQbY4wxxhhTSf6eRE9xOwAXBOI9Q2DedyDeMwTufQeCQP3ZBuJ92z0HDr+9b7+eE22MMcYYY0xN\n8PeRaGOMMcYYY6qdXybRInKpiGwQkc0i8ojb8dQGEWkpIl+LyFoRWSMi97gdU20REY+IfC8iH7kd\nS20RkfoiMkdE1ovIOhEZ4HZMNU1E7nP+314tIm+KSLjbMZnqYX229dn+zvps/+yz/S6JFhEP8ALw\nC6ALMEpEurgbVa0oBB5Q1S5Af+DOALlvgHuAdW4HUcueAz5R1SSgB35+/yLSHPgNkKyq3fCuMX+d\nu1GZ6mB9tvXZAcL6bD/ss/0uiQb6AptVdauq5gOzgGEux1TjVHWfqi53Xmfi/Qfa3N2oap6ItAAu\nB6a6HUttEZF6wHnAKwCqmq+qR92NqlYEAxEiEgxEAntdjsdUD+uzrc/2a9Zn+2+f7Y9JdHNgV5nt\n3QRAx1SWiCQCvYAl7kZSK54FHgKK3Q6kFrUB0oF/Ox+JThWRKLeDqkmqugd4CtgJ7AOOqepn7kZl\nqon12dZn+zvrs/20z/bHJDqgiUg08A5wr6pmuB1PTRKRK4ADqprqdiy1LBg4G5ikqr2AbMCv55GK\nSAO8o5NtgGZAlIjc4G5Uxpw567MDgvXZftpn+2MSvQdoWWa7hbPP74lICN7O+HVVfdfteGrBOcAv\nRWQ73o+AzxeR19wNqVbsBnarasmo1Ry8HbQ/uxDYpqrpqloAvAsMdDkmUz2sz7Y+299Zn+2nfbY/\nJtHLgA4i0kZEQvFOZJ/rckw1TkQE73yrdar6tNvx1AZVfVRVW6hqIt6f81eq6nd/6Z5IVdOAXSLS\nydl1AbDWxZBqw06gv4hEOv+vX4CfP5gTQKzPtj7br1mf7b99drDbAVQ3VS0UkbuAT/E+DTpNVde4\nHFZtOAcYDfwgIiucfb9T1Y9djMnUnLuB152kYytwk8vx1ChVXSIic4DleFc1+B4/roIVSKzPtj47\nQFif7Yd9tlUsNMYYY4wxppL8cTqHMcYYY4wxNcqSaGOMMcYYYyrJkmhjjDHGGGMqyZJoY4wxxhhj\nKsmSaGOMMcYYYyrJkmjjc0RkkfM9UUSur+Zz/668axljjKka67ONv7Il7ozPEpEhwG9V9YpKtAlW\n1cJTvJ+lqtHVEZ8xxpgfWZ9t/I2NRBufIyJZzsuJwCARWSEi94mIR0T+ISLLRGSViNzmHD9ERBaK\nyFycKlEi8r6IpIrIGhG51dk3EYhwzvd62WuJ1z9EZLWI/CAiI8uce56IzBGR9SLyulOdyRhjDNZn\nG//ldxULTUB5hDKjGk7HekxV+4hIGPCtiHzmHHs20E1VtznbN6vqYRGJAJaJyDuq+oiI3KWqPcu5\n1tVAT6AH0Mhps8B5rxfQFdgLfIu3Etk31X+7xhjj06zPNn7FRqKNP7kYuNEpobsEaAh0cN5bWqYz\nBviNiKwEvgNaljnuZM4F3lTVIlXdD8wH+pQ5925VLQZWAInVcjfGGOPfrM82Ps1Goo0/EeBuVf30\nJzu98/CyT9i+EBigqsdFZB4QfgbXzSvzugj7d2WMMRVhfbbxaTYSbXxZJhBTZvtT4HYRCQEQkY4i\nElVOu3rAEaczTgL6l3mvoKT9CRYCI505fPHAecDSarkLY4wJDNZnG79if30ZX7YKKHI+4nsVeA7v\nx3LLnQdF0oHh5bT7BJggIuuADXg/HiwxBVglIstV9ddl9r8HDABWAgo8pKppTodujDHm9KzPNn7F\nlrgzxhhjjDGmkmw6hzHGGGOMMZVkSbQxxhhjjDGVZEm0McYYY4wxlWRJtDHGGGOMMZVkSbQxxhhj\njDGVZEm0McYYY4wxlWRJtDHGGGOMMZVkSbQxxhhjjDGV9P+vyT0HMM0v9AAAAABJRU5ErkJggg==\n", "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "reload(bilstm);\n", "bilstm.plot_results(losses, accuracies)" ] }, { "cell_type": "code", "execution_count": 170, "metadata": { "collapsed": true }, "outputs": [], "source": [ "reload(tagger_base);\n", "reload(bilstm);\n", "confusion = tagger_base.eval_model(model,'bilstm_crf-dev-nr.preds', word_to_ix_nr, \n", " trainfile=NR_TRAIN_FILE, testfile=NR_DEV_FILE)" ] }, { "cell_type": "code", "execution_count": 171, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "0.344578313253012\n" ] } ], "source": [ "print (scorer.accuracy(confusion))" ] }, { "cell_type": "code", "execution_count": 172, "metadata": { "collapsed": true }, "outputs": [], "source": [ "tagger_base.apply_model(model,'bilstm_crf-te-nr.preds',word_to_ix_nr, all_tags_nr, \n", " trainfile=NR_TRAIN_FILE, testfile=NR_TEST_FILE_HIDDEN)\n" ] }, { "cell_type": "code", "execution_count": 173, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "0.3550631280597784\n" ] } ], "source": [ "# you don't have no_bokmaal-ud-test.conllu, so you can't run this\n", "te_confusion = scorer.get_confusion(NR_TEST_FILE,'bilstm_crf-te-nr.preds')\n", "print (scorer.accuracy(te_confusion))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "7. Improve your Taggers performance\n", "==================\n", "**Deliverable 7.1** (*2.0 points*)\n", "Improve your taggers performance to get atleast `87.00%` on the dev dataset for both English and Norwegian.\n", "You can use any model that you have implemented in this assignment: HMM, BiLSTM, or BiLSTM-CRF.\n", "- HMM is unlikely to score high enough for full credit\n", "- BiLSTM-CRF is more accurate than BiLSTM in theory, but it takes a long time to train. So iterating on BiLSTM might be better. \n", "- Ideas: try varying the input (character-level features?), the optimizer, the model architecture, the pretrained embeddings, dropout, regularization, etc. Here are some more pretrained embeddings to try: [fastText](https://github.com/facebookresearch/fastText), [Polyglot](polyglot.readthedocs.io/en/latest/Embeddings.html), [word2vec](https://code.google.com/archive/p/word2vec/), [Glove](https://nlp.stanford.edu/projects/glove/).\n", "- For Norwegian, there are officially [two written forms of language](https://en.wikipedia.org/wiki/Norwegian_language): `Bokmål` and `Nynorsk`. The dataset we are using is for the `Bokmål` language. So, when using pretrained embeddings make sure to use it for that specific form.\n", "\n", "**Output files:** Make sure to name your files as follows: The unit tests will check for these files.\n", "- `model-dev-en.preds` ( Predictions for the dev dataset of english ) \n", "- `model-te-en.preds` ( Predictions for the test dataset of english ) \n", "- `model-dev-nr.preds` ( Predictions for the dev dataset of norwegian ) \n", "- `model-te-nr.preds` ( Predictions for the test dataset of norwegian ) \n", "\n", "**Tests**: `test_performance.py`: \n", "- `test_en_dev_accuracy()`\n", "- `test_en_test_accuracy()` #you cannot run this unit test.\n", "- `test_nr_dev_accuracy()`\n", "- `test_nr_test_accuracy()` #you cannot run this unit test." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Rubric\n", "\n", "Dev Set (both languages)\n", "- $\\geq$ 85% (*0.15 points*)\n", "- $\\geq$ 86% (*0.25 points*)\n", "- $\\geq$ 87% (*0.5 points*)\n", "\n", "Test Set (both languages)\n", "- $\\geq$ 84% (*0.15 points*)\n", "- $\\geq$ 85% (*0.25 points*)\n", "- $\\geq$ 86% (*0.5 points*)\n" ] }, { "cell_type": "markdown", "metadata": { "collapsed": true }, "source": [ "8. Bakeoff (Extra Credit)\n", "==================" ] }, { "cell_type": "markdown", "metadata": { "collapsed": true }, "source": [ "Try to get the best accuracy possible on both `English` and `Norwegian` tagging.\n", "\n", "Some ideas:\n", "\n", "- Better features, such as characters\n", "- Better optimization\n", "- Number of Layers and Hidden units in the BiLSTM\n", "- More hidden layers in the Fully Connected Layer\n", "- Better loss function: like structured perceptron\n", "- Better preprocessing\n", "- Using Pretrained Embeddings like [fastText](https://github.com/facebookresearch/fastText), [Polyglot](polyglot.readthedocs.io/en/latest/Embeddings.html), [word2vec](https://code.google.com/archive/p/word2vec/), [Glove](https://nlp.stanford.edu/projects/glove/)\n", "- Dropout or other regularization scheme\n", "\n", "The current best accuracies from the staff are:\n", "- For English data: \n", " - `88% dev`, \n", " - `87.5% test`\n", "- For Norwegian data: \n", " - `89% dev`, \n", " - `87.8% test`\n", "\n", "However, these accuracies may improve before the end of the bakeoff! \n", " \n", "\n", "**Output files:** In addition to participating in the kaggle bakeoff, make sure to name your files as follows:\n", "\n", "- `bakeoff-dev-en.preds` ( Predictions for the dev dataset of english ) \n", "- `bakeoff-te-en.preds` ( Predictions for the test dataset of english ) \n", "- `bakeoff-dev-nr.preds` ( Predictions for the dev dataset of norwegian ) \n", "- `bakeoff-te-nr.preds` ( Predictions for the test dataset of norwegian ) \n", "\n", "The unit tests will check for these files. To get the extra credit, you must both participate in Kaggle and submit these files.\n", "\n", "**Rubric**\n", "\n", "- Top 3 final score on Kaggle in English: +0.5%\n", "- Top 3 final score on Kaggle in Norwegian: +0.5%\n", "- Beat best staff final score on Kaggle in English: +0.5%\n", "- Best best staff final score on Kaggle in Norwegian: +0.5%" ] }, { "cell_type": "markdown", "metadata": { "collapsed": true }, "source": [ "# 9. 7650 Research Question \n", "\n", "(*1 point*)\n", "\n", "You will select a recent research paper that performs **sequence labeling** either POS tagging or some other task within the last five years (2014-2018). Summarize the paper, answering the following questions:\n", "\n", "- List the title, author(s) and venue of the paper.\n", "- What is the task they are trying to solve?\n", "- What tagging methods do they use? HMMs, CRF?, max-margin markov networks, deep learning models?\n", "- What features do they use and why?\n", "- What methods and features are most effective?\n", "- Give a one-line summary of the paper that the authors are trying to leave for the reader.\n", "\n", "Your selection of papers is determined by the last digit of your GTID.\n", "\n", "- Digits 0-4: choose from NAACL, EMNLP, or NIPS\n", "- Digits 5-9: choose from ACL, EACL, or TACL\n", "\n", "You must choose a paper in the main conference (not workshops). The paper must be at least four pages long. All papers from these conferences are available for free online." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.3" } }, "nbformat": 4, "nbformat_minor": 2 }