{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"cell_id": "00000-4c943a74-89a4-4936-a64a-1f160563a0c3",
"deepnote_cell_type": "markdown",
"id": "CBvij-E8ws6i"
},
"source": [
"# Programming Assignment 5: Neural Networks"
]
},
{
"cell_type": "markdown",
"metadata": {
"cell_id": "00001-72aa6fcd-ab1c-45dd-98a5-90cf247739d7",
"deepnote_cell_type": "markdown",
"id": "pNH8-FIUSSIC"
},
"source": [
"In the previous assignments, we learned about **text preprocessing tools** like tokenization and regular expressions, **NLP tasks** like information retrieval and sentiment\n",
"analysis, and **task-solving tools** like word vectors, Naive Bayes, and logistic regression.\n",
"\n",
"This time, we’ll integrate and expand these ideas to build a new and\n",
"powerful tool for your NLP toolkit: a simple neural network!\n",
"\n",
"Neural networks are extremely useful and powerful tools that are behind some\n",
"really impressive advances in NLP over the last few years. You'll see plenty of\n",
"them in future NLP classes (if you choose to take them), but hopefully this\n",
"assignment will give you a first taste of how they work and what you can do\n",
"with them!"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Part 0. Environment Check\n",
"\n",
"Before we do anything else, let's quickly check that you're running the correct\n",
"version of Python and are in the right environment!"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"pycharm": {
"name": "#%%\n"
}
},
"outputs": [],
"source": [
"import os\n",
"from typing import Dict, List, Tuple\n",
"\n",
"import gensim.models\n",
"\n",
"assert os.environ['CONDA_DEFAULT_ENV'] == \"cs124\"\n",
"\n",
"import sys\n",
"assert sys.version_info.major == 3 and sys.version_info.minor == 10"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"If the above cell complains, it means that you're using the wrong environment\n",
"or Python version!\n",
"\n",
"If so, please exit this notebook, kill the notebook server with CTRL-C, and\n",
"try running\n",
"\n",
"$ conda activate cs124\n",
"\n",
"then restarting your notebook server with\n",
"\n",
"$ jupyter notebook\n",
"\n",
"If that doesn't work, you should go back and follow the installation\n",
"instructions in PA0."
]
},
{
"cell_type": "markdown",
"metadata": {
"cell_id": "00002-b76b2fef-04ed-469b-8893-a3daa5c6502d",
"deepnote_cell_type": "markdown",
"id": "umaY6CwjwnSt"
},
"source": [
"## Part 1.1: Why not Logistic Regression?\n",
"\n",
"Before we build a simple neural network, let's take a brief look back at\n",
"logistic regression (from PA2). Recall that the output of a logistic regression classifier is given by:\n",
"\n",
"$y = \\sigma(XW + b)$ \n",
"\n",
"$X$ is a matrix of shape (`num_examples`, `num_features`), so that each row is a training example.\n",
"\n",
"$W$ is a vector of shape (`num_features`, 1), so that it transfers each example in X from a `num_features`-dimensional vector to a 1-dimensional scalar output. In other words, the computation result $XW$ is of shape (`num_examples`, 1).\n",
"\n",
"Finally, $b$ is a scalar added to every element of the vector $XW$, and $\\sigma()$ is the\n",
"sigmoid function.\n",
"\n",
"\n",
"This approach worked really well on the text classification task in PA2. Yet in this section, we will reveal its weakness. \n",
"\n",
"To see that, let's bring out a toy dataset."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cell_id": "00003-9f3bde15-181e-44a8-88e9-5fc68598900e",
"deepnote_cell_type": "code",
"deepnote_to_be_reexecuted": false,
"execution_millis": 888,
"execution_start": 1619508614561,
"id": "KB6ZUXp441N6",
"output_cleared": true,
"source_hash": "202f447a"
},
"outputs": [],
"source": [
"# imports\n",
"import numpy as np\n",
"\n",
"np.random.seed(1)\n",
"\n",
"from util import *"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"tags": [
"exploration"
]
},
"outputs": [],
"source": [
"%matplotlib inline"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cell_id": "00004-3daf0648-9e59-4496-a81a-dcddd5112911",
"deepnote_cell_type": "code",
"deepnote_to_be_reexecuted": false,
"execution_millis": 13,
"execution_start": 1619508616449,
"id": "G5GOoYtrwzmR",
"output_cleared": true,
"source_hash": "1d785f64",
"tags": [
"exploration"
]
},
"outputs": [],
"source": [
"# Construct a toy dataset\n",
"X, y = generate_2d_xor_dataset()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cell_id": "00005-2cd87dc4-71c6-4d31-ab8a-0614defa67b3",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 265
},
"deepnote_cell_type": "code",
"deepnote_to_be_reexecuted": false,
"execution_millis": 337,
"execution_start": 1619508620002,
"id": "fkKWlBvdw4Dk",
"outputId": "7de01a83-6377-468b-8b1d-e90655b08b44",
"output_cleared": true,
"source_hash": "d1b799c0",
"tags": [
"exploration"
]
},
"outputs": [],
"source": [
"# Plot our example dataset\n",
"plot_2d_dataset_points(X, y)"
]
},
{
"cell_type": "markdown",
"metadata": {
"cell_id": "00006-631ebd22-1b38-4255-bc25-8b716ed9eba8",
"deepnote_cell_type": "markdown",
"id": "IqQMOcX1xLWj"
},
"source": [
"This distribution may look a bit unusual at first sight, but datasets with this sort of property actually appear quite often in real life. \n",
"\n",
"As an example, consider the case of doing sentiment analysis, but in the presence of sarcasm. Let each data point represent a sentence, with the $x$ value representing\n",
"the average sentiment score of the words in the sentence, and the $y$ value\n",
"representing the amount of sarcasm detected in the sentence (assume this is measurable). Again, a datapoint is labeled as 0 (blue) if its true\n",
"sentiment is negative, and it's labeled as 1 (red) if its true sentiment is positive.\n",
"\n",
"Under this setting, a sentence with high sentiment score and high sarcasm score, such as \"Woooooow, what a FANTASTIC day, I hope it NEVER ends.\" will appear on the top right of the figure and be labeled as negative. Meanwhile, a genuinely good review – \"The food is good and the service is kind\" – will have a high sentiment score and low sarcasm score. So, this second review will appear on the bottom right of the figure and be labeled as positive.\n",
"\n",
"NOTE: In case you're curious, this dataset was actually generated from 4 normal\n",
"distributions with different means and unit variances."
]
},
{
"cell_type": "markdown",
"metadata": {
"pycharm": {
"name": "#%% md\n"
}
},
"source": [
"Here, let's learn this task with Scikit-learn's pre-written Logsitic Regression."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cell_id": "00007-8ff93d54-4077-4cbf-b75a-197eb6b55e7e",
"deepnote_cell_type": "code",
"deepnote_to_be_reexecuted": false,
"execution_millis": 1,
"execution_start": 1619496574863,
"id": "XKJZ9KXixF7p",
"output_cleared": true,
"source_hash": "f1714c1d",
"tags": [
"exploration"
]
},
"outputs": [],
"source": [
"from sklearn.linear_model import LogisticRegression\n",
"\n",
"# Train the logistic regression classifier on our data\n",
"logistic_regression_classifier = LogisticRegression().fit(X, np.squeeze(y))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cell_id": "00010-d2f78405-7d19-41ba-82a1-ad7edc95fa7d",
"deepnote_cell_type": "code",
"deepnote_to_be_reexecuted": false,
"execution_millis": 299,
"execution_start": 1619496579999,
"output_cleared": true,
"source_hash": "678180cf",
"tags": [
"exploration"
]
},
"outputs": [],
"source": [
"plot_points_with_classifier_predictions(X, y, logistic_regression_classifier)"
]
},
{
"cell_type": "markdown",
"metadata": {
"cell_id": "00010-3d43f862-803d-4b1b-90b7-38488568fceb",
"deepnote_cell_type": "markdown",
"id": "zsKNS6NzxXja"
},
"source": [
"In this plot, the region shaded in blue is where the classifier predicts as 0 (negative sentiment). The region shaded in red is where it predicts as 1. As one can observe, the prediction is way off!"
]
},
{
"cell_type": "markdown",
"metadata": {
"cell_id": "00012-4a4987a3-fbda-428a-9848-f5d453d018c9",
"deepnote_cell_type": "markdown",
"id": "iyE8LP0-xdS7"
},
"source": [
"## Part 1.2 One Solution: A Simple Neural Network\n",
"\n",
"A neural network is made up of a series of layers. Each layer looks\n",
"almost exactly like a logistic regression classifier, written as:\n",
"\n",
"$a_l = f(a_{l-1}W_l + b_l)$\n",
"\n",
"Where $a_l$ represents the output of layer $l$, $a_{l-1}$ represents the output from the previous layer (or input $X$ if $l-1=0$), and $f()$ is a non-linear\n",
"function called the __activation function__. The sigmoid function used in logistic regression is a kind of activation function. In fact, you can think of logistic regression as a single-layer neural network!"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Implementation roadmap\n",
"\n",
"\n",
"The following notebook code cell defines a multi-layer neural network class `StackedLogisticRegressionNetwork` you are going to implement. It is a big chunk of code and can be intimidating, since it contains all of its necessary features and methods in one piece -- an unfortunate fact of python class definitions. But do not fear! We've broken it down into steps, and have written intermediate unit tests and spot checks to guide you along the way.\n",
"\n",
"Follow the following steps and you can implement a neural network of your own! \n",
"\n",
"For your convenience, we've provided links to jump across to cells in this notebook, but in some contexts, such as Google Colab, these links do not work. In that case, please follow the \"**STEP x**' labels in the notebook.\n",
"\n",
"- **[STEP 1](#step1): Implement the `__init__()` method.** This defines and initializes the weight matrices/vectors of the neural network. You can find more in the description below.\n",
"- **[STEP 2](#step2): Run a spot check for the `__init__()` method**.\n",
"- **[STEP 3](#step3): Implement the `forward_pass()` method**. This is the main body of the neural network's computation. \n",
"- **[STEP 4](#step4): Debug/run a spot check for the the `forward_pass()` method**. This code cell provides some sample code to debug `forward_pass()`, if you're stuck.\n",
"- **[STEP 5](#step5): Implement the `backward_pass()` method**. You will implement backward propogation to compute gradients for weights in the neural network.\n",
"- **[STEP 6](#step6): Debug/run a spot check for the the `backward_pass()` method**. This code cell provides some sample code to debug `forward_pass()`, if you're stuck.\n",
"- **[STEP 7](#step7): Implement and test the `predict()` method**. This basically converts the model's final output into a binary prediction.\n",
"- **[STEP 8](#step8): Training and evaluating your neural network**. Run the training and evaluation code that we provide to train and evaluate your neural network!\n",
"\n",
"**Important coding tip 1:** When writing code for neural networks, it is useful to frequently check the shape of any intermediate vectors or tensors as you are writing them, instead of implementing the whole forward/backward method without testing in one setting and try to debug later. To do this, use a toy input to call the `forward_pass()` or `backward_pass()` function, even though it might not be completely implemented, and print out as much information as you can! We've provided some code in STEP 4 and STEP 6 for you to do this.\n",
"\n",
"**Important coding tip 2:** At all costs, avoid using for loops when you are implementing computations involving vectors or matrices. Instead, use the various matrix/vector operations that numpy provides.\n",
"\n",
"__WARNING:__ The spot checks provided above do __NOT__ provide\n",
"a guarantee that your implementation is 100% correct. We encourage you to\n",
"do additional checking and testing to convince yourself that everything\n",
"is working correctly. You can see how the checks are implemented in `util.py`."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### (STEP 1) The network architecture and initializing weights. \n",
"\n",
"The first step is to complete the `__init__()` method to defines and initializes the weight matrices/vectors of the neural network.\n",
"\n",
"Recall the previous definition of a neural network layer:\n",
"\n",
"$a_l = f(a_{l-1}W_l + b_l)$\n",
"\n",
"Where $a_l$ represents the output of layer $l$, $a_{l-1}$ represents the output from the previous layer (or input $X$ if $l-1=0$), and we use the sigmoid function for $f()$.\n",
"\n",
"Note that we are computing with examples in batch. That means, we are running multiple examples through the network at the same time. Each example occupies one row in $a_l$. \n",
"\n",
"Suppose $a_{l-1}$ has shape (`N`, `D1`) where `N` is the number of examples in $a_l$ and `D1` is the size of each example's current feature vector. Suppose $W$ has shape (`D1`, `D2`). Then $a_l$ will be of shape (`N`, `D2`). Essentially, the matrix multiplication between $a_{l-1}$ and $W$ maps each example's features of size `D1` (as a row vector in $a_{l-1}$) into a new set of features of size `D2` (as a row vector in $a_{l}$). The transformation is non-linear because $f()$ is a non-linear function, and all examples are transformed in parallel without interfering with each other. (Think about why this is the case!)\n",
"\n",
"We want to implement a feed-forward neural network with 3 layers, including 2 hidden layers and 1 output layer. We denote the output of each layer as $a_1$, $a_2$, and $a_3$. Each layer has two trainable parameters, $W$ and $b$, as we have explained before.\n",
"\n",
"After you implement `__init__()`, be sure to proceed to [STEP 2](#step2) to check your implementation is correct!"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
""
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cell_id": "00015-6cd15ba1-9492-422b-935a-be1b66bbe234",
"deepnote_cell_type": "code",
"deepnote_to_be_reexecuted": false,
"execution_millis": 42,
"execution_start": 1619508800186,
"id": "gbVnL9MBxTpK",
"output_cleared": true,
"source_hash": "c8678370"
},
"outputs": [],
"source": [
"class StackedLogisticRegressionNetwork:\n",
" def __init__(self, input_size: int, layer_1_size: int, layer_2_size: int,\n",
" seed: int = 1234):\n",
" \"\"\"\n",
" STEP 1 TODO: Initialize the weights and biases of the 3 layers. You should\n",
" initialize the weights randomly with initialize_weights() for each\n",
" layer, and initialize the biases to zero for each layer.\n",
"\n",
" Args:\n",
" input_size (int): The size of the input (number of input features/\n",
" number of dimensions of the input points).\n",
" layer_1_size (int): The number of hidden units in the first hidden\n",
" layer.\n",
" layer_2_size (int): The number of hidden units in the second hidden\n",
" layer.\n",
" seed (int): Random seed to use when initializing weights.\n",
" \"\"\"\n",
" np.random.seed(seed)\n",
"\n",
" ######################## STEP 1: YOUR CODE STARTS HERE #########################\n",
" # HINT1: You should use self.initialize_weights() to initialize Ws \n",
" # (read this function's header below for details),\n",
" # and use np.zeros() to initialize bs. \n",
" # (https://numpy.org/doc/stable/reference/generated/numpy.zeros.html)\n",
" \n",
" # HINT2: The shape of the bias for each layer should be (1, layer_size).\n",
"\n",
" # WARNING: Do NOT change the order in which the weights are initialized.\n",
" \n",
" self.W_1 = None\n",
" self.b_1 = None\n",
" self.W_2 = None\n",
" self.b_2 = None\n",
" self.W_3 = None\n",
" self.b_3 = None\n",
" ######################## STEP 1: YOUR CODE ENDS HERE ###########################\n",
" # Now proceed to STEP 2 to check your implementation\n",
" ################################################################################\n",
" \n",
" self.variables = {\n",
" \"W_1\": self.W_1,\n",
" \"b_1\": self.b_1,\n",
" \"W_2\": self.W_2,\n",
" \"b_2\": self.b_2,\n",
" \"W_3\": self.W_3,\n",
" \"b_3\": self.b_3\n",
" }\n",
"\n",
" @staticmethod\n",
" def initialize_weights(num_inputs: int, num_outputs: int) -> np.ndarray:\n",
" \"\"\"\n",
" Initialize the weights in a specific way to help gradient descent\n",
" converge better (Glorot initialization).\n",
"\n",
" Args:\n",
" num_inputs (int): The number of inputs to the layer.\n",
" num_outputs (int): The number of outputs (units) in the layer.\n",
"\n",
" Returns:\n",
" np.ndarray: A fully initialized NumPy array of weights, of shape\n",
" (num_inputs, num_outputs)\n",
" \"\"\"\n",
" bound = np.sqrt(6 /(num_inputs + num_outputs))\n",
" return np.random.uniform(-bound, bound, (num_inputs, num_outputs))\n",
"\n",
" @staticmethod\n",
" def sigmoid(x: np.ndarray) -> np.ndarray:\n",
" \"\"\"\n",
" The sigmoid function, exactly the same as in PA2. NOTE: Applied\n",
" element-wise to each element in the array x.\n",
"\n",
" Args:\n",
" x (np.ndarray): A NumPy array.\n",
"\n",
" Returns:\n",
" np.ndarray: An array of the sigmoid function applied to each element\n",
" of x. Shape is the same as x.\n",
" \"\"\"\n",
" return 1 / (1 + np.exp(-x))\n",
"\n",
" def forward_pass(self, X: np.ndarray) ->\\\n",
" Tuple[np.ndarray, np.ndarray, np.ndarray]:\n",
" \"\"\"\n",
" STEP 3 TODO: Calculate the outputs of each of the three layers in the model\n",
" using the inputs, weights, and biases.\n",
"\n",
" Args:\n",
" X (np.ndarray): The NumPy array of input examples, of shape\n",
" (num_examples, input_size).\n",
"\n",
" Returns:\n",
" Tuple[np.ndarray, np.ndarray, np.ndarray]: A tuple of three NumPy\n",
" arrays, corresponding to the output (after activation) of the\n",
" first layer ($a_1$), second layer ($a_2$), and third/output\n",
" layer ($a_3$).\n",
" \"\"\"\n",
"\n",
" ######################## STEP 3: YOUR CODE STARTS HERE #########################\n",
" # HINT1: Your output a_1 should look exactly like the output of a logistic\n",
" # regression classifier on X. Your output a_2 should look exactly like\n",
" # the output of another logistic regression classifier on the input a_1.\n",
" # And same again for a_3. Note that you should use the appropriate\n",
" # weights and biases for each part though.\n",
"\n",
" # HINT2: Don't forget to apply self.sigmoid() between each layer!\n",
" \n",
" a_1 = None\n",
" a_2 = None\n",
" a_3 = None\n",
" \n",
" ######################## STEP 3: YOUR CODE ENDS HERE ###########################\n",
" # Now proceed to STEP 4 to check your implementation\n",
" ################################################################################\n",
" \n",
" return a_1, a_2, a_3\n",
"\n",
" def backward_pass(self,\n",
" X : np.ndarray,\n",
" y : np.ndarray,\n",
" a_1: np.ndarray,\n",
" a_2: np.ndarray,\n",
" a_3: np.ndarray\n",
" ) -> Dict[str, np.ndarray]:\n",
" \"\"\"\n",
" STEP 5 TODO: Use the examples, labels, and model outputs to compute the\n",
" gradient for each of the weights and biases.\n",
"\n",
" Args:\n",
" X (np.ndarray): The NumPy array of input examples, of shape\n",
" (num_examples, input_size).\n",
" y (np.ndarray): The NumPy array of true labels for each\n",
" example in X. Has shape\n",
" (num_examples, 1).\n",
" a_1 (np.ndarray): The output of the first logistic regression\n",
" layer (after activation). Computed in\n",
" forward_pass().\n",
" a_2 (np.ndarray): The output of the second logistic regression\n",
" layer (after activation). Computed in\n",
" forward_pass().\n",
" a_3 (np.ndarray): The output of the third logistic regression\n",
" layer (after activation). Computed in\n",
" forward_pass().\n",
"\n",
" Returns:\n",
" Dict[str, np.ndarray]: A dictionary of the gradients computed for\n",
" each weight and bias parameter. The keys\n",
" are strings, the values are NumPy arrays\n",
" (see the names of the keys below).\n",
" \"\"\"\n",
"\n",
" ######################## STEP 5: YOUR CODE STARTS HERE #########################\n",
" # NOTE: We use dW_3 to denote the loss J's derivative with respect to W_3,\n",
" # db_3 to denote its derivative with respect to b_3, and so on.\n",
"\n",
" # HINT: You'll probably find it useful to compute:\n",
" # - dz_3\n",
" # - da_2\n",
" # - dz_2\n",
" # - da_1\n",
" # - dz_1\n",
" # as you do your calculations.\n",
" dW_3 = None\n",
" db_3 = None\n",
" dW_2 = None\n",
" db_2 = None\n",
" dW_1 = None\n",
" db_1 = None\n",
" ######################## STEP 5: YOUR CODE ENDS HERE ###########################\n",
" # Now proceed to STEP 6 to check your implementation.\n",
" ################################################################################\n",
"\n",
" # Return the gradients\n",
" gradients = {\n",
" \"W_3\": dW_3,\n",
" \"b_3\": db_3,\n",
" \"W_2\": dW_2,\n",
" \"b_2\": db_2,\n",
" \"W_1\": dW_1,\n",
" \"b_1\": db_1\n",
" }\n",
" return gradients\n",
"\n",
" @staticmethod\n",
" def loss(y_hat : np.ndarray,\n",
" y : np.ndarray,\n",
" epsilon: float=1e-16) -> float:\n",
" \"\"\"\n",
" Computes the average binary cross-entropy loss over all examples. This\n",
" is the same loss function we implemented for logistic regression in PA2.\n",
"\n",
" Args:\n",
" y_hat (np.ndarray): The NumPy array of our prediction scores. This\n",
" is the same as the output a_3 from\n",
" forward_pass(). It is of shape\n",
" (num_examples, 1), and\n",
" each element is between 0.0 and 1.0.\n",
" y (np.ndarray): The NumPy array of true labels for each\n",
" example in X. Has shape\n",
" (num_examples, 1). Each element is either 0.0 or\n",
" 1.0.\n",
" epsilon (float): A tiny value used to clip y_hat so that it is never\n",
" accidentally rounded to 1.0 or 0.0, causing\n",
" the log to become undefined.\n",
"\n",
" Returns:\n",
" float: The average binary cross-entropy loss over the num_examples\n",
" examples.\n",
" \"\"\"\n",
" # We clip the predictions so they can never become exactly 0 or 1.\n",
" # This would cause the log of the predictions to be undefined.\n",
" y_hat = np.clip(y_hat, epsilon, 1 - epsilon)\n",
" return -1 * np.mean(y * np.log(y_hat) + (1.0 - y) * np.log(1.0 - y_hat))\n",
"\n",
" def predict(self, X: np.ndarray) -> np.ndarray:\n",
" \"\"\"\n",
" STEP 7 TODO: Convert the prediction scores (output a_3 in self.forward_pass())\n",
" to binary predictions.\n",
"\n",
" Args:\n",
" X (np.ndarray): The NumPy array of input examples, of shape\n",
" (num_examples, input_size).\n",
"\n",
" Returns:\n",
" np.ndarray: A NumPy array of shape (num_examples, 1) containing the\n",
" predicted class for each example. Each element should\n",
" be either 0 or 1.\n",
" \"\"\"\n",
" ######################## STEP 7: YOUR CODE STARTS HERE #########################\n",
" # HINT: The predictions scores will be numbers between 0 and 1. \n",
" # For example, a prediction score of 0.8 means that the neural network thinks\n",
" # there is an 80% chance it is in class 1 and a 20% chance that it is in\n",
" # class 0. \n",
" \n",
" # Your job is to take this score and round it to an INTEGER 0 or 1.\n",
" # You should convert all scores >= 0.5 to 1 and others to 0.\n",
"\n",
" predictions = None\n",
" ######################## STEP 7: YOUR CODE ENDS HERE ###########################\n",
" # Be sure to run the spot check after STEP 7 !\n",
" ################################################################################\n",
" return predictions\n",
"\n",
" def train(self,\n",
" X_train: np.ndarray,\n",
" y_train: np.ndarray,\n",
" learning_rate: float = 0.01,\n",
" num_epochs: int = 1000,\n",
" print_every: int = 5,\n",
" verbose: bool=True):\n",
" \"\"\"\n",
" Trains the neural network using gradient descent.\n",
"\n",
" Args:\n",
" X_train (np.ndarray): The NumPy array of training examples, of shape\n",
" (num_examples, input_size).\n",
" y_train (np.ndarray): The NumPy array of true labels for each\n",
" example in X_train. Has shape\n",
" (num_examples, 1). Each element is either 0.0 or\n",
" 1.0.\n",
" learning_rate (np.ndarray): Step size parameter for gradient\n",
" descent.\n",
" num_epochs (int): The number of epochs (full passes through the\n",
" provided training data) to train the model for.\n",
" print_every (int): Number of epochs to wait between printouts of the\n",
" training loss and accuracy during training.\n",
" verbose (bool): Whether to print any training output.\n",
"\n",
" This code should look very familiar to you from PA2. It's pretty much\n",
" exactly the same code we used to train our logistic regression model,\n",
" except this time you implemented the part that computes the gradients!\n",
" \"\"\"\n",
" for epoch in range(1, num_epochs + 1):\n",
" a_1, a_2, a_3 = self.forward_pass(X_train)\n",
" gradients = self.backward_pass(X_train, y_train, a_1, a_2, a_3)\n",
" # Update the weights with gradient descent\n",
" for name, grad in gradients.items():\n",
" self.variables[name] -= learning_rate * grad\n",
" # Occasionally print out the training loss and accuracy\n",
" if verbose and epoch % print_every == 0:\n",
" loss = self.loss(a_3, y_train)\n",
" accuracy = np.mean(self.predict(X_train) == y_train)\n",
" print(\"Epoch: {} | Training Loss: {} | Training Accuracy: {}\".format(\n",
" epoch, loss, accuracy))\n",
" print(\"Finished training!\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### (STEP 2) Spot check for `__init__()` \n",
"\n",
"Run the following cell (`check_parameter_shapes`) to check if you've implemented `__init__()` correctly.\n",
"\n",
"Specifically, it checks the shapes of all trainable parameters Ws and bs.\n",
"\n",
"If all things are correct, proceed to **[STEP 3](#step3)** below!\n",
"\n",
"(Click [here](#roadmap) to return to the implementation roadmap that contains an outline of all the steps.)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"tags": [
"exploration"
]
},
"outputs": [],
"source": [
"# STEP 2\n",
"check_parameter_shapes(StackedLogisticRegressionNetwork)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### (STEP 3) The forward pass \n",
"\n",
"The next step is to implement the method `forward_pass()`, i.e. the main body of the neural network's computation.\n",
"\n",
"The network runs the input $X$ ($\\in \\mathbb{R}^{\\text{num\\_examples} \\times \\text{input\\_size}}$)\n",
"through its layers to get a probability vector $y$\n",
"($\\in \\mathbb{R}^{\\text{num\\_examples} \\times 1}$).\n",
"The $i$-th value in $y$ should indicate the probability of the $i$-th example\n",
"in $X$ to express a positive sentiment.\n",
"\n",
"We have done all the math for you, so you only need to implement these equations in numpy. Think about the shape of each intermediate variable, and use `print(x.shape)` to debug if you encounter \"shape mismatch\" errors.\n",
"\n",
"$z_1 = X W_1 + b_1$\n",
"\n",
"$a_1 = \\sigma(z_1)$\n",
"\n",
"$z_2 = a_1 W_2 + b_2$\n",
"\n",
"$a_2 = \\sigma(z_2)$\n",
"\n",
"$z_3 = a_2 W_3 + b_3$\n",
"\n",
"$y = a_3 = \\sigma(z_3)$\n",
"\n",
"**Important coding tip**: All the multiplications here are matrix multiplication. There are three ways to perform matrix multiplication in numpy. Suppose you want to multiply $A \\in R^{mxn}$ and $B \\in R^{nxk}$, the following expressions are equivalent:\n",
"\n",
"`A @ B`\n",
"\n",
"`A.dot(B)`\n",
"\n",
"`np.matmul(A,B)`\n",
"\n",
"After implementing STEP 3, proceed to [STEP 4](#step4) to check your implementation. \n",
"\n",
"**Important coding tip:** as mentioned before, when implementing the above, it is useful to call `forward_pass()` with toy data, and print out the shape of any intermediate vectors or tensors as you are writing them, instead of implementing the whole forward/backward method without testing in one setting and try to debug later. To do this, check out the commented out code in STEP 4.\n",
"\n",
"- Click [here](#nn_code) to jump to the notebook cell containing the neural network definition.\n",
"- Click [here](#roadmap) to return to the implementation roadmap."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### (STEP 4) Debugging and spot check for forward pass \n",
"\n",
"This checks if the outputs `a_1`, `a_2` and `a_3` of the forward function are correct.\n",
"\n",
"If the spot check passes, and you've double-checked your implementation, proceed to [STEP 5](#step5)!"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"tags": [
"exploration"
]
},
"outputs": [],
"source": [
"# STEP 4: spot check for forward_pass()\n",
"check_forward_pass(StackedLogisticRegressionNetwork, X)\n",
"\n",
"# you can uncomment and play around with the following code to \n",
"# temporarily construct a neural network, and call forward_pass()\n",
"# with some toy data (X) for debugging purposes!\n",
"\n",
"# test_network = StackedLogisticRegressionNetwork(2, 2, 2)\n",
"# a_1, a_2, a_3 = test_network.forward_pass(X[:3, :])\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### (STEP 5) The backward pass \n",
"\n",
"The next step is to implement the backward propogation for your neural network in the method `backward_pass()`.\n",
"\n",
"Recall from the modules and reading that we train a neural network via stochastic gradient \n",
"descent -- the exact same way we trained logistic regression in PA2. \n",
"\n",
"For this problem, we define the loss using the binary cross entropy\n",
"cost function, defined as:\n",
"\n",
"$J = -\\frac{1}{m} \\sum_{i=1}^m(y \\log(a_3)+(1-y) \\log(1-a_3)) _i$\n",
"\n",
"(Note m is the batch size, i.e., the number of examples in the input $X$. We divide the summed loss by $m$ to average over examples.)\n",
"\n",
"During the backward pass, we compute the gradient of the loss with respect to each \n",
"trainable parameter, and use this gradient to update the parameter. For example,\n",
"The gradient of $J$ with respect to $W_1$ is denoted as $\\frac{dJ}{dW_1}$. This gradient should have the same shape as $W_1$. After we obtain this value, we update $W_1$ via:\n",
"\n",
"$W_1 = W_1 - \\eta \\frac{dJ}{dW_1}$\n",
"\n",
"Where $\\eta$ is the learning rate. The good news is that you don't need to code these\n",
"updates by hand -- we've already implemented it for you in train()!\n",
"\n",
"(Note: You may have learned in a calculus class that $\\frac{dJ}{dW_1}$ denotes a 'vanilla' derivative,\n",
"while $\\frac{\\partial J}{\\partial W_1}$ denotes a partial derivative. In this PA,\n",
"we use the two notations interchangeably -- you don't need to worry about the difference!)\n",
"\n",
"In `backward_pass()`, all you need to compute are the gradients of the loss\n",
"with respect to each $W$ and $b$! We understand that matrix calculus can be challenging to\n",
"many of you, so we provide the computated results below. If you are interested in learning to do\n",
"these computations by hand, you can refer to this tutorial by CS231N: https://cs231n.github.io/optimization-2/\n",
"\n",
"When you translate these formulas into numpy code, pay special attention to the shapes of your derivatives! A derivative of a scalar (remember that $J$ is a scalar) with respect to a variable should have the same shape of that variable.\n",
"\n",
"**Important coding tip:** as mentioned before, when implementing the above, it is useful to call `backward_pass()` with toy data, and print out the shape of any intermediate vectors or tensors as you are writing them, instead of implementing the whole forward/backward method without testing in one setting and try to debug later. To do this, check out the commented out code in [STEP 6](#step6).\n",
"\n",
"---\n",
"\n",
"**Mathematical equations for back-propogation**\n",
"\n",
"_Life hack: to avoid scrolling up and down between the code and the following equations, you can open this notebook in two separate windows side by side._ :p\n",
"\n",
"$\\frac{dJ}{dz_3} = \\frac{1}{m} (a_3 - y)$ \n",
"\n",
"(Note $m$ is the batch size, i.e., the original number of examples in input $X$)\n",
"\n",
"$\\frac{dJ}{dW_3} = a_{2}^T \\frac{dJ}{dz_3}$\n",
"\n",
"(Check out `np.transpose()`)\n",
"\n",
"$\\frac{dJ}{db_3} = \\sum_{i=1}^m (\\frac{dJ}{dz_3})_i$\n",
"\n",
"(Hint: check out `np.sum()`)\n",
"\n",
"$\\frac{dJ}{da_2} = \\frac{dJ}{dz_3} W_3^T$\n",
"\n",
"(This denotes a product between a vector and another vector _transposed_. Generally, given any two vectors $x \\in \\mathbb{R}^n, y \\in \\mathbb{R}^m$, when we compute $x y^T$, we treat $x$ and $y$ as matrices with dimensions $(n, 1)$ and $(m, 1)$ respectively. Then $y^T$ should have a dimension of $(1, m)$. If we treat $x$ and $y^T$ both as matrices, what is the dimension of their product?)\n",
"\n",
"$\\frac{dJ}{dz_2} = \\frac{dJ}{da_2} \\circ a_2 \\circ (\\mathbb{1}-a_2)$\n",
"\n",
"($\\mathbb{1}$ indicates an all-1 vector of your desired shape. However, you don't need to construct this vector. Try running \"1 - np.array([1, 4, 5])\" in a separate cell and check out the result.)\n",
"\n",
"($\\circ$ denotes element-wise multiplication. You can do this with the `*` operator in python.)\n",
"\n",
"$\\frac{dJ}{dW_2} = (a_{1}^T \\frac{dJ}{dz_2})$\n",
"\n",
"$\\frac{dJ}{db_2} = \\sum_{i=1}^m (\\frac{dJ}{dz_2})_i$\n",
"\n",
"(Since $\\frac{dJ}{dz_2}$ is a matrix, here we are summing up all its rows, because each row corresponds to an input. Checkout the `axis` argument in `np.sum()`.)\n",
"\n",
"$\\frac{dJ}{da_1} = \\frac{dJ}{dz_2} W_2^T$\n",
"\n",
"$\\frac{dJ}{dz_1} = \\frac{dJ}{da_1} \\circ a_1 \\circ (\\mathbb{1}-a_1)$\n",
"\n",
"$\\frac{dJ}{dW_1} = (X^T \\frac{dJ}{dz_1})$\n",
"\n",
"$\\frac{dJ}{db_1} = \\sum_{i=1}^m (\\frac{dJ}{dz_1})_i$\n",
"\n",
"(Similar to the above, we are summing up rows in the matrix $\\frac{dJ}{dz_1}$.)\n",
"\n",
"---\n",
"\n",
"After implementing STEP 5, proceed to [STEP 6](#step6) to check your implementation.\n",
"\n",
"- Click [here](#nn_code) to jump to the notebook cell containing the neural network definition.\n",
"- Click [here](#roadmap) to return to the implementation roadmap."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### (STEP 6) Spot check for backward pass \n",
"\n",
"The following spot check will test if the gradients computed by your `backward_pass()` are correct."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"tags": [
"exploration"
]
},
"outputs": [],
"source": [
"# STEP 6: spot check for forward_pass()\n",
"check_backward_pass(StackedLogisticRegressionNetwork, X, y)\n",
"\n",
"# you can uncomment and play around with the following code to \n",
"# temporarily construct a neural network, and call backward_pass()\n",
"# with some toy data (X). With this you can manually check \n",
"# or debug your implementation!\n",
"\n",
"# Note: you should make sure that the forward_pass implementation \n",
"# is working.\n",
"\n",
"# test_network = StackedLogisticRegressionNetwork(2, 2, 2)\n",
"# a_1, a_2, a_3 = test_network.forward_pass(X[:3, :])\n",
"# test_network_gradients = test_network.backward_pass(X[:3, :], y[:3], a_1, a_2, a_3)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### (STEP 7) Implement and test `predict()` \n",
"Finally, implement `predict()`, which convert's the model's final output into a binary prediction. Read the method's description for what you need to do. Hint: You will need to call `forward_pass()`.\n",
"\n",
"Don't forget to run the follwing spot check!\n",
"\n",
"After this step, proceed to [STEP 8](#step8) where you run the script that we've provided to train and test your neural network!\n",
"\n",
"- Click [here](#nn_code) to jump to the notebook cell containing the neural network definition.\n",
"- Click [here](#roadmap) to return to the implementation roadmap."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"tags": [
"exploration"
]
},
"outputs": [],
"source": [
"# STEP 7: spot check for predict()\n",
"check_predict(StackedLogisticRegressionNetwork, X)\n",
"\n",
"# you can uncomment and play around with the following code to \n",
"# temporarily construct a neural network, and call predict()\n",
"# with some toy data (X). With this you can manually check \n",
"# or debug your implementation!\n",
"\n",
"# test_network = StackedLogisticRegressionNetwork(2, 2, 2)\n",
"# test_predict = test_network.predict(X[:3, :])"
]
},
{
"cell_type": "markdown",
"metadata": {
"cell_id": "00024-12f5fe45-a38d-44cb-a060-d9fbba2be4f9",
"deepnote_cell_type": "markdown",
"tags": []
},
"source": [
"### (STEP 8) Training and evaluating your neural network \n",
"\n",
"If you completed all of the above steps, congratulations! In all likelihood, you have\n",
"just written a working neural network! Now for the final test:\n",
"__training and testing it__.\n",
"\n",
"Just like our logistic regression classifier in PA2, we can train our model\n",
"with gradient descent. You can check out the train() function to see how it\n",
"works (basically the same as in PA2)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cell_id": "00020-7f172fb5-0575-4c07-9dd7-e81f240701ae",
"colab": {
"base_uri": "https://localhost:8080/"
},
"deepnote_cell_type": "code",
"deepnote_to_be_reexecuted": false,
"execution_millis": 2480,
"execution_start": 1619497092993,
"id": "Hqx5hPqCxwUT",
"outputId": "c5507e9d-1875-4f51-ab99-aafe94798f9a",
"output_cleared": true,
"source_hash": "9c83eb1",
"tags": [
"exploration"
]
},
"outputs": [],
"source": [
"# We choose the size of the 2nd and 3rd layers to be 10. More on this below.\n",
"neural_network_classifier = StackedLogisticRegressionNetwork(2, 10, 10)\n",
"neural_network_classifier.train(X, y,\n",
" learning_rate=0.1,\n",
" num_epochs=10000,\n",
" print_every=1000)"
]
},
{
"cell_type": "markdown",
"metadata": {
"cell_id": "00021-d7b10a16-e4c4-44b4-86fe-57254bdf6b90",
"deepnote_cell_type": "markdown",
"id": "HutayYClxrk5"
},
"source": [
"Now let's plot the resulting predictions, just like we did with the logistic\n",
"regression classifier:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cell_id": "00022-3f2565ba-c19c-4505-8a7c-0147727c9bab",
"deepnote_cell_type": "code",
"deepnote_to_be_reexecuted": false,
"execution_millis": 2700,
"execution_start": 1619497097594,
"id": "RdfY6_O_x1r8",
"output_cleared": true,
"source_hash": "ed6a4ad4",
"tags": [
"exploration"
]
},
"outputs": [],
"source": [
"plot_points_with_classifier_predictions(X, y, neural_network_classifier)"
]
},
{
"cell_type": "markdown",
"metadata": {
"cell_id": "00023-db94a56d-a2e4-4cce-afc8-4edb1ac9697c",
"deepnote_cell_type": "markdown",
"id": "PLwgtmy2x4Bv"
},
"source": [
"Wow! This worked a lot better than logistic regression!\n",
"\n",
"The non-linear boundary is what allows us to correctly\n",
"classify this dataset. No single straight line could correctly put all the red\n",
"points on one side and all blue ones on the other side, but a curved line can!"
]
},
{
"cell_type": "markdown",
"metadata": {
"cell_id": "00029-54c1f607-7082-4804-ba90-4b9f53452237",
"deepnote_cell_type": "markdown",
"tags": []
},
"source": [
"### Some Final Thoughts on Part 1\n",
"\n",
"We chose input size of 2 and hidden layer sizes of 10 to learn this task.\n",
"\n",
"The input size (size of the first logistic regression layer)\n",
"is determined by $X$. We had 2-dimensional inputs, so the input size\n",
"can only be 2.\n",
"\n",
"But for the second and third logistic regression layers, we're free\n",
"to choose any size we'd like! Choosing larger sizes means more parameters, resulting in potentially more complex and fancy decision boundaries. But it also means more numbers to crunch when calculating the gradients, and thus a longer training time and potential overfitting.\n",
"\n",
"In practice, there's no strict rule for chooosing layer sizes.\n",
"Usually you just have to experiment with some different choices\n",
"and see what works well!\n",
"\n",
"__[Try this:]__ We'd encourage you to go back and try changing the layer sizes\n",
"and see what happens to your model's training and predictions! What happens\n",
"when you choose super tiny layer sizes `(1, 2)`? What about super large ones\n",
"`(100, 1000, 10000)`?\n",
"\n",
"Finally, note that no matter what sizes we choose for the second and third\n",
"layers, the final output has to be a single number (prediction score) for each\n",
"example in $X$, which is why the last layer's size is determined for us."
]
},
{
"cell_type": "markdown",
"metadata": {
"cell_id": "00024-d61e9093-b673-4114-a264-fe7fd937cf65",
"deepnote_cell_type": "markdown",
"id": "C9v6cGlr90hL"
},
"source": [
"## Part 2: Yelp Review Classification\n",
"\n",
"Now that we've seen what neural networks can do, it's time to apply them to\n",
"some real language data! More specifically, we'll return to sentiment analysis,\n",
"which you should be very familiar with from PA2. However, now we'll see how a\n",
"neural network fares on this task.\n",
"\n",
"More specifically, our goal will be to take in data in the form of Yelp reviews\n",
"of various businesses and predict, just from the text of the review itself,\n",
"whether the reviewer's opinion was positive or negative!\n",
"\n",
"Before so, let's take one last detour to the world of vector semantics."
]
},
{
"cell_type": "markdown",
"metadata": {
"cell_id": "00025-5297a1aa-23c6-44c2-9648-4fe198273d16",
"deepnote_cell_type": "markdown",
"id": "KBSmYXvU5hUu"
},
"source": [
"### Word Embeddings\n",
"\n",
"You'll recall from PA4 that semantic word vectors, or word embeddings, are a\n",
"very useful and powerful way to represent words.\n",
"We showed how you can look at the geometric properties of these vectors and use\n",
"them to show relationships between words such as analogy.\n",
"\n",
"Embeddings are a key building block for solving all sorts of other language and\n",
"NLP-related problems. Once we've transformed words and text into numerical\n",
"values, we can use them to construct example features at the level of word, sentence, or longer texts.\n",
"\n",
"For this assignment, we use GloVe, a set of pre-computed word embeddings and apply bag-of-words to construct sentence features."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cell_id": "00028-e834f9b0-e9ab-4975-bab3-4c927bf64c7f",
"deepnote_cell_type": "code",
"deepnote_to_be_reexecuted": false,
"execution_millis": 2706,
"execution_start": 1619508857627,
"id": "Ya7K0WAk5qzQ",
"output_cleared": true,
"source_hash": "bcfc1941",
"tags": [
"exploration"
]
},
"outputs": [],
"source": [
"# Load in our embeddings from a text file\n",
"embeddings = load_embeddings(\"data/glove.dataset_small.50d.txt\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"pycharm": {
"name": "#%% md\n"
}
},
"source": [
"First let's learn to look up individual word embeddings!\n",
"If you compute a dot product, you will find that \"good\" is much more similar to \"nice\" than \"terrible\" in terms of cosine similarity."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cell_id": "00029-0a4b0dc6-1319-427c-a241-b59d106c94f1",
"deepnote_cell_type": "code",
"deepnote_to_be_reexecuted": false,
"execution_millis": 10,
"execution_start": 1619497150059,
"id": "GRCNuGgk5sRA",
"output_cleared": true,
"source_hash": "87f61896",
"tags": [
"exploration"
]
},
"outputs": [],
"source": [
"# Print out some of our embeddings\n",
"words = ['terrible', 'good', 'nice']\n",
"\n",
"for word in words:\n",
" print(\"{} : {}\".format(word, embeddings[word]))"
]
},
{
"cell_type": "markdown",
"metadata": {
"cell_id": "00030-cc0dfb7d-13ce-4575-9f03-f91b839b83da",
"deepnote_cell_type": "markdown",
"id": "npNN-WzK5uMG"
},
"source": [
"This embedding set also contains a special token, ``, which represents any word that isn't in the vocabulary. The embedding for `` is just the average of the embeddings of all the other words."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cell_id": "00031-5479adc4-c31d-484d-95d3-e8438c125c5e",
"deepnote_cell_type": "code",
"deepnote_to_be_reexecuted": false,
"execution_millis": 10,
"execution_start": 1619497153647,
"id": "WFNWajr15wDc",
"output_cleared": true,
"source_hash": "80a3fe75",
"tags": [
"exploration"
]
},
"outputs": [],
"source": [
"print(\" embedding : {}\".format(embeddings[\"\"]))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"You can check whether a word is in vocabulary following the example below:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"tags": [
"exploration"
]
},
"outputs": [],
"source": [
"print(\"nice\" in embeddings)\n",
"print(\"cs124\" in embeddings)"
]
},
{
"cell_type": "markdown",
"metadata": {
"cell_id": "00040-4c4b4c7c-e79b-4e5d-819c-e13db2b175f3",
"deepnote_cell_type": "markdown",
"tags": []
},
"source": [
"### Preparing the Data\n",
"\n",
"Now we have a way to translate our text data into a form that our neural\n",
"network can use! Let's load up some data and do just that.\n",
"\n",
"The code below loads our dataset from the file (see `util.py` for details):"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cell_id": "00041-e9880d42-a75c-45df-a13f-6124c35e542b",
"deepnote_cell_type": "code",
"deepnote_to_be_reexecuted": false,
"execution_millis": 1468,
"execution_start": 1619508836599,
"output_cleared": true,
"source_hash": "1f06d4ea",
"tags": [
"exploration"
]
},
"outputs": [],
"source": [
"train_examples, train_labels = load_dataset(\"data/train.csv\")\n",
"dev_examples, dev_labels = load_dataset(\"data/dev.csv\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"cell_id": "00043-1ae2f230-dfb4-45f8-99ed-b28f7fa4b1ed",
"deepnote_cell_type": "markdown",
"tags": []
},
"source": [
"Each element in `train_examples` is a Yelp review that we've preprocessed into a\n",
"list of words. We've already taken out punctuation, segmented the words, and\n",
"converted them all to lower case for you, to make them easier to work with.\n",
"\n",
"Each element in `train_labels` is the review's corresponding positive/negative\n",
"label. We've transformed the original star ratings into two labels (0 or 1) by collapsing 1 and 2-star reviews into the negative class (label 0) and 4 and 5-star reviews into the positive class (label 1). We ignore 3-star reviews for simplicity.\n",
"\n",
"`dev_examples` and `dev_labels` are a separate development set with the exact same format.\n",
"\n",
"We can examine a couple reviews and their ratings:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cell_id": "00044-7461922c-3eaa-41ac-a02c-d3e9e1f27340",
"deepnote_cell_type": "code",
"deepnote_to_be_reexecuted": false,
"execution_millis": 9,
"execution_start": 1619508838826,
"output_cleared": true,
"source_hash": "6862c178",
"tags": [
"exploration"
]
},
"outputs": [],
"source": [
"EXAMPLES_TO_PRINT = 5\n",
"\n",
"for text, label in zip(train_examples[:EXAMPLES_TO_PRINT],\n",
" train_labels[:EXAMPLES_TO_PRINT]):\n",
" print(\"Review: {}\".format(\" \".join(text)))\n",
" print(\"Sentiment: {}\".format(label))\n",
" print(\"\\n\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"cell_id": "00045-893663f0-3d4d-42cc-b226-d31ce1c7968e",
"deepnote_cell_type": "markdown",
"tags": []
},
"source": [
"### Now's your turn! Let's convert these reviews into fixed-dimensional feature vectors!\n",
"\n",
"We can do this in two steps:\n",
"\n",
"First, convert each word in the review into word embeddings. Make sure you handle unknown words with the `` embedding because not all words are in the vocabulary. You'll implement this step in `examples_to_array()`.\n",
"\n",
"Next, we aggregate all word embeddings to construct an overall representation for the review, so that different reviews can have the same feature size despite of varying lengths. You'll implement three aggregation modes in `aggregate_embeddings()`, including mean, sum, and max."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cell_id": "00041-0d21fdbe-9336-4f51-90c1-4ce0cb7f2458",
"deepnote_cell_type": "code",
"deepnote_to_be_reexecuted": false,
"execution_millis": 2,
"execution_start": 1619508840946,
"output_cleared": true,
"source_hash": "bfcaebbd",
"tags": []
},
"outputs": [],
"source": [
"def aggregate_embeddings(list_of_embeddings: List[np.ndarray],\n",
" mode: str=\"mean\") -> np.ndarray:\n",
" \"\"\"\n",
" TODO: Take a review in the form of a list of embedding vectors\n",
" and return a vector (of the same shape as a single embedding vector)\n",
" that represents the entire sentence.\n",
"\n",
" You need to support 3 modes: \"mean\", \"sum\", and \"max\".\n",
"\n",
" Args:\n",
" list_of_embeddings (List[np.ndarray]): A list of word embeddings, which\n",
" collectively represent a single\n",
" Yelp review. The list includes\n",
" one word embedding for each\n",
" word in the review. Each word\n",
" embedding is a vector of\n",
" shape (50,).\n",
" mode (str): How to aggregate all of the word embeddings in the review\n",
" together. There are only 3 valid modes: mean, sum, and max.\n",
" Mean and sum are self-explanatory. Max takes the\n",
" element-wise max of the embeddings.\n",
"\n",
" Returns:\n",
" np.ndarray: A single vector representing the entire Yelp review. It\n",
" should be of shape (50,), the same shape as a single word embedding.\n",
"\n",
" HINT: Refer to the following methods. Note that you will need to fill in the \n",
" axis optional parameter to get desired outputs.\n",
" np.mean():\n",
" https://numpy.org/doc/stable/reference/generated/numpy.mean.html\n",
" np.sum():\n",
" https://numpy.org/doc/stable/reference/generated/numpy.sum.html\n",
" np.amax():\n",
" https://numpy.org/doc/stable/reference/generated/numpy.amax.html\n",
" \"\"\"\n",
"\n",
" ######################## YOUR CODE STARTS HERE #############################\n",
" if mode == \"mean\":\n",
" aggregated = None\n",
" elif mode == \"sum\":\n",
" aggregated = None\n",
" elif mode == \"max\":\n",
" aggregated = None\n",
" else:\n",
" raise ValueError(\"Invalid mode: {}\".format(mode))\n",
" ######################## YOUR CODE END HERE ################################\n",
" return aggregated\n",
"\n",
"def examples_to_array(examples: List[List[str]],\n",
" embeddings: gensim.models.KeyedVectors,\n",
" mode: str=\"mean\"):\n",
" \"\"\"\n",
" TODO: Take a list of Yelp reviews, where each review is a list\n",
" of words, and convert it into an array of shape (num_examples,\n",
" embedding size) where each row is the aggregated embedding representing\n",
" that review.\n",
"\n",
" Args:\n",
" examples (List[List[str]]): A list of examples, where each example is\n",
" a Yelp review. Each example/review is\n",
" represented as a list of words, where each\n",
" word is a separate string. This is the\n",
" first output of the load_dataset() function\n",
" above.\n",
" embeddings (gensim.models.KeyedVectors): The embeddings object loaded\n",
" earlier with load_embeddings().\n",
"\n",
" mode (str): The mode argument to pass in to aggregate_embeddings().\n",
"\n",
" Returns:\n",
" np.ndarray: A single NumPy array containing the vector for each example\n",
" (review) in the list of examples. Should be of shape\n",
" (num_examples, 50).\n",
"\n",
" HINT: You need to convert EACH review example in examples to a 50-dimensional \n",
" feature vector. You'll need the aggregate_embeddings() function implemented above.\n",
" \"\"\"\n",
" ######################## YOUR CODE STARTS HERE #############################\n",
" examples_array = None\n",
" ######################## YOUR CODE END HERE ################################\n",
" return examples_array"
]
},
{
"cell_type": "markdown",
"metadata": {
"cell_id": "00047-2abf4de2-48ca-486c-a6cb-fc3bc51b6509",
"deepnote_cell_type": "markdown",
"tags": []
},
"source": [
"You can test your implementation with this check (again, this is not a foolproof\n",
"guarantee of correctness):"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cell_id": "00048-a2211057-464f-4ac1-b4bb-83ed12f6cb92",
"deepnote_cell_type": "code",
"deepnote_to_be_reexecuted": false,
"execution_millis": 11,
"execution_start": 1619508863998,
"output_cleared": true,
"source_hash": "848123eb",
"tags": [
"exploration"
]
},
"outputs": [],
"source": [
"check_examples_to_array(examples_to_array, embeddings)"
]
},
{
"cell_type": "markdown",
"metadata": {
"cell_id": "00054-40f5937c-e4c4-4ada-b87e-ec61fb27555c",
"deepnote_cell_type": "markdown",
"tags": []
},
"source": [
"Fantastic! Let's finish up by pre-processing our entire dataset with the code\n",
"that you wrote:\n",
"\n",
"__This may take a few seconds.__"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cell_id": "00033-3726aa27-a13c-4dc1-8666-457ae87e953b",
"deepnote_cell_type": "code",
"deepnote_to_be_reexecuted": false,
"execution_millis": 11627,
"execution_start": 1619508866473,
"id": "IHjHbmKw50CD",
"output_cleared": true,
"source_hash": "1d5f358",
"tags": [
"exploration"
]
},
"outputs": [],
"source": [
"# Note that we use expand_dims to change the shape of the labels from\n",
"# (num_examples,) to (num_examples, 1) as our model requires.\n",
"train_examples_array = examples_to_array(train_examples, embeddings)\n",
"train_labels_array = np.expand_dims(np.array(train_labels), axis=1)\n",
"\n",
"dev_examples_array = examples_to_array(dev_examples, embeddings)\n",
"dev_labels_array = np.expand_dims(np.array(dev_labels), axis=1)"
]
},
{
"cell_type": "markdown",
"metadata": {
"cell_id": "00034-996a205e-b71f-401e-bd38-d6d67d1004cc",
"deepnote_cell_type": "markdown",
"id": "U1nNK_Rz51qC"
},
"source": [
"### TODO: Yelp Review Classification Model\n",
"\n",
"And now all we need to do is tweak our previous model slightly to work on our\n",
"new problem. We'll need a couple of things:\n",
"\n",
"1. New activation function\n",
"\n",
"In our previous network, we applied the sigmoid function 3 times, once for\n",
"each layer. This time, we'll use a different non-linear function (the __ReLU__\n",
"function) for hidden layers, but still use sigmoid on our final output. ReLU is \n",
"a very common activation function in real-world neural networks, so it should help our model work better on this more difficult task).\n",
"\n",
"2. New training scheme\n",
"\n",
"We'll use a slightly different training method (mini-batch gradient descent\n",
"instead of gradient descent), because our dataset is much larger now and it\n",
"will be difficult to do the calculations on the whole dataset at once. \n",
"\n",
"3. New weight initialization\n",
"\n",
"We'll use a slightly different way of initializing the weights, so that the\n",
"initialization is better suited to our new ReLU activation functions.\n",
"\n",
"**We've taken care of 2 and 3 for you, so your only need to focus on task 1!**"
]
},
{
"cell_type": "markdown",
"metadata": {
"cell_id": "00057-d333c5fc-b733-4d65-844d-7819a9b15d2e",
"deepnote_cell_type": "markdown",
"tags": []
},
"source": [
"### Some Helpful Information:\n",
"\n",
"__ReLU__: The ReLU (rectified linear unit) function is a piecewise function, defined as:\n",
"\n",
"$ReLU(x) = \\max(0, x)$\n",
"\n",
"The gradient of relu with respect to input is:\n",
"\n",
"$\\frac{d}{dx} ReLU(x) = \\begin{cases}\n",
" 0 &\\text{if } x < 0 \\\\\n",
" 1 &\\text{if } x \\geq 0\n",
"\\end{cases}$\n",
"\n",
"Similar to sigmoid, when we apply the ReLU function to an array, we apply it to every\n",
"element in the array separately. The same applies to gradient computation.\n",
"\n",
"For example, if $x=[1,-5,-3,2]$, we will have\n",
"\n",
"$ReLU(x) = [1,0,0,2]$\n",
"\n",
"$\\frac{d}{dx} ReLU(x) = [1,0,0,1]$"
]
},
{
"cell_type": "markdown",
"metadata": {
"cell_id": "00054-8e538535-5cc9-4a49-b3e0-28d7dee6b2d2",
"deepnote_cell_type": "markdown",
"tags": []
},
"source": [
"### The Math, Version 2\n",
"\n",
"The computations required in the forward and backward pass in this new model\n",
"are very similar to those in Part 1. The only difference comes in where we change\n",
"the sigmoid() into ReLU(). Again, we have done all the computation for you, but \n",
"you should try to do them by hand if you are interested. These values are not that\n",
"hard to compute, and can provide great intro to your future neural network classes\n",
"if you keep on studying them!\n",
"\n",
"For our modified network, the forward pass equations are:\n",
"\n",
"$z_1 = X W_1 + b_1$\n",
"\n",
"$a_1 = ReLU(z_1)$\n",
"\n",
"$z_2 = a_1 W_2 + b_2$\n",
"\n",
"$a_2 = ReLU(z_2)$\n",
"\n",
"$z_3 = a_2 W_3 + b_3$\n",
"\n",
"$a_3 = \\sigma(z_3)$\n",
"\n",
"The loss function is still the same (binary cross-entropy loss):\n",
"\n",
"$J = \\frac{-1}{m} \\sum_{i=1}^m(y \\log(a_3)+(1-y) \\log(1-a_3)) _i$\n",
"\n",
"And the gradients computed during the backward pass are:\n",
"\n",
"$\\frac{dJ}{dz_3} = \\frac{1}{m} (a_3 - y)$\n",
"\n",
"$\\frac{dJ}{dW_3} = a_{2}^T \\frac{dJ}{dz_3}$\n",
"\n",
"$\\frac{dJ}{db_3} = \\sum_{i=1}^m (\\frac{dJ}{dz_3})_i$\n",
"\n",
"$\\frac{dJ}{da_2} = \\frac{dJ}{dz_3} W_3^T$\n",
"\n",
"$\\frac{dJ}{dz_2} = \\frac{dJ}{da_2} \\circ 1\\{a_2 > 0\\}$\n",
"\n",
"(1\\{a_2 > 0\\} is an identity function, yielding an output of the exact same shape as $a_2$. The positions where $a_2$ is greater than 0 become 1 in the output, and those not greater than 0 become 0. Read the ReLU definition and derivative above for reference.)\n",
"\n",
"$\\frac{dJ}{dW_2} = a_{1}^T \\frac{dJ}{dz_2}$\n",
"\n",
"$\\frac{dJ}{db_2} = \\sum_{i=1}^m (\\frac{dJ}{dz_2})_i$\n",
"\n",
"$\\frac{dJ}{da_1} = \\frac{dJ}{dz_2} W_2^T$\n",
"\n",
"$\\frac{dJ}{dz_1} = \\frac{dJ}{da_1} \\circ 1\\{a_1 > 0\\}$\n",
"\n",
"$\\frac{dJ}{dW_1} = X^T \\frac{dJ}{dz_1}$\n",
"\n",
"$\\frac{dJ}{db_1} = \\sum_{i=1}^m (\\frac{dJ}{dz_1})_i$"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cell_id": "00035-617a352f-3788-4a7c-aaec-73718e1952ba",
"deepnote_cell_type": "code",
"deepnote_to_be_reexecuted": false,
"execution_millis": 2,
"execution_start": 1619508809426,
"id": "WGLteCTl54uv",
"output_cleared": true,
"source_hash": "b185568d"
},
"outputs": [],
"source": [
"class YelpClassificationNeuralNetwork(StackedLogisticRegressionNetwork):\n",
" def __init__(self, *args, **kwargs):\n",
" super(YelpClassificationNeuralNetwork, self).__init__(*args, **kwargs)\n",
"\n",
" @staticmethod\n",
" def initialize_weights(num_inputs: int, num_outputs: int) -> np.ndarray:\n",
" \"\"\"\n",
" We overwrite the previous weight initialization approach with a slightly\n",
" different one that works better with our new ReLU activations (He\n",
" initialization).\n",
"\n",
" Args:\n",
" num_inputs (int): The number of inputs to the layer.\n",
" num_outputs (int): The number of outputs (units) in the layer.\n",
"\n",
" Returns:\n",
" np.ndarray: A fully initialized NumPy array of weights, of shape\n",
" (num_inputs, num_outputs)\n",
" \"\"\"\n",
" stddev = np.sqrt(2 /(num_inputs + num_outputs))\n",
" return np.random.normal(0, stddev, (num_inputs, num_outputs))\n",
"\n",
" @staticmethod\n",
" def relu(x: np.ndarray) -> np.ndarray:\n",
" \"\"\"\n",
" TODO: Implement the ReLU function.\n",
"\n",
" Args:\n",
" x (np.ndarray): A NumPy array of any shape.\n",
"\n",
" Returns:\n",
" np.ndarray: A new NumPy array where each element is the output\n",
" of applying ReLU to the corresponding element of x.\n",
" The shape of the output should be identical to the\n",
" shape of x.\n",
" \"\"\"\n",
" ######################## YOUR CODE STARTS HERE #########################\n",
" relu_x = None\n",
"\n",
" ######################## YOUR CODE END HERE ############################\n",
" return relu_x\n",
"\n",
" def forward_pass(self, X: np.ndarray\n",
" ) -> Tuple[np.ndarray, np.ndarray, np.ndarray]:\n",
" \"\"\"\n",
" TODO: Calculate the outputs of each of the three layers in the model\n",
" using the inputs, weights, and biases.\n",
"\n",
" Args:\n",
" X (np.ndarray): The NumPy array of input examples, of shape\n",
" (num_examples, input_size).\n",
"\n",
" Returns:\n",
" Tuple[np.ndarray, np.ndarray, np.ndarray]: A tuple of three NumPy\n",
" arrays, corresponding to the output (after activation) of the\n",
" first layer ($a_1$), second layer ($a_2$), and third/output\n",
" layer ($a_3$).\n",
"\n",
" HINT: You implementation should look almost like Part1, except for replacing \n",
" some (not all) sigmoids with ReLU.\n",
" \"\"\"\n",
"\n",
" ######################## YOUR CODE STARTS HERE #########################\n",
" a_1 = None\n",
" a_2 = None\n",
" a_3 = None\n",
" ######################## YOUR CODE END HERE ############################\n",
" return a_1, a_2, a_3\n",
"\n",
" def backward_pass(self,\n",
" X: np.ndarray,\n",
" y: np.ndarray,\n",
" a_1: np.ndarray,\n",
" a_2: np.ndarray,\n",
" a_3: np.ndarray) -> Dict[str, np.ndarray]:\n",
" \"\"\"\n",
" TODO: Use the data examples, labels, and model outputs to compute all of\n",
" the gradients with respect to the weights and biases.\n",
"\n",
" Args:\n",
" X (np.ndarray): The NumPy array of input examples, of shape\n",
" (num_examples, input_size).\n",
" y (np.ndarray): The NumPy array of true labels for each\n",
" example in X. Has shape\n",
" (num_examples, 1).\n",
" a_1 (np.ndarray): The output of the first logistic regression\n",
" layer (after activation). Computed in\n",
" forward_pass().\n",
" a_2 (np.ndarray): The output of the second logistic regression\n",
" layer (after activation). Computed in\n",
" forward_pass().\n",
" a_3 (np.ndarray): The output of the third logistic regression\n",
" layer (after activation). Computed in\n",
" forward_pass().\n",
"\n",
" Returns:\n",
" Dict[str, np.ndarray]: A dictionary of the gradients computed for\n",
" each weight and bias parameter. The keys\n",
" are strings, the values are NumPy arrays\n",
" (see the names of the keys below).\n",
"\n",
" HINT: You can use np.where() for the identity function.\n",
" https://numpy.org/doc/stable/reference/generated/numpy.where.html \n",
" \"\"\"\n",
"\n",
" ######################## YOUR CODE STARTS HERE #########################\n",
" dW_3 = None\n",
" db_3 = None\n",
" dW_2 = None\n",
" db_2 = None\n",
" dW_1 = None\n",
" db_1 = None\n",
" ######################## YOUR CODE END HERE ############################\n",
"\n",
" # Return the gradients\n",
" gradients = {\n",
" \"W_3\": dW_3,\n",
" \"b_3\": db_3,\n",
" \"W_2\": dW_2,\n",
" \"b_2\": db_2,\n",
" \"W_1\": dW_1,\n",
" \"b_1\": db_1\n",
" }\n",
"\n",
" return gradients\n",
"\n",
" def train_batch(self,\n",
" X_train: np.ndarray,\n",
" y_train: np.ndarray,\n",
" X_dev: np.ndarray,\n",
" y_dev: np.ndarray,\n",
" batch_size: int=1000,\n",
" learning_rate: float=0.001,\n",
" num_epochs: int=100,\n",
" print_every: int=10,\n",
" verbose: bool=True):\n",
" \"\"\"\n",
" Trains the model with batch gradient descent.\n",
"\n",
" Args:\n",
" X_train (np.ndarray): The NumPy array of training examples, of shape\n",
" (num_examples, input_size).\n",
" y_train (np.ndarray): The NumPy array of true labels for each\n",
" example in X_train. Has shape\n",
" (num_examples, 1). Each element is either 0.0 or\n",
" 1.0.\n",
" X_dev (np.ndarray): The NumPy array of dev examples. Properties the\n",
" same as X_train.\n",
" y_dev (np.ndarray): The NumPy array of true labels for each\n",
" example in X_dev. Properties the same as y_train.\n",
" batch_size (int): The number of examples in each batch during\n",
" mini-batch gradient descent.\n",
" learning_rate (np.ndarray): Step size parameter for gradient\n",
" descent.\n",
" num_epochs (int): The number of epochs (full passes through the\n",
" provided training data) to train the model for.\n",
" print_every (int): Number of epochs to wait between printouts of the\n",
" training loss and accuracy during training.\n",
" verbose (bool): Whether to print any training output.\n",
"\n",
" This is just a slightly modified version of the train() function for\n",
" our earlier model.\n",
" \"\"\"\n",
" for epoch in range(1, num_epochs + 1):\n",
" total_loss = 0.0\n",
" total_correct = 0\n",
" num_batches = 0\n",
" for batch_start in range(0, X_train.shape[0], batch_size):\n",
" batch_end = min(X_train.shape[0], batch_start + batch_size)\n",
" train_X_batch = X_train[batch_start:batch_end]\n",
" train_y_batch = y_train[batch_start:batch_end]\n",
" a_1, a_2, a_3 = self.forward_pass(train_X_batch)\n",
" gradients = self.backward_pass(train_X_batch, train_y_batch, a_1,\n",
" a_2, a_3)\n",
"\n",
" for name, grad in gradients.items():\n",
" self.variables[name] -= learning_rate * grad\n",
"\n",
" total_loss += self.loss(a_3, train_y_batch)\n",
" total_correct += np.sum(\n",
" self.predict(train_X_batch) == train_y_batch)\n",
" num_batches += 1\n",
" if verbose and epoch % print_every == 0:\n",
" train_loss = total_loss / num_batches\n",
" train_accuracy = total_correct / X_train.shape[0]\n",
" dev_loss = self.loss(self.forward_pass(X_dev)[2], y_dev)\n",
" dev_accuracy = np.mean(self.predict(X_dev) == y_dev)\n",
" print(\"Epoch: {} | Train Loss: {} | Train Acc: {} | Dev Loss: {} | Dev Acc: {}\".format(epoch,\n",
" train_loss,\n",
" train_accuracy,\n",
" dev_loss,\n",
" dev_accuracy))\n",
" print(\"Finished training!\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"cell_id": "00059-5b86d472-3f48-4725-8c4e-25579e2a7f74",
"deepnote_cell_type": "markdown",
"tags": []
},
"source": [
"Here are some more spot checks to make sure you've gotten everything working.\n",
"As usual, they don't guarantee your solution is correct, but they're a helpful\n",
"way to check if it's wrong:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cell_id": "00062-e85d62ce-ad98-41d9-b0a4-3734af4300c3",
"deepnote_cell_type": "code",
"deepnote_to_be_reexecuted": false,
"execution_millis": 8,
"execution_start": 1619508950155,
"output_cleared": true,
"source_hash": "68e9fd5a",
"tags": [
"exploration"
]
},
"outputs": [],
"source": [
"check_forward_pass_yelp(YelpClassificationNeuralNetwork, train_examples_array)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cell_id": "00058-2c7d1cc8-2fbf-4bbe-a7c3-42cb045b9363",
"deepnote_cell_type": "code",
"deepnote_to_be_reexecuted": false,
"execution_millis": 16,
"execution_start": 1619509301393,
"output_cleared": true,
"source_hash": "360d2b2b",
"tags": [
"exploration"
]
},
"outputs": [],
"source": [
"check_backward_pass_yelp(YelpClassificationNeuralNetwork,\n",
" train_examples_array, train_labels_array)"
]
},
{
"cell_type": "markdown",
"metadata": {
"cell_id": "00059-e42e4aa2-1233-4850-8326-b2319f5c3ef9",
"deepnote_cell_type": "markdown",
"tags": []
},
"source": [
"If you've passed all the spot checks above, your model is most likely\n",
"working correctly.\n",
"\n",
"Let's train it!\n",
"\n",
"__NOTE:__ Training this model may take some time. We recommend taking a break\n",
"while it runs, or doing something else."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cell_id": "00037-6dd0ba0a-66a5-417a-8d15-a0d4cab83be9",
"deepnote_cell_type": "code",
"deepnote_to_be_reexecuted": false,
"execution_millis": 327393,
"execution_start": 1619499654992,
"id": "7_7O_NPy6CZ_",
"output_cleared": true,
"source_hash": "382f731b",
"tags": [
"exploration"
]
},
"outputs": [],
"source": [
"model = YelpClassificationNeuralNetwork(50, 100, 100)\n",
"\n",
"model.train_batch(\n",
" train_examples_array,\n",
" train_labels_array,\n",
" dev_examples_array,\n",
" dev_labels_array,\n",
" learning_rate=0.1,\n",
" batch_size=5000,\n",
" num_epochs=500,\n",
" print_every=10)"
]
},
{
"cell_type": "markdown",
"metadata": {
"cell_id": "00069-88427d70-3bc3-409d-b3b6-6c71233a3402",
"deepnote_cell_type": "markdown",
"tags": []
},
"source": [
"We encourage you to play around with your trained classifier and try out different\n",
"sentences using the evaluate() function below. When does your classifier succeed or fail? Are there weaknesses in this approach? How does it compare with the Logistic Regression model?\n",
"\n",
"__[Try this:]__ Although we will only grade your model's\n",
"performance/implementation using\n",
"the \"mean\" aggregation method, we encourage you to try re-processing the data\n",
"and re-training your model with the other aggregation strategies (sum or max).\n",
"\n",
"Once you do that, consider the following questions:\n",
"\n",
"How do the different settings affect your results? Do any of the other methods\n",
"seem to perform better/worse than using the mean? Can you explain the results\n",
"that you're seeing?"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cell_id": "00068-346c05e8-7624-4915-b2e7-c7b237a034e1",
"deepnote_cell_type": "code",
"deepnote_to_be_reexecuted": false,
"execution_millis": 8,
"execution_start": 1619509458587,
"output_cleared": true,
"source_hash": "659a8f5e",
"tags": [
"exploration"
]
},
"outputs": [],
"source": [
"evaluate(model, examples_to_array, embeddings, \"The food was fantastic.\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cell_id": "00064-0f9f2e8f-bbbe-4e30-a764-28b894cd11bf",
"deepnote_cell_type": "code",
"deepnote_to_be_reexecuted": false,
"execution_millis": 18,
"execution_start": 1619509460555,
"output_cleared": true,
"source_hash": "a8b84228",
"tags": [
"exploration"
]
},
"outputs": [],
"source": [
"evaluate(model, examples_to_array, embeddings, \"I did not like it.\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"cell_id": "00059-f3790d10-c2d3-4ed0-894e-c4cb182230c6",
"deepnote_cell_type": "markdown",
"tags": []
},
"source": [
"### Evaluating our Model\n",
"\n",
"Your classifier is not perfect, but it's not too shabby either!\n",
"\n",
"To give a rough ballpark estimate of what you should expect to see,\n",
"our implementation was able to achieve a training and dev accuracy of around the low to mid 80-percent using the exact training settings above.\n",
"\n",
"The autograder will train up your model implementation using the same\n",
"training settings above and evaluate your performance on the dev set, as well as a separate hidden test set.\n",
"\n",
"**You will be given up to 3 points for dev performance and 3 points for test performance.** On both datasets, you will get 1 point if you score at least 50% accuracy (this is a freebie, as in a binary classification task with balanced classes, guessing all class 1 will give you 50% accuracy), a further 1 point for scoring at least 80% accuracy, and a final 1 point for scoring above 84% accuracy.\n",
"\n",
"We've observed performance on dev and test to be very similar, as one would expect, so if you are able to meet the dev thresholds when training locally, you are almost guaranteed to get the 3 points for test as well.\n",
"\n",
"__NOTE:__ As part of evaluating your model, the autograder will train it for 500 epochs. This will take a bit of time to run, so **we don't recommend submitting rapid-fire to the autograder to test out every small change and fix you make to your model**. Instead, you should try to test each part locally (the spot checks should help with that), and verify that your model trains and performs well locally before submitting to Gradescope."
]
},
{
"cell_type": "markdown",
"metadata": {
"pycharm": {
"name": "#%% md\n"
}
},
"source": [
"### Interpreting our Results, and Final Thoughts\n",
"\n",
"As a final note, you may print out the trained model parameters using commands like \"print(model.W_1)\". However, these values are no longer easily interpretable as the weights in PA2, and the reason is complicated.\n",
"\n",
"First, it's no longer clear what each input feature even means, as\n",
"they come from dense word vectors/embeddings (what does the 5th or 48th element\n",
"in one of our word vectors really represent?).\n",
"\n",
"Secondly, even if we knew what each input feature was, the multi-layer\n",
"nature of our model means there's no longer a clear and interpretable connection\n",
"between oinput and output. \n",
"\n",
"This is not just a mathematical or theoretical observation. It's a serious\n",
"issue that comes up often when we apply neural network models in the real world. \n",
"When we use neural networks to make important decisions that have\n",
"an impact on the lives of real people, these decisions not only need to be right, but also need to be transparent and interpretable by humans.\n",
"\n",
"To mention just a couple of serious real-world examples, neural networks have been trained that [inadvertently discriminate\n",
"against women in hiring practices](https://www.reuters.com/article/us-amazon-com-jobs-automation-insight/amazon-scraps-secret-ai-recruiting-tool-that-showed-bias-against-women-idUSKCN1MK08G)\n",
"or [use zip code as a proxy for race](https://engineering.cmu.edu/news-events/news/2018/12/11-datta-proxies.html)\n",
"because the engineers building the systems were not able to fully understand\n",
"which features in the training data the network was using, or how it was\n",
"reaching its decisions.\n",
"\n",
"Based on what you're learned so far about neural networks and NLP, what\n",
"are some other potential downsides to networks being opaque/uninterpretable?\n",
"Do you have any ideas for how we could increase neural network interpretability?"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Part 3: Ethics in Sentiment Analysis\n",
"\n",
"Now that you’ve built a model that can \"read\" sentiment, it’s important to step back and look at the data and the task itself. In the real world, a high accuracy score on a test set doesn't always mean a model is fair, robust, or even solving the right problem.\n",
"\n",
"### Question 1: The \"Hidden\" Feature Problem\n",
"In Part 1, you saw how logistic regression failed on the XOR dataset because it couldn't capture non-linear relationships. In Part 2, your neural network achieved high accuracy on Yelp reviews by aggregating word embeddings.\n",
"\n",
"Suppose you are choosing between two restaurants:\n",
"\n",
"Restaurant A: 10 reviews, all 5-stars. \\\n",
"Restaurant B: 500 reviews, average rating of 4.3 stars."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Part A: \n",
"\n",
"Your neural network takes a single review's text and predicts a 0 or 1. If you were to aggregate these predictions to score a restaurant, how might the model’s confidence in a 5-star prediction for Restaurant A differ from its confidence for Restaurant B?"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"def model_confidence_response():\n",
" \"\"\"\n",
" TODO: Write your response below.\n",
" \"\"\"\n",
" response = \"\"\n",
" return response"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Part B: \n",
"A model can achieve high accuracy on a training set while still missing features critical for real-world decision-making (like sample size or rating variance). How do you know if your model will be robust once it is deployed in the world? Propose one additional feature or a new measurement (beyond the text itself) that you would include to help a user understand how much they should trust the model's output."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"def model_trust_response():\n",
" \"\"\"\n",
" TODO: Write your response below.\n",
" \"\"\"\n",
" response = \"\"\n",
" return response"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Question 2: The Meaning of a \"Star\"\n",
"\n",
"To train your model, you performed a common NLP shortcut: collapsing labels. You turned 1 and 2-star reviews into 0 (negative) and 4 and 5-star reviews into 1 (positive), ignoring 3-star reviews entirely.\n",
"\n",
"Consider these two 1-star reviews: \n",
"\n",
"Review 1: \"The pasta was overcooked and bland. I've had better frozen meals. Truly disappointing food.\" \\\n",
"Review 2: \"The food was okay, but the waiter was incredibly rude to my grandmother. Then we saw a cockroach near the bar. Never coming back to this filth.\"\n",
"\n",
"### Part A: \n",
"Your model treats both of these reviews as the same mathematical label: 0. Is there a \"right\" answer to what a 1-star review represents? By collapsing all negative experiences into a single dimension, what assumptions are you making about how humans value different aspects of a business (e.g., food quality vs. hygiene vs. service)?\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"def model_assumptions_response():\n",
" \"\"\"\n",
" TODO: Write your response below.\n",
" \"\"\"\n",
" response = \"\"\n",
" return response"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Part B: \n",
"Do you think both of these reviews should be \"learned from\" equally by your neural network? How might you go about filtering or restructuring your training data to deal with the fact that different reviewers value different things?"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"def data_filtering_response():\n",
" \"\"\"\n",
" TODO: Write your response below.\n",
" \"\"\"\n",
" response = \"\"\n",
" return response"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Question 3: Perspectivism and Noisy Labels\n",
"\n",
"Data labels are often treated as ground truth, but they can reflect the perspectives and assumptions of those who created them. This is especially true in subjective areas like Yelp reviews or Course Evaluations.\n",
"\n",
"Imagine applying your neural network to Stanford’s course evaluation data. You find that:\n",
"\n",
"Professor A (a tough grader) receives many 1-2 star reviews from students who received a C. \\\n",
"Professor B (a lenient grader) receives many 5-star reviews from students who praise their charming personality but rarely mention the actual course material.\n",
"\n",
"### Part A: \n",
"A high score on an evaluation can be driven by many factors: pedagogical clarity, a professor's personality, or even how leniently they grade. Since every student weighs these factors differently, what does a model's predicted score actually communicate to a future student?\n",
"If you saw a course predicted to receive a 1.5/5, what would you infer about it? What would you want to know that the score alone doesn't tell you? How might another student with different academic goals or learning preferences interpret that same 1.5 differently than you would?"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"def score_interpretation_response():\n",
" \"\"\"\n",
" TODO: Write your response below.\n",
" \"\"\"\n",
" response = \"\"\n",
" return response"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Part B: \n",
"When the ground truth labels in your data contain such varied personal motivations and biases, the data is considered noisy. Propose a way you might redesign the labels (not the model) to better account for this noisiness. For example, how might you incorporate the perspective of the reviewer into the training process?"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"def noisy_data_response():\n",
" \"\"\"\n",
" TODO: Write your response below.\n",
" \"\"\"\n",
" response = \"\"\n",
" return response"
]
},
{
"cell_type": "markdown",
"metadata": {
"pycharm": {
"name": "#%% md\n"
}
},
"source": [
"CONGRATS! You're done!\n",
"\n",
"Once you're ready to submit, you can run the cell below to prepare and zip\n",
"up your solution:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cell_id": "00053-55919144-e9a1-4ec6-b723-145420e564a2",
"deepnote_cell_type": "code",
"deepnote_to_be_reexecuted": true,
"id": "vAsNmB39ynRB",
"output_cleared": true,
"source_hash": "fc295dda",
"tags": [
"exploration"
]
},
"outputs": [],
"source": [
"%%bash\n",
"\n",
"if [[ ! -f \"./pa5.ipynb\" ]]\n",
"then\n",
" echo \"WARNING: Did not find notebook in Jupyter working directory. This probably means you're running on Google Colab. You'll need to go to File->Download .ipynb to download your notebok and other files, then zip them locally. See the README for more information.\"\n",
"else\n",
" echo \"Found notebook file, creating submission zip...\"\n",
" zip -r submission.zip pa5.ipynb deps/\n",
"fi\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"cell_id": "00054-005f5170-ce17-49d0-b17b-807bf1047813",
"deepnote_cell_type": "markdown",
"id": "MlR7fvySysAL",
"pycharm": {
"name": "#%% md\n"
}
},
"source": [
"If you're running on Google Colab, see the README for instructions on\n",
"how to submit.\n",
"\n",
"__Best of luck!__\n",
"\n",
"__Some reminders for submission:__\n",
"* If you have any extra files required for your implementation to work, make\n",
" sure they are in a `deps/` folder on the same level as `pa5.ipynb` and\n",
" include that folder in your submission zip file.\n",
" * Make sure you didn't accidentally change the name of your notebook file,\n",
" (it should be `pa5.ipynb`) as that is required for the autograder to work.\n",
"* Go to Gradescope (gradescope.com), find the pa5 Neural Networks assignment and\n",
"upload your zip file (`submission.zip`) as your solution.\n",
"* Wait for the autograder to run (it should only take a minute or so) and check\n",
"that your submission was graded successfully! If the autograder fails, or you\n",
"get an unexpected score it may be a sign that your zip file was incorrect."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": []
}
],
"metadata": {
"celltoolbar": "Tags",
"colab": {
"collapsed_sections": [],
"name": "cs124_pa_5_ramp_up.ipynb",
"provenance": []
},
"deepnote": {
"is_reactive": false
},
"deepnote_execution_queue": [],
"deepnote_notebook_id": "5e42aa6c-1fdf-4499-ae69-7de303b68aae",
"kernelspec": {
"display_name": "base",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.0"
}
},
"nbformat": 4,
"nbformat_minor": 4
}