{ "cells": [ { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "view-in-github" }, "source": [ "\"Open" ] }, { "cell_type": "markdown", "metadata": { "id": "Rdy1FtrRpGcC" }, "source": [ "# Getting Started\n", "\n", "## Overview\n", "This semester, all homeworks will be conducted through Google Colab notebooks. All code for the homework assignment will be written and run in this notebook. Running in Colab will automatically provide a GPU, but you may also run this notebook locally by following [these instructions](https://research.google.com/colaboratory/local-runtimes.html) if you wish to use your own GPU.\n", "\n", "You will save images in the notebooks to use and fill out a given LaTeX template which will be submitted to Gradescope, along with your notebook code.\n", "\n", "## Using Colab\n", "On the left-hand side, you can click the different icons to see a Table of Contents of the assignment, as well as local files accessible through the notebook.\n", "\n", "Make sure to go to **Runtime -> Change runtime type** and select **GPU** as the hardware accelerator. This allows you to use a GPU. Run the cells below to get started on the assignment. Note that a session is open for a maximum of 12 hours, and using too much GPU compute may result in restricted access for a short period of time. Please start the homework early so you have ample time to work.\n", "\n", "**If you load this notebook by clicking \"Open in Colab\" from github, you will need to save it to your own Google Drive to keep your work.**\n", "\n", "## General Tips\n", "In each homework problem, you will implement autoregressive models and run it on various datasets. Oftentime you will run it on two datasets (dataset 1 and dataset 2). In these cases, the expected outputs for dataset 1 are already provided to help as a sanity check.\n", "\n", "Feel free to print whatever output (e.g. debugging code, training code, etc) you want, as the graded submission will be the submitted pdf with images.\n", "\n", "After you complete the assignment, download all of the images outputted in the results/ folder and upload them to the figure folder in the given latex template.\n", "\n", "There is a lot of freedom in this homework to design write and design your own models. Hyperparameters are given as a guide to show what worked for us, but feel free to explore and use what you find is best!\n", "\n", "Run the cells below to download and load up the starter code." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "wUVy2glDtoaR", "outputId": "25ae66e6-0a38-4c5d-89ad-91b696af731c" }, "outputs": [], "source": [ "!if [ -d deepul ]; then rm -Rf deepul; fi\n", "!git clone https://github.com/rll/deepul.git \n", "!unzip -qq deepul/homeworks/hw1/data/hw1_data.zip -d deepul/homeworks/hw1/data/\n", "!pip install ./deepul" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "ZHWosWrbpO5Y" }, "outputs": [], "source": [ "import numpy as np\n", "from deepul.hw1_helper import (\n", " # Q1\n", " visualize_q1_data,\n", " q1_sample_data_1,\n", " q1_sample_data_2,\n", " q1_save_results,\n", " # Q2\n", " q2a_save_results,\n", " q2b_save_results,\n", " visualize_q2a_data,\n", " visualize_q2b_data,\n", " # Q3\n", " q3ab_save_results,\n", " q3c_save_results,\n", " # Q4\n", " q4a_save_results,\n", " q4b_save_results,\n", " # Q5\n", " visualize_q5_data,\n", " q5a_save_results,\n", " # Q6\n", " visualize_q6_data,\n", " q6a_save_results,\n", ")" ] }, { "cell_type": "markdown", "metadata": { "id": "7E4CMktzo100" }, "source": [ "# Question 1: 1D Data\n", "\n", "In this question, we will train simple generative models on discrete 1D data.\n", "\n", "Execute the cell below to visualize our datasets" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 607 }, "id": "ehhv2FZGo4_b", "outputId": "9d7fcbbc-5b19-47e8-8f9f-23a2c9ca6744" }, "outputs": [], "source": [ "visualize_q1_data(dset_type=1)\n", "visualize_q1_data(dset_type=2)" ] }, { "cell_type": "markdown", "metadata": { "id": "kSGTVznZqAR3" }, "source": [ "## Part (a) Fitting a Histogram\n", "\n", "Let $\\theta = (\\theta_0, \\dots, \\theta_{d-1}) \\in \\mathbb{R}^{d}$ and define the model $p_\\theta(x) = \\frac{e^{\\theta_x}}{\\sum_{x'}e^{\\theta_{x'}}}$\n", "\n", "Fit $p_\\theta$ with maximum likelihood via stochastic gradient descent on the training set, using $\\theta$ initialized to zero. Use your favorite version of stochastic gradient descent, and optimize your hyperparameters on a validation set of your choice.\n", "\n", "**You will provide these deliverables**\n", "\n", "\n", "1. Over the course of training, record the average negative log-likelihood (nats / dim) of the training data (per minibatch) and test data (for your entire test set). Code is provided that automatically plots the training curves. \n", "2. Report the final test set performance of your final model\n", "3. Plot the model probabilities in a bar graph with $\\{0,\\dots,d-1\\}$ on the x-axis and a real number in $[0,1]$ on the y-axis.\n", "\n", "\n" ] }, { "cell_type": "markdown", "metadata": { "id": "Yg0Jmo1PSaE4" }, "source": [ "Fill out the function below and return the necessary arguments. Feel free to create more cells if need be." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "BJNa6dHKpEQU" }, "outputs": [], "source": [ "def q1_a(train_data, test_data, d, dset_id):\n", " \"\"\"\n", " train_data: An (n_train,) numpy array of integers in {0, ..., d-1}\n", " test_data: An (n_test,) numpy array of integers in {0, .., d-1}\n", " d: The number of possible discrete values for random variable x\n", " dset_id: An identifying number of which dataset is given (1 or 2). Most likely\n", " used to set different hyperparameters for different datasets\n", "\n", " Returns\n", " - a (# of training iterations,) numpy array of train_losses evaluated every minibatch\n", " - a (# of epochs + 1,) numpy array of test_losses evaluated once at initialization and after each epoch\n", " - a numpy array of size (d,) of model probabilities\n", " \"\"\"\n", "\n", " return train_losses, test_losses, distribution" ] }, { "cell_type": "markdown", "metadata": { "id": "SiGBSP-ESeIj" }, "source": [ "### Results\n", "\n", "Once you've implemented `q1_a`, execute the cells below to visualize and save your results\n", "\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 590 }, "id": "qjK_KReXsqYa", "outputId": "a4c6bd70-96fb-430a-8312-8cfab7169ca4" }, "outputs": [], "source": [ "q1_save_results(1, 'a', q1_a)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 590 }, "id": "sJVOUEaaZXcA", "outputId": "eaded21d-fc51-4f23-ea1a-ab63f3b829e1" }, "outputs": [], "source": [ "q1_save_results(2, 'a', q1_a)" ] }, { "cell_type": "markdown", "metadata": { "id": "DiyFXlj0rfcr" }, "source": [ "## Part (b) Fitting Discretized Mixture of Logistics\n", "\n", "Let us model $p_\\theta(x)$ as a **discretized** mixture of 4 logistics such that $p_\\theta(x) = \\sum_{i=1}^4 \\pi_i[\\sigma((x+0.5 - \\mu_i)/s_i) - \\sigma((x-0.5-\\mu_i)/s_i)]$\n", "\n", "For the edge case of when $x = 0$, we replace $x-0.5$ by $-\\infty$, and for $x = 99$, we replace $x+0.5$ by $\\infty$.\n", "\n", "You may find the [PixelCNN++](https://arxiv.org/abs/1701.05517) helpful for more information on discretized mixture of logistics.\n", "\n", "**Provide the same set of corresponding deliverables as part (a)**" ] }, { "cell_type": "markdown", "metadata": { "id": "f4dnQIg_TDx6" }, "source": [ "Fill out the function below and return the necessary arguments. Feel free to create more cells if need be." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "uAvMQDJJrjNo" }, "outputs": [], "source": [ "def q1_b(train_data, test_data, d, dset_id):\n", " \"\"\"\n", " train_data: An (n_train,) numpy array of integers in {0, ..., d-1}\n", " test_data: An (n_test,) numpy array of integers in {0, .., d-1}\n", " d: The number of possible discrete values for random variable x\n", " dset_id: An identifying number of which dataset is given (1 or 2). Most likely\n", " used to set different hyperparameters for different datasets\n", "\n", " Returns\n", " - a (# of training iterations,) numpy array of train_losses evaluated every minibatch\n", " - a (# of epochs + 1,) numpy array of test_losses evaluated once at initialization and after each epoch\n", " - a numpy array of size (d,) of model probabilities\n", " \"\"\"\n", " return train_losses, test_losses, distribution" ] }, { "cell_type": "markdown", "metadata": { "id": "VwZyhlewTHH4" }, "source": [ "### Results\n", "\n", "Once you've implemented `q1_b`, execute the cells below to visualize and save your results\n", "\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 591 }, "id": "wnnQORaG6Ouf", "outputId": "3890f9ce-84cb-4c48-f2ca-9a1746acc424" }, "outputs": [], "source": [ "q1_save_results(1, 'b', q1_b)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 590 }, "id": "1jLGoDa46RM6", "outputId": "000a56aa-4811-44ef-a8bb-a2e672cbc106" }, "outputs": [], "source": [ "q1_save_results(2, 'b', q1_b)" ] }, { "cell_type": "markdown", "metadata": { "id": "dP8lmmk7Xrct" }, "source": [ "# Question 2 PixelCNNs" ] }, { "cell_type": "markdown", "metadata": { "id": "4wnyhDNqcAcw" }, "source": [ "Now, you will train more powerful PixelCNN models on the shapes dataset and MNIST. In addition, we will extend to modeling colored datasets.\n", "\n", "Run the cell below to visualize the two datasets binary datasets" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "visualize_q2a_data(1)\n", "visualize_q2a_data(2)" ] }, { "cell_type": "markdown", "metadata": { "id": "50WsEzhx4Uua" }, "source": [ "## Part (a) PixelCNN on Shapes and MNIST\n", "In this part, implement a simple PixelCNN architecture to model binary MNIST and shapes images (same as Q2(b), but with a PixelCNN).\n", "\n", "We recommend the following network design:\n", "* A $7 \\times 7$ masked type A convolution\n", "* $5$ $7 \\times 7$ masked type B convolutions\n", "* $2$ $1 \\times 1$ masked type B convolutions\n", "* Appropriate ReLU nonlinearities in-between\n", "* 64 convolutional filters\n", "\n", "And the following hyperparameters:\n", "* Batch size 128\n", "* Learning rate $10^{-3}$\n", "* 10 epochs\n", "* Adam Optimizer (this applies to all PixelCNN models trained in future parts)\n", "\n", "Your model should output logits, after which you could apply a sigmoid over 1 logit, or a softmax over two logits (either is fine). It may also help to scale your input to $[-1, 1]$ before running it through the network. \n", "\n", "Training on the shapes dataset should be quick, and MNIST should take around 10 minutes\n", "\n", "Checkout the Paper for more details: https://arxiv.org/abs/1601.06759\n", "\n", "**You will provide these deliverables**\n", "\n", "\n", "1. Over the course of training, record the average negative log-likelihood (nats / dim) of the training data (per minibatch) and test data (for your entire test set). Code is provided that automatically plots the training curves. \n", "2. Report the final test set performance of your final model\n", "3. 100 samples from the final trained model\n", "\n" ] }, { "cell_type": "markdown", "metadata": { "id": "EleefdNuciyc" }, "source": [ "Fill out the function below and return the necessary arguments. Feel free to create more cells if need be." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "NWualafa-tpD" }, "outputs": [], "source": [ "def q2_a(train_data, test_data, image_shape, dset_id):\n", " \"\"\"\n", " train_data: A (n_train, H, W, 1) uint8 numpy array of binary images with values in {0, 1}\n", " test_data: A (n_test, H, W, 1) uint8 numpy array of binary images with values in {0, 1}\n", " image_shape: (H, W), height and width of the image\n", " dset_id: An identifying number of which dataset is given (1 or 2). Most likely\n", " used to set different hyperparameters for different datasets\n", "\n", " Returns\n", " - a (# of training iterations,) numpy array of train_losses evaluated every minibatch\n", " - a (# of epochs + 1,) numpy array of test_losses evaluated once at initialization and after each epoch\n", " - a numpy array of size (100, H, W, 1) of samples with values in {0, 1}\n", " \"\"\"\n", " return train_losses, test_losses, samples" ] }, { "cell_type": "markdown", "metadata": { "id": "v0EPVfz1cpq0" }, "source": [ "### Results\n", "\n", "Once you've implemented `q2_a`, execute the cells below to visualize and save your results\n", "\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 559 }, "id": "xNxXqVZpAd_V", "outputId": "e2b74bd6-fde3-41f3-9e3a-e24a8cdee92e" }, "outputs": [], "source": [ "q2a_save_results(1, q2_a)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 559 }, "id": "OCyQzhJdAfiJ", "outputId": "1bf0668e-43b8-457c-db3b-7ea970af4777" }, "outputs": [], "source": [ "q2a_save_results(2, q2_a)" ] }, { "cell_type": "markdown", "metadata": { "id": "8J7qlqlODNgL" }, "source": [ "## Part (b) PixelCNN on Colored Shapes and MNIST: Independent Color Channels\n", "\n", "For the next part, we'll work with color images (shapes and MNIST). Run the cell below to visualize the dataset." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 511 }, "id": "80f_7uZWkDSv", "outputId": "61dde7b4-00aa-49e0-bbd3-96bd19f8f8bf" }, "outputs": [], "source": [ "visualize_q2b_data(1)\n", "visualize_q2b_data(2)" ] }, { "cell_type": "markdown", "metadata": { "id": "3Y6NggR6gmU9" }, "source": [ "Now, implement a PixelCNN to support RGB color channels (or augment your existing implementation). **First, implement a PixelCNN that assumes color channels as independent.** More formally, we model the following parameterized distribution:\n", "\n", "$$p_\\theta(x) = \\prod_{i=1}^{HW}\\prod_{c=1}^C p_\\theta(x_i^c | x_{** or beginning of sequence token at the start of every sequence every image. Make sure to include this in your implementation as well. You can generate unconditional sample by conditioning with the **\\** token.\n", " * While dropout is a common feature in transformer models, you do not need to add it (but may if you wish!).\n", " * Prebuilt transformers exist in some frameworks (i.e. pytorch). Don't just use an off the shelf implementation as the point of the exercise is to better understand the transformer architecture. Building the transformer from the ground up (use primitives such as Linear/Dense layers, LayerNorm, GeLU, Embedding)\n", " * Learning rate warmup and cos learning rate decay are often used when training transformers to improve training stability and improve performance. See if this helps your model! Try 1000 steps of warmup with a cosine learning rate decay.\n", "\n", "Paper references\n", "* [Attention Is All You Need](https://arxiv.org/abs/1706.03762) \n", "* [Generative Pretraining from Pixels](https://cdn.openai.com/papers/Generative_Pretraining_from_Pixels_V2.pdf) \n", "* [Language Models are Unsupervised Multitask Learners](https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf)\n", "\n", "We recommend the following network design parameters:\n", "* $d_{model}$: 128\n", "* heads: 4\n", "* layers: 2\n", "* GeLU nonlinearities\n", "\n", "And the following hyperparameters:\n", "* Batch size: 64 or 32 or 16 (whichever fits in your GPU)\n", "* Learning rate: $10^{-3}$\n", "* 15 epochs or more\n", "* Adam Optimizer (this applies to all Transformers models trained in future parts)\n", "\n", "**You will provide these deliverables**\n", "\n", "1. Over the course of training, record the average negative log-likelihood (nats / dim) of the training data (per minibatch) and test data (for your entire test set). Code is provided that automatically plots the training curves. \n", "2. Report the final test set performance of your final model\n", "3. 100 samples from the final trained model\n", "\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def q3_a(train_data, test_data, image_shape, dset_id):\n", " \"\"\"\n", " train_data: A (n_train, H, W, 1) uint8 numpy array of color images with values in {0, 1}\n", " test_data: A (n_test, H, W, 1) uint8 numpy array of color images with values in {0, 1}\n", " image_shape: (H, W, 1), height, width, and # of channels of the image\n", " dset_id: An identifying number of which dataset is given (1 or 2). Most likely\n", " used to set different hyperparameters for different datasets\n", "\n", " Returns\n", " - a (# of training iterations,) numpy array of train_losses evaluated every minibatch\n", " - a (# of epochs + 1,) numpy array of test_losses evaluated once at initialization and after each epoch\n", " - a numpy array of size (100, H, W, 1) of samples with values in {0, 1}\n", " \"\"\"\n", " return train_losses, test_losses, samples" ] }, { "cell_type": "markdown", "metadata": { "id": "mGp2OsLKiToN" }, "source": [ "### Results\n", "\n", "Once you've implemented `q3_a`, execute the cells below to visualize and save your results" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 577 }, "id": "kW-k-59qJaKN", "outputId": "78b08f6a-12db-46b7-ac27-a07693a9cd09" }, "outputs": [], "source": [ "q3ab_save_results(1, 'a', q3_a)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 577 }, "id": "It_iPXaZjlk0", "outputId": "3ba138b1-71c9-4f67-8413-b5ef23db8204" }, "outputs": [], "source": [ "q3ab_save_results(2, 'a', q3_a)" ] }, { "cell_type": "markdown", "metadata": { "id": "50WsEzhx4Uua" }, "source": [ "## Part (b) iGPT on Colored Shapes and MNIST\n", "\n", "Now, implement an iGPT that models color. In order to reduce the length of token sequences, iGPT models each RGB pixel as a **single** token. This effectively reduces the context length from H*W*C to just H*W. iGPT does this through a k-means clustering approach. Because our images only each can only take on 4 values (2 bits) per channel, we can represent each pixel with 64 values (6 bits). Convert the dataset into an image of tokens and train iGPT on the colored shapes and MNIST dataset.\n", "\n", "Checkout the iGPT paper for more details: [Generative Pretraining from Pixels](https://cdn.openai.com/papers/Generative_Pretraining_from_Pixels_V2.pdf) \n", "\n", "Training times and hyperparameter settings should be the same as part (a), except train for longer (15 epochs)\n", "\n", "**You will provide these deliverables**\n", "\n", "1. Over the course of training, record the average negative log-likelihood (nats / dim) of the training data (per minibatch) and test data (for your entire test set). Code is provided that automatically plots the training curves. \n", "2. Report the final test set performance of your final model\n", "3. 100 samples from the final trained model\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def q3_b(train_data, test_data, image_shape, dset_id):\n", " \"\"\"\n", " train_data: A (n_train, H, W, C) uint8 numpy array of color images with values in {0, 1, 2, 3}\n", " test_data: A (n_test, H, W, C) uint8 numpy array of color images with values in {0, 1, 2, 3}\n", " image_shape: (H, W, C), height, width, and # of channels of the image\n", " dset_id: An identifying number of which dataset is given (1 or 2). Most likely\n", " used to set different hyperparameters for different datasets\n", "\n", " Returns\n", " - a (# of training iterations,) numpy array of train_losses evaluated every minibatch\n", " - a (# of epochs + 1,) numpy array of test_losses evaluated once at initialization and after each epoch\n", " - a numpy array of size (100, H, W, C) of samples with values in {0, 1, 2, 3}\n", " \"\"\"\n", " return train_losses, test_losses, samples" ] }, { "cell_type": "markdown", "metadata": { "id": "mGp2OsLKiToN" }, "source": [ "### Results\n", "\n", "Once you've implemented `q3_b`, execute the cells below to visualize and save your results" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 577 }, "id": "kW-k-59qJaKN", "outputId": "78b08f6a-12db-46b7-ac27-a07693a9cd09" }, "outputs": [], "source": [ "q3ab_save_results(1, 'b', q3_b)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 577 }, "id": "It_iPXaZjlk0", "outputId": "3ba138b1-71c9-4f67-8413-b5ef23db8204" }, "outputs": [], "source": [ "q3ab_save_results(2, 'b', q3_b)" ] }, { "cell_type": "markdown", "metadata": { "id": "50WsEzhx4Uua" }, "source": [ "## Part (c) K, V Caching for Improved Inference\n", "You may have noticed that generation from the transformer is quite slow. Part of this is just due to the autoregressive nature. However, another part is due to some computational inefficiency. At each forward pass of the model, we are performing repeat computation of the past sequence. Specifically, we can cache the key and values at the multi attention layer to more quickly predict at each step.\n", "\n", "In self-attention, a sequence is processed by generating three vectors for each element in the sequence: a Query (Q), a Key (K), and a Value (V). These vectors are then used to compute attention scores and subsequently the output of the attention layer.\n", "Mathematically, this can be represented as:\n", " * For each index $i$, compute $Q_i$, $K_i$, $V_i$ for the current element\n", " * Retrieve $K_{ int:\n", "# \"\"\"The size of the token vocabulary\"\"\"\n", "# \n", "# def quantize(self, x: np.ndarray) -> np.ndarray:\n", "# \"\"\"Quantize an image x.\n", "#\n", "# Args:\n", "# x (np.ndarray, dtype=int): Image to quantize. shape=(batch_size, 28, 28, 3). Values in [0, 3].\n", "#\n", "# Returns:\n", "# np.ndarray: Quantized image. shape=(batch_size, 7, 7). Values in [0, n_embeddings]\n", "# \"\"\"\n", "# \n", "# def decode(self, z_index: np.ndarray) -> np.ndarray:\n", "# \"\"\"Decode a quantized image.\n", "#\n", "# Args:\n", "# z_index (np.ndarray, dtype=int): Quantized image. shape=(batch_size, 7, 7). Values in [0, n_embeddings].\n", "#\n", "# Returns:\n", "# np.ndarray: Decoded image. shape=(batch_size, 28, 28, 3). Values in [0, 3].\n", "# \"\"\"\n", "# " ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def q4_a(images, vqvae):\n", " \"\"\"\n", " images: (B, H, W, C), the images to pass through the encoder and decoder of the vqvae\n", " vqvae: a vqvae model, trained on the relevant dataset\n", "\n", " Returns\n", " - a numpy array of size (2, H, W, C) of the decoded image\n", " \"\"\"\n", " return autoencoded_images" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "q4a_save_results(1, q4_a)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "q4a_save_results(2, q4_a)" ] }, { "cell_type": "markdown", "metadata": { "id": "50WsEzhx4Uua" }, "source": [ "## Part (b) Autoregressive Transformer on Colored Shapes and MNIST with Vector Quantization\n", "\n", "We can use the VQVAE to tokenize an image dataset. This will result in a much smaller sequence length than the approach we tried in Question 3(b). For this part, train a transformer on the dataset tokenized by the VQVAE.\n", "\n", "This is a simplified version of the approach used in VQGAN [VQGAN](https://arxiv.org/abs/2012.09841) -> Section 3.2: Learning the Composition of Images with Transformers (Again, we will cover this in more detail in a future lecture!)\n", "\n", "Update the following hyperparameters:\n", "* layers: 4 (we can train a bigger transformer now since less memory is used per input!)\n", "* 30 epochs\n", "\n", "**You will provide these deliverables**\n", "\n", "1. Over the course of training, record the average negative log-likelihood (nats / dim) of the training data (per minibatch) and test data (for your entire test set). Code is provided that automatically plots the training curves. \n", "2. Report the final test set performance of your final model\n", "3. 100 samples from the final trained model" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def q4_b(train_data, test_data, image_shape, dset_id, vqvae):\n", " \"\"\"\n", " train_data: A (n_train, H, W, C) uint8 numpy array of color images with values in {0, 1, 2, 3}\n", " test_data: A (n_test, H, W, C) uint8 numpy array of color images with values in {0, 1, 2, 3}\n", " image_shape: (H, W, C), height, width, and # of channels of the image\n", " dset_id: An identifying number of which dataset is given (1 or 2). Most likely\n", " used to set different hyperparameters for different datasets\n", " vqvae: a vqvae model, trained on dataset dset_id\n", "\n", " Returns\n", " - a (# of training iterations,) numpy array of train_losses evaluated every minibatch\n", " - a (# of epochs + 1,) numpy array of test_losses evaluated once at initialization and after each epoch\n", " - a numpy array of size (100, H, C, W) of samples with values in {0, 1, 2, 3}\n", " \"\"\"\n", " return train_losses, test_losses, samples" ] }, { "cell_type": "markdown", "metadata": { "id": "mGp2OsLKiToN" }, "source": [ "### Results\n", "\n", "Once you've implemented `q4_b`, execute the cells below to visualize and save your results\n", "\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 577 }, "id": "kW-k-59qJaKN", "outputId": "78b08f6a-12db-46b7-ac27-a07693a9cd09" }, "outputs": [], "source": [ "q4b_save_results(1, q4_b)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 577 }, "id": "It_iPXaZjlk0", "outputId": "3ba138b1-71c9-4f67-8413-b5ef23db8204" }, "outputs": [], "source": [ "q4b_save_results(2, q4_b)" ] }, { "cell_type": "markdown", "metadata": { "id": "LMZLcaHwLNNL" }, "source": [ "# Question 5: Causal Transformer: Text" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now lets consider text! You are probably already fimilar with autoregressive transformers for text, now more commonly known as Large Language Modesl (LLMs).\n", "We will now implement a simplified version.\n", "\n", "We will be detailing with a [small poetry dataset](https://huggingface.co/datasets/merve/poetry). See some of the data below." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "data = visualize_q5_data()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Part (a) Modeling Text\n", "Train a transformer on the poetry dataset.\n", "\n", "Data Preprocessing:\n", "* We will use a simple method to tokenize the data. We will convert each unique character into a token. (Current LLMs use more sophisticated tokenizers, most commonly, [byte-pair encoding](https://huggingface.co/learn/nlp-course/chapter6/5?fw=pt))\n", "* Previously we have leveraged a **\\** as part of the model, just like iGPT. For text, we may not always sample a sequence that starts at the beginning. Instead, we will add the **\\** token to the beginning of every sequence in the dataset, and remove the **\\** token from the model.\n", "* Another problem is that the model must know when to stop sampling. This is done by appending an **\\**, or end of sequence token at the end of every sequence in the dataset.\n", "* We can now convert the sequence into subsequences of size context_length, for training!\n", "\n", "We recommend the following hyperparameters:\n", "* Sequence length: 128\n", "* 5 epochs\n", "\n", "**You will provide these deliverables**\n", "\n", "1. Over the course of training, record the average negative log-likelihood (nats / dim) of the training data (per minibatch) and test data (for your entire test set). Code is provided that automatically plots the training curves. \n", "2. Report the final test set performance of your final model\n", "3. Provide **5 unconditional samples** of **128 characters** showcasing the model text generation capabilities (text samples should stop after **\\**. Text after **\\** can be removed in post processing)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import torch.utils.data as data\n", "def q5_a(train_text, test_text):\n", " \"\"\"\n", " train_text: list[str] Train text sequences.\n", " test_text: list[str] Test text sequences.\n", "\n", " Returns\n", " - a (# of training iterations,) numpy array of train_losses evaluated every minibatch\n", " - a (# of epochs + 1,) numpy array of test_losses evaluated once at initialization and after each epoch\n", " - a list of 5 (str), 5 generated samples from the model.\n", " \"\"\"\n", " return train_losses, test_losses, text_samples" ] }, { "cell_type": "markdown", "metadata": { "id": "mGp2OsLKiToN" }, "source": [ "### Results\n", "\n", "Once you've implemented `q5_a`, execute the cells below to visualize and save your results\n", "\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 577 }, "id": "kW-k-59qJaKN", "outputId": "78b08f6a-12db-46b7-ac27-a07693a9cd09" }, "outputs": [], "source": [ "q5a_save_results(q5_a)" ] }, { "cell_type": "markdown", "metadata": { "id": "LMZLcaHwLNNL" }, "source": [ "# Question 6: Causal Transformer: Multimodal" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "So far, we have been dealing only with autoregressive generation of a single modality. Now we will train a model that operates on multiple modalities!\n", "\n", "We will use the text labeled colored MNIST dataset, which has a text description of the MNIST image. Run the cell below to visualize the data along with the text annotation. This is the Colored MNIST v2 dataset, which also comes with these text labels." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "visualize_q6_data()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Part (a) Multimodal Text and Image Generation\n", "Implement and train an autoregressive (AR) model capable of handling both text and image data. The model should be designed to process sequences composed of concatenated text and image tokens in both orders (text followed by images and images followed by text). Additionally, the model should be capable of generating unconditional text and image samples.\n", "\n", "Data Preprocessing:\n", "* Text Tokens: Map each unique word in the text data to a unique token. (Note that all text descriptions contain the exact same amount of words. This simplifies text processing, as you won't have to deal with sequences of different lengths as in Question 5)\n", "* Image Tokens: Quantize the image data into tokens using the VQVAE tokenizer from Problem 4.\n", "* In this problem, we have 2 modalities. Introduce an **\\** token and an **\\** token. After seeing such a token, the model should switch to sampling the next modality.\n", "* Formulate batches as sequences of concat([**\\**, text_tokens, **\\**, image_tokens]) and concat([**\\**, image_tokens, **\\**, text_tokens]). With a 50/50 split between each ordering.\n", "\n", "Inference:\n", "* During inference, we cannot mix modality tokens. During sampling we can restrict the logits to only be within the relevant modality.\n", "* After **\\**, only allow the model to sample text tokens (including **\\**)\n", "* After **\\**, only allow the model to sample image tokens (including **\\**)\n", "* At the very start (conditioned on the **\\** token, only allow the model to sample one of (**\\** or **\\**))\n", "* As the model may not always correctly sample the **\\** token before the image ends, you may add a rule to force the model to always sample the correct number of image tokens (49 tokens).\n", "\n", "You can use the same hyperparameters as in 4(b) (but of course, feel free to tune your model to achieve better performance)\n", "\n", "**You will provide these deliverables**\n", "\n", "1. Over the course of training, record the average negative log-likelihood (nats / dim) of the training data (per minibatch) and test data (for your entire test set). Code is provided that automatically plots the training curves. \n", "2. Report the final test set performance of your final model\n", "3. 9 conditional samples based on provided text.\n", "4. 9 conditional samples based on provided images.\n", "5. 9 unconditional samples showcasing the model's capability in generating standalone text and images." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def q6_a(train_data, test_data, image_shape, train_text, test_text, image_test_prompt, text_test_prompt, vqvae):\n", " \"\"\"\n", " train_data: A (n_train, H, W, C) uint8 numpy array of color images with values in {0, 1, 2, 3}\n", " test_data: A (n_test, H, W, C) uint8 numpy array of color images with values in {0, 1, 2, 3}\n", " image_shape: tuple (H, W, C) The shape of the images in the dataset, indicating height, width, and number of color channels.\n", " train_text: list[str] Text data associated with each training image.\n", " test_text: list[str] Text data associated with each test image.\n", " image_test_prompt: (9, H, W, C) Image data used for generating conditional text samples during testing.\n", " text_test_prompt: list of 9 strings Text prompts used for generating conditional image samples during testing.\n", " vqvae: a vqvae model, trained on the relevant dataset\n", "\n", " Returns\n", " - a (# of training iterations,) numpy array of train_losses evaluated every minibatch\n", " - a (# of epochs + 1,) numpy array of test_losses evaluated once at initialization and after each epoch\n", " - a list of 9 (image, text), corresponding to the image conditioned samples\n", " - a list of 9 (image, text), corresponding to the text conditions samples\n", " - a list of 9 (image, text), corresponding to unconditional samples\n", " \"\"\"\n", " return train_losses, test_losses, samples_image_conditioned, samples_text_conditioned, samples_unconditioned" ] }, { "cell_type": "markdown", "metadata": { "id": "mGp2OsLKiToN" }, "source": [ "### Results\n", "\n", "Once you've implemented `q6_a`, execute the cells below to visualize and save your results\n", "\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 577 }, "id": "kW-k-59qJaKN", "outputId": "78b08f6a-12db-46b7-ac27-a07693a9cd09" }, "outputs": [], "source": [ "q6a_save_results(q6_a)" ] } ], "metadata": { "accelerator": "GPU", "colab": { "collapsed_sections": [], "include_colab_link": true, "name": " Homework 1 Autoregressive Models (Solutions).ipynb", "provenance": [], "toc_visible": true }, "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.9" } }, "nbformat": 4, "nbformat_minor": 4 }