{ "cells": [ { "cell_type": "markdown", "metadata": { "id": "WBk0ZDWY-ff8" }, "source": [ "\n", " \n", " \n", " \n", "
\n", " \n", " Visit MIT Deep Learning\n", " Run in Google Colab\n", " View Source on GitHub
\n", "\n", "# Copyright Information\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "3eI6DUic-6jo" }, "outputs": [], "source": [ "# Copyright 2026 MIT Introduction to Deep Learning. All Rights Reserved.\n", "#\n", "# Licensed under the MIT License. You may not use this file except in compliance\n", "# with the License. Use and/or modification of this code outside of MIT Introduction\n", "# to Deep Learning must reference:\n", "#\n", "# © MIT Introduction to Deep Learning\n", "# http://introtodeeplearning.com\n", "#" ] }, { "cell_type": "markdown", "metadata": { "id": "57knM8jrYZ2t" }, "source": [ "# Lab 1: Intro to PyTorch and Music Generation with RNNs\n", "\n", "In this lab, you'll get exposure to using PyTorch and learn how it can be used for deep learning. Go through the code and run each cell. Along the way, you'll encounter several ***TODO*** blocks -- follow the instructions to fill them out before running those cells and continuing.\n", "\n", "\n", "# Part 1: Intro to PyTorch\n", "\n", "## 0.1 Install PyTorch\n", "\n", "[PyTorch](https://pytorch.org/) is a popular deep learning library known for its flexibility and ease of use. Here we'll learn how computations are represented and how to define a simple neural network in PyTorch. For all the labs in Introduction to Deep Learning 2026, there will be a PyTorch version available.\n", "\n", "Let's install PyTorch and a couple of dependencies." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "LkaimNJfYZ2w" }, "outputs": [], "source": [ "import torch\n", "import torch.nn as nn\n", "\n", "# Download and import the MIT Introduction to Deep Learning package\n", "!pip install mitdeeplearning --quiet\n", "import mitdeeplearning as mdl\n", "\n", "import numpy as np\n", "import matplotlib.pyplot as plt" ] }, { "cell_type": "markdown", "metadata": { "id": "2QNMcdP4m3Vs" }, "source": [ "## 1.1 What is PyTorch?\n", "\n", "PyTorch is a machine learning library, like TensorFlow. At its core, PyTorch provides an interface for creating and manipulating [tensors](https://pytorch.org/docs/stable/tensors.html), which are data structures that you can think of as multi-dimensional arrays. Tensors are represented as n-dimensional arrays of base datatypes such as a string or integer -- they provide a way to generalize vectors and matrices to higher dimensions. PyTorch provides the ability to perform computation on these tensors, define neural networks, and train them efficiently.\n", "\n", "The [```shape```](https://pytorch.org/docs/stable/generated/torch.Tensor.shape.html#torch.Tensor.shape) of a PyTorch tensor defines its number of dimensions and the size of each dimension. The `ndim` or [```dim```](https://pytorch.org/docs/stable/generated/torch.Tensor.dim.html#torch.Tensor.dim) of a PyTorch tensor provides the number of dimensions (n-dimensions) -- this is equivalent to the tensor's rank (as is used in TensorFlow), and you can also think of this as the tensor's order or degree.\n", "\n", "Let’s start by creating some tensors and inspecting their properties:\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "tFxztZQInlAB" }, "outputs": [], "source": [ "integer = torch.tensor(1234)\n", "decimal = torch.tensor(3.14159265359)\n", "\n", "print(f\"`integer` is a {integer.ndim}-d Tensor: {integer}\")\n", "print(f\"`decimal` is a {decimal.ndim}-d Tensor: {decimal}\")\n" ] }, { "cell_type": "markdown", "metadata": { "id": "-dljcPUcoJZ6" }, "source": [ "Vectors and lists can be used to create 1-d tensors:" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "oaHXABe8oPcO" }, "outputs": [], "source": [ "fibonacci = torch.tensor([1, 1, 2, 3, 5, 8])\n", "count_to_100 = torch.tensor(range(100))\n", "\n", "print(f\"`fibonacci` is a {fibonacci.ndim}-d Tensor with shape: {fibonacci.shape}\")\n", "print(f\"`count_to_100` is a {count_to_100.ndim}-d Tensor with shape: {count_to_100.shape}\")\n" ] }, { "cell_type": "markdown", "metadata": { "id": "gvffwkvtodLP" }, "source": [ "Next, let’s create 2-d (i.e., matrices) and higher-rank tensors. In image processing and computer vision, we will use 4-d Tensors with dimensions corresponding to batch size, number of color channels, image height, and image width." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "tFeBBe1IouS3" }, "outputs": [], "source": [ "### Defining higher-order Tensors ###\n", "\n", "'''TODO: Define a 2-d Tensor'''\n", "matrix = # TODO\n", "\n", "assert isinstance(matrix, torch.Tensor), \"matrix must be a torch Tensor object\"\n", "assert matrix.ndim == 2\n", "\n", "'''TODO: Define a 4-d Tensor.'''\n", "# Use torch.zeros to initialize a 4-d Tensor of zeros with size 10 x 3 x 256 x 256.\n", "# You can think of this as 10 images where each image is RGB 256 x 256.\n", "images = # TODO\n", "\n", "assert isinstance(images, torch.Tensor), \"images must be a torch Tensor object\"\n", "assert images.ndim == 4, \"images must have 4 dimensions\"\n", "assert images.shape == (10, 3, 256, 256), \"images is incorrect shape\"\n", "print(f\"images is a {images.ndim}-d Tensor with shape: {images.shape}\")" ] }, { "cell_type": "markdown", "metadata": { "id": "wkaCDOGapMyl" }, "source": [ "As you have seen, the `shape` of a tensor provides the number of elements in each tensor dimension. The `shape` is quite useful, and we'll use it often. You can also use slicing to access subtensors within a higher-rank tensor:" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "FhaufyObuLEG" }, "outputs": [], "source": [ "row_vector = matrix[1]\n", "column_vector = matrix[:, 1]\n", "scalar = matrix[0, 1]\n", "\n", "print(f\"`row_vector`: {row_vector}\")\n", "print(f\"`column_vector`: {column_vector}\")\n", "print(f\"`scalar`: {scalar}\")" ] }, { "cell_type": "markdown", "metadata": { "id": "iD3VO-LZYZ2z" }, "source": [ "## 1.2 Computations on Tensors\n", "\n", "A convenient way to think about and visualize computations in a machine learning framework like PyTorch is in terms of graphs. We can define this graph in terms of tensors, which hold data, and the mathematical operations that act on these tensors in some order. Let's look at a simple example, and define this computation using PyTorch:\n", "\n", "![alt text](https://raw.githubusercontent.com/MITDeepLearning/introtodeeplearning/master/lab1/img/add-graph.png)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "X_YJrZsxYZ2z" }, "outputs": [], "source": [ "# Create the nodes in the graph and initialize values\n", "a = torch.tensor(15)\n", "b = torch.tensor(61)\n", "\n", "# Add them!\n", "c1 = torch.add(a, b)\n", "c2 = a + b # PyTorch overrides the \"+\" operation so that it is able to act on Tensors\n", "print(f\"c1: {c1}\")\n", "print(f\"c2: {c2}\")\n" ] }, { "cell_type": "markdown", "metadata": { "id": "Mbfv_QOiYZ23" }, "source": [ "Notice how we've created a computation graph consisting of PyTorch operations, and how the output is a tensor with value 76 -- we've just created a computation graph consisting of operations, and it's executed them and given us back the result.\n", "\n", "Now let's consider a slightly more complicated example:\n", "\n", "![alt text](https://raw.githubusercontent.com/MITDeepLearning/introtodeeplearning/master/lab1/img/computation-graph.png)\n", "\n", "Here, we take two inputs, `a, b`, and compute an output `e`. Each node in the graph represents an operation that takes some input, does some computation, and passes its output to another node.\n", "\n", "Let's define a simple function in PyTorch to construct this computation function:" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "PJnfzpWyYZ23", "scrolled": true }, "outputs": [], "source": [ "### Defining Tensor computations ###\n", "\n", "# Construct a simple computation function\n", "def func(a, b):\n", " '''TODO: Define the operation for c, d, e.'''\n", " c = # TODO\n", " d = # TODO\n", " e = # TODO\n", " return e\n" ] }, { "cell_type": "markdown", "metadata": { "id": "AwrRfDMS2-oy" }, "source": [ "Now, we can call this function to execute the computation graph given some inputs `a,b`:" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "pnwsf8w2uF7p" }, "outputs": [], "source": [ "# Consider example values for a,b\n", "a, b = 1.5, 2.5\n", "# Execute the computation\n", "e_out = func(a, b)\n", "print(f\"e_out: {e_out}\")" ] }, { "cell_type": "markdown", "metadata": { "id": "6HqgUIUhYZ29" }, "source": [ "Notice how our output is a tensor with value defined by the output of the computation, and that the output has no shape as it is a single scalar value." ] }, { "cell_type": "markdown", "metadata": { "id": "1h4o9Bb0YZ29" }, "source": [ "## 1.3 Neural networks in PyTorch\n", "We can also define neural networks in PyTorch. PyTorch uses [``torch.nn.Module``](https://pytorch.org/docs/stable/generated/torch.nn.Module.html), which serves as a base class for all neural network modules in PyTorch and thus provides a framework for building and training neural networks.\n", "\n", "Let's consider the example of a simple perceptron defined by just one dense (aka fully-connected or linear) layer: $ y = \\sigma(Wx + b) $, where $W$ represents a matrix of weights, $b$ is a bias, $x$ is the input, $\\sigma$ is the sigmoid activation function, and $y$ is the output.\n", "\n", "![alt text](https://raw.githubusercontent.com/MITDeepLearning/introtodeeplearning/master/lab1/img/computation-graph-2.png)\n", "\n", "We will use `torch.nn.Module` to define layers -- the building blocks of neural networks. Layers implement common neural networks operations. In PyTorch, when we implement a layer, we subclass `nn.Module` and define the parameters of the layer as attributes of our new class. We also define and override a function [``forward``](https://pytorch.org/docs/stable/generated/torch.nn.Module.html#torch.nn.Module.forward), which will define the forward pass computation that is performed at every step. All classes subclassing `nn.Module` should override the `forward` function.\n", "\n", "Let's write a dense layer class to implement a perceptron defined above." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "HutbJk-1kHPh" }, "outputs": [], "source": [ "### Defining a dense layer ###\n", "\n", "# num_inputs: number of input nodes\n", "# num_outputs: number of output nodes\n", "# x: input to the layer\n", "\n", "class OurDenseLayer(torch.nn.Module):\n", " def __init__(self, num_inputs, num_outputs):\n", " super(OurDenseLayer, self).__init__()\n", " # Define and initialize parameters: a weight matrix W and bias b\n", " # Note that the parameter initialize is random!\n", " self.W = torch.nn.Parameter(torch.randn(num_inputs, num_outputs))\n", " self.bias = torch.nn.Parameter(torch.randn(num_outputs))\n", "\n", " def forward(self, x):\n", " '''TODO: define the operation for z (hint: use torch.matmul).'''\n", " z = # TODO\n", "\n", " '''TODO: define the operation for out (hint: use torch.sigmoid).'''\n", " y = # TODO\n", " return y\n" ] }, { "cell_type": "markdown", "metadata": { "id": "GqeEbn959hV_" }, "source": [ "Now, let's test the output of our layer." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "2yxjCPa69hV_" }, "outputs": [], "source": [ "# Define a layer and test the output!\n", "num_inputs = 2\n", "num_outputs = 3\n", "layer = OurDenseLayer(num_inputs, num_outputs)\n", "x_input = torch.tensor([[1, 2.]])\n", "y = layer(x_input)\n", "\n", "print(f\"input shape: {x_input.shape}\")\n", "print(f\"output shape: {y.shape}\")\n", "print(f\"output result: {y}\")" ] }, { "cell_type": "markdown", "metadata": { "id": "Jt1FgM7qYZ3D" }, "source": [ "Conveniently, PyTorch has defined a number of ```nn.Modules``` (or Layers) that are commonly used in neural networks, for example a [```nn.Linear```](https://pytorch.org/docs/stable/generated/torch.nn.Linear.html) or [`nn.Sigmoid`](https://pytorch.org/docs/stable/generated/torch.nn.Sigmoid.html) module.\n", "\n", "Now, instead of using a single ```Module``` to define our simple neural network, we'll use the [`nn.Sequential`](https://pytorch.org/docs/stable/generated/torch.nn.Sequential.html) module from PyTorch and a single [`nn.Linear` ](https://pytorch.org/docs/stable/generated/torch.nn.Linear.html) layer to define our network. With the `Sequential` API, you can readily create neural networks by stacking together layers like building blocks." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "7WXTpmoL6TDz" }, "outputs": [], "source": [ "### Defining a neural network using the PyTorch Sequential API ###\n", "\n", "# define the number of inputs and outputs\n", "n_input_nodes = 2\n", "n_output_nodes = 3\n", "\n", "# Define the model\n", "'''TODO: Use the Sequential API to define a neural network with a\n", " single linear (dense!) layer, followed by non-linearity to compute z'''\n", "model = nn.Sequential( ''' TODO ''' )\n" ] }, { "cell_type": "markdown", "metadata": { "id": "HDGcwYfUyR-U" }, "source": [ "We've defined our model using the Sequential API. Now, we can test it out using an example input:" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "zKhp6XqCFFa0" }, "outputs": [], "source": [ "# Test the model with example input\n", "x_input = torch.tensor([[1, 2.]])\n", "model_output = model(x_input)\n", "print(f\"input shape: {x_input.shape}\")\n", "print(f\"output shape: {y.shape}\")\n", "print(f\"output result: {y}\")" ] }, { "cell_type": "markdown", "metadata": { "id": "596NvsOOtr9F" }, "source": [ "With PyTorch, we can create more flexible models by subclassing [`nn.Module`](https://pytorch.org/docs/stable/generated/torch.nn.Module.html). The `nn.Module` class allows us to group layers together flexibly to define new architectures.\n", "\n", "As we saw earlier with `OurDenseLayer`, we can subclass `nn.Module` to create a class for our model, and then define the forward pass through the network using the `forward` function. Subclassing affords the flexibility to define custom layers, custom training loops, custom activation functions, and custom models. Let's define the same neural network model as above (i.e., Linear layer with an activation function after it), now using subclassing and using PyTorch's built in linear layer from `nn.Linear`." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "K4aCflPVyViD" }, "outputs": [], "source": [ "### Defining a model using subclassing ###\n", "\n", "class LinearWithSigmoidActivation(nn.Module):\n", " def __init__(self, num_inputs, num_outputs):\n", " super(LinearWithSigmoidActivation, self).__init__()\n", " '''TODO: define a model with a single Linear layer and sigmoid activation.'''\n", " self.linear = '''TODO: linear layer'''\n", " self.activation = '''TODO: sigmoid activation'''\n", "\n", " def forward(self, inputs):\n", " linear_output = self.linear(inputs)\n", " output = self.activation(linear_output)\n", " return output\n" ] }, { "cell_type": "markdown", "metadata": { "id": "goKCQ9dEGzRn" }, "source": [ "Let's test out our new model, using an example input, setting `n_input_nodes=2` and `n_output_nodes=3` as before." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "V-eNhSyRG6hl" }, "outputs": [], "source": [ "n_input_nodes = 2\n", "n_output_nodes = 3\n", "model = LinearWithSigmoidActivation(n_input_nodes, n_output_nodes)\n", "x_input = torch.tensor([[1, 2.]])\n", "y = model(x_input)\n", "print(f\"input shape: {x_input.shape}\")\n", "print(f\"output shape: {y.shape}\")\n", "print(f\"output result: {y}\")" ] }, { "cell_type": "markdown", "metadata": { "id": "HTIFMJLAzsyE" }, "source": [ "Importantly, `nn.Module` affords us a lot of flexibility to define custom models. For example, we can use boolean arguments in the `forward` function to specify different network behaviors, for example different behaviors during training and inference. Let's suppose under some instances we want our network to simply output the input, without any perturbation. We define a boolean argument `isidentity` to control this behavior:" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "P7jzGX5D1xT5" }, "outputs": [], "source": [ "### Custom behavior with subclassing nn.Module ###\n", "\n", "class LinearButSometimesIdentity(nn.Module):\n", " def __init__(self, num_inputs, num_outputs):\n", " super(LinearButSometimesIdentity, self).__init__()\n", " self.linear = nn.Linear(num_inputs, num_outputs)\n", "\n", " '''TODO: Implement the behavior where the network outputs the input, unchanged,\n", " under control of the isidentity argument.'''\n", " def forward(self, inputs, isidentity=False):\n", " ''' TODO '''\n" ] }, { "cell_type": "markdown", "metadata": { "id": "Ku4rcCGx5T3y" }, "source": [ "Let's test this behavior:" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "NzC0mgbk5dp2" }, "outputs": [], "source": [ "# Test the IdentityModel\n", "model = LinearButSometimesIdentity(num_inputs=2, num_outputs=3)\n", "x_input = torch.tensor([[1, 2.]])\n", "\n", "'''TODO: pass the input into the model and call with and without the input identity option.'''\n", "out_with_linear = # TODO\n", "\n", "out_with_identity = # TODO\n", "\n", "print(f\"input: {x_input}\")\n", "print(\"Network linear output: {}; network identity output: {}\".format(out_with_linear, out_with_identity))" ] }, { "cell_type": "markdown", "metadata": { "id": "7V1dEqdk6VI5" }, "source": [ "Now that we have learned how to define layers and models in PyTorch using both the Sequential API and subclassing `nn.Module`, we're ready to turn our attention to how to actually implement network training with backpropagation." ] }, { "cell_type": "markdown", "metadata": { "id": "dQwDhKn8kbO2" }, "source": [ "## 1.4 Automatic Differentiation in PyTorch\n", "\n", "In PyTorch, [`torch.autograd`](https://pytorch.org/docs/stable/autograd.html) is used for [automatic differentiation](https://en.wikipedia.org/wiki/Automatic_differentiation), which is critical for training deep learning models with [backpropagation](https://en.wikipedia.org/wiki/Backpropagation).\n", "\n", "We will use the PyTorch [`.backward()`](https://pytorch.org/docs/stable/generated/torch.Tensor.backward.html) method to trace operations for computing gradients. On a tensor, the [`requires_grad`](https://pytorch.org/docs/stable/generated/torch.Tensor.requires_grad_.html) attribute controls whether autograd should record operations on that tensor. When a forward pass is made through the network, PyTorch builds a computational graph dynamically; then, to compute the gradient, the `backward()` method is called to perform backpropagation.\n", "\n", "Let's compute the gradient of $ y = x^2 $:" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "tdkqk8pw5yJM" }, "outputs": [], "source": [ "### Gradient computation ###\n", "\n", "# y = x^2\n", "# Example: x = 3.0\n", "x = torch.tensor(3.0, requires_grad=True)\n", "y = x ** 2\n", "y.backward() # Compute the gradient\n", "\n", "dy_dx = x.grad\n", "print(\"dy_dx of y=x^2 at x=3.0 is: \", dy_dx)\n", "assert dy_dx == 6.0\n" ] }, { "cell_type": "markdown", "metadata": { "id": "JhU5metS5xF3" }, "source": [ "In training neural networks, we use differentiation and stochastic gradient descent (SGD) to optimize a loss function. Now that we have a sense of how PyTorch's autograd can be used to compute and access derivatives, we will look at an example where we use automatic differentiation and SGD to find the minimum of $ L=(x-x_f)^2 $. Here $x_f$ is a variable for a desired value we are trying to optimize for; $L$ represents a loss that we are trying to minimize. While we can clearly solve this problem analytically ($ x_{min}=x_f $), considering how we can compute this using PyTorch's autograd sets us up nicely for future labs where we use gradient descent to optimize entire neural network losses." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "attributes": { "classes": [ "py" ], "id": "" }, "id": "7g1yWiSXqEf-" }, "outputs": [], "source": [ "### Function minimization with autograd and gradient descent ###\n", "\n", "# Initialize a random value for our intial x\n", "x = torch.randn(1)\n", "print(f\"Initializing x={x.item()}\")\n", "\n", "learning_rate = 1e-2 # Learning rate\n", "history = []\n", "x_f = 4 # Target value\n", "\n", "\n", "# We will run gradient descent for a number of iterations. At each iteration, we compute the loss,\n", "# compute the derivative of the loss with respect to x, and perform the update.\n", "for i in range(500):\n", " x = torch.tensor([x], requires_grad=True)\n", "\n", " # TODO: Compute the loss as the square of the difference between x and x_f\n", " loss = # TODO\n", "\n", " # Backpropagate through the loss to compute gradients\n", " loss.backward()\n", "\n", " # Update x with gradient descent\n", " x = x.item() - learning_rate * x.grad\n", "\n", " history.append(x.item())\n", "\n", "# Plot the evolution of x as we optimize toward x_f!\n", "plt.plot(history)\n", "plt.plot([0, 500], [x_f, x_f])\n", "plt.legend(('Predicted', 'True'))\n", "plt.xlabel('Iteration')\n", "plt.ylabel('x value')\n", "plt.show()\n" ] }, { "cell_type": "markdown", "metadata": { "id": "pC7czCwk3ceH" }, "source": [ "Now, we have covered the fundamental concepts of PyTorch -- tensors, operations, neural networks, and automatic differentiation. Fire!!\n" ] } ], "metadata": { "accelerator": "GPU", "colab": { "collapsed_sections": [ "WBk0ZDWY-ff8" ], "name": "PT_Part1_Intro.ipynb", "provenance": [] }, "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.6" }, "vscode": { "interpreter": { "hash": "31f2aee4e71d21fbe5cf8b01ff0e069b9275f58929596ceb00d14d90e3e16cd6" } } }, "nbformat": 4, "nbformat_minor": 0 }