Final cleanup

34d7c2df · Marco Kuhlmann · ff23dad8 · 34d7c2df · 34d7c2df
Commit 34d7c2df authored 4 years ago by Marco Kuhlmann
--- a/lectures/lecture-04/Basic_text_processing.ipynb
+++ b/lectures/lecture-04/Basic_text_processing.ipynb
@@ -83,7 +83,7 @@
    "\n",
    "In English, words are often separated from each other by whitespace, but whitespace is neither necessary nor sufficient to define what constitutes a word. In our example, names such as *Latin America*, *Michael Wood*, and *London Review of Books* could arguably be treated as large words even though they contain spaces, while segments such as *day-to-day* need to be separated into several parts.\n",
    "\n",
-    "In real applications, we would have to tokenize the raw text ourselves, or use some library such as [spaCy](https://spacy.io) or [Stanford CoreNLP](https://stanfordnlp.github.io/CoreNLP/). The WikiText dataset, however, is pre-tokenized, so that we can extract the tokens from a given line using a simple helper function that splits on whitespace:"
+    "In real applications, we would have to tokenize the raw text ourselves, perhaps using some library such as [spaCy](https://spacy.io) or [Stanford CoreNLP](https://stanfordnlp.github.io/CoreNLP/). The WikiText dataset, however, is pre-tokenized, so that we can extract the tokens from a given line using a simple helper function that splits on whitespace:"
   ]
  },
  {
@@ -359,7 +359,7 @@
   "source": [
    "**🤔 Problem 4: Visualize Heaps’ law**\n",
    "\n",
-    "> Produce a plot that visualizes Heaps’ law for the WikiText dataset: on the $x$-axis, plot the text length; on the $y$-axis, plot the number of unique words at any given position in the text. How many new words, approximately, appear in the second half of the text?"
+    "> Produce a plot that visualizes Heaps’ law for the WikiText dataset: on the $x$-axis, plot the text length; on the $y$-axis, plot the number of unique words encountered up to a given position in the text. How many new words, approximately, appear in the second half of the text?"
   ]
  },
  {
@@ -393,7 +393,7 @@
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
-   "version": "3.8.2"
+   "version": "3.9.0"
  }
 },
 "nbformat": 4,

 %% Cell type:markdown id: tags:
 # Basic text processing
 %% Cell type:markdown id: tags:
 The purpose of this notebook is to introduce you to some basic concepts and techniques for processing text data.
 %% Cell type:markdown id: tags:
 ## Dataset
 %% Cell type:markdown id: tags:
 The dataset for this notebook is [WikiText](https://blog.einstein.ai/the-wikitext-long-term-dependency-language-modeling-dataset/), which is based on text extracted from ‘Good’ and ‘Featured’ articles on English Wikipedia. More specifically, we will be using the training portion of the WikiText-2 dataset.
 We start by opening the file and store its contents in a list:
 %% Cell type:code id: tags:
 ``` python
 with open('wiki.train.tokens', encoding='utf-8') as fp:
    wikitext = fp.readlines()
 ```
 %% Cell type:markdown id: tags:
 Here is a sample line from the file:
 %% Cell type:code id: tags:
 ``` python
 wikitext[1234]
 ```
 %% Cell type:markdown id: tags:
 **🤔 Problem 1: A close look at the data**
 > Take a close look at this line and compare it with the [original](https://en.wikipedia.org/wiki/The_Feast_of_the_Goat#Critical_reception). You will notice several differences between the two texts. What are your hypotheses as to how these differences come about?
 %% Cell type:markdown id: tags:
 ## Tokenization
 %% Cell type:markdown id: tags:
 Many tasks in natural language processing rely on first segmenting the raw text into words or word-like units. This task is called **tokenization**.
 In English, words are often separated from each other by whitespace, but whitespace is neither necessary nor sufficient to define what constitutes a word. In our example, names such as *Latin America*, *Michael Wood*, and *London Review of Books* could arguably be treated as large words even though they contain spaces, while segments such as *day-to-day* need to be separated into several parts.
-In real applications, we would have to tokenize the raw text ourselves, or use some library such as [spaCy](https://spacy.io) or [Stanford CoreNLP](https://stanfordnlp.github.io/CoreNLP/). The WikiText dataset, however, is pre-tokenized, so that we can extract the tokens from a given line using a simple helper function that splits on whitespace:
+In real applications, we would have to tokenize the raw text ourselves, perhaps using some library such as [spaCy](https://spacy.io) or [Stanford CoreNLP](https://stanfordnlp.github.io/CoreNLP/). The WikiText dataset, however, is pre-tokenized, so that we can extract the tokens from a given line using a simple helper function that splits on whitespace:
 %% Cell type:code id: tags:
 ``` python
 def tokens(lines):
    for line in lines:
        for token in line.rstrip().split():
            yield token
 ```
 %% Cell type:markdown id: tags:
 With this, we can count the number of tokens in the dataset:
 %% Cell type:code id: tags:
 ``` python
 sum(1 for t in tokens(wikitext))
 ```
 %% Cell type:markdown id: tags:
 ## Vocabulary
 %% Cell type:markdown id: tags:
 The term *word* is used ambiguously in natural language processing. It can either refer to a *word token*, a word as it occurs in the running text; or to a *word type*, a unique word. To illustrate the distinction: We are interested in *word tokens* when we want to count the length of a text, and in *word types* when we want to compile a dictionary of all words in the text material.
 The set of unique words in a dataset or task is called the **vocabulary**.
 %% Cell type:code id: tags:
 ``` python
 vocab = set(tokens(wikitext))
 ```
 %% Cell type:markdown id: tags:
 Show the size of the vocabulary:
 %% Cell type:code id: tags:
 ``` python
 len(vocab)
 ```
 %% Cell type:markdown id: tags:
 **🤔 Problem 2: Missing words**
 > When you compare this number to the corresponding number on the [WikiText website](https://blog.einstein.ai/the-wikitext-long-term-dependency-language-modeling-dataset/), you will notice a small difference. Try to find out what causes this difference, and how you can change the code so that you get the same number as on the website.
 %% Cell type:markdown id: tags:
 For some natural language processing tasks, tokenization is the only thing we need to do in order to compile the vocabulary. For other tasks, we may want to apply additional pre-processing, and/or include only the most frequent words into the vocabulary.
 %% Cell type:markdown id: tags:
 **🤔 Problem 3: Normalization**
 > The term **text normalization** refers to all kinds of pre-processing that we apply to a text in order to bring it into a more standard, convenient form. Two standard normalization techniques are lowercasing and the filtering-out of non-alphabetical tokens. What effect do these techniques have on the size of the vocabulary?
 %% Cell type:markdown id: tags:
 ## String-to-index mapping
 %% Cell type:markdown id: tags:
 One common task when processing text with neural networks is to encode words and other strings into contiguous ranges of integers, so that we can map them to the components of a vector. A standard pattern for doing this looks as follows:
 %% Cell type:code id: tags:
 ``` python
 w2i = {}
 for token in tokens(wikitext):
    if token not in w2i:
        w2i[token] = len(w2i)
 ```
 %% Cell type:markdown id: tags:
 Assuming that we already have a representation of the vocabulary, we can also use a different pattern:
 %% Cell type:code id: tags:
 ``` python
 w2i = {w: i for i, w in enumerate(sorted(vocab))}
 ```
 %% Cell type:markdown id: tags:
 What index did the word *language* get?
 %% Cell type:code id: tags:
 ``` python
 w2i['language']
 ```
 %% Cell type:markdown id: tags:
 Sometimes we also need to invert the string-to-integer mapping:
 %% Cell type:code id: tags:
 ``` python
 i2w = {i: s for s, i in w2i.items()}
 ```
 %% Cell type:markdown id: tags:
 Decoding should be the inverse of encoding:
 %% Cell type:code id: tags:
 ``` python
 assert i2w[w2i['language']] == 'language'
 ```
 %% Cell type:markdown id: tags:
 ## Power-law behaviour of language
 %% Cell type:markdown id: tags:
 Language data often exhibits a power-law behaviour. We want to check whether this also applies to the WikiText data. In particular, we want to check whether the word frequencies follow the expected Zipfian pattern. We start by collecting the frequencies:
 %% Cell type:code id: tags:
 ``` python
 from collections import Counter
 counter = Counter(tokens(wikitext))
 ```
 %% Cell type:markdown id: tags:
 How often does the word *language* occur in the data?
 %% Cell type:code id: tags:
 ``` python
 counter['language']
 ```
 %% Cell type:markdown id: tags:
 Prepare `matplotlib` for plotting:
 %% Cell type:code id: tags:
 ``` python
 %matplotlib inline
 import matplotlib.pyplot as plt
 %config InlineBackend.figure_format = 'retina'
 ```
 %% Cell type:markdown id: tags:
 Produce the rank/frequency plot for the 30,000 most common words:
 %% Cell type:code id: tags:
 ``` python
 words, counts = zip(*counter.most_common(30000))
 plt.xlabel('Rank')
 plt.yscale('log')
 plt.ylabel('Frequency')
 plt.plot(range(len(words)), counts)
 plt.show()
 ```
 %% Cell type:markdown id: tags:
 **🤔 Problem 4: Visualize Heaps’ law**
-> Produce a plot that visualizes Heaps’ law for the WikiText dataset: on the $x$-axis, plot the text length; on the $y$-axis, plot the number of unique words at any given position in the text. How many new words, approximately, appear in the second half of the text?
+> Produce a plot that visualizes Heaps’ law for the WikiText dataset: on the $x$-axis, plot the text length; on the $y$-axis, plot the number of unique words encountered up to a given position in the text. How many new words, approximately, appear in the second half of the text?
 %% Cell type:markdown id: tags:
 That’s all, folks!

--- a/lectures/lecture-06/Logistic_regression.ipynb
+++ b/lectures/lecture-06/Logistic_regression.ipynb
@@ -15,7 +15,7 @@
    "id": "2ZEewrx2YD2K"
   },
   "source": [
-    "In this notebook you will learn how to implement logistic regression in PyTorch and apply it to the task of sentiment analysis."
+    "In this notebook you will learn how to implement logistic regression in PyTorch, and apply it to the task of sentiment analysis."
   ]
  },
  {
@@ -35,7 +35,7 @@
   "source": [
    "Our dataset is derived from the [Stanford Sentiment Treebank](https://nlp.stanford.edu/sentiment/), which consists of 11,855 sentences extracted from movie reviews. Each sentence has been manually labelled with a rating between 0 (very negative) and 4 (very positive) towards the movie at hand. For this data, sentiment analysis can be framed as a multi-class classification problem. We have pre-processed the original data by tokenization, removing non-alphabetical tokens, and lowercasing.\n",
    "\n",
-    "The following helper function loads the rating-labelled movie reviews from a tab-separated file. It returns two lists: one containing the tokenized reviews (lists of strings), and one containing the corresponding ratings (integers in the range 0–4)."
+    "The following helper function loads the rating-labelled movie reviews from a tab-separated file. It returns a list of pairs where the first component of each pair is a tokenized review (represented a lists of string tokens) and the second component is the corresponding ratings (an integer in the range 0–4)."
   ]
  },
  {
@@ -46,14 +46,30 @@
   },
   "outputs": [],
   "source": [
-    "def load_data(filename):\n",
+    "def load_data(filename, max_length=20):\n",
-    "    reviews, ratings = [], []\n",
+    "    items = []\n",
    "    with open(filename, 'rt', encoding='utf-8') as fp:\n",
    "        for line in fp:\n",
    "            review, rating = line.rstrip().split('\\t')\n",
-    "            reviews.append(review.split())\n",
+    "            items.append((review.split()[:max_length], int(rating)))\n",
-    "            ratings.append(int(rating))\n",
+    "    return items"
-    "    return reviews, ratings"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "We use this function to load the training data and the development data:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "train_data = load_data('sst-5-train.txt')\n",
+    "dev_data = load_data('sst-5-dev.txt')"
   ]
  },
  {
@@ -62,7 +78,7 @@
    "id": "jzvqnD4dYD2P"
   },
   "source": [
-    "We use this function to load the training data and the validation data:"
+    "The next cell prints the number of examples in the training data:"
   ]
  },
  {
@@ -77,10 +93,7 @@
   },
   "outputs": [],
   "source": [
-    "train_reviews, train_ratings = load_data('sst-5-train.txt')\n",
+    "print(len(train_data))"
-    "dev_reviews, dev_ratings = load_data('sst-5-dev.txt')\n",
-    "\n",
-    "len(train_reviews), len(dev_reviews)"
   ]
  },
  {
@@ -111,22 +124,38 @@
   },
   "outputs": [],
   "source": [
-    "def make_vocab(reviews):\n",
+    "def make_vocab(data):\n",
    "    vocab = {}\n",
-    "    for review in reviews:\n",
+    "    for review, rating in data:\n",
    "        for t in review:\n",
    "            if t not in vocab:\n",
    "                vocab[t] = len(vocab)\n",
    "    return vocab"
   ]
  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "We create the vocabulary from the training data:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "vocab = make_vocab(train_data)"
+   ]
+  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "P5jO6laPYD2Q"
   },
   "source": [
-    "Running the cell below shows the size of our vocabulary:"
+    "Running the cell below prints the size of the vocabulary:"
   ]
  },
  {
@@ -141,9 +170,7 @@
   },
   "outputs": [],
   "source": [
-    "vocab = make_vocab(train_reviews)\n",
+    "print(len(vocab))"
-    "\n",
-    "len(vocab)"
   ]
  },
  {
@@ -165,15 +192,17 @@
   "source": [
    "import torch\n",
    "\n",
-    "def vectorize(vocab, reviews, ratings):\n",
+    "def vectorize(vocab, data):\n",
    "    xs = []\n",
-    "    for review, rating in zip(reviews, ratings):\n",
+    "    ys = []\n",
+    "    for review, rating in data:\n",
    "        x = [0] * len(vocab)\n",
    "        for w in review:\n",
    "            if w in vocab:\n",
    "                x[vocab[w]] += 1\n",
    "        xs.append(x)\n",
-    "    return torch.FloatTensor(xs), torch.LongTensor(ratings)"
+    "        ys.append(rating)\n",
+    "    return torch.FloatTensor(xs), torch.LongTensor(ys)"
   ]
  },
  {
@@ -182,7 +211,7 @@
    "id": "OL51CWX9YD2R"
   },
   "source": [
-    "We vectorize the training data:"
+    "We vectorize the training data and the development data:"
   ]
  },
  {
@@ -197,9 +226,8 @@
   },
   "outputs": [],
   "source": [
-    "train_x, train_y = vectorize(vocab, train_reviews, train_ratings)\n",
+    "train_x, train_y = vectorize(vocab, train_data)\n",
-    "\n",
+    "dev_x, dev_y = vectorize(vocab, dev_data)"
-    "train_x.size(), train_y.size()"
   ]
  },
  {
@@ -208,7 +236,7 @@
    "id": "H1vZmcwsYD2R"
   },
   "source": [
-    "We also vectorize the development data."
+    "The next cell prints the shapes of the resulting tensors:"
   ]
  },
  {
@@ -223,9 +251,8 @@
   },
   "outputs": [],
   "source": [
-    "dev_x, dev_y = vectorize(vocab, dev_reviews, dev_ratings)\n",
+    "print('Training data:', train_x.size(), train_y.size())\n",
-    "\n",
+    "print('Development data:', dev_x.size(), dev_y.size())"
-    "dev_x.size(), dev_y.size()"
   ]
  },
  {
@@ -264,7 +291,7 @@
    "id": "2-fntYQuYD2S"
   },
   "source": [
-    "By always predicting the majority class in the training data (rating&nbsp;3), we can get a validation accuracy of slightly above 25%."
+    "By always predicting the majority class in the training data (rating&nbsp;3), we can get an accuracy on the development data of slightly above 25%."
   ]
  },
  {
@@ -297,7 +324,7 @@
    "id": "uX1ArQIMYD2S"
   },
   "source": [
-    "We are now ready to set up a logistic regression model and train it using the cross-entropy loss function. Recall that a logistic regression model consists of a linear layer followed by the softmax function. In PyTorch, linear layers are implemented by the class [`nn.Linear`](https://pytorch.org/docs/stable/generated/torch.nn.Linear.html), and cross-entropy loss is implemented by the function [`cross_entropy()`](https://pytorch.org/docs/stable/nn.functional.html#cross-entropy). The softmax function will be computed inside the loss function; see the code below."
+    "We are now ready to set up a logistic regression model and train it using the categorical cross-entropy loss function. Recall that a logistic regression model consists of a linear layer followed by the softmax function. In PyTorch, linear layers are implemented by the class [`nn.Linear`](https://pytorch.org/docs/stable/generated/torch.nn.Linear.html), and cross-entropy loss is implemented by the function [`cross_entropy()`](https://pytorch.org/docs/stable/nn.functional.html#cross-entropy). The softmax function will be computed inside the loss function; see the note below."
   ]
  },
  {
@@ -354,18 +381,19 @@
   },
   "outputs": [],
   "source": [
-    "def train(vocab, x, y, n_epochs=40, batch_size=1068, lr=1e-4):\n",
+    "def train(n_epochs=20, batch_size=48, lr=1e-1):\n",
    "    # Initialize the model\n",
    "    model = nn.Linear(len(vocab), 5)\n",
    "    \n",
-    "    # Initialize the optimizer. Here we will use Adam instead of plain SGD.\n",
+    "    # Initialize the optimizer\n",
-    "    optimizer = optim.Adam(model.parameters(), lr=lr)\n",
+    "    optimizer = optim.SGD(model.parameters(), lr=lr)\n",
    "    \n",
-    "    # We will train for n_epochs epochs\n",
+    "    # We train for several epochs\n",
    "    for t in range(n_epochs):\n",
    "        \n",
    "        # In each epoch, we loop over all the minibatches\n",
-    "        for bx, by in minibatches(x, y, batch_size):\n",
+    "        for bx, by in minibatches(train_x, train_y, batch_size):\n",
+    "            \n",
    "            # Reset the accumulated gradients\n",
    "            optimizer.zero_grad()\n",
    "            \n",
@@ -401,7 +429,7 @@
    "id": "g508BX-SYD2U"
   },
   "source": [
-    "Here is an embellished version of the training loop that plots the per-epoch losses and the per-epoch accuracies on the validation data."
+    "Here is an embellished version of the training loop that plots the per-epoch losses and the per-epoch accuracies on the development data."
   ]
  },
  {
@@ -419,46 +447,43 @@
    "%matplotlib inline\n",
    "%config InlineBackend.figure_format = 'retina'\n",
    "\n",
-    "def train(vocab, x, y, n_epochs=40, batch_size=1068, lr=1e-4, validation_data=None):\n",
+    "def train(n_epochs=20, batch_size=48, lr=1e-1):\n",
    "    model = nn.Linear(len(vocab), 5)\n",
-    "    optimizer = optim.Adam(model.parameters(), lr=lr)\n",
+    "    optimizer = optim.SGD(model.parameters(), lr=lr)\n",
    "    losses = []\n",
-    "    valid_losses = []\n",
+    "    dev_losses = []\n",
-    "    valid_accuracies = []\n",
+    "    dev_accuracies = []\n",
-    "    if validation_data:\n",
-    "        valid_x, valid_y = validation_data\n",
    "    for t in range(n_epochs):\n",
    "        model.train()\n",
    "        running_loss = 0\n",
-    "        for bx, by in minibatches(x, y, batch_size):\n",
+    "        for bx, by in minibatches(train_x, train_y, batch_size):\n",
    "            optimizer.zero_grad()\n",
    "            output = model.forward(bx)\n",
    "            loss = F.cross_entropy(output, by)\n",
    "            loss.backward()\n",
    "            optimizer.step()\n",
    "            running_loss += loss.item() * len(bx)\n",
-    "        losses.append(running_loss / len(x))\n",
+    "        losses.append(running_loss / len(train_x))\n",
-    "        if validation_data:\n",
+    "        model.eval()\n",
-    "            model.eval()\n",
+    "        with torch.no_grad():\n",
-    "            with torch.no_grad():\n",
+    "            dev_output = model.forward(dev_x)\n",
-    "                valid_output = model.forward(valid_x)\n",
+    "            dev_loss = F.cross_entropy(dev_output, dev_y)\n",
-    "                valid_y_pred = torch.argmax(valid_output, axis=1)\n",
+    "            dev_losses.append(dev_loss)\n",
-    "                valid_loss = F.cross_entropy(valid_output, valid_y)\n",
+    "            dev_y_pred = torch.argmax(dev_output, axis=1)\n",
-    "                valid_losses.append(valid_loss)\n",
+    "            dev_acc = accuracy(dev_y_pred, dev_y)\n",
-    "                valid_acc = accuracy(valid_y_pred, valid_y)\n",
+    "            dev_accuracies.append(dev_acc)\n",
-    "                valid_accuracies.append(valid_acc)\n",
+    "            print('\\repoch {}, dev loss {:.4f}, dev acc {:.4f}'.format(t, dev_loss, dev_acc), end='')\n",
-    "                print('\\repoch {}, valid loss {:.4f}, valid acc {:.4f}'.format(t, valid_loss, valid_acc), end='')\n",
    "    print()\n",
    "    plt.figure(figsize=(15, 6))\n",
    "    plt.subplot(121)\n",
    "    plt.plot(losses)\n",
-    "    plt.plot(valid_losses)\n",
+    "    plt.plot(dev_losses)\n",
    "    plt.xlabel('Epoch')\n",
    "    plt.ylabel('Average loss')\n",
    "    plt.subplot(122)\n",
-    "    plt.plot(valid_accuracies)\n",
+    "    plt.plot(dev_accuracies)\n",
    "    plt.xlabel('Epoch')\n",
-    "    plt.ylabel('Validation set accuracy')\n",
+    "    plt.ylabel('Development set accuracy')\n",
    "    return model"
   ]
  },
@@ -485,7 +510,7 @@
   },
   "outputs": [],
   "source": [
-    "model = train(vocab, train_x, train_y, validation_data=(dev_x, dev_y))"
+    "train()"
   ]
  },
  {
@@ -526,26 +551,20 @@
    "# CHANGED: Set the device variable\n",
    "device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n",
    "\n",
-    "def train(vocab, x, y, n_epochs=40, batch_size=1068, lr=1e-4, validation_data=None):\n",
+    "def train(n_epochs=20, batch_size=48, lr=1e-1):\n",
+    "    \n",
    "    # CHANGED: Send the model to the available device\n",
-    "    model = nn.Linear(len(vocab), 10).to(device)\n",
+    "    model = nn.Linear(len(vocab), 5).to(device)\n",
    "    \n",
-    "    optimizer = optim.Adam(model.parameters(), lr=lr)\n",
+    "    optimizer = optim.SGD(model.parameters(), lr=lr)\n",
    "    losses = []\n",
-    "    valid_losses = []\n",
+    "    dev_losses = []\n",
-    "    valid_accuracies = []\n",
+    "    dev_accuracies = []\n",
-    "    \n",
-    "    if validation_data:\n",
-    "        valid_x, valid_y = validation_data\n",
-    "        \n",
-    "        # CHANGED: Send the validation data to the available device\n",
-    "        valid_x = valid_x.to(device)\n",
-    "        valid_y = valid_y.to(device)\n",
-    "    \n",
    "    for t in range(n_epochs):\n",
    "        model.train()\n",
    "        running_loss = 0\n",
-    "        for bx, by in minibatches(x, y, batch_size):\n",
+    "        for bx, by in minibatches(train_x, train_y, batch_size):\n",
+    "            \n",
    "            # CHANGED: Send the minibatches to the available device\n",
    "            bx = bx.to(device)\n",
    "            by = by.to(device)\n",
@@ -556,28 +575,27 @@
    "            loss.backward()\n",
    "            optimizer.step()\n",
    "            running_loss += loss.item() * len(bx)\n",
-    "        losses.append(running_loss / len(x))\n",
+    "        losses.append(running_loss / len(train_x))\n",
-    "        if validation_data:\n",
+    "        model.eval()\n",
-    "            model.eval()\n",
+    "        with torch.no_grad():\n",
-    "            with torch.no_grad():\n",
+    "            dev_output = model.forward(dev_x.to(device))\n",
-    "                valid_output = model.forward(valid_x)\n",
+    "            dev_loss = F.cross_entropy(dev_output, dev_y.to(device))\n",
-    "                valid_y_pred = torch.argmax(valid_output, axis=1)\n",
+    "            dev_losses.append(dev_loss)\n",
-    "                valid_loss = F.cross_entropy(valid_output, valid_y)\n",
+    "            dev_y_pred = torch.argmax(dev_output, axis=1)\n",
-    "                valid_losses.append(valid_loss)\n",
+    "            dev_acc = accuracy(dev_y, dev_y_pred)\n",
-    "                valid_acc = accuracy(valid_y, valid_y_pred)\n",
+    "            dev_accuracies.append(dev_acc)\n",
-    "                valid_accuracies.append(valid_acc)\n",
+    "            print('\\repoch {}, dev loss {:.4f}, dev acc {:.4f}'.format(t, dev_loss, dev_acc), end='')\n",
-    "                print('\\repoch {}, valid loss {:.4f}, valid acc {:.4f}'.format(t, valid_loss, valid_acc), end='')\n",
    "    print()\n",
    "    plt.figure(figsize=(15, 6))\n",
    "    plt.subplot(121)\n",
    "    plt.plot(losses)\n",
-    "    plt.plot(valid_losses)\n",
+    "    plt.plot(dev_losses)\n",
    "    plt.xlabel('Epoch')\n",
    "    plt.ylabel('Average loss')\n",
    "    plt.subplot(122)\n",
-    "    plt.plot(valid_accuracies)\n",
+    "    plt.plot(dev_accuracies)\n",
    "    plt.xlabel('Epoch')\n",
-    "    plt.ylabel('Validation set accuracy')\n",
+    "    plt.ylabel('Development set accuracy')\n",
    "    return model"
   ]
  },
@@ -603,7 +621,7 @@
   },
   "outputs": [],
   "source": [
-    "model = train(vocab, train_x, train_y, validation_data=(dev_x, dev_y))"
+    "model = train()"
   ]
  },
  {

 %% Cell type:markdown id: tags:
 # Implementing logistic regression
 %% Cell type:markdown id: tags:
-In this notebook you will learn how to implement logistic regression in PyTorch and apply it to the task of sentiment analysis.
+In this notebook you will learn how to implement logistic regression in PyTorch, and apply it to the task of sentiment analysis.
 %% Cell type:markdown id: tags:
 ## Loading the data
 %% Cell type:markdown id: tags:
 Our dataset is derived from the [Stanford Sentiment Treebank](https://nlp.stanford.edu/sentiment/), which consists of 11,855 sentences extracted from movie reviews. Each sentence has been manually labelled with a rating between 0 (very negative) and 4 (very positive) towards the movie at hand. For this data, sentiment analysis can be framed as a multi-class classification problem. We have pre-processed the original data by tokenization, removing non-alphabetical tokens, and lowercasing.
-The following helper function loads the rating-labelled movie reviews from a tab-separated file. It returns two lists: one containing the tokenized reviews (lists of strings), and one containing the corresponding ratings (integers in the range 0–4).
+The following helper function loads the rating-labelled movie reviews from a tab-separated file. It returns a list of pairs where the first component of each pair is a tokenized review (represented a lists of string tokens) and the second component is the corresponding ratings (an integer in the range 0–4).
 %% Cell type:code id: tags:
 ``` python
-def load_data(filename):
+def load_data(filename, max_length=20):
-    reviews, ratings = [], []
+    items = []
    with open(filename, 'rt', encoding='utf-8') as fp:
        for line in fp:
            review, rating = line.rstrip().split('\t')
-            reviews.append(review.split())
+            items.append((review.split()[:max_length], int(rating)))
-            ratings.append(int(rating))
+    return items
-    return reviews, ratings
 ```
 %% Cell type:markdown id: tags:
-We use this function to load the training data and the validation data:
+We use this function to load the training data and the development data:
 %% Cell type:code id: tags:
 ``` python
-train_reviews, train_ratings = load_data('sst-5-train.txt')
+train_data = load_data('sst-5-train.txt')
-dev_reviews, dev_ratings = load_data('sst-5-dev.txt')
+dev_data = load_data('sst-5-dev.txt')
+```
+%% Cell type:markdown id: tags:
+The next cell prints the number of examples in the training data:
+%% Cell type:code id: tags:
-len(train_reviews), len(dev_reviews)
+``` python
+print(len(train_data))
 ```
 %% Cell type:markdown id: tags:
 ## Vectorizing the data
 %% Cell type:markdown id: tags:
 To process the movie reviews using neural networks, we need to convert them into vectors of numerical values. A simple but common vector representation in natural language processing is the **bag-of-words**. Under this representation, we record *which* words occur in a review and *how often* they occur, but completely ignore their ordering. This is the representation that we will use in this notebook.
 We first construct a mapping from words to vector indices. The domain of this mapping will be our vocabulary.
 %% Cell type:code id: tags:
 ``` python
-def make_vocab(reviews):
+def make_vocab(data):
    vocab = {}
-    for review in reviews:
+    for review, rating in data:
        for t in review:
            if t not in vocab:
                vocab[t] = len(vocab)
    return vocab
 ```
 %% Cell type:markdown id: tags:
-Running the cell below shows the size of our vocabulary:
+We create the vocabulary from the training data:
 %% Cell type:code id: tags:
 ``` python
-vocab = make_vocab(train_reviews)
+vocab = make_vocab(train_data)
+```
+%% Cell type:markdown id: tags:
+Running the cell below prints the size of the vocabulary:
-len(vocab)
+%% Cell type:code id: tags:
+``` python
+print(len(vocab))
 ```
 %% Cell type:markdown id: tags:
 Next, we map each review to a vector that holds the counts for all the words in the vocabulary, and collect all sentence vectors into a PyTorch tensor. We do the same thing for all ratings, which results in a vector of integers.
 %% Cell type:code id: tags:
 ``` python
 import torch
-def vectorize(vocab, reviews, ratings):
+def vectorize(vocab, data):
    xs = []
-    for review, rating in zip(reviews, ratings):
+    ys = []
+    for review, rating in data:
        x = [0] * len(vocab)
        for w in review:
            if w in vocab:
                x[vocab[w]] += 1
        xs.append(x)
-    return torch.FloatTensor(xs), torch.LongTensor(ratings)
+        ys.append(rating)
+    return torch.FloatTensor(xs), torch.LongTensor(ys)
 ```
 %% Cell type:markdown id: tags:
-We vectorize the training data:
+We vectorize the training data and the development data:
 %% Cell type:code id: tags:
 ``` python
-train_x, train_y = vectorize(vocab, train_reviews, train_ratings)
+train_x, train_y = vectorize(vocab, train_data)
+dev_x, dev_y = vectorize(vocab, dev_data)
-train_x.size(), train_y.size()
 ```
 %% Cell type:markdown id: tags:
-We also vectorize the development data.
+The next cell prints the shapes of the resulting tensors:
 %% Cell type:code id: tags:
 ``` python
-dev_x, dev_y = vectorize(vocab, dev_reviews, dev_ratings)
+print('Training data:', train_x.size(), train_y.size())
+print('Development data:', dev_x.size(), dev_y.size())
-dev_x.size(), dev_y.size()
 ```
 %% Cell type:markdown id: tags:
 ## Evaluation
 %% Cell type:markdown id: tags:
 The standard evaluation measure for our dataset is **accuracy** – the percentage of reviews for which the classifier predicts the correct rating.
 %% Cell type:code id: tags:
 ``` python
 def accuracy(y_pred, y):
    return torch.mean(torch.eq(y_pred, y).float()).item()
 ```
 %% Cell type:markdown id: tags:
-By always predicting the majority class in the training data (rating&nbsp;3), we can get a validation accuracy of slightly above 25%.
+By always predicting the majority class in the training data (rating&nbsp;3), we can get an accuracy on the development data of slightly above 25%.
 %% Cell type:code id: tags:
 ``` python
 accuracy(torch.full_like(dev_y, 3), dev_y)
 ```
 %% Cell type:markdown id: tags:
 ## Training the model
 %% Cell type:markdown id: tags:
-We are now ready to set up a logistic regression model and train it using the cross-entropy loss function. Recall that a logistic regression model consists of a linear layer followed by the softmax function. In PyTorch, linear layers are implemented by the class [`nn.Linear`](https://pytorch.org/docs/stable/generated/torch.nn.Linear.html), and cross-entropy loss is implemented by the function [`cross_entropy()`](https://pytorch.org/docs/stable/nn.functional.html#cross-entropy). The softmax function will be computed inside the loss function; see the code below.
+We are now ready to set up a logistic regression model and train it using the categorical cross-entropy loss function. Recall that a logistic regression model consists of a linear layer followed by the softmax function. In PyTorch, linear layers are implemented by the class [`nn.Linear`](https://pytorch.org/docs/stable/generated/torch.nn.Linear.html), and cross-entropy loss is implemented by the function [`cross_entropy()`](https://pytorch.org/docs/stable/nn.functional.html#cross-entropy). The softmax function will be computed inside the loss function; see the note below.
 %% Cell type:code id: tags:
 ``` python
 import torch.nn as nn
 import torch.nn.functional as F
 import torch.optim as optim
 ```
 %% Cell type:markdown id: tags:
 We will train our model using minibatch gradient descent. The following function splits the data into randomly sampled minibatches of the specified size.
 %% Cell type:code id: tags:
 ``` python
 def minibatches(x, y, batch_size):
    random_indices = torch.randperm(x.size(0))
    for i in range(0, x.size(0) - batch_size + 1, batch_size):
        batch_indices = random_indices[i:i+batch_size]
        yield x[batch_indices], y[batch_indices]
 ```
 %% Cell type:markdown id: tags:
 With this we can now write our training loop:
 %% Cell type:code id: tags:
 ``` python
-def train(vocab, x, y, n_epochs=40, batch_size=1068, lr=1e-4):
+def train(n_epochs=20, batch_size=48, lr=1e-1):
    # Initialize the model
    model = nn.Linear(len(vocab), 5)
-    # Initialize the optimizer. Here we will use Adam instead of plain SGD.
+    # Initialize the optimizer
-    optimizer = optim.Adam(model.parameters(), lr=lr)
+    optimizer = optim.SGD(model.parameters(), lr=lr)
-    # We will train for n_epochs epochs
+    # We train for several epochs
    for t in range(n_epochs):
        # In each epoch, we loop over all the minibatches
-        for bx, by in minibatches(x, y, batch_size):
+        for bx, by in minibatches(train_x, train_y, batch_size):
            # Reset the accumulated gradients
            optimizer.zero_grad()
            # Forward pass
            output = model.forward(bx)
            # Compute the loss
            loss = F.cross_entropy(output, by)
            # Backward pass; propagates the loss and computes the gradients
            loss.backward()
            # Update the parameters of the model
            optimizer.step()
    return model
 ```
 %% Cell type:markdown id: tags:
 **⚠️ The softmax is implicit!**
 One thing that you will note when you go through the code of the training loop is that there is no explicit call to the softmax function. The outputs of the network are not normalized probabilities but just scores (logits). The softmax is computed inside `cross_entropy()`.
 %% Cell type:markdown id: tags:
-Here is an embellished version of the training loop that plots the per-epoch losses and the per-epoch accuracies on the validation data.
+Here is an embellished version of the training loop that plots the per-epoch losses and the per-epoch accuracies on the development data.
 %% Cell type:code id: tags:
 ``` python
 # Same training loop with evaluation and plotting
 import matplotlib.pyplot as plt
 %matplotlib inline
 %config InlineBackend.figure_format = 'retina'
-def train(vocab, x, y, n_epochs=40, batch_size=1068, lr=1e-4, validation_data=None):
+def train(n_epochs=20, batch_size=48, lr=1e-1):
    model = nn.Linear(len(vocab), 5)
-    optimizer = optim.Adam(model.parameters(), lr=lr)
+    optimizer = optim.SGD(model.parameters(), lr=lr)
    losses = []
-    valid_losses = []
+    dev_losses = []
-    valid_accuracies = []
+    dev_accuracies = []
-    if validation_data:
-        valid_x, valid_y = validation_data
    for t in range(n_epochs):
        model.train()
        running_loss = 0
-        for bx, by in minibatches(x, y, batch_size):
+        for bx, by in minibatches(train_x, train_y, batch_size):
            optimizer.zero_grad()
            output = model.forward(bx)
            loss = F.cross_entropy(output, by)
            loss.backward()
            optimizer.step()
            running_loss += loss.item() * len(bx)
-        losses.append(running_loss / len(x))
+        losses.append(running_loss / len(train_x))
-        if validation_data:
+        model.eval()
-            model.eval()
+        with torch.no_grad():
-            with torch.no_grad():
+            dev_output = model.forward(dev_x)
-                valid_output = model.forward(valid_x)
+            dev_loss = F.cross_entropy(dev_output, dev_y)
-                valid_y_pred = torch.argmax(valid_output, axis=1)
+            dev_losses.append(dev_loss)
-                valid_loss = F.cross_entropy(valid_output, valid_y)
+            dev_y_pred = torch.argmax(dev_output, axis=1)
-                valid_losses.append(valid_loss)
+            dev_acc = accuracy(dev_y_pred, dev_y)
-                valid_acc = accuracy(valid_y_pred, valid_y)
+            dev_accuracies.append(dev_acc)
-                valid_accuracies.append(valid_acc)
+            print('\repoch {}, dev loss {:.4f}, dev acc {:.4f}'.format(t, dev_loss, dev_acc), end='')
-                print('\repoch {}, valid loss {:.4f}, valid acc {:.4f}'.format(t, valid_loss, valid_acc), end='')
    print()
    plt.figure(figsize=(15, 6))
    plt.subplot(121)
    plt.plot(losses)
-    plt.plot(valid_losses)
+    plt.plot(dev_losses)
    plt.xlabel('Epoch')
    plt.ylabel('Average loss')
    plt.subplot(122)
-    plt.plot(valid_accuracies)
+    plt.plot(dev_accuracies)
    plt.xlabel('Epoch')
-    plt.ylabel('Validation set accuracy')
+    plt.ylabel('Development set accuracy')
    return model
 ```
 %% Cell type:markdown id: tags:
 We are ready to train:
 %% Cell type:code id: tags:
 ``` python
-model = train(vocab, train_x, train_y, validation_data=(dev_x, dev_y))
+train()
 ```
 %% Cell type:markdown id: tags:
 ## Using a GPU
 %% Cell type:markdown id: tags:
 The data set that we used in this notebook is very small, and the model is very simple. For larger datasets and/or models, you may want to use a GPU, say on a service such as [Colab](http://colab.research.google.com). Fortunately, making a model ready for GPU training is rather straightforward in PyTorch: You only need to ‘send’ the model and its input to the correct device.
 The next code cell contains a GPU-enabled version of the (embellished) training loop. Modified lines are marked as `CHANGED`.
 %% Cell type:code id: tags:
 ``` python
 # GPU-enabled training loop
 import matplotlib.pyplot as plt
 %matplotlib inline
 %config InlineBackend.figure_format = 'retina'
 # CHANGED: Set the device variable
 device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
-def train(vocab, x, y, n_epochs=40, batch_size=1068, lr=1e-4, validation_data=None):
+def train(n_epochs=20, batch_size=48, lr=1e-1):
    # CHANGED: Send the model to the available device
-    model = nn.Linear(len(vocab), 10).to(device)
+    model = nn.Linear(len(vocab), 5).to(device)
-    optimizer = optim.Adam(model.parameters(), lr=lr)
+    optimizer = optim.SGD(model.parameters(), lr=lr)
    losses = []
-    valid_losses = []
+    dev_losses = []
-    valid_accuracies = []
+    dev_accuracies = []
-    if validation_data:
-        valid_x, valid_y = validation_data
-        # CHANGED: Send the validation data to the available device
-        valid_x = valid_x.to(device)
-        valid_y = valid_y.to(device)
    for t in range(n_epochs):
        model.train()
        running_loss = 0
-        for bx, by in minibatches(x, y, batch_size):
+        for bx, by in minibatches(train_x, train_y, batch_size):
            # CHANGED: Send the minibatches to the available device
            bx = bx.to(device)
            by = by.to(device)
            optimizer.zero_grad()
            output = model.forward(bx)
            loss = F.cross_entropy(output, by)
            loss.backward()
            optimizer.step()
            running_loss += loss.item() * len(bx)
-        losses.append(running_loss / len(x))
+        losses.append(running_loss / len(train_x))
-        if validation_data:
+        model.eval()
-            model.eval()
+        with torch.no_grad():
-            with torch.no_grad():
+            dev_output = model.forward(dev_x.to(device))
-                valid_output = model.forward(valid_x)
+            dev_loss = F.cross_entropy(dev_output, dev_y.to(device))
-                valid_y_pred = torch.argmax(valid_output, axis=1)
+            dev_losses.append(dev_loss)
-                valid_loss = F.cross_entropy(valid_output, valid_y)
+            dev_y_pred = torch.argmax(dev_output, axis=1)
-                valid_losses.append(valid_loss)
+            dev_acc = accuracy(dev_y, dev_y_pred)
-                valid_acc = accuracy(valid_y, valid_y_pred)
+            dev_accuracies.append(dev_acc)
-                valid_accuracies.append(valid_acc)
+            print('\repoch {}, dev loss {:.4f}, dev acc {:.4f}'.format(t, dev_loss, dev_acc), end='')
-                print('\repoch {}, valid loss {:.4f}, valid acc {:.4f}'.format(t, valid_loss, valid_acc), end='')
    print()
    plt.figure(figsize=(15, 6))
    plt.subplot(121)
    plt.plot(losses)
-    plt.plot(valid_losses)
+    plt.plot(dev_losses)
    plt.xlabel('Epoch')
    plt.ylabel('Average loss')
    plt.subplot(122)
-    plt.plot(valid_accuracies)
+    plt.plot(dev_accuracies)
    plt.xlabel('Epoch')
-    plt.ylabel('Validation set accuracy')
+    plt.ylabel('Development set accuracy')
    return model
 ```
 %% Cell type:markdown id: tags:
 We can now train on the GPU. (For this very simple model however, the training times are almost identical.)
 %% Cell type:code id: tags:
 ``` python
-model = train(vocab, train_x, train_y, validation_data=(dev_x, dev_y))
+model = train()
 ```
 %% Cell type:markdown id: tags:
 That’s all folks!