Skip to content
Snippets Groups Projects
Commit c9174583 authored by Marco Kuhlmann's avatar Marco Kuhlmann
Browse files

Remove explicit <bos> padding; update performance targets

parent b3683bff
No related branches found
No related tags found
No related merge requests found
Hide whitespace changes
Inline Side-by-side
labs/l1-basic/nlp-l1-basic.ipynb
+ 12
− 12
View file @ c9174583
%% Cell type:markdown id: tags:
# L1: Language modelling
%% Cell type:markdown id: tags:
In this lab you will implement and train two neural language models: the fixed-window model and the recurrent neural network model. You will evaluate these models by computing their perplexity on a benchmark dataset.
%% Cell type:code id: tags:
``` python
import torch
```
%% Cell type:markdown id: tags:
For this lab, you should use the GPU if you have one:
%% Cell type:code id: tags:
``` python
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
```
%% Cell type:markdown id: tags:
## Data
%% Cell type:markdown id: tags:
The data for this lab is [WikiText](https://blog.einstein.ai/the-wikitext-long-term-dependency-language-modeling-dataset/), a collection of more than 100 million tokens extracted from the “Good” and “Featured” articles on Wikipedia. We will use the small version of the dataset, which contains slightly more than 2.5 million tokens.
The next cell contains code for an object that will act as a container for the “training” and the “validation” section of the data. We fill this container by reading the corresponding text files. The only processing we do is to whitespace-tokenise, and to enclose each non-empty line within `<bos>` (beginning-of-sentence) and `<eos>` (end-of-sentence) tokens.
The next cell contains code for an object that will act as a container for the “training” and the “validation” section of the data. We fill this container by reading the corresponding text files. The only processing we do is to whitespace-tokenise and to replace each newline with an end-of-sentence token.
%% Cell type:code id: tags:
``` python
class WikiText(object):
def __init__(self):
self.vocab = {}
self.train = self.read_data('wiki.train.tokens')
self.valid = self.read_data('wiki.valid.tokens')
def read_data(self, path):
ids = []
with open(path, encoding='utf-8') as source:
for line in source:
line = line.rstrip()
if line:
for token in ['<bos>'] + line.split() + ['<eos>']:
for token in line.split() + ['<eos>']:
if token not in self.vocab:
self.vocab[token] = len(self.vocab)
ids.append(self.vocab[token])
return ids
```
%% Cell type:markdown id: tags:
The cell below loads the data and prints the total number of tokens and the size of the vocabulary.
%% Cell type:code id: tags:
``` python
wikitext = WikiText()
print('Tokens in train:', len(wikitext.train))
print('Tokens in valid:', len(wikitext.valid))
print('Vocabulary size:', len(wikitext.vocab))
```
%% Cell type:markdown id: tags:
## Problem 1: Fixed-window model
%% Cell type:markdown id: tags:
In this section, you will implement and train the fixed-window neural language model proposed by [Bengio et al. (2003)](http://www.jmlr.org/papers/volume3/bengio03a/bengio03a.pdf) and presented in the lectures. Recall that an input to the network takes the form of a vector of $n-1$ integers representing the preceding words. Each integer is mapped to a vector via an embedding layer. (All positions share the same embedding.) The embedding vectors are then concatenated and sent through a two-layer feed-forward network with a non-linearity in the form of a rectified linear unit (ReLU) and a final softmax layer.
%% Cell type:markdown id: tags:
### Problem 1.1: Vectorise the data (1&nbsp;point)
### Problem 1.1: Vectorise the data (2&nbsp;point)
%% Cell type:markdown id: tags:
Your first task is to write code for transforming the data in the WikiText container into a vectorised form that can be fed to the fixed-window model. Concretely, you will implement a [collate function](https://pytorch.org/docs/stable/data.html#dataloader-collate-fn) in the form of a callable vectoriser object. Complete the skeleton code in the cell below:
%% Cell type:code id: tags:
``` python
class FixedWindowVectorizer(object):
def __init__(self, n):
# n-gram order
self.n = n
def __call__(self, data):
# TODO: Replace the following line with your own code
return None, None
```
%% Cell type:markdown id: tags:
Your code should implement the following specification:
**__init__** (*self*, *n*)
> Creates a new vectoriser with n-gram order $n$. Your code should be able to handle arbitrary n-gram orders $n \geq 1$.
**__call__** (*self*, *data*)
> Transforms WikiText *data* (a list of word ids) into a pair of tensors $\mathbf{X}$, $\mathbf{y}$ that can be used to train the fixed-window model. Let $N$ be the total number of $n$-grams from the token list; then $\mathbf{X}$ is a matrix with shape $(N, n-1)$ and $\mathbf{y}$ is a vector with length $N$.
%% Cell type:markdown id: tags:
#### 🤞 Test your code
Test your implementation by running the code in the next cell. Does the output match your expectation?
%% Cell type:code id: tags:
``` python
valid_x, valid_y = FixedWindowVectorizer(3)(wikitext.valid)
print(valid_x.size(), valid_y.size())
```
%% Cell type:markdown id: tags:
### Problem 1.2: Implement the model (2&nbsp;points)
%% Cell type:markdown id: tags:
Your next task is to implement the fixed-window model based on the graphical specification given in the lecture.
%% Cell type:code id: tags:
``` python
import torch.nn as nn
class FixedWindowModel(nn.Module):
def __init__(self, n, n_words, embedding_dim=50, hidden_dim=50):
super().__init__()
# TODO: Add your own code
def forward(self, x):
# TODO: Replace the next line with your own code
raise NotImplemented
```
%% Cell type:markdown id: tags:
Here is the specification of the two methods:
**__init__** (*self*, *n*, *n_words*, *embedding_dim*=50, *hidden_dim*=50)
> Creates a new fixed-window neural language model. The argument *n* specifies the model&rsquo;s $n$-gram order. The argument *n_words* is the number of words in the vocabulary. The arguments *embedding_dim* and *hidden_dim* specify the dimensionalities of the embedding layer and the hidden layer of the feedforward network, respectively; their default value is 50.
**forward** (*self*, *x*)
> Computes the network output on an input batch *x*. The shape of *x* is $(B, n-1)$, where $B$ is the batch size. The output of the forward pass is a tensor of shape $(B, V)$ where $V$ is the number of words in the vocabulary.
**Hint:** The most efficient way to implement the vector concatenation in this model is to use the [`view()`](https://pytorch.org/docs/stable/tensors.html#torch.Tensor.view) method.
#### 🤞 Test your code
Test your code by instantiating the model and feeding it a batch of examples from the training data.
%% Cell type:markdown id: tags:
### Problem 1.3: Train the model (3&nbsp;points)
### Problem 1.3: Train the model (4&nbsp;points)
%% Cell type:markdown id: tags:
Your final task is to write code to train the fixed-window model using minibatch gradient descent and the cross-entropy loss function. This should be a straightforward generalisation of the training loops that you have seen so far. Complete the skeleton code in the cell below:
%% Cell type:code id: tags:
``` python
def train_fixed_window(n, n_epochs=1, batch_size=3200, lr=1e-2):
# TODO: Replace the following line with your own code
return None
```
%% Cell type:markdown id: tags:
Here is the specification of the training function:
**train_fixed_window** (*n*, *n_epochs* = 1, *batch_size* = 3200, *lr* = 0.01)
> Trains a fixed-window neural language model of order *n* using minibatch gradient descent and returns it. The parameters *n_epochs* and *batch_size* specify the number of training epochs and the minibatch size, respectively. Training uses the cross-entropy loss function and the [Adam optimizer](https://pytorch.org/docs/stable/optim.html#torch.optim.Adam) with learning rate *lr*. After each epoch, prints the perplexity of the model on the validation data.
%% Cell type:markdown id: tags:
The code in the cell below trains a bigram model.
The code in the cell below trains a trigram model.
%% Cell type:code id: tags:
``` python
model_fixed_window = train_fixed_window(2)
model_fixed_window = train_fixed_window(3)
```
%% Cell type:markdown id: tags:
#### Performance goal
**Your submitted notebook must contain output demonstrating a validation perplexity of at most 350.** If you do not reach this perplexity after the first epoch, try training for a second epoch.
**Your submitted notebook must contain output demonstrating a validation perplexity of at most 360.** If you do not reach this perplexity after the first epoch, try training for a second epoch.
⚠️ Computing the validation perplexity in one go (for the full validation set) will probably exhaust your computer’s memory and/or take a lot of time. If you run into this problem, do the computation at the minibatch level and aggregate the results.
%% Cell type:markdown id: tags:
#### 🤞 Test your code
To see whether your network is learning something, print or plot the loss and/or the perplexity on the training data. If the two values do not decrease during training, try to find the problem before wasting time (and electricity) on useless computation.
Training and even evaluation will take some time – on a CPU, you should expect several minutes per epoch, depending on hardware. Our reference implementation uses a GPU and runs in less than 30 seconds per epoch on [Colab](http://colab.research.google.com).
%% Cell type:markdown id: tags:
## Problem 2: Recurrent neural network model
%% Cell type:markdown id: tags:
In this section, you will implement the recurrent neural network language model. Recall that an input to this model is a vector of word ids. Each integer is mapped to an embedding vector. The sequence of embedded vectors is then fed into an unrolled LSTM. At each position $i$ in the sequence, the hidden state of the LSTM at that position is sent through a linear transformation into a final softmax layer representing the probability distribution over the words at position $i+1$. In theory, the input vector could represent the complete training data; for practical reasons, however, we will truncate the input to some fixed value *bptt_len*. This length is called the **backpropagation-through-time horizon**.
%% Cell type:markdown id: tags:
### Problem 2.1: Vectorise the data (1&nbsp;point)
### Problem 2.1: Vectorise the data (2&nbsp;points)
%% Cell type:markdown id: tags:
As in the previous problem, your first task is to transform the data in the WikiText container into a vectorised form that can be fed to the model.
%% Cell type:code id: tags:
``` python
class RNNVectorizer(object):
def __init__(self, bptt_len):
# backpropagation-through-time horizon
self.bptt_len = bptt_len
def __call__(self, data):
# TODO: Replace the following line with your own code
return None, None
```
%% Cell type:markdown id: tags:
Your vectoriser should meet the following specification:
**__init__** (*self*, *bptt_len*)
> Creates a new vectoriser. The parameter *bptt_len* specifies the backpropagation-through-time horizon.
**__call__** (*self*, *data*)
> Transforms a list of token indexes *data* into a pair of tensors $\mathbf{X}$, $\mathbf{Y}$ that can be used to train the recurrent neural language model. The rows of both tensors represent contiguous subsequences of token indexes of length *bptt_len*. Compared to the sequences in $\mathbf{X}$, the corresponding sequences in $\mathbf{Y}$ are shifted one position to the right. More precisely, if the $i$th row of $\mathbf{X}$ is the sequence that starts at token position $j$, then the same row of $\mathbf{Y}$ is the sequence that starts at position $j+1$.
%% Cell type:markdown id: tags:
#### 🤞 Test your code
Test your implementation by running the following code:
%% Cell type:code id: tags:
``` python
valid_x, valid_y = RNNVectorizer(32)(wikitext.valid)
print(valid_x.size(), valid_y.size())
```
%% Cell type:markdown id: tags:
### Problem 2.2: Implement the model (2&nbsp;points)
%% Cell type:markdown id: tags:
Your next task is to implement the recurrent neural network model based on the graphical specification.
%% Cell type:code id: tags:
``` python
import torch.nn as nn
class RNNModel(nn.Module):
def __init__(self, n_words, embedding_dim=50, hidden_dim=50):
super().__init__()
# TODO: Add your own code
def forward(self, x):
# TODO: Replace the next line with your own code
raise NotImplemented
```
%% Cell type:markdown id: tags:
Your implementation should follow this specification:
**__init__** (*self*, *n_words*, *embedding_dim* = 50, *hidden_dim* = 50)
> Creates a new recurrent neural network language model based on an LSTM. The argument *n_words* is the number of words in the vocabulary. The arguments *embedding_dim* and *hidden_dim* specify the dimensionalities of the embedding layer and the LSTM hidden layer, respectively; their default value is 50.
**forward** (*self*, *x*)
> Computes the network output on an input batch *x*. The shape of *x* is $(B, H)$, where $B$ is the batch size and $H$ is the length of each input sequence. The shape of the output tensor is $(B, H, V)$, where $V$ is the size of the vocabulary.
%% Cell type:markdown id: tags:
#### 🤞 Test your code
Test your code by instantiating the model and feeding it a batch of examples from the training data.
%% Cell type:markdown id: tags:
### Problem 2.3: Train the model (3&nbsp;points)
### Problem 2.3: Train the model (4&nbsp;points)
%% Cell type:markdown id: tags:
The training loop for the recurrent neural network model is essentially identical to the loop that you wrote for the feed-forward model. The only thing to note is that the cross-entropy loss function expects its input to be a two-dimensional tensor; you will therefore have to re-shape the output tensor from the LSTM as well as the gold-standard output tensor in a suitable way. The most efficient way to do so is to use the [`view()`](https://pytorch.org/docs/stable/tensors.html#torch.Tensor.view) method.
%% Cell type:code id: tags:
``` python
def train_rnn(n_epochs=1, batch_size=1000, bptt_len=32, lr=1e-2):
# TODO: Replace the next line with your own code
return None
```
%% Cell type:markdown id: tags:
Here is the specification of the training function:
**train_rnn** (*n_epochs* = 1, *batch_size* = 1000, *bptt_len* = 32, *lr* = 0.01)
> Trains a recurrent neural network language model on the WikiText data using minibatch gradient descent and returns it. The parameters *n_epochs* and *batch_size* specify the number of training epochs and the minibatch size, respectively. The parameter *bptt_len* specifies the length of the backpropagation-through-time horizon, that is, the length of the input and output sequences. Training uses the cross-entropy loss function and the [Adam optimizer](https://pytorch.org/docs/stable/optim.html#torch.optim.Adam) with learning rate *lr*. After each epoch, prints the perplexity of the model on the validation data.
%% Cell type:markdown id: tags:
Evaluate your model by running the following code cell:
%% Cell type:code id: tags:
``` python
model_rnn = train_rnn()
```
%% Cell type:markdown id: tags:
#### Performance goal
**Your submitted notebook must contain output demonstrating a validation perplexity of at most 280.** If you do not reach this perplexity after the first epoch, try training for a second epoch.
**Your submitted notebook must contain output demonstrating a validation perplexity of at most 300.** If you do not reach this perplexity after the first epoch, try training for a second epoch.
%% Cell type:markdown id: tags:
## Problem 3: Parameter initialisation (3&nbsp;points)
## Problem 3: Parameter initialisation (6&nbsp;points)
%% Cell type:markdown id: tags:
The error surfaces explored when training neural networks can be very complex. Because of this, it is important to choose “good” initial values for the parameters. In PyTorch, the weights of the embedding layer are initialised by sampling from the standard normal distribution $\mathcal{N}(0, 1)$. Test how changing the initialisation affects the perplexity of your feed-forward language model. Find research articles that propose different initialisation strategies.
Write a short (150&nbsp;words) report about your experiments and literature search. Use the following prompts:
* What different initialisation did you try? What results did you get?
* How do your results compare to what was suggested by the research articles?
* What did you learn? How, exactly, did you learn it? Why does this learning matter?
You are allowed to consult sources for this problem if you appropriately cite them. If in doubt, please read the [Academic Integrity Policy](https://www.ida.liu.se/~TDDE09/logistics/policies.html#academic-integrity-policy).
%% Cell type:markdown id: tags:
*TODO: Enter your text here*
......
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment