Add the Chocolate Box Challenge

challenge/challenge.ipynb

+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Feature engineering for part-of-speech tagging"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "In this challenge, you will practice your skills in feature engineering, the task of identifying useful features for a machine learning system."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## The data set"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "The data for this challenge and their representation is the same as for lab L4 on dependency parsing, but for this lab we have converted the data into a simpler format: words and their part-of-speech tags are separated by tabs, sentences are separated by empty lines. The code in the next cell defines a container class for data with this format."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "class Dataset():\n",
+    "\n",
+    "    def __init__(self, filename):\n",
+    "        self.filename = filename\n",
+    "\n",
+    "    def __iter__(self):\n",
+    "        tmp = []\n",
+    "        with open(self.filename, 'rt', encoding='utf-8') as lines:\n",
+    "            for line in lines:\n",
+    "                line = line.rstrip()\n",
+    "                if line:\n",
+    "                    tmp.append(tuple(line.split('\\t')))\n",
+    "                else:\n",
+    "                    yield tmp\n",
+    "                    tmp = []"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "We load the training data and the development data for this lab:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "train_data = Dataset('train.txt')\n",
+    "dev_data = Dataset('dev.txt')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Both data sets consist of **tagged sentences**. On the Python side of things, a tagged sentence is represented as a list of string pairs, where the first component of each pair represents a word token and the second component represents the word’s tag. The possible tags are listed and exemplified in the [Annotation Guidelines](http://universaldependencies.org/u/pos/all.html) of the Universal Dependencies Project."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Baseline tagger"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "The baseline tagger that you will use in this lab is a pure Python implementation of the perceptron tagger that was presented in the video lectures on [Part-of-speech tagging with the perceptron](https://web.microsoftstream.com/video/1a846d64-57e9-41ba-a3ca-3ab74cf32039) and [The perceptron learning algorithm](https://web.microsoftstream.com/video/1a69c6d6-35e1-42dc-81ba-8b5e52208831). To understand what the code provided here does, and how it might be extended with new features, you should watch these two lectures."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Linear model"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from collections import defaultdict\n",
+    "\n",
+    "class Linear(object):\n",
+    "\n",
+    "    def __init__(self, classes):\n",
+    "        self.classes = sorted(classes)\n",
+    "        self.weight = {c: defaultdict(float) for c in self.classes}\n",
+    "        self.bias = {c: 0.0 for c in self.classes}\n",
+    "\n",
+    "    def forward(self, features):\n",
+    "        scores = {}\n",
+    "        for c in self.classes:\n",
+    "            scores[c] = self.bias[c]\n",
+    "            for f, v in features.items():\n",
+    "                scores[c] += v * self.weight[c][f]\n",
+    "        return scores"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Perceptron learning algorithm"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "class PerceptronTrainer(object):\n",
+    "\n",
+    "    def __init__(self, model):\n",
+    "        self.model = model\n",
+    "        self._acc = Linear(model.classes)\n",
+    "        self._counter = 1\n",
+    "\n",
+    "    def update(self, features, gold):\n",
+    "        scores = self.model.forward(features)\n",
+    "        pred = max(self.model.classes, key=lambda c: scores[c])\n",
+    "        if pred != gold:\n",
+    "            self.model.bias[gold] += 1\n",
+    "            self.model.bias[pred] -= 1\n",
+    "            self._acc.bias[gold] += self._counter\n",
+    "            self._acc.bias[pred] -= self._counter\n",
+    "            for f, v in features.items():\n",
+    "                self.model.weight[gold][f] += v\n",
+    "                self.model.weight[pred][f] -= v\n",
+    "                self._acc.weight[gold][f] += v * self._counter\n",
+    "                self._acc.weight[pred][f] -= v * self._counter\n",
+    "        self._counter += 1\n",
+    "\n",
+    "    def finalize(self):\n",
+    "        for c in self.model.classes:\n",
+    "            delta_b = self._acc.bias[c] / self._counter\n",
+    "            self.model.bias[c] -= delta_b\n",
+    "            for feat in self.model.weight[c]:\n",
+    "                delta_w = self._acc.weight[c][feat] / self._counter\n",
+    "                self.model.weight[c][feat] -= delta_w"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Perceptron tagger"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "This is the part of the code that you will have to modify."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "class PerceptronTagger(object):\n",
+    "\n",
+    "    def __init__(self, tags):\n",
+    "        self.model = Linear(tags)\n",
+    "\n",
+    "    def featurize(self, words, i, pred_tags):\n",
+    "        # TODO: This is the only method that you are allowed to change!\n",
+    "        feats = []\n",
+    "        feats.append(words[i])\n",
+    "        feats.append(words[i-1] if i > 0 else '<bos>')\n",
+    "        feats.append(words[i+1] if i + 1 < len(words) else '<eos>')\n",
+    "        feats.append(pred_tags[i-1] if i > 0 else '<bos>')\n",
+    "        return {(i, f): 1 for i, f in enumerate(feats)}\n",
+    "\n",
+    "    def predict(self, words):\n",
+    "        pred_tags = []\n",
+    "        for i, _ in enumerate(words):\n",
+    "            features = self.featurize(words, i, pred_tags)\n",
+    "            scores = self.model.forward(features)\n",
+    "            pred_tag = max(self.model.classes, key=lambda c: scores[c])\n",
+    "            pred_tags.append(pred_tag)\n",
+    "        return pred_tags"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Training loop"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from tqdm import tqdm\n",
+    "\n",
+    "def train_perceptron(train_data, n_epochs=1):\n",
+    "    # Collect the tags in the training data\n",
+    "    tags = set()\n",
+    "    for tagged_sentence in train_data:\n",
+    "        words, gold_tags = zip(*tagged_sentence)\n",
+    "        tags.update(gold_tags)\n",
+    "\n",
+    "    # Initialise and train the perceptron tagger\n",
+    "    tagger = PerceptronTagger(tags)\n",
+    "    trainer = PerceptronTrainer(tagger.model)\n",
+    "    for epoch in range(n_epochs):\n",
+    "        with tqdm(total=sum(1 for s in train_data)) as pbar:\n",
+    "            for tagged_sentence in train_data:\n",
+    "                words, gold_tags = zip(*tagged_sentence)\n",
+    "                pred_tags = []\n",
+    "                for i, gold_tag in enumerate(gold_tags):\n",
+    "                    features = tagger.featurize(words, i, pred_tags)\n",
+    "                    trainer.update(features, gold_tag)\n",
+    "                    pred_tags.append(gold_tag)\n",
+    "                pbar.update()\n",
+    "    trainer.finalize()\n",
+    "\n",
+    "    return tagger"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Evaluation"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "The following function that computes the accuracy of the tagger on gold-standard data."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Feature engineering"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Your task now is to try to improve the performance of the perceptron tagger by adding new features. The only part of the code that you are allowed to change is the `featurize` method. Provide a short (ca. 150&nbsp;words) report on what features you added and what results you obtained."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "**⚠️ To claim the bonus points for this challenge, your submitted notebook must contain output demonstrating at least 91% accuracy on the development set.**"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "tagger = train_perceptron(train_data, n_epochs=3)\n",
+    "print('{:.4f}'.format(accuracy(tagger, dev_data)))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "*TODO: Insert your report here*"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Chocolate Box Challenge"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "To participate in the [Chocolate Box Challenge](https://www.kaggle.com/t/ce6f010bfb99478fb20830735c6f4a03), run the next code cell to produce a file `submission.csv` and upload this file to Kaggle."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Load the test data (without the tags)\n",
+    "test_data = Dataset('test-notags.txt')\n",
+    "\n",
+    "# Generate submission.csv with results on both the dev data and the test data\n",
+    "with open('submission.csv', 'w') as target:\n",
+    "    target.write('Id,Tag\\n')\n",
+    "    for p, data in [('D', dev_data), ('T', test_data)]:\n",
+    "        for i, tagged_sentence in enumerate(data):\n",
+    "            words, _ = zip(*tagged_sentence)\n",
+    "            predicted_tags = tagger.predict(words)\n",
+    "            for j, tag in enumerate(predicted_tags):\n",
+    "                target.write('{}-{:04d}-{:04d},{}\\n'.format(p, i, j, tag))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Please observe the following rules for the Chocolate Box Challenge:\n",
+    "\n",
+    "> The point of the challenge is to come up with interesting features. You are not allowed to change the tagger in any other way.\n",
+    "\n",
+    "Good luck, and may the best team win! 🙂"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.10"
+  },
+  "latex_envs": {
+   "bibliofile": "biblio.bib",
+   "cite_by": "apalike",
+   "current_citInitial": 1,
+   "eqLabelWithNumbers": true,
+   "eqNumInitial": 0
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 1
+}