{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*This notebook is from https://github.com/neubig/anlp-code* by Graham Neubig"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "# Build a Rule-based Sentiment Classifier\n\nThis is a notebook for [CMU CS11-711 Advanced NLP](https://cmu-l3.github.io/anlp-fall2025/), in which you can attempt to build a rule-based sentiment classifier. It will take in a text `X` and return a `label` of \"1\" if the sentiment of the text is positive, \"-1\" if the sentiment of the text is negative, and \"0\" if the sentiment of the text is neutral. You can test the accuracy of your classifier on the [Stanford Sentiment Treebank](http://nlp.stanford.edu/sentiment/index.html) by running the notebook all the way to end.\n\nThe only thing that you should change in this notebook is the following cell which contains two important elements. The first is `extract_features(X)`, which will extract a dictionary of (named) feature values from the text. You should create this by hand, and a simple example is shown for you. The second is `feature_weights`, a dictionary which will assign a weight to each extracted feature.\n\nThe final way the classifier decides whether to assign a positive, negative, or neutral label is by calculating the dot product `feature_weights * extract_features(X)`, and if the value is greater than zero, return 1, less than zero return -1, and if exactly zero return 0.\n\nLet's have some fun trying to design a classifier 😁"
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "def extract_features(x: str) -> dict[str, float]:\n",
    "    features = {}\n",
    "    x_split = x.split(' ')\n",
    "    \n",
    "    # Count the number of \"good words\" and \"bad words\" in the text\n",
    "    good_words = ['love', 'good', 'nice', 'great', 'enjoy', 'enjoyed']\n",
    "    bad_words = ['hate', 'bad', 'terrible', 'disappointing', 'sad', 'lost', 'angry']\n",
    "    for x_word in x_split:\n",
    "        if x_word in good_words:\n",
    "            features['good_word_count'] = features.get('good_word_count', 0) + 1\n",
    "        if x_word in bad_words:\n",
    "            features['bad_word_count'] = features.get('bad_word_count', 0) + 1\n",
    "    \n",
    "    # The \"bias\" value is always one, to allow us to assign a \"default\" score to the text\n",
    "    features['bias'] = 1\n",
    "    \n",
    "    return features\n",
    "\n",
    "feature_weights = {'good_word_count': 1.0, 'bad_word_count': -1.0, 'bias': 0.5}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Reading\n",
    "\n",
    "Read in the data from the training and dev (or finally test) sets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "def read_xy_data(filename: str) -> tuple[list[str], list[int]]:\n",
    "    x_data = []\n",
    "    y_data = []\n",
    "    with open(filename, 'r') as f:\n",
    "        for line in f:\n",
    "            label, text = line.strip().split(' ||| ')\n",
    "            x_data.append(text)\n",
    "            y_data.append(int(label))\n",
    "    return x_data, y_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_train, y_train = read_xy_data('./data/train.txt')\n",
    "x_test, y_test = read_xy_data('./data/dev.txt')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The Rock is destined to be the 21st Century 's new `` Conan '' and that he 's going to make a splash even greater than Arnold Schwarzenegger , Jean-Claud Van Damme or Steven Segal .\n",
      "1\n"
     ]
    }
   ],
   "source": [
    "print(x_train[0])\n",
    "print(y_train[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Run the Classifier and Calculate Accuracy\n",
    "\n",
    "Run the classifier over the training and dev (test) sets and calculate accuracy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "def run_classifier(x: str) -> int:\n",
    "    score = 0\n",
    "    for feat_name, feat_value in extract_features(x).items():\n",
    "        score = score + feat_value * feature_weights.get(feat_name, 0)\n",
    "    if score > 0:\n",
    "        return 1\n",
    "    elif score < 0:\n",
    "        return -1\n",
    "    else:\n",
    "        return 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "def calculate_accuracy(x_data: list[str], y_data: list[int]) -> float:\n",
    "    total_number = 0\n",
    "    correct_number = 0\n",
    "    for x, y in zip(x_data, y_data):\n",
    "        y_pred = run_classifier(x)\n",
    "        total_number += 1\n",
    "        if y == y_pred:\n",
    "            correct_number += 1\n",
    "    return correct_number / float(total_number)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{1: 444, 0: 229, -1: 428}\n"
     ]
    }
   ],
   "source": [
    "label_count = {}\n",
    "for y in y_test:\n",
    "    if y not in label_count:\n",
    "        label_count[y] = 0\n",
    "    label_count[y] += 1\n",
    "print(label_count)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train accuracy: 0.4345739700374532\n",
      "Dev/test accuracy: 0.4214350590372389\n"
     ]
    }
   ],
   "source": [
    "train_accuracy = calculate_accuracy(x_train, y_train)\n",
    "test_accuracy = calculate_accuracy(x_test, y_test)\n",
    "print(f'Train accuracy: {train_accuracy}')\n",
    "print(f'Dev/test accuracy: {test_accuracy}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Error Analysis\n",
    "\n",
    "An important part of improving any system is figuring out where it goes wrong. The following two functions allow you to randomly observe some mistaken examples, which may help you improve the classifier. Feel free to write more sophisticated methods for error analysis as well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "import random\n",
    "def find_errors(x_data, y_data):\n",
    "    error_ids = []\n",
    "    y_preds = []\n",
    "    for i, (x, y) in enumerate(zip(x_data, y_data)):\n",
    "        y_preds.append(run_classifier(x))\n",
    "        if y != y_preds[-1]:\n",
    "            error_ids.append(i)\n",
    "    for _ in range(5):\n",
    "        my_id = random.choice(error_ids)\n",
    "        x, y, y_pred = x_data[my_id], y_data[my_id], y_preds[my_id]\n",
    "        print(f'{x}\\ntrue label: {y}\\npredicted label: {y_pred}\\n')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The thing looks like a made-for-home-video quickie .\n",
      "true label: 0\n",
      "predicted label: 1\n",
      "\n",
      "Less front-loaded and more shapely than the two-hour version released here in 1990 .\n",
      "true label: 0\n",
      "predicted label: 1\n",
      "\n",
      "A smug and convoluted action-comedy that does n't allow an earnest moment to pass without reminding audiences that it 's only a movie .\n",
      "true label: -1\n",
      "predicted label: 1\n",
      "\n",
      "While it 's all quite tasteful to look at , the attention process tends to do a little fleeing of its own .\n",
      "true label: -1\n",
      "predicted label: 1\n",
      "\n",
      "entertaining enough , but nothing new\n",
      "true label: -1\n",
      "predicted label: 1\n",
      "\n"
     ]
    }
   ],
   "source": [
    "find_errors(x_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "prototype",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}