{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Lecture 6: Pretraining\n", "\n", "Lecture 6 | CMU ANLP Fall 2025 | Instructor: Sean Welleck\n", "\n", "\n", "This is a notebook for [CMU CS11-711 Advanced NLP](https://cmu-l3.github.io/anlp-fall2025/) that generates text with a pretrained model, and shows an example from a pretraining dataset." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Install [HuggingFace Transformers](https://huggingface.co/docs/transformers/en/index)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "!pip install transformers --upgrade" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Load tokenizer and model\n", "\n", "- Find models at https://huggingface.co/" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from transformers import AutoTokenizer, AutoModelForCausalLM\n", "\n", "model = \"HuggingFaceTB/SmolLM2-360M\"\n", "\n", "tokenizer = AutoTokenizer.from_pretrained(model)\n", "model = AutoModelForCausalLM.from_pretrained(model)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Generate" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "The meaning of life is to be happy.\n", "\n", "The meaning of life is to be happy.\n", "\n", "The meaning of\n", "====\n", "142 + 29 = 161\n", "\n", "(ii) 123 + 29 = 1\n", "====\n", "On February 12th 2025, the temperature in Pittsburgh was 100 degrees Fahrenheit.\n", "\n", "The temperature in Pittsburgh was 100 degrees Fahrenheit\n", "====\n" ] } ], "source": [ "prompts = [\n", " \"The meaning of life is\",\n", " \"142 + 29 =\",\n", " \"On February 12th 2025, the temperature in Pittsburgh was\"\n", "]\n", "\n", "for prompt in prompts:\n", " inputs = tokenizer(prompt, return_tensors=\"pt\")\n", " outputs = model.generate(\n", " **inputs, pad_token_id=tokenizer.eos_token_id\n", " )\n", " print(tokenizer.decode(outputs[0]))\n", " print(\"====\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Inspect DOLMA data\n", "\n", "- Paper: https://arxiv.org/abs/2402.00159\n", "- Data: https://huggingface.co/datasets/allenai/dolma\n" ] }, { "cell_type": "code", "execution_count": 26, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "--2025-01-30 12:59:01-- https://olmo-data.org/dolma-v1_5-1913B-sample/cc_en_head/cc_en_head-0000.json.gz\n", "Resolving olmo-data.org (olmo-data.org)... 172.67.175.213, 104.21.83.123\n", "Connecting to olmo-data.org (olmo-data.org)|172.67.175.213|:443... connected.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...\n", "To disable this warning, you can either:\n", "\t- Avoid using `tokenizers` before the fork if possible\n", "\t- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "HTTP request sent, awaiting response... 200 OK\n", "Length: 1947344553 (1.8G) [application/json]\n", "Saving to: ‘cc_en_head-0000.json.gz’\n", "\n", "cc_en_head-0000.jso 100%[===================>] 1.81G 31.7MB/s in 59s \n", "\n", "2025-01-30 13:00:01 (31.6 MB/s) - ‘cc_en_head-0000.json.gz’ saved [1947344553/1947344553]\n", "\n" ] } ], "source": [ "# From https://huggingface.co/datasets/allenai/dolma/blob/main/urls/v1_5-sample.txt\n", "!wget https://olmo-data.org/dolma-v1_5-1913B-sample/cc_en_head/cc_en_head-0000.json.gz\n" ] }, { "cell_type": "code", "execution_count": 30, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...\n", "To disable this warning, you can either:\n", "\t- Avoid using `tokenizers` before the fork if possible\n", "\t- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)\n" ] } ], "source": [ "!head -n 1 cc_en_head-0000.json > sample.json" ] }, { "cell_type": "code", "execution_count": 31, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "added 2023-04-11T09:57:03.044571+00:00\n", "attributes {'random_number_v1__random_number_v1__random': [[0, 9626, 0.11918]]}\n", "created 2020-01-17T12:48:23Z\n", "id http://250news.theexplorationplace.com/www.250news.com/65595.html\n", "metadata {'bucket': 'head', 'cc_segment': 'crawl-data/CC-MAIN-2020-05/segments/1579250589560.16/wet/CC-MAIN-20200117123339-20200117151339-00000.warc.wet.gz', 'date_download': '2020-01-17T12:48:23Z', 'digest': 'sha1:K2RJ3HQ73U7W4LKKI2JVVGZ4ABQFRF6Q', 'language': 'en', 'language_score': 0.97, 'length': 13925, 'line_ids': [27, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 62, 63, 64, 65, 67, 68, 69, 70, 71, 72, 73, 75, 76, 77, 78, 79, 80, 81, 82, 84, 85, 86, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 102, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 116, 119, 120, 121, 122, 123, 124], 'nlines': 85, 'original_length': 15502, 'original_nlines': 130, 'perplexity': 321.5, 'source_domain': '250news.theexplorationplace.com', 'title': '250 News | RiverBend Project Shows PG is “Open for Business”', 'url': 'http://250news.theexplorationplace.com/www.250news.com/65595.html'}\n", "source common-crawl\n", "text Prince George, B.C.- Construction of the new RiverBend Seniors housing project in Prince George is well underway, but it’s a project that almost didn’t happen.\n", "The $33 million dollar project was first presented to Mayor and Council in 2013 “We didn’t get much traction at that time” says developer Frank Quinn of A&T Ventures. He says it wasn’t until Mayor Lyn Hall was elected that the wheels were set in motion.\n", "Hall and key members of the City Staff, arranged to meet with Quinn in Kamloops and laid out the possible benefits of developing such a project in P.G. Those benefits include: a ten year tax break, and just over $265 thousand dollars in development cost charges relief plus an affordable housing grant of $855 thousand.\n", "“This project comes at the perfect time for us” says Gwen Norheim. She and her husband are among those who have already purchased a unit and expect to move to their new condo in the fall of 2017. “We were at the point of knowing we would need to move, but didn’t want to move into any of the existing facilities in Prince George.” Norheim was a teacher at the Connaught Junior High School which was on the neighbouring property until it burned to the ground. “In all my years of teaching I never imagined I would be living in its backyard.” She is also pleased their little dog, a small poodle terrier mix, will be able to live with them.\n", "Quinn says they did make an interesting discovery when they started construction, “The creek ran right through the property. Even though we had done our geo-tech work, we had no idea until we got into the ground how much of a challenge that was going to be for an excavation.” But he says their past experience in dealing with alpine construction and avalanche trenches gave them the know-how to deal with it.\n", "That’s it, big smiles Shirley and Mike… there is an election coming.\n", "This is an excellent and well needed project!\n", "If the NDP was in power (god forbid) and it was NDP MLA’s in the picture, you would be the first to be all over it, praising them and the NDP to no end!!\n", "Go ahead and deny it if you want, but we know better!\n", "What we do know with the liberals is they are always raising fees and medical costs for retired seniors..and also charging them more every year for staying in senior housing.. All the seniors really appreciate being driven into poverty after paying taxes all their lives.. Thanks liberals.\n", "grow up galt.\n", "Okay, so we get rid of all our debt. Let us say that the last little bit will be taken care of in 20 years from now. As time progresses, there will be progressively more money available to spend on parks, to spend on no tax increase for a few years until inflation starts catching up, and so on.\n", "We will then buy new buildings, additions, infrastructure, enhancements of one sort or other, rebuilding old stock, etc. at 2036 dollars which will have 20 years inflation on them. EXCEPT, we cannot do that until we have saved for these new buildings and other infrastructure so we can see such development 40 years from now.\n", "Great planning! Good understanding of how the operation of a City works and how financing city development works.\n", "Green parks???? forget abut it. Learn to design them so that they are as natural as possible.\n", "Have a walk sometimes on the cover of the First Avenue landfill. That is the way to design low maintenance “parks”. Forget about Rainbow, Connaught Hill, Fort George/Lheitli Tenneh, etc. Just like they have forgotten about Cottonwood Island Park and the Heritage Riverfront Trail system.\n", "Yess the owners of the condos can save money too if they tap into the creek that runs underneath.\n", "I see we really do not give a chit about housing for seniors, do we???\n", "Seems like a great project to me . I could be wrong but what is wrong with keeping more seniors in Prince George?\n", "Nothing, just the usual curmudgeons that believe every project is a personal assault on their lives, all change is bad and anytime you even see a politician it must be a self serving photo-op.\n", "What is great about a project that only got built due to all the corporate welfare that it received? Cons just love to go on and on about the wonders of Capitalism over Socialism, but this thing only got built due to corporate socialism. The mayor and council of this city should put a sign that reads “For Sale…cheap” outside of council Chambers.\n", "The income tax being paid by construction workers on this project will more than pay for any so called corporate welfare.\n", "Remember, the City does not collect any income tax. Now it will not collect any property tax for 10 years either.\n", "This is the slow incursion of the City getting into the housing business, another download from the provincial government. Anything BC Housing has been getting into in the recent past reeks of Provincial Government download of responsibility without the necessary transfer of funds to the City. This is simply a hidden Triple P project.\n", "18 of the market units have already been sold! Seems that demand for these units is already there!\n", "I for one have parents who are in their very golden years. A facility like this, especially next door to Gateway, will be well utilized and much appreciated by those that will live there and by those who will be able to easily visit elderly family members who will choose to stay here instead of moving south!\n", "Sounds like a win for those that will live there, those who’s family will live there, those who will work there and those who will benefit from the economic activity generated from the retirement funds that will be spent here instead of in the Okanagan or other points down south!\n", "What is so special about being close to Gateway? Are they in the market for cars, mufflers, tires, motel room rentals? Do they need to be close to a new Firehall?\n", "Perhaps they eat a lot of Big Macs or Popa and Moma Burgers or 711 wedges.\n", "The Gateway might be a clean-look route to drive through, but it is not exactly a shopping, restaurant or entertainment mecca.\n", "Come to think of it, which part of PG is, other than parts of downtown plus the major shopping centres.\n", "1? What about all the other projects that developers walk away from, because the city is truly not ‘open for business’. Change the archaic zoning bylaws. Get Ospika built. \n", "Someone might want to ask a few questions. As an example do you think that a 10 year tax break, along with $265,000.00 in development cost charges relief, plus the $855,000.00 affordable housing grant is a little excessive.\n", "Who will own the rental units when the project is completed. My understanding is that the Government will own 33 units and Oncore Seniors Society, received financing from the Government over a long period of time, and they will own the other 33 units. The contractor of course will sell the other 107 Units. One assumes at a hefty profit.\n", "My understanding is that these rental units will be available for low to moderate income seniors. Hmmmm. Where does one go to get on the waiting list.\n", "These are one bedroom units with a space for a micro wave (Not provided) and while some aspects of it are pretty good, in actual fact, we end up with putting all this money into a project and end up with 33 units. Hmmmm.\n", "Perhaps gopg2015 could do some of his research and come up with a dollar number for the cost per unit, and perhaps some inkling as to how one can access these rentals.\n", "The Government has similar projects in Squamish, Kamloops, Peachland, etc; so this is not exactly new.\n", "Palopu, your comment shows us the amount of homework and research you did to support your assertions… my thumbs up!\n", "There is insufficient information to come up with a cost per unit. The proper measure of such a development is the cost per square foot and then attribute that to units of varying sizes.\n", "There would also be a difference between the market housing and the subsidized housing portion.\n", "Finally, I would want to want some info on which tax dollars are attributable to what portion of the development.\n", "What I am really trying to get at is why should anyone in the public have to do such calculations? This is a mixed housing project, and the taxpayer is funding part of it.\n", "This City has not yet learned the notion transparency.\n", "In this case they and the province need to prepare a prospectus of the project, the amount of subsidies provided by the taxpayer to something which few us will see any direct benefits from, the benefits which will accrue to the general population, who will be the source of the majority of the funds, and which services we will not see for general application to our annual budget deliberations, and so forth.\n", "I think THAT is the message we need to keep hammering the City as well as the province with. We should not have to second guess such information or spend more than a minute on finding the information.\n", "This looks like its going to be a successfull project with obvious demand for the units. How come it took a company from Kamloops to make this happen? The developers in PG seem to be very risk adverse and lack the vision and capasity to grow this city to its potential.\n", "This is one of those statements that could actually be very good or very bad . . .\n", "In my estimation it is more often bad from the taxpayer’s point of view.\n", "There needs to be a reporting out to the citizens of PG of the actual cost to the City of the subsidy provided and the measurable value received in return stated both in quality and in monetary quantity..\n", "Like to get the names of the voters who voted for A. The Harper Conservatives B. The BC Liberals. Just get a good look.\n", "I voted for both. I’ll show you my name if you show me yours.\n" ] } ], "source": [ "import json\n", "with open(\"sample.json\") as f:\n", " data = json.load(f)\n", "\n", "for k, v in data.items():\n", " print(k, v)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "prototype", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.13" } }, "nbformat": 4, "nbformat_minor": 2 }