Every AI project starts with data — and the source’s whole workflow is one library, one cache directory and one seed. This lesson follows a dataset from the Hugging Face Hub into ~/.cache/huggingface/datasets/, through CSV, JSON, Parquet and Arrow, into a reproducible 80/10/10 split, then adds the model cache and the three ways to keep big files out of git.
load_dataset("stanfordnlp/imdb") finds the dataset on the Hugging Face Hub, downloads it, converts it to Apache Arrow and caches the working copy at ~/.cache/huggingface/datasets/. The first load pays the transfer once; every later load — tomorrow, in another project, offline — is a local read. Model files work the same way, cached under ~/.cache/huggingface/hub/.
first load: network · later loads: diskThe datasets library works in Apache Arrow — columnar, zero-copy, fastest to read. Parquet is the same layout on disk with compression: typically several times smaller than the same CSV and faster to read back, which is why it is the storage format for AI work. CSV and JSON stay for interchange — spreadsheets, APIs, humans. Convert with to_csv, to_json, to_parquet; read back with from_csv, from_json, from_parquet.
Parquet = storage · Arrow = memory · CSV/JSON = interchangeTrain learns (typically 80%), validation checks progress during training (10%), test is the final unbiased evaluation (10%). Hold out the val+test pool first, then split that pool into validation and test — and pass seed=42, because train_test_split shuffles before it cuts. The same seed gives the same three piles on every machine; the second test_size is a fraction of the pool, not of the original.
same seed · same split · every machineThe Hub is the warehouse, the datasets library is the courier, and ~/.cache/huggingface/ is your pantry: you buy once or browse on demand (download vs streaming), you shelve what you keep (Parquet), you read from the shelf (Arrow), and the seed is the receipt that says which pile every row went to.
By the end you will be able to load a dataset from the Hugging Face Hub and explain what the cache is doing, stream a dataset that does not fit on disk, choose between CSV, JSON, Parquet and Arrow with a reason, make a reproducible train/validation/test split and check its arithmetic, download model files with hf_hub_download and snapshot_download, keep big files out of git with .gitignore, Git LFS or DVC, and name the sizes of the datasets this course uses.
Every AI (artificial intelligence) project starts with data. You find it, download it, convert it, split it, version it — and the source says the quiet part out loud: doing that by hand every time is slow and error-prone. The fix is not discipline. It is one library and one workflow, drawn as a seven-node diagram.
The source’s diagram is the spine of this whole lesson, so read it as a route with seven stops: Hugging Face Hub → the datasets library → load / stream → local cache → format conversion → data splits → your training pipeline. Hugging Face (HF) is the company behind the Hub — the hosted collection where datasets and models live — and the datasets library is the Python client that speaks to it. Everything after the first stop happens on your machine, and most of it happens automatically.
Two of the seven stops are decisions rather than machinery. Load or stream decides whether a full copy lands on your disk before you touch a row — the answer matters the moment a dataset is bigger than your free space. And which format decides how fast everything after it runs: the same rows can cost you 84 MB as CSV or about a third of that as Parquet, and can take about half a second to read as text or a few hundredths of a second in a columnar format — the format comparator in chapter 04 runs the arithmetic. The remaining five stops are the library’s job.
| stage | what it is | what it looks like in practice |
|---|---|---|
| Hugging Face Hub | the hosted home of shared datasets and models | huggingface.co/datasets, addressed as stanfordnlp/imdb |
| datasets library | the client that downloads, caches, converts and streams | pip install datasets huggingface_hub |
| Load / Stream | download a working copy, or iterate rows on demand | load_dataset(...) vs load_dataset(..., streaming=True) |
| Local cache | the working copy that makes every later load instant | ~/.cache/huggingface/datasets/ |
| Format conversion | rewrite the data for the tool that needs it | to_csv, to_json, to_parquet; Arrow inside |
| Data splits | train / validation / test, reproducible with a seed | train_test_split(test_size=0.2, seed=42) |
| Your training pipeline | the consumer: batches, tensors, gradients | a DataLoader over the split you made |
One honest note before the walkthrough: none of this is the only way to get data into a program. Python can read a CSV with the standard library, pandas has its own readers, and PyTorch has a DataLoader. What makes the datasets library the source’s centerpiece is that it handles downloading, caching, format conversion and streaming out of the box — four separate chores behind one function call — and every one of those four is a place a homemade pipeline quietly breaks.
The source draws the workflow as one diagram: Hugging Face Hub → datasets → load/stream → local cache → format conversion → data splits → your training pipeline. Walk it stage by stage and watch where the 84 MB actually lives at each point.
Stage 1 of 7 · 6 stages left to the training loop
The hosted collection where datasets and models live, addressed by an owner/name path. Nothing is on your machine yet.
Nothing in this walkthrough is specific to IMDb — the same seven stages carry every dataset in the course, and each one is introduced by the stage it belongs to.
In the source's pipeline, which piece is responsible for downloading a dataset, caching it, converting between formats and streaming it — all four?
The source’s first two steps are a two-line install and a three-line script. Everything the diagram promised happens behind them: the Hub is contacted, the data is fetched, and a working copy lands in a cache directory that makes the second load — and the hundredth — a local read.
Start with the packages. Two of them: datasets for the data, huggingface_hub for the file-level downloads later. Install them into the environment you built in Phase 0, Lesson 06 — the library is exactly the kind of dependency that belongs in a project environment, not in the system Python.
pip install datasets huggingface_hub
# with uv, the course default:
# uv pip install datasets huggingface_hubThen load something. The source picks IMDB — 50,000 labelled movie reviews with binary sentiment labels (25,000 train + 25,000 test, plus 50,000 unlabelled ones) — because it is small enough to arrive in seconds and real enough to train on:
from datasets import load_dataset
dataset = load_dataset("stanfordnlp/imdb")
print(dataset)
print(dataset["train"][0])
# example output (illustrative — the review text is truncated here)
# DatasetDict({
# train: Dataset({
# features: ['text', 'label'],
# num_rows: 25000
# })
# test: Dataset({
# features: ['text', 'label'],
# num_rows: 25000
# })
# unsupervised: Dataset({
# features: ['text', 'label'],
# num_rows: 50000
# })
# })
# {'text': 'I rented I AM CURIOUS-YELLOW from my video store…', 'label': 0}Read the printout as a table of contents, not a wall of text. The dataset is a DatasetDict — a dictionary of named splits, which is the same three-way idea this lesson makes reproducible in Chapter 05 — and each entry is a Dataset with a shape (25,000 rows), a column list (text, label) and a data type per column. Indexing a row gives an ordinary dictionary, so the training code that consumes it needs no special handling at all.
| split | rows | what it is |
|---|---|---|
train | 25,000 | labelled reviews the model learns from |
test | 25,000 | labelled reviews for the final evaluation |
unsupervised | 50,000 | unlabelled reviews, for pretraining-style experiments |
Now the part of the workflow that pays for itself. The source’s sentence is one line long: after the first download, it loads from cache at ~/.cache/huggingface/datasets/. That directory holds an Arrow working copy of everything you have ever loaded, keyed by a fingerprint of how the dataset was built. The practical consequences are the ones beginners discover the hard way: the second run of your script skips the download — but it still resolves metadata, so a genuinely offline load needs HF_HUB_OFFLINE=1 or HF_DATASETS_OFFLINE=1; a deleted cache directory costs you exactly one re-download and no data loss; and the same dataset loaded by two different projects on the same machine is downloaded once, not twice.
Same call, four different costs. Run it cold, run it again, stream it instead, then delete the cache folder and watch the bill come back. Times are typical figures for an 84 MB download at 100 Mbps — your connection decides the real numbers.
The first load is the only load that pays the download. Everything after it — in this project, in another project, in a notebook, next month — is a local read from ~/.cache/huggingface/datasets/, though the library still resolves metadata unless HF_HUB_OFFLINE=1 is set.
You run `load_dataset('stanfordnlp/imdb')` today, then run the same script again next month. What does the library do the second time?
The source’s third step exists because of an honest fact: some datasets are too large to fit on disk. Streaming loads them row by row from the source, and the number that makes it special is not speed — it is memory, which stays constant no matter how large the dataset is.
The example is the one the source uses: the English Wikipedia dump, configured as wikimedia/wikipedia with the 20220301.en config. That is roughly 6.4 million articles and on the order of 20 gigabytes of text — approximately, since the exact figure depends on the snapshot. Downloading it to read five titles would cost about 27 minutes of transfer at a typical 100 Mbps connection and 20 GB of disk. Streaming costs neither:
dataset = load_dataset("wikimedia/wikipedia", "20220301.en", split="train", streaming=True)
for i, example in enumerate(dataset):
print(example["title"])
if i >= 4:
break
# example output (illustrative)
# Anarchism
# Autism
# Albedo
# Alabama
# AbolitionismWhat comes back is an IterableDataset — an object you iterate, not an object you index. There is no len(dataset), no dataset[0], and no guarantee about the order rows arrive in beyond what the source provides. What it does give you is the guarantee the source states: memory usage stays constant regardless of dataset size, because only the rows currently being read exist in memory. The loop above exits after five titles having touched five titles’ worth of data, out of 6.4 million.
The price is visible in the same sentence. Streaming is a cursor over a remote source: when the loop ends, the cursor is gone, and nothing promises that tomorrow’s run reads the same bytes without touching the network again. (The library may keep partial download buffers as it reads; what it never does is require a full local copy.) So the practical pattern is to stream into something smaller — a filtered subset, a count, a statistics pass — and then work with that.
from datasets import Dataset, load_dataset
stream = load_dataset("wikimedia/wikipedia", "20220301.en", split="train", streaming=True)
# one streaming pass: collect the rows you actually need
matches = [row for row in stream if "quantum" in row["title"].lower()]
# now you own a small Dataset — cache it, convert it, split it
small = Dataset.from_list(matches)
small.to_parquet("quantum_articles.parquet")
print(f"kept {len(small)} of ~6,400,000 articles")One pass over a dataset, drawn as the memory the process holds. The download workflow materialises rows as it goes — the line climbs. The streaming workflow holds one row plus a prefetch buffer — the line is flat no matter how big the dataset is. Drag the progress slider to move through the pass.
One honest caveat: a downloaded Dataset is backed by memory-mapped Arrow files, so a carefully written pipeline can keep memory low too. The flat line is what streaming guarantees without tuning — and it is the only option when the dataset does not fit on the disk at all.
The library uses Apache Arrow under the hood — a columnar, in-memory format built for zero-copy reads. You can convert to anything from there, but the source has a clear ranking, and the reasons are worth one paragraph of layout theory.
Four formats, two families. CSV (comma-separated values) and JSON (JavaScript Object Notation) are row-oriented text: human-readable, spreadsheet-friendly, and slow, because every read re-parses every character. Parquet (Apache Parquet) and Arrow (Apache Arrow) are columnar binary: the data for one column is stored together, so reading one column touches one region, and the similar bytes next to each other compress well. The library uses Arrow in memory — that is what you are actually holding when a dataset is loaded — and in its cache; Parquet is for the copy you keep on disk, the same layout with compression.
Converting is a method call away, exactly as the source shows. The to_* methods write files; the from_* constructors read them back:
dataset = load_dataset("stanfordnlp/imdb", split="train")
dataset.to_csv("imdb_train.csv")
dataset.to_json("imdb_train.json")
dataset.to_parquet("imdb_train.parquet")
# reading one back (no Hub, no cache — just the file)
from datasets import Dataset
reloaded = Dataset.from_parquet("imdb_train.parquet")
print(reloaded.column_names) # ['text', 'label']| format | size | read speed | best for |
|---|---|---|---|
| CSV | large | slow | human readability, spreadsheets |
| JSON | large | slow | APIs (application programming interfaces), nested data |
| Parquet | small | fast | analytics, columnar queries |
| Arrow | small | fastest | in-memory processing (what datasets uses internally) |
Be careful with the word “small”, because it is a typical figure and not a law. On text data, a Parquet file is often several times smaller than the CSV of the same rows — the compression sees repeated keys and similar strings — and reading it back is faster because nothing has to be parsed character by character. The exact ratio depends on your data and the writer’s settings; the ordering is what the source claims and what the lab beside this paragraph shows.
The source’s conclusion is three sentences and it is the rule to keep: for AI work, Parquet is the best storage format; Arrow is what you work with in memory; CSV and JSON are for interchange. A spreadsheet from a collaborator arrives as CSV and becomes Parquet once; an API answers in JSON and becomes Parquet once; the training loop never sees either.
The same text data written four ways. Sizes are typical factors (Parquet ≈ 0.30 × CSV, JSON ≈ 1.15 × CSV) and read bars use typical parse throughputs — the ordering is the lesson, the exact ratios are not. Set a dataset size and watch the four bars move together.
For AI work the source’s rule is three rules: Parquet for storage (small and fast to read back), Arrow for in-memory work (what datasets already uses), and CSV/JSON for interchange — the formats a spreadsheet or an API will hand you.
You convert a dataset to Parquet, then want to eyeball three rows in a text editor. What happens?
Every machine-learning project needs three piles of data, and the source’s shares are the industry default: 80% train, 10% validation, 10% test. Some datasets arrive pre-split; when they do not, you cut them yourself — and the two-line recipe has one subtlety that catches almost everyone.
The three sets have three different jobs, and the source states them as a sequence in time. Train is what the model learns from. Validation is what you check progress against during training — comparing two learning rates, or deciding when to stop. Test is the final, unbiased evaluation after all decisions are made. The reason the test set must stay untouched is the same reason a rehearsal is not a performance: the moment you tune anything against the test set, its number stops measuring generalization and starts measuring how well you fit it.
| split | typical share | what it is for | when it runs |
|---|---|---|---|
| train | 80% | the model learns from these rows | every training step |
| validation | 10% | check progress, compare choices | between training steps |
| test | 10% | the final unbiased evaluation | once, after training |
IMDb is the worked case: it arrives with a train and a test split (25,000 rows each) but no validation set, so you still have to carve one out. The source’s recipe cuts twice from a single loaded split — and the second cut is where the subtlety lives:
dataset = load_dataset("stanfordnlp/imdb", split="train")
split = dataset.train_test_split(test_size=0.2, seed=42)
train_val = split["train"].train_test_split(test_size=0.125, seed=42)
train_ds = train_val["train"]
val_ds = train_val["test"]
test_ds = split["test"]
print(f"Train: {len(train_ds)}, Val: {len(val_ds)}, Test: {len(test_ds)}")
# example output (illustrative — computed in the worked example below)
# Train: 14000, Val: 2000, Test: 4000The two recipes differ in where the second cut is applied. The doc snippet applies it to the train remainder and takes 12.5% of it — producing 14,000 / 2,000 / 4,000 (70/10/20) even though the surrounding text says 80/10/10. The course utility applies it to the held-out pool and splits the pool instead, which lands the stated target exactly: 16,000 / 2,000 / 2,000. The lesson’s split calculator lets you replay both recipes on any dataset size — the counts are computed live, not asserted.
That pool-then-halve shape is the general recipe, and it is exactly what code/data_utils.py implements in its make_splits helper — the version to copy:
def make_splits(ds, train_ratio=0.8, val_ratio=0.1, seed=42):
test_ratio = 1.0 - train_ratio - val_ratio
assert test_ratio > 0, "train_ratio + val_ratio must be less than 1.0"
test_size = val_ratio + test_ratio # hold out the pool
split1 = ds.train_test_split(test_size=test_size, seed=seed)
train_ds = split1["train"]
val_fraction = val_ratio / test_size # val's share OF THE POOL
split2 = split1["test"].train_test_split(test_size=(1.0 - val_fraction), seed=seed)
val_ds = split2["train"]
test_ds = split2["test"]
return {"train": train_ds, "val": val_ds, "test": test_ds}
# defaults on 20,000 rows → 16,000 / 2,000 / 2,000 (80 / 10 / 10)Now the seed. The source’s line is short and absolute: always set a seed for reproducibility — the same seed produces the same split every time. It matters because train_test_split shuffles before it cuts: the sizes are determined by the ratios, but which rows land in which pile is decided by the shuffle. Without a seed, tomorrow’s run gives the model a different training set and a different test set — and across a week of experiments, the test set you are “evaluating on” has quietly been in training. With seed=42, the same three piles come back on your laptop, your teammate’s, and the CI (continuous integration) runner.
A second worked example, for the 70/15/15 split the lesson’s exercises ask for, uses the same shrinking-pie rule on 20,000 rows: cut 1 holds out 0.15 + 0.15 = 0.3 → 6,000 rows, leaving 14,000; cut 2 takes half of that pool (0.5) → 3,000 validation and 3,000 test. Result: 14,000 / 3,000 / 3,000 = 70/15/15. On 1,000 rows the same recipe gives 700 / 150 / 150 — the shapes scale, the rounding is the only wobble.
Set a dataset size and the three ratios, and watch the two cuts produce the three counts. The recipe switch replays the doc’s literal snippet (0.2 then 0.125, cut on the train remainder) beside the course utility’s pool-then-split structure — the arithmetic the lesson corrects becomes visible. The seed changes which rows land where; the counts stay put.
The one sentence to keep: the second cut’s test_size is a fraction of what remains, not of the original dataset. The course’s utility derives it for you, and the seed is what makes the three piles identical on every machine.
A dataset has 20,000 rows. You call `train_test_split(test_size=0.2, seed=42)` and then split the result's train part with `test_size=0.125, seed=42`. What are the three sizes?
Datasets are not the only large files an AI project downloads. Model weights are frequently bigger, and they come from the same Hub through a sibling library — huggingface_hub — with the same promise: download once, load instantly afterwards.
The source’s sixth step names the two functions you will use for the rest of the course. hf_hub_download fetches one file from a repository by name — a config, a tokenizer, a single weights file. snapshot_download fetches the whole repository and hands back a directory. Both take a repo id like sentence-transformers/all-MiniLM-L6-v2 and both cache what they fetch:
from huggingface_hub import hf_hub_download, snapshot_download
model_path = hf_hub_download(
repo_id="sentence-transformers/all-MiniLM-L6-v2",
filename="config.json"
)
print(f"Cached at: {model_path}")
model_dir = snapshot_download("sentence-transformers/all-MiniLM-L6-v2")
print(f"Full model at: {model_dir}")
# example output (illustrative — the hash is not reproducible)
# Cached at: /Users/you/.cache/huggingface/hub/models--sentence-transformers--all-MiniLM-L6-v2/
# snapshots/5c38ec7c405ec4b44b94cc5a9bb96e735b38267a/config.json
# Full model at: /Users/you/.cache/huggingface/hub/models--sentence-transformers--all-MiniLM-L6-v2/
# snapshots/5c38ec7c405ec4b44b94cc5a9bb96e735b38267aThe source’s summary sentence is the one to remember: models cache to ~/.cache/huggingface/hub/, and once downloaded, they load instantly on subsequent runs. That directory is the twin of ~/.cache/huggingface/datasets/ from Chapter 02 — same roof, two shelves — and the same rules apply: deleting it costs one re-download, copying it between machines saves time, and the first run in any fresh environment (a Docker container, a rented machine with a GPU (graphics processing unit) — Phase 0, Lesson 07 and Lesson 03) pays the network.
Size is why this chapter exists at all. Model weights are stored per parameter, and the arithmetic is unforgiving: one parameter in 32-bit floats is 4 bytes, so 22 million parameters — the small sentence-transformer above — is about 88 MB before any metadata, and a 7-billion-parameter model is about 28 GB in 32-bit precision, or 14 GB in 16-bit. Those are computed sizes, not measurements, and they are the reason model files never belong in git — which is exactly what the next chapter settles.
| model shape | parameters | 32-bit weights | 16-bit weights | where it shows up |
|---|---|---|---|---|
| sentence encoder (MiniLM) | 22M | ≈ 88 MB | ≈ 44 MB | embeddings, search, retrieval-augmented generation (RAG) |
| BERT-base shape | 110M | ≈ 440 MB | ≈ 220 MB | classification, named entity recognition (NER) |
| 7B open model | 7,000M | ≈ 28 GB | ≈ 14 GB | generation, instruction following |
Model weights and large datasets should not go into git — the source states it flatly, then gives three ways to keep them out, from a one-line ignore rule to a full data-versioning system. The choice is a decision about who needs the bytes and whether they can be rebuilt.
Start with why, because “just commit it” is tempting. Git keeps every version of every file forever: a 250 MB checkpoint committed ten times is 2.5 GB of history that every clone downloads and nobody can delete without rewriting history. The dataset is 84 MB today and 840 MB after a few revisions. And the files themselves — weights, Arrow files, Parquet — are binary blobs that git cannot diff, compress, or merge, so it stores them whole. The source’s three options differ in how much machinery they add around that fact.
The simplest answer, and the one the source recommends for this course. List the patterns once and the files become invisible to git status:
*.bin
*.safetensors
*.pt
*.onnx
data/*.parquet
data/*.csv
models/Git LFS (Large File Storage) is an extension that stores large files on a separate server while keeping small pointer files in the repository. You still git add the file; what gets committed is a few lines of text naming its hash, and the bytes are fetched on checkout. The source’s commands:
git lfs install
git lfs track "*.bin"
git lfs track "*.safetensors"
git add .gitattributesDVC (Data Version Control) takes the pointer idea one step further: the data lives in your own storage — S3 (Amazon Simple Storage Service) or GCS (Google Cloud Storage) — and the small .dvc file that names its version is what git tracks. That pointer file is the version: check out an old commit and dvc checkout restores the exact data that experiment ran on.
pip install dvc
dvc init
dvc add data/training_set.parquet
git add data/training_set.parquet.dvc data/.gitignore
git commit -m "Track training data with DVC"
# then, once, point it at storage and push:
dvc remote add -d myremote s3://my-bucket/dvc-store
dvc push| approach | complexity | best for |
|---|---|---|
| .gitignore | low | personal projects, downloaded data you can re-fetch |
| Git LFS | medium | teams sharing model weights via git |
| DVC | high | reproducible experiments, large datasets, teams |
The source’s recommendation is refreshingly specific: for this course, .gitignore is enough — every dataset the lessons use is a download away, so the script is the version. Reach for DVC the day an experiment must reproduce exactly across machines, and for Git LFS the day the weights themselves are the deliverable. The storage pattern follows the same logic: local storage works for datasets under about 10 GB (the source’s number, and the HF cache handles it automatically); anything larger, or shared across machines, belongs in cloud storage — and the source notes that the moment arrives when you fine-tune on remote GPU instances, where the machine is not yours and the cache starts empty.
import os
local_path = os.path.expanduser("~/.cache/huggingface/datasets/")
print(local_path) # /Users/you/.cache/huggingface/datasets/
# cloud paths exist, but DVC is what manages them:
# s3_path = "s3://my-bucket/datasets/"
# gcs_path = "gs://my-bucket/datasets/"
# and from the shell:
# dvc remote add -d myremote s3://my-bucket/dvc-store
# dvc pushSix situations a real project hits, and the three answers from the source: ignore the file, track it with Git LFS, or version it with DVC. Choose one per situation; the feedback says why it wins and what the alternatives cost.
Situations answered: 0 / 6 · wrong picks: 0
A load script fetches the dataset and writes a Parquet copy. Nobody edits it by hand, and it can be rebuilt with one command.
The source’s recommendation for this course: .gitignore is enough. Reach for DVC when an experiment must reproduce exactly across machines — and for Git LFS when the weights themselves are what the team shares.
One last table before the check — the datasets this course actually downloads, with the sizes the source lists. None of them need to be fetched now; each lesson names what it needs, and the cache keeps them across the whole curriculum.
| dataset | lessons | size | what it teaches |
|---|---|---|---|
| IMDB | tokenization, classification | 84 MB | text classification basics |
| WikiText | language modeling | 181 MB | next-token prediction |
| SQuAD | question answering | 35 MB | question answering, spans |
| Common Crawl (subset) | embeddings | varies | large-scale text processing |
| MNIST | vision basics | 21 MB | image classification fundamentals |
| COCO (subset) | multimodal | varies | image-text pairs |
Answer before you look. The streaming question and the split question are the two that separate “I read the lesson” from “I can hand a teammate a data pipeline that still works next month”.
0 / 5 answered · 0 correct
01Why is it important to have separate train, validation, and test splits in machine learning?
02What is the Hugging Face Hub primarily used for in AI/ML workflows?
03What advantage does the Parquet format have over CSV for storing ML datasets?
04What does 'streaming=True' do when loading a dataset with the Hugging Face datasets library?
05When should you use DVC (Data Version Control) instead of just .gitignore for large files?
Click a card to swap the lazy description for what it actually means.
Four small drills — inspect a second dataset, stream one that is too large, compare Parquet against CSV with your own numbers, and build a 70/15/15 split. Try first; a worked answer is one click away.
Two lines do the loading (the first call downloads once; later calls read the cache): ```python from datasets import load_dataset ds = load_dataset("glue", "mrpc", split="train") print(ds) print(ds.features) print(ds[:5]) ``` Read the output in the same order as Chapter 02 taught. `print(ds)` shows a `Dataset` with 3,668 rows (the standard MRPC train split — the numbers come from the dataset card, and the config string `"mrpc"` is how a Hub dataset with several configurations is selected). `ds.features` shows the schema: `idx` (int32), `label` (a `ClassLabel` with names `['not_equivalent', 'equivalent']`), `sentence1` and `sentence2` (strings) — so a label of 1 means the two sentences are paraphrases of each other. `ds[:5]` shows five dictionaries at once; run `ds[0]` when you want a single row and print it slowly. Note what did *not* need to happen: no file paths, no manual download, and the second run of the cell is a local read from `~/.cache/huggingface/datasets/`.
```python import time from datasets import load_dataset # the Hub dataset id today is allenai/c4; the source's exercise says "c4" ds = load_dataset("allenai/c4", "en", split="train", streaming=True) start = time.time() count = 0 for example in ds: count += 1 if time.time() - start > 10: break print(f"processed {count:,} examples in ~10 s") ``` The honest part of the answer is what the number depends on: the network between you and the Hub's storage, how large each example's `text` field is, and whether the library is spending the first seconds finding the shard to read from. In practice you will see hundreds to a few thousand examples in ten seconds on a home connection — treat any single number as illustrative, not as a benchmark. What *is* reproducible is the shape of the result: the process never allocates memory for C4's full size (hundreds of gigabytes), the counter climbs while resident memory stays flat, and stopping early costs nothing. Compare it with the download path in one sentence for your notes: `streaming=True` traded the full local copy for a cursor that re-reads the source on the next pass.
```python from datasets import load_dataset from pathlib import Path ds = load_dataset("stanfordnlp/imdb", split="train") ds.to_csv("imdb_train.csv") ds.to_parquet("imdb_train.parquet") csv_mb = Path("imdb_train.csv").stat().st_size / 1e6 parquet_mb = Path("imdb_train.parquet").stat().st_size / 1e6 print(f"CSV {csv_mb:.1f} MB · Parquet {parquet_mb:.1f} MB · {csv_mb / parquet_mb:.1f}x smaller") ``` Expect Parquet to land *several times smaller* than CSV on this text data — in the ballpark of 84 MB of CSV to 25–30 MB of Parquet, so roughly 3×, and treat that as a typical figure from the lesson's teaching model rather than a promise (your numbers depend on the writer's compression settings and the data's own redundancy). Three reasons the difference exists: the columnar layout groups similar bytes so compression works better; Parquet stores typed columns instead of re-parsing text; and repeated values (like the label column) compress to almost nothing. Two footnotes worth writing down: the round trip is lossless — `Dataset.from_parquet("imdb_train.parquet")` gives back the same rows and features — and the comparison is not free of context, because CSV is the format a human or a spreadsheet can open directly, which is exactly why it stays in the interchange role.
Use the pool-then-split structure from `code/data_utils.py` rather than the doc snippet, and verify the arithmetic before trusting it: ```python from datasets import load_dataset ds = load_dataset("stanfordnlp/imdb", split="train") # 25,000 rows # cut 1: hold out the val + test pool (0.15 + 0.15 = 0.30) first = ds.train_test_split(test_size=0.30, seed=42) train_ds, pool = first["train"], first["test"] # cut 2: split the pool in half — 50% val, 50% test second = pool.train_test_split(test_size=0.5, seed=42) val_ds, test_ds = second["train"], second["test"] print(f"Train: {len(train_ds)}, Val: {len(val_ds)}, Test: {len(test_ds)}") # verification worth keeping in the notebook assert len(train_ds) == 17_500 assert len(val_ds) == 3_750 assert len(test_ds) == 3_750 assert len(train_ds) + len(val_ds) + len(test_ds) == len(ds) ``` On IMDb's 25,000-row train split, 70/15/15 is 17,500 / 3,750 / 3,750; on the lesson's rounder example of 20,000 rows it is 14,000 / 3,000 / 3,000. The reason the second `test_size` is 0.5 is the shrinking-pie rule: cut 1 put 30% of the whole dataset on the table, and cut 2 needs half of *that* to make 15% of the original. Run the cell twice with `seed=42` and the same three piles come back — change the seed and the counts stay identical while the membership moves, which is the entire point of a seed. The asserts are the habit worth copying: a split that does not add up is a data bug, and it is much cheaper to catch here than after training.
You do not need to master these here. Each one gets a proper treatment in its own lesson; the one-line meaning is enough to keep reading. Orange dotted underlines in the prose point back to this list.
This lesson is a port of an open course. Everything here traces back to it — and the next step is running the code yourself.
Lesson text, quiz and the data_utils.py walkthrough are adapted from AI Engineering from Scratch (Phase 00, Lesson 09). Everything the source states is kept as-is: the seven-node pipeline (Hugging Face Hub → datasets → load/stream → local cache → format conversion → data splits → your training pipeline), the install and load commands with the cache at ~/.cache/huggingface/datasets/, the Wikipedia streaming example and the constant-memory guarantee it makes, the Apache Arrow basis of the library with to_csv / to_json / to_parquet conversion and the four-row format table (CSV/JSON large and slow, Parquet small and fast, Arrow fastest and in-memory), the three-split explanation with the source's two train_test_split calls and seed=42, the hf_hub_download / snapshot_download pair with the hub cache at ~/.cache/huggingface/hub/, the three large-file options (.gitignore, Git LFS with GitHub's 1 GB free, DVC with .dvc pointers to S3 or GCS) and their comparison table, the local-under-~10-GB storage guidance, and the course dataset table (IMDB 84 MB, WikiText 181 MB, SQuAD 35 MB, MNIST 21 MB, COCO subsets, Common Crawl). The quiz's five questions keep their correct answers. Original to this page: the six lab components (the pipeline board walkthrough, the SVG cache timeline, the canvas streaming-versus-download simulator, the canvas format comparator, the canvas split calculator and the large-file strategy chooser); the split arithmetic worked end to end on 20,000 rows, including the honest correction that the doc snippet's 0.2 → 0.125 cut on the train remainder lands 14,000 / 2,000 / 4,000 = 70/10/20 while the course's code/data_utils.py::make_splits cuts the val+test pool first and lands the stated 16,000 / 2,000 / 2,000 = 80/10/10, with the exercise's 70/15/15 variant at 14,000 / 3,000 / 3,000; the shrinking-pie and cache-as-pantry memory hooks; IMDB's own structure (25,000 / 25,000 / 50,000 rows, no validation split); the typical format factors used by the comparator (Parquet ≈ 0.30 × CSV, JSON ≈ 1.15 × CSV, Arrow fastest) labelled as teaching estimates; the model-size arithmetic (4 bytes per parameter at 32-bit precision: 22M ≈ 88 MB, 110M ≈ 440 MB, 7B ≈ 28 GB / ≈ 14 GB at 16-bit) with the HF_HOME / HF_HUB_CACHE / HF_DATASETS_CACHE note and the gated-model login line; the streaming trade-offs and the stream-once-then-save-Parquet pattern; the row-versus-columnar figure and the "the format is the pipeline's speed limit" AI connection; the .gitignore-is-not-versioning trap and the checkout-that-took-forty-minutes scenario. Version numbers, timings, file sizes, format ratios and example command outputs are labelled teaching estimates or illustrative examples; the labs compute their numbers live, and every simulation says so on the canvas.