EVERYTHING AIAI engineering, made visual
0/12 complete
LESSON 11 · SETUP & TOOLING × AI · LEARN

The cloud box has no screen.
You have a cursor and a meter.

You develop on macOS or Windows; the moment you SSH into a rented GPU box you land in Ubuntu, where the terminal is the only interface and idle minutes cost money. This lesson is the survival guide: one filesystem tree under /, the commands that cover 95% of the work, permissions, apt, processes, disk, tmux, and the macOS habits that quietly break.

30 MIN · 7 CHAPTERS + CHECKPREREQ · PHASE 0 · LESSON 01
FIG. 11 / A SIMULATED SESSION · TREE, PROCESSES, PERMISSIONS
filesystem processes · GPU denied allowed
LESSON 11TYPE · LEARN~30 MINPREREQ · PHASE 0 · LESSON 01ORIGINAL LESSON ↗
THE 60-SECOND VERSIONSee the problem ↓
01 / ONE TREE, ONE DOOR

Everything lives under a single /.

There is no C:\ and no /Volumes on the box: one hierarchy, and every disk or volume is attached somewhere inside it. home holds your files, tmp is wiped on reboot, usr and etc belong to the system, var/log is where breakage leaves a trail, mnt and media is where a dataset volume appears, and proc and sys are the kernel talking. Your work lives in ~, which is just /home/your-username.

~ → /home/your-username · cd ~ takes you there from anywhere
02 / THE COMMANDS ARE THE UI

A dozen verbs cover 95% of the work.

The source calls them the 15 commands that cover 95%: pwd, ls, cd, mkdir, cp, mv, rm, cat, head, tail, less, grep, find — plus the flags that do the real work (ls -la, mkdir -p, cp -r, rm -rf, tail -f, grep -r, find -size +1G). Permissions decide who may run what (chmod +x, chmod 755, chmod 644), and apt installs system software after apt update refreshes the index.

chmod +x train.sh · sudo apt update && sudo apt install -y tmux
03 / SESSIONS OUTLIVE LAPTOPS

tmux keeps the run alive; rsync moves the data.

Closing your laptop ends the SSH session, and anything that was not inside tmux or nohup goes with it — a five-hour fine-tune dies at hour four with no checkpoint. Transfer large files with rsync, not scp: it sends only changed bytes and resumes an interrupted transfer. And your macOS muscle memory lies: brew is apt, ~/.zshrc is ~/.bashrc, sed -i needs no empty string, file names are case-sensitive.

tmux new -s train → Ctrl+B, d → tmux attach -t train
MENTAL MODEL IN ONE SENTENCE

The remote box is one tree under / and the terminal is the only door: your job is to know which branch you are standing on (pwd, ls, ~), who owns and may touch what (chmod, chown, sudo), what is still running and what it costs (htop, nvidia-smi, tmux), and how full the disk is (df -h, du -sh).

By the end you will be able to navigate any Linux box from the command line (pwd, ls -la, cd), create, copy, move and delete files safely, read and search logs (tail -f, grep -r, find -size +1G), decode a permission string like -rwxr-xr-- into chmod 755 and fix a “Permission denied”, install a fresh GPU box’s software with apt, find and stop a runaway process with htop / kill, free disk space with du and apt clean, move data with rsync, and keep a training run alive across disconnects with tmux.

THE BOX WITH NO SCREEN

You rented a GPU.
It came with a terminal.

You develop on macOS or Windows. But the moment you SSH (Secure Shell) into a cloud GPU box — a Lambda instance, an EC2 machine, a university cluster — you land in Ubuntu. The terminal is your only interface: no Finder, no Explorer, no GUI (graphical user interface). This chapter is about what that costs, and what the rest of the lesson gives you so it stops costing.

The source states the stakes plainly: if you cannot navigate the file system, install packages and manage processes from the command line, you are stuck paying for idle GPU hours while you google how to unzip a file. A rented GPU (graphics processing unit) bills by the hour — or by the minute, priced up — whether it is training a model, downloading a dataset, or sitting there while you read an error message twice. The box does not care that you have never seen a Linux prompt before.

The fix is not to learn Linux. It is to learn enough Linux — the exact subset a remote AI box makes you use. That is the frame the source insists on: this is a survival guide, and it covers exactly what you need to operate on a remote Linux machine for AI work. Nothing more. Everything in the next seven chapters is something you will type in your first week on a rented box; nothing else is invited.

  1. Navigate a Linux filesystem and do the file operations that matter — from the command line, without a file manager.
  2. Decode and change permissions, so “Permission denied” becomes a two-second diagnosis instead of a wall.
  3. Install system software with apt and bring a fresh GPU box up to working order with one command block.
  4. Recognise the macOS-to-Linux differences that silently break scripts, imports and habits.

The meter is the whole argument. Every other lesson in this course would let you stop and come back; a rented box punishes the pause. Pick a card class and the minutes a detour eats — the rates are illustrative teaching numbers, not quotes: the source only says the box bills by the hour, and Lesson 03 (GPU Setup & Cloud) discusses the wider range — and the arithmetic below is the one thing in this lesson worth memorising before any command: cost = minutes ÷ 60 × rate.

EXTRA INTERACTIVE · THE METER IS RUNNING

The box bills whether or not you are learning. Pick a card class and the minutes a detour costs you — reading an error, hunting a file, guessing a command — and watch what the detour is worth.

card class · example rate
$0.5530 min × $1.10/h on A10G · 24 GB
role a common fine-tuning class card rate $1.10/hour (illustrative teaching rate) minutes 30 arithmetic 30 / 60 × $1.10 = $0.55 five detours this week $2.75 twenty detours this month $11.00 the fix a survival guide: pwd, ls, chmod, apt update, htop, df -h, tmux attach

At this rate the detour costs more than the run. The commands that stop the bleed: pwd, ls, htop, df -h, tmux attach.

ONE TREE UNDER /

One tree, seven branches.
You will meet them all in a week.

Linux organizes everything under a single root: /. There is no C:\ and no /Volumes — one hierarchy, and every disk, volume or network mount is attached somewhere inside it. Almost everything you do as a GPU user happens in one subtree of it: your home directory.

Your home directory is ~ — a shell shortcut, not a directory name. The shell expands it to /home/your-username before the command ever runs, which is why cd ~ works from anywhere and why du -sh ~/.cache means the same thing as du -sh /home/ubuntu/.cache. On a cloud box the username is usually ubuntu (or root, whose home is /root). Cloning repositories, building virtual environments, running training and keeping logs — all of it lives here unless you decide otherwise.

Two kinds of path, one rule between them: a path that starts with / is absolute — it means the same place from anywhere. A path that does not is relative to where you currently are, which pwd (print working directory) will tell you. . means “here”, .. means “the parent directory”, so cd ../.. climbs two levels, and cd ~ is the escape hatch back to familiar ground. Being lost on a remote box is almost always a question with a one-word answer: pwd.

Where the gigabytes actually live. A GPU box’s boot disk is small compared to the data AI work produces, and the filesystem layout tells you where to look when it fills. ~/.cache holds pip’s wheels and HuggingFace model weights, which for a single 7B-parameter model is about 14 GB in fp16 (7 billion parameters × 2 bytes — plus tokenizer and config files), and a cache with two or three models reaches 40 GB without trying. Training checkpoints land wherever your script writes them — if that is ~/project, two checkpoints at 3.2 GB and 3.1 GB are 6.3 GB of your home directory. Datasets that do not fit at all belong on a mounted volume under /mnt, which is why df -h /mnt/data is the first command on a new box. (All sizes here are illustrative examples, not measurements of your machine.)

The one directory to be careful with is /tmp: it is scratch space, and it is cleared on reboot. Unpack an archive there, not an only copy of anything. The map below is the whole chapter in one picture — click around it until the seven names feel like places rather than strings.

The filesystem explorer

One tree, seven directories you will actually meet. Click a node — or focus it and press Enter — to see what lives there and when a GPU user touches it. The tree is simplified: `/var/log` is drawn as one node, and a real box has dozens more entries.

/the roothome/ubuntu · ~your hometmptemporary filesusrsystem programsetcconfigurationvar/loglogsmnt · mediamounted storageproc · systhe kernel talkingSIMPLIFIED MAP — EVERY PATH ON THE BOX STARTS AT /
home/ubuntu · ~ what lives here Your files. Your username's directory is what `~` expands to, so `~/project` and `/home/ubuntu/project` are the same place. Almost everything you do on the box happens here. when you touch it Constantly: you clone repositories here, build virtual environments here, run training here, and read `~/.bashrc` when a shell setting needs changing. a command to try cd ~ && pwd && ls -la

On a GPU box: `~/.cache` is where pip wheels and HuggingFace model weights land by default — it is the first place to look when disk fills up.

Quick check

You are in /home/ubuntu/projects/train and you type `cd ..`. Where are you now?

THE COMMANDS THAT COVER 95%

A dozen verbs.
Almost everything you will ever type.

The source calls this list “the 15 commands that cover 95% of what you’ll do on a remote GPU box”. Counted as distinct verbs it is thirteen; counting the flag-first forms it treats as separate tools — ls -la, tail -f — is how it reaches fifteen. Keep the shape, not the number: a short list, learned properly, replaces a file manager.

Read the blocks below the way you would read a phrasebook: the command first, the comment after it explaining why you would reach for it. The $ lines are illustrative example output, not a recording of your machine — your paths and sizes will differ, but the shapes will be the same. Everything here is the source’s own command set.

Moving around

movement · where am I, what is here, go therebash
pwd                         # Where am I?
ls                          # What's here?
ls -la                      # What's here, including hidden files with details?
cd /path/to/dir             # Go there
cd ~                        # Go home
cd ..                       # Go up one level

# illustrative output — yours will differ
$ pwd
/home/ubuntu/projects
$ ls -la
drwxr-xr-x  2 ubuntu ubuntu 4096 Sep 14 09:12 .
drwxr-xr-x 14 ubuntu ubuntu 4096 Sep 14 08:55 ..
-rw-rw-r--  1 ubuntu ubuntu 1024 Sep 14 09:11 notes.txt
-rwxr-xr-x  1 ubuntu ubuntu 2048 Sep 14 09:11 train.sh
From the source. `ls -la` is the command you will type ten thousand times: `-l` asks for the long form with permissions, owner, size and date; `-a` includes the hidden dotfiles (`.`, `..`, `.bashrc`) that plain `ls` omits. The first column is what Chapter 04 decodes.

Files and directories

files · mkdir, cp, mv, rmbash
mkdir my-project            # Create a directory
mkdir -p a/b/c              # Create nested directories in one shot

cp file.txt backup.txt      # Copy a file
cp -r src/ src-backup/      # Copy a directory (recursive)

mv old.txt new.txt          # Rename a file
mv file.txt /tmp/           # Move a file

rm file.txt                 # Delete a file (no trash, it's gone)
rm -rf my-dir/              # Delete a directory and everything inside
From the source. `mkdir -p` is the flag people discover late: without it, making a/b/c requires three commands. `cp -r` is the recursive flag — copy a directory without it and cp refuses. `rm -rf` is permanent: there is no trash, no undo, and no confirmation prompt.

Reading files

On a box with no GUI there is no double-click. These five commands are how you look at a file, from a quick peek to watching a log grow while training runs.

reading · cat, head, tail, tail -f, lessbash
cat file.txt                # Print entire file
head -20 file.txt           # First 20 lines
tail -20 file.txt           # Last 20 lines
tail -f log.txt             # Follow a log file in real time (Ctrl+C to stop)
less file.txt               # Scroll through a file (q to quit)

# illustrative output
$ tail -3 training.log
2026-09-15 09:41:02 epoch 12 loss 0.412
2026-09-15 09:44:18 epoch 13 loss 0.398
2026-09-15 09:47:31 epoch 14 loss 0.391
From the source. The distinction that matters: `cat` is for small files, `less` for anything long (it loads lazily and lets you scroll, and `q` quits), and `tail -f` is the training-run companion — it keeps printing new lines as they are written. Ctrl+C stops the follow, not the training.

Searching

searching · grep and findbash
grep "error" training.log           # Find lines containing "error"
grep -r "learning_rate" .           # Search all files in current directory
grep -i "cuda" config.yaml          # Case-insensitive search

find . -name "*.py"                 # Find all Python files under current dir
find . -name "*.ckpt" -size +1G     # Find checkpoint files larger than 1GB

# illustrative output
$ find . -name "*.ckpt" -size +1G
./checkpoints/epoch_01/model.ckpt
./checkpoints/epoch_02/model.ckpt
From the source. `grep -r` walks a directory tree; `grep -i` ignores case (`CUDA`, `Cuda`, `cuda` all match — add `-n` for line numbers). `find` matches names; `-size +1G` keeps only large files, which is how you hunt the checkpoints that filled the disk in Chapter 06. GNU find rounds each size up to the unit, so `+1G` means 'more than 1 GiB': a file of exactly 1 GiB does not match, and 1 GiB + 1 byte does.
Quick check

A training job is running in another terminal and writes to training.log. You want to watch new lines appear as they are written. Which command?

WHO MAY DO WHAT

-rwxr-xr-- is not noise.
It is an arithmetic problem.

Every file on Linux has an owner and a set of permission bits. You meet them the first time a script will not run or a dataset will not open — and “Permission denied” is almost always this one mechanism, not a broken install or a corrupted file.

Start with the string. ls -l prints ten characters: one type character, then three groups of three. In -rwxr-xr-- the leading - says “regular file” (d would mean directory, l a symbolic link); rwx is what the owner may do; r-x is what anyone in the file’s group may do; r-- is what everyone else may do. r is read, w is write, x is execute — and a dash means the permission is absent. On a directory, x means “enter or traverse”, not “run”: a directory you may read but not enter gives you a listing without the ability to cd into it.

Then the arithmetic. Each identity’s triple is a three-bit number: read = 4, write = 2, execute = 1. Add the bits an identity holds and you get one octal digit. For -rwxr-xr--: owner is 4 + 2 + 1 = 7, group is 4 + 0 + 1 = 5, other is 4 + 0 + 0 = 4 — the whole mode is 754. Run it the other way and chmod 755 spells rwxr-xr-x, while chmod 644 spells rw-r--r--. A second worked example, because this is the step people blur: -rw-rw-r-- is 6 (4 + 2) for the owner, 6 for the group, 4 for other → 664, which is what a project file looks like when a shared group may edit it. And the mode for a secrets file is 600: rw------- — owner reads and writes, nobody else gets anything.

reading and changing permissionsbash
ls -l train.py
# -rwxr-xr-- 1 user group 2048 Mar 19 10:00 train.py
#  ^^^             owner permissions: read, write, execute
#     ^^^          group permissions: read, execute
#        ^^        everyone else: read only

chmod +x train.sh           # Make a script executable
chmod 755 deploy.sh         # Owner: full, others: read+execute
chmod 644 config.yaml       # Owner: read+write, others: read only

chown user:group file.txt   # Change who owns a file (needs sudo)
From the source. `chmod +x` is symbolic: add the execute bit, leaving the rest of the mode alone. `chmod 755` is absolute: set the whole mode to exactly those three digits. `chown` changes ownership and is a root operation — hence `sudo` in front of it.
One octal digit per identity. The digit is a sum, not a code you memorise: 4 = read, 2 = write, 1 = execute.
digitbitswhat it grants
7rwxread, write and execute — everything
6rw-read and write, but not execute
5r-xread and execute — run it, do not edit it
4r--read only
0---no access at all

One rule that most tutorials skip: the kernel does not add the classes together. It checks the owner class first, then the group, then other, and stops at the first class that matches you. If you are the owner and the owner triple is ---, you are denied even though the group triple says rwx and the world triple says rwx — membership in a class ends the search. That is why the diagnosis for “Permission denied” is mechanical: ls -l the file, work out which class you are in, and read the bit you need. Fixing it is a chmod on that class, or a chown to move the file into a class that has the bit. Two practical footnotes: bash train.sh runs a script even without the execute bit, because bash only needs to read it — which is why a script can run one way and fail the other; and chmod +x grants the bit to all three classes unless you say chmod u+x.

The permission decoder

Flip the nine mode bits and watch the symbolic string and the octal number change together — the two spellings of the same fact. Then ask as the owner, the group or the world, and see whether the kernel would let the request through. This is the classic Unix mode model as a teaching simulation.

Who is asking
Wants to
File type
Mode bits · click to flip
owner
group
other
Common modes
ls -l train.sh -rwxr-xr-x 1 you users 4096 Sep 15 09:12 train.sh symbolic -rwxr-xr-x octal 755 as owner (the user who owns the file) asking for x (execute) verdict ✓ allowed owner rwx = 4 + 2 + 1 = 7 group r-x = 4 + 1 = 5 other r-x = 4 + 1 = 5 fix chmod 755 train.sh preset note scripts and directories everyone may run or enter

The one arithmetic worth owning: read is 4, write is 2, execute is 1, and each identity’s bits add up to one digit — 4 + 2 + 1 = 7 for rwx, 4 + 1 = 5 for r-x, 4 for r--.

INSTALLING SOFTWARE

Fresh box, missing tools.
apt fixes it in one line.

Ubuntu installs system-level software with apt — the Advanced Package Tool. It is how a bare GPU image gets a compiler, a downloader, a process viewer and a session manager. The one rule the source puts first: run apt update before every install.

update and upgrade are not the same command. sudo apt update refreshes apt’s local index — the catalogue of what versions exist in the repositories — and installs nothing. sudo apt upgrade installs newer versions of packages you already have. On a fresh box whose index has never been fetched, the catalogue is empty, so apt install answers Unable to locate package for software that absolutely exists. The -y flag answers the confirmation prompt for you, which is what makes one-line setup commands possible.

You are not root — and you should not be. On a cloud GPU instance you are typically the only user, and you already have sudo (superuser do) rights: prefix a single command with sudo and it runs with root’s privileges. The source is explicit about the balance — don’t run everything as root; use sudo only when needed. The cost of ignoring that is concrete: files created by root are owned by root, so the next git pull or pip install run as your user fails with “Permission denied” on files you thought were yours; and a global pip install as root writes packages into the system interpreter that every project shares — the dependency-hell setup Phase 0, Lesson 06 exists to undo. whoami tells you which side you are on; sudo su opens an interactive root shell and is best kept for short, deliberate visits (exit takes you back).

What a fresh AI box actually needs. The source’s one-liner installs the build toolchain (many Python packages compile against it), the network tools, a terminal multiplexer for sessions, a process viewer, an unpacker and the Python virtual-environment module. In the lab’s illustrative model that resolves to 18 packages, about 96 MB of downloads and about 404 MB on disk — and the toolchain is most of it, because build-essential pulls in gcc, g++, make and the C headers. Undoing any of it is one apt remove; clearing the downloaded .deb files is one sudo apt clean, which frees the cache without touching the installed packages.

apt · install, inspect, removebash
sudo apt update             # Refresh the package list (always do this first)
sudo apt install -y htop    # Install a package (-y skips confirmation)
sudo apt install -y build-essential  # C compiler, make, etc. Needed by many Python packages
sudo apt install -y tmux    # Terminal multiplexer (keep sessions alive after disconnect)

apt list --installed        # What's installed?
sudo apt remove htop        # Uninstall
sudo apt clean              # Clear the download cache (.deb files)
From the source, plus `apt clean` from its disk-space section. `apt list --installed` needs no sudo; `apt remove` and `apt clean` do. A package that other packages depend on cannot be removed without taking the dependents too — apt refuses rather than leaving a broken toolchain.
the fresh-GPU-box one-linerbash
sudo apt update && sudo apt install -y \
    build-essential \
    git \
    curl \
    wget \
    tmux \
    htop \
    unzip \
    python3-venv
From the source verbatim. Read it as four groups: the compiler (build-essential), the network and repository tools (git, curl, wget), the survival tools (tmux, htop, unzip), and the Python environment module (python3-venv) that Phase 0, Lesson 06 then uses. Run it once on a new box and every later lesson assumes it.

Users and sudo

who am I, and who may I becomebash
whoami                      # What user am I?
sudo command                # Run a single command as root
sudo su                     # Become root (exit to go back, use sparingly)
From the source. `whoami` returning `root` is the tell that you are in a root shell — run `exit` and go back to your normal user before continuing. On cloud boxes you usually have sudo already; you do not need to become root to use it.

The apt dependency resolver

A fresh GPU box, a stale package index, and the source’s install one-liner. Pick packages, run sudo apt update first, then install and watch apt resolve dependencies, download archives and grow the disk footprint. Every version and size here is an illustrative teaching estimate, not a quote from a real repository — and the Python row is simplified, for instance: Ubuntu marks pip as a recommendation of the venv package rather than a hard dependency.

PACKAGE INDEX · STALEillustrative 40 GB boot disk · 12.6 GB used before you start

12.60 GB / 40 GB used · 27.40 GB free · apt cache 0 MB

build tools
networking
sessions & monitoring
file utilities
Python
RESOLUTION · DEPENDENCIES COME FIRST

Nothing to install yet. Select packages above, then press sudo apt install -y.

INSTALLED · CLICK A PACKAGE TO REMOVE IT

The box is bare. The source’s fresh-GPU-box stack is 8 packages; the one-liner is sudo apt update && sudo apt install -y \.

index stale — installs will fail selected none resolution nothing queued download 0 MB on disk 0 MB installed 0 package(s) · 0 MB apt cache 0 MB of .deb files disk 12.60 GB / 40 GB · 27.40 GB free sudo apt install -y # select packages first
ubuntu@cloud-gpu:~$ # fresh box — the apt index has never been refreshed

Two lessons live in the numbers: apt update refreshes the index (nothing is installed by it), and the toolchain is the heavy part — build-essential drags in a compiler and headers. Removing a package something else depends on is refused, which is apt protecting you from a broken toolchain.

Quick check

On a brand-new Ubuntu box you run `sudo apt install -y htop` and it fails with `E: Unable to locate package htop`. What is the most likely cause?

PROCESSES, SERVICES, DISK

Something is running.
Find it, judge it, then stop it.

Three questions end most remote-box panics: what is running, what is holding the GPU, and how much disk is left. The source’s tools for those questions are five commands and one habit — read the whole table before you kill anything.

htop is an interactive process viewer: CPU and memory per process, sorted how you like, with q to quit. ps aux | grep python is the same idea in one line — ps (process status) lists processes, and the pipe sends the list to grep to keep the lines mentioning Python. You now have a PID (process identifier): the number you need to talk to the process. nvidia-smi (the NVIDIA System Management Interface) adds the GPU’s view — which PIDs hold GPU memory and how much of the card is in use, reported in MiB (mebibytes, the powers-of-two unit GPU tools use) — and it is the first command to run when a training job claims there is no memory left.

Stopping a process is a conversation with signals. kill 12345 does not mean “destroy” — it sends SIGTERM (signal 15), which asks the process to stop and lets it clean up: flushing files, closing connections, writing a checkpoint. A well-behaved training loop uses that chance. When the process is wedged and ignores the request, kill -9 12345 sends SIGKILL (signal 9), which the kernel delivers whether the process wants it or not — no handler runs, no cleanup happens, and unsaved work is gone. Your shell reports a signal death in the 128 + signal convention: 143 = 128 + 15 when SIGTERM’s default action kills the process, 137 = 128 + 9 for SIGKILL. A process that catches SIGTERM and exits cleanly reports its own exit code instead — often 0 — so 143 tells you which signal landed, not that cleanup was skipped. Neither number is a problem in itself; they are the receipt.

processes · see, then signalbash
htop                        # Interactive process viewer (q to quit)
ps aux | grep python        # Find running Python processes
kill 12345                  # Gracefully stop process with PID 12345 (SIGTERM, 15)
kill -9 12345               # Force kill (SIGKILL, 9) — use when graceful doesn't work
nvidia-smi                  # GPU processes and memory usage
From the source. Reading the table first is the discipline: CPU, memory and GPU memory can each belong to a different process, and the one you want to stop is not always the one at the top. `nvidia-smi` also answers the most common training question — is the GPU actually being used, or is the job limping on the CPU?

The process manager

A simulated htop for a busy GPU box: sort by memory, look at what holds the GPU, then ask one process to stop with SIGTERM and force it with SIGKILL. The training process saves a checkpoint when asked politely; the inference server in this teaching model does not. Nothing here touches a real machine.

Sort the table
Select a process
simulated 6 process(es) · sorted by %MEM (memory) gpu memory 21308 / 23028 MiB (93%) gpu holders 6702 (7100 MiB), 4211 (14208 MiB) 4211 you 87.4% CPU 6120 MB RAM 14208 MiB GPU python train.py --epochs 50 the training run: SIGTERM asks it to stop, and a well-written loop writes a checkpoint before exiting exit codes SIGTERM → handled: the program's own code (train.py: 0) · default: 143 = 128 + 15 SIGKILL → 137 = 128 + 9
$ ps aux | grep python you 4211 87.4 python train.py --epochs 50 you 6702 34.1 python -m vllm.entrypoints.openai.api_server you 5122 21.3 python data_prep.py --shard 3/8

The takeaway is a habit: read the whole table before killing anything. Here the biggest RAM user (the inference server) is not the biggest GPU user — kill the wrong one and you free memory while the card stays full.

Services are processes that start themselves. An inference server or a web dashboard is not launched by hand; it is a service managed by systemd (the system and service manager), and it is controlled with systemctl. The verbs are worth knowing cold: start, stop, restart after a config change, status to see whether it is running (and read the last log lines it produced), and enable to make it start again automatically at boot — the one people forget, which is why their server works until the box reboots and then mysteriously does not.

systemd · services in five verbsbash
sudo systemctl start nginx          # Start a service
sudo systemctl stop nginx           # Stop it
sudo systemctl restart nginx        # Restart it
sudo systemctl status nginx         # Check if it's running
sudo systemctl enable nginx         # Start automatically on boot
From the source. `status` is the debugging verb — it shows whether the unit is active and prints its recent log lines, which is often where the actual error is. Nothing here is AI-specific; your model's inference server is just another service.

Then there is the disk. GPU boxes usually have a small boot disk, and models, datasets, caches and checkpoints fill it fast. Two commands, one letter apart, answer different questions: df -h (disk free) reports each mounted filesystem’s size and free space, while du -sh (disk usage) reports how much a path you name is using. df tells you the tank is nearly empty; du tells you what has been loaded into it. The source’s space-hog hunt is one pipeline: du -h --max-depth=1 / 2>/dev/null | sort -hr | head -20 — measure one level down, throw away the permission-denied noise, sort by human-readable size in reverse, keep the top twenty.

An illustrative afternoon on a 200 GB box: df -h /home reports 186 GB used and 14.2 GB free — 93% full, and training is about to write another checkpoint. du -sh * in the project shows checkpoints/epoch_01 at 3.2 GB and checkpoints/epoch_02 at 3.1 GB, both superseded. du -sh ~/.cache shows the pip and HuggingFace caches at 2.7 GB of wheels you can re-download, and sudo apt clean can return 0.6 GB of cached .deb files. Delete the two old checkpoints, purge the pip cache, clean apt — 9.6 GB back, and the box goes from 14.2 GB free to about 23.8 GB free without touching a single training artifact you still need. (Sizes and outputs are illustrative.)

disk · df tells you it's full, du tells you whybash
df -h                       # Disk usage for all mounted drives
df -h /home                 # Disk usage for /home specifically

du -sh *                    # Size of each item in current directory
du -sh ~/.cache             # Size of your cache (pip, huggingface models land here)
du -sh /data/checkpoints/   # Check how big your checkpoints are

# Find the biggest space hogs
du -h --max-depth=1 / 2>/dev/null | sort -hr | head -20

# Common space savers
pip cache purge             # Clear pip cache (pip = the Python package installer)
sudo apt clean              # Clear apt cache
rm -rf checkpoints/epoch_01/ checkpoints/epoch_02/   # Remove old checkpoints
From the source. In the pipeline, `--max-depth=1` stops the walk at the top level, `2>/dev/null` discards the permission errors from directories you cannot read, `sort -hr` sorts numerically in reverse using human units (so 9G beats 800M), and `head -20` keeps the top of the list. `pip cache purge` and `apt clean` delete caches only — both re-download what they need later.
Quick check

A training process is stuck and `kill 4211` did nothing — the process is still running. What is the correct next step?

SESSIONS THAT OUTLIVE YOU

The box keeps working.
Your laptop does not have to.

The last survival skill is continuity: moving files between your laptop and the box, hitting an API (application programming interface) from the command line, and making sure a five-hour training run survives the moment you close the lid.

Downloads are one command. wget fetches a URL; curl -O does the same and writes the file under its remote name. curl’s real superpower on an AI box is talking to APIs: curl -s https://api.example.com/health fetches quietly, and piping the result through python3 -m json.tool pretty-prints the JSON (JavaScript Object Notation) response so you can actually read it. That one line is the fastest way to answer “is the endpoint up, and what is it saying?” before you blame your own code.

Transfers come in two flavours. scp (secure copy) copies a file or directory over SSH in one shot — fine for a single small file you expect to move in one attempt, in either direction. rsync is the professional choice for anything large or retry-expensive: it compares the two sides and transfers only what differs, and it can carry on from where an interrupted transfer stopped. The source’s exact flags: rsync -avz --progress -a archive mode (preserve permissions, timestamps, recurse), -v verbose, -z compress in transit, and --progress so you can see it moving. One flag the honest version of “resumes on failure” needs: --partial keeps the part of the file that did arrive, so the next run continues into it instead of deleting the temp file. scp starts a dropped transfer over from zero; rsync with --partial continues — on a flaky link or a 40 GB checkpoint, that is the difference between minutes and hours. And one detail that saves people: a trailing slash on the source means “the contents of this directory”, no slash means “the directory itself.”

networking · download, query, transferbash
# Download files
wget https://example.com/model.bin                   # Download a file
curl -O https://example.com/data.tar.gz              # Same thing with curl
curl -s https://api.example.com/health | python3 -m json.tool  # Hit an API, pretty-print JSON

# Transfer files between machines
scp model.bin user@remote:/data/                     # Copy file to remote machine
scp user@remote:/data/results.csv .                  # Copy file from remote to local
scp -r user@remote:/data/checkpoints/ ./local-dir/   # Copy directory

# Sync directories (changed bytes only; add --partial to resume a dropped transfer)
rsync -avz --progress ./data/ user@remote:/data/
rsync -avz --progress user@remote:/results/ ./results/
Commands from the source, with the resume caveat the lesson's rule needs. The trailing slash on the source means 'the contents of this directory': `./data/` copies what is inside `./data` into the remote `/data/`. The second line mirrors the direction — everything inside the remote `./results/` lands in the local directory. `-z` compresses in transit: a big win on text, a small one on archives that are already compressed.

tmux is the answer to the closed laptop. The mechanics — sessions, detach, attach, panes — are Lesson 10’s territory (“tmux, the keeper”); what matters on a rented box is the consequence. A tmux session runs on the server, so a dropped SSH connection closes a view, not the job; without one, a five-hour fine-tune dies at hour four — and the meter kept running the whole time. The source’s instruction is unqualified: always run long training jobs inside tmux. Always. Detach with Ctrl+B then d, and come back with tmux attach -t train.

tmux · the survival reminderbash
tmux new -s train           # Start a session on the box (Lesson 10 has the mechanics)
# ... start your training, then:
# Ctrl+B, then d            # Detach (training keeps running)

tmux attach -t train        # Reattach from any connection
The source's rule, kept short: the session lives on the server, so closing the laptop cannot kill what it does not own. Lesson 10 (Terminal & Shell) covers the full workflow — panes, `tmux ls`, kill-session and the 2,000-line scrollback default.

On Windows, WSL2 is a real Linux box in a window. The Windows Subsystem for Linux (version 2) runs a genuine Linux kernel, so everything in this lesson works inside it — same filesystem tree, same apt, same tmux. Install a distribution from PowerShell as administrator, restart, and open Ubuntu from the Start menu. Your Windows files appear inside Linux at /mnt/c/Users/YourName/, and GPU passthrough works with the Windows NVIDIA driver installed (not the Linux one) — CUDA then becomes available inside WSL2, which is how most Windows users reach a GPU locally.

WSL2 · Linux on a Windows machinebash
# In PowerShell (admin)
wsl --install -d Ubuntu-24.04

# After restart, open Ubuntu from Start menu
sudo apt update && sudo apt upgrade -y
From the source. The Windows files at /mnt/c are reachable but slow for training (cross-filesystem I/O) — keep projects and datasets inside the Linux home directory, and treat /mnt/c as a transfer window.

The macOS-to-Linux gotchas

These are the habits that break silently when you move from a Mac to a GPU box. None of them is hard; all of them waste an afternoon if they surprise you.

The source’s macOS-to-Linux table: the macOS habit on the left, the Linux reality in the middle, the consequence underneath.
macOS habitLinux realitywhat happens
brew installsudo apt installDifferent package managers. Often the same name (htop is htop), sometimes not: brew install readline is sudo apt install libreadline-dev.
open file.txtxdg-open file.txtBut a remote box has no GUI, so both are useless over SSH — use cat for a peek, less for scrolling.
pbcopy / pbpastenot availableThere is no clipboard to pipe to over SSH: the clipboard belongs to the machine in front of you, not the server.
~/.zshrc~/.bashrcmacOS defaults to zsh; most Linux servers run bash, where interactive shells read ~/.bashrc (and ~/.profile at login).
/opt/homebrew//usr/bin/, /usr/local/bin/Homebrew's prefix on Apple Silicon versus the distribution's binary directories. which <tool> is the portable answer.
sed -i '' 's/a/b/' filesed -i 's/a/b/' fileBSD sed requires an empty string after -i; GNU sed does not. A script that crosses systems breaks on this line exactly.
case-insensitive filesystemcase-sensitive filesystemModel.py and model.py are two different files on Linux — an import written on a Mac can fail on the box.
line endings \nline endings \n — until a Windows tool writes \r\nbash reports \r: command not found when a script carries CRLF. Run dos2unix train.sh to convert it.

The macOS ↔ Linux translator

You type the mac command; the box expects something else. Pick the direction, choose the equivalent in the other system, and get the gotcha as feedback — the eight habits from the source’s macOS-to-Linux table. Facts, not a shell: no command runs here.

TASK 01 / 8 · INSTALL A PACKAGE

On macOS you have this:

brew install htop

What is the Linux equivalent?

Pick an answer — the gotcha behind this row appears either way.

direction macOS → Linux task 1 / 8 · Install a package score 0 / 0 answered correctly the eight gotchas 1. Install a package ← practising 2. Open a file with its default app 3. Copy command output to your clipboard 4. Where shell startup lives 5. Where installed binaries live 6. Edit a file in place 7. Are file names case-sensitive? 8. Line endings in a bash script
Reveal the whole macOSLinux map
The source’s macOS-to-Linux table, both directions, with the gotchas this lesson keeps coming back to.
taskmacOSLinux
Install a packagebrew install htopsudo apt install htop
Open a file with its default appopen report.pdfxdg-open report.pdf
Copy command output to your clipboardcat model.txt | pbcopy# no clipboard over SSH
Where shell startup lives~/.zshrc~/.bashrc
Where installed binaries live/opt/homebrew/bin/usr/bin and /usr/local/bin
Edit a file in placesed -i '' 's/a/b/' filesed -i 's/a/b/' file
Are file names case-sensitive?Model.py = model.pyModel.py ≠ model.py
Line endings in a bash scriptLF (\n)LF (\n) — until a Windows tool writes CRLF (\r\n)

The pattern behind the table: the box is a different machine, not a slower Mac. Package managers, startup files, path conventions and even file-name case all change — and every one of them fails with an error that sounds like your fault, not the platform’s.

The quick reference card

The source closes with a card worth pinning above the desk — or scrolling to when the box is billing and the command is on the tip of your tongue.

quick reference cardtext
Navigation:     pwd, ls, cd, find
Files:          cp, mv, rm, mkdir, cat, head, tail, less
Search:         grep, find
Permissions:    chmod, chown, sudo
Packages:       apt update, apt install
Processes:      htop, ps, kill, nvidia-smi
Services:       systemctl start/stop/restart/status
Disk:           df -h, du -sh
Network:        curl, wget, scp, rsync
Sessions:       tmux new/attach/detach
From the source. If you learn one line per row, you have the whole lesson at your fingertips — and the meter stops running while you think.
CHECK YOURSELF

Six questions.
Then the terms worth keeping.

Answer before you look. The permission question and the tmux question are the two that separate “I read the lesson” from “I can land on a GPU box at 2 a.m. and keep working”.

0 / 6 answered · 0 correct

01What does the '~' symbol represent in a Linux file path?

02What is the purpose of 'sudo' before a command?

03You get 'Permission denied' when trying to run a shell script. What command fixes this?

04On a remote GPU box, your training data fills the disk. Which command shows the largest directories consuming space?

05What is a key difference between the macOS and Linux versions of 'sed -i'?

06You start a six-hour training run in a plain SSH terminal and close your laptop. What happens, and what prevents it?

Key terms, demystified

Click a card to swap the lazy description for what it actually means.

Exercises from the lesson

Five small drills — build a project and read an ls -la, install and sort htop, prove a tmux session survives a detach, clean a full disk with the arithmetic shown, and compare scp with rsync on a transfer that drops. Try first; a worked answer is one click away.

  1. SSH into any Linux machine (or open WSL2) and navigate to your home directory. Create a project folder, create three empty files inside it with `touch`, then list them with `ls -la` — the source's exercise 1.
    Show one worked answer

    The drill is five commands, and the last one is the lesson: `cd ~` (or just `cd`) takes you home from anywhere; `pwd` proves it, printing something like `/home/ubuntu`; `mkdir -p ~/linux-drill/files` creates a project with a nested folder in one shot (`-p` makes the parents and does not complain if they exist); `touch ~/linux-drill/notes.txt ~/linux-drill/files/a.txt ~/linux-drill/files/b.txt` creates three empty files — zero bytes each, because `touch` creates the file if it is missing and updates its timestamp if it is not; and `ls -la ~/linux-drill/` prints the long listing. Read that listing rather than skimming it: the `.` and `..` entries at the top are the directory itself and its parent (that is what `cd ..` uses); `drwxr-xr-x` on the two directory lines starts with `d` where files start with `-`; every new file shows `-rw-r--r--` (mode 644) because that is what your shell's umask produces by default; and the third and fourth columns are owner and group — you, for files you created. `ls -la` is the command you will type more than any other; this exercise makes its output legible before anything goes wrong with it.

  2. Install `htop` with apt, run it, and identify which process is using the most memory — the source's exercise 2.
    Show one worked answer

    From a fresh box, the sequence is the lesson in miniature: `sudo apt update` (refresh the index — nothing installs), then `sudo apt install -y htop`. Launch it with `htop` and press `F6` (or the on-screen Sort menu) to sort by `MEM%`; the process at the top of the list is the memory user. To see the same information without an interactive tool, `ps aux --sort=-%mem | head` prints the top ten by memory. Two readings worth practising: the `RES` column is the resident set size (RAM actually held, in MB here), and `%MEM` is that as a fraction of the machine's total RAM — a 9,840 MB process on a 32 GB box reads about 30%. Cross-check the PID against `nvidia-smi` if the machine has a GPU: a process can be small in RAM and enormous in GPU memory, and `nvidia-smi` is where that shows. Quit htop with `q` (not Ctrl+C). If `htop` is missing on a box you do not control, `top` ships with the system and does the same job less prettily — but the source's point stands: install the tool you will actually use, then read its table before ever killing anything.

  3. Start a tmux session, run `sleep 300` inside it, detach, list sessions, and reattach — the source's exercise 3.
    Show one worked answer

    Type `tmux new -s drill`. The screen clears to a fresh shell with a green status bar at the bottom; the `-s drill` names the session so you can find it later. Run `sleep 300` — a stand-in for a training job; it occupies the terminal for five minutes. Now press Ctrl+B, release, then press d — the default detach key (uppercase D would open the choose-client picker instead). tmux detaches: you are back in your original shell, and the status line confirming the session is gone, but the `sleep` is still running on the server. Prove it from outside tmux: `tmux ls` prints `drill: 1 windows (created ...)`, and `ps aux | grep sleep` shows the process with a PID — a process does not know or care whether a terminal is watching it. Reattach with `tmux attach -t drill` and the sleeping shell is exactly where you left it; when `sleep` finishes and the prompt returns, type `exit` (or Ctrl+D) to close the session, and `tmux ls` will report no server running. The exercise is really about the failure it prevents: a five-hour run started in a plain SSH session dies when the laptop closes, while the same run inside tmux keeps going and reattaches with its scrollback intact.

  4. Use `df -h` to check available disk space, then `du -sh ~/.cache/*` to find what is taking up space in your cache — the source's exercise 4. Then decide what is safe to delete and compute what you get back.
    Show one worked answer

    Start at the mount level: `df -h` lists every filesystem with Size, Used, Avail, Use% and its mount point; `df -h /home` narrows to the one your work lives on. Suppose it reports 186G used, 14.2G available, 93% on a 200 GB disk — tight enough that a checkpoint write is a gamble. Now find the load: `du -sh ~/.cache/*` prints one line per entry, sorted however the shell expanded them, so pipe it through sort: `du -sh ~/.cache/* | sort -hr | head`. Typical findings are `huggingface` (model weights — 42 GB if you have pulled a few 7B models at ~14 GB each in fp16), `pip` (downloaded wheels), and sometimes `torch` (compiled kernels). Decide with a rule rather than a vibe: downloaded artifacts you can re-fetch with one command are deletable (`pip cache purge` empties the pip cache; HuggingFace models re-download on next use), while anything you generated — checkpoints, logs, processed data — needs a deliberate decision. Then do the arithmetic before and after: clearing a 2.7 GB pip cache, `sudo apt clean` for 0.6 GB of cached .deb files, and deleting two superseded 3.2 GB and 3.1 GB checkpoints returns 9.6 GB, taking the example box from 14.2 GB free to about 23.8 GB free. `df -h` again to confirm; the numbers are illustrative, the method is not.

  5. Transfer a file from your local machine to a remote one with `scp`, then again with `rsync -avz --partial --progress`, interrupting the second transfer and restarting it. Compare the two experiences — the source's exercise 5.
    Show one worked answer

    Pick a file worth moving — the lesson's 3.2 GB checkpoint is the example, but any large file shows the shape. First: `scp model.pt user@remote:/data/` copies it in one shot; if the link drops at 60%, scp starts again from byte zero, because it has no record of what already arrived. Second: `rsync -avz --partial --progress model.pt user@remote:/data/` — `-a` preserves permissions and times, `-v` prints each file, `-z` compresses in transit (a small win on an already-compressed `.pt`), `--progress` gives a live meter, and `--partial` keeps the bytes that did arrive when the transfer is interrupted. Press Ctrl+C at roughly 60% and run the same rsync again: it resumes from the remaining 40% rather than restarting, which is what resumes-on-failure actually means. The rule to keep is the lesson's: a single small file that fits in one attempt goes with scp; anything large or retry-expensive goes with rsync `--partial --progress`, because the second attempt is where the cost lives. No remote box handy? `rsync -avz --partial --progress ./big.bin /tmp/copy/` shows the same behaviour locally, and `--dry-run` lists what would move before anything does.

Terms this lesson borrows from later lessons (or outside)

You do not need to master these here. Each one gets a proper treatment in its own lesson; the one-line meaning is enough to keep reading. Orange dotted underlines in the prose point back to this list.

  • Shell, PATH and pipesThis lesson leans on pipes (`|`), redirection (`2>/dev/null`), `$PATH` and `~` expansion without teaching the shell model behind them. Phase 0, Lesson 10 (Terminal & Shell) is where that lives — and it is the reason `du ... | sort -hr | head` can be read as one sentence.
  • SSH and Remote-SSHThe lesson assumes you already connected to the box and that the terminal you type in is the remote one. Phase 0, Lesson 08 (Editor Setup) covers Remote-SSH so your editor opens a folder on the server and its integrated terminal is the remote shell — the same one tmux attaches to.
  • CUDA drivers and the GPU stackThis lesson only reads `nvidia-smi` (processes and GPU memory). Phase 0, Lesson 03 (GPU Setup & Cloud) explains what the driver reports, what CUDA is, and how a framework wheel is matched to it — including the Windows-driver requirement for WSL2 GPU passthrough mentioned in Chapter 07.
  • Docker containersA container image is a whole Linux filesystem in a file: same `/`, `/home`, `/tmp` and permission ideas, but disposable and reproducible. Phase 0, Lesson 07 (Docker for AI) builds that abstraction; this lesson's filesystem tree is what lives inside the image you will train in.
  • HuggingFace and data caches`~/.cache` filling with model weights is the recurring disk story of AI work. Phase 0, Lesson 09 (Data Management) covers datasets, caches and keeping large binaries out of your repository — the strategy layer under this lesson's `du -sh ~/.cache` diagnosis.
KEEP GOING

A picture is a start.
Practice is the rest.

This lesson is a port of an open course. Everything here traces back to it — and the next step is running the code yourself.

Original lessonLinux for AIAI Engineering from Scratch · Phase 00, Lesson 11 — the source text and its five-question quiz. The survival-guide framing, the filesystem layout, the 15 commands, the permission decoding and chmod/chown fixes, the apt section with the fresh-GPU-box one-liner, users and sudo, htop/ps/kill/nvidia-smi and systemd, the disk-space commands and space savers, wget/curl/scp/rsync, tmux, WSL2, the macOS-to-Linux gotchas table and the quick reference card are all from here.Official docsUbuntu Server documentationCanonical · the distribution this lesson's commands assume. The authoritative reference for users and sudo, `apt update` versus `apt upgrade`, installing and removing packages, systemd services, and where Ubuntu places binaries — the ground truth behind Chapter 05's apt model and Chapter 06's systemctl verbs.Official docsGNU coreutils manualThe Free Software Foundation · the reference for the file commands this lesson calls "the 15": `ls`, `cp`, `mv`, `rm`, `mkdir`, `cat`, `head`, `tail`, `chmod`, `chown`, and the flags (`-la`, `-r`, `-p`, `-f`) that make them useful. When this lesson says a flag does something, this is the manual that says so at length.Official docsWindows Subsystem for Linux documentationMicrosoft · everything behind Chapter 07's WSL2 section: `wsl --install -d Ubuntu-24.04`, the `/mnt/c` mount that exposes Windows files inside Linux, and the GPU passthrough setup that uses the Windows NVIDIA driver so CUDA becomes available inside WSL2.Official docstmux wikiThe tmux project · sessions, windows and panes from the tool's own documentation: the Ctrl+B prefix, `new -s`, detach and `attach -t`, and the split-pane keys (`%` and `"`) the key-terms card decodes. The source's rule — always run long training jobs inside tmux — is a one-line consequence of how sessions work here, and Chapter 07 keeps only that consequence plus the two session commands a rented box needs.

Lesson text and quiz are adapted from AI Engineering from Scratch (Phase 00, Lesson 11). Everything the source states is kept as-is: the survival-guide framing of a remote Ubuntu box with no GUI and idle GPU hours as money; the single tree under / with its seven directories and ~; "the 15 commands that cover 95%" and every command in the source's four groups, including rm -rf's no-undo warning; the permission string -rwxr-xr-- decoded into chmod +x / 755 / 644 and chown; apt with update first plus the fresh-GPU-box one-liner (build-essential, git, curl, wget, tmux, htop, unzip, python3-venv); users and sudo; htop, ps aux | grep python, kill vs kill -9, nvidia-smi and the systemctl verbs; df -h, du -sh, the max-depth pipeline, pip cache purge, apt clean and deleting old checkpoints; wget, curl -O, curl | python3 -m json.tool, scp and rsync -avz --progress with the reason rsync wins; tmux new/attach/detach and split panes; WSL2 with /mnt/c and Windows-driver GPU passthrough; the macOS-to-Linux gotchas table; and the quick reference card. Original to this page: the five labs (the canvas permission decoder, the SVG filesystem explorer, the apt dependency resolver, the canvas process manager and the macOS-to-Linux translator) plus the idle-cost meter; the honest count behind the "15 commands" (13 distinct verbs, 15 counting the flag-first forms it treats as separate tools); the permission arithmetic worked twice (-rwxr-xr-- = 754, -rw-rw-r-- = 664, the 600 .env habit) with the kernel's owner-then-group-then-other rule and "first matching class decides"; the apt footprint arithmetic (18 packages, ~96 MB download, ~404 MB on disk — illustrative); the disk-recovery arithmetic (3.2 GB + 3.1 GB checkpoints, 2.7 GB pip cache, 0.6 GB apt cache -> 9.6 GB back, 14.2 GB -> ~23.8 GB free — illustrative); signal exit codes 143 = 128 + 15 and 137 = 128 + 9 with the uninterruptible-I/O caveat; the file-size examples (7B fp16 model ~14 GB, HuggingFace cache ~40 GB); grep -n and find's -size rounding rule; the df-vs-du memory hook; and the sixth quiz question about tmux, written for this page from the source's "always run long training jobs inside tmux" rule. Example command outputs, versions, sizes and times are illustrative teaching estimates; the labs are simulations, not benchmarks.