The terminal is
where the work runs.

The word “terminal” gets used for two programs, and the distinction pays off immediately. The terminal emulator is the window — it draws text, handles the scrollbar, and forwards keystrokes. The shell is the program running inside it: bash, zsh, fish. The shell reads what you type, decides what it means, and starts programs. One line checks which one you have:

$ echo $SHELL
/bin/zsh

# illustrative: on a Linux box the same command usually prints /bin/bash.
# Both shells run every command in this course.

Here is the honest scale of it: you will type more commands than you write lines of Python on some days. Four jobs live in the shell, and the rest of this lesson is one chapter per job.

Do the arithmetic on that terminal window. A training loop prints one line per optimiser step. The source’s example pace is modest — three steps a second — so one hour of training prints 3 × 3,600 = 10,800 lines, and a six-hour run prints 6 × 10,800 = 64,800 lines. The editor lesson configures a 10,000-line scrollback, which at three lines a second holds 10,000 ÷ 3 ÷ 60 ≈ 56 minutes of history. If the loss went bad at step 8,400 — forty-seven minutes in — it is still on screen at breakfast only because the process stopped printing; a run that keeps going buries it within the hour.

Nothing here is a warning against the terminal. It is an instrument panel: the run, the hardware and the evidence, all in one window. The skill is knowing which part of the panel remembers things — and that is what the labs below let you feel.

The shell answers two questions constantly: where am I? and what is here? Three commands handle both, and they are the ones you will type thousands of times:

• cd — change directory. cd ~/projects/ai-engineering-from-scratch moves you there; cd .. goes up one level; cd alone goes home. ~ is shorthand for your home directory.
• pwd — print working directory. It prints the absolute path of where you are, which is the answer to “why did that command not find my file?” more often than anything else.
• ls — list. ls -la adds two flags: -l for the long format (permissions, owner, size, date) and -a for all entries, including the dotfiles like .venv and .gitignore that the plain version hides. In an AI project that hidden .venv is usually the thing you were looking for.

Of those four, Ctrl+R is the one that pays for itself daily. The commands worth repeating in AI work are the long, exact ones — python train.py --epochs 10 --lr 1e-4 --batch-size 64 is forty-odd keystrokes to retype and one Ctrl+R plus train to recall. The search is incremental: every keystroke narrows the fragment and jumps to the most recent match; pressing Ctrl+R again without typing walks one match older. Enter runs the line, and the arrow keys drop it at the prompt so you can edit it.

Ctrl+C and Ctrl+Z are different animals. Ctrl+C sends SIGINT — interrupt — and most programs, including Python, exit. Ctrl+Z sends SIGTSTP — terminal stop — which suspends the process: it stops using the central processing unit (CPU), keeps every byte of memory it had, and waits. fg brings it back to the foreground; bg lets it continue in the background; kill %1 is how you actually end it.

Every process starts with three standard streams: standard input (file descriptor 0), standard output (1, the normal results) and standard error (2, warnings and tracebacks). By default all three point at your terminal, which is why output and errors look identical on screen.

The shell’s superpower is that it can re-point those descriptors before a program starts. The pipe operator | points the left command’s standard output at the right command’s standard input — and the two run at the same time, with the kernel moving bytes between two file descriptors. It is not a temporary file, and the two programs never need to agree on anything beyond “lines of text”. Here are the recipes the source gives, with the shape of their output:

Read the recipes one at a time; each one is a single idea.

• Count. wc -l counts the lines it receives, so the pipeline’s answer is one number: 12,480 step lines in the illustrative run. Note the redundant cat — it is in the source, and grep "loss" train.log | wc -l is equivalent and starts one fewer process. Copy whichever you will remember.
• Extract. awk’s $NF is the last whitespace-separated field of each line, so '{print $NF}' turns epoch 1/10 · step 100 · loss 1.8821 into 1.8821. Twelve thousand four hundred eighty of those, at roughly seven bytes each, is a losses.txt of about 85 KiB — small enough to plot, mail or commit.
• Watch. tail -f follows a file as it grows, and grep keeps the lines you asked for. The flag matters: when grep’s output is a pipe rather than a terminal, it block-buffers — it collects output in a buffer of several KiB before flushing (GNU grep uses the C library’s BUFSIZ, commonly 8 KiB). At ≈ 90 bytes per log line that is on the order of a hundred lines of silence, so a rare ERROR can sit invisible while the buffer waits to fill. --line-buffered makes grep flush every line as it matches. (An estimate, not a specification — the buffer size is a GNU grep implementation detail.)
• Sort. sort -t= -k2 -n -r is four flags doing one job: -t= says fields are separated by =, -k2 sorts on the second field, -n compares them as numbers (not text, which would put 0.9 above 0.88), and -r reverses to descending. Because grep searched several files, each line carries its filename — which is exactly what you want when the top line names the winning experiment.
• Split. > output.log 2> errors.log keeps the megabytes of normal output away from the few kilobytes of errors, so reading the failures is a one-second command instead of a search.
• Merge. > train_full.log 2>&1 puts both streams in one file, in the order they happened — the format you want when a traceback needs the line that caused it.

Three more patterns you will meet constantly. The first is tee, which writes a stream in two directions at once — into a file and onwards — so you can watch a run and keep the record. The source’s version even pings you when it is done:

A trailing & tells the shell “start this program, then give me my prompt back”. The process becomes a background job of that shell: it keeps printing to the terminal (which gets messy), and the shell keeps a handle on it — jobs lists it, fg %1 brings it back to the foreground, and kill %1 ends it. All of that machinery lives inside that one shell, which is the detail that decides everything that follows.

What happens when the terminal closes? The window is the terminal emulator; when it exits, the kernel closes the controlling terminal and sends SIGHUP — signal 1, “hang up”, a name inherited from telephone modems — to the processes in that terminal’s session. The default action for SIGHUP is to terminate.

nohup changes exactly one thing: the process starts with SIGHUP ignored. It does not detach the process from your terminal in any other way, and it does not give you a way back in — that is what tmux (next chapter) is for. One documented detail worth knowing before it surprises you: when a nohup’d command’s output still points at a terminal, nohup quietly appends it to a file called nohup.out in the current directory (or in $HOME if that is not writable). That is why the source writes > train.log 2>&1 explicitly: it makes the destination a decision instead of a surprise.

What a hangup costs, roughly. Suppose the run is six hours long and the machine costs an example $2.34 per GPU-hour. If SIGHUP lands at hour five, the money spent is about 5 × 2.34 ≈ $11.70 — and every hour since the last checkpoint is work you will pay for twice. If checkpoints land hourly, the loss is about an hour of compute; if the script only saves at the end, it is all of it. That is the arithmetic behind the lesson’s rule: anything longer than a few minutes goes in tmux. (Rates and durations here are illustrative; your provider’s invoice is the real one.)

Two closing details for the toolbox. kill sends SIGTERM by default — a polite request that a well-written program can catch, clean up on, and honour; kill -9 sends SIGKILL, which cannot be caught and exists for the case where SIGTERM was ignored. And if you started a job in the background of a shell you want to close anyway, disown removes the job from the shell’s table so it will not be hung up with it — a bash/zsh builtin, and the third escape hatch the source does not mention because tmux makes it obsolete.

The mental model has three parts. A server process owns pseudo-terminals and runs independently of any window. A session is a named collection of those terminals inside the server. A client is your window attaching to the server — a view, nothing more. That split is why closing your terminal, losing Wi-Fi or moving to another machine changes nothing about the run: the server never depended on the client.

Every command is prefixed with one chord, Ctrl+B by default: you press it, release, then press the command key. It feels clumsy for about a day and then becomes muscle memory.

The source’s three-pane workflow is the reason to learn all of it. A training run occupies one pane and prints for hours; the hardware wants a second pane with watch -n1 nvidia-smi; the evidence wants a third with tail -f logs/experiment.log. One window, three live views, and — the important part — you can detach from all three, go home, SSH back in from another machine, and find them exactly where you left them.

Detach is not stop, and close is not detach. The simulator above exists to make that difference visible: pressing Ctrl+B d closes your view while every process keeps running and every pane keeps its scrollback. Killing the session — tmux kill-session -t training — does the opposite: the panes are destroyed and their processes receive SIGHUP, exactly as if you had closed the window on a background job. And tmux keeps its own scrollback per pane, with a documented default history-limit of 2,000 lines: the same lesson as the terminal’s 10,000-line buffer, one layer down. (Need more? set -g history-limit 50000 before creating panes; need to see one pane full-window? Ctrl+B z zooms it, and Ctrl+B x closes it.)

A small trap to file away: screen — the older tool with the same idea — uses Ctrl+A as its prefix, so advice written for screen will look wrong in tmux. Same concept, different chord; when a keybinding from a tutorial does nothing, check which multiplexer it was written for.

Start with htop, the source’s preferred version of top: a live list of processes with colour-coded central processing unit (CPU) and memory meters, and — the part that makes it a tool rather than a display — keys for finding, sorting and killing. On a GPU box those processes include the python running your training, and the reason htop matters is that “the machine is slow” usually turns out to be a specific number about a specific process.

Then nvidia-smi — the NVIDIA System Management Interface — for the card itself, and nvtop, which is htop’s idea applied to GPUs: per-process utilisation and memory, updating live. Two numbers carry almost all the information you need: utilisation (what fraction of the last sample window the GPU spent executing kernels) and memory used out of total.

Now the shortcuts. The commands above are long, exact and repeated — the ideal candidates for aliases. The lesson ships a file of them; the four the source highlights are these:

Disk space is a training problem too. Datasets and checkpoints fill a disk without asking, and a full disk turns a working run into a crash whose message mentions everything except space. The two commands to reach for: df -h for a whole-filesystem view in human units, and du -sh ./data/* to find which directory inside the project is the culprit. When you know it is models, the source’s find . -name "*.pt" -o -name "*.safetensors" | xargs du -h | sort -rh | head -20 ranks the biggest files first — -size +100M narrows the search if the list is long.

And before blaming your code for a CUDA error, ask the environment: env | grep -i cuda and env | grep -i torch print the environment variables that decide which CUDA libraries are even visible — CUDA_VISIBLE_DEVICES is the one that silently hides a GPU from a process.

SSH stands for Secure Shell: an encrypted protocol for running commands on a remote machine. ssh user@host opens an interactive shell there; everything you learned in the last six chapters applies to it unchanged, because it is a shell — just one whose window is a network connection. A private key replaces the password, and -i names it explicitly:

scp or rsync? scp copies what you point it at, every time — perfect for a single small file you will move in one attempt. rsync -avz compares the two sides first and sends only what differs, so it is the tool for anything large or retry-expensive — a dataset, a results folder, a checkpoint. The three flags: -a archive mode (recurse, preserve permissions and times), -v verbose (print what is happening), -z compress on the wire — a real win for text and a small one for already-compressed files like .npz. Add --partial and a dropped transfer resumes instead of starting over.

Price it once, and the habit sticks. An illustrative dataset directory: 240 files × 12 MB = 2,880 MB. Send it the first time and both tools move all 2,880 MB — rsync has nothing to compare against. Edit three files and send it again: scp -r moves 2,880 MB again, while rsync -avz moves 3 × 12 = 36 MB. That is 80× less traffic, and at an example 10 MB/s link it is the difference between about 4 min 48 s and about 3.6 seconds. The lab below lets you change the number of changed files and rerun both. (Times and sizes are teaching estimates on a constant link; the comparison is the point, not the stopwatch.)

Port forwarding is how remote tools become local ones. A Jupyter server started on the box with --no-browser is deliberately unreachable from the internet; ssh -L 8888:localhost:8888 user@gpu-box-ip opens port 8888 on your machine and carries the traffic, encrypted, to a service listening on the box’s own loopback. Read the three parts as local-port : remote-host : remote-port — and note that localhost in the middle is resolved on the remote side, which is why the same command works for any service bound to the box’s loopback. TensorBoard’s default port, 6006, is the other one you will forward this way. By default the local end of -L binds to 127.0.0.1, so only your machine can use the tunnel; that is a feature, and it is why you should not add the option that exposes it unless you mean to share.

Getting data onto the box is usually three commands: download the archive on the remote (or locally), unpack it, and check the disk before and after. wget fetches a URL — wget https://huggingface.co/model/resolve/main/model.safetensors is the source’s model example — and tar xzf dataset.tar.gz -C ./data/ unpacks a gzipped tarball into place. The flags decode as x extract, z gzip, f use this file, -C change to this directory first. Then df -h and du -sh ./data/* confirm that a 200 GB dataset did not fill the disk you need for checkpoints.

One honest update to the source. Since OpenSSH 9.0, scp is implemented on top of the SFTP protocol (SSH File Transfer Protocol) rather than the old SCP protocol — same command, same flags for the common cases, a different engine underneath. Nothing in this lesson changes because of it; the practical consequence is that sftp and rsync-over-SSH are the forward-looking ways to move files, while scp stays as the quick answer it has always been.