A click says where.
A concept prompt says what — and returns every one.

The original SAM is a visual-prompt-only model. Its interface is a click, a box, or a rough mask: something that identifies an object by location. The image encoder turns the photo into embeddings once; a small prompt encoder and mask decoder then answer in about 50 ms per prompt in a browser, which is why clicking feels instant. What no part of that model can do is read the string "yellow school bus" and find every instance. The released weights do not take text at all.

So the field built a cascade. A text-conditioned open-vocabulary detector — Grounding DINO and its descendants — turns the phrase into boxes. SAM turns each box into a mask. Wrap them together and you get Grounded SAM, and after 2024 the video-aware Grounded SAM 2. It works, it ships, and it has a structural cost: two frozen models, one forward pass per prompt for the detector, one mask-decoder pass per proposed box, and two ways to be wrong that multiply instead of cancelling.

SAM 3 (Meta, November 2025) collapsed the cascade. Its task is Promptable Concept Segmentation: give it a short noun phrase — or an image exemplar — and it returns masks and unique instance IDs for every matching object in one forward pass. Not “segment this thing I pointed at” but “segment this concept”. The paper’s own framing is that the model has an image-level detector and a memory-based video tracker sharing a single backbone, with a presence head that decides whether the concept is even there.

Three differences matter when you write code against SAM 3. First, no per-instance prompting: one phrase returns all matches, so you stop looping over objects. Second, open vocabulary: the concept can be anything describable in a short noun phrase, not a fixed label set. Third, many instances at once: the output is a set of masks with distinct IDs, not one mask per call. The interface you build around those three facts is a different shape from a detector loop.

Keep the cascade’s trade-off in view, though, because it never disappears: SAM 3 is monolithic. You cannot swap its detector for a domain-specific one, you cannot tune the detector threshold the way you can when the detector is a separate package, and the released weights are gated on Hugging Face under a custom license. Grounded SAM 2 remains the modular option — and the lesson comes back to that when the choices get real.

Rung one: SAM (April 2023). A ViT encodes the image once — ViT-H is a 636M-parameter image encoder inside a 641.1M-parameter checkpoint whose fp32 weights are 2.56 GB, the file people quote as “2.4 GB” because 2.56 GB is 2.38 GiB. Then the prompt encoder and mask decoder answer each click in ~50 ms in a browser. A single click is ambiguous (the wheel of a bike, or the whole bike?), so SAM returns up to three masks with predicted IoU scores and lets a follow-up click disambiguate. The interface is a gesture, and the gesture cannot name anything.

Rung two: Grounded SAM 2 (2024). A text-conditioned detector turns a phrase into boxes, SAM 2 turns each box into a mask and — this is the 2024 addition — tracks it through video as a masklet, using a streaming memory that attends to previous frames. Promptable visual segmentation (PVS) lets you prompt on any frame and refine later. The cost structure is the cascade’s: one detector pass per prompt string, one image encode, and one mask decoder pass per box. Twelve instances of a concept are twelve decodes in rung two and a single concept pass in rung three.

Rung three: SAM 3 (November 2025). The phrase goes straight into the model. Promptable Concept Segmentation returns every matching instance with a mask and an instance ID, plus a presence score for the concept as a whole. The paper describes one shared backbone feeding an image-level detector and a memory-based video tracker, with a presence head decoupling “is it here?” from “where is it?”. Visual prompts still work; text and exemplars are added, not substituted.

Watch the comparator cycle: the same five-object scene answers three different questions. A point returns one instance because a click can only mean “the thing I touched”. Text through a detector returns one mask per proposed box — the detector’s recall is the ceiling, and every box pays a decode. Text asked directly returns every match of the concept in one pass, and it can also answer nothing when the concept is absent, which the cascade cannot say cleanly.

The official paper describes SAM 3 as an image-level detector and a memory-based video tracker that share a single backbone. The blog adds the ancestry: the vision and text encoders come from Meta’s Perception Encoder, the detector is DETR-based, and the tracker builds on SAM 2’s memory bank and memory encoder. Four jobs sit on that one encode:

• Recognise — the text (or exemplar) encoder turns the concept into a query, and the detector scores image regions against it.
• Localise — the same detector regresses the boxes and instance queries that the mask decoder turns into masks.
• Decide presence — a separate scalar head answers “does this concept exist in this image?” before anything is localised. Meta’s own summary says recognition and localisation are decoupled with a presence head, which boosts detection accuracy.
• Track — on video, per-instance memory carries identity across frames, and an occlusion head marks frames where the instance is not visible instead of deleting it.

The decoupling is not cosmetic. Re-detecting an instance every frame wants features that are invariant across viewpoints; keeping two similar instances apart wants features that are discriminative. Those pull in opposite directions, which is why the detector and the tracker are separate heads over shared features rather than one head doing both. The same instinct is why the presence head exists: a detector forced to answer every query will rank something highest even when the concept is absent, and a scalar veto fixes that cleanly.

SAM 2 (2024) generalised promptable segmentation to video with a streaming architecture: frames are consumed one at a time, and the mask decoder’s input is not the raw image embedding but an embedding conditioned on memories of past frames. The memory encoder downsamples the predicted mask, fuses it with the frame’s image embedding, and stores the result in a memory bank: a FIFO queue of up to six recent frames plus a second queue of prompted frames. Cross-attention reads that bank, and a per-object object pointer (a 256-dimensional vector split into four 64-dimensional tokens) carries the high-level “which object is this” signal.

The second new piece is the occlusion head — SAM 2’s answer to frames where the object is genuinely not visible. SAM always assumed there was a valid object given a positive prompt; in video that is false, and a tracker that has to emit a mask on every frame will hallucinate one behind the building. The occlusion head adds a learned embedding to the memory features of hidden frames and produces a visibility score, so the model can say “not visible” while keeping the object alive in memory. That is the difference the stepper lab is built to show.

SAM 3 keeps the architecture and changes the economics. The tracker is still memory-based, reading the same shared per-frame features; what used to be expensive was that every object ran its own memory bank and decoder, so cost grew linearly with the number of instances. The March 2026 SAM 3.1 update, Object Multiplex, replaces the per-instance banks with one shared memory plus per-instance queries: up to 16 objects in a single forward pass, and for a medium number of objects, throughput on one H100 doubles from 16 to 32 FPS. For crowds and dense scenes that is the difference between tractable and not — and it is where this lesson hands off to Lesson 27: the classical trackers (SORT and ByteTrack) solve the same problem by detecting every frame and associating boxes, and this memory-based family is what took over when identity, not boxes, became the product.

One honesty note before you deploy it: the transformers video configuration documents a hotstart window of 15 frames that removes unmatched and duplicate tracks using future frames — and streaming inference, by definition, has no future. The docs say streaming “may result in more false positive detections and duplicate object tracks”, so use pre-loaded inference when the clip is available and accept the trade-off only when it is not.

The first job is turning “what the user typed” into “what the model consumes”. SAM 3 takes one concept per forward pass, so a sentence with conjunctions has to be split into short noun phrases — heuristic, imperfect, and worth doing explicitly instead of letting a model guess. The source’s splitter normalises the common separators and returns a list.

The second job is the output contract. SAM 3, Grounded SAM 2 and YOLO-World all return boxes, labels and scores; the segmentation models add masks, and SAM 3 adds unique instance IDs. Your pipeline should not care which backend ran, so the source defines one dataclass and encodes masks as run-length runs — the same format SAM 2, SAM 3 and COCO use, and the reason a mask can travel through JSON at all.

That 53× is the whole reason RLE is on the wire, and it comes with a warning worth internalising: the win is a property of the image, not of the format. Encode a checkerboard and every pixel alternates, so runs are one pixel long and each one costs about four characters — the “compressed” payload is four times the raw bitmap. Mask formats that also accept polygons and PNG exist for exactly this reason: pick the encoding that matches your shapes.

Behind the contract sits a real model call. Hugging Face’s transformers integration exposes Sam3Processor and Sam3Model for images, and — this is the part that makes the pass-count arithmetic real — get_vision_features and get_text_features so the vision encode is paid once per image and reused across concepts.

Video uses the detector and tracker as one object: Sam3VideoModel wraps a detector (SAM 3) and a SAM 2-style tracker, and you drive it through an inference session. Text prompts are added once; the tracker then propagates the masklets frame by frame, and each frame’s output maps a prompt to its object IDs. Multiple prompts are processed in a single pass with shared vision features, which is the multiplexing idea already visible at the API level.

The last piece of the interface is a fallback. When the real model is not loaded — in unit tests, in CI, on the machine of the person reviewing your PR — you still want the pipeline to run. The source wraps every backend behind one abstract class and ships a StubOpenVocabSeg that returns deterministic detections for a given image size. Your downstream code never learns whether it is talking to SAM 3, Grounded SAM 2, or a stub; that is the point.

Axis one: the output. Boxes-only at high frame rate is a different problem from masks. YOLO-World is the detection-only tool — the paper reports 35.4 AP zero-shot on LVIS at 52.0 FPS on a V100 at 640², which is 19.2 ms per frame — and it has no mask decoder to pay for. If your dashboard counts objects and draws rectangles, adding masks is pure cost. If a downstream step needs silhouettes — area, shape, touching-instance separation — you need a segmentation model and the comparison starts at rung three.

Axis two: concept complexity. Common single-word concepts are solved roughly equally by everything; the differences appear on rare and compositional phrases — “striped red umbrella”, “person holding a mug” — and on concepts that are absent from the image, where the presence head is the structural advantage. But the axis has a far end: Meta documents SAM 3 as weak on fine-grained out-of-domain concepts even zero-shot, naming terms that need domain knowledge such as “platelet” in medical imagery. When your concepts are like that, no stack choice saves you; the answer is fine-tuning on in-domain annotations, and the release ships fine-tuning code for exactly this reason.

Axis three: the deployment. SAM 3 is monolithic — one model, one license, one threshold behaviour. Grounded SAM 2 is a composition you can rewire: swap in DINO-X, Florence-2, or Grounding DINO 1.5; tune the detector threshold yourself; keep research ablations valid while you hold SAM fixed. Add the practical constraints — the released SAM 3 weights are gated on Hugging Face under a custom license, and the model is ≈860M parameters / 3.44 GB in fp32 — and modularity stops looking like nostalgia.

When you migrate from Grounded SAM 2 to SAM 3, the source’s advice is to audit three things instead of assuming a clean upgrade. Latency is usually net-neutral or a slight win — you delete a detector pass, but the surviving model is heavier — so measure the end-to-end pipeline, not the model card. Accuracy shifts by concept: rare and compositional phrases gain the most, common single-word concepts look similar, and your own long tail is the only sample that matters. Flexibility is a real loss: you can no longer choose the detector, so if a domain-specific detector was carrying your accuracy, do that comparison before you swap. The source packages both decisions as two artifacts — a stack-picker prompt and a concept-prompt designer skill that turns user utterances into well-formed prompts with disambiguation and fallbacks — and this lesson’s chooser and splitter are the same two shapes.

When you do not want to write the session plumbing yourself, Ultralytics wraps the weights behind the interface it already uses for YOLO and SAM 2 — same call shape, different model name. That is a perfectly good production choice for straightforward image work; the lesson spells out the transformers path because it is the one you extend.

Start with what open vocabulary can and cannot mean. A concept prompt can be any short noun phrase, but the model’s ability to ground it comes from training data. Meta’s own limitations section says SAM 3 struggles to generalise to fine-grained out-of-domain concepts zero-shot — the example given is “platelet” in medical imagery — and that it does not support longer, complex phrases like “the second to last book from the right on the top shelf”. That second limitation is architectural: PCS takes short phrases, so reasoning-heavy queries are handled by a multimodal model that proposes noun phrases and inspects the masks, iterating until satisfied. Meta calls that pattern SAM 3 Agent, and it is a useful mental model for your own product: the language model does the reasoning, the segmentation model does the pixels.

The second honesty problem is per-concept ambiguity. “A player in white” and “a player in red” are two concepts and two sets of instances; evaluating them together as “player” hides exactly the errors that matter. This is why the presence head helps — it sharpens closely related prompts — and why SA-Co contains hard negatives (concepts deliberately not present) alongside positives. If your evaluation averages over a whole vocabulary, the number will look fine while the concepts you care about are broken. Audit per concept, and include absent concepts: a false positive on “hard hat” in a safety system is not noise, it is a wrong alert.

The third is simply dating. “SAM 3”, “SAM 3.1” and “SAM 2.1” are different checkpoints with different numbers; the mask-decoder timings in the 2023 paper were taken in a browser; the frame rates in the SAM 2 release were measured on an A100 at 1024 px; the SAM 3 figures come from an H200 and an H100. A production decision that quotes any of them without the hardware and version is a decision with a hidden variable. This page checked its numbers on 2026-09-16 and says so everywhere it repeats them.

Finally, the boring constraint that ends more projects than accuracy: licensing and access. The facebook/sam3 checkpoint on Hugging Face is gated (manual approval) under a custom license — verified on this page’s check date — while SAM 2’s checkpoints and training code are Apache 2.0 and YOLO-World’s repo is public. If your legal team needs a permissive licence, Grounded SAM 2 is the rung that ships today; if it needs the best open-vocabulary masks and can accept the terms, SAM 3 is the default. Neither answer is a formula — it is a decision with a date on it.