ML Lecture — ImageLab

Overview · ~5 min The Pipeline and Where ML Fits

ImageLab processes each photograph through five independent signals. Four of them are deterministic calculations — pixel arithmetic that returns the same answer every time. The fifth is a language model call. The ML libraries sit in the middle of the stack, between fast pixel operations and the expensive language model:

folder ──► ingest ──► prepare ──► [ hash ] ──► [ embed ] ──► [ features ] ──► [ describe ] ──► SQLite │ │ │ │ SHA-256 CLIP YOLO + OCR + Claude dHash (torch + colour + (non-det.) (PIL) open-clip) blur, edges det. repr.inf. (EasyOCR, ultralytics)

The five signals at a glance

Signal	Library	Question answered	Deterministic?
SHA-256 / dHash	`pillow`	Have I seen this exact file before? Is it a near-duplicate?	Yes
CLIP embedding	`torch` + `open-clip`	Is this the same subject from a different angle?	Reproducible
OCR text	`easyocr`	What text is visible in this image?	Yes
Object classes	`ultralytics`	What categories of object appear?	Yes
Description	`anthropic`	What is it, qualitatively, and is it worth keeping?	No

The layering is intentional: each signal is more expensive than the last, so expensive checks only run on what survives cheaper ones. SHA-256 is free. dHash is ~15 lines of PIL. Embeddings and features are local model inference — no API cost. Descriptions cost money per call, and are cached so changing ranking logic costs nothing.

Module 1 · ~12 min PyTorch — the Computation Framework

What it is

PyTorch is an open-source numerical computing library built around tensors — multidimensional arrays — with two defining properties: automatic differentiation (autograd) and first-class GPU/Apple Silicon support. Most deep learning models in production today are built in PyTorch.

ImageLab does not train anything. It uses PyTorch purely as a runtime: loading pre-trained weights and running forward passes. This is the simplest possible use of the library — but understanding what is happening helps interpret both correctness and performance.

Core concept: tensors and the computation graph

A tensor is a generalisation of matrices to arbitrary dimensions. A grayscale image is a 2D tensor (height × width). An RGB image is 3D (channels × height × width). A batch of 8 RGB images is 4D (batch × channels × height × width).

During training, every operation on a tensor records itself in a computation graph so that gradients can be backpropagated. This is expensive memory-wise. During inference we do not need gradients at all — so ImageLab disables the graph entirely with torch.no_grad(), making inference faster and lighter.

Device selection

PyTorch can execute on CPU or GPU. On Apple Silicon Macs, the unified memory architecture is exposed as the mps (Metal Performance Shaders) backend. ImageLab prefers MPS when available, falling back to CPU:

# from imagelab/embedding.py def select_device() -> str: import torch if torch.backends.mps.is_available(): return "mps" return "cpu"

Lazy model loading

The CLIP weights are ~600 MB. Loading them on every request would be unacceptable. ImageLab loads them exactly once, the first time embed_image() is called, then holds the model in a module-level variable protected by a threading lock:

# from imagelab/embedding.py _model: Any = None _preprocess: Any = None _device: str | None = None _lock = threading.Lock() def _load() -> tuple[Any, Any, str]: global _model, _preprocess, _device with _lock: if _model is None: import open_clip import torch device = select_device() model, _, preprocess = open_clip.create_model_and_transforms( config.EMBEDDING_MODEL_NAME, pretrained=config.EMBEDDING_PRETRAINED, cache_dir=str(weights_dir), ) model.eval() # switch BatchNorm / Dropout to inference mode model.to(device) # move weights to MPS or CPU torch.set_grad_enabled(False) _model, _preprocess, _device = model, preprocess, device return _model, _preprocess, _device

Three points worth noting: model.eval() changes the behaviour of layers like BatchNormalisation and Dropout — they work differently during training vs inference. model.to(device) moves the entire parameter set to the chosen hardware. torch.set_grad_enabled(False) is a global switch that tells PyTorch not to build the computation graph at all — more aggressive than torch.no_grad() (which is a context manager).

The forward pass

A forward pass is the computation of the model's output given an input. No learning occurs. The input image is preprocessed into a normalised tensor (the preprocess transform handles resize, centre-crop, and channel normalisation), passed through the model, and the resulting feature vector is extracted:

# from imagelab/embedding.py def embed_image(image: Image.Image) -> np.ndarray: import torch model, preprocess, device = _load() # preprocess: resize → centre crop → tensor → normalise channels tensor = preprocess(image.convert("RGB")).unsqueeze(0).to(device) # ^ add batch dimension (1, C, H, W) with torch.no_grad(): features = model.encode_image(tensor) # features: (1, 512) tensor on device → move to CPU → numpy float32 vector = features.detach().to("cpu").numpy().astype(np.float32).reshape(-1) return l2_normalise(vector)

Input tensor shape

(1, 3, 224, 224)

batch × RGB channels × height × width

Output vector

512 dims

L2-normalised float32

Device

MPS / CPU

no GPU required

Gradient graph

disabled

set_grad_enabled(False)

Module 2 · ~12 min open-clip — Contrastive Image–Language Pretraining

What CLIP is

CLIP (Contrastive Language–Image Pretraining) is a model published by OpenAI in 2021, trained on 400 million (image, text) pairs scraped from the internet. The key idea is contrastive learning: given a batch of N image–text pairs, train the model so that the embedding of each image is close to its paired text caption and far from the other N-1 captions in the batch.

open_clip_torch is an open-source reimplementation with weights trained on the LAION-2B dataset (two billion English image–text pairs). ImageLab uses the ViT-B/32 variant trained on the laion2b_s34b_b79k checkpoint.

Architecture: two encoders, one shared space

IMAGE ──► Vision Transformer (ViT-B/32) ──► 512-dim vector │ cosine similarity │ TEXT ──► Transformer text encoder ──► 512-dim vector

The vision encoder is a Vision Transformer. The input image is divided into 32×32 pixel patches (hence "B/32"), each patch is projected to a token embedding, and the sequence of patch tokens is processed by a standard transformer. The output is a single 512-dimensional vector representing the image.

The text encoder takes a tokenised string and produces another 512-dimensional vector. Training pushes the image and text vectors for matching pairs close together in this shared space. After training, images of similar concepts end up near each other, even if they look different.

Why L2-normalise?

Cosine similarity measures the angle between two vectors: cos(θ) = a·b / (|a| |b|). If both vectors are L2-normalised to unit length (|a| = |b| = 1), the denominator is always 1, and cosine similarity reduces to a plain dot product: a·b. This makes ranking 250 vectors a single matrix multiply — microseconds, no vector database required.

# from imagelab/embedding.py def l2_normalise(vector: np.ndarray) -> np.ndarray: norm = float(np.linalg.norm(vector)) if norm == 0.0: return vector.astype(np.float32) return (vector / norm).astype(np.float32) def cosine(left: np.ndarray, right: np.ndarray) -> float: """Plain dot product: both vectors are L2-normalised at write time.""" return float(np.dot(left, right))

Text queries and cross-modal search

Because the image and text encoders share a vector space, you can search an image library with a text query. The text is embedded into the same 512-dimensional space, then ranked against every stored image vector by cosine similarity — no fine-tuning required. ImageLab implements this via embed_text():

# from imagelab/embedding.py def embed_text(query: str) -> np.ndarray: """Encode a text query with the CLIP text tower, L2-normalised.""" import open_clip import torch model, _preprocess, device = _load() tokens = open_clip.tokenize([query]).to(device) with torch.no_grad(): features = model.encode_text(tokens) vector = features.detach().to("cpu").numpy().astype(np.float32).reshape(-1) return l2_normalise(vector)

This is how the Search page's text input works: the query string goes through the text tower, producing a 512-dim vector, which is then dot-producted against every stored image embedding. The results are images whose visual content aligns with the query concept.

Deep dive: what "ViT-B/32" means

The model name encodes the architecture. ViT = Vision Transformer. B = Base size (86M parameters — there are also Small, Large, and Huge variants). /32 = patch size of 32×32 pixels.

A 224×224 input image divided into 32×32 patches gives (224/32)² = 49 patch tokens. A learnable [CLS] token is prepended, making the sequence length 50. The transformer operates over these 50 tokens; the final [CLS] output is the 512-dim image representation.

A larger patch size (ViT-B/16, ViT-L/14) means finer-grained patch tokens and better accuracy, at the cost of more compute. ViT-B/32 is the fast variant — appropriate for a local tool running on CPU or MPS over a library of a few hundred images.

Module 3 · ~10 min EasyOCR — Reading Text in Images

What OCR is

Optical Character Recognition (OCR) converts pixel data containing text into a machine-readable string. Classical OCR (Tesseract) required clean, printed text in a uniform font. Modern neural OCR handles handwriting, stamps, logos, curved text, noise, and arbitrary orientations.

EasyOCR is a Python library that wraps a two-stage deep learning pipeline. It returns a list of detected text strings without requiring any configuration. In ImageLab it is used to extract any readable text from a photograph — part numbers stamped on hardware, labels, markings — and store the result as a searchable string.

Architecture: CRAFT + CRNN

Stage 1 — Text Detection (CRAFT) image ──► VGG-16 backbone ──► region score map + affinity map │ detected text bounding polygons Stage 2 — Text Recognition (CRNN) cropped region ──► CNN (ResNet) ──► feature maps │ BiLSTM sequence model │ CTC decoder ──► "PART-7734-A"

Stage 1 uses CRAFT (Character Region Awareness for Text Detection) — a VGG-16 backbone trained to output a heatmap of character regions and another heatmap of character affinities (which characters belong to the same word). Connected-component analysis on these maps produces bounding polygons around text regions.

Stage 2 crops each detected region and passes it to a CRNN: a ResNet feature extractor produces a sequence of column features, a bidirectional LSTM reads the sequence in both directions and models character dependencies, and a CTC (Connectionist Temporal Classification) decoder converts the output probability sequence into a final string without needing character-level alignment labels during training.

How ImageLab uses it

The reader is loaded once, lazily, on the first call, using the same pattern as the CLIP model. The GPU is disabled (gpu=False) because the model is small enough that CPU inference is fast, and it avoids requiring CUDA:

# from imagelab/features.py _ocr_reader = None _ocr_lock = threading.Lock() def _get_ocr_reader(): global _ocr_reader with _ocr_lock: if _ocr_reader is None: import easyocr _ocr_reader = easyocr.Reader(["en"], gpu=False, verbose=False) return _ocr_reader def ocr_text(image: Image.Image) -> str: """Extract all readable text via EasyOCR.""" reader = _get_ocr_reader() rgb = np.array(image.convert("RGB")) results = reader.readtext(rgb, detail=0) return " ".join(results).strip()

readtext(rgb, detail=0) returns only the text strings, not the bounding boxes or confidence scores. The results are joined into a single space-separated string stored in the features table as ocr_text. This string is indexed for full-text search: if an image contains the text "PART-7734-A", a query for that string will find it directly without an LLM call.

Detection model

CRAFT

VGG-16 backbone

Recognition model

CRNN

ResNet + BiLSTM + CTC

GPU

disabled

cpu-only inference

Output

plain string

stored as ocr_text column

Deep dive: what CTC decoding is

CTC (Connectionist Temporal Classification) solves a specific problem: the CRNN outputs one probability distribution over characters at each time step (each column of the feature map), but we do not know how many time steps each character spans. CTC introduces a special blank token and a rule that collapses repeated characters and blanks into a single character. This lets the model be trained end-to-end on (image, string) pairs without needing per-character position labels.

The beam search decoder finds the most probable string by exploring multiple character hypotheses at each time step and pruning low-probability paths. For short strings on clean images (part numbers, labels) this is fast and reliable.

Module 4 · ~10 min Ultralytics / YOLOv8 — Object Detection

What object detection is

Object detection answers: what categories of object are present and where are they (bounding box)? It is distinct from image classification (what is the single most likely label?) and from segmentation (which pixels belong to each object). Detection is the right tool for asking "does this photo contain a person, a car, a backpack?"

YOLO: single-pass detection

YOLO (You Only Look Once) changed object detection in 2016 by framing it as a single regression problem. Earlier two-stage detectors (R-CNN, Faster R-CNN) first proposed candidate regions, then classified each region — accurate but slow. YOLO passes the image through a backbone network once and directly predicts bounding boxes and class probabilities from a grid of cells.

YOLOv8 (Ultralytics, 2023) is the eighth major revision. The n variant ("nano") has 3.2M parameters and runs at ~80 FPS on a CPU — far faster than the larger models while retaining good accuracy on the 80 COCO object classes (person, car, bicycle, bottle, cup, chair, …). The pre-trained weights file is yolov8n.pt, 6 MB.

How ImageLab uses it

The same lazy-loading pattern as CLIP and EasyOCR. The model is loaded from the local yolov8n.pt file once:

# from imagelab/features.py _yolo_model = None _yolo_lock = threading.Lock() def _get_yolo_model(): global _yolo_model with _yolo_lock: if _yolo_model is None: from ultralytics import YOLO _yolo_model = YOLO("yolov8n.pt") return _yolo_model def yolo_detect(image: Image.Image) -> list[dict[str, Any]]: """Run YOLOv8n and return detections as [{class, confidence, bbox}, ...].""" model = _get_yolo_model() results = model(image, verbose=False) detections: list[dict[str, Any]] = [] for result in results: for box in result.boxes: cls_id = int(box.cls[0]) conf = float(box.conf[0]) xyxy = box.xyxy[0].tolist() detections.append({ "class": result.names[cls_id], "confidence": round(conf, 3), "bbox": [round(v, 1) for v in xyxy], }) detections.sort(key=lambda d: d["confidence"], reverse=True) return detections

The result is stored as a JSON array in the features table: e.g. [{"class": "bicycle", "confidence": 0.91, "bbox": [42.0, 18.3, 380.1, 290.7]}]. The bounding box is in (x1, y1, x2, y2) pixel coordinates.

Comparing two images by detected objects

For similarity search, what matters is not whether the same object instance appears, but whether the same categories appear. Two photos of bicycles should score higher against each other than a bicycle photo scores against a kitchen photo. Jaccard similarity on the set of detected classes achieves this:

# from imagelab/features.py def yolo_class_jaccard(a: list[dict], b: list[dict]) -> float: """Jaccard similarity on detected object class sets.""" set_a = {d["class"] for d in a} set_b = {d["class"] for d in b} if not set_a or not set_b: return 0.0 return len(set_a & set_b) / len(set_a | set_b)

If image A contains {bicycle, person} and image B contains {bicycle, car}, the Jaccard score is |{bicycle}| / |{bicycle, person, car}| = 1/3 ≈ 0.33. If both contain exactly {bicycle, person}, the score is 1.0. This is used as one of the weighted signals in the combined similarity score, alongside cosine embedding distance and dHash Hamming distance.

Model

YOLOv8n

3.2M parameters · 6 MB

Classes

80

COCO dataset categories

Similarity signal

Jaccard

intersection / union of class sets

Weights file

yolov8n.pt

local, no download on use

Module 5 · ~8 min Determinism Across the Stack

Why it matters

A pipeline that produces different results each time you run it is hard to debug, hard to trust, and hard to cache. ImageLab is designed around a clear distinction between what is deterministic (can be run again, will return the same answer, can be stored and trusted as a number) and what is not (should be labelled as an opinion, cached explicitly, never presented as a measurement).

The determinism spectrum in ImageLab

Operation	Level	Why
`sha256`, `dhash`	Fully deterministic	Pure pixel arithmetic. Same bytes → identical output, every time, on every machine.
`blur_score`, `color_histogram`, `edge_histogram`	Fully deterministic	Convolutions and histograms over fixed-format arrays. No randomness anywhere.
CLIP / YOLOv8 inference	Reproducible	Deterministic function of weights + input. May differ in the last few decimal places across machines/hardware due to floating-point precision differences in GPU vs CPU implementations.
EasyOCR inference	Reproducible	Same as above. Beam search is deterministic given a fixed beam width.
`dominant_palette` (k-means)	Seed-dependent	MiniBatchKMeans uses `random_state=42` — deterministic given the same scikit-learn version. Technically reproducible, but depends on an explicit seed.
Claude descriptions	Non-deterministic	LLM sampling. Even at temperature 0 there is no bit-for-bit reproducibility guarantee. Cached by `(sha256, model, prompt_version)` so the same call is never made twice.

The design rule

The P1 spec states it explicitly about the quality metrics: "Measured quality scores track the deliberately poor images perfectly — they're deterministic, they must." This is a hard requirement, not an aspiration. If a blur score or exposure clip value changes between runs on the same image, something is wrong.

The principle flows through the whole system. Look at how the detail page is designed: deterministic numbers (blur score, aspect ratio, exposure) are shown as numbers with no qualification. Claude's output is clearly labelled as a model opinion from a specific model at a specific prompt version. The two are never mixed.

# The cache key for Claude results: UNIQUE (sha256, model, prompt_version) # This means: # - Change the ranking logic: $0 cost, all existing results still valid # - Change the prompt profile: only new-profile images re-run # - Change the image: only that image re-runs # - Same (image, model, prompt): served from SQLite, no API call

Inference reproducibility in practice

For CLIP and YOLO, "reproducible" means: given the same model weights and the same input image, the output vector or detection list will be the same. This holds because:

No sampling or random state is involved in a forward pass.
The model is in .eval() mode — BatchNorm uses running statistics, not batch statistics; Dropout is turned off.
Gradients are disabled — the computation is a pure function.
The only caveat is hardware-level floating-point non-associativity (GPU thread ordering) — this can cause differences in the last 1–2 decimal places across different hardware, but cosine similarity is robust to this.

Module 6 · ~5 min Putting It Together — One Image Through the Pipeline

When you import a folder of images and run a full ingest, each image passes through every stage in sequence. Here is the complete journey, with the library responsible at each step:

Ingest walk-through

Step	Library	Output stored	Det.?
1. Read file, compute SHA-256	`hashlib` (stdlib)	`image.sha256`	Yes
2. Read EXIF, correct orientation	`pillow`	`image.captured_at`, `camera`	Yes
3. Resize to 1568px working copy, generate 256px thumb	`pillow`	Files in `cache/working/`, `cache/thumbs/`	Yes
4. Compute dHash (perceptual hash)	`pillow` + numpy	`image.dhash`	Yes
5. Embed image with CLIP	`torch` + `open-clip`	`embedding.vector` (512-dim BLOB)	Repr.
6. Extract OCR text	`easyocr`	`features.ocr_text`	Yes
7. Run YOLO detection	`ultralytics`	`features.yolo_objects` (JSON)	Yes
8. Compute colour histogram, edges, blur, palette	`numpy`, `scipy`, `sklearn`	`features.*`	Yes
9. Send to Claude for structured description	`anthropic`	`analysis.result_json`	No

Cost structure

Steps 1–8 are local and free. Only step 9 costs money, and only when the cache key is new. A full re-run after changing the similarity weights: $0. A full re-run after changing the prompt profile: only new-profile images re-run. This means the cost of experimentation is bounded and predictable.

Steps 1–8

$0.00

local, deterministic / reproducible

Step 9 (Sonnet)

~$0.01

per image · cached by key

100 images

~$1.00

first run only

Re-rank / tune

$0.00

all results already cached

The separation is what makes this useful as a research tool: you can iterate on the ML-side signals (embeddings, YOLO weights, OCR postprocessing) freely and cheaply, then only spend on Claude when you have something worth describing. The deterministic foundation is what makes the LLM caching trustworthy.

How the ML Stack Works