Introducing a new way to review your PRs.

Read the announcement

immich-app / immich

High performance self-hosted photo and video management solution.

View on GitHub

21 documents · Updated Oct 19, 2026

docs / vision-and-text-model-capabilities

Vision and text model capabilities

The vision and text model layer turns images and text into signals for search and asset analysis. It provides shared image and text embeddings through CLIP, face locations and face embeddings through facial recognition, and detected text with scores through OCR. The machine learning service exposes these capabilities to the server, while automated asset analysis stores and uses the results.

Capability boundary

The model service accepts one image or one text value with a list of requested model tasks. It selects the model family for each task, prepares the input, runs the required stages, and returns task-specific results together with image dimensions when the input is an image.

The server sends requests through server/…/machine-learning.repository.ts. That boundary keeps model execution separate from product workflows. Search asks for embeddings, people analysis asks for faces, and OCR jobs ask for text regions and recognized text.

The main capabilities have these roles:

CapabilityInputProduct signal
CLIPImage or textA vector used to compare images and text for smart search
Facial recognitionImageFace boxes, confidence scores, and face embeddings
OCRImageText strings, polygon boxes, and detection and recognition scores

The request and result flow is organized as follows.

flowchart LR
    A[Asset or search text] --> B[Server model repository]
    B --> C[Machine learning request]
    C --> D[Input transform]
    D --> E[Model pipeline]
    E --> F[Structured result]
    F --> G[Search, people, or OCR analysis]

Shared model abstraction

The shared model abstraction gives every model a common life cycle and runtime boundary. A model has a task and type identity, loads its model data when first used, and returns a result in the shape expected by the request. InferenceModel in machine-learning/…/base.py is the common contract.

Models can use more than one execution format. The service selects a format supported by the environment, downloads model data when it is not cached, and creates a session for inference. A model can warm its session before use, and the model cache reuses loaded instances across requests.

Model selection and dependencies

Model selection maps a model name, task, and type to one of the supported capability implementations. The model source map in machine-learning/…/constants.py identifies CLIP, InsightFace, and Paddle model families for visual, textual, detection, or recognition roles. The supported names include newer CLIP and Paddle OCR model families, so the map remains the contract that connects configured names to their model source.

Some roles depend on an earlier result. Facial recognition depends on face detection, and OCR recognition depends on text detection. The service runs independent stages first, then supplies their results to dependent stages. This lets one request express a complete pipeline without making the caller manage intermediate data.

References: machine-learning/…/constants.py• L4-92 machine-learning/…/constants.py• L167-182 machine-learning/…/recognition.py• L24-27 machine-learning/…/recognition.py• L23-25

Model loading and caching

Model loading keeps downloaded model data separate from live inference sessions. InferenceModel loads a model once under a load lock, while ModelCache returns the same model for matching model identity and graph options. Temporary models can expire after a time-to-live; preloaded models remain available until the cache is cleared.

This cache boundary also accounts for options that change a model graph. For example, OCR detection resolution is part of the cache key, so requests with different resolution choices do not share an incompatible model instance.

References: machine-learning/…/base.py• L27-92 machine-learning/…/base.py• L122-192 machine-learning/…/cache.py• L11-43 machine-learning/…/detection.py• L19-44

Input and output transforms

Transforms make source data match the model contract and make model results useful to the server. Shared image handling decodes bytes, arrays, or images to RGB data. Model-specific stages then resize, crop, pad, normalize, or change tensor layout as required by the selected model.

Text inputs are cleaned before tokenization. CLIP tokenizers pad or truncate text to the model context and can add a language code for multilingual models. Numeric vectors are serialized so the HTTP response can carry them without requiring the server to understand the model runtime. Detection and recognition results keep semantic fields such as boxes, landmarks, text, and confidence scores.

References: machine-learning/…/transforms.py• L15-79 machine-learning/…/transforms.py• L82-97 machine-learning/…/textual.py• L28-60 machine-learning/…/visual.py• L25-36

CLIP embeddings

CLIP provides a shared representation for visual and textual content. The visual encoder turns an asset image into a vector, and the textual encoder turns a search phrase into a vector in the same comparison space. The server can then compare the two forms without matching the words in the phrase directly to stored labels.

Visual encoding

The visual pipeline decodes an image, applies the preprocessing settings for the selected model, and returns one serialized embedding. Smart search stores that embedding for an asset and later compares it with a text embedding or with another asset embedding.

The visual preprocessing settings are part of the model contract. A change in image size, resize mode, interpolation, normalization, or model choice can change the meaning and dimension of the vector. The server therefore checks the selected CLIP model and keeps the search index dimension aligned with it.

References: machine-learning/…/visual.py• L25-36 machine-learning/…/visual.py• L62-89 machine-learning/…/transforms.py• L15-41 server/…/machine-learning.repository.ts• L205-209 server/…/smart-info.service.ts• L34-64

Text encoding

The textual pipeline cleans a search phrase, tokenizes it with the selected CLIP family, and returns a serialized embedding. OpenCLIP and multilingual CLIP use their own token input shapes, while multilingual OpenCLIP can add a language code when the model supports it. The search service caches these text embeddings during query handling and uses them to search stored asset embeddings.

References: machine-learning/…/textual.py• L28-60 machine-learning/…/textual.py• L101-142 server/…/machine-learning.repository.ts• L211-215 server/…/search.service.ts• L325-353

Facial recognition pipeline

Facial recognition converts an image into face records that can support people grouping and face search. It has two linked stages: detection finds candidate faces and landmarks, then recognition aligns each face and produces an embedding for that face.

Face detection

The detector prepares the image with aspect-preserving padding and pixel normalization. It decodes detector output into face boxes, scores, and five-point landmarks, then removes low-score and overlapping candidates. It maps the kept coordinates back from the detector canvas to the source image before the recognition stage receives them.

References: machine-learning/…/detection.py• L13-38 machine-learning/…/_ops.py• L18-56

Face recognition

The recognizer uses detector landmarks to align each face to a common face crop before it creates an embedding. Its result joins the embedding with the face box and detection score, so the server can both locate the face in the source image and compare it with known people.

The server scales returned boxes to the asset dimensions that it stores. It uses the embeddings for face search and keeps the face records linked to the asset and, when known, to a person.

References: machine-learning/…/recognition.py• L24-72 machine-learning/…/_ops.py• L59-73 server/…/machine-learning.repository.ts• L190-203 server/…/person.service.ts• L341-369 server/…/asset-face.table.ts• L40-88

OCR pipeline

OCR converts visible text in an image into searchable text and its position. It has two linked stages: detection finds text polygons, then recognition reads each cropped polygon and returns text with a recognition score.

Text detection

The detector scales an image to a model-friendly canvas and produces a probability map for text. It selects a compatible compiled canvas for the requested maximum resolution and skips images that are too small to yield a useful result. Post-processing turns the map into quadrilateral boxes and detection scores. The boxes are ordered from top to bottom and then from left to right, which gives the recognition stage a stable reading sequence.

References: machine-learning/…/detection.py• L19-36 machine-learning/…/detection.py• L46-59 machine-learning/…/detection.py• L61-92 machine-learning/…/detection.py• L111-117 machine-learning/…/postprocess.py• L45-52

Text recognition

The recognizer crops each detected polygon and corrects its perspective before resizing the crops for recognition. It groups similar crop shapes into batches, decodes the model output with a CTC decoder, and filters results below the recognition threshold. Before returning, it normalizes polygon coordinates to the image width and height so the server can store them in a stable range.

References: machine-learning/…/recognition.py• L23-55 machine-learning/…/recognition.py• L56-113 machine-learning/…/ctc.py• L62-85

OCR model compatibility

OCR supports both the current model-graph path and older model exports. The compatibility layer keeps older installations on the ONNX format and obtains their model files through the older OCR model source. Current model graphs use the shared model loading and session path.

The model source map in machine-learning/…/constants.py recognizes the supported Paddle OCR model names. The legacy path is used only when the older model setting is enabled. This keeps older exports available without making them the default for current model graphs.

References: machine-learning/…/legacy.py• L6-42 machine-learning/…/constants.py• L78-92 machine-learning/…/constants.py• L167-182 machine-learning/…/detection.py• L19-44 machine-learning/…/recognition.py• L23-45

Analysis outputs

The server turns model results into durable search data and user-visible asset data. The model service does not decide how a face is assigned to a person or how OCR text is searched. It provides the signals and scores that those server workflows use.

Search embeddings

Smart search creates an image embedding for each eligible asset and stores it in the search index. A text query uses the selected textual encoder to create a comparable embedding. A query can also use an existing asset embedding, which supports image-to-image search.

Changing the CLIP model can change both vector size and vector meaning. The smart-information workflow validates the model, updates the stored dimension when needed, and removes old embeddings when the model changes. Assets then need new image embeddings before the new model can give complete search results.

References: server/…/smart-info.service.ts• L67-118 server/…/search.service.ts• L325-350 server/…/machine-learning.repository.ts• L205-215

Face records

Face analysis stores each detected face with its asset location, image dimensions, visibility state, and optional person link. The face embedding supports later matching. The person workflow also uses the returned image dimensions to map model coordinates to the asset record.

References: server/…/person.service.ts• L324-369 server/…/asset-face.table.ts• L36-88 server/…/machine-learning.repository.ts• L68-74

OCR records

OCR analysis stores each recognized string with its quadrilateral, detection score, and recognition score. It also builds search tokens from the recognized text, so OCR contributes to text search as well as to an overlay on the asset.

References: server/…/ocr.service.ts• L31-55 server/…/ocr.service.ts• L57-80 server/…/asset-ocr.table.ts• L23-65

Change checklist

When these files or sections are changed, remember to consider these:

  • Keep the Python model task and type identities aligned with the TypeScript request and response types in server/…/machine-learning.repository.ts.
  • When a pipeline output changes, update the server consumer that stores or compares it. This includes smart-search vectors, face coordinate scaling, and OCR polygon normalization.
  • When a CLIP model or preprocessing contract changes, keep the search vector dimension and existing embedding refresh path aligned with the new model.
  • When a model graph option changes, keep it in the model cache key so requests cannot share an incompatible loaded model.
  • When a detector output changes, preserve the dependency input expected by its recognition stage.
  • When OCR decoding, coordinate, or score meaning changes, update OcrService.parseOcrResults and the asset_ocr column contract together.
  • When legacy OCR support changes, keep the old export path separate from the current model-graph path until older installations no longer need it.
  • When adding or removing a configured model name, update the model source map and the matching source-specific loading path together.

Important failure modes

TriggerConsequenceGuard
A model name does not map to a known source, task, and typeThe prediction request cannot create the requested modelValidate the model name and use a supported model combination before sending work
A dependent recognition stage runs without its detector resultThe machine learning request returns a bad-request error instead of a partial resultSend detection and recognition entries as one complete pipeline
A cached model is reused across incompatible graph optionsInference uses the wrong compiled shape or resolutionInclude every graph-changing option in the model cache key
A CLIP model changes without matching search dimensions or fresh embeddingsSearch can reject vectors or return incomplete resultsValidate the model, align the stored dimension, and re-encode affected assets
Face landmarks or OCR polygons are interpreted in the wrong coordinate spaceStored overlays and later face matches point to the wrong image areaPreserve the detector-to-source scaling and OCR normalized-coordinate contracts
An asset has no usable preview or is hiddenFace or OCR analysis skips or fails the jobKeep asset eligibility checks before queuing or processing analysis
All configured machine learning servers fail health or prediction checksThe server cannot complete the analysis requestKeep more than one healthy endpoint when high availability is required and monitor the ping state
OCR detection or recognition scores fall below configured limitsText is omitted from stored OCR resultsSet thresholds with the expected model and image quality in mind
A configured Paddle model name is missing from the source mapOCR model selection returns no source and the request cannot startKeep Paddle model names and legacy/current OCR loading paths aligned
Legacy OCR is enabled for a model export that does not match its expected language or sizeModel download or inference can fail for older installationsKeep legacy model names in the expected version, language, and size form

Key files

FileWhy to read it
machine-learning/…/main.pyShows the request boundary and how independent and dependent model stages run.
machine-learning/…/base.pyDefines the shared model life cycle, model loading, runtime session boundary, and model format choice.
machine-learning/…/cache.pyShows how loaded models are shared and how temporary models expire.
machine-learning/…/transforms.pyDefines common image, text, normalization, coordinate, and serialization steps.
machine-learning/…/constants.pyMaps model names to CLIP, facial-recognition, and OCR model sources and defines supported model families.
machine-learning/…/visual.py and textual.pyShow the two sides of the shared CLIP embedding space.
machine-learning/…/detection.py, recognition.py, and _ops.pyShow the detector-to-alignment-to-embedding face pipeline.
machine-learning/…/detection.py, recognition.py, ctc.py, and legacy.pyShow OCR box detection, text decoding, and support for older model exports.
server/…/machine-learning.repository.tsDefines the server-to-model-service contract and product-facing result types.
server/…/smart-info.service.tsShows how image embeddings enter the smart-search index.
server/…/person.service.tsShows how face results become asset and person data.
server/…/ocr.service.tsShows how OCR results become stored text and search tokens.