Introducing a new way to review your PRs.

Read the announcement

immich-app / immich

High performance self-hosted photo and video management solution.

View on GitHub

21 documents · Updated Oct 19, 2026

docs / machine-learning-service

Machine learning service

The machine learning service is a separate Python runtime that turns images and text into face data, OCR data, and search embeddings for the Immich server. It keeps model files, inference sessions, and device-specific execution outside the server process. The vision and text model capabilities page explains the user-facing model tasks in more detail. The automated asset analysis page explains how server jobs use these results.

Service boundary

The service exposes one small HTTP boundary for inference. The Immich server sends a multipart request to /predict, and the service returns JSON keyed by the requested task. The /ping endpoint gives the server a low-cost health check.

Runtime process

The runtime starts an ASGI application behind Gunicorn and Uvicorn workers. The entry process binds the service to its configured host and port, applies worker and timeout settings, and assigns configured device IDs across workers. A container image selects the device support at build time, so the same service shape can run with CPU, GPU, or supported embedded-device execution. The process also keeps memory release and clean shutdown outside the server's control.

References: machine-learning/…/__main__.py• L29-61 machine-learning/…/config.py• L163-177 machine-learning/…/gunicorn_conf.py• L8-16 machine-learning/Dockerfile• L27-40 machine-learning/Dockerfile• L117-173 machine-learning/…/allocator.py• L1-10

Request and response contract

Each request contains one payload and an entries value. The payload is either an image or text. The entries select a task, a model role, a model name, and optional thresholds or size settings. Image responses also carry the source height and width so the server can map returned coordinates to its asset.

The main result shapes are stable across model formats:

TaskResult supplied to the server
clipA serialized visual or text embedding
facial-recognitionFace boxes, scores, and serialized face embeddings
ocrText, normalized text boxes, and detection and recognition scores

References: machine-learning/…/main.py• L141-181 machine-learning/…/schemas.py• L110-160 server/…/machine-learning.repository.ts• L29-75

Inference pipeline

The service parses the request, resolves the model family, loads the needed sessions, and combines the task results before it replies. Independent entries can run in the same stage. A dependent entry waits for the output it names, which lets face recognition use face detection and OCR recognition use text detection.

This diagram shows how a server request becomes one response while preserving model dependencies.

flowchart LR
    Server[Immich server] -->|multipart image or text| API[Predict endpoint]
    API --> Parse[Parse task entries]
    Parse --> Resolve[Resolve model family]
    Resolve --> Detect[Run independent model roles]
    Detect --> Dependent[Run dependent roles]
    Dependent --> Response[Return task JSON]
    Response --> Server

References: machine-learning/…/main.py• L184-209 machine-learning/…/__init__.py• L13-42

Runtime configuration and lifecycle

The runtime reads its own settings from the environment. These settings control service resources and model behavior. The Immich server separately controls feature use, model choices, URLs, and result thresholds.

Runtime settings

The service settings define the bind address, log level, cache location, model lifetime, worker and thread counts, preload list, batch limits, model source, and model revision. Device selection can assign one or more device IDs to workers. The model revision also separates current model exports from legacy model behavior.

References: machine-learning/…/config.py• L21-38 machine-learning/…/config.py• L43-98 machine-learning/…/gunicorn_conf.py• L8-16 machine-learning/Dockerfile• L129-137

Worker lifecycle

At startup the service creates the request thread pool and can preload selected models. Gunicorn assigns configured device IDs to workers in round-robin order. During normal use it tracks active requests and refreshes model expiry. After an idle period it can stop the worker so a deployment can release device memory. Shutdown clears the model cache and closes request resources.

References: machine-learning/…/main.py• L59-87 machine-learning/…/main.py• L89-120 machine-learning/…/main.py• L126-139 machine-learning/…/gunicorn_conf.py• L8-16 machine-learning/…/main.py• L252-258

Request schemas and model sessions

The service uses a shared vocabulary between the HTTP boundary and model code. A task names the product capability. A model type names the role within that capability. A model session gives each model a common way to inspect inputs, select a supported shape, warm the runtime, and run inference.

Task and model roles

The request schema supports three task groups:

TaskModel rolesPurpose
clipvisual, textualCreate image and text embeddings for search
facial-recognitiondetection, recognitionFind faces and create face representations
ocrdetection, recognitionFind text regions and read their text

The role is part of the contract because the service can compose more than one model into one product result. Model names are mapped to the matching model family before the request runs.

References: machine-learning/…/schemas.py• L23-54 machine-learning/…/__init__.py• L13-42 server/…/machine-learning.repository.ts• L14-75

Session contract

An execution session is the device-facing part of a model. It reports input and output information, supports the shapes that the model can accept, and runs prepared numeric inputs. ONNX Runtime is the general path. Arm NN and RKNN provide compatible paths for supported devices. Model code can therefore focus on task inputs and outputs instead of device selection.

The ONNX Runtime path can keep one graph for a provider that supports dynamic dimensions or build separate graphs for fixed supported shapes. TensorRT-RTX shares a serialized boundary for graph creation and inference because its capture and device context cannot safely overlap across request threads. The provider list also avoids combining TensorRT-RTX with the legacy CUDA provider in the same session path.

References: machine-learning/…/schemas.py• L57-108 machine-learning/…/ort.py• L63-103 machine-learning/…/ort.py• L294-333 machine-learning/…/__init__.py• L15-69 machine-learning/…/__init__.py• L40-81

Shape and batch policy

Each model declares the input shapes it can use. The service selects a supported shape for the request and splits face or text crops into supported batches. OCR detection derives its supported canvas sizes from the configured maximum resolution, then fits the source image to one of those canvases before it runs. This keeps one request valid for models with fixed accelerator shapes while allowing the same task contract across runtimes.

For providers that use dynamic dimensions, the session can reuse one graph while the request keeps its selected shape policy. Other providers build and retain a graph for each admitted shape. The session warms those graphs when needed so the first real request does not carry all device preparation work.

References: machine-learning/…/policy.py• L9-32 machine-learning/…/recognition.py• L20-50 machine-learning/…/detection.py• L19-45 machine-learning/…/detection.py• L61-105 machine-learning/…/recognition.py• L27-45 machine-learning/…/ort.py• L294-347

Model loading and caching

Model reuse has a disk layer and a memory layer. The disk layer avoids repeated downloads and stores prepared runtime artifacts. The memory layer keeps model sessions available for later requests. The service can preload models or load them on first use.

Model acquisition

When a model is absent from the configured cache, the service downloads the model snapshot for the selected organization and revision. It keeps files for the preferred model format and ignores files for other formats when possible. If the preferred format is not available, the service can fall back to ONNX.

The model cache is cleared and rebuilt when a model file or prepared runtime artifact is unreadable. A repeated failure becomes an inference error instead of returning a partial result.

References: machine-learning/…/base.py• L20-25 machine-learning/…/base.py• L53-78 machine-learning/…/main.py• L218-249

In-memory model lifetime

The in-memory cache keys a model by its task, role, name, and any option that changes its graph. This matters for settings such as OCR maximum resolution because two graph variants cannot safely share one loaded object. For OCR detection, the graph identity is derived from the resolution used to choose its supported canvas set. A request refreshes the model time-to-live. Preloaded models remain resident until shutdown.

References: machine-learning/…/cache.py• L11-29 machine-learning/…/cache.py• L31-43 machine-learning/…/detection.py• L19-45

Prepared runtime artifacts

Some execution providers need a model graph prepared for a device, shape, and runtime version. The service records those facts in a manifest and reuses the prepared graph only while the facts still match. Preparation runs in a child process so its temporary memory does not remain in the request worker. This gives device caches a clear boundary without changing the server request contract.

References: machine-learning/…/ort.py• L105-169 machine-learning/…/ort.py• L259-276 machine-learning/…/prepare.py• L18-38

Immich server integration

The Immich server treats the Python service as a remote inference provider. Its machine-learning repository builds the shared request shape, sends the asset or query text, and maps the response into server workflows. Feature settings decide whether a workflow calls the boundary.

Server adapter

MachineLearningRepository is the integration boundary. It sends image files for face detection, OCR, and image embeddings. It sends text for search queries. The service returns embeddings as strings and returns image coordinates with source dimensions. The repository exposes these results in forms used by people, OCR, smart-search, and search services.

References: server/…/machine-learning.repository.ts• L162-241 server/…/person.service.ts• L324-359 server/…/ocr.service.ts• L31-64 server/…/smart-info.service.ts• L87-118 server/…/search.service.ts• L325-354

Availability and failover

The server can configure more than one machine-learning URL. It checks each URL with /ping, tries healthy URLs first, and marks a URL unhealthy after a failed request. It then tries the other URLs. This gives the server basic failover without changing the request sent to the service.

References: server/…/machine-learning.repository.ts• L96-187 machine-learning/…/main.py• L155-163

Feature consumers

Background jobs use the service for asset analysis. Face results update asset faces and face search data. OCR results become stored text and searchable regions. Image embeddings support smart search. A text query is encoded at search time, while an asset query uses an embedding stored by an earlier job. Each consumer skips the call when its feature is disabled. The automated asset analysis page describes those server-side workflows.

References: server/…/person.service.ts• L324-359 server/…/ocr.service.ts• L31-55 server/…/smart-info.service.ts• L87-118 server/…/search.service.ts• L325-353 server/…/config.dto.ts• L207-262

Change checklist

When these files or sections are changed, remember to consider these:

  • Keep task and model-role values aligned between machine-learning/…/schemas.py and server/…/machine-learning.repository.ts.
  • Keep response fields aligned with server consumers. This includes image dimensions, face data, normalized OCR boxes, scores, and serialized embeddings.
  • When a request option changes a model graph, keep the option in the model cache key and in the server request that supplies it.
  • When a model's input shape or output changes, review the shape policy, session adapters, model preprocessing, and prepared-graph facts together.
  • When changing model organization, revision, format, or cache paths, review downloads, legacy model handling, preload settings, and existing disk caches together.
  • When changing device support or worker assignment, keep the container build variant, available execution provider, device IDs, model artifact path, and session contract aligned. For TensorRT-RTX, also preserve the serialized graph and inference boundary and its provider selection.
  • When changing service availability, preserve the relationship between /ping, URL health state, retry order, and the server's configured URL list.
  • When changing feature switches or score thresholds, review both the server job guard and the options sent in the inference request.

Important failure modes

TriggerConsequenceGuard
A request has neither an image nor textThe service rejects it with a client errorSend exactly one supported payload type
An entry names an unsupported task, role, model family, or option combinationModel resolution fails before useful inferenceKeep the Python and server request contracts aligned
A model snapshot, model file, or prepared graph is missing or unreadableThe service clears the affected cache and retries; repeated failure returns an errorUse a writable persistent cache with model artifacts for the selected device and revision
A model is requested with a shape that its session does not supportThe model cannot run the requestKeep model shape policies and exported accelerator artifacts aligned
Concurrent TensorRT-RTX graph creation or inference crosses its capture boundarySession setup or inference can fail because CUDA graph capture and device context use overlapKeep TensorRT-RTX graph creation and inference serialized and do not add the legacy CUDA provider to that path
The configured OCR resolution differs from a preloaded detector's graph optionThe request creates a different model cache entry and does not reuse the preloaded graphKeep the server OCR resolution and preload resolution aligned
Worker device IDs do not match the available device supportA worker can start with an unusable device assignment and fail model executionConfigure device IDs that exist in the selected runtime and keep worker assignment aligned with the container build
Every configured machine-learning URL is unhealthy or unreachableThe server fails the inference operation after trying all URLsKeep /ping reachable and configure at least one working service
The CLIP model changes while an embedding job runsThe generated embedding is skipped instead of being stored for the new modelRecheck the configured model before upserting the embedding
A machine-learning feature is disabledIts background job skips the remote callEnable the service and the matching capability before queueing work

Key files

FileWhy to read it
machine-learning/…/main.pyShows the HTTP endpoints, request parsing, dependency order, cache use, and lifecycle.
machine-learning/…/__main__.pyShows how the service process starts Gunicorn and binds the runtime address.
machine-learning/…/schemas.pyDefines the task, model-role, session, request, and response vocabulary.
machine-learning/…/config.pyShows the settings that control workers, device binding, logging, cache lifetime, preload, batches, and model revision.
machine-learning/…/gunicorn_conf.pyShows how configured device IDs map to service workers.
machine-learning/…/base.pyShows the shared model lifecycle from acquisition to session creation and prediction.
machine-learning/…/cache.pyShows model identity, graph-option separation, and in-memory expiry.
machine-learning/…/detection.pyShows how OCR detection maps a resolution option to supported canvases and returns ordered text boxes.
machine-learning/…/ort.pyShows the general session path, provider choice, shape handling, prepared graph reuse, and TensorRT-RTX serialization.
machine-learning/…/prepare.pyShows how runtime-specific model artifacts are created and marked complete.
machine-learning/…/__init__.pyShows the Arm NN session that follows the common session contract.
machine-learning/…/__init__.pyShows the RKNN session and its device-specific model selection.
machine-learning/DockerfileShows how the service image selects device support and exposes the runtime cache.
server/…/machine-learning.repository.tsDefines the HTTP contract, server request builders, health checks, and failover.
server/…/config.dto.tsDefines server-side service URLs, model choices, feature switches, and task thresholds.
machine-learning/test_main.pyShows the guarantees for request behavior, model caching, session providers, shape handling, and graph preparation.