Machine learning service
The machine learning service is a separate Python runtime that turns images and text into face data, OCR data, and search embeddings for the Immich server. It keeps model files, inference sessions, and device-specific execution outside the server process. The vision and text model capabilities page explains the user-facing model tasks in more detail. The automated asset analysis page explains how server jobs use these results.
Service boundary
The service exposes one small HTTP boundary for inference. The Immich server sends a multipart request to /predict, and the service returns JSON keyed by the requested task. The /ping endpoint gives the server a low-cost health check.
Runtime process
The runtime starts an ASGI application behind Gunicorn and Uvicorn workers. The entry process binds the service to its configured host and port, applies worker and timeout settings, and assigns configured device IDs across workers. A container image selects the device support at build time, so the same service shape can run with CPU, GPU, or supported embedded-device execution. The process also keeps memory release and clean shutdown outside the server's control.
References: machine-learning/…/__main__.py• L29-61 machine-learning/…/config.py• L163-177 machine-learning/…/gunicorn_conf.py• L8-16 machine-learning/Dockerfile• L27-40 machine-learning/Dockerfile• L117-173 machine-learning/…/allocator.py• L1-10
Request and response contract
Each request contains one payload and an entries value. The payload is either an image or text. The entries select a task, a model role, a model name, and optional thresholds or size settings. Image responses also carry the source height and width so the server can map returned coordinates to its asset.
The main result shapes are stable across model formats:
| Task | Result supplied to the server |
|---|---|
clip | A serialized visual or text embedding |
facial-recognition | Face boxes, scores, and serialized face embeddings |
ocr | Text, normalized text boxes, and detection and recognition scores |
References: machine-learning/…/main.py• L141-181 machine-learning/…/schemas.py• L110-160 server/…/machine-learning.repository.ts• L29-75
Inference pipeline
The service parses the request, resolves the model family, loads the needed sessions, and combines the task results before it replies. Independent entries can run in the same stage. A dependent entry waits for the output it names, which lets face recognition use face detection and OCR recognition use text detection.
This diagram shows how a server request becomes one response while preserving model dependencies.
flowchart LR
Server[Immich server] -->|multipart image or text| API[Predict endpoint]
API --> Parse[Parse task entries]
Parse --> Resolve[Resolve model family]
Resolve --> Detect[Run independent model roles]
Detect --> Dependent[Run dependent roles]
Dependent --> Response[Return task JSON]
Response --> Server
References: machine-learning/…/main.py• L184-209 machine-learning/…/__init__.py• L13-42
Runtime configuration and lifecycle
The runtime reads its own settings from the environment. These settings control service resources and model behavior. The Immich server separately controls feature use, model choices, URLs, and result thresholds.
Runtime settings
The service settings define the bind address, log level, cache location, model lifetime, worker and thread counts, preload list, batch limits, model source, and model revision. Device selection can assign one or more device IDs to workers. The model revision also separates current model exports from legacy model behavior.
References: machine-learning/…/config.py• L21-38 machine-learning/…/config.py• L43-98 machine-learning/…/gunicorn_conf.py• L8-16 machine-learning/Dockerfile• L129-137
Worker lifecycle
At startup the service creates the request thread pool and can preload selected models. Gunicorn assigns configured device IDs to workers in round-robin order. During normal use it tracks active requests and refreshes model expiry. After an idle period it can stop the worker so a deployment can release device memory. Shutdown clears the model cache and closes request resources.
References: machine-learning/…/main.py• L59-87 machine-learning/…/main.py• L89-120 machine-learning/…/main.py• L126-139 machine-learning/…/gunicorn_conf.py• L8-16 machine-learning/…/main.py• L252-258
Request schemas and model sessions
The service uses a shared vocabulary between the HTTP boundary and model code. A task names the product capability. A model type names the role within that capability. A model session gives each model a common way to inspect inputs, select a supported shape, warm the runtime, and run inference.
Task and model roles
The request schema supports three task groups:
| Task | Model roles | Purpose |
|---|---|---|
clip | visual, textual | Create image and text embeddings for search |
facial-recognition | detection, recognition | Find faces and create face representations |
ocr | detection, recognition | Find text regions and read their text |
The role is part of the contract because the service can compose more than one model into one product result. Model names are mapped to the matching model family before the request runs.
References: machine-learning/…/schemas.py• L23-54 machine-learning/…/__init__.py• L13-42 server/…/machine-learning.repository.ts• L14-75
Session contract
An execution session is the device-facing part of a model. It reports input and output information, supports the shapes that the model can accept, and runs prepared numeric inputs. ONNX Runtime is the general path. Arm NN and RKNN provide compatible paths for supported devices. Model code can therefore focus on task inputs and outputs instead of device selection.
The ONNX Runtime path can keep one graph for a provider that supports dynamic dimensions or build separate graphs for fixed supported shapes. TensorRT-RTX shares a serialized boundary for graph creation and inference because its capture and device context cannot safely overlap across request threads. The provider list also avoids combining TensorRT-RTX with the legacy CUDA provider in the same session path.
References: machine-learning/…/schemas.py• L57-108 machine-learning/…/ort.py• L63-103 machine-learning/…/ort.py• L294-333 machine-learning/…/__init__.py• L15-69 machine-learning/…/__init__.py• L40-81
Shape and batch policy
Each model declares the input shapes it can use. The service selects a supported shape for the request and splits face or text crops into supported batches. OCR detection derives its supported canvas sizes from the configured maximum resolution, then fits the source image to one of those canvases before it runs. This keeps one request valid for models with fixed accelerator shapes while allowing the same task contract across runtimes.
For providers that use dynamic dimensions, the session can reuse one graph while the request keeps its selected shape policy. Other providers build and retain a graph for each admitted shape. The session warms those graphs when needed so the first real request does not carry all device preparation work.
References: machine-learning/…/policy.py• L9-32 machine-learning/…/recognition.py• L20-50 machine-learning/…/detection.py• L19-45 machine-learning/…/detection.py• L61-105 machine-learning/…/recognition.py• L27-45 machine-learning/…/ort.py• L294-347
Model loading and caching
Model reuse has a disk layer and a memory layer. The disk layer avoids repeated downloads and stores prepared runtime artifacts. The memory layer keeps model sessions available for later requests. The service can preload models or load them on first use.
Model acquisition
When a model is absent from the configured cache, the service downloads the model snapshot for the selected organization and revision. It keeps files for the preferred model format and ignores files for other formats when possible. If the preferred format is not available, the service can fall back to ONNX.
The model cache is cleared and rebuilt when a model file or prepared runtime artifact is unreadable. A repeated failure becomes an inference error instead of returning a partial result.
References: machine-learning/…/base.py• L20-25 machine-learning/…/base.py• L53-78 machine-learning/…/main.py• L218-249
In-memory model lifetime
The in-memory cache keys a model by its task, role, name, and any option that changes its graph. This matters for settings such as OCR maximum resolution because two graph variants cannot safely share one loaded object. For OCR detection, the graph identity is derived from the resolution used to choose its supported canvas set. A request refreshes the model time-to-live. Preloaded models remain resident until shutdown.
References: machine-learning/…/cache.py• L11-29 machine-learning/…/cache.py• L31-43 machine-learning/…/detection.py• L19-45
Prepared runtime artifacts
Some execution providers need a model graph prepared for a device, shape, and runtime version. The service records those facts in a manifest and reuses the prepared graph only while the facts still match. Preparation runs in a child process so its temporary memory does not remain in the request worker. This gives device caches a clear boundary without changing the server request contract.
References: machine-learning/…/ort.py• L105-169 machine-learning/…/ort.py• L259-276 machine-learning/…/prepare.py• L18-38
Immich server integration
The Immich server treats the Python service as a remote inference provider. Its machine-learning repository builds the shared request shape, sends the asset or query text, and maps the response into server workflows. Feature settings decide whether a workflow calls the boundary.
Server adapter
MachineLearningRepository is the integration boundary. It sends image files for face detection, OCR, and image embeddings. It sends text for search queries. The service returns embeddings as strings and returns image coordinates with source dimensions. The repository exposes these results in forms used by people, OCR, smart-search, and search services.
References: server/…/machine-learning.repository.ts• L162-241 server/…/person.service.ts• L324-359 server/…/ocr.service.ts• L31-64 server/…/smart-info.service.ts• L87-118 server/…/search.service.ts• L325-354
Availability and failover
The server can configure more than one machine-learning URL. It checks each URL with /ping, tries healthy URLs first, and marks a URL unhealthy after a failed request. It then tries the other URLs. This gives the server basic failover without changing the request sent to the service.
References: server/…/machine-learning.repository.ts• L96-187 machine-learning/…/main.py• L155-163
Feature consumers
Background jobs use the service for asset analysis. Face results update asset faces and face search data. OCR results become stored text and searchable regions. Image embeddings support smart search. A text query is encoded at search time, while an asset query uses an embedding stored by an earlier job. Each consumer skips the call when its feature is disabled. The automated asset analysis page describes those server-side workflows.
References: server/…/person.service.ts• L324-359 server/…/ocr.service.ts• L31-55 server/…/smart-info.service.ts• L87-118 server/…/search.service.ts• L325-353 server/…/config.dto.ts• L207-262
Change checklist
When these files or sections are changed, remember to consider these:
- Keep task and model-role values aligned between machine-learning/…/schemas.py and server/…/machine-learning.repository.ts.
- Keep response fields aligned with server consumers. This includes image dimensions, face data, normalized OCR boxes, scores, and serialized embeddings.
- When a request option changes a model graph, keep the option in the model cache key and in the server request that supplies it.
- When a model's input shape or output changes, review the shape policy, session adapters, model preprocessing, and prepared-graph facts together.
- When changing model organization, revision, format, or cache paths, review downloads, legacy model handling, preload settings, and existing disk caches together.
- When changing device support or worker assignment, keep the container build variant, available execution provider, device IDs, model artifact path, and session contract aligned. For TensorRT-RTX, also preserve the serialized graph and inference boundary and its provider selection.
- When changing service availability, preserve the relationship between
/ping, URL health state, retry order, and the server's configured URL list. - When changing feature switches or score thresholds, review both the server job guard and the options sent in the inference request.
Important failure modes
| Trigger | Consequence | Guard |
|---|---|---|
| A request has neither an image nor text | The service rejects it with a client error | Send exactly one supported payload type |
| An entry names an unsupported task, role, model family, or option combination | Model resolution fails before useful inference | Keep the Python and server request contracts aligned |
| A model snapshot, model file, or prepared graph is missing or unreadable | The service clears the affected cache and retries; repeated failure returns an error | Use a writable persistent cache with model artifacts for the selected device and revision |
| A model is requested with a shape that its session does not support | The model cannot run the request | Keep model shape policies and exported accelerator artifacts aligned |
| Concurrent TensorRT-RTX graph creation or inference crosses its capture boundary | Session setup or inference can fail because CUDA graph capture and device context use overlap | Keep TensorRT-RTX graph creation and inference serialized and do not add the legacy CUDA provider to that path |
| The configured OCR resolution differs from a preloaded detector's graph option | The request creates a different model cache entry and does not reuse the preloaded graph | Keep the server OCR resolution and preload resolution aligned |
| Worker device IDs do not match the available device support | A worker can start with an unusable device assignment and fail model execution | Configure device IDs that exist in the selected runtime and keep worker assignment aligned with the container build |
| Every configured machine-learning URL is unhealthy or unreachable | The server fails the inference operation after trying all URLs | Keep /ping reachable and configure at least one working service |
| The CLIP model changes while an embedding job runs | The generated embedding is skipped instead of being stored for the new model | Recheck the configured model before upserting the embedding |
| A machine-learning feature is disabled | Its background job skips the remote call | Enable the service and the matching capability before queueing work |
Key files
| File | Why to read it |
|---|---|
| machine-learning/…/main.py | Shows the HTTP endpoints, request parsing, dependency order, cache use, and lifecycle. |
| machine-learning/…/__main__.py | Shows how the service process starts Gunicorn and binds the runtime address. |
| machine-learning/…/schemas.py | Defines the task, model-role, session, request, and response vocabulary. |
| machine-learning/…/config.py | Shows the settings that control workers, device binding, logging, cache lifetime, preload, batches, and model revision. |
| machine-learning/…/gunicorn_conf.py | Shows how configured device IDs map to service workers. |
| machine-learning/…/base.py | Shows the shared model lifecycle from acquisition to session creation and prediction. |
| machine-learning/…/cache.py | Shows model identity, graph-option separation, and in-memory expiry. |
| machine-learning/…/detection.py | Shows how OCR detection maps a resolution option to supported canvases and returns ordered text boxes. |
| machine-learning/…/ort.py | Shows the general session path, provider choice, shape handling, prepared graph reuse, and TensorRT-RTX serialization. |
| machine-learning/…/prepare.py | Shows how runtime-specific model artifacts are created and marked complete. |
| machine-learning/…/__init__.py | Shows the Arm NN session that follows the common session contract. |
| machine-learning/…/__init__.py | Shows the RKNN session and its device-specific model selection. |
machine-learning/Dockerfile | Shows how the service image selects device support and exposes the runtime cache. |
| server/…/machine-learning.repository.ts | Defines the HTTP contract, server request builders, health checks, and failover. |
| server/…/config.dto.ts | Defines server-side service URLs, model choices, feature switches, and task thresholds. |
| machine-learning/test_main.py | Shows the guarantees for request behavior, model caching, session providers, shape handling, and graph preparation. |