Introducing a new way to review your PRs.

Read the announcement

immich-app / immich

High performance self-hosted photo and video management solution.

View on GitHub

21 documents · Updated Oct 19, 2026

docs / automated-asset-analysis

Automated asset analysis

Automated asset analysis turns uploaded media into information that people can view, search, group, and manage. Background jobs read file metadata, call optional machine learning services, and save the results next to the asset so later search and organization features can use them. This page focuses on that analysis layer; see Background jobs and operations for the wider job system, Asset lifecycle and media processing for upload and media preparation, Asset organization and discovery for user-facing discovery features, and Storage and system operations for the storage services that hold source and derived files.

Analysis pipeline

Analysis runs after an asset has the files needed by a task, such as a preview image. The work is split into queues so metadata work, model work, and follow-up work can run independently while still recording when each task is complete.

Asset analysis lifecycle

The main post-upload path creates several analysis tasks from the available asset files. Some tasks then create the next task, such as smart information leading to duplicate detection.

flowchart TD
    A[Asset and source file] --> B[Metadata extraction]
    A --> C[Preview and thumbnail ready]
    C --> D[Smart information]
    C --> E[OCR]
    C --> F[Face detection]
    F --> G[Face recognition and people]
    D --> H[Duplicate detection]
    B --> I[Metadata and tags]
    D --> J[Search data]
    E --> J
    G --> J
    I --> J
    J --> K[Asset search and discovery]

The system can queue a task for one asset or queue a scan for all assets. A scan selects assets that have not completed the requested work unless the task is forced. The asset_job_status record gives each analysis type its own completion time, so one failed or disabled feature does not make all other analysis stale.

References: server/…/job.service.ts• L167-188 server/…/job.service.ts• L251-255 server/…/asset-job.repository.ts• L196-229 server/…/asset-job-status.table.ts• L4-20

Queue control

Analysis queues run in the microservices worker. The API watches worker state and exposes queue controls for starting, pausing, resuming, and clearing work. Queue concurrency is read from system configuration, while some work such as duplicate detection and facial recognition uses a single worker path to protect shared search and person data.

Nightly work can also refresh selected analysis. For example, it can cluster new faces and generate missing thumbnails. The queue service owns this schedule and prevents more than one worker from claiming the nightly run.

References: server/…/queue.service.ts• L44-103 server/…/queue.service.ts• L191-265 server/…/queue.service.ts• L267-300 server/…/microservices.ts• L12-31

Derived asset information

Each analysis feature produces a different kind of asset information. The system keeps source metadata, model output, and task state separate so users can update one kind of information without replacing the others.

Metadata and sidecars

Metadata extraction builds the asset record from the source file and any matching XMP sidecar. It supplies dates, time zones, dimensions, camera or device details, GPS data, descriptions, ratings, media details, tags, and grouping hints. GPS data can also produce a country, state, or city through reverse geocoding.

The result updates the asset and its EXIF data, then marks metadata extraction complete. Tags from the file become application tags. Metadata can also identify motion photos and link their embedded video as a hidden companion asset. Named face regions in metadata can create or refresh face records and person groups. Changes made in the application can be written back to a sidecar. A later sidecar check updates the stored sidecar association when the discovered path changes, and later metadata extraction can read the new sidecar while preserving locked user values.

References: server/…/metadata.service.ts• L215-246 server/…/metadata.service.ts• L260-317 server/…/metadata.service.ts• L361-417 server/…/metadata.service.ts• L420-533 server/…/metadata.service.ts• L669-817 server/…/metadata.service.ts• L820-1001

Smart information

Smart information is the visual search data for an asset. The system sends a preview image to the machine learning service and stores the returned image embedding. An embedding is a numeric representation that lets the search system compare visual meaning rather than only matching file fields or words.

The feature is optional. A full scan skips when it is disabled, ignores hidden assets, and avoids replacing an embedding when the model changed during processing. A model change can require the stored search data to be cleared and rebuilt because embeddings from different models or dimensions cannot be compared safely.

References: server/…/smart-info.service.ts• L34-64 server/…/smart-info.service.ts• L67-118 server/…/machine-learning.repository.ts• L190-215 server/…/smart-search.table.ts• L4-18

Optical character recognition (OCR) reads text that appears inside an image. The OCR task sends the asset preview to the machine learning service, then saves both the detected text regions and a normalized text value for search.

The saved regions support user-visible text overlays. The saved search text supports queries for words that do not exist in the file name or metadata. Hidden assets are excluded, and the task records its completion time only after the OCR result is stored.

References: server/…/ocr.service.ts• L13-55 server/…/ocr.service.ts• L57-80 server/…/ocr-search.table.ts• L4-19 server/…/asset-job.repository.ts• L254-260

Faces and people

Face analysis has two related goals. Detection finds face boxes and face embeddings in a preview, while recognition compares those embeddings with other faces to assign faces to person groups. A person is the user-visible identity attached to a group of related faces.

Detection refreshes machine-found faces for an asset and records the face-analysis state after it stores the current result. New faces then enter recognition. Recognition waits for earlier preview and detection work before a broad run, and a nightly run skips when no newer faces need processing. It searches within the asset's cluster group, which is the shared user boundary for related assets. It can reuse a matching person group, create a new group for the asset owner, or defer a weak match until more evidence exists. A forced run can be limited to one cluster group, while a full run covers all eligible groups. The system also accepts named face regions from asset metadata, which can create or update people without a new model result.

Face refresh can remove old machine-found faces, then clean up person groups and cluster groups that no longer have valid faces. Faces are subject to asset visibility and ownership rules. Hidden assets are skipped. A person record is created for the asset owner even when a matching group came from another user in the same cluster group. Person cleanup removes groups that no longer have faces.

References: server/…/person.service.ts• L355-367 server/…/person.service.ts• L369-392 server/…/person.service.ts• L395-477 server/…/person.service.ts• L496-551 server/…/person.service.ts• L554-646 server/…/metadata.service.ts• L902-1001 server/…/face-search.table.ts• L4-17

Duplicate detection

Duplicate detection uses the visual embedding already created for an asset. It compares the asset with other assets owned by the same user and groups close matches by a configured distance limit. It skips hidden, locked, stacked, or unembedded assets.

The result is a duplicate group that users can review. Resolving a group can keep one asset, move the other assets to the trash, and merge safe metadata, album, and tag information into the keeper. A new upload can trigger this check after smart information succeeds; a full queue scan can process existing assets.

References: server/…/duplicate.service.ts• L302-374 server/…/duplicate.service.ts• L376-400 server/…/duplicate.service.ts• L158-236 server/…/job.service.ts• L251-255

Search indexing

Search indexing connects derived information to asset discovery. Metadata fields are queried from the asset and EXIF records, OCR stores searchable text, smart information stores visual embeddings, and face analysis stores face embeddings linked to asset faces. The search service combines these sources with asset filters and access scope before returning results.

The main search modes are:

  • Metadata search, which filters by dates, file state, location, camera data, tags, albums, and OCR text.
  • Smart search, which compares a text query or another asset embedding with stored visual embeddings.
  • People search, which finds person groups and uses face similarity during recognition.
  • Place and suggestion search, which uses extracted location and camera fields to help users form filters.

The search layer applies visibility and sharing rules while it builds the search scope. Locked assets remain personal, and shared-link searches must stay within an allowed album. This means analysis can create data for an asset, but search still decides whether a given user may see it.

References: server/…/search.service.ts• L74-113 server/…/search.service.ts• L168-198 server/…/search.service.ts• L281-353 server/…/search.service.ts• L302-323 server/…/search.repository.ts• L317-341 server/…/search.repository.ts• L353-397

Machine learning service boundary

The server treats machine learning as a separate service boundary. It sends image or text input for one of three supported tasks: visual search, OCR, or facial recognition. The boundary returns embeddings, text regions, or face boxes with embeddings. The server then validates the result against the current asset and stores it in the database.

The repository can check configured model servers for health and tries healthy servers first. If all configured servers fail, the related job fails and its derived data is not treated as complete. See Machine learning service and Vision and text model capabilities for the model service and its supported capabilities.

References: server/…/machine-learning.repository.ts• L14-75 server/…/machine-learning.repository.ts• L96-187 server/…/machine-learning.repository.ts• L190-225

Change checklist

When these files or sections are changed, remember to consider these:

  • When a derived field or embedding changes, update the matching search source and the query path that consumes it. The job status must still show whether the new value is current.
  • When a model name or embedding dimension changes, plan the cleanup and rebuild of existing smart-search or face-search data.
  • When a queue name or follow-up job changes, update both the queue command mapping and the completion path that starts dependent jobs.
  • When asset visibility or cluster-group scope changes, review every analysis stream and every search scope. Hidden and locked assets do not follow the same path, and a forced face run may target one group or all groups.
  • When sidecar fields or tag rules change, keep read, write, and re-extraction behavior aligned so a sidecar update does not lose locked user values.
  • When person-group matching or cleanup changes, keep detection state, recognition scope, owner records, face assignment, and thumbnail jobs aligned.

Important failure modes

TriggerConsequenceGuard
Machine learning is disabled or all configured servers failSmart search, OCR, or face work is skipped or fails. The related derived result is not marked complete.Feature checks run before queueing, and the model boundary tracks server health and fails only after trying all configured URLs.
A smart-search model changes while an embedding job is runningThe result may not match the database dimension or the active model.The service checks the current model again before storing the embedding.
An asset has no preview fileOCR or face detection cannot produce a model input and the job fails.The asset selection streams only assets with the needed preview for these tasks.
An asset is hidden, locked, stacked, or lacks an embeddingThe related face or duplicate analysis does not run.Each task applies its visibility, stack, and prerequisite checks before model or vector search work.
A sidecar changes after initial extractionApplication metadata can become stale until the sidecar association is checked and metadata extraction runs again.Sidecar checks compare the discovered path with the stored path, and metadata extraction reads the associated sidecar while honoring locked properties.
A face run uses the wrong cluster-group scopeRecognition can miss valid matches or process faces outside the intended user group.Forced recognition passes the selected cluster group through unassignment and face selection.
A face match belongs to another user in the same cluster groupThe asset owner's face may remain without a person record or may be assigned to the wrong owner.Recognition keeps the shared group identity but creates the person record for the asset owner before assigning the face.
Nightly recognition runs before new face work is ready or runs again with no new facesRecognition can waste worker capacity or leave new faces for a later run.The queue waits for thumbnail and face detection completion, then compares the last run with the newest face date before a nightly run.

Key files

FileWhy to read it
server/…/job.service.tsShows how thumbnail completion starts analysis and how smart information starts duplicate detection.
server/…/queue.service.tsShows the queue controls, concurrency setup, manual queue starts, and nightly analysis work.
server/…/asset-job.repository.tsShows which assets qualify for each analysis task and which prerequisites each task needs.
server/…/metadata.service.tsShows how source metadata, sidecars, tags, locations, motion photos, imported face regions, and device details become asset data.
server/…/smart-info.service.tsShows how visual embeddings are created and kept consistent with the active model.
server/…/ocr.service.tsShows how detected text becomes both visible OCR regions and searchable text.
server/…/person.service.tsShows face detection, recognition scheduling, cluster-scoped matching, person grouping, cleanup, and owner-specific person record lifecycle.
server/…/duplicate.service.tsShows how visual matches become duplicate groups and how users resolve those groups.
server/…/search.service.tsShows how metadata, smart, people, and access-scope search paths are combined.
server/…/machine-learning.repository.tsDefines the boundary between background jobs and the separate machine learning service.
server/…/person.service.spec.tsSummarizes the guarantees for face queue scope, cleanup, recognition, and owner-specific person records.