Asset lifecycle and media processing
An asset is a photo, video, or related file that the server records, enriches, and serves to library users. The lifecycle starts with an upload or a file found in an external library. It then adds metadata and derived media in background work before the asset is viewed, streamed, downloaded, restored, or removed. See Server API and application runtime for the wider server flow, Identity, access, and sharing for permission boundaries, Automated asset analysis for analysis that builds on stored asset metadata, Data model and persistence for durable asset state, and Web library workflows for the user-facing library use of these assets.
Asset lifecycle
The asset lifecycle connects file storage, the asset record, background jobs, and access checks. The original file remains the source of truth while the server keeps a record for dates, type, checksum, visibility, links, and generated files.
The main lifecycle stages are:
- Ingestion: Accept an uploaded file or import a path from an external library.
- Registration: Create or update the asset record and link sidecar files or live-photo motion files.
- Enrichment: Read file tags and stream data. Store dates, dimensions, location, camera data, and video data.
- Derivation: Generate thumbnails, previews, full-size images, edited variants, and encoded video when policy requires them.
- Delivery: Serve originals, derived images, video playback, HLS streams, asset files, or ZIP archives.
- Retention: Move assets to trash, restore them, or delete database records and tracked files after the retention period.
This diagram shows how an asset moves from a source file to user-facing media and then through retention.
flowchart LR
A[Upload or external library file] --> B[Asset record]
B --> C[Metadata extraction]
C --> D[Thumbnail and preview generation]
C --> E[Video conversion when needed]
D --> F[View and download]
E --> F
B --> G[Trash]
G --> H[Restore]
G --> I[Permanent deletion]
Asset records and file variants
The asset record identifies the source file and its state. Generated outputs are tracked as asset files so readers can distinguish the original from thumbnails, previews, full-size copies, edited copies, sidecars, and encoded video.
| Asset data | Purpose |
|---|---|
| Original path and name | Locate and name the source file. |
| Checksum | Detect duplicate uploads within the relevant owner and library scope. |
| Type and visibility | Control how the asset appears and which media paths can be used. |
| Dates and dimensions | Support timeline display, search, and media layout. |
| Asset files | Track derived images, sidecars, and encoded video. |
| Deleted and offline state | Distinguish trash or missing external files from active media. |
An asset file also records whether an image is edited, progressive, or transparent. This lets delivery and regeneration use the correct variant without treating every stored file as an original.
References: server/…/asset.table.ts• L23-148 server/…/asset-file.table.ts• L16-49 server/…/asset.util.ts• L16-31 server/…/media.service.ts• L776-824
Access-controlled asset behavior
Asset operations apply permission checks before reading or changing records and files. The same asset can therefore expose different data paths: a viewer may see a thumbnail, a downloader may receive the original, and a shared link may receive edited media with selected metadata removed.
The server also protects special relationships during changes. A live-photo motion file can be hidden while it belongs to a still image, and shared-link responses can omit owner, people, or EXIF data according to the link settings.
References: server/…/asset.service.ts• L67-98 server/…/asset-media.service.ts• L229-315 server/…/asset.util.ts• L145-190
Asset ingestion
Asset ingestion creates the record that connects a source file to its owner and later processing jobs. Uploads use a temporary upload path before registration, while external libraries register an existing path without copying the file into the managed media area.
Uploaded assets
The upload path accepts an asset file and an optional XMP sidecar. It classifies files by supported image, video, or sidecar type and rejects other upload types. It checks upload permission, records the checksum and file dates, applies initial flags such as favorite or visibility, and queues metadata extraction.
The same upload boundary supports a checksum check before upload and a duplicate result after a unique checksum conflict. It also enforces the user quota, links an optional live-photo video, adds an upload made through a shared link to that link, and schedules cleanup if registration fails.
Profile images use a separate accepted image set and are reduced to the configured profile thumbnail format. They do not enter the asset record and media-derivation lifecycle.
References: server/…/asset-media.controller.ts• L51-89 server/…/asset-media.service.ts• L59-227 server/…/mime-types.ts• L102-194 server/…/profile-image.ts• L14-40
External library assets
An external library is a set of user-managed import paths. A scan or file watcher finds supported files and creates asset records that point to those paths. The server does not treat the external path as a managed upload, so the source stays in place and can later become offline if it is missing or inaccessible.
After an import, sidecar discovery and metadata extraction bring the external asset into the same enrichment path as an upload. Later scans compare file access and modification time to decide whether to mark the asset offline, restore it online, or refresh its metadata.
References: server/…/library.service.ts• L91-163 server/…/library.service.ts• L252-294 server/…/library.service.ts• L401-432 server/…/library.service.ts• L477-613
Ingestion follow-up jobs
Ingestion is intentionally separate from media generation. The initial request records enough data for the asset to exist, then queues work for metadata, sidecars, thumbnails, and video conversion. This keeps file acceptance independent from the time needed to inspect and transform large media.
The queue service exposes these stages as distinct operational queues. A manual or scheduled run can refresh one stage without rebuilding all other derived data.
References: server/…/asset-media.service.ts• L169-195 server/…/library.service.ts• L422-432 server/…/queue.service.ts• L199-230
Metadata extraction
Metadata extraction turns file tags and media stream information into searchable asset facts. It reads the original file and any sidecar, chooses trusted values for dates and related fields, and stores EXIF, audio, video, and keyframe data for later services.
The extraction pass combines several sources with clear precedence:
- Sidecar dates take priority over dates in the original file. Sidecar time-zone data follows the selected date.
- Video probing supplies stream dimensions, rotation, duration, codec data, and packet timing when file tags do not provide enough detail.
- HEIF orientation uses the container rotation value when the normal EXIF orientation is missing or unreliable.
- File creation and modification times provide a date when no usable media date exists.
The resulting facts include capture and modification dates, time-zone information, image size, orientation, projection, color data, camera details, GPS coordinates, optional reverse-geocoded location, descriptions, ratings, tags, and video or audio data. Keyframe timing is stored for later playback work.
Metadata can also reveal relationships. A matching live-photo identifier links the still image and motion asset, hides the motion asset from the normal timeline, and removes it from albums while the link exists. Tagged faces and other analysis data can be applied as part of this pass.
References: server/…/metadata.service.ts• L215-418 server/…/metadata.service.ts• L1004-1152 server/…/media.stub.ts• L51-495
Sidecar synchronization
Sidecars provide a file-based exchange for selected asset metadata. The server discovers sidecars beside an original or from an already tracked sidecar file, and it gives sidecar date values priority when it extracts metadata.
When a user changes supported asset fields, the server marks those fields as controlled values and queues a sidecar write. A later extraction preserves controlled values instead of silently replacing them with the source file values.
References: server/…/metadata.service.ts• L420-554 server/…/asset.service.ts• L489-522 server/…/mime-types.ts• L141-143
Motion photo extraction
Some image files contain a motion video inside the image container. Metadata extraction identifies that relationship, creates or reuses a hidden video asset by checksum, writes the extracted video when needed, and sends it through the same metadata and encoding path as other videos.
This keeps the still image as the visible library item while preserving the motion part for live-photo viewing and download. If a later extraction finds a different motion video, the old linked video is sent through normal asset deletion.
References: server/…/metadata.service.ts• L175-204 server/…/metadata.service.ts• L397-418 server/…/metadata.service.ts• L669-817
Derived media
Derived media makes source files usable across browsers, devices, and view sizes. The media worker creates the needed output, records its path as an asset file, and removes replaced outputs through a file-delete job.
Images and previews
Image processing can extract an embedded preview from a RAW file or decode the original into a configured image format. It creates a thumbnail and a preview. It can also create a full-size image when the original is not suitable for direct web use or when configuration requires one.
Image type rules separate web-supported images from formats that need conversion. They also identify RAW files, possibly animated images, and formats that can contain transparency. These categories guide whether the server extracts an embedded image, preserves transparency, or creates a browser-ready variant.
Video and animated images use a representative frame or transcode path to create their thumbnail and preview. Edited image variants are generated separately. The server stores a thumbhash with the asset so clients can show a quick visual placeholder while the real image loads.
References: server/…/media.service.ts• L191-230 server/…/media.service.ts• L256-389 server/…/media.service.ts• L493-537 server/…/mime-types.ts• L37-100 server/…/storage.core.ts• L120-134
Image edits
Image edits are stored as a sequence of actions rather than as a replacement of the original. The edit request validates the image and crop bounds, then queues edited thumbnail generation. The original and edited files can therefore be served independently.
When a crop changes the visible area, face and OCR visibility are recalculated against the new image bounds. This keeps related analysis aligned with the displayed image.
References: server/…/asset.service.ts• L524-608 server/…/media.service.ts• L154-190 server/…/media.service.ts• L826-853
Video conversion
Video conversion creates an encoded video only when the configured policy requires a codec, pixel format, resolution, bitrate, audio, or container change. The result is tracked separately from the original, so the original remains available for download and later policy changes.
The conversion decision uses the stored video and audio metadata. It also accounts for rotation, target size, HDR tone mapping, and the available hardware path. The conversion path can use hardware acceleration and falls back to software encoding when an accelerated attempt fails. If the current policy no longer needs an encoded copy, the old derived file is removed.
References: server/…/media.service.ts• L540-638 server/…/media.service.ts• L640-731 server/…/media.ts• L32-45 server/…/media.ts• L145-220
Media delivery
Media delivery maps an access-checked request to the best available file. It favors a derived file when one exists, redirects suitable full-size image requests to the original, and downgrades a missing full-size preview to a preview instead of failing when that fallback is valid.
| Request | Result |
|---|---|
| Original | Original or edited source file with download permission. |
| Thumbnail or preview | Stored image variant with view permission. |
| Full-size image | Full-size derived file, original web-safe image, or preview fallback. |
| Video playback | Encoded video when available, otherwise the original video. |
| Asset file | A tracked file such as a sidecar or derived output. |
| Archive | A streamed ZIP of selected originals or edited files. |
References: server/…/asset-media.controller.ts• L92-177 server/…/asset-media.service.ts• L229-315 server/…/asset-file.controller.ts• L22-71
Video streaming
The HLS interface provides adaptive video playback through a short-lived streaming session. The API worker creates playlists and waits for requested segments, while the media worker starts or pauses a transcode process and writes the requested segments into a session directory.
This sequence shows the relationship between a client request, the API worker, and the media worker.
sequenceDiagram
participant C as Client
participant A as API worker
participant M as Media worker
C->>A: Request main playlist
A->>M: Start HLS session
M-->>A: Session ready
A-->>C: Main and variant playlists
C->>A: Request segment
A->>M: Request segment if missing
M-->>A: Segment ready event
A-->>C: Segment file
C->>A: End session
A->>M: Stop and clean session
The session has a lease and an inactivity cleanup path. Heartbeats extend active sessions and provide enough demand information for backpressure, which prevents the encoder from running far ahead of the viewer.
References: server/…/video-stream.controller.ts• L30-105 server/…/hls.service.ts• L49-127 server/…/transcoding.service.ts• L60-180 server/…/media.ts• L173-220
Downloads and archives
Single-asset downloads return the original or edited source with a safe content type and cache policy. Archive downloads first compute groups that fit the configured archive size, then stream a ZIP with unique names when several assets share a file name.
Live-photo downloads include the linked motion file when the user preference allows it. Access is checked at the requested asset or album boundary before the server reads source paths.
References: server/…/download.controller.ts• L16-40 server/…/download.service.ts• L20-131
Asset deletion and retention
Deletion separates user-visible trash state from physical cleanup. A normal delete marks assets as trashed. Restore clears that state, while empty-trash, forced deletion, or the retention job queues permanent deletion.
The retention check processes expired trash in batches and logs how many assets it queues for permanent deletion. Permanent deletion removes the asset record, updates quota usage for managed assets, repairs stack membership, and queues deletion of tracked derived files and the original when the file is owned by the managed store. Offline external assets keep their source path untouched.
The same cleanup handles linked motion files when no other asset uses them. The deletion worker combines the request to remove files with the asset's offline state, so retention cleanup can use one path while external-library sources remain on disk.
References: server/…/asset.service.ts• L292-381 server/…/trash.service.ts• L35-60 server/…/library.service.ts• L363-398
File cleanup
File cleanup is asynchronous and idempotent at the workflow level. Media regeneration first updates the tracked file set, then queues old paths for removal. Asset deletion uses the same file-delete job for thumbnails, previews, edited variants, encoded video, sidecars, and managed originals.
References: server/…/media.service.ts• L776-824 server/…/storage.service.ts• L137-155 server/…/asset-media.service.ts• L299-333
Change checklist
When these files or sections are changed, remember to consider these:
- Keep the upload checksum contract aligned between the pre-upload check, the asset uniqueness rule, and duplicate responses.
- Keep every generated
AssetFileTypevariant aligned across path generation, lookup, media delivery, regeneration, and deletion. - Keep file-type classification aligned across upload validation, asset type detection, profile images, metadata extraction, and content-type responses.
- If metadata field precedence changes, review sidecar discovery, sidecar writes, locked fields, video probing, and external-library refresh behavior together.
- If date or orientation handling changes, review timeline dates, image dimensions, video rotation, HEIF handling, and tagged-face coordinates together.
- If video metadata or stream fields change, review stored keyframes, encoded-video policy, HLS playlist generation, and segment timing.
- If image conversion rules change, review RAW extraction, web support classification, transparency handling, full-size generation, and client format support together.
- If trash or library deletion changes, preserve the distinction between managed files that can be removed and external files that must remain on disk.
- If shared-link media behavior changes, review original downloads, thumbnail redirects, edited media selection, and metadata redaction together.
Important failure modes
| Trigger | Consequence | Guard |
|---|---|---|
| Upload checksum matches an existing asset | The new file is rejected as a duplicate and the temporary upload is cleaned up. | Pre-upload checksum checks and the unique owner/library checksum rule. |
| Upload file type is not in the shared media classification | The upload is rejected before asset registration. | Central image, video, sidecar, and profile type maps. |
| No usable media date exists | The asset can appear at the wrong time if file-time fallback is not applied. | Use the ordered metadata date list, then fall back to file creation or modification time. |
| Metadata is not ready for a video | Conversion or HLS setup cannot build a valid stream. | Queue metadata extraction first and report missing metadata to the job or stream caller. |
| Requested full-size media does not exist | The request falls back to a preview when the endpoint supports that path. | Media lookup checks the full-size file and selects a valid target. |
| Source format cannot preserve transparency in the configured output | Transparent pixels can be lost in a generated image. | Detect transparency-capable sources and warn when the output format does not support it. |
| Accelerated video conversion fails | The encoded output may fail unless a fallback path succeeds. | Retry with software decoding and then without hardware acceleration. |
| External library source disappears | The asset becomes offline or is removed from the library view without deleting the source path. | The deletion worker suppresses original and sidecar removal when the asset is offline. |
| Motion-photo metadata points to a changed embedded video | The old hidden motion asset can remain linked or consume storage. | Match the extracted checksum, relink the still image, and queue the old motion asset for deletion. |
| Asset is permanently deleted while in a stack or live photo | A stale stack or unreferenced motion asset can remain. | Deletion repairs the stack and queues unused linked motion for deletion. |
| HLS session becomes inactive or expires | The transcode process and temporary segments can remain until cleanup. | Heartbeats extend active leases and cleanup ends expired sessions. |
Key files
| File | Why to read it |
|---|---|
| server/…/asset-media.controller.ts | Shows the public upload, original, thumbnail, playback, and duplicate-check interfaces. |
| server/…/asset-media.service.ts | Explains upload registration, duplicate handling, media selection, and original delivery. |
| server/…/asset.service.ts | Connects asset updates, jobs, edits, trash, and permanent deletion. |
| server/…/library.service.ts | Shows how external paths enter the asset lifecycle and become offline or online. |
| server/…/metadata.service.ts | Defines metadata precedence, media probing, sidecar synchronization, motion-photo extraction, and metadata relationships. |
| server/…/media.service.ts | Defines thumbnail, edited media, migration, and video conversion work. |
| server/…/mime-types.ts | Defines the shared file categories that control upload acceptance and media conversion choices. |
| server/…/profile-image.ts | Shows the separate path for creating user profile thumbnails. |
| server/…/hls.service.ts | Shows the API side of on-demand video streaming. |
| server/…/transcoding.service.ts | Shows the media-worker side of HLS sessions and segment production. |
| server/…/download.service.ts | Explains archive sizing and streamed ZIP downloads. |
| server/…/asset.table.ts | Defines the durable asset state that the lifecycle updates. |
| server/…/asset-file.table.ts | Defines how derived and related files are tracked. |
| server/…/storage.core.ts | Defines the shared path model for managed media, generated images, encoded video, and HLS sessions. |