Media artifact lifecycle
The media artifact lifecycle turns image data found in agent activity into safe, durable artifacts that can support observation inference, remain linked to retained memory, replicate between devices, and appear in the local viewer. The current v1 contract accepts PNG, JPEG, and WebP images; it keeps raw pixels out of observation text and exposes only bounded metadata and converted variants at each boundary.
Media contract
The media contract separates a stable reference from private source detail. This lets observations and sync records identify an artifact without carrying bytes, paths, object keys, or event-private provenance.
Canonical media references
The canonical media namespace is metadata.cmem_media_v1. It carries a version, attachment IDs, bounded labels, and an inspected or uninspected state. A reference reports that an artifact belongs to an observation; it does not report whether its bytes are ready. Readiness is resolved when a client asks for the artifact.
The contract keeps labels stable across capture, inference, storage, and viewer display. Labels use an event-and-image form such as event1_image1, while UUID attachment IDs provide durable identity. A manifest can hold up to 32 references; extra references remain on the source event and set overflow.
References: src/…/media-contract.ts• L7-29 src/…/media-contract.ts• L57-69 src/…/media-contract.ts• L136-155 src/…/types.ts• L20-35
Private provenance
Private provenance explains where an attachment came from and how it can be replayed safely. It includes the event identity, source shape, source location within the event, source checksum, and conversion recipe. This data stays in the owner media plane and does not enter generic observations, prompts, logs, analytics, or sync content.
Ownership checks happen before the system reads a private file or exposes owner-only details. A provenance value can validate the shape of a source, but it cannot grant permission to read that source.
References: src/…/media-contract.ts• L71-101 src/…/media-contract.ts• L158-199 src/…/store.ts• L98-106 src/…/MediaRoutes.ts• L121-135
Ingress and capture pipeline
The ingress pipeline removes image bytes from agent-facing text while preserving useful surrounding information. It recognizes typed native image shapes instead of guessing from paths or fetching remote URLs.
Source recognition
The scanner accepts inline base64 images from supported provider shapes and a typed local file locator from a trusted Read result. It replaces retained bytes with a small status descriptor before the event reaches observation storage. Captured bytes remain available to the media worker through the capture boundary, while ordinary text keeps its original meaning.
Capture reserves the size of a local file before it reads the file. A worker-wide source-byte budget makes captures wait without holding image buffers when other captures already use the budget. This limits memory use across sessions while allowing queued captures to proceed.
The scanner also bounds its walk through nested payloads and bounds the event size. If it cannot safely inspect a deep or oversized value, it replaces the possible image-bearing value with a byte-free descriptor rather than allowing hidden bytes into stored text.
References: src/…/media-ingress.ts• L19-47 src/…/media-ingress.ts• L75-137 src/…/media-ingress.ts• L157-179 src/…/media-ingress.ts• L297-338 src/…/media-ingress.ts• L387-413 src/…/capture.ts• L239-333 src/…/source-budget.ts• L3-40
Privacy and event identity
Capture runs only for accepted, non-private activity. Images inside memory-private regions are skipped before their pixels are read, stored, uploaded, or sent for inference. A trusted local file must also remain inside the project boundary and match the tool input that named it.
The lifecycle uses a stable platform event identity when one exists. Platform tool-use IDs are namespaced by platform and immutable content-session identity. For replayable transcript records it uses the transcript identity and checksum. The replay key combines that event identity with the source location, source checksum, and recipe, so a repeated event reuses its attachment while changed bytes fail instead of silently replacing the old artifact.
This diagram shows how an agent event becomes either a retained artifact reference or a safe text-only failure descriptor.
flowchart LR
A[Agent tool event] --> B[Ingress scan]
B --> C{Private or unsupported?}
C -->|Yes| D[Byte-free text descriptor]
C -->|No| E[Stable event identity]
E --> F{Capture enabled?}
F -->|No| D
F -->|Yes| G[Read and validate source bytes]
G --> H[Durable artifact conversion]
H --> I[Attachment reference on source event]
H --> J[Bounded failure code]
References: src/…/capture.ts• L107-137 src/…/capture.ts• L181-235 src/…/capture.ts• L239-333 tests/…/capture.test.ts• L138-186 tests/…/capture.test.ts• L203-289
Conversion and inference
Conversion creates the two forms needed by the product: a full viewer image and a smaller image for visual observation inference. Inference is independent from capture, so an artifact can remain durable and uninspected when the model or endpoint cannot accept images.
Artifact conversion
The converter accepts only PNG, JPEG, and WebP still images within the source and decoder bounds. It checks the actual bytes and image dimensions, rejects animation and invalid pixels, applies orientation, removes source metadata, and writes lossy WebP viewer and inference variants. Oversized accepted frames are fitted to the contract bounds instead of being cropped. The source is not retained as a stored artifact.
The viewer variant preserves the full frame within its size limit. The inference variant fits the full frame to the smaller model-input bound. Both variants carry checksums, dimensions, byte length, MIME type, recipe, and encoder version so later reads and cloud transfers can verify what they received.
References: src/…/converter.ts• L26-42 src/…/converter.ts• L66-96 src/…/converter.ts• L166-218 src/…/converter.ts• L230-298 docs/media-contract-v1.md• L38-60
Visual inspection
Visual inspection happens only on the current observation turn. The worker sends the retained inference derivative as a WebP data URL after it qualifies the selected model or gateway for image input and text output. Earlier history and later turns remain text-only, so pixels do not become conversation history.
If inference is disabled, unavailable, or not qualified, the worker sends the normal text request. The artifact stays retained and uninspected. The same result applies to a text-only retry after a gateway rejects an image request.
References: src/…/inference.ts• L5-13 src/…/inference.ts• L82-153 src/…/media-capability.ts• L12-24 src/…/media-capability.ts• L310-334 docs/media-contract-v1.md• L182-228 docs/media-contract-v1.md• L272-315
Attachment linkage
The observer receives bounded labels with the current turn's image parts. An accepted observation can name those labels in its attachment block. The system validates each label against the request and links only images from one source event; it never assigns an image to an observation by guessing after a rejected reference.
An image can link to more than one observation. An image that receives no valid link remains attached to its source event and remains uninspected. A successful link upgrades inspection state, but a later merge cannot downgrade an already inspected reference.
References: src/…/inference.ts• L155-265 docs/media-contract-v1.md• L204-211 docs/media-contract-v1.md• L230-270 src/…/linkage.ts• L42-89
Durable artifact storage
Local storage keeps artifact files separate from observation text and manages them through durable database state. The database records identity, provenance, conversion state, variant metadata, event references, and cleanup work; the filesystem holds only the converted WebP variants.
Local retention
A captured artifact starts as a converting record and becomes ready only after both variants are published and their metadata is committed. Publication is treated as one unit, so a reader does not receive only one half of an artifact. A lease and reconciliation pass recover abandoned conversions and remove temporary files after failures.
The store uses database uniqueness to resolve concurrent replay attempts. It does not use a process-memory ID or a filename as artifact identity. A repeated event therefore returns the same durable reference across worker restarts, while identical bytes from a different event keep separate event provenance.
Second-device artifacts start as unresolved placeholders. They become ready only after cloud bytes for both variants pass validation and are published locally. Replica rows remain marked as cloud-owned, so the receiving device never uploads them.
References: src/…/store.ts• L29-73 src/…/store.ts• L85-119 src/…/store.ts• L132-213 src/…/store.ts• L216-260
Deletion and cleanup
An artifact remains available while an event or observation still references it. Deleting an observation removes only that observation link; deleting the last relevant event or session queues cleanup and tombstones the artifact. Cleanup is durable and retryable, so a storage outage does not lose the deletion intent.
The same reference checks protect shared artifacts. A file is not removed while another event or observation still uses it. A late replay cannot recreate an artifact after its source event has been deleted.
References: src/…/store.ts• L263-327 src/…/media.ts• L23-67 src/…/media.ts• L136-151 tests/…/capture.test.ts• L384-434
Observation linkage
Observation linkage joins retained media to searchable memory without putting image bytes into the memory record. The observation stores a validated manifest and the local database stores junctions that preserve the relationship between an observation, an event, and an attachment.
Manifest association
When an observation is stored, new media references merge with its existing media namespace. Existing order stays stable, other metadata namespaces remain unchanged, and inspection can move from uninspected to inspected. If the observation is full, the new reference stays at event level rather than being silently dropped.
Native rows update their sync revision when their media manifest changes. Replica rows are not mutated by local inference; their references remain event-level until the owning device publishes the accepted observation state.
When a turn stores observations without naming its images, the retained images from that turn link to each observation as uninspected. If the turn stores no observation, the event waits for the next storing turn with the same session and prompt. This preserves capture linkage without treating an image as inspected merely because it was present in the activity.
References: src/…/linkage.ts• L42-89 src/…/linkage.ts• L112-176 src/…/linkage.ts• L179-290 docs/media-contract-v1.md• L398-434
Replay recovery
The media event stores both the text result and the observation IDs produced by that result. The store commits these results with observation and media-link writes before the worker confirms its in-memory batch. If the worker stops after the database commit, a replay recovers the result without another model call.
This boundary keeps artifact linkage and observation retention aligned. A failed link write can leave the text observation durable without claiming media linkage, while an uncommitted event remains eligible for normal retry.
References: src/…/linkage.ts• L292-318 src/…/media.ts• L111-118 tests/…/inference-observation-turn.test.ts• L517-542 tests/…/inference-observation-turn.test.ts• L631-667
Cloud replication
Cloud replication moves converted artifacts independently from canonical text sync. Text sync can publish a stable attachment ID before its bytes arrive, while a separate media worker uploads the owner copy and resolves it on other devices.
Owner upload
The owner uploads the canonical WebP variant with a manifest that contains safe provenance, checksums, dimensions, recipe, and encoder version. The local file locator is removed before upload. The cloud service creates the other retained variant and returns a descriptor that the client validates.
Uploads require both media capture and cloud sync to be enabled. Deletion work drains independently so a setting that stops new uploads does not strand cloud cleanup. Upload state has its own durable queue and retry schedule. Network errors, temporary service errors, authorization problems, quota limits, and not-ready results back off without losing the work. Deterministic format, checksum, manifest, or tombstone errors become bounded terminal failures.
References: src/…/cloud.ts• L16-30 src/…/cloud.ts• L46-88 src/…/cloud.ts• L193-252 src/…/cloud.ts• L285-445 tests/…/media-sync.test.ts• L270-329 tests/…/media-sync.test.ts• L498-629
Replica resolution
When another device receives a synced observation, it creates a placeholder for each referenced attachment. The placeholder carries the canonical ID and manifest relation but no bytes, and it is never uploaded by the receiving device. The first viewer read asks the owner cloud plane for the descriptor and both variants.
The receiving device verifies every downloaded variant against the cloud descriptor before publishing it locally. Missing credentials, an offline owner, or an unuploaded artifact remain a pending download rather than a false conversion failure. A newer synced revision can remove or restore a replica without deleting the owner's cloud copy.
References: src/…/cloud.ts• L448-504 src/…/store.ts• L216-260 src/…/replica.ts• L6-24 src/…/replica.ts• L42-88 tests/…/media-sync.test.ts• L287-329 tests/…/media-sync.test.ts• L424-474
Viewer delivery
Viewer delivery exposes safe metadata first and image bytes only through private local routes. The viewer uses the same attachment ID from the observation manifest to request readiness, a thumbnail derivative, or the full viewer image.
Private media reads
The media routes require the trusted local worker page boundary and reject forwarded or cross-site reads before storage access. Generic metadata contains state, dimensions, checksums, variant sizes, recipe, and encoder version, but no source locator, object key, or event identity. Owner source details require an explicit request and use the same private guard.
Variant reads return WebP bytes only after the artifact is ready and the stored checksum matches. The route checks state again after the file read so a deletion that wins during I/O cannot return bytes from a deleted artifact.
References: src/…/MediaRoutes.ts• L13-55 src/…/MediaRoutes.ts• L57-70 src/…/MediaRoutes.ts• L91-165 tests/…/media-routes.test.ts• L92-128 tests/…/media-routes.test.ts• L131-209
Viewer states
The viewer treats processing and replica download as pending states, not as permanent failures. It polls only those states for a bounded time, then offers a manual retry. Rejection states describe source or image limits; read errors describe a temporary route or storage problem.
Observation cards show bounded thumbnails and an overflow count. The lightbox requests the full viewer variant and can show inspection status. Source paths appear only after the owner explicitly asks for source details, while synced replicas show that only a converted copy is present.
References: src/…/media.ts• L3-31 src/…/media.ts• L41-85 src/…/media.ts• L101-115 src/…/ObservationMedia.tsx• L39-177 tests/…/observation-media.test.tsx• L56-120 tests/…/observation-media.test.tsx• L145-206
The parent page Capture and memory lifecycle explains how agent activity enters retained memory. The related CLI hook event ingestion, Transcript and observation processing, Memory ingest and reinforcement, Native memory bridge, Remote worker capture, History and prompt import, and Automatic context clearing pages describe the surrounding activity and memory flows.
Change checklist
When these files or sections are changed, remember to consider these:
- Keep src/…/media-contract.ts aligned with every local, cloud, and viewer validator. A new manifest field or error code must fail or pass consistently at all boundaries.
- Keep capture labels, event identity, replay keys, and observation request labels aligned. A mismatch can create duplicate artifacts or attach an image to the wrong observation.
- Keep the privacy gate ahead of file reads and conversion. A source inside a private memory region must not reach local storage, inference, cloud upload, or viewer metadata.
- Keep the source-byte budget and conversion admission limits aligned with the worker's memory and time limits. A queue change must not allow waiting captures to retain unbounded file buffers.
- Keep the viewer and inference limits aligned with the gateway request limits. A client that sends a derivative outside those bounds can force a text-only retry and leave the image uninspected.
- Keep event-level references and observation junctions consistent with manifest overflow. A reference that does not fit an observation must remain recoverable from its source event.
- Keep capture-only waiting events scoped to the same session and prompt. A broad match can attach an image to an unrelated observation.
- Keep upload state separate from text sync state. Text must not wait for media, and a replica must never re-upload a cloud-owned artifact.
- Keep deletion ordering ahead of late upload completion. A cloud copy that lands after local deletion needs a durable cloud deletion.
- Keep generic feeds free of provenance and source paths. Owner-only details and raw media reads need the same local-page trust boundary.
Important failure modes
| Trigger | Consequence | Guard |
|---|---|---|
| Capture is disabled | The event keeps a bounded disabled descriptor and no artifact is stored. | Capture and inference are separate flags; the capture path records converter_disabled. |
| Image bytes are inside a private memory region | The image is not read or retained. | Privacy filtering checks stripped event fields before candidate bytes are read. |
| Concurrent file captures exceed the worker source-byte budget | Later captures wait without holding image buffers. | The source budget reserves file size before reading and releases it when capture ends. |
| A replay uses changed bytes at the same event and source location | The old artifact remains and the replay is rejected. | The replay key and source checksum are durable. |
| Model capability is unavailable or unqualified | The observation runs as text-only and the image stays uninspected. | Capability resolution fails closed and does not retry inference automatically. |
| An observation response names an invalid, unknown, or cross-event label | No image is linked by that rejected reference. | Linkage validates label grammar, request membership, and source-event scope. |
| Conversion stops after temporary files are written | The artifact remains non-ready until reconciliation removes abandoned files. | Conversion leases, atomic publication, and durable cleanup jobs recover the state. |
| A second device receives text before the owner uploads bytes | The viewer shows a pending download rather than a missing permanent image. | Replica placeholders resolve lazily and treat not-ready cloud results as pending. |
| Cloud upload acknowledgement is lost after the cloud commits | The next attempt may repeat the upload but does not create a second artifact. | The stable attachment ID makes upload replay idempotent. |
| A linked observation is deleted while another event still references the artifact | The artifact remains readable. | Cleanup checks both event references and observation links before tombstoning. |
| A foreign page or forwarded request asks for media | No metadata or image bytes are returned. | Media routes require the trusted loopback socket and own-page origin. |
Key files
| File | Why to read it |
|---|---|
| docs/media-contract-v1.md | Defines the v1 media contract, limits, inference rules, linkage rules, and cloud boundary. |
| src/…/media-contract.ts | Defines the shared manifest, provenance, failure, recipe, and limit contract. |
| src/…/media-ingress.ts | Shows how native image shapes become bounded capture candidates and byte-free event text. |
| src/…/capture.ts | Shows the privacy, identity, replay, source-validation, and source-byte admission boundary for capture. |
| src/…/source-budget.ts | Shows the worker-wide memory budget for local file sources waiting for capture. |
| src/…/converter.ts | Shows the supported source checks and the viewer and inference derivative model. |
| src/…/inference.ts | Shows how one observation turn selects bounded derivatives and validates response labels. |
| src/…/store.ts | Shows durable artifact identity, local retention, replica publication, variant reads, and cleanup. |
| src/…/linkage.ts | Shows how media manifests, capture-only links, and observation links are committed with memory rows. |
| src/…/cloud.ts | Shows independent cloud upload, deletion, retry, and second-device resolution. |
| src/…/replica.ts | Shows how synced observation references become lazy local replica placeholders. |
| src/…/MediaRoutes.ts | Shows the private metadata, owner-details, and variant delivery boundary. |
| src/…/media.ts | Shows how route results become viewer states and bounded retry behavior. |
| src/…/ObservationMedia.tsx | Shows thumbnail and lightbox delivery without exposing private provenance by default. |