Introducing a new way to review your PRs.

Read the announcement

immich-app / immich

High performance self-hosted photo and video management solution.

View on GitHub

21 documents · Updated Oct 19, 2026

docs / native-media-core

Native media core

The native media core is a Rust library that gives the mobile clients one place for low-level image work. It converts decoded pixels into Flutter-readable RGBA data, applies image orientation, and rebuilds small ThumbHash previews while platform code manages media access and memory ownership.

System role

The core sits between Flutter image requests and the native media APIs on Android and iOS. It keeps pixel rules consistent where both platforms need the same result, while each platform remains responsible for reading local media and exposing it through the Flutter bridge.

flowchart LR
    A[Flutter image request] --> B[Platform image bridge]
    B --> C[Platform media source]
    C --> D[Native media core]
    D --> E[RGBA buffer and image metadata]
    E --> F[Flutter image frame]
    A --> G[ThumbHash value]
    G --> D

References: native/README.md• L3-10 mobile/…/local_image_api.dart• L14-30 mobile/…/image_request.dart• L119-163

Media capabilities

The core supports three media operations. Together they cover the path from platform-decoded pixels to a frame that Flutter can display.

CapabilityPurposeMobile result
Pixel conversionChanges Android 10-bit RGBA pixels into 8-bit RGBA pixelsCorrect colors for high-bit-depth image sources
Orientation transformApplies all EXIF orientation values, including flips and quarter turnsA dense pixel buffer with the correct width and height
ThumbHash decodeRebuilds a small RGBA preview from a ThumbHash stringA fast placeholder while the full image loads

The core accepts image dimensions and row stride as part of the pixel contract. This lets it handle padded source rows and return a compact output buffer without making Flutter understand platform bitmap layouts.

References: native/…/convert.rs• L1-14 native/…/rotate.rs• L1-35 native/…/thumbhash.rs• L8-25 native/…/bitmap.rs• L34-43

Binding boundary

The Flutter package builds the Rust FFI library for the mobile target and exposes generated Dart bindings. The Rust FFI layer provides the C entry points and Android JNI entry points; platform code calls those entry points and returns buffer addresses plus image metadata through the existing image bridge.

The binding boundary also defines ownership. A successful native operation returns memory that the caller must release after Flutter creates its image buffer. An invalid input, unsupported bitmap format, allocation failure, or native error returns no usable buffer so the platform can use its fallback path or report an error.

References: native/…/build.dart• L7-29 native/…/ffigen.yaml• L1-9 native/…/lib.rs• L5-44 mobile/…/image_request.dart• L119-145

Android image path

Android uses the core for bitmap work that needs direct pixel access. The platform decodes an image, checks whether EXIF orientation still needs to be applied, and sends a supported bitmap to the native transform; the returned buffer becomes a Flutter raw RGBA image.

The path has safe fallbacks. A 10-bit bitmap can use native conversion, other formats can use the platform's 8-bit conversion, and unsupported native operations return control to the platform path. Raw image orientation is read before decoding because some Android raw-media paths do not apply EXIF orientation themselves.

sequenceDiagram
    participant F as Flutter
    participant A as Android image bridge
    participant R as Rust core
    F->>A: Request image or placeholder
    A->>A: Decode bitmap and inspect orientation
    A->>R: Convert or rotate pixels
    R-->>A: Buffer address and dimensions
    A-->>F: Pointer, width, height, row bytes
    F->>F: Build RGBA image and free buffer

References: mobile/…/NativeImage.kt• L5-27 mobile/…/LocalImagesImpl.kt• L76-113 mobile/…/LocalImagesImpl.kt• L224-259 mobile/…/RawOrientation.kt• L45-67

iOS ThumbHash path

iOS uses the shared Rust symbol for ThumbHash decoding and keeps normal image decoding in the platform image stack. The iOS bridge base64-decodes the value, asks the core for RGBA pixels, and sends the pointer and dimensions through the same Flutter image contract.

The shared ThumbHash decoder returns the preview aspect ratio and supports hashes with or without alpha. The mobile loader then treats the result as a raw image rather than as an animated or encoded image.

References: mobile/…/NativeCore.swift• L3-21 mobile/…/LocalImagesImpl.swift• L40-54 mobile/…/thumbhash_image_request.dart• L3-20 native/…/thumbhash.rs• L16-52

Flutter image contract

The local image API is the shared contract between Flutter and the platform bindings. It carries either encoded bytes or decoded pixel data, plus the dimensions and row byte count needed to construct a Flutter image; ThumbHash requests use the same decoded-pixel form.

Flutter owns the final handoff. It wraps the native memory as an immutable buffer, creates a raw RGBA image descriptor, and frees the native pointer in a finally path. Cancellation therefore affects both the request and the buffer lifetime rather than only stopping the platform task.

References: mobile/…/local_image_api.dart• L14-30 mobile/…/image_request.dart• L41-79 mobile/…/image_request.dart• L119-163

Build and version boundary

The Rust workspace contains the shared media logic and its FFI adapter. The Flutter package builds that adapter for each supported Android and iOS target, then uses generated C and Dart bindings so application code does not depend on Rust types.

The core exposes a version value for a small runtime compatibility check. Changes to the Rust ABI must flow through the generated header and Dart bindings before the mobile package can use them.

References: native/Cargo.toml• L1-4 native/…/lib.rs• L1-7 native/…/lib.rs• L16-39 native/…/native_core.dart• L1-12

Change checklist

When these files or sections are changed, remember to consider these:

  • Keep the FFI header asset name and the generated Dart binding asset name aligned between native/native_core/hook/build.dart and native/…/ffigen.yaml.
  • Keep the pointer metadata contract aligned across mobile/pigeon/local_image_api.dart, both platform image bridges, and mobile/lib/infrastructure/loaders/image_request.dart.
  • Keep pixel format and row-byte assumptions aligned between native bitmap transforms and Flutter's PixelFormat.rgba8888 descriptor.
  • Keep EXIF orientation handling aligned with the Android decode paths so raw images are not left unrotated or rotated twice.
  • Update the ThumbHash decoder compatibility samples when its output changes, because the tests compare decoded dimensions and pixel values with existing mobile placeholders.

Important failure modes

TriggerConsequenceGuard
Native bitmap format is not supportedNo native buffer is returnedAndroid falls back to a platform conversion path
EXIF orientation is applied twiceThe image appears flipped or rotatedOnly raw decode paths report an orientation for the native transform
ThumbHash base64 or payload is invalidPlaceholder loading failsBoth platform bridges report a failed request instead of building an image
Returned pixel pointer is not freed after Flutter consumes itNative memory grows with image requestsFlutter frees decoded buffers after creating the immutable buffer
FFI header or generated bindings are staleThe mobile build or native call contract breaksRegenerate the header and Dart bindings after ABI changes

Key files

FileWhy to read it
native/…/rotate.rsDefines the shared orientation model and output dimensions
native/…/convert.rsDefines high-bit-depth pixel conversion
native/…/thumbhash.rsDefines ThumbHash preview reconstruction
native/…/bitmap.rsShows the native bitmap-to-buffer boundary
native/native_core/hook/build.dartShows how Flutter builds the Rust library for mobile targets
mobile/pigeon/local_image_api.dartDefines the platform-to-Flutter image contract
mobile/…/LocalImagesImpl.ktShows Android media access, fallback paths, and buffer delivery
mobile/…/LocalImagesImpl.swiftShows the iOS image and ThumbHash bridge
mobile/lib/infrastructure/loaders/image_request.dartShows how Flutter consumes and frees native image buffers