Native media core
The native media core is a Rust library that gives the mobile clients one place for low-level image work. It converts decoded pixels into Flutter-readable RGBA data, applies image orientation, and rebuilds small ThumbHash previews while platform code manages media access and memory ownership.
System role
The core sits between Flutter image requests and the native media APIs on Android and iOS. It keeps pixel rules consistent where both platforms need the same result, while each platform remains responsible for reading local media and exposing it through the Flutter bridge.
flowchart LR
A[Flutter image request] --> B[Platform image bridge]
B --> C[Platform media source]
C --> D[Native media core]
D --> E[RGBA buffer and image metadata]
E --> F[Flutter image frame]
A --> G[ThumbHash value]
G --> D
References: native/README.md• L3-10 mobile/…/local_image_api.dart• L14-30 mobile/…/image_request.dart• L119-163
Media capabilities
The core supports three media operations. Together they cover the path from platform-decoded pixels to a frame that Flutter can display.
| Capability | Purpose | Mobile result |
|---|---|---|
| Pixel conversion | Changes Android 10-bit RGBA pixels into 8-bit RGBA pixels | Correct colors for high-bit-depth image sources |
| Orientation transform | Applies all EXIF orientation values, including flips and quarter turns | A dense pixel buffer with the correct width and height |
| ThumbHash decode | Rebuilds a small RGBA preview from a ThumbHash string | A fast placeholder while the full image loads |
The core accepts image dimensions and row stride as part of the pixel contract. This lets it handle padded source rows and return a compact output buffer without making Flutter understand platform bitmap layouts.
References: native/…/convert.rs• L1-14 native/…/rotate.rs• L1-35 native/…/thumbhash.rs• L8-25 native/…/bitmap.rs• L34-43
Binding boundary
The Flutter package builds the Rust FFI library for the mobile target and exposes generated Dart bindings. The Rust FFI layer provides the C entry points and Android JNI entry points; platform code calls those entry points and returns buffer addresses plus image metadata through the existing image bridge.
The binding boundary also defines ownership. A successful native operation returns memory that the caller must release after Flutter creates its image buffer. An invalid input, unsupported bitmap format, allocation failure, or native error returns no usable buffer so the platform can use its fallback path or report an error.
References: native/…/build.dart• L7-29 native/…/ffigen.yaml• L1-9 native/…/lib.rs• L5-44 mobile/…/image_request.dart• L119-145
Android image path
Android uses the core for bitmap work that needs direct pixel access. The platform decodes an image, checks whether EXIF orientation still needs to be applied, and sends a supported bitmap to the native transform; the returned buffer becomes a Flutter raw RGBA image.
The path has safe fallbacks. A 10-bit bitmap can use native conversion, other formats can use the platform's 8-bit conversion, and unsupported native operations return control to the platform path. Raw image orientation is read before decoding because some Android raw-media paths do not apply EXIF orientation themselves.
sequenceDiagram
participant F as Flutter
participant A as Android image bridge
participant R as Rust core
F->>A: Request image or placeholder
A->>A: Decode bitmap and inspect orientation
A->>R: Convert or rotate pixels
R-->>A: Buffer address and dimensions
A-->>F: Pointer, width, height, row bytes
F->>F: Build RGBA image and free buffer
References: mobile/…/NativeImage.kt• L5-27 mobile/…/LocalImagesImpl.kt• L76-113 mobile/…/LocalImagesImpl.kt• L224-259 mobile/…/RawOrientation.kt• L45-67
iOS ThumbHash path
iOS uses the shared Rust symbol for ThumbHash decoding and keeps normal image decoding in the platform image stack. The iOS bridge base64-decodes the value, asks the core for RGBA pixels, and sends the pointer and dimensions through the same Flutter image contract.
The shared ThumbHash decoder returns the preview aspect ratio and supports hashes with or without alpha. The mobile loader then treats the result as a raw image rather than as an animated or encoded image.
References: mobile/…/NativeCore.swift• L3-21 mobile/…/LocalImagesImpl.swift• L40-54 mobile/…/thumbhash_image_request.dart• L3-20 native/…/thumbhash.rs• L16-52
Flutter image contract
The local image API is the shared contract between Flutter and the platform bindings. It carries either encoded bytes or decoded pixel data, plus the dimensions and row byte count needed to construct a Flutter image; ThumbHash requests use the same decoded-pixel form.
Flutter owns the final handoff. It wraps the native memory as an immutable buffer, creates a raw RGBA image descriptor, and frees the native pointer in a finally path. Cancellation therefore affects both the request and the buffer lifetime rather than only stopping the platform task.
References: mobile/…/local_image_api.dart• L14-30 mobile/…/image_request.dart• L41-79 mobile/…/image_request.dart• L119-163
Build and version boundary
The Rust workspace contains the shared media logic and its FFI adapter. The Flutter package builds that adapter for each supported Android and iOS target, then uses generated C and Dart bindings so application code does not depend on Rust types.
The core exposes a version value for a small runtime compatibility check. Changes to the Rust ABI must flow through the generated header and Dart bindings before the mobile package can use them.
References: native/Cargo.toml• L1-4 native/…/lib.rs• L1-7 native/…/lib.rs• L16-39 native/…/native_core.dart• L1-12
Change checklist
When these files or sections are changed, remember to consider these:
- Keep the FFI header asset name and the generated Dart binding asset name aligned between
native/native_core/hook/build.dartand native/…/ffigen.yaml. - Keep the pointer metadata contract aligned across
mobile/pigeon/local_image_api.dart, both platform image bridges, andmobile/lib/infrastructure/loaders/image_request.dart. - Keep pixel format and row-byte assumptions aligned between native bitmap transforms and Flutter's
PixelFormat.rgba8888descriptor. - Keep EXIF orientation handling aligned with the Android decode paths so raw images are not left unrotated or rotated twice.
- Update the ThumbHash decoder compatibility samples when its output changes, because the tests compare decoded dimensions and pixel values with existing mobile placeholders.
Important failure modes
| Trigger | Consequence | Guard |
|---|---|---|
| Native bitmap format is not supported | No native buffer is returned | Android falls back to a platform conversion path |
| EXIF orientation is applied twice | The image appears flipped or rotated | Only raw decode paths report an orientation for the native transform |
| ThumbHash base64 or payload is invalid | Placeholder loading fails | Both platform bridges report a failed request instead of building an image |
| Returned pixel pointer is not freed after Flutter consumes it | Native memory grows with image requests | Flutter frees decoded buffers after creating the immutable buffer |
| FFI header or generated bindings are stale | The mobile build or native call contract breaks | Regenerate the header and Dart bindings after ABI changes |
Key files
| File | Why to read it |
|---|---|
| native/…/rotate.rs | Defines the shared orientation model and output dimensions |
| native/…/convert.rs | Defines high-bit-depth pixel conversion |
| native/…/thumbhash.rs | Defines ThumbHash preview reconstruction |
| native/…/bitmap.rs | Shows the native bitmap-to-buffer boundary |
native/native_core/hook/build.dart | Shows how Flutter builds the Rust library for mobile targets |
mobile/pigeon/local_image_api.dart | Defines the platform-to-Flutter image contract |
| mobile/…/LocalImagesImpl.kt | Shows Android media access, fallback paths, and buffer delivery |
| mobile/…/LocalImagesImpl.swift | Shows the iOS image and ThumbHash bridge |
mobile/lib/infrastructure/loaders/image_request.dart | Shows how Flutter consumes and frees native image buffers |