Client video decoding and rendering
Overview
The native client video path is assembled in app/…/scrcpy.c by wiring trait-style sources and sinks: the video sc_demuxer reads encoded packets from server.video_socket, optional packet consumers are attached, sc_decoder converts packets to AVFrames, optional sc_video_regulator instances delay frames for display or V4L2 buffering, and sc_screen or sc_v4l2_sink consume the decoded frames. The owned files for this flow are the demuxer/decoder/regulator/buffer/screen/texture/hardware-interop/V4L2 pieces; the orchestration in app/…/scrcpy.c is the entry point that determines which of those pieces are active.
app/…/demuxer.c owns stream ingestion. sc_demuxer_start() creates a scrcpy-demuxer thread which first reads a 4-byte codec id, maps it to an FFmpeg codec id, then for video reads an initial 12-byte session header carrying width, height, and a client_resized flag. It opens all packet sinks via sc_packet_source_sinks_open() with AVCodecParameters and the initial sc_stream_session, then loops over 12-byte packet/session headers. Session packets are forwarded with sc_packet_source_sinks_push_session(). Media packets are read into a reusable AVPacket, receive pts/dts, key-frame flags, and optionally an opaque_ref receive timestamp when video buffering is enabled. H.264 and H.265 streams pass through app/…/packet_merger.c, which stores config packets (pts == AV_NOPTS_VALUE) and prepends the latest config payload to the next media packet while still forwarding config packets as standalone packets for downstream consumers that need extradata.
app/…/decoder.c is both a packet sink and a frame source. It opens an AVCodecContext from the demuxer parameters, starts a dedicated scrcpy-vid-dec thread, and keeps producer/decoder ownership separate with a mutex-protected sc_decoder_queue. Packet sink callbacks run on the demuxer thread: they ref-copy non-config AVPackets or enqueue session updates, signal the condition variable, and return quickly. The decoder thread drains the queue, applies downstream backpressure before avcodec_send_packet(), receives all available frames, updates video session/frame-size state, pushes frames to frame sinks via sc_frame_source_sinks_push(), and unrefs its reusable AVFrame immediately after pushing. If hardware decoding was selected but the first hardware send fails and the user did not force hardware decoding, it recreates the codec context for software decoding; AV1 hardware sessions also reopen the codec context on a new session because FFmpeg may not reinitialize the hardware decoder on size changes.
Hardware decoding is split between decoder setup and rendering interop. app/…/hwdec.c creates the FFmpeg hardware device (VAAPI, D3D11VA, VideoToolbox, or none), finds a decoder that supports the requested AVHWDeviceType, attaches hw_device_ctx to the codec context, and optionally refuses FFmpeg's software fallback when hardware decoding is forced. app/…/screen.c creates the SDL renderer before sc_init_video_hwdec() is called by app/…/scrcpy.c, so sc_screen_get_hw_type() and sc_screen_get_hw_device() can reflect the selected texture interop. app/…/texture.c chooses an app/src/interop/** implementation with sc_interop_new(), exposes its hw_type and device string, and on each frame verifies that the decoded AVFrame.format matches the selected interop. If a hardware interop later receives an incompatible software frame, sc_texture_update() switches to software interop and continues with YUV420P.
The renderer side lives on the SDL/UI thread. app/…/screen.c implements a frame sink, but its push() callback is called from the decoder/regulator thread and only writes to screen->fb under screen->mutex, records whether an older frame was overwritten, and posts SC_EVENT_NEW_FRAME when needed. app/…/frame_buffer.c is a one-frame latest-frame buffer: push() refs the incoming frame into pending_frame, replacing any unconsumed frame, and consume() move-refs it to a destination frame. sc_screen_handle_event() consumes SC_EVENT_NEW_FRAME on the UI thread, updates screen->frame, handles pause/resume, applies frame-size/content-size changes, updates the SDL texture through sc_texture_update(), and renders with SDL_RenderTexture() or SDL_RenderTextureRotated() according to display orientation and render_fit. The screen initially creates a hidden SDL window; sc_screen_frame_sink_open() posts SC_EVENT_OPEN_WINDOW, and the UI thread sizes/positions/shows the window once the first video session size is known.
Texture upload paths are selected by app/…/interop.c. Software interop (app/…/interop_software.c) creates a YV12 SDL texture and updates it with SDL_UpdateYUVTexture(), optionally enabling OpenGL mipmaps via app/…/opengl.c when the renderer is OpenGL/OpenGL ES. VA-API interop (app/…/interop_vaapi.c) requires an OpenGL renderer, EGL image support from app/…/egl.c, DMA-BUF import, and NV12 frames; it maps VAAPI frames to DRM PRIME, imports Y and UV planes as EGLImages, then attaches them to the SDL texture's OpenGL plane texture ids. D3D11VA interop (app/…/interop_d3d11va.c) requires SDL's Direct3D 11 renderer and copies the decoded texture-array slice into SDL's D3D11 texture. VideoToolbox interop (app/…/interop_videotoolbox.c) requires SDL's Metal renderer and creates a new SDL texture around each frame's CVPixelBuffer. app/…/vaapi_shim.c is only for static builds: it weakly defines the libva/libdrm entry points FFmpeg references, loads the system libraries at runtime, and lets VA-API device creation fail cleanly so software decoding can be used.
app/…/video_regulator.c is an optional frame source/sink inserted between the decoder and a frame sink when --video-buffer or --v4l2-buffer is configured. It requires demuxer receive timestamps to have been stored in AVPacket.opaque_ref and copied by the decoder to AVFrame.opaque_ref. Its sink push() updates a clock from (recv_date, pts), optionally forwards the first frame immediately, otherwise ref-copies frames or sessions into a queue. A scrcpy-vruf thread pops queued items, waits until the stream clock time plus configured delay (capped at now + delay), then forwards to its frame source. Its apply_backpressure() blocks upstream when the queue exceeds the configured threshold; app/…/scrcpy.c disables one of two thresholds when both display and V4L2 regulators are active so only the tighter/outer regulator controls pressure.
FPS reporting is owned by app/…/fps_counter.c and embedded in sc_screen. It starts a scrcpy-fps thread lazily when --print-fps is active or toggled, counts rendered frames from sc_screen_apply_frame() and skipped frames from sc_screen_frame_sink_push(), and logs once per second. started is atomic so hot-path frame accounting avoids taking the mutex when the counter is disabled; the counters and next timestamp are protected by the mutex.
The optional decoded-video sink in this area is app/…/v4l2_sink.c. It is a frame sink attached from app/…/scrcpy.c when a V4L2 device is configured, optionally behind its own sc_video_regulator. On open it creates a latest-frame sc_frame_buffer, FFmpeg V4L2 muxer, rawvideo encoder, output AVIOContext, reusable frame/packet, and a scrcpy-v4l2 thread. Pushes from the decoder/regulator store only the latest frame and signal the thread if it was idle. The thread consumes frames, encodes rawvideo packets, writes the first packet as stream extradata/header, rescales later packet timestamps from scrcpy's microsecond time base to the output stream, and writes them to the device.
Coupling
- If the video stream header or session metadata changes, update both app/…/demuxer.c and consumers of
struct sc_stream_sessionin app/…/decoder.c, app/…/screen.c, and app/…/video_regulator.c. The demuxer is the only parser for the 12-byte session packet, but the screen usessession.video.client_resizedto suppress automatic resize and the decoder compares decoded frame sizes againstsession.video.width/height. - If video buffering is enabled or changed, keep app/…/scrcpy.c, app/…/demuxer.c, app/…/decoder.c, and app/…/video_regulator.c aligned: the regulator asserts every delayed frame has an
opaque_refreceive timestamp, sosc_demuxer_init(... set_recv_date=true ...)andsc_decoder_init(... copy_opaque=true ...)must both be selected whenever a regulator is inserted. - If adding another packet or frame consumer, check
SC_PACKET_SOURCE_MAX_SINKSin app/…/packet_source.h andSC_FRAME_SOURCE_MAX_SINKSin app/…/frame_source.h. The video pipeline already uses fan-out for recorder/decoder at the packet level and display/V4L2 at the frame level, so adding a third sink silently exceeds the fixed source capacity unless the trait limit and source implementation are updated. - If an interop advertises a hardware
hw_typein app/…/interop*.c, verify app/…/hwdec.c can create a matching FFmpeg hardware device and that app/…/texture.c expects the sameAVPixelFormat. A mismatch makes the decoder output frames the texture layer rejects, causing rendering to fail or forcing software interop. - If changing SDL renderer, texture properties, or color-space handling in app/…/texture.c or
app/src/interop/**, verify the screen render path in app/…/screen.c still uses the correct source rectangle. D3D11VA may allocate surfaces larger than the visible frame and relies onsc_texture_get_frame_size()to render only the frame area. - If changing app/…/frame_buffer.c, audit both app/…/screen.c and app/…/v4l2_sink.c. The buffer deliberately has no internal mutex; each owner must hold its own mutex around
has_frame(),push(), andconsume()to preserve the one-latest-frame contract. - If changing H.264/H.265 config packet handling in app/…/demuxer.c or app/…/packet_merger.c, verify both decoder startup and packet consumers. Config packets must remain standalone for extradata-style consumers, but the next media packet must receive prepended config bytes for correct H.26x decoding/recording.
Risks
- Hardware interop is fragile across platforms: app/…/hwdec.c, app/…/texture.c, and
app/src/interop/**have recent targeted fixes for H.265 hardware decoding on Windows (f6dce15c), AV1 hardware session reset (f05949d6), VA-API renderer GPU selection (fd583a4e), and VA-API NV12 chroma layout (2e8363a7). If a decoder/interop format or device-selection change is wrong, hardware frames either cannot be imported to the SDL texture or fall back to software unexpectedly. - Screen resizing and frame delivery cross thread boundaries in app/…/screen.c and app/…/frame_buffer.c. History includes fixes for data races (
612947cd), frame-triggered resize loops (9c4d2bbf), and physical/logical size confusion (aecd902e); if frame-buffer locking or resize-event filtering is changed, the UI thread can consume stale frames, skip resize suppression, or send spurious flex-display resize requests back to the device. - V4L2 output drops intermediate frames by design in app/…/v4l2_sink.c via app/…/frame_buffer.c, and its history includes race/leak fixes (
5caeab5f,84f17fde,1cde68a1). If thehas_frameflag and frame-buffer operations are not kept undervs->mutex, the sink thread can miss a wakeup or consume an overwritten frame while writing to the device.