Introducing a new way to review your PRs.

Read the announcement

genymobile / scrcpy

Display and control your Android device

View on GitHub

13 documents · Updated Nov 16, 2026

docs / client-audio-playback-and-regulation

Client Audio Playback and Regulation

Overview

Client-side audio playback is wired as a frame-sink at app/…/audio_player.c: sc_audio_player_init() installs open, push, and close callbacks on struct sc_audio_player.frame_sink, and app/…/scrcpy.c attaches that sink to s->audio_decoder.frame_source when options->audio_playback is enabled. The audio decoder is initialized in app/…/scrcpy.c with no hardware decoder and pushes decoded FFmpeg AVFrames to the player via the generic frame-source path; app/…/decoder.c opens downstream frame sinks with the AVCodecContext, then calls sc_frame_source_sinks_push() for every decoded frame.

sc_audio_player_frame_sink_open() in app/…/audio_player.c derives the SDL/FFmpeg-facing playback format from the decoder context. It asserts a positive channel count and sample rate, computes sample_size as nb_channels * av_get_bytes_per_sample(SC_AV_SAMPLE_FMT), and initializes struct sc_audio_regulator with the decoder AVCodecContext and a target buffering delay converted from ticks to samples. The regulator output sample format is fixed by app/…/audio_regulator.h as SC_AV_SAMPLE_FMT (AV_SAMPLE_FMT_FLT), and app/…/audio_player.c maps the same format to SDL as SDL_AUDIO_F32LE; changes to either side must keep those formats in sync. The player also sets SDL's SDL_HINT_AUDIO_DEVICE_SAMPLE_FRAMES from output_buffer_duration, allocates a temporary output buffer at least 1024 samples long, opens the default SDL playback device stream, and resumes the device.

Runtime data flow has two threads meeting at the regulator. The decoder/producer side calls sc_audio_player_frame_sink_push() in app/…/audio_player.c, which forwards each decoded AVFrame to sc_audio_regulator_push() in app/…/audio_regulator.c. The SDL/consumer side invokes sc_audio_player_stream_callback(), which may split SDL's requested byte count into chunks no larger than ap->aout_buffer_size, asks sc_audio_regulator_pull() for the corresponding sample count, then writes the filled bytes to the SDL_AudioStream with SDL_PutAudioStreamData().

app/…/audio_regulator.c performs format conversion and clock compensation with FFmpeg/libswresample. sc_audio_regulator_init() configures a SwrContext with the decoder channel layout, input sample rate, output sample rate, input format from ctx->sample_fmt, and output format SC_AV_SAMPLE_FMT; the input and output sample rates are intentionally equal, and drift is handled by swr_set_compensation() rather than changing the SDL device rate. The regulator owns a struct sc_audiobuf sized to target_buffering + one second of samples, plus a reusable swr_buf for converted frames. On each push, it detects PTS discontinuities over 100 ms, may insert silence to restore target buffering, resets smoothed buffering and compensation state, converts the incoming frame with swr_convert(), and writes converted samples into the ring buffer.

app/…/audiobuf.c is the sample ring buffer shared by producer and consumer. It allocates capacity + 1 sample slots so head == tail means empty and the full condition is unambiguous. sc_audiobuf_write() advances the writer head after copying sample bytes, while sc_audiobuf_read() advances the reader tail after optionally copying to the caller; passing NULL to read discards samples, which app/…/audio_regulator.c uses when the audio buffer is too full. The buffer is mostly single-producer/single-consumer and relies on atomic head/tail cursors with acquire/release ordering; the regulator adds ar->mutex only for operations where the producer may also consume/drop old samples or where the consumer pull must not race that drop.

Playback startup and underflow behavior are controlled in sc_audio_regulator_pull(). Until the buffer reaches target_buffering, pulls return silence without marking playback as started. Once playing, a short read is treated as an underrun: the regulator fills the rest of SDL's request with silence and increments an atomic underflow counter, but only after at least one packet has been received. It does not drop late real samples on underrun; instead, later producer-side compensation absorbs the extra latency.

Drift regulation runs on the producer side in sc_audio_regulator_push(). After playback has started, the regulator updates avg_buffering with the current readable sample count, applies immediate corrections for resampling compensation, inserted silence, and dropped samples, and once per second compares the smoothed average to target_buffering. Differences below 4 ms are ignored when compensation is inactive, differences below 1 ms are ignored when compensation is active, acceleration is suppressed when the instantaneous buffer is already below target, and the requested compensation is clamped to 2% over a four-second distance before calling swr_set_compensation(). This keeps SDL consumption continuous while slowly pulling average buffering back toward the configured audio buffer target.

Shutdown is symmetric in app/…/audio_player.c: sc_audio_player_frame_sink_close() pauses the SDL audio device, destroys the SDL stream that owns the device ID, destroys the regulator, and frees the temporary output buffer. app/…/decoder.c waits for the decoder thread to stop using frame sinks before closing them, so the player close path runs after decoded-frame pushes have stopped.

Coupling

  • If SC_AV_SAMPLE_FMT changes in app/…/audio_regulator.h, update SC_SDL_SAMPLE_FMT in app/…/audio_player.c and verify sc_audio_player_frame_sink_open() still computes sample_size from the same packed FFmpeg format that SDL receives. The player asserts the FFmpeg format is not planar and passes the same byte layout through sc_audio_regulator_pull() into SDL.
  • If SDL callback sizing or output_buffer_duration handling changes in app/…/audio_player.c, verify sc_audio_regulator_pull() still receives whole samples. The callback asserts additional_amount % ap->audioreg.sample_size == 0 and converts bytes to samples before pulling.
  • If app/…/audiobuf.c is changed from a single-reader/single-writer ring buffer, re-check the regulator lock boundaries in app/…/audio_regulator.c. Normal read/write paths intentionally run lock-free, but sample dropping uses ar->mutex because the producer temporarily acts as a reader.
  • If decoder frame ownership changes in app/…/decoder.c, verify sc_audio_regulator_push() in app/…/audio_regulator.c still copies all needed data before av_frame_unref(). The decoder reuses one AVFrame, and the regulator currently consumes it synchronously via swr_convert() during the push call.
  • If audio PTS units change upstream, update the discontinuity logic in app/…/audio_regulator.c. sc_audio_regulator_push() treats frame->pts and computed packet duration as microseconds and uses a 100000 µs gap as the discontinuity threshold.

Risks

  • app/…/audio_regulator.c and app/…/audiobuf.c: the buffer is safe only under the current producer/consumer assumptions. If another thread starts reading or writing struct sc_audiobuf directly, the relaxed single-owner cursor updates can race and corrupt audio ordering or drop samples silently.
  • app/…/audio_regulator.c: discontinuity recovery writes silence with sc_audiobuf_write_silence() without checking the returned count. If the buffer is unexpectedly near capacity when a large PTS gap is detected, less silence than requested may be inserted and the regulator's reset average can no longer match actual buffered samples.