Introducing a new way to review your PRs.

Read the announcement

genymobile / scrcpy

Display and control your Android device

View on GitHub

13 documents · Updated Nov 16, 2026

docs / server-audio-capture-encoding

Server audio capture and encoding

Overview

The Android server builds the audio pipeline from server/…/Server.java: when audio is enabled, it reads Options.getAudioSource() and chooses either AudioDirectCapture for every source except playback, or AudioPlaybackCapture for playback; it then wraps the audio socket in a Streamer and chooses AudioRawRecorder for AudioCodec.RAW or AudioEncoder for encoded codecs.

server/…/AudioSource.java maps client source names to Android MediaRecorder.AudioSource constants. output uses REMOTE_SUBMIX; microphone and voice variants map directly to their recorder source constants; playback is special and has no direct source value, so isDirect() routes it to AudioPlaybackCapture instead. server/…/AudioCodec.java defines the stream codec ids and MIME types for OPUS, AAC, FLAC, and RAW; these ids are what server/…/Streamer.java can write as the audio stream header before packet data.

All capture implementations use the fixed PCM format from server/…/AudioConfig.java: 48 kHz, stereo, 16-bit PCM, and reads capped at 1024 stereo samples (MAX_READ_SIZE) to bound latency. AudioConfig.createAudioFormat() provides the AudioFormat used by both direct and playback capture.

Direct capture is implemented by server/…/AudioDirectCapture.java. checkCompatibility() disables audio below Android 11. On Android 12 and later, createAudioRecord() passes FakeContext.get() to AudioRecord.Builder; on older versions it relies on Workarounds.apply() having filled app info before the server constructs capture. It uses AudioRecord.getMinBufferSize() only if the returned value is positive, and multiplies it by 8 for the recorder buffer. If AudioRecord.Builder.build() throws NullPointerException, it falls back to Workarounds.createAudioRecord(...), which exists for Vivo/third-party ROM constructor behavior. Android 11 has a foreground-recording workaround: start() launches com.android.shell.HeapDumpActivity, retries startRecording() up to five times with 100 ms sleeps, and force-stops the shell package afterward; if recording still cannot start, it logs that the device should be unlocked and throws AudioCaptureException.

Playback capture is implemented by server/…/AudioPlaybackCapture.java and is only allowed on Android 13 and later. It uses reflection over hidden android.media.audiopolicy APIs to build an AudioMixingRule that captures many usages, including media, games, notifications, alarms, accessibility/navigation/sonification, assistant, unknown, and voice communication/signalling. It calls voiceCommunicationCaptureAllowed(true), builds an AudioMix using AudioConfig.createAudioFormat(), and registers an AudioPolicy through AudioManager.registerAudioPolicyStatic(...). The audio_dup option controls whether the route flag is ROUTE_FLAG_LOOP_BACK_RENDER (keep playing on device) or ROUTE_FLAG_LOOP_BACK (capture only). Any reflection, policy registration, or sink creation failure logs "Could not capture audio playback" and becomes AudioCaptureException.

server/…/AudioRecordReader.java is the common reader for both capture paths. It reads at most AudioConfig.MAX_READ_SIZE bytes into a direct ByteBuffer, fills a MediaCodec.BufferInfo, and timestamps packets from AudioRecord.getTimestamp(..., TIMEBASE_MONOTONIC). If an initial timestamp is unavailable, it logs a warning and estimates from System.nanoTime(). Later packets may mix recorder timestamps and estimated timestamps, so it enforces strictly increasing PTS by bumping any non-increasing timestamp by one 48 kHz sample.

Raw audio streaming is handled by server/…/AudioRawRecorder.java. It refuses to run below Android 11 by writing Streamer.writeDisableStream(false), starts the selected AudioCapture, writes the audio codec header, then repeatedly reads PCM chunks and forwards them through Streamer.writePacket(buffer, bufferInfo). If capture startup fails, it writes writeDisableStream(false) so the desktop can continue without audio. Negative read results become IOException; non-broken-pipe I/O failures are logged as audio capture errors. stop() interrupts the recorder thread and finally always calls capture.stop().

Encoded streaming is handled by server/…/AudioEncoder.java. It also refuses to run below Android 11, then calls capture.checkCompatibility() before creating the MediaCodec. If the user supplied an audio_encoder, createMediaCodec() creates by name and verifies the encoder MIME type matches the selected AudioCodec; a missing encoder or mismatched type becomes a ConfigurationException. Without an explicit name, it creates the default encoder by MIME type. The MediaFormat is built from the selected codec MIME type, the configured audio bitrate, the fixed sample rate/channel count, and any audio_codec_options applied via CodecUtils.setCodecOption().

AudioEncoder uses asynchronous MediaCodec callbacks on a dedicated HandlerThread. Callback events enqueue input-buffer indices and output-buffer indices into bounded queues; an audio-in thread blocks for input tasks, reads from AudioCapture, and queues PCM to the codec with the capture PTS; an audio-out thread writes the stream header once, then writes encoded output packets through Streamer. For the platform OPUS and FLAC encoders named c2.android.opus.encoder and c2.android.flac.encoder, AudioEncoder regenerates output PTS (except codec-config packets) because those encoders overwrite input PTS with sample-count-derived timestamps, hiding clock drift and hard silences. On capture errors it writes writeDisableStream(false); on other initialization or runtime failures it writes writeDisableStream(true) so the desktop treats the failure as fatal.

server/…/Streamer.java is the socket-facing boundary. For audio it may first write the 4-byte codec id from AudioCodec; for disabled streams, four zero bytes mean audio was disabled non-fatally and {0,0,0,1} means a configuration/fatal error. For packet metadata it maps MediaCodec.BufferInfo flags to config/key-frame bits. OPUS and FLAC codec-config packets are rewritten before transmission: OPUS is sliced to the AOPUSHDR block and FLAC is sliced to the metadata block after fLaC, so changes to audio config packet handling must account for the desktop-side extradata expectations.

Coupling

  • If server/…/AudioConfig.java changes sample rate, channel count, encoding, or max read size, also verify AudioRecordReader duration math and AudioEncoder.createFormat(); both derive packet PTS and encoder format from these constants, while AudioDirectCapture passes the same constants to AudioRecord construction and the Vivo fallback.
  • If server/…/AudioCodec.java adds or changes an audio codec, update the selection path in Server.java, the MIME handling in AudioEncoder.createMediaCodec(), and Streamer.writePacket() if that codec needs config-packet rewriting like OPUS or FLAC.
  • If server/…/AudioSource.java changes whether a source is direct, verify Server.java still routes it to the correct AudioCapture implementation; only PLAYBACK is currently handled by AudioPlaybackCapture, and every other source must provide a valid MediaRecorder.AudioSource value for AudioDirectCapture.
  • If server/…/AudioPlaybackCapture.java changes captured usages, route flags, or hidden audiopolicy reflection, verify desktop behavior for audio_dup; ROUTE_FLAG_LOOP_BACK_RENDER is the only path here that keeps device playback active while streaming.
  • If Streamer.writeDisableStream(...) semantics change in server/…/Streamer.java, update both AudioRawRecorder and AudioEncoder; they distinguish non-fatal capture unavailability from fatal configuration/runtime failure by choosing false or true.
  • If server/…/AudioEncoder.java changes threading or termination, verify all three worker contexts still unblock each other: the MediaCodec callback thread feeds queues, audio-in may block on input tasks or capture reads, and audio-out may block on output tasks or socket writes.

Risks

  • server/…/AudioPlaybackCapture.java depends on hidden android.media.audiopolicy classes and reflected methods (AudioMixingRule, AudioMix, AudioPolicy, and AudioManager.registerAudioPolicyStatic). If a vendor or Android release changes those APIs or policy permissions, playback capture fails before streaming and the server sends a non-fatal disabled-audio marker. The file itself references issue #4380, and history includes playback-capture fixes such as e71cdb87 and 8b05a616.
  • server/…/AudioDirectCapture.java has multiple device/version workarounds: Android 11 foreground activity retries and a Vivo AudioRecord construction fallback. If these paths regress, direct sources can fail only on specific devices or lock-screen states; the consequence is silent audio disablement via AudioCaptureException rather than a full server crash.
  • server/…/AudioEncoder.java contains a targeted PTS workaround for c2.android.opus.encoder and c2.android.flac.encoder, added in history by 24598128. If encoder canonical names change or another encoder starts overwriting PTS, recordings can drift or mishandle hard silences because the desktop receives encoded packet timestamps that no longer reflect capture time.
  • server/…/Streamer.java rewrites OPUS and FLAC config packets by parsing codec-specific headers and throwing IOException on unexpected structure. If Android changes MediaCodec CSD layout, encoded audio startup can fail as an audio encoding error even though capture and encoding succeeded.