Server video capture, encoding, and display
Overview
The server video pipeline is built around server/…/SurfaceEncoder.java: it owns the MediaCodec encoder, asks a SurfaceCapture implementation to render into the codec input Surface, then drains encoded buffers to com.genymobile.scrcpy.device.Streamer. SurfaceEncoder.streamCapture() creates the encoder selected by VideoCodec/options, writes the stream codec id header, loops through capture sessions, writes per-session video dimensions via Streamer.writeSessionMeta(), and forwards each encoded packet from MediaCodec.dequeueOutputBuffer() to the desktop with frame metadata. VideoSource.java only distinguishes display capture from camera capture; the concrete source is one of the SurfaceCapture subclasses.
server/…/SurfaceCapture.java defines the lifecycle contract used by the encoder: init() receives a shared CaptureControl and VideoConstraints, each retry/reset calls prepare() before getSize(), start(surface) renders into the encoder surface, stop() tears down the active rendering path, and release() frees source resources. Resets are cooperative: CaptureControl.reset() records bit flags and, if an encoder is currently running, calls MediaCodec.signalEndOfInputStream() so SurfaceEncoder.encode() exits its drain loop and the outer loop can recreate the codec/capture session.
For screen mirroring, server/…/ScreenCapture.java reads DisplayInfo from ServiceManager.getDisplayManager() in prepare(), derives crop/orientation/angle transforms with VideoFilter, constrains the output size with VideoConstraints, and starts a DisplayMonitor for the selected display during init(). In start(), if a transform is required, it inserts OpenGLRunner with an AffineOpenGLFilter so the system display is first rendered to an intermediate SurfaceTexture at the full display size and then drawn into the encoder surface at videoSize. Without a transform, the display is rendered directly at the encoder size. It prefers DisplayManager.createVirtualDisplay("scrcpy", ..., displayId, surface) and falls back to SurfaceControl.createDisplay()/setDisplaySurface() with the original display layer stack; the fallback matters for devices where DisplayManager virtual display creation fails. After creating the mirror surface, it notifies VirtualDisplayListener with the active display id and a PositionMapper based on the same inverse transform used for video coordinates.
For creating and streaming a new Android display, server/…/NewDisplayCapture.java also implements SurfaceCapture, but it owns a persistent VirtualDisplay. init() resolves requested NewDisplay size and dpi, optionally using main-display defaults from DisplayInfo when not supplied. In flex-display mode it creates a DisplayPropertiesTracker plus DisplayResizeDebouncer and forbids crop, because client-driven logical display resizing is the core behavior. prepare() computes the display size, physical size, event transform, and display transform separately: eventTransform maps client input back to the logical display, while displayTransform additionally compensates for the virtual display's physical rotation before rendering to video. start() inserts OpenGLRunner when displayTransform is non-null, then either creates the new display with createNewVirtualDisplay() and flags such as public/presentation/own-content/supports-touch/rotates-with-content, or reuses the existing virtual display by swapping its surface.
Display property changes drive encoder resets. server/…/DisplayMonitor.java watches one display id and compares (Size, rotation) pairs from DisplayProperties.java. On Android versions before 14 it uses DisplayManager.registerDisplayListener() on a handler thread; on Android 14+ it uses WindowManager.registerDisplayWindowListener() because DisplayListener can stop sending events. ScreenCapture.init() resets with RESET_REASON_DISPLAY_PROPERTIES_CHANGED whenever the mirrored display changes. NewDisplayCapture.startNew() monitors the created virtual display; in flex mode, DisplayPropertiesTracker.onChanged() classifies whether the change matches a recent client resize request, causing either RESET_REASON_CLIENT_RESIZED or RESET_REASON_DISPLAY_PROPERTIES_CHANGED. Client resize requests go through NewDisplayCapture.requestResize(), are coalesced by DisplayResizeDebouncer for 300 ms, and triggerResize() pushes the expected DisplayProperties before calling VirtualDisplay.resize().
Camera capture is handled by server/…/CameraCapture.java. init() starts a camera HandlerThread, selects a camera by explicit id or CameraFacing, opens it through CameraManager, and stores the initial VideoConstraints. prepare() selects the largest compatible camera output size from StreamConfigurationMap (or high-speed video sizes), respecting explicit size, max size, and optional CameraAspectRatio; then it builds a VideoFilter for crop/orientation/angle and aligns the encoded videoSize. Camera output surfaces must match the camera capture size, so if constraints change the output dimensions without an existing transform, it forces an identity transform and inserts OpenGLRunner. start() builds a camera2 recording session targeting either the encoder surface or the OpenGL input surface, applies fps, torch, and zoom options to a repeating request, and marks the capture closed plus resets with termination when the device disconnects or session configuration fails.
server/…/VideoFilter.java is the shared geometry builder for display, new-display, and camera sources. It tracks the output Size and an affine transform as operations are appended: crop, orthogonal rotation, orientation flip/rotation, arbitrary angle, and resize. Its inverse transform is reused both by AffineOpenGLFilter for sampling texture coordinates and by capture classes to build input-event mappings. server/…/OpenGLRunner.java is the bridge when a source cannot render directly to the encoder surface in the final geometry: it creates a shared handler thread, EGL context/window surface around the encoder output, an external OES SurfaceTexture input, and renders every frame through OpenGLFilter.draw(). AffineOpenGLFilter.java compiles a simple ES2 shader that multiplies the SurfaceTexture transform matrix with the user affine matrix and outputs transparent pixels outside [0,1]; camera capture passes an override vertical-flip matrix because SurfaceTexture.getTransformMatrix() is unreliable for that path.
Encoder constraints are negotiated in SurfaceEncoder.java and expressed by server/…/VideoConstraints.java. Initial constraints include the requested max size and the stricter of codec width/height alignment and minSizeAlignment, but intentionally do not apply full MediaCodecInfo.VideoCapabilities until encoding actually fails. On failures before the first frame, or for a small number of consecutive post-frame errors, prepareRetry() may apply the encoder capabilities to the capture, downsize through fixed fallback maxima, and retry. ScreenCapture and NewDisplayCapture accept new constraints; CameraCapture.applyNewVideoConstraints() refuses them if a camera size was explicitly requested, because the capture size cannot be reselected safely around that fixed request.
Coupling
- If
CaptureControlreset reasons in server/…/CaptureControl.java change, update the reset handling inSurfaceEncoder.streamCapture()and the producers in ScreenCapture.java/NewDisplayCapture.java; the session metadata sent byStreamer.writeSessionMeta()marks client resizes only whenRESET_REASON_CLIENT_RESIZEDis present without a display-properties reset. - If
VideoFiltertransform semantics change, verify ScreenCapture.java, NewDisplayCapture.java, CameraCapture.java, and AffineOpenGLFilter.java together. The same inverse transform is used for OpenGL texture sampling and for input coordinate mapping throughVirtualDisplayListener, so video fixes can silently break pointer coordinates. - If display monitoring changes in DisplayMonitor.java, test both mirrored display and new-display/flex-display resets. NewDisplayCapture.java relies on DisplayPropertiesTracker.java to distinguish client-requested resize acknowledgements from external display property changes, and wrong classification changes the desktop resize behavior.
- If codec size/alignment logic changes in SurfaceEncoder.java or VideoConstraints.java, verify all three capture implementations. Display capture constrains final output, new-display flex constrains logical display size, and camera capture may need an OpenGL identity transform because camera sessions require the target surface to match the selected camera output size.
- If OpenGLRunner.java threading or lifecycle changes, check all
stop()/release()paths in ScreenCapture.java, NewDisplayCapture.java, and CameraCapture.java; they assumestopAndRelease()synchronously prevents further frame rendering before source or codec surfaces are released.
Risks
- server/…/DisplayMonitor.java has platform-specific event selection because Android 14/15
DisplayListenercan miss events. If that fallback regresses, display rotation/resize changes will not interruptSurfaceEncoder.encode(), so the desktop may continue decoding with stale dimensions until another reset occurs. - server/…/SurfaceEncoder.java retries after codec failures by mutating capture constraints and recreating the codec loop. If a capture implementation accepts constraints but does not actually change
getSize(), repeated configuration failures can either exhaust fallback sizes or fail after consecutive errors without a useful recovery path. - server/…/NewDisplayCapture.java treats a display-properties event as a client resize only if it exactly matches a pending
DisplayPropertieswithinDisplayPropertiesTracker's 3-second cache. If rotation or size normalization changes, client resizes may be misclassified as external display changes, cancelling pending resize requests and clearing the client-resize metadata bit. - server/…/CameraCapture.java performs camera session state changes on a handler thread while
SurfaceEncodercontrolsstart()/stop()from the video thread. If session callbacks arrive afterstop()or after camera disconnect, missedstarted/currentSessionchecks can leave the encoder waiting on a source that will not produce frames.