Introducing a new way to review your PRs.

Read the announcement

genymobile / scrcpy

Display and control your Android device

View on GitHub

13 documents · Updated Nov 16, 2026

docs / client-control-input-and-hid

Client Control Input and HID

Overview

This area turns local SDL input into either scrcpy control-socket messages or USB HID traffic. The event entry point is app/…/input_manager.c: sc_input_manager_handle_event() receives SDL keyboard, text, mouse, touch, gamepad, drop-file, and device-disconnect events, filters shortcuts/paused/disconnected/camera states, converts window coordinates to device-frame coordinates, and dispatches accepted events to key, mouse, and gamepad processor traits. The internal event model lives in app/…/input_events.h; it deliberately mirrors SDL keycodes, scancodes, modifiers, mouse buttons, and gamepad enums while changing pointer coordinates from SDL window coordinates to struct sc_position in device frame space from app/…/coords.h.

The control-socket path is managed by app/…/controller.c. sc_controller_push_msg() queues struct sc_control_msg values from app/…/control_msg.h, dropping most events when the queue exceeds SC_CONTROL_MSG_QUEUE_LIMIT but preserving UHID_CREATE and UHID_DESTROY so UHID device state cannot desynchronize. SC_CONTROL_MSG_TYPE_RESIZE_DISPLAY is special: sc_controller_resize_display() stores one pending size outside the normal queue, overwriting older resize requests and giving resize priority in the controller thread. run_controller() serializes each message with sc_control_msg_serialize() from app/…/control_msg.c and writes it to the control socket with net_send_all().

app/…/control_msg.c defines the wire layout for client-to-server control messages. The first byte is always the enum sc_control_msg_type; numeric fields are written big-endian. SDK key injection serializes action, Android keycode, repeat count, and Android metastate. SDK pointer injection serializes action, pointer id (SC_POINTER_ID_MOUSE, SC_POINTER_ID_GENERIC_FINGER, or SC_POINTER_ID_VIRTUAL_FINGER), frame position, pressure as fixed-point, action button, and button state. Scroll serializes position plus normalized fixed-point horizontal/vertical scroll clamped from local [-16, 16] units. Strings for text, clipboard, scan-file, and app-start messages are UTF-8-truncated to their protocol maxima before the length-prefixed payload is written. UHID messages on this socket are also control messages: UHID_CREATE sends id, vendor/product, optional tiny name, and static report descriptor; UHID_INPUT sends id, size, and up to SC_HID_MAX_SIZE bytes; UHID_DESTROY sends only the id.

Device-to-client messages are the reverse side of the same control connection. app/…/device_msg.c deserializes DEVICE_MSG_TYPE_CLIPBOARD, DEVICE_MSG_TYPE_ACK_CLIPBOARD, and DEVICE_MSG_TYPE_UHID_OUTPUT from a byte stream; the receiver that consumes them is owned by app/…/controller.c and configured via sc_controller_configure(). app/…/receiver.c is outside this doc's ownership but is the direct consumer: clipboard messages are posted to the SDL main thread, ACK messages signal struct sc_acksync, and UHID output is posted back to app/…/uhid_output.c. Only keyboard UHID output is currently handled there, and app/…/keyboard_uhid.c uses it to track CapsLock/NumLock LED state.

There are three backend families, selected before app/…/input_manager.c is initialized:

  • SDK injection uses the Android control protocol. app/…/keyboard_sdk.c maps struct sc_key_event to SC_CONTROL_MSG_TYPE_INJECT_KEYCODE using Android keycodes/metastates from app/…/keycodes.h and app/…/input.h, or text input to SC_CONTROL_MSG_TYPE_INJECT_TEXT depending on the configured key injection mode. In text mode, letter/space key events are suppressed unless Ctrl is held so text can arrive through SDL text input; in raw mode punctuation/numpad mappings are also sent as keycodes and text events are ignored. app/…/mouse_sdk.c maps mouse/touch events to Android touch and scroll control messages with absolute frame coordinates and optional hover motion.
  • UHID injection sends HID reports through the Android server over the control socket. app/…/keyboard_uhid.c, app/…/mouse_uhid.c, and app/…/gamepad_uhid.c implement the processor traits by generating generic HID reports from app/src/hid/**, wrapping them as UHID_CREATE and UHID_INPUT control messages, and letting the server create virtual input devices on Android. UHID mouse is relative mode, so touch events are not supported; UHID keyboard ignores SDL repeats because the host handles HID key repeat.
  • AOA injection bypasses the scrcpy server for HID events and talks directly to the USB device with Android Open Accessory HID control transfers. app/…/keyboard_aoa.c, app/…/mouse_aoa.c, and app/…/gamepad_aoa.c share the same HID report generators as UHID, but enqueue OPEN, INPUT, and CLOSE events to app/…/aoa_hid.c. That thread issues ACCESSORY_REGISTER_HID, ACCESSORY_SET_HID_REPORT_DESC, ACCESSORY_SEND_HID_EVENT, and ACCESSORY_UNREGISTER_HID via libusb. app/…/scrcpy_otg.c is the OTG-only entry point and builds an input-only screen around the AOA processors; the regular scrcpy path can also use AOA processors when USB HID is enabled.

The shared HID report layer in app/src/hid/** is backend-agnostic. app/…/hid_keyboard.c keeps a 102-key pressed-state array and emits 8-byte boot-keyboard reports: one modifier byte, one reserved byte, and up to six pressed scancodes, with rollover filled by 0x01. It can also synthesize CapsLock/NumLock reports for modifier-lock synchronization. app/…/hid_mouse.c emits 5-byte relative reports: buttons, X, Y, vertical wheel, and horizontal pan; scroll residuals are accumulated until they produce integer HID wheel deltas. app/…/hid_gamepad.c maintains up to SC_MAX_GAMEPADS slots, assigns HID ids starting at SC_HID_ID_GAMEPAD_FIRST, and emits 15-byte Xbox-like reports with little-endian axes/triggers, button bits, and a hat-switch byte for DPAD combinations.

Mouse capture is separate from message serialization. app/…/mouse_capture.c toggles SDL relative mouse mode when the configured shortcut modifier key is pressed and released alone, consumes capture keys so they are not forwarded, captures after the first mouse-button release, disables capture on focus loss, and consumes touch events because relative-mode processors cannot use absolute touch coordinates. This pairs with struct sc_mouse_processor.relative_mode: app/…/input_manager.c omits absolute positions when the active mouse backend is relative and asserts that virtual-finger gestures are not used in relative mode.

Keyboard, mouse, and gamepad handling in app/…/input_manager.c is also responsible for high-level scrcpy shortcuts and clipboard sequencing. Shortcut modifiers are normalized through app/…/shortcut_mod.h, including treating SDL AltGr mode as right Alt. Device-affecting shortcuts push control messages for HOME/BACK/APP_SWITCH/MENU/POWER/volume, display power, panels, rotation, hard-keyboard settings, reset video, and camera torch/zoom. Normal Ctrl+V can first push SET_CLIPBOARD; when the key processor advertises async_paste, the input manager assigns a sequence number and passes it to process_key(). AOA keyboard then waits in app/…/aoa_hid.c for the device ACK before sending the HID Ctrl+V event because clipboard synchronization uses the control socket while AOA HID uses USB. SDK and UHID set async_paste = false because their key events are serialized on the same control channel as clipboard synchronization.

Coupling

Risks

  • app/…/controller.c intentionally drops droppable events once the queue has 60 items, and app/…/control_msg.c only marks UHID_CREATE and UHID_DESTROY as non-droppable. Under sustained input load, UHID_INPUT, SDK touch, scroll, key, and clipboard messages may be discarded; if a backend starts depending on every input report being delivered, device-visible state can lag or stick even though the controller keeps running.
  • app/…/keyboard_uhid.c acknowledges that CapsLock/NumLock synchronization is inherently racy because HID output LED reports arrive asynchronously. If device_mod is updated from stale DEVICE_MSG_TYPE_UHID_OUTPUT data while local modifier lock state has already changed, the next key press may briefly inject the wrong lock synchronization report.
  • app/…/aoa_hid.c waits at most 500 ms for a clipboard ACK before discarding an AOA HID event with ack_to_wait. If the server is slow to acknowledge SET_CLIPBOARD, Ctrl+V is silently skipped rather than pasting stale content.
  • app/…/hid_keyboard.c supports only six simultaneous non-modifier keys in its boot-keyboard report. Pressing more than six emits a rollover error report; games or tests requiring high key rollover will not receive the individual keys through UHID or AOA keyboard backends.