Media, Calls, Webviews, Editor, and Third-Party Media Boundaries
Overview
This area owns Telegram Desktop's high-bandwidth UI and native-integration surfaces: media playback and transcoding under Telegram/SourceFiles/media/**, one-to-one and group calls under Telegram/SourceFiles/calls/**, photo/video editors under Telegram/SourceFiles/editor/**, FFmpeg wrappers under Telegram/SourceFiles/ffmpeg/**, WebAuthn/caBLE under Telegram/SourceFiles/webauthn/**, Instant View / markdown / Ton Site webview launching under Telegram/SourceFiles/iv/**, and the checked-out third-party boundary directories Telegram/ThirdParty/**. The Telegram/lib_lottie, Telegram/lib_qr, Telegram/lib_webrtc, Telegram/lib_webview, and sampled Telegram/ThirdParty/{tgcalls,libfido2,libcbor,QR,MicroTeX,cmark-gfm} directories are empty in this checkout, so references to those libraries enter through include boundaries such as lottie/lottie_icon.h, qr/qr_generate.h, webrtc/..., webview/..., <tgcalls/...>, latex.h, and platform FIDO APIs.
Media playback has two main paths. Audio tracks are coordinated by the singleton Media::Player::Instance in Telegram/…/media_player_instance.h and Telegram/…/media_player_instance.cpp; start() initializes Media::Audio::Start, audio capture, and the singleton, while Instance tracks separate song and voice queues, sparse shared-media playlist slices, seek state, repeat/shuffle, listen tracking, and last-playback-position persistence. It also subscribes to Core::App().calls().currentCallValue() and currentGroupCallValue() and pauses/resumes voice and music when calls start or end. The low-level audio mixer and device lifecycle live in Telegram/…/media_audio.h, with FFmpeg-backed loaders in Telegram/…/media_audio_ffmpeg_loader.cpp reading from memory, byte buffers, or files through custom AVIOContext callbacks.
Video and round-media playback use the streaming subsystem. Media::Streaming::Instance in Telegram/…/media_streaming_instance.h and .cpp is a lightweight handle around a shared Media::Streaming::Document obtained from Data::Session::streaming() for documents, alternate-quality documents, and photos. It forwards play/pause/resume/stop, frame requests, waiting-spinner state, speed, quality-switch requests, and per-view priority/locking to a shared Media::Streaming::Player. Telegram/…/media_streaming_player.h defines the player as the bridge between a file-reading thread and the main thread: FileDelegate callbacks open streams, push FFmpeg packets, report buffering/full-cache state, and compute audio/video received and played positions; the main thread renders frames, drives timers, and emits Update events. Remote media loads through Telegram/…/media_streaming_loader_mtproto.cpp, which subclasses the MTProto download task, queues requested byte offsets by priority, reads already cached parts from Storage::StreamedFileDownloader, and emits speed estimates based on completed download parts.
Animated previews, GIF/WebM-style clips, and thumbnails share FFmpeg helpers. Telegram/…/ffmpeg_utility.h and .cpp own RAII wrappers for AV packets, formats, codecs, frames, scaling, resampling, error logging, custom IO, and hardware decode setup. RestrictToCustomIO() in ffmpeg_utility.cpp clears FFmpeg's protocol_whitelist before demuxing so DASH/HLS or other demuxers cannot fetch referenced network URLs or local files. Telegram/…/ffmpeg_frame_generator.cpp implements Ui::FrameGenerator for byte-backed video/WebM data, bounds decoded area, handles rotation, scales into premultiplied BGRA QImages, and loops by seeking back to the start. Telegram/…/media_clip_ffmpeg.cpp is the clip reader used for animated inline media: it skips a small number of invalid packets, seeks back at EOF to loop, enforces different frame-area caps for inspecting vs inline playback, and treats VP9/WebM alpha specially. Transcoding and send-time processing are centralized in Telegram/…/media_video_encode.cpp: it creates temporary files in a private tdtranscode directory, chooses H.264 via libopenh264 or FFmpeg's H.264 encoder and VP9 WebM via libvpx-vp9, estimates output size, applies crop/trim/quality data, transcodes audio into AAC when needed, and has a special WebM-sticker CRF ladder.
The editor layer consumes those media services rather than owning codecs. Telegram/…/photo_editor.h and .cpp create a PhotoEditor widget with PhotoEditorContent, controls, brush/tool state, color picker, and PhotoModifications done/cancel streams. Brush and text preferences are serialized into Core::App().settings().photoEditorBrush() using a versioned QDataStream format; new fields are appended so older state can still deserialize. The drawable model is Telegram/…/scene.h, a QGraphicsScene that owns numbered items, canvas, shape drafting, text editing, undo/redo save states, and blur-source callbacks. Telegram/…/video_editor.h and .cpp wrap local video content with Media::Streaming::Instance and media_streaming_loader_local, a Crop, VideoTimeline, quality slider, frame capture/bubble preview, and VideoModifications collection. Video editor frames are requested through Media::Streaming::FrameRequest; final send/export behavior depends on Media::Encode::SourceInfo and media_video_encode.cpp.
Calls are orchestrated by Calls::Instance in Telegram/…/calls_instance.h and .cpp. It starts outgoing one-to-one calls, joins or creates group/conference calls, handles MTProto call updates, owns the current Calls::Call and Calls::GroupCall, requests microphone/camera permissions, caches DH config/server config, plays call sounds through Media::Audio::Track, and shares a weakly cached tgcalls::VideoCaptureInterface across camera/screen use. One-to-one calls live in Telegram/…/calls_call.h and .cpp: they run the MTProto DH handshake (phone.requestCall, phone.acceptCall, phone.confirmCall), validate hashes/fingerprints, then create a tgcalls controller using the agreed auth key and protocol layer. Telegram/…/calls_controller_webrtc.cpp adapts Telegram call endpoint/config structs into Webrtc::CallContext::Config; signaling bytes from updatePhoneCallSignalingData are forwarded into the running controller.
Group calls are much larger and are centered on Telegram/…/calls_group_call.h and .cpp. GroupCall joins via MTProto phone.joinGroupCall/connection updates, parses join-response JSON into either RTC video endpoints or broadcast/RTMP stream details, and creates tgcalls::GroupInstanceCustomImpl descriptors with audio level callbacks, media-device IDs, camera capture, video codec preferences, optional TdE2E encrypt/decrypt, and tasks for current time, audio/video broadcast parts, and media-channel descriptions. Broadcast playback requests map back into MTPupload_GetFile with MTP_inputGroupCallStream in broadcastPartStart(). Screen sharing is a second GroupInstanceCustomImpl with a loopback audio device module and VideoContentType::Screencast. Group-call chat/message support and conference encryption are adjacent in Telegram/SourceFiles/calls/group/calls_group_messages*.{h,cpp} and calls_group_message_encryption.*.
WebAuthn/passkeys have two native branches under Telegram/SourceFiles/webauthn/**. Telegram/…/webauthn_common.h exposes caBLE and security-key entry points; comments document that Linux/macOS always use libfido2 and Windows uses webauthn.dll where present with libfido2 fallback. caBLE public API is Telegram/…/cable.h, with register/login requests carrying RP/user/client-data-hash/algorithm fields and outcomes for QR success, cancellation, security-key fallback, Bluetooth absence, or failure. Telegram/…/cable_core.h and .cpp implement CBOR, SHA/HMAC/HKDF, P-256 ECDH, AES-GCM, Noise KNpsk0/NKpsk0 handshakes, caBLE QR/advert derivation constants, and encrypted CTAP framing. Telegram/…/cable_tunnel.cpp implements the TLS WebSocket tunnel to caBLE-assigned domains with subprotocol fido.cable, validates the Sec-WebSocket-Accept header, bounds upgrade and frame payload sizes, masks client frames, and surfaces binary messages. Telegram/…/cable_box.cpp is the UI boundary: it renders a QR with qr/qr_generate.h, uses a lottie passkey icon via lottie/lottie_icon.h, offers security-key fallback, and drives sheet state for QR, connecting, continue, and error screens.
Instant View and embedded web content are mediated by Telegram/SourceFiles/iv/**. Iv::Instance in Telegram/…/iv_instance.h and .cpp opens native IV pages, rich-message markdown, local markdown files, and Ton Sites; it tracks active windows per Main::Session, in-flight full-page and rich-message requests, HTML exports, and markdown windows. Rich-message media activation in iv_instance.cpp routes photos/documents into the media viewer and external URLs through click handlers, while Ton Site URLs are delegated to a webview controller. Telegram/…/iv_controller.h owns an Iv::Controller with Webview::Window, Webview::StorageId, navigation buttons, and event types for close/open/join/report; actual webview implementation is outside the covered source in the webview/... boundary. Markdown Instant View is handled by Telegram/…/iv_markdown_controller.h, which opens top-level Ui::RpWindows with history, search, zoom, source/share options, and prepared article content. The prepare pipeline in Telegram/…/iv_markdown_prepare.cpp converts either parsed markdown or native rich pages into bounded render data; it caps table dimensions, visual nesting, and total prepared blocks. Formula rendering crosses into MicroTeX through Telegram/…/iv_markdown_microtex.cpp, which initializes bundled resources, refuses very long TeX strings, rewrites full-width equation environments, caps logical/physical image size, and paints formulas into Qt images.
Coupling
- If call lifecycle behavior changes in Telegram/…/calls_instance.cpp or Telegram/…/calls_group_call.cpp, verify Telegram/…/media_player_instance.cpp: the audio player pauses/resumes on
Core::App().calls().currentCallValue()andcurrentGroupCallValue(), so stale call-state signals leave music or voice messages playing during calls. - If FFmpeg demuxing is added outside Telegram/…/ffmpeg_utility.cpp, Telegram/…/media_audio_ffmpeg_loader.cpp, or Telegram/…/media_clip_ffmpeg.cpp, also enforce
FFmpeg::RestrictToCustomIO()beforeavformat_open_input(). That invariant prevents media files from causing FFmpeg demuxers to open external URLs or local paths. - If
Media::Streaming::Instanceconstruction orData::Session::streaming().sharedDocument(...)changes in Telegram/…/media_streaming_instance.cpp, review both media viewer playback and Telegram/…/video_editor.cpp; the video editor relies on the sameframe()and update path as playback for crop preview, cover preview, and trim UI. - If editor
PhotoModifications, brush tools, or text preferences change in Telegram/…/photo_editor.cpp, preserve the versionedQDataStreamdeserializer. Existing local settings may contain old brush payloads, and the code intentionally accepts both legacy and negative-version formats. - If group-call join-response JSON changes in Telegram/…/calls_group_call.cpp, update both RTC endpoint handling and broadcast/RTMP handling.
ParseJoinResponse()feedssetJoinResponsePayload()for tgcalls RTC instances and separately toggles stream/RTMP mode, URL, key, and broadcast DC selection. - If video capture creation changes in Telegram/…/calls_instance.cpp, verify one-to-one calls, group camera, and screen share paths.
Instance::getVideoCapture()reuses one weak cachedtgcalls::VideoCaptureInterfaceand switches devices when a caller passes a device ID; changing reuse semantics can break camera switching or screen-capture selection. - If caBLE crypto constants or CBOR layout change in Telegram/…/cable_core.h or .cpp, check Telegram/…/cable_tunnel.cpp and Telegram/…/cable_box.cpp: QR payload derivation, tunnel IDs, Noise traffic keys, and UI timeout/outcome reporting must continue to agree with authenticators.
- If Instant View markdown limits change in Telegram/…/iv_markdown_prepare.cpp or formula caps change in Telegram/…/iv_markdown_microtex.cpp, test local markdown, rich messages, and native IV pages. They share the same prepared-content model but have different table limits and media runtimes.
- If webview storage, navigation, or availability behavior changes at the
webview/...boundary, re-check Telegram/…/iv_controller.h and Ton Site opening in Telegram/…/iv_instance.cpp; IV keeps only aWebview::Windowpointer and surfaces navigation/actions throughIv::Controller::Event, so failures appear as viewer-level errors rather than compile-time contract breaks.
Risks
- Telegram/…/ffmpeg_utility.cpp, Telegram/…/media_audio_ffmpeg_loader.cpp, and Telegram/…/media_clip_ffmpeg.cpp: if any demux path skips
RestrictToCustomIO(), a crafted media file can make FFmpeg resolve referenced network segments or local files. The code comments explicitly call this an IP-leak/local-file-read vector. - Telegram/…/calls_group_call.cpp: group-call streaming depends on several asynchronous tgcalls callbacks returning to the main thread and then into MTProto requests (
requestAudioBroadcastPart,requestVideoBroadcastPart,requestMediaChannelDescriptions). If task cancellation or_broadcastPartsbookkeeping is wrong, tgcalls can wait on a part whose MTProto request was already cancelled or whoseLoadPartTaskhas been erased. - Telegram/…/cable_tunnel.cpp: the WebSocket tunnel is implemented manually, including HTTP upgrade parsing, frame lengths, masking, ping/pong, and close handling. If length bounds or accept-header validation regress, passkey ceremonies can fail silently or accept malformed tunnel traffic.
- Telegram/…/iv_markdown_microtex.cpp: MicroTeX parsing happens synchronously before render caps apply. The file mitigates this with
kMaxFormulaTexLength; raising or removing that cap can stall the UI on hostile markdown before image-size limits are reached.