Storage persistence and media transfer
Overview
This area owns Telegram Desktop's local tdata/ persistence format, in-memory media indexes, file upload/download plumbing, local media preparation, and the export takeout download path.
Local persistence is split into global, domain, and per-account boundaries:
- Global settings are read and written by Telegram/…/localstorage.cpp.
Local::start()createstdata/, readssettings, falls back to oldconfigand old user/MTP files when needed, and applies global theme/background/language/fallback-config state.Local::writeSettings()writes the current global settings undertdata/settings*with an encrypted payload and also saves AyuGram settings before serializing the Telegram settings blocks. - Domain/account discovery is handled by Telegram/…/storage_domain.cpp. The modern domain key file is
tdata/key_<dataName>*: it stores passcode salt, an encrypted local storage key, and a local-key-encrypted account list plus active account index.Domain::start()first tries this modern file, falls back to legacy single-account startup throughAccount::legacyStart(), and rewrites accounts when the stored version is older thanAppVersion. - Per-account storage is in Telegram/…/storage_account.cpp.
Accountcomputes an account base pathtdata/<dataName hash>/, a per-account database pathtdata/user_<dataName>/, and a temp pathtdata/temp_<dataName>/. Its encryptedmapfile is an index of generatedFileKeyvalues for drafts, locations/download state, prefs, stickers, trusted peers, export settings, search suggestions, playback positions, bot storage, and other account-local state. Each value file is separately encrypted with the domain local key; the map only records which generated key belongs to which logical bucket. - The wire format utilities live in Telegram/…/storage_file_utilities.cpp. Files have a
TDF$header, app version, QDataStream payload, and md5 trailer.FileWriteDescriptorstages payloads in memory and hands them to an asyncWriteManager, which writes a modern safe...sfile usingQSaveFileand removes legacy...0/...1backups on success.ReadFile()prefers the...sfile, otherwise tries the legacy pair by modification time and validates magic/version/md5 before exposing aQDataStream. - Encryption is local-key based in storage_file_utilities.cpp:
CreateLocalKey()derives the modern key with salted SHA-512/PBKDF2-HMAC-SHA512, whileCreateLegacyLocalKey()uses the older SHA1 PBKDF2 path.PrepareEncrypted()prefixes plaintext length, pads to AES block size, stores the first 16 bytes of SHA1 as the data key, and encrypts withMTP::aesEncryptLocal;DecryptLocal()reverses this and rejects size or hash mismatches.
The per-account map is intentionally a directory of encrypted blobs rather than one monolith. For example, Account::writeLocations() serializes file-location records, aliases, and the downloads-manager snapshot into the _locationsKey blob, then writeMap() persists the _locationsKey pointer. Account::readLocations() restores local file paths into _fileLocations and _fileLocationPairs; writeFileLocation(), readFileLocation(), and removeFileLocation() are the boundary for downloaded media path persistence. Export settings are another map-managed blob: Account::writeExportSettings()/readExportSettings() serialize Export::Settings, including media flags, output format/path, availability time, optional single peer, and date bounds.
File locking is a separate platform abstraction in Telegram/…/storage_file_lock.h, storage_file_lock_posix.cpp, and storage_file_lock_win.cpp. Both implementations reserve the first four bytes (FileLock::kSkipBytes) as the lock range. POSIX uses fcntl(F_SETLK) and may terminate the locking pid; Windows uses LockFile and Restart Manager shutdown against the file owner.
In-memory media indexes are exposed through Storage::Facade in Telegram/…/storage_facade.cpp. It wraps SharedMedia (storage_shared_media.cpp) and UserPhotos (storage_user_photos.cpp), which keep sparse message/photo id slices in memory and publish RPL updates. The shared-media lists use SparseIdsList from storage_sparse_ids_list.cpp, where known no-skip ranges merge into slices and queries can return skipped counts when the top or bottom boundary is known. These indexes are not the encrypted on-disk store; they cache server-backed slices for UI queries.
The generic download entry point is CreateFileLoader() in Telegram/…/file_download.cpp. It dispatches a DownloadLocation to:
mtpFileLoaderin file_download_mtproto.cpp for Telegram storage files, bot/web files served through MTProto, geo-point tiles, and audio-album thumbnails.webFileLoaderin file_download_web.cpp for plain HTTP(S) URLs.- an in-memory loader for already available bytes.
FileLoader first tries local cache when LoadToCacheAsWell is enabled, using session->data().cache() and a subclass-provided cache key. Cache values may be full bytes or partial: prefixed bytes. If local cache misses and cloud loading is allowed, the subclass starts network loading. Incoming parts are written either to _data for in-memory loads or to _file for LoadToFileOnly; finalizeResult() stores local file locations through session->local().writeFileLocation(), writes small completed or partial results back into the media cache, closes downloaded files, and fires task completion.
MTProto downloads are coordinated by Telegram/…/download_manager_mtproto.cpp. DownloadManagerMtproto keeps per-DC queues, priorities, and adaptive download session counts. DownloadMtprotoTask::sendRequest() issues upload.getFile, upload.getWebFile, or CDN requests with a fixed kDownloadPartSize of 128 KiB. It handles CDN redirects by switching DC/token/key state, decrypting CDN parts with AES-CTR, verifying CDN file hashes, and reuploading/requesting hashes when the server asks. mtpFileLoader implements the task callbacks: it advances offsets, writes parts through FileLoader::writeResultPart(), marks completion on short/empty parts or requested size, and refreshes stale file references through Data::UpdatedFileReferences before retrying.
Plain web downloads use WebLoadManager in file_download_web.cpp, a shared manager with its own QNetworkAccessManager thread, at most eight concurrent requests, and a 5-redirect limit. Only http and https URLs are accepted; redirects are resolved and revalidated to avoid unsafe schemes. Non-streaming plain web loads are capped to Storage::kMaxFileInMemory, while streaming-to-file loads are capped to 4 GiB and pass incremental QNetworkReply::readAll() chunks to webFileLoader::loadProgress().
Streamed media save-as/download is bridged by Telegram/…/streamed_file_downloader.cpp. StreamedFileDownloader wraps a media streaming Reader, requests up to 32 streaming parts concurrently, writes each loaded part through FileLoader, exposes readLoadedPart() for already saved chunks, and finalizes only after every streaming part has been saved.
Local upload preparation starts with TaskQueue and FileLoadTask in Telegram/…/localimageloader.cpp, with drag/drop and list preparation helpers in storage_media_prepare.cpp. TaskQueue processes media inspection and thumbnail work on a worker thread and calls finish() back on the queue owner thread. FileLoadTask::process() determines MIME/media type from a file path or in-memory bytes, prepares images/photos, videos, voices, stickers, thumbnails, attached-sticker metadata, optional animation/video transcode jobs, and archive requests. Its result is FilePrepareResult, which carries file bytes/path, generated ids, parts/md5, thumbnail parts, MTP photo/document stubs, and send options.
Uploads are sent by Storage::Uploader in Telegram/…/file_upload.cpp. Uploader::upload() creates local upload state on PhotoData/DocumentData, queues optional video transcode or folder archive preparation, then maybeSend() schedules file parts across upload DC sessions. Photos and thumbnails use pre-sliced 32 KiB in-memory parts from FilePrepareResult; documents are streamed from memory or QFile with a size-dependent part size and use upload.saveFilePart or upload.saveBigFilePart above 30 MiB. The uploader adaptively adds/removes upload sessions based on fast/slow request timing, emits progress/failure RPL streams, and fires ready events that call ApiWrap::sendUploadedPhoto, sendUploadedDocument, secure upload completion, secondary-file completion, or video-cover upload.
Export is a separate takeout/download pipeline under Telegram/SourceFiles/export/. Export::Manager (export_manager.cpp) owns one Controller and UI panel at a time, starting whole-account, single-peer, or topic export. ControllerObject in export_controller.cpp normalizes settings, creates an output writer, drives high-level steps (initialization, dialogs list, personal info, userpics, stories, profile music, contacts, sessions, other data, dialogs, or a single topic), maps progress into ProcessingState, and stops on API or output errors.
The export MTProto work is in Telegram/…/export_api_wrap.cpp. ApiWrap::startMainSession() opens an account takeout session, setting flags according to requested data and media size limit. Subsequent requests are wrapped in InvokeWithTakeout; dialog requests may also be wrapped in message split ranges. Userpics, stories, profile music, messages, custom emoji, rich messages, and other data are converted into export/data/export_data_types.* models, then file payloads are downloaded by processFileLoad(). Export files write through Output::File (export/…/export_output_file.cpp), which prepares a collision-free relative path, opens/appends the output file, writes blocks, flushes, and increments export stats.
Export media download is intentionally independent from FileLoader. ApiWrap::loadFile() creates a FileProcess, then loadFilePart() requests upload.getFile with kFileChunkSize (8 * 128 KiB) through the takeout session and shifted export media DC. filePartDone() writes ordered chunks into Output::File, reports progress, caches previously downloaded locations in LoadedFileCache, and finishes when the known size is reached. Export does not support CDN redirects in this path. If a FILE_REFERENCE_* error occurs, filePartRefreshReference() reloads the owning story/message/custom emoji/rich message and retries with a refreshed Data::FileLocation; otherwise unavailable files are marked with a skip reason.
Coupling
- If the on-disk file envelope changes in Telegram/…/storage_file_utilities.cpp, update all readers and writers that assume
TDF$, version, payload, and md5 semantics: storage_domain.cpp, storage_account.cpp, and localstorage.cpp all callReadFile()/FileWriteDescriptordirectly and will reject files if the envelope or QDataStream version drifts. - If account-map key tags change in Telegram/…/storage_account.cpp, update both
readMapWith()andwriteMap()together. The map is the only index from logical state (drafts, locations, export settings, prefs, bot storage) to encrypted value-file keys, so adding a writer without a reader leaks data, and adding a reader without a map writer loses the state on the next rewrite. - If
Core::FileLocationserialization changes, verifyAccount::writeLocations()andAccount::readLocations()in Telegram/…/storage_account.cpp together withFileLoader::finalizeResult()in Telegram/…/file_download.cpp. Downloads rely on this path both to remember saved-to-disk media and to recover the downloads-manager serialized snapshot. - If
DownloadLocation,StorageFileLocation, or cache-key generation changes, updateCreateFileLoader()(file_download.cpp),mtpFileLoader::cacheKey()/fileLocationKey()(file_download_mtproto.cpp), and the MTProto request construction in download_manager_mtproto.cpp; otherwise a media object can download successfully but be cached under the wrong key or lose its persisted local file location. - If MTProto download part sizing changes, check
Storage::kDownloadPartSizein download_manager_mtproto.h, CDN hash checking in download_manager_mtproto.cpp, partial cache resume inmtpFileLoader::startLoadingWithPartial(), andFileLoader::finalizeResult()partial-cache writes. CDN hash validation and partial-cache offsets assume one fixed part size. - If upload file-size thresholds or part sizes change in Telegram/…/file_upload.cpp, verify
FilePrepareResult::setFileData()in localimageloader.cpp and MTProto input-file construction inUploader::finishFront(). Small files need md5 hashes andinputFile; files abovekUseBigFilesFromuseinputFileBigand no md5. - If local media MIME/type detection changes in Telegram/…/localimageloader.cpp or storage_media_prepare.cpp, verify
Uploader::upload()andUploader::finishFront()in file_upload.cpp. The uploader trustsFilePrepareResult::type,fileparts,thumbparts,document, andphotofields to choose progress state and final API calls. - If
Export::Settingschanges in Telegram/…/export_settings.h, update both persistence inAccount::writeExportSettings()/readExportSettings()(storage_account.cpp) and normalization/step selection in export_controller.cpp/export_api_wrap.cpp. Saved export settings and takeout flags must agree on media masks, single-peer/topic fields, date limits, and size limits. - If export file locations or file-reference refresh rules change in
export/data/export_data_types.*, also updateApiWrap::ComputeLocationKey(),processFileLoad(),filePartRefreshReference(), and theData::RefreshFileReference()call sites in export_api_wrap.cpp. Export de-duplicates downloaded files and retries stale references by comparing parsedData::FileLocationvalues. - If the output writer contract changes in export/…/export_output_file.cpp, verify all
Output::AbstractWriterimplementations andApiWrap::writePreloadedFile()/filePartDone(). Export progress and final stats count files/bytes throughOutput::File, not through the genericFileLoadercache.
Risks
- Telegram/…/storage_sparse_ids_list.cpp contains an AyuGram-specific guard replacing an
Expects(noSkipRange.from <= noSkipRange.till)crash with an early return inSparseIdsList::addRangeItemsAndCountNew(). If shared-media no-skip ranges are changed to rely on invalid ranges being visible, those entries will now be silently dropped and shared media counts can diverge from server state. - Telegram/…/file_download_web.cpp explicitly restricts web downloads and redirects to HTTP(S) because other schemes can expose local/UNC paths. If
IsSupportedWebUrl()or redirect handling is relaxed, plain web media loads can become a credential-disclosure or local-file access path. - Telegram/…/export_api_wrap.cpp treats
upload.fileCdnRedirectas a hard export error infilePartDone(), while the normal app downloader supports CDN redirects in download_manager_mtproto.cpp. If export begins receiving CDN redirects for takeout media, affected exports will stop instead of downloading those files. - Telegram/…/storage_file_lock_posix.cpp and storage_file_lock_win.cpp can terminate processes holding the target file lock. If this lock is reused for user-selected files rather than app-owned coordination files, another application holding the file may be killed unexpectedly.