Storage and SQLite
Overview
This doc owns the Telegram SQLite wrapper and the persistent storage surfaces covered by:
TMessagesProj/src/main/java/org/telegram/SQLite/**- TMessagesProj/…/MessagesStorage.java
- TMessagesProj/…/DatabaseMigrationHelper.java
- TMessagesProj/…/FilePathDatabase.java
- TMessagesProj/…/FilesMigrationService.java
TMessagesProj/schemas/**TMessagesProj/jni/sqlite/**- TMessagesProj/…/SqliteWrapper.cpp
MessagesStorage.java is the main per-account message database owner. Each MessagesStorage singleton creates a DispatchQueue named storageQueue_<account> and opens cache4.db on that queue from the constructor. Account 0 stores the database under ApplicationLoader.getFilesDirFixed(), while nonzero accounts route to account<id>/cache4.db; WAL and SHM sidecars are tracked as cache4.db-wal and cache4.db-shm. openDatabase() waits briefly for NativeLoader, opens the database through the custom SQLiteDatabase wrapper, applies secure_delete, memory temp store, journal_mode = AyuConfig.getWALMode(), and a 10 MiB journal size limit. New databases are initialized by MessagesStorage.createTables(), and existing databases read PRAGMA user_version, load sync parameters from params, and migrate when below LAST_DB_VERSION (117).
The message database stores more than message rows. MessagesStorage.createTables() creates dialog and message state (dialogs, messages_v2, messages_topics, hole tables, scheduled messages), media indexes (media_v4, media_topics, media counts and holes), users/chats/encrypted chats, contacts, bot keyboards, dialog filters, download/sent-file queues, pending tasks, reactions, topics, emoji/sticker caches, wallpapers, app config, and other feature caches. Serialized Telegram objects are written as NativeByteBuffer BLOBs and read back with the corresponding TLRPC.*.TLdeserialize() methods; examples include message reads in getMessagesInternal(), sent-media lookups in getSentFile(), download queue reads in getDownloadQueue(), and user/chat persistence through putUsersAndChatsInternal().
Most public storage entry points either post work to storageQueue or take an explicit useQueue flag. putMessages(ArrayList<TLRPC.Message>...) posts to putMessagesInternal() when requested; putMessages(TLRPC.messages_Messages...) always posts and writes fetched history, updates hole ranges, stores message/media rows, stores users/chats, and schedules file deletion through FileLoader when media changes. getMessages() reads rows on storageQueue, resolves users/chats/animated emoji, and returns to MessagesController.processLoadedMessages(). loadPendingTasks() rehydrates pending_tasks after open and dispatches controller work to MessagesController, SendMessagesHelper, UI thread, or Utilities.stageQueue depending on task type. Download queue APIs (getDownloadQueue(), removeFromDownloadQueue(), clearDownloadQueue()) persist queued media and call DownloadController on the UI thread with deserialized DownloadObjects.
DatabaseMigrationHelper.java is the ordered migration script for cache4.db. MessagesStorage.updateDbToLastVersion() toggles NotificationCenter.onDatabaseMigration, calls DatabaseMigrationHelper.migrate(), and expects it to advance PRAGMA user_version to MessagesStorage.LAST_DB_VERSION. The migration helper contains historical step blocks from very old schemas through version 117, including table renames such as messages to messages_v2, dialog-id normalization for encrypted dialogs, topic tables, reaction tables, and recent additions like app_config, bot_keyboard_topics, and reply indexes. It also owns recoverDatabase(): on malformed database errors, MessagesStorage.checkSQLException() can build a fresh database in recover_database_<account>/, attach the old cache4.db, copy safe whole tables, selectively preserve secret-chat rows and last messages, recreate holes, then replace the broken database files.
FilePathDatabase.java is a separate per-account SQLite database named file_to_path.db with a backup file file_to_path_backup.db. It maps downloaded document IDs/DC/type triples to actual paths in paths, and maps filesystem paths to dialog/message metadata in paths_by_dialog_id for cache-by-chat cleanup and media existence checks. It lazily creates its own DispatchQueue named files_database_queue_<account>, opens only after ensureDatabaseCreated(), and has an independent LAST_DB_VERSION of 4. On open it creates paths and paths_by_dialog_id, migrates older versions by adding the path index and message metadata columns, and writes a copy backup unless it opened from backup. On repeated open failures it restores from backup or deletes the local files and retries.
FilesMigrationService.java is not a SQLite migration; it is an Android R scoped-storage file mover. It starts as a foreground service, locates the old root folder using SharedConfig.storageCacheDir or external storage plus AyuConstants.APP_NAME, moves files into applicationContext.getExternalFilesDir(null)/<APP_NAME>, updates progress notification 301, and marks systemConfig.migration_to_scoped_storage_finished. Its FilesMigrationBottomSheet gates the service behind read/write permissions and tracks how often the prompt is shown.
The Java SQLite wrapper in TMessagesProj/src/main/java/org/telegram/SQLite/ is a thin JNI-backed API over the vendored SQLite source in TMessagesProj/jni/sqlite/. SQLiteDatabase owns the native database handle, opens with a temp directory from ApplicationLoader.getFilesDirFixed(), exposes executeFast(), queryFinalized(), executeInt(), explicit BEGIN/COMMIT, and closes in close()/finalize(). SQLitePreparedStatement owns a native statement, binds Java primitives, String, ByteBuffer, and NativeByteBuffer, logs slow queries over 500 ms, and must be disposed to finalize native resources. SQLiteCursor steps a prepared statement, maps SQLite row values back to Java and NativeByteBuffer, and retries SQLITE_BUSY six times with 500 ms sleeps before throwing SQLiteException. SqliteWrapper.cpp implements these JNI methods with sqlite3_prepare_v2, sqlite3_step, sqlite3_bind_*, sqlite3_column_*, sqlite3_open, and sqlite3_close; BLOB reads for NativeByteBuffer allocate via BuffersStorage and copy SQLite column bytes.
TMessagesProj/…/*.json are Room schema snapshots for a separate Ayu database, not the MessagesStorage cache4.db schema. The current snapshot version 21 records EditedMessage, DeletedMessage, and DeletedMessageReaction tables with Room identity hash metadata. These schemas are still storage-facing because Ayu features referenced from MessagesStorage.java (for example edited-message handling around AyuMessagesController and AyuSavePreferences) may serialize related message fields and paths outside cache4.db.
Coupling
- If
MessagesStorage.LAST_DB_VERSIONchanges in MessagesStorage.java, update bothMessagesStorage.createTables()and the ordered steps inDatabaseMigrationHelper.migrate(). New installs usecreateTables()directly, while existing installs only see the migration path andPRAGMA user_version; a mismatch silently produces different schemas for fresh and upgraded users. - If a table is added, removed, or changes column order in
MessagesStorage.createTables(), also updateMessagesStorage.DATABASE_TABLESandDatabaseMigrationHelper.recoverDatabase(). Recovery copies many tables withINSERT OR IGNORE INTO %s SELECT * FROM old.%s, so column-order drift or missing table entries breaks malformed-database recovery. - If message row columns change in
messages_v2,messages_topics,scheduled_messages_v2,media_v4, ormedia_topics, audit every prepared statement in MessagesStorage.java and migration copy block in DatabaseMigrationHelper.java. The code uses positionalREPLACE INTO ... VALUES(...)and positional cursor reads, not named columns, so adding a column requires updating bind counts and read indexes together. - If the serialization format of stored
TLRPCobjects orNativeByteBufferlayout changes, verify all BLOB readers and writers in MessagesStorage.java, DatabaseMigrationHelper.java,SQLiteCursor.byteBufferValue(), and SqliteWrapper.cpp. Stored rows are deserialized by constructor IDs from raw blobs; incompatible buffers surface later as empty or malformed messages rather than as SQL errors. - If
FilePathDatabase.paths_by_dialog_idchanges, updatesaveFileDialogId(),getFileDialogId(),removeFiles(),lookupFiles(), andmigrateDatabase()together. Cache cleanup and chat-scoped media retention depend on path-to-dialog metadata, while older databases must add new columns explicitly. - If file storage roots or scoped-storage behavior change in FilesMigrationService.java, check FilePathDatabase.java path mappings and any
FileLoaderpath users called from MessagesStorage.java. Moving files without keepingfile_to_path.dband cache cleanup metadata coherent can make existing downloads appear missing or make cleanup delete the wrong files. - If JNI signatures change in SQLiteDatabase.java, SQLitePreparedStatement.java, or SQLiteCursor.java, update TMessagesProj/…/SqliteWrapper.cpp with the exact Java symbol names. The wrapper does not use reflection or registration tables; stale JNI names fail only when native methods are invoked.
- If Ayu Room entities change, commit the matching TMessagesProj/…/<version>.json snapshot. These JSON files are the migration/audit record for the Ayu database and are separate from
cache4.dbmigrations in DatabaseMigrationHelper.java.
Risks
- DatabaseMigrationHelper.java contains long manual migration chains with positional data transforms. If a mid-chain migration is edited without testing upgrade paths from older
PRAGMA user_versionvalues, users can land onLAST_DB_VERSIONwith missing columns or partially copied rows; migrations around message table rewrites and dialog-id normalization are especially sensitive because they deserialize and reserializeNativeByteBufferrows. - MessagesStorage.java and DatabaseMigrationHelper.java use many raw SQL strings built with
String.format()and positionalVALUES(...). If a path, ID list, or schema value is not sanitized consistently, queries can break on quoting or column-count changes;FilePathDatabase.shield()only strips quotes for some path queries and is not general SQL binding. SQLitePreparedStatement.bindByteBuffer()in Java and SqliteWrapper.cpp bind blobs withSQLITE_STATIC. If the backingByteBufferorNativeByteBufferis reused before SQLite finishes stepping the statement, stored blob contents can become corrupted. Existing callers usually callstep()beforereuse(), so preserve that order when adding writes.SQLiteCursor.next()blocks for up to about three seconds onSQLITE_BUSY. If storage code is accidentally run on the main thread or a synchronousCountDownLatchcall is made from the same dispatch queue, UI stalls or deadlocks are possible;FilePathDatabase.getPath()has a debug guard for same-queue use, but many MessagesStorage.java APIs rely on callers choosinguseQueuecorrectly.