Introducing a new way to review your PRs.

Read the announcement

twentyhq / twenty

The open alternative to Salesforce, designed for AI.

View on GitHub

22 documents · Updated Nov 20, 2026

docs / backend-data-events

Data persistence and event processing

The backend stores workspace data, turns committed changes into events, and moves slow work to workers. These paths support audit and usage views, webhooks, workflows, logic functions, agent triggers, live subscriptions, deferred wake-ups, billing state, exports, and safe cleanup without crossing workspace boundaries. See the backend platform page for the wider server boundary and API, metadata, and workspace runtime for request metadata and permission rules.

Persistence boundary

The persistence boundary gives each workspace a data view that combines its records with workspace metadata and access context. Normal reads and writes use the active workspace's object definitions and access rules. System jobs can use a controlled bypass for maintenance work.

Workspace data boundary

The workspace ORM manager selects the data source for the active workspace and provides repository and transaction access. A transaction commits record changes before their events reach other parts of the system. Workspace cache state is cleared after maintenance work so a failed callback cannot leave later work with stale metadata.

The workspace repository also builds the record snapshots needed for change events. Bulk system writes may suppress those events only when the caller accepts that webhooks, workflow triggers, timeline activity, and similar consumers will not run.

References: packages/…/global-workspace-orm.manager.ts• L33-69 packages/…/global-workspace-orm.manager.ts• L124-198 packages/…/workspace-repository.ts• L126-164 packages/…/workspace-repository.ts• L1393-1423 packages/…/twenty-orm.module.ts• L12-24

Database initialization

Database initialization creates the shared public and core schemas and enables the extensions needed by the server. Optional foreign-data wrappers are added only when that integration mode is enabled. The truncation path removes workspace schemas in small groups while keeping shared system schemas.

References: packages/…/setup-db.ts• L5-61 packages/…/setup-db-utils.ts• L6-34 packages/…/truncate-db.ts• L5-45

Record data contracts

Record data contracts turn workspace field metadata into the value and filter shapes used by record tools and queries. They omit system-managed fields and fields that the current role cannot update. They require fields that have no default and represent writable many-to-one relations through their join identifiers.

Filter rules follow the field type. Relation filters use record identifiers and option fields use their current option values. This keeps record filtering and record writes tied to the same metadata contract.

References: packages/…/field-filters.json-schema.ts• L14-33 packages/…/record-properties.json-schema.ts• L26-46 packages/…/record-properties.json-schema.ts• L159-213

Application metadata state

An application manifest is converted into workspace metadata maps before the application can use it. The resulting state covers objects, fields, indexes, permissions, logic functions, views, page layouts, variables, and application UI declarations. Secret workspace variables are stored in encrypted form; user-scoped variables keep only a default at workspace level.

References: packages/…/compute-application-manifest-all-universal-flat-entity-maps.service.ts• L64-143 packages/…/compute-application-manifest-all-universal-flat-entity-maps.service.ts• L171-229 packages/…/compute-application-manifest-all-universal-flat-entity-maps.service.ts• L602-649

Schema upgrade processing

Schema upgrades keep existing instances and workspaces aligned with current data and metadata rules. Instance changes add or repair shared tables, columns, keys, permissions, and indexes. Workspace changes run per workspace, use dry-run validation when needed, and apply only safe additions or cleanup.

The upgrade path favors bounded, repeatable work. It provisions new workspace objects and relations, backfills standard data, reconciles catalog state, repairs permission and command metadata, and removes replaced or expired data. Workspace iteration selects eligible workspaces, supports shards and limits, records failures separately from deleted workspaces, and evicts local cache state after each workspace.

Recent upgrades continue this model. They demote conflicting application index views before restoring engine-owned views, add record-share access options with rollback protection, backfill agent-turn indexes without blocking writes, and move stored field options into their normalized rows. These changes are registered as instance or workspace commands so older workspaces receive the same state as new ones.

Raw upgrade writes can change stored data without using the normal event path. Such commands recompute the affected cache maps after success. Compare-and-set writes and transactions protect changes made while an upgrade is running.

References: packages/…/workspace-iterator.service.ts• L22-43 packages/…/workspace-iterator.service.ts• L79-169 packages/…/2-59-upgrade-version-command.module.ts• L12-24 packages/…/2-59-workspace-command-1794548649004-backfill-agent-turn-agent-id-created-at-index.command.ts• L44-71 packages/…/2-59-instance-command-fast-1794590652080-backfill-field-metadata-option-rows-from-stored-options.ts• L36-55 packages/…/2-59-instance-command-slow-1794588217126-add-is-internal-import-enabled-to-workspace.ts• L14-39 packages/…/instance-commands.constant.ts• L303-308 packages/…/instance-commands.constant.ts• L611-616

Agent history migration

Agent history can move between shared and workspace storage without losing active conversations. A durable route and migration cursor identify where each workspace's history lives. The migration copies data in resumable batches, validates active streams and linked records, and changes the route only after the target is ready.

Missing route state is treated as unsafe when a legacy history table contains data. This prevents the server from silently reading an empty snapshot. The migration also preserves legacy fields for older upgrade paths and refuses a rollback that would discard sender identity or linked targets. Tests cover crash recovery, concurrent runners, exact values, rollback safety, and protection of active streams.

The legacy field set is reconstructed from standard object metadata when an upgrade still needs columns that newer standard definitions have removed. This keeps intermediate history moves compatible with the upgrade versions that read or copy those fields.

References: packages/…/agent-history-migration-state.service.ts• L13-101 packages/…/get-legacy-agent-history-flat-field-metadatas.util.ts• L10-131 packages/…/agent-history-migration.postgres.spec.ts• L565-697 packages/…/agent-history-migration.postgres.spec.ts• L1196-1286

Workspace export

Workspace export produces a self-contained SQL file for one workspace. It includes the shared rows needed by that workspace, metadata-derived workspace schema definitions, permission tables, and workspace records in bounded batches. The export can limit which workspace objects it includes and streams output so large workspaces do not need to fit in memory.

The schema is rebuilt from current metadata rather than copied as an opaque database shape. Field option rows are resolved into the field metadata used to build that schema, which keeps exported option values aligned with the current model. The export also separates core data, workspace data, and permission storage so unrelated workspace rows do not enter the file.

References: packages/…/workspace-export.service.ts• L74-119 packages/…/workspace-export.service.ts• L160-202 packages/…/workspace-export.service.ts• L409-467 packages/…/workspace-export.service.ts• L516-589 packages/…/generate-workspace-schema-ddl.util.ts• L75-157

Billing and usage state

Billing state links a workspace to an external subscription and stores the local customer, subscription, item, entitlement, and credit records. Webhook and direct-sync paths update these records under a workspace-specific lock, then refresh the derived billing cache. Subscription status can suspend or reactivate a workspace, so billing persistence affects workspace availability.

Usage limits combine stored workspace rules with plan defaults and billing entitlements. Quota and stock counters use fast shared state while usage events provide the durable usage record. Credit grants form an idempotent ledger with effective and expiry times, and derived subscription state includes only grants that are still usable.

References: packages/…/billing-webhook-subscription.service.ts• L76-87 packages/…/billing-webhook-subscription.service.ts• L127-218 packages/…/billing-credit-grant.service.ts• L17-99 packages/…/usage-limit.service.ts• L59-99

Database event flow

Database events are the shared change signal for workspace records. A record change becomes a batch with the workspace, object metadata, action, and one event per affected record. Updates carry before and after values plus changed fields. Create, delete, restore, destroy, and upsert actions carry the state that is available for that action.

Event creation

The ORM builds event data from the affected records while it performs the write. The workspace event emitter names the event from the object and action, then sends it through the in-process event bus. A committed change can therefore feed several downstream paths without coupling the write to each consumer.

Change calculation ignores storage-only fields such as updatedAt and searchVector. A relation change keeps its logical field name and also reports the canonical join-column name so field watchers can match the physical change. Position changes remain visible when they are the only changed field.

This diagram shows how one committed workspace change reaches independent downstream features.

flowchart LR
    A[Workspace record change] --> B[Workspace-aware ORM]
    B --> C[Database event batch]
    C --> D[In-process event listeners]
    D --> E[Live subscriptions]
    D --> F[Webhook and workflow jobs]
    D --> G[Audit and timeline jobs]
    D --> H[Logic function and agent trigger jobs]

References: packages/…/workspace-repository.ts• L1393-1423 packages/…/workspace-repository.ts• L1752-1800 packages/…/format-twenty-orm-event-to-database-batch-event.util.ts• L33-51 packages/…/workspace-event-emitter.ts• L30-71 packages/…/object-record-changed-values.ts• L37-156 packages/…/format-result.util.ts• L51-69 packages/…/format-result.util.ts• L71-229

Event consumers

The central record listener fans one database event batch into the paths that need it. It publishes a live record event at once, then queues webhook delivery and database-triggered workflow work. Audit-enabled objects also queue event-log ingestion and timeline activity.

Agent and logic-function triggers apply role access, row access, and field masking before they enqueue runs. Webhook delivery uses a workspace-wide access gate because a webhook has no user identity, then applies rate limits and sends bounded retryable jobs. Metadata changes use a related path that updates cache state, enriches events, and broadcasts them to the right workspace recipients.

References: packages/…/entity-events-to-db.listener.ts• L35-161 packages/…/call-database-event-trigger-jobs.job.ts• L44-187 packages/…/call-agent-database-event-triggers.job.ts• L39-179 packages/…/call-webhook-jobs.job.ts• L37-107 packages/…/workflow-database-event-trigger.listener.ts• L85-161

Pending wake-ups

Pending wake-ups let a workflow, agent run, or other owner wait for a time, record event, or answer without keeping a request open. The wait is durable in the workspace data boundary. A delayed queue job carries the event or answer that can resume the owner.

Each owner can replace a wait for the same owner key. Re-arming gives the row a new identifier, so a stale job cannot claim the newer wait. Resolution claims the row by deleting it, which makes duplicate delivery harmless. Record events wake matching waits only inside the same workspace and after permission checks.

Owner handlers report WAITING, NOT_READY, or GONE. A not-ready owner keeps the wait and retries later. A gone owner drops it. The resolver reads event records with the owner's permissions and removes fields that the owner cannot read.

This diagram shows the durable wait path and its recovery boundary.

flowchart LR
    A[Owner arms a wait] --> B[Persist pending wake-up]
    B --> C{Time, event, or answer}
    C -->|Time| D[Delayed resolution job]
    C -->|Record event| E[Match and queue resolution]
    C -->|Answer| D
    E --> D
    D --> F[Check owner state and readable record]
    F --> G[Claim wait and deliver outcome]
    H[Overdue sweep] --> D

References: packages/…/pending-wake-up.entity.ts• L17-57 packages/…/pending-wake-up.service.ts• L18-205 packages/…/pending-wake-up-resolver.service.ts• L33-155 packages/…/pending-wake-up-database-event.listener.ts• L25-105

Worker queues

Worker queues move work that does not need to finish inside the record request out of the request path. A job has a name, typed data, and an explicit scope. The queue service sends it to a driver, while the worker registers the handler for that queue and job name.

Queue execution

The normal driver uses BullMQ over Redis. It stores waiting work, applies retries and backoff, retains recent job state, and reports waiting, completed, failed, and stalled work. A synchronous driver runs the same contract directly for local development and tests.

Queue settings control priority, parallel work, stalled-job recovery, retries, delay, deduplication, and shutdown behavior. Jobs can report progress and can name a workspace user who receives selected state changes through the subscription channel. Queue scope is explicit: system jobs have no workspace, workspace jobs carry a workspace, and application jobs carry both a workspace and application identifier.

Fair scheduling gives each scope a fair place in the queue. Group rate pacing spaces jobs that share a workspace or application budget. A limiter allows a short burst and then delays later jobs. The pacing slot is scoped to the workspace and group, so applications in one workspace share that group's schedule while another workspace has its own schedule. Urgent jobs bypass pacing. If Redis cannot calculate pacing, the queue records a fallback and continues without the extra delay.

When Redis memory fairness is enabled, the queue estimates each job's size and charges workspace or application groups against shared memory limits before enqueue. The whole bulk batch is refused when a charged group would cross its share. System jobs and critical maintenance queues are not charged. A completed or final failed job releases its charge, while a retryable failure keeps it for the next attempt.

Queue removal also protects the charge record. If removal cannot show that a job is gone, the charge stays until reconciliation can check it. A restore step can put the charge back when a removal race leaves the job unfinished. A reconcile command releases charges for jobs that are gone or finished, with a grace period so a charge is not released during the short gap before a job appears in Redis.

Each queue has a complete worker policy. Deployment settings can enable or exclude queues. Shutdown drains active workers, while long-running stream work can use a bounded drain and abort after its limit. Queue metrics separate runnable backlog, delayed work, memory use, completion, failure, and stalled jobs.

References: packages/…/job-options.interface.ts• L1-65 packages/…/message-queue-driver.interface.ts• L19-100 packages/…/message-queue-worker-options.interface.ts• L1-11 packages/…/message-queue-worker-config.constant.ts• L4-31 packages/…/bullmq.driver.ts• L262-531 packages/…/bullmq.driver.ts• L621-795 packages/…/bullmq.driver.ts• L1028-1157 packages/…/fair-queue-scheduler.ts• L54-174 packages/…/group-rate-pacer.ts• L39-143 packages/…/group-rate-pace-script.constant.ts• L1-45 packages/…/build-pace-slot-key.util.ts• L5-15 packages/…/is-valid-message-queue-limiter.util.ts• L3-10 packages/…/bullmq.driver.limiter.spec.ts• L19-330 packages/…/bullmq.driver.memory.spec.ts• L39-671

Queue workload groups

Queues group work by business purpose rather than by one event type:

Queue roleTypical work
Change deliveryDatabase-event persistence, webhooks, workflow triggers, and logic function or agent triggers
External syncCalendar, messaging, connected-account, and contact work
Campaign and mailCampaign materialization, delivery, email sending, and stats
MaintenanceCron work, workspace cleanup, cascades, and delayed jobs
Deferred resumptionPending wake-ups that resume owners after time, events, or answers
Long-running workLogic functions and AI streams

Fair scheduling uses scope keys so system work, workspace work, and application work remain distinct. Bulk fan-out can enqueue one job per workspace, which limits the effect of a large upgrade or import on unrelated tenants.

References: packages/…/build-queue-job-group-keys.util.ts• L5-30 packages/…/build-workspace-group-key.util.ts• L1-2 packages/…/build-application-group-key.util.ts• L1-3 packages/…/message-queue-driver.interface.ts• L26-32 packages/…/message-queue.constants.ts• L5-35

Scheduled maintenance jobs

Scheduled jobs turn recurring checks into the same queue-backed work model. The database command layer registers cleanup, wake-up, event-log, migration-recovery, workspace, billing, export, integration, session, and usage jobs as one set. Configuration can skip optional groups, while registration runs each enabled command independently and reports successes, failures, and skips instead of hiding a partial setup.

The shared registration command loads the command groups that own these jobs. This includes workspace iteration, export, migration, trash cleanup, event-log cleanup, pending wake-ups, queue-memory reconciliation, and workspace cleanup. It also registers pending file and session cleanup, workspace export cleanup, deferred workspace operation recovery, and billing and usage maintenance. Marketplace sync, signing-key rotation, and billing-related jobs are gated by configuration. Each command is attempted independently so one registration error does not hide the rest.

References: packages/…/cron-register-all.command.ts• L43-109 packages/…/cron-register-all.command.ts• L167-255 packages/…/cron-register-all.command.ts• L263-297 packages/…/database-command.module.ts• L69-120 packages/…/jobs.module.ts• L61-68

Billing and usage jobs

Recurring billing work checks its enablement before it acts. Subscription and entitlement changes update local workspace state, refresh caches, and use locks so two external updates cannot create conflicting transitions. Billing reminders record the boundary after sending so one boundary produces one reminder.

Usage admission checks subscription state, plan allowance, workspace limits, and spender scope before work runs. If a supporting read fails, quota enforcement records the degraded state and admits the operation rather than blocking a paying workspace.

References: packages/…/billing-reminder.service.ts• L61-177 packages/…/billing-reminder.service.ts• L204-269 packages/…/usage-quota-admission.service.ts• L32-67 packages/…/usage-limit-quota.service.ts• L318-429

Event log persistence

Event logs preserve workspace activity for audit, usage, page-view, and application-log views. Record events are validated into event envelopes and pass through the entity-event and event-log queues before they reach configured sinks. The sink stores the batch and publishes the same events to live event-log watchers.

Event log storage

Object events cover create, update, delete, upsert, restore, and permanent destroy activity. A destroyed record keeps its record and object identifiers but not its properties. Clients can also submit analytics and custom object events through the GraphQL event emitter, which attaches workspace and user context before using the same event-log path.

ClickHouse is the optional analytical store for these event rows. Its service can keep a main client and separate clients for registered external stores. Inserts use bounded chunks and can report failure without pretending that an unavailable client wrote the batch. Select paths either return an empty result for an unavailable store or raise a typed unavailable-store error when the caller requires the store.

ClickHouse migrations create the configured database, keep a migration table, skip files already applied, and run the remaining SQL in order. Event-log queries require an available analytical store and apply workspace retention and entitlement rules. A missing sink means the worker does not persist the event.

References: packages/…/event-log-emitter.resolver.ts• L31-117 packages/…/build-event-envelope.ts• L15-66 packages/…/create-event-log-from-internal-event.ts• L24-141 packages/…/clickhouse.service.ts• L29-105 packages/…/clickhouse.service.ts• L142-180 packages/…/run-migrations.ts• L29-79 packages/…/run-migrations.ts• L81-148 packages/…/event-logs.service.ts• L48-220

Application telemetry

Application telemetry separates an application log, which explains a notable event, from an application metric event, which counts repeated activity for trend views. The recorder keeps the first matching log in a short window and rolls up metric occurrences by application, workspace, source, function, level, error, and minute. It flushes metric batches on a timer, when its buffer is full, and during shutdown.

The metric table keeps application identity, source, severity, error code, duration, and occurrence count. It has a 90-day table-level expiry. Events without an application registration are not stored as application metrics. The console sink displays application logs but omits metric rows.

References: packages/…/application-repeated-event-recorder.service.ts• L19-135 packages/…/application-metric-event-rollup-buffer.ts• L5-51 packages/…/console-event.sink.ts• L9-40 packages/…/events.type.ts• L70-112

Event log retention

Event-log retention is derived from workspace settings and the audit-log entitlement. Cleanup groups workspaces that share a retention period, then removes old rows from each registered event table in bounded workspace batches. Application metric rows have their own analytical-store expiry, so their lifecycle can be shorter than the general event-log setting.

Transient analytical-store failures have a bounded recovery path. The event-log worker allows one explicit transient retry and then records a warning and drops the batch instead of retrying forever. Cleanup skips the operation when ClickHouse is not configured.

References: packages/…/event-log-retention.service.ts• L20-60 packages/…/event-log-cleanup.service.ts• L28-125 packages/…/create-event-log-from-internal-event.ts• L89-141

Live subscriptions

Live subscriptions let a client keep one GraphQL event stream for record and metadata changes. Redis pub/sub carries each payload. Cache storage keeps the stream identity, subscriber auth context, query signatures, and active lifetime.

Event stream

The event stream resolver creates and authorizes a stream, adds or removes query signatures, and refreshes the stream while the client sends heartbeats. A stream has a time limit and is removed on disconnect or expiry. This prevents inactive clients from remaining in the active-stream set.

Record events pass through permission checks before publication. The publisher checks object access, row filters, watched query filters, and restricted fields. It also keeps application access in the subscriber context and fails closed when that application is no longer active. Metadata events use workspace-wide delivery unless an event names specific recipient members.

Metadata publication enriches selected records before it broadcasts them. Field metadata events gain current field and option context and remove duplicate morph-relation changes. Navigation items can gain a target record identifier. Private workflows, unlisted views, and user-owned navigation items are limited to their intended recipients; missing parent metadata fails closed for child events.

Event-log subscriptions add a permission check for the selected table. Their heartbeat keeps the table marked as watched. Application-log rows are filtered again for the calling application before delivery.

This sequence shows stream setup, permission-aware publication, heartbeat refresh, and expiry.

sequenceDiagram
    participant Client
    participant Resolver
    participant Stream as Event stream state
    participant Publisher
    participant Redis

    Client->>Resolver: Open stream and add query
    Resolver->>Stream: Store auth context and query signature
    Publisher->>Stream: Read active streams and query state
    Publisher->>Publisher: Apply access and query filters
    Publisher->>Redis: Publish matching payload
    Redis-->>Client: Record or metadata event
    Client->>Resolver: Heartbeat
    Resolver->>Stream: Refresh TTL

References: packages/…/event-stream.resolver.ts• L43-285 packages/…/event-stream.service.ts• L65-260 packages/…/object-record-event-publisher.ts• L86-313 packages/…/metadata-event-publisher.ts• L75-115 packages/…/metadata-event-publisher.ts• L117-211 packages/…/metadata-event-publisher.ts• L214-328 packages/…/event-logs-live.resolver.ts• L109-168

Queue job subscriptions

Some queued jobs expose progress to one workspace user. The queue event listener converts selected job state into a workspace event and publishes it in order. Jobs without a client recipient remain internal worker state.

References: packages/…/queue-job-event.listener.ts• L16-66 packages/…/bullmq.driver.ts• L473-517

Cleanup processing

Cleanup uses scheduled jobs and commands to turn workspace retention settings into bounded maintenance work. A cross-workspace scheduler selects eligible workspaces and enqueues workspace-scoped work. This keeps one large workspace or one stale row from blocking the full system.

Workspace cleanup

Trash cleanup permanently removes soft-deleted records older than the workspace retention period. It works across known workspace objects in batches and stops at a per-workspace limit. Workspace cleanup also warns members about inactive suspended workspaces, soft-deletes them after the configured period, and queues final destruction after the grace period. It skips final destruction when billing still has an active subscription.

Onboarding cleanup handles unfinished or already deleted workspaces. The cleaner supports dry runs and bounds the number of workspace destructions per execution. Orphan cleanup finds workspace-related rows whose workspace identifier no longer points to a workspace and deletes them in dependency-aware batches.

References: packages/…/trash-cleanup.cron.job.ts• L21-91 packages/…/trash-cleanup.service.ts• L29-151 packages/…/cleaner.workspace-service.ts• L43-115 packages/…/cleaner.workspace-service.ts• L324-504 packages/…/list-and-delete-orphaned-workspace-entities.command.ts• L112-417

Wake-up recovery

Pending wake-up recovery is maintenance rather than data deletion. Every five minutes it finds waits that are more than five minutes overdue and queues them again in bounded batches. Queue deduplication makes repeated sweeps safe until a resolution job claims the wait.

References: packages/…/pending-wake-up-sweep.cron.job.ts• L12-61 packages/…/pending-wake-up.service.ts• L92-128

Retention cleanup

Event-log cleanup applies the effective retention period to each registered analytical event table. The effective period can depend on both workspace settings and the audit-log entitlement. Application metric events use a separate 90-day analytical-store expiry. Billing reminder state uses a stored boundary so recurring scans do not send the same reminder twice.

References: packages/…/event-log-cleanup.service.ts• L28-125 packages/…/event-log-retention.service.ts• L20-60 packages/…/billing-reminder.service.ts• L204-269

Change checklist

When these files or sections are changed, remember to consider these:

  • Keep the database event action and payload shape aligned between the ORM formatter, workspace emitter, listeners, event-log envelope mapping, pending wake-up matching, and logic or agent trigger payloads.
  • Preserve the relation diff contract. A relation field change must keep its logical field name and canonical join-column name aligned with metadata and record schemas.
  • Preserve workspace scoping whenever event data is read again for workflow enrichment, trigger filtering, subscription filtering, event-log queries, pending wake-up resolution, billing state, export, or cleanup.
  • When adding a downstream consumer, choose whether it runs in the request process or through a queue. For queued work, register the queue, its worker settings, its processor, and the worker module that loads it.
  • Keep queue scope keys aligned with fairness groups, pacing slot keys, and rate-limiter keys. A workspace or application job must not lose the scope that protects other work from starvation or bypasses its rate budget.
  • Keep queue identifiers, prefix checks, generated suffixes, bulk enqueue behavior, and deduplication options aligned so an active job is not lost or run twice. A re-added paced job must not reserve a second pacing slot.
  • Keep queue priority defaults, per-queue concurrency, global concurrency, rate limits, group pacing, stall policy, backlog metrics, memory admission, enabled and excluded queue filters, processor discovery, and shutdown drain behavior aligned when queue policy changes.
  • Keep job size accounting, workspace and application memory shares, critical-queue exclusions, charge release and restore, reconciliation grace periods, and refusal metrics aligned when Redis memory fairness changes.
  • Keep group limiter keys, burst and interval math, delayed-job handling, urgent priority behavior, Redis fallback metrics, and caller delay behavior aligned when queue pacing changes.
  • Keep pending wake-up owner registration, owner states, outcome types, resume-job options, claim behavior, and handler delivery aligned when a new owner is added.
  • If a ClickHouse event table or row shape changes, update the registry, envelope mapping, migration sequence, sink write shape, query normalization, and retention cleanup together.
  • Keep ClickHouse client availability behavior aligned between event ingestion, query access, migration setup, and cleanup.
  • Keep application metric rollup keys, registration filtering, duration normalization, metric table columns, and the table TTL aligned.
  • If event-log pagination changes, keep the timestamp cursor and same-timestamp page completion aligned.
  • Keep stream TTL refresh and active-stream cleanup in step with stream lifecycle and heartbeat behavior.
  • Keep metadata event enrichment, morph-relation duplicate filtering, recipient scoping, required permission flags, and workspace cache state aligned.
  • If billing state changes, keep the external subscription event, local rows, entitlement cache, credit ledger, usage admission, and workspace activation transition aligned.
  • Keep suspended-workspace warning, soft-delete, destruction grace periods, billing checks, destruction queue identifiers, and dry-run behavior aligned.
  • Keep versioned instance and workspace upgrade commands registered in their lists or provider modules. Keep repair commands idempotent and keep dry-run and per-workspace batch behavior intact.
  • Keep schema upgrade additions, optimistic metadata maps, cache updates, and database DDL aligned. Do not publish mutated cache state before a migration succeeds.
  • Keep migration checkpoints, route state, cutover guards, rollback validation, workspace locks, and active-stream checks aligned when storage moves between schemas.
  • If database setup changes, keep shared schema creation, required extensions, optional foreign-data wrappers, and schema truncation boundaries aligned.
  • If workspace export changes, keep metadata-derived DDL, generated columns, JSON and copy formatting, row filters, and workspace isolation aligned.
  • When a normalized metadata table replaces stored JSON, keep the backfill, cache flush, export reconstruction, and later metadata reads aligned.
  • When a cron or upgrade command changes, keep its owning module, registration list, command name, enablement gate, and compatibility path aligned.

Important failure modes

TriggerConsequenceGuard
A database event lacks its workspace or event nameWorkflow dispatch or trigger matching cannot select the right listenersListener validation and workspace-scoped event formatting reject incomplete batches
Event emission is suppressed for a write that downstream features needWebhooks, workflow triggers, timeline activity, and pending wake-ups do not runSuppression is limited to controlled bulk system writes
A relation diff uses a stale join-column nameField watchers can miss a relation update or react to the wrong fieldRelation metadata supplies the canonical join-column name
A trigger receives records outside its role or row accessAn agent or application can act on data it cannot readTrigger jobs resolve admitted records and mask restricted fields before enqueue
A queue is added without worker settings or a loaded processorJobs fail to enqueue or remain unprocessedQueue registration, explicit worker configuration, processor discovery, and worker filters cover the queue
A cron command is omitted, fails during registration, or is disabled by the wrong gateA recurring cleanup, recovery, sync, or billing task does not runThe shared registration list names each command, applies explicit config gates, and reports successes, failures, and skips
Fairness or pacing keys omit workspace or application scopeOne noisy scope can delay unrelated work or consume another scope's rate budgetQueue job scope builds system, workspace, application, and group keys before scheduling
Redis pacing failsJobs run without their group rate delayThe pacer records a fallback and the queue still accepts the work
An invalid group limiter is acceptedPacing can produce bad delays or fail to protect a groupLimiter validation requires a positive integer count and a finite positive duration
A queued job exceeds the configured size or memory shareThe job is refused before it can overload RedisSize checks and Redis memory admission reject the job and record the refusal
A job memory charge remains after its job is goneLater jobs see less capacity than is availableReconciliation uses two passes with a grace period before releasing a charge
A removal race loses a charge restoreA live delayed job can run without its memory chargeThe restore script checks the BullMQ job and restores only unfinished jobs
A pending wake-up is matched without an admitted or readable recordA wait can resume from a record the owner cannot accessMatching checks workspace and admission; resolution reads with owner permissions
A wake-up owner is not ready when its job runsThe outcome can arrive before the owner records its waitNOT_READY keeps the wait and schedules a bounded retry
A delayed wake-up job is lostA durable wait stays overdueThe five-minute sweep requeues overdue waits with deduplication
ClickHouse is missing or unavailableEvent-log writes and retention cleanup cannot use the analytical storeSink selection omits an unavailable client and access checks reject unavailable event-log views
A ClickHouse migration is applied twice or out of orderEvent-log tables can fail to start or hold an inconsistent shapeThe migration table records applied files and the runner skips known files
A monitoring or metric row shape changes without its registry and migrationMonitoring or usage data is lost or cannot be queriedEvent envelopes, table migrations, registry entries, sink writes, and retention rules change together
A recurring analytical-store error continuesAn event-log batch can retry forever or block later workThe event-log path uses one explicit transient retry and then drops the batch with a warning
A workspace upgrade publishes optimistic metadata before DDL succeedsLater reads use metadata for tables or fields that do not existUpgrade services refresh cache only after migration success
A versioned upgrade command is not registeredExisting workspaces or instances do not receive the required repairVersion modules and command registries load the command before deployment runs upgrades
A workspace migration resumes with an invalid route, stale checkpoint, or active streamData can be copied to the wrong store or cut over with incomplete stateMigration state is validated, batches are resumable, and cutover and rollback checks fail closed
A workspace upgrade updates stored data but not its derived cache mapsWorkflow runs, triggers, permissions, or UI metadata can read an old stateUpgrade commands invalidate and recompute the maps affected by raw writes
A migration assumes an object, field, or page exists in every workspaceA partial workspace upgrade stops instead of allowing later workspaces to proceedUpgrade commands detect missing entities and skip safely; dry runs expose incomplete prerequisites
An export writes records without metadata-derived schema or workspace filtersThe export cannot be restored reliably or can include another workspace's dataExport builds DDL from the workspace metadata and applies workspace-specific row filters
Stored field options are backfilled without cache invalidationQueries and exports can use old option maps after the rows changeThe backfill flushes field metadata and option maps for affected workspaces
A suspended workspace has an active subscriptionDestruction can remove data that billing still considers activeThe cleaner checks active and trialing subscriptions before destruction
A stream expires or its cached state is goneThe broadcaster cannot deliver to that streamHeartbeats refresh the TTL and active-stream cleanup removes stale entries
Metadata enrichment loses a parent view or workflowA private or unlisted metadata event can be delivered to the wrong audience or to no valid audienceRecipient resolution uses current workspace maps and fails closed when a required parent is missing
An orphan cleanup deletes parent rows firstForeign-key cleanup fails or leaves partial metadataCleanup uses dependency-aware child-first batches and dry-run checks

Key files

FileWhy to read it
packages/…/global-workspace-orm.manager.tsShows how workspace-aware repositories, transactions, and contexts reach persistence
packages/…/workspace-repository.tsShows permission-aware record writes, event snapshots, and mutation event emission
packages/…/format-result.util.tsShows how stored values are shaped from metadata for nested relations, composite fields, and date values
packages/…/format-twenty-orm-event-to-database-batch-event.util.tsDefines the record states carried by each database event action
packages/…/entity-events-to-db.listener.tsShows the main fan-out from record events to live, webhook, workflow, audit, timeline, and trigger work
packages/…/workspace-iterator.service.tsShows bounded, isolated execution across workspaces with dry-run and shard support
packages/…/workspace-command-provider.module.tsShows how versioned workspace upgrades are made available to the runner
packages/…/2-59-upgrade-version-command.module.tsShows the current workspace upgrade group and its per-workspace repair commands
packages/…/instance-commands.constant.tsShows how instance upgrades are registered and kept in deployment order
packages/…/cron-register-all.command.tsShows the shared registration boundary for recurring cleanup, recovery, sync, and billing jobs
packages/…/database-command.module.tsShows which database, export, migration, cleanup, and queue-maintenance command groups are loaded together
packages/…/agent-history-migration-state.service.tsShows how storage routes and migration cursors prevent unsafe history reads
packages/…/workspace-export.service.tsShows how workspace metadata, normalized options, and rows become a portable SQL export
packages/…/clickhouse.service.tsShows analytical-store availability, client selection, bounded inserts, and query behavior
packages/…/run-migrations.tsShows ordered and repeat-safe analytical-store schema setup
packages/…/message-queue.service.tsShows the common interface for enqueue, bulk work, cron work, inspection, and workers
packages/…/message-queue-core.module.tsShows driver selection and the per-queue service boundary
packages/…/bullmq.driver.tsExplains asynchronous execution, retries, deduplication, progress, fairness, pacing, memory limits, metrics, and shutdown drain
packages/…/fair-queue-scheduler.tsShows how queue priority and group pacing are calculated before jobs enter Redis
packages/…/group-rate-pacer.tsShows workspace and group rate slots and the safe fallback when Redis pacing fails
packages/…/job-memory-limiter.tsShows workspace and application memory admission and charge reconciliation
packages/…/job-memory-restore-script.constant.tsShows how a failed removal restores a charge only for an unfinished job
packages/…/job-options.interface.tsDefines queue scope, retry, deduplication, recipient, group, and limiter contracts
packages/…/pending-wake-up.service.tsShows how waits are armed, replaced, queued, found, and claimed
packages/…/create-event-log-from-internal-event.tsShows record-action mapping, sink enablement, and bounded transient retry
packages/…/event-logs.service.tsDefines access checks, retention filters, normalization, and timestamp-safe pagination
packages/…/event-stream.resolver.tsShows stream authorization, query registration, heartbeat, and expiry
packages/…/object-record-event-publisher.tsShows permission-aware record event publication
packages/…/metadata-event-publisher.tsShows metadata enrichment and recipient-aware broadcast behavior
packages/…/trash-cleanup.service.tsShows bounded deletion of expired soft-deleted records
packages/…/cleaner.workspace-service.tsShows suspended-workspace cleanup and destruction guards