Gateway session lifecycle and recovery
The gateway keeps a conversation usable across client reconnects, process restarts, and short-lived resource cleanup. It separates durable conversation identity from a live client attachment, protects each active turn with a cross-process lease, and uses persisted history plus bounded recovery steps to avoid duplicate work or lost replies. See Gateway service, inbound turn routing, and platform delivery for the surrounding service flows.
Session identity
The session model has two identities with different lifetimes. A runtime ID identifies one live record in the gateway process. A session key identifies the stored conversation that can be opened again after that record or process is gone.
Identity model
session_id is the live ID used by RPC calls and event routing. stored_session_id and session_key refer to the durable conversation key. A new draft can have a runtime ID before it has a database row. The first real prompt normally creates that row, while a seeded session is stored at creation time so its history survives a restart.
The durable key can outlive a runtime ID. Resume creates or reuses a live record for the stored key. Compression can move the active conversation to a continuation record, so lifecycle code uses the agent's current durable ID when it closes or persists a session.
References: tui_gateway/…/sessions.py• L73-99 tui_gateway/methods_session.py• L337-380 tui_gateway/methods_session.py• L452-477 tui_gateway/session_lifecycle.py• L396-425
Live session states
The gateway reports a small set of states to clients. These states describe the live record rather than the full history of the stored conversation.
| State | Meaning |
|---|---|
idle | The session can accept a new turn. |
starting | The agent or session services are being built. |
waiting | The agent is waiting for user input or a server request. |
working | A turn is running. |
streaming | A live child or isolated run is producing output. |
resuming | Stored history is being loaded for a reconnect or resume. |
References: tui_gateway/…/sessions.py• L64-99 tui_gateway/server.py• L2313-2404
Lifecycle transitions
The gateway moves a conversation between a client-facing live record and a durable stored record. The transition keeps drafts cheap, makes active turns exclusive, and leaves stored history available when the live record is reclaimed.
Creation and resume
When a client creates a session, the gateway mints a runtime ID and a stored key. It returns quickly so the client can render the session while the agent builds in the background. An empty draft has no durable row yet. A seeded session or a remote workspace session is persisted earlier because the client expects it to be resumable immediately.
When a client resumes, the gateway first reuses a matching live record in the same profile. Otherwise it reads the stored history and creates a new live record. A live reattach binds the new transport without taking control from other viewers. A failed or stale reattach is rejected rather than silently attaching to the wrong generation.
This sequence shows the normal relationship between a client attachment, a live record, and durable history.
flowchart TD
A[Client creates or resumes] --> B{Live record exists?}
B -- Yes --> C[Rebind transport]
B -- No --> D[Read stored session]
D --> E[Build live record]
E --> F{First real turn?}
C --> F
F -- No --> G[Keep draft or resumed view]
F -- Yes --> H[Acquire active-session lease]
H --> I[Run turn and persist history]
I --> J[Idle live record]
J --> K[Reattach, close, or reclaim]
K --> D
References: tui_gateway/methods_session.py• L638-669 tui_gateway/methods_session.py• L721-745 tui_gateway/methods_session.py• L876-899 tui_gateway/…/sessions.py• L118-152 tui_gateway/…/sessions.py• L182-215
Turn admission
The gateway admits a turn only after it has a live session record and an active-session lease. Create and resume do not reserve a lease because a painted draft or a reconnecting viewer must not consume a writer slot. The first real turn claims the slot before work starts.
The lease makes one conversation single-writer across processes. Its identity combines the process identity with the live runtime ID. If the registry cannot prove ownership, the gateway refuses the turn. This fail-closed rule prevents two backends from writing the same conversation at the same time.
References: tui_gateway/session_lifecycle.py• L69-85 tui_gateway/session_lifecycle.py• L121-137 hermes_cli/active_sessions.py• L101-117 hermes_cli/active_sessions.py• L137-148 hermes_cli/active_sessions.py• L516-574
Durable session state
The state database is the source for stored session rows and transcript history. In-memory records hold the current agent, transport, turn state, and unsaved message view. Persistence joins these two forms without making a short client disconnect close the conversation.
Transcript persistence
The agent writes the current message snapshot through its persistence boundary. The gateway also performs incremental flushes for idle sessions. These writes use marker-based deduplication, so a repeated flush does not create duplicate transcript rows.
A session close first snapshots history and persists the agent message state. It then runs end hooks, commits memory state, and closes the agent. Automatic Desktop cleanup differs from an explicit close: it releases the live runtime but keeps the durable row open so the conversation can be resumed.
References: tui_gateway/session_reaper.py• L23-81 tui_gateway/session_lifecycle.py• L353-425 tui_gateway/…/sessions.py• L388-398
Compression lineage
Compression may replace the active history segment with a continuation. The session key remains the conversation anchor, while the agent's current session ID identifies the active durable segment. Close and resume paths therefore follow the current tip instead of ending or reopening an older parent.
References: tui_gateway/session_lifecycle.py• L396-425 tui_gateway/methods_session.py• L831-847 tui_gateway/…/sessions.py• L227-249
Lease lifecycle
An active-session lease protects the write slot shared by the TUI, Desktop, dashboard, and other gateway surfaces. The lease is a coordination record, not the transcript itself, so it can be released while the stored conversation remains resumable.
Lease ownership
The gateway attaches a lease to a live session only when a turn needs admission. A normal close releases it. A lease transfer keeps the slot while compression or a detached-runtime takeover changes the durable key or live owner. A borrowed lease in an isolated child is inert: the child can use the parent's authority but cannot release or transfer it.
If a client reconnects in a new runtime while an old runtime in the same process has no client, the new runtime can take over that detached lease. The old turn is asked to stop. A live foreign process or a still-attached sibling remains protected.
References: tui_gateway/session_lifecycle.py• L88-119 tui_gateway/session_lifecycle.py• L145-186 tui_gateway/session_lifecycle.py• L280-307 tui_gateway/compute_host_bridge.py• L80-94
Lease release and orphan cleanup
The reaper compares the leases owned by live session records with the registry. It releases entries that this process no longer backs. A lease held for an isolated child turn is kept after the parent closes until the child reports completion. If that report never arrives, the reaper releases the deferred lease after its safety limit instead of leaving a permanent capacity block.
References: tui_gateway/session_lifecycle.py• L234-277 tui_gateway/session_lifecycle.py• L587-648 tui_gateway/session_reaper.py• L307-320 hermes_cli/active_sessions.py• L640-708
Disconnect and reconnect behavior
WebSocket loss does not immediately destroy a normal session. The gateway parks a session on a detached transport so a short network drop can recover the same live record and continue receiving buffered events.
Transport parking
When a WebSocket closes, the gateway first removes that transport from each session. A session with another live viewer stays attached. A clientless session with close_on_disconnect is torn down at once. Other clientless sessions use the detached sentinel and enter the orphan grace period.
Resume and transport rebind share a lock with the reap claim. A successful rebind cancels the pending reap. A late request from a dead socket is not treated as a reconnect, so it cannot cancel cleanup for a newer disconnect.
References: tui_gateway/ws.py• L429-467 tui_gateway/session_lifecycle.py• L994-1051 tui_gateway/session_lifecycle.py• L818-847 tui_gateway/methods_session.py• L876-899
Stall recovery
The orphan reaper gives a detached session time to reconnect. A detached running turn is allowed to continue while its activity clock is fresh. Activity includes model waits, stream output, and tool heartbeats. If the clock becomes stale, the gateway interrupts the turn and polls for settlement.
Pending delegations delay orphan cleanup because they still own work for the session. A turn that does not settle after the poll budget is force-reaped. This breaks a deadlock caused by a stuck thread while keeping the normal finalization path for turns that respond to the interrupt.
References: tui_gateway/server.py• L130-162 tui_gateway/session_lifecycle.py• L850-875 tui_gateway/session_lifecycle.py• L915-979 tui_gateway/session_lifecycle.py• L771-800
Event and request replay
A reconnecting client rebuilds state from the live session snapshot and replays events after its sequence watermark. The snapshot can include an in-flight turn, queued prompt, pending approval, open server request, and pending connection operation. If the replay buffer is truncated or the gateway epoch changed, the client must refetch the session state instead of assuming that missing events did not occur.
References: tui_gateway/…/sessions.py• L20-99 tui_gateway/…/sessions.py• L720-734 tui_gateway/ws.py• L362-369
Restart recovery
Restart recovery uses durable evidence of an unfinished turn. It does not infer failure from recent activity alone. This distinction prevents a second gateway from replaying a turn that another live backend still owns.
Interrupted-turn markers
Before a turn starts, the gateway records a small marker under the session's profile home. The marker contains the prompt, start time, retry count, and writer identity. A normal conclusion clears it before the terminal client frame. A marker left behind means that the process may have stopped before the client received a terminal result.
On resume, the gateway checks the marker writer's liveness. A live sibling writer keeps the marker and blocks auto-continue. A dead or unknown writer does not grant unlimited retries: the marker must be fresh, recovery must be enabled, and the attempt count must remain below its limit. Otherwise the marker is cleared and the user can continue manually.
References: tui_gateway/turn_marker.py• L1-8 tui_gateway/turn_marker.py• L39-74 tui_gateway/turn_marker.py• L115-154 tui_gateway/session_auto_continue.py• L58-88
Auto-continue
For a fresh eligible marker, resume schedules one continuation after the agent is rebuilt. It reacquires the active-session lease before it starts. The continuation uses a recovery note that tells the agent to inspect work already done before it repeats the request.
The retry count is part of the durable recovery contract. A failed build or ownership refusal leaves the marker for a later resume. A stale marker, a disabled setting, or a retry limit clears it without starting a duplicate turn.
References: tui_gateway/session_auto_continue.py• L80-139 tui_gateway/session_auto_continue.py• L101-120 tui_gateway/session_lifecycle.py• L121-137
Startup orphan sweep
An in-process orphan timer disappears when the gateway process dies, so a later gateway performs a conservative startup sweep. The sweep closes only old eligible rows that are not held in memory and are not plausibly owned by another live backend. Backend heartbeat rows distinguish a live but idle backend from a dead process.
The sweep starts after the WebSocket grace period. This gives a client that reconnects after a restart time to resume its row before cleanup considers it orphaned.
References: tui_gateway/session_reaper.py• L375-422 tui_gateway/session_reaper.py• L425-499 tui_gateway/session_reaper.py• L502-517 tui_gateway/ws.py• L377-387
Shutdown flushing
Shutdown protects the durable transcript before the process exits. It combines periodic writes with a bounded exit flush, then interrupts active work so the final persistence path can record partial results.
Flush sequence
The normal reaper flushes idle sessions during its scan. A SIGTERM or SIGINT handler then runs a bounded flush for all in-memory sessions, including running sessions, before handing control to the server or supervisor shutdown handler. The default exit flush budget is five seconds. A slow or incomplete flush is logged as possible transcript loss.
The shutdown path also interrupts active turns, waits briefly for them, and kills live foreground processes that did not stop. This gives tool results a chance to reach the final session flush without allowing one stuck operation to block process exit.
References: tui_gateway/session_reaper.py• L18-21 tui_gateway/session_reaper.py• L64-113 tui_gateway/session_reaper.py• L116-175
Finalization boundary
All close paths converge on finalization. Finalization stops notifications, persists the agent snapshot, commits memory state, ends owned resources, cancels background review work, and closes the agent. The gateway keeps gateway-owned durable rows from being ended by a viewer-only backend. It also preserves a row when another runtime has taken over its lease.
After finalization, the live record is removed from the process registry. Its stored history remains the recovery source for a later resume unless the session was explicitly ended or deleted by an owning lifecycle.
References: tui_gateway/session_lifecycle.py• L353-463 tui_gateway/session_lifecycle.py• L515-535 tui_gateway/session_lifecycle.py• L551-584
Change checklist
When these files or sections are changed, remember to consider these:
- Keep
session_id,session_key, andstored_session_idaligned across live snapshots, resume replies, event routing, and lease records. - Preserve the first-turn lazy-row rule. Changes that persist drafts early must also preserve idempotent create retries and seeded-session resume.
- Update every transport rebind path when changing orphan-reap rules. A missed path can leave a timer armed against a live client or leave a detached session without cleanup.
- Keep lease release and isolated-turn settlement coupled. Releasing too early allows a second writer while the child can still write; releasing too late creates a zombie capacity slot.
- Keep turn-marker writer identity checks with auto-continue. A marker from a live sibling backend must never start a second turn.
- Keep profile selection bound to the session for off-turn persistence, finalization, and startup cleanup. A launch-profile lookup can write valid data to the wrong profile.
Important failure modes
| Trigger | Consequence | Guard |
|---|---|---|
| Active-session registry cannot be read | The turn is refused rather than risking two writers. | Fail-closed lease admission. |
| WebSocket disconnect during a live turn | The turn may be interrupted after its activity becomes stale. | Grace period, activity check, interrupt polling, then bounded force-reap. |
| Reconnect races with orphan cleanup | A live client could be detached or a new runtime could reuse stale state. | Shared resume lock, session lock, and transport revalidation. |
| Process dies after turn start | The prompt could be lost or run twice after restart. | Durable marker with writer liveness, freshness, and attempt limits. |
| Isolated child completion is lost | The active lease could remain forever. | Deferred lease tracking and stale-lease TTL cleanup. |
| Exit flush exceeds its budget | Recent in-memory transcript data may not reach the state database. | Bounded flush warning and periodic idle flush. |
Key files
| File | Why to read it |
|---|---|
| tui_gateway/methods_session.py | Defines create, resume, live reuse, and the durable-to-live session boundary. |
| tui_gateway/…/sessions.py | Defines the client-visible session states, snapshots, resume contract, and event replay contract. |
| tui_gateway/session_lifecycle.py | Owns leases, finalization, transport rebind, orphan recovery, and close races. |
| tui_gateway/session_reaper.py | Owns incremental and exit flushes, idle cleanup, backend heartbeats, and startup orphan sweeps. |
| tui_gateway/session_auto_continue.py | Defines restart continuation and queued recovery behavior. |
| tui_gateway/turn_marker.py | Defines durable interrupted-turn evidence and writer liveness checks. |
| tui_gateway/ws.py | Shows WebSocket disconnect handling and the handoff to session teardown. |
| hermes_cli/active_sessions.py | Defines cross-process lease identity, exclusivity, transfer, and orphan release. |