Introducing a new way to review your PRs.

Read the announcement

nousresearch / hermes-agent

The agent that grows with you

View on GitHub

38 documents · Updated Nov 20, 2026

docs / user-interfaces

User interfaces and local API

Hermes presents one agent and session system through browser, desktop, and terminal experiences. A shared local API carries agent work, session state, configuration, onboarding, and integration actions to each client while each surface controls its own input and display model.

User experience model

The interfaces serve different needs but act on the same profile and session state. The browser manages the system and can host chat, the desktop provides a native chat surface, and the terminal gives direct access to live agent work.

ExperienceMain roleConnection to the local system
Browser dashboardManage profiles, settings, integrations, and sessionsHTTP management routes plus chat or gateway WebSocket paths
Browser-hosted chatUse chat from a browser connected to one serving gatewayServing-origin HTTP and ticketed WebSocket paths
Desktop applicationProvide a native chat and control surfaceElectron connection routing plus the shared gateway contract
Terminal user interfaceRun live conversations and review streamed workJSON-RPC over standard input and output

The browser chat path can run the terminal conversation experience behind a browser WebSocket. The browser-hosted desktop path instead uses the serving gateway as its one connection. The native desktop has its own chat surface and does not embed the TUI.

References: web/README.md• L1-4 hermes_cli/…/chat_ws.py• L1-6 apps/…/index.tsx• L94-126 apps/…/browser-connection.ts• L16-55

Browser experience

The browser experience is a web control surface for local agent use. It combines profile and system management with chat access while keeping authentication and gateway state on the serving web origin.

Dashboard management

The dashboard manages installation status, profiles, configuration, credentials, integrations, session history, billing, and usage. It also offers chat entry points that use the same agent and session services as the other clients.

The selected profile remains part of the browser context. This lets management views and chat address the same profile after navigation or refresh. Profile configuration, secrets, and sessions remain separate so one dashboard can manage more than one profile.

References: web/…/api.ts• L63-145 hermes_cli/…/config_env.py• L80-140 hermes_cli/…/sessions.py• L175-180 apps/…/gateway-contract.openrpc.json• L154-169

Browser chat

Browser chat can present the terminal conversation experience through a pseudo-terminal WebSocket. A selected or resumed session supplies the conversation state while the server forwards terminal input and output.

The browser can also use a structured gateway WebSocket for events and session updates. The server checks chat settings, credentials, origin rules, and client access before it accepts the connection.

References: hermes_cli/…/chat_ws.py• L90-100 hermes_cli/…/chat_ws.py• L126-183 apps/…/gateway-contract.openrpc.json• L51908-51918

Browser-hosted desktop

The browser-hosted desktop treats the serving origin as its only connection. It can view the profiles served by that gateway and uses a fresh ticket for each gateway or audio socket.

An installed profile is not automatically reachable from every gateway. A browser tab waits when its selected profile is not in the gateway's served-profile set. Web access also reports configured state separately from the state currently being served because changing access can restart the gateway.

References: apps/…/browser-connection.ts• L85-223 apps/…/browser-served-profiles.ts• L27-98 apps/…/webapp-access.ts• L3-69

For browser routes, profile behavior, authentication, chat transport, and web-app access, see Web dashboard.

Desktop application

The desktop application is a native chat and control surface for the same agent and session system. Electron owns machine and connection access, the renderer owns navigation and presentation, and the gateway owns sessions, tools, model calls, and streamed work.

Native application boundary

The renderer reaches machine and gateway services through a narrow preload bridge. The bridge provides connection lookup, profile and connection scope, gateway access, HTTP requests, session windows, and native controls without giving the renderer general Node access. Browser-hosted mode supplies safe same-origin equivalents for this bridge.

The desktop can use local, remote, cloud, or SSH-backed sources through the same user experience. A connection identifies an agent source while a profile identifies the agent context served by that source. Requests that address a session keep both identities when the session owner is not the active source.

References: apps/…/preload.ts• L28-70 apps/…/client.ts• L645-687 apps/…/client.ts• L818-899

Capability management

The desktop uses profile-scoped API actions to manage the capabilities available to an agent. These actions cover skills, learning data, model choices, memory providers, audio, maintenance, and other profile settings. Long-running actions return status that the desktop can follow instead of treating request acceptance as completion.

Onboarding uses the same profile and connection scope as the chat that will use the new credentials. It supports provider sign-in, device-code or external sign-in, API-key setup, and model selection. The selected model is tied to the provider that serves it, so a model from another configured provider is not paired with the wrong sign-in.

References: apps/…/system.ts• L60-177 apps/…/flow.tsx• L238-362 apps/…/messaging.ts• L23-59

Session browsing

The desktop can read sessions for one profile or aggregate sessions across profiles. The sidebar groups recent, scheduled, and messaging sessions, and newer backends can return those groups in one request. Older backends remain usable through separate profile-scoped reads when the combined route is unavailable.

Session reads and writes carry the profile that the user is viewing. A session row also retains its owning connection when more than one source exposes similar profile names. This keeps transcript reads, session edits, and later resumes on the backend that owns the session.

References: apps/…/sessions.ts• L60-75 apps/…/sessions.ts• L136-177 apps/…/sessions.ts• L179-203 apps/…/sessions.ts• L324-386 apps/…/sessions-list-profile-scope.test.ts• L18-64

Native chat surface

The desktop chat keeps the conversation, composer, session tiles, transcript, attachments, tools, voice input, and profile context in one native surface. It sends user turns through the gateway and renders streamed events as messages, tool activity, status, and input requests.

The chat can show several session surfaces at once. Each surface keeps its own session owner scope and transcript view while background work continues without taking focus from the current conversation. The gateway remains responsible for agent work; the renderer adapts that work to the native interaction model.

References: apps/…/index.tsx• L94-126 apps/…/index.tsx• L250-469 apps/…/index.tsx• L518-808

Integration management

The desktop exposes profile-scoped messaging and webhook controls beside the chat. Users can test messaging platforms, link relay accounts, complete Telegram pairing, approve or revoke users, and manage webhook subscriptions.

Webhook state belongs to the selected backend profile. The UI reloads state after changes, shows the new secret only after creation, and reports when enabling the receiver needs a gateway restart. A profile change replaces the old view so pending form state cannot cross into another profile.

References: apps/…/messaging.ts• L62-241 apps/…/index.tsx• L86-240 apps/…/index.tsx• L262-373

For the native application lifecycle, onboarding, chat behavior, and desktop-specific capabilities, see Desktop application.

Terminal user interface

The TUI is the interactive terminal front end for live agent work. It shows the transcript, streamed output, tool activity, approvals, clarification prompts, session controls, and command results while the gateway owns agent and session decisions.

The terminal and gateway exchange newline-delimited JSON-RPC messages over standard input and output. The terminal owns screen state and user input. The gateway owns model calls, tools, session persistence, and the rules that turn user actions into agent work.

References: tui_gateway/entry.py• L273-361 tui_gateway/rpc_dispatch.py• L45-90

The same TUI can run in a terminal or as the child process behind browser chat. This shared path keeps prompt behavior, commands, tool progress, and user questions consistent across both experiences.

References: tui_gateway/entry.py• L57-64 hermes_cli/…/chat_ws.py• L176-183 website/…/tui.md• L7-11

For the terminal wire format, request types, event flow, reconnect behavior, and gateway method families, see TUI gateway RPC.

Local RPC bridge

The local RPC bridge is the shared API for interactive clients. It turns typed client requests into agent, session, configuration, onboarding, integration, billing, display, and scheduled-work operations, then returns results, progress events, and requests for user decisions. A local canonical gateway is the authority for shared agent state; a client remains responsible for local presentation and for answering requests that need a user or a local device.

The contract exposes a broad gateway method catalog and a separate canonical method catalog for operations served by the session authority. Clients use capability discovery to decide which authority methods they can rely on instead of treating every gateway method as interchangeable.

The bridge supports two transport forms without changing the operation model:

  • Stdio: A local TUI starts the gateway as its peer and exchanges one JSON-RPC frame per line.
  • WebSocket: Desktop, browser-hosted, and other clients connect to a running gateway. Each connection uses the credentials and profile scope for its source.

The desktop normally reaches the bridge through a gateway endpoint resolved by Electron. Browser-hosted clients use the serving origin. Session, connection, and profile scope keep requests on the intended source. The desktop also retains narrow compatibility paths for older gateways and rejects unsupported authority semantics instead of silently changing the user's action.

References: apps/…/sessions.ts• L60-69 apps/…/messaging.ts• L21-27 apps/…/system.ts• L59-77 apps/…/gateway-contract.generated.ts• L7170-7252 apps/…/gateway-contract.openrpc.json• L50418-50436

Operation groups

The RPC catalog groups user actions by the resource they affect. These groups form one contract surface, so a session or configuration change can affect later agent turns in the same profile.

GroupUser capabilityRepresentative operations
Agent workSubmit prompts, stream replies, run commands, and control running workprompt.submit, prompt.cancel, prompt.receipt, shell.exec
SessionsCreate, attach, resume, inspect, branch, archive, close, export, and remove conversationssession.create, session.resume, session.list, session.info, session.mutate, session.transcript
ConfigurationRead and change effective settings and display-safe summariesconfig.get, config.set, config.show
Models and toolsSelect model access and manage tools, MCP servers, skills, plugins, and delegation routesmodel.options, config.set, mcp.servers.*, commands.catalog
IntegrationsView, connect, and manage external connector accounts, messaging links, pairing, and webhooksconnectors.catalog, connectors.list, connectors.connect, billing.state
Scheduled workCreate, inspect, run, pause, resume, and stop scheduled agent workcron.manage, cron.submit, cron.cancel
Profiles and onboardingStart setup, record its state, and manage profile identity and assetssetup.status, profiles.list, onboarding.start, onboarding.state
Local device featuresControl desktop display, files, images, clipboard, and machine-facing actionsdisplay.observe, file.attach, image.attach, clipboard.paste, machine.facts

The canonical catalog adds authority-owned prompt admission, session lifecycle, replay, transcript, runtime description, and revision-fenced session mutation operations. These methods give clients stable controls for queue state, owner identity, replay, and session edits while the general catalog continues to expose the wider gateway surface.

References: apps/…/gateway-contract.generated.ts• L6710-6759 apps/…/gateway-contract.generated.ts• L7170-7252 apps/…/gateway-contract.openrpc.json• L50418-50436 apps/…/gateway-contract.openrpc.json• L50888-51318

Wire contracts

The bridge defines client calls with results, backend events, and server requests that ask the user or client to make a decision. The OpenRPC catalog and generated shared types describe the general gateway catalog, while the canonical method map describes operations served by the session authority.

The first gateway.ready event identifies the resolved client surface, change-event support, and replay epoch. The client then advertises the server requests it handles through client.capabilities. A canonical runtime description supplies the authority identity and the canonical capabilities a client may use. The contract is the boundary for version differences: unsupported methods or invalid message shapes become protocol errors instead of silent UI failures. The shared generated types come from the gateway contract source, so the terminal, browser, desktop client, and gateway use the same method names and data shapes.

References: apps/…/gateway-contract.openrpc.json• L334-349 apps/…/gateway-contract.openrpc.json• L51014-51029 apps/…/gateway-contract.openrpc.json• L51908-51918 apps/…/gateway-contract.generated.ts• L6412-6413 apps/…/gateway-contract.generated.ts• L7418-7420 apps/…/gateway-contract.generated.ts• L7226-7227

Events and user questions

Events carry streamed replies, tool progress, session changes, onboarding completion, connector changes, billing or display state, and gateway state to connected views. Server requests have a different role: they pause or guide agent work until a client answers an approval, clarification, secret, connector, setup, or desktop capability question.

Canonical approval and clarification events carry the session's execution identity. The client keeps each question tied to its owning session and can recover an unanswered question from authoritative state after reconnect. This prevents another window or an old session view from answering the wrong request.

References: apps/…/gateway-types.ts• L42-85 apps/…/gateway-contract.generated.ts• L6997-7028 apps/…/gateway-contract.generated.ts• L7084-7090 apps/…/json-rpc-channel.ts• L20-59 apps/…/client.ts• L123-182

Session continuity

Session continuity lets a client reconnect without turning a short transport loss into a new conversation. The client tracks a per-session event watermark and replay identity, requests events after its last known point, and resumes from authoritative session state when the event window is incomplete or the gateway restarts.

Canonical session methods make this authority explicit: clients can inspect runtime and session state, attach or resume a session on the current socket, request events since a watermark, read transcript pages, and use fenced session mutations. These paths keep replay, controls, and writes tied to the session owner rather than to a stale client view.

Session writes retain the request identity and the revision read before the write until the intent is acknowledged. A retry or profile switch must not silently replace that precondition. This keeps an uncertain response from applying the same user action to newer session state.

References: apps/…/gateway-contract.generated.ts• L5408-5456 apps/…/gateway-contract.generated.ts• L7213-7252 apps/…/gateway-contract.openrpc.json• L51104-51318 apps/…/json-rpc-gateway.ts• L549-737 apps/…/sessions.ts• L389-435 apps/…/client.ts• L424-500

Transport health

The shared channel gives each transport request IDs, deadlines, cancellation, typed errors, and server-request handling. WebSocket clients also use a JSON-RPC heartbeat and treat inbound streamed frames as proof that the connection is alive.

The channel is transport-neutral. A WebSocket client and a TUI stdio client use the same frame handling for responses, events, and server requests. Reconnect owners decide when to replace a transport; the channel rejects calls from the old generation so stale results cannot update a new connection.

References: apps/…/json-rpc-channel.ts• L107-156 apps/…/json-rpc-channel.ts• L200-277 apps/…/json-rpc-channel.ts• L464-555 apps/…/json-rpc-gateway.ts• L112-117 apps/…/json-rpc-gateway.ts• L222-371

Interaction flow

This diagram shows how browser, desktop, and terminal actions reach the same agent and session services through their supported connections.

flowchart LR
    TUI[Terminal TUI] -->|stdio JSON-RPC| RPC[Local RPC bridge]
    Browser[Browser dashboard] -->|HTTP management API| Web[Dashboard web server]
    Browser -->|PTY WebSocket| TUIChild[Embedded TUI]
    TUIChild -->|stdio JSON-RPC| RPC
    BrowserApp[Browser-hosted chat] -->|gateway WebSocket| RPC
    Desktop[Desktop application] -->|gateway WebSocket| RPC
    RPC --> Agent[Agent and tool runtime]
    RPC --> Sessions[Session authority]
    RPC --> Config[Profile configuration]
    RPC --> Integrations[Models and integrations]
    RPC -->|events and user questions| Clients[Connected user interfaces]

The shared bridge keeps agent actions in one service while each client chooses how to display them. Browser management can run without chat, browser-hosted chat stays bound to its serving origin, and the desktop can connect to a local or remote gateway without serving browser assets. Profile and connection scope add the routing needed when several sources expose similarly named profiles.

References: apps/…/sessions.ts• L324-386 apps/…/client.ts• L424-500 apps/…/index.tsx• L518-808 hermes_cli/web_server_dashboard.py• L103-140

Change checklist

When these files or sections are changed, remember to consider these:

  • Keep connection, profile, and session-owner scope aligned across browser connection creation, Electron route resolution, REST helpers, gateway requests, and event handling.
  • When changing a gateway method or event, update the source contract, OpenRPC document, generated shared types, and every client handler together.
  • When changing canonical authority methods, keep the canonical method list, capability discovery, adapter translation, and client routing behavior aligned with the general gateway catalog.
  • When changing profile-scoped REST routes, keep query scope, body scope, cache keys, and view lifetime aligned so a profile switch cannot reuse another profile's state.
  • When changing session listing, keep combined sidebar reads, older per-slice fallback, profile selectors, owner stamps, and retry or partial-failure state aligned.
  • When changing messaging, billing, or onboarding flows, keep every step on the same profile and connection owner, including pairing, relay links, webhook state, provider sign-in, model selection, and payment recovery.
  • When changing replay or prompt behavior, preserve event watermarks, replay identity, open-question recovery, execution identity, and request ownership across reconnects.
  • Keep the PTY byte stream used by browser chat separate from structured JSON-RPC traffic used by gateway clients.
  • Keep desktop-native capabilities tied to the owning session and client surface, not only to the backend host.
  • Keep scheduled-work controls and run-session rows on the same connection and profile scope as the job that owns them.
  • Keep generated contract types and gateway type projections aligned with the OpenRPC source when canonical session, control, capability, or readiness fields change.
  • When changing long-running audio, maintenance, webhook, skill, connector, display, or billing actions, keep accepted, running, completed, and failed states distinct.

Important failure modes

TriggerConsequenceGuard
A request omits its connection, profile, or session-owner scopeThe action can reach the wrong backend or fail as ambiguousCarry the full owner scope through routing and request helpers
A single-profile list selector is sent with the ambient profile of another profileThe gateway rejects the read or the sidebar shows the wrong sessionsPut the named profile in the request scope and verify the pair
A newer sidebar route is assumed to exist on an older backendSession lists fail even though the older routes still workTreat a missing route as a capability signal and use the scoped per-slice reads
A session read or write uses the active source instead of the session ownerA same-named profile can return a missing or unrelated transcriptPin session reads and mutations to the recorded owner
A profile switch leaves webhook, billing, or onboarding state mountedA later mutation applies to the previous profileKey profile-owned views and carry the captured profile through every call
Webhook enable saves configuration but the gateway does not restartThe UI reports a working receiver while the old gateway still servesReport restart state separately and offer a retry or explicit restart
Telegram pairing starts on one profile and applies on anotherThe backend cannot find the in-progress pairing stateCarry the same profile scope through start, status, apply, and cancel
A model from one provider is paired with another provider's sign-inChat starts with credentials that cannot serve the selected modelSave the selected model with the provider that owns it
A client calls a canonical method on a gateway that does not serve itThe request fails instead of silently producing an incomplete session actionUse runtime and capability discovery, then surface an explicit protocol error
A client sends a method or message shape from another contract versionThe request is rejected or the UI cannot interpret the resultUse the shared generated contract and surface an explicit protocol error
A client misses events during a disconnectThe transcript or pending question becomes staleReplay from the per-session watermark, then resume authoritative state when a gap exists
A live frame races a replay responseThe same event can render twice or a gap can be skippedHold live frames for the session until replay applies and sequence-gate dispatch
A session mutation retries with a new revision after an uncertain resultThe same user intent can apply twice or overwrite a newer changeRetain request identity and the original revision until acknowledgement
A canonical prompt reply arrives after reconnect or after its generation changesThe reply can answer a stale question or be lost as an unsafe actionTie prompts to their owner and execution identity, then recover from a fresh snapshot
Browser web-app access restarts a busy gatewayThe saved setting changes but the serving state does not changeReport restart refusal or error separately from the saved configuration
Browser chat expects the headless gateway to serve dashboard assetsThe gateway works but the browser application is unavailableKeep dashboard serving and headless API or WebSocket mode separate
A long-running action is treated as complete when it only returns an action handleThe UI shows a capability, integration, display, or billing change as ready before the backend finishesFollow action status and use its final result for displayed state

Key files

FileWhy to read it
web/README.mdDefines the browser dashboard's user-facing role and runtime split.
hermes_cli/web_server.pyShows how HTTP routers and WebSocket routes join the web service.
hermes_cli/…/chat_ws.pyShows browser chat and sidecar connection gates.
apps/…/preload.tsShows the narrow renderer-to-main capability bridge.
apps/…/client.tsShows desktop request routing, readiness, owner scope, and REST access.
apps/…/sessions.tsShows profile-aware session listing, fallback reads, owner routing, and fenced writes.
apps/…/messaging.tsShows profile-scoped messaging, pairing, relay, and webhook access.
apps/…/system.tsShows profile-scoped system, audio, memory, and maintenance actions.
apps/…/index.tsxShows the desktop's independent chat surface and session-scoped rendering.
apps/…/index.tsxShows profile-owned webhook management and restart state.
apps/…/flow.tsxShows provider sign-in and profile-scoped model selection.
apps/…/index.tsShows the plugin-facing state and gateway request door.
apps/…/gateway-contract.openrpc.jsonDefines the published RPC catalog, canonical authority methods, events, readiness data, and wire schemas.
apps/…/gateway-contract.generated.tsProvides generated RPC methods, capability payloads, readiness payloads, canonical methods, event names, and payload types.
apps/…/gateway-types.tsDefines shared session, prompt, and canonical event projections.
apps/…/json-rpc-channel.tsShows transport-neutral request, response, heartbeat, and server-request handling.
apps/…/json-rpc-gateway.tsShows WebSocket lifecycle, event fan-out, and reconnect replay.
tui_gateway/rpc_dispatch.pyShows request admission and dispatch for the local RPC bridge.