Scheduled automation
Scheduled automation lets Hermes define recurring agent work, decide when each occurrence may run, and keep that work safe across failures and restarts. It combines a durable job definition with a profile-owned scheduler, run limits, provider-quota handling, and health records. Scheduled job execution describes the work after the scheduler accepts an occurrence.
Automation model
The cron subsystem turns a user-authored task into a controlled recurring process. A job stores the task and its schedule. A scheduler claims due occurrences, applies policy, and hands accepted work to the execution and delivery flow.
This diagram shows the main path from a definition to a monitored result.
flowchart LR
A[Job definition] --> B[Schedule validation]
B --> C[Profile-owned scheduler]
C --> D{Due and allowed?}
D -- No --> E[Keep or defer occurrence]
D -- Yes --> F[Claim occurrence]
F --> G[Agent execution and delivery]
G --> H[Run history and health state]
E --> H
H --> C
Job definitions
A job definition states what Hermes must do, when it must do it, and where its result belongs. The definition is separate from scheduler-owned state such as the next run, last result, repeat count, and active claims. This separation lets configuration changes preserve useful run history and ownership data.
Schedule forms
The scheduler accepts both simple human time phrases and standard cron expressions. It normalizes each accepted form into a schedule that can produce the next occurrence.
- A one-shot delay such as
in 30mruns once. - An interval such as
every 2hrepeats at a fixed cadence. - A calendar rule such as
every monday 9amor0 9 * * *repeats on matching times. - An ISO timestamp runs once at a specified instant.
References: cron/jobs.py• L643-680 cron/jobs.py• L787-860
Work inputs
The task can use a prompt, reusable skills, or a script. A script can provide context before an agent run, or become the whole job in no-agent mode. A job can also set its working directory, model route, tool access, repeat limit, and result target. These inputs describe the intended work; the scheduler still decides if the job is safe and eligible at fire time.
References: cron/job_definition.py• L13-18 website/…/cron.md• L136-166 website/…/cron.md• L168-198
Schedule validation
Validation prevents a job from consuming an agent run when its setup cannot succeed. The scheduler checks the resolved provider access, required skill setup, delivery target, and job-requested tool connections before it creates agent work.
When validation finds a clear configuration problem, the job records blocked_config, sends one alert, and makes no model call. A later healthy check can clear the blocked condition so a new configuration problem can alert again.
References: cron/scheduler_preflight.py• L80-121 cron/scheduler_preflight.py• L236-293 cron/scheduler_preflight.py• L303-405 website/…/cron.md• L77-102
Scheduler ownership
The scheduler owns the decision to claim an occurrence, not the complete agent task. A built-in ticker normally checks due jobs every minute. An optional provider can supply another trigger, but all providers use the same shared execution and delivery path after a claim.
Profile ownership
Each profile has its own cron store and runs its jobs with that profile's settings and credentials. A host gateway can serve several profiles, but it scopes each tick to one profile and uses that profile's adapters. A profile with its own gateway is left to that gateway so two processes do not compete for its work.
References: cron/jobs.py• L75-85 cron/scheduler_provider.py• L118-133 cron/scheduler_provider.py• L430-461 cron/scheduler_ownership.py• L79-148
Trigger ownership
The selected scheduler provider is a trigger only. If a named provider is missing or unavailable, Hermes falls back to the built-in ticker instead of leaving jobs without a trigger. This keeps the job store and the execution rules stable when the deployment changes its wake-up mechanism.
References: cron/scheduler_provider.py• L136-194 cron/scheduler_provider.py• L387-427
Claim ownership
A tick lock and a per-job ownership guard prevent overlapping schedulers or manual actions from starting the same work at the same time. The guard is profile-scoped, which allows two profiles to use the same job identifier without sharing ownership.
References: cron/jobs.py• L288-319 cron/scheduler.py• L560-576 cron/scheduler.py• L806-855
Occurrence handling
An occurrence is the specific scheduled instant represented by a recurring rule. Hermes advances the next occurrence before dispatch and records enough claim information to account for a crash during the handoff. The execution ledger can then identify an occurrence that already completed and stop it from running twice after a restart.
The scheduler handles missed work in a small set of controlled ways:
- It runs a late occurrence inside the grace window.
- It collapses an older backlog into one catch-up run.
- It records a skip when catch-up is disabled.
- It restores one unclaimed occurrence when its previous owner is proven gone.
- It does not fire paused, disabled, completed, or expired one-shot jobs.
References: website/…/cron-internals.md• L117-164 cron/jobs.py• L937-963
Resource and quota controls
The subsystem limits pressure from both the agent runtime and the provider. Per-profile parallelism prevents a large due batch from taking over the host. An inactivity watchdog stops a stalled agent run, while active work can continue beyond that limit. Scripts use a separate execution limit.
Provider usage limits follow a different rule. When a provider gives a reliable future reopening time and no fallback provider is available, the scheduler parks recurring work until a safe recovery point. It avoids repeated failures during the closed window and resumes at a natural occurrence or a useful sparse-schedule recovery point. A run that reaches the model clears the hold.
References: cron/scheduler.py• L560-565 cron/scheduler_liveness.py• L25-49 cron/quota_hold.py• L1-14 cron/quota_hold.py• L92-163
Recovery rules
Recovery aims to preserve truthful state without replaying uncertain side effects. A durable attempt begins before dispatch. On restart, Hermes marks an attempt unknown only when the owner is proven dead, or when a live owner has stopped making progress past a derived stale bound. Unknown attempts are audit records, not automatic retries.
Recurring jobs can receive limited re-runs for transient provider or network failures that happen before any model call. Repeated failures also create grouped incidents, so Hermes alerts on a new failure pattern and uses a cooldown for the same pattern instead of sending the same alert on every tick.
References: cron/executions.py• L1-5 cron/executions.py• L152-185 cron/executions.py• L359-428 cron/incidents.py• L1-10 cron/incidents.py• L156-197
Health monitoring
Monitoring exposes both scheduler liveness and job health. Heartbeats show that the ticker is alive, while a separate success marker shows whether it completed a clean tick. The execution ledger records each attempt and its terminal result. Incident records group repeated failures without hiding the underlying run history.
Operators can use scheduler status to find a stopped or overdue ticker, run history to inspect attempts, incidents to review repeated errors, and the cron doctor to find configuration, dispatch, and late-run problems. These views describe different layers of health and should not be treated as one status value.
References: cron/scheduler_provider.py• L467-532 cron/executions.py• L213-335 cron/incidents.py• L242-276 website/…/cron.md• L533-558
Execution and delivery boundary
The cron subsystem ends its scheduling responsibility when it has accepted and claimed an occurrence. The agent then runs in its own scheduled session, and the result is delivered through the configured target while the scheduler records execution and delivery outcomes. Read Scheduled job execution for that downstream lifecycle.
References: cron/scheduler_provider.py• L183-237 cron/executions.py• L213-233 website/…/cron-internals.md• L210-212
Change checklist
When these files or sections are changed, remember to consider these:
- Keep the schedule parser, next-run calculation, and missed-occurrence rules aligned so a newly accepted schedule cannot become permanently due or silently skip an occurrence.
- Keep profile scoping aligned across the job store, execution ledger, ownership guard, worker pool, and delivery adapters.
- Preserve the distinction between a failed agent run, a failed delivery, a blocked configuration, and an unknown interrupted attempt.
- Keep trigger-provider changes limited to wake-up and claim behavior; execution and delivery must continue through the shared path.
- Keep quota holds and transient network retries separate so a known provider closure does not create repeated probes or duplicate side effects.
Important failure modes
| Trigger | Consequence | Guard |
|---|---|---|
| Provider credentials, skills, tools, or delivery target are not ready | The scheduled run is blocked before an agent call | Pre-dispatch validation records blocked_config and alerts once |
| Two scheduler paths inspect the same due job | The job could run twice concurrently | Per-store locks and profile-scoped in-flight ownership |
| Gateway restarts during a claimed occurrence | The result may be uncertain | Durable execution states and dead-owner recovery mark the attempt unknown |
| A recurring job is due during a closed provider usage window | Repeated failures and alerts would consume runs | Quota hold parks the job until a safe recovery point |
| Ticker work stops while the gateway process remains alive | Jobs become overdue without an obvious process failure | Per-profile heartbeat, success markers, status, and doctor checks |
Key files
| File | Why to read it |
|---|---|
| cron/jobs.py | Defines the durable job record, schedule parsing, occurrence state, and store locking. |
| cron/scheduler.py | Coordinates due-job claims, shared run admission, and scheduler-owned state changes. |
| cron/scheduler_provider.py | Defines trigger ownership and the built-in or external scheduler boundary. |
| cron/scheduler_preflight.py | Defines the checks that can refuse a run before agent work starts. |
| cron/scheduler_ownership.py | Explains profile ownership and how competing gateways stand aside. |
| cron/quota_hold.py | Defines provider-window parking and safe recovery timing. |
| cron/executions.py | Defines durable attempt history and restart recovery. |
| cron/incidents.py | Defines grouped failure alerts, cooldowns, resolution, and acknowledgement. |
| website/…/cron-internals.md | Provides the wider cron lifecycle and the contract between scheduling, execution, and delivery. |