Scheduled job execution
Scheduled job execution turns a saved job definition into one controlled run. The scheduler finds a due occurrence, checks that the run is safe and usable, builds the agent request, selects a provider, starts work in the right profile, delivers the result, and records what happened. This page complements Scheduled automation by focusing on the execution path.
Execution model
The execution model separates when a job fires from how it runs. A scheduler provider owns the trigger, while the shared cron execution path owns prompts, agent work, delivery, and run state.
Job definitions
A job definition describes the task and its schedule. It can also select skills, a model or provider, a pre-run script, a working directory, context from other jobs, toolsets, and result or failure targets. The store keeps scheduler state with these authored fields so an edit changes the task without losing run history or ownership state.
Schedules support one-time dates, delayed one-shots, recurring intervals, and cron-style calendar rules. A recurring job keeps its next occurrence anchored to the configured profile time zone. A finite one-shot claims its dispatch before side effects begin, which prevents a restart from running the same limited dispatch more than once.
References: cron/jobs.py• L1818-1948 cron/jobs.py• L1178-1237 cron/jobs.py• L2590-2642 cron/job_definition.py• L13-45
Due-job admission
The ticker checks each profile's store on its schedule. A per-profile tick lock prevents two processes from scanning the same store at the same time. For recurring work, the ticker advances the next occurrence before it submits the run. This gives the system at-most-once behavior across a crash, while a durable pending-slot marker preserves an occurrence that was advanced but never claimed.
The fire claim is the final admission gate. It rejects paused, completed, or already claimed work, and binds the run to one scheduled instant when the fire is a real scheduled occurrence. Manual runs remain separate from future scheduled occurrences.
The main path is:
flowchart LR
A[Job definition] --> B[Profile ticker]
B --> C[Due occurrence]
C --> D[Recurring advance or one-shot claim]
D --> E[Fire claim]
E --> F[Execution attempt]
F --> G[Output and delivery]
G --> H[Job state and execution ledger]
References: cron/scheduler_provider.py• L430-532 cron/scheduler_tick.py• L73-184 cron/occurrences.py• L58-103 cron/jobs.py• L2741-2807
Scheduler providers
The scheduler provider is a trigger interface. It may be the built-in in-process ticker or an external provider. Providers decide when a job fires; they do not create a second execution or delivery system. Every provider uses the same claim and run_one_job path, so provider choice does not change job semantics.
If a configured provider is missing, unavailable, or fails to load, the system uses the built-in ticker. Multiplexed profiles also use the built-in ticker when an external provider cannot identify which profile owns a job. A saved job is registered with the active provider after creation; a registration error keeps the job and asks the operator to retry registration rather than create a duplicate.
References: cron/scheduler_provider.py• L1-5 cron/scheduler_provider.py• L136-240 cron/scheduler_provider.py• L387-427 cron/scheduler.py• L4022-4033
Preflight and prompt construction
Preflight prevents a job from spending an agent call when a clear configuration problem can be found first. Prompt construction then combines the saved task with the run's current data and applies safety checks before an agent starts.
Preflight checks
Preflight checks the provider credential, delivery targets, attached skill readiness, and requested tool availability. It blocks only on a positive misconfiguration result. A failed check of the checker itself does not block the job, because an uncertain preflight result must not turn a temporary inspection problem into a permanent scheduler stop.
A blocked configuration produces one operator-facing notice and leaves the recurring job for a later scheduled attempt. The notice states that no model call was made. A missing or unsafe payload is different: the scheduler pauses an unrunnable job so it does not fail on every tick.
References: cron/scheduler_preflight.py• L80-122 cron/scheduler_preflight.py• L236-293 cron/scheduler_preflight.py• L303-329 cron/scheduler_preflight.py• L387-406 cron/scheduler_prompt.py• L423-443
Prompt assembly
The effective prompt starts with the job's user text and the scheduled-job guidance. The scheduler can add one-run context, monitor data, pre-run script output, earlier job output, the job's own prior output, a durable notepad, and named skills. A script can also act as the whole job when agent execution is disabled.
The final assembled text is scanned for prompt injection after skills and runtime data are added. The scanner uses a stricter path for user-only text and a safer data-aware path when loaded content is present. A blocked prompt is saved as a blocked run and the agent is not started.
flowchart TD
A[Saved prompt] --> B[Run context]
B --> C[Script and monitor data]
C --> D[Earlier job output]
D --> E[Skills and scheduled-job guidance]
E --> F[Injection scan]
F -->|allowed| G[Agent prompt]
F -->|blocked| H[Blocked failure record]
References: cron/scheduler_prompt.py• L129-190 cron/scheduler_prompt.py• L193-257 cron/scheduler_prompt.py• L260-395 cron/scheduler.py• L2224-2311
Provider and agent setup
After prompt checks pass, the run resolves its model, provider, credentials, fallback policy, reasoning settings, and tool access. A pinned job stays on its selected provider and does not use the global provider fallback chain. An unpinned job can use the configured chain.
The agent runs in a cron-specific session. The session has no inbound chat identity, uses the job's working directory when set, and skips memory and background review work that is not part of a scheduled run. The job's profile supplies its configuration and secrets, including when one gateway serves several profiles.
References: cron/scheduler.py• L91-129 cron/scheduler.py• L2423-2499 cron/scheduler.py• L2320-2388 cron/scheduler.py• L3245-3281
Worker execution
The execution layer gives each fire a durable identity and an isolated place to run. This lets the gateway survive a worker handoff and lets recovery distinguish a completed result from work whose side effects are unknown.
Execution ledger
Each attempt is written to the profile-local execution ledger before dispatch. Its state moves from claimed to running and then to completed, failed, or unknown. The ledger also records the scheduled instant, owner process, delivery outcome, and error detail.
The job record and ledger serve different needs. The job record controls future scheduling and operator-visible latest status. The ledger is the audit trail for one attempt and is not itself a retry queue. Terminal ledger states are not rewritten.
References: cron/executions.py• L1-6 cron/executions.py• L213-335 cron/executions.py• L359-428
Worker bootstrap
In a managed gateway, the run can move to a restart-safe external worker. The gateway writes a private payload, marks the ledger handoff as pending, and starts the worker with the owning profile. The worker must first adopt the ledger row; only the adopting worker may run side effects and publish its ready acknowledgement.
The worker restores the Hermes source and dependency environment before it imports the job system. It then loads the owning profile's secrets and hooks, completes the normal Hermes bootstrap, and runs the payload. This order lets a source update or dependency re-selection restart the whole worker without losing the payload or creating an unowned run.
sequenceDiagram
participant G as Gateway
participant L as Execution ledger
participant W as External worker
participant P as Provider and agent
G->>L: Create attempt and mark handoff pending
G->>W: Write profile-scoped payload and start process
W->>L: Adopt claimed attempt
W-->>G: Publish ownership acknowledgement
W->>P: Bootstrap profile and execute job
P->>L: Record terminal outcome
G->>L: Observe completion or recover uncertainty
References: cron/scheduler.py• L3626-3864 cron/scheduler.py• L3867-3969 cron/worker_bootstrap.py• L1-67 cron/scheduler_worker_env.py• L64-85
Run watchdog
The run keeps its fire claim fresh while it works. The inactivity watchdog measures silence rather than total run time, so an active long task can continue while a stalled task is interrupted. Agent resources stay alive through result delivery and are closed after delivery; this preserves the live transport needed by some adapters.
References: cron/scheduler.py• L2688-2769 cron/scheduler.py• L2636-2657 cron/scheduler_detached_worker.py• L1-53
Result delivery and failure reporting
A run produces a saved output document before it completes its bookkeeping. The delivery layer then sends the result or failure notice through the configured target rules, while the job record and execution ledger preserve the outcome even when no target is available.
Result delivery
Delivery resolves local, origin, home-channel, explicit, broadcast, and Bot Chat targets. A live gateway adapter is preferred when available, with a standalone sender as fallback where that is safe. A restart-safe worker has no live adapters, so it places the delivery into a durable queue for the gateway.
The delivery path treats a response as confirmed only when the adapter reports success. It records unverified acknowledgements, target errors, queued work, and delivery warnings separately from agent failure. Successful deliveries may also mirror the brief or seed a continuation session when the target is eligible.
References: cron/scheduler_delivery.py• L1079-1111 cron/scheduler_delivery.py• L1646-1842 cron/scheduler_delivery.py• L1973-2007 cron/scheduler.py• L3050-3125
Failure reporting
Failure reporting distinguishes agent failure from delivery failure. A failed run creates or updates a failure incident, builds a short notice with a useful operator action, and sends it through failure_deliver when that override exists. The full run output and diagnostic detail remain available in the job's saved output.
Blocked configuration alerts are sent once because repeating the same warning on every tick adds noise. A run that returns the literal scheduled-job silence marker is successful but has no outward delivery. An agent can also declare a task failure with the strict [CRON_FAILURE] first-line marker; that result follows the normal failure path.
References: cron/scheduler.py• L2924-2970 cron/scheduler.py• L3207-3242 cron/scheduler_failure_copy.py• L88-148 cron/scheduler.py• L533-558
Outcome bookkeeping
The final bookkeeping step updates the job's last status, failure streak, delivery error, repeat count, and next occurrence. It then writes the terminal ledger state. Owner fencing prevents a stale worker from overwriting a newer fire.
An agent failure and a delivery failure remain distinct. A successful agent run with a failed send is recorded as delivery failure, while an agent failure with a sent warning records both the failed run and its delivery outcome. This distinction lets operators decide whether to repair the task or the target channel.
References: cron/jobs.py• L2455-2522 cron/scheduler.py• L3153-3204 cron/executions.py• L312-335
Unreachable-work retries
Unreachable-work retries cover a narrow outage: the provider could not be reached because of a transient network or DNS error, and the agent made zero model calls. The retry is safe because the run did not execute model work or spend a provider call.
Retry ladder
For recurring jobs, the scheduler can place up to three silent re-runs at five, fifteen, and thirty minutes. The ladder does not replace the job's natural schedule when that occurrence comes first. A retry does not consume another repeat slot because it re-runs the same scheduled occurrence.
The failure notice stays suppressed while a retry is pending. If a retry reaches the model, the ladder resets whether the model succeeds or fails. If all rungs fail, the job returns to its normal schedule and the next failure can start a fresh ladder. Operators can disable this behavior with cron.retry_unreachable: false.
flowchart LR
A[Transient network or DNS failure] --> B{Zero model calls?}
B -->|No| C[Normal failure reporting]
B -->|Yes| D{Recurring and retry enabled?}
D -->|No| C
D -->|Yes| E[Retry after 5 minutes]
E --> F[Retry after 15 minutes]
F --> G[Retry after 30 minutes]
G --> H[Natural schedule]
E --> I[Model reached: reset ladder]
F --> I
G --> I
References: cron/unreachable_retry.py• L1-18 cron/unreachable_retry.py• L42-65 cron/unreachable_retry.py• L68-107 cron/unreachable_retry.py• L110-166 cron/jobs.py• L2476-2481
Uncertain worker outcomes
A worker that dies after adoption cannot safely be called a normal failure because side effects may already have run. The ledger marks the attempt unknown after the owner is proved dead or has exceeded the derived stale bound. The scheduler then reports the uncertain outcome, retires the job claim, and does not replay the side effect automatically.
References: cron/executions.py• L338-483 cron/scheduler.py• L3520-3590 cron/scheduler_worker_failure.py• L4-61
Change checklist
When these files or sections are changed, remember to consider these:
- Keep scheduler-provider fires on the shared claim, execution, delivery, and bookkeeping path. A provider-specific execution path can break repeat limits or ledger ownership.
- Keep
pending_slot, the scheduled occurrence in the ledger, and completed-occurrence checks aligned. Changing one can cause a missed occurrence or a duplicate fire after a crash. - Keep the prompt scanner's treatment of user text, skills, and runtime data aligned with the actual assembled prompt. A new prompt source can bypass the intended safety check.
- Keep the owning profile scope through provider resolution, worker environment creation, agent work, and delivery. A scope gap can use another profile's credentials or delivery target.
- Keep failure delivery and result delivery on the same target-resolution contract when adding a delivery mode. Failure notices must not silently use a different target set.
- Keep the unreachable retry ladder spend-neutral. Any new retry trigger must still prove a transient provider error and zero model calls.
Important failure modes
| Trigger | Consequence | Guard |
|---|---|---|
| Recurring schedule advances but dispatch never claims the slot | The next scan could skip the occurrence | pending_slot restores an unclaimed slot once after its owner is gone. |
| External worker adopts no ledger ownership | The gateway must not run the payload a second time | The worker must win the claimed-to-running transition before acknowledgement or side effects. |
| Adopted worker exits before a terminal ledger state | Side effects may have happened, so a replay can duplicate them | The attempt becomes unknown; recovery reports the uncertainty without automatic replay. |
| Provider credential, delivery, or skill setup is clearly missing | The job could spend a call and fail in a predictable way | Preflight blocks before agent construction and sends a one-time configuration notice. |
| Live adapter returns no positive delivery evidence | The job could be shown as delivered when no message was sent | Delivery requires a positive success result and records unverified acknowledgements. |
| Transient network failure occurs after a model call | A retry could repeat work or side effects | The unreachable ladder requires zero model calls and only applies to recurring jobs. |
Key files
| File | Why to read it |
|---|---|
| cron/jobs.py | Defines job records, schedule state, occurrence claims, and final job bookkeeping. |
| cron/scheduler_tick.py | Shows how the ticker admits due jobs and submits them without losing recurring slots. |
| cron/scheduler_provider.py | Defines the trigger boundary and the built-in profile-aware ticker. |
| cron/scheduler.py | Connects execution, worker handoff, delivery, failure handling, and terminal bookkeeping. |
| cron/scheduler_preflight.py | Explains the checks that can stop a run before agent work begins. |
| cron/scheduler_prompt.py | Explains how saved prompts become safe, run-specific agent input. |
| cron/scheduler_delivery.py | Explains target resolution, live and standalone sending, queueing, and session continuation. |
| cron/executions.py | Defines durable attempt ownership, terminal states, and recovery of abandoned work. |
| cron/worker_bootstrap.py | Shows the dependency and Hermes bootstrap required by restart-safe workers. |
| cron/unreachable_retry.py | Defines the bounded retry policy for work that never reached the model. |