Background jobs and operations
Background jobs and operations provide the server's asynchronous work layer. They move slow or repeatable work away from request handling and coordinate asset processing, user-defined workflows, scheduled cleanup, maintenance actions, and administrator controls. The two focused areas are automated asset analysis and storage and system operations.
Asynchronous execution model
The server separates API work from background work so a client request can enqueue a task and return while a worker processes it. Queues hold work in a shared job system, and each job has one named handler that performs the domain action.
Worker roles
The server runs as separate worker roles with different responsibilities. The API role serves client requests and watches for the background worker. The microservices role owns queue workers and executes jobs. The maintenance role serves a small, protected surface while the main server is in maintenance mode. This split lets background work scale and lets maintenance actions run without the normal application path.
| Worker role | System responsibility |
|---|---|
| API | Serves the normal HTTP and WebSocket interface and reports worker availability. |
| Microservices | Runs the queue workers and handles asynchronous jobs. |
| Maintenance | Serves maintenance status, backup, restore, and restart actions. |
References: server/…/app.module.ts• L105-151 server/…/api.ts• L9-23 server/…/microservices.ts• L12-32 server/…/maintenance.ts• L9-24
Queue and job lifecycle
A job is one unit of deferred work. A queue is a named stream of jobs with its own status and concurrency rules. Services enqueue jobs through the job boundary, a microservices worker claims them, and the job service emits start, success, error, and completion events around the handler. A successful job can enqueue follow-up work, which lets one user action produce a controlled chain of tasks.
The main flow moves from a request or event to a queued job and then to domain work. The result can update client state or create the next job.
flowchart LR
A[Request or server event] --> B[Queue job]
B --> C[Microservices worker]
C --> D[Job handler]
D --> E[Result and completion event]
D --> F[Follow-up jobs]
F --> B
The queue layer groups work by broad purpose. Asset analysis queues feed the search, face, duplicate, OCR, and media preparation flows. Background task queues handle cleanup and other general server work. Migration, backup, workflow, and integrity queues isolate work with different load or safety needs.
References: server/…/job.repository.ts• L40-99 server/…/job.repository.ts• L137-155 server/…/job.service.ts• L79-99 server/…/job.service.ts• L101-188 server/…/enum.ts• L815-835
Queue concurrency
Queue concurrency controls how many jobs of a queue can run at the same time. The microservices role starts one worker for each queue, then applies configured concurrency to queues that can safely run in parallel. Some work remains single-file or single-run because it changes shared state or must not overlap, such as database backup and storage template migration.
References: server/…/queue.service.ts• L44-103 server/…/queue.service.ts• L258-265 server/…/job.repository.ts• L90-100
Workflow execution
Workflows let users connect an event to a sequence of plugin actions. The workflow system stores the trigger and steps, checks that each action supports that trigger, and runs the workflow as a background job when a matching asset event occurs.
Workflow triggers
An asset event starts a workflow run without blocking the action that caused the event. Asset creation, metadata extraction, and tagging can find matching workflows and enqueue one workflow job for each match. The workflow queue therefore acts as the bridge between normal asset activity and user-defined automation.
References: server/…/workflow-execution.service.ts• L307-340 server/…/workflow.service.ts• L23-61 server/…/workflow.controller.ts• L25-56
Workflow steps
Each workflow run reads the current asset state, passes it through its ordered steps, and writes accepted changes back to the server. A step can stop the run, update later input, or fail the run. Optional logs record whether the run completed, halted, or failed so an operator can inspect the outcome without tracing the queue itself.
References: server/…/workflow-execution.service.ts• L390-516 server/…/workflow.service.ts• L64-124 server/…/workflow.controller.ts• L107-131
Scheduled and maintenance work
The operations layer keeps the system healthy after the original request ends. It schedules recurring tasks, exposes safe manual starts for administrators, and provides a separate mode for work that must pause normal application activity.
Nightly task scheduling
One server instance claims the nightly-task lock and schedules the configured nightly run. At the configured time, that run adds only the enabled tasks to the background queue. The task set covers database cleanup, memory generation, quota sync, missing thumbnails, and face clustering. A lock prevents multiple server instances from starting the same nightly work.
References: server/…/queue.service.ts• L34-63 server/…/queue.service.ts• L66-80 server/…/queue.service.ts• L267-300 server/…/enum.ts• L977-995
Manual operational jobs
Administrators can start selected maintenance and integrity jobs without waiting for the schedule. The server maps a small public set of manual job names to internal queue work. This keeps the operational interface stable while the job graph can grow behind it.
References: server/…/job.controller.ts• L32-58 server/…/job.service.ts• L11-83 server/…/enum.ts• L437-455
Maintenance mode
Maintenance mode replaces the normal application path with a protected control surface. It supports status checks, database backup management, database restore, and a controlled return to normal service. The mode stores its state so a restart can continue the selected action, and it uses a separate token and route guard for access.
The maintenance flow changes the stored state, restarts into the maintenance worker, performs the requested action, and broadcasts a restart when it ends.
sequenceDiagram
participant O as Operator
participant A as API
participant M as Maintenance worker
participant D as Database and storage
O->>A: Start maintenance or restore
A->>D: Store maintenance state
A->>A: Restart application
M->>D: Read state and run action
M-->>O: Report status and progress
M->>D: Clear maintenance state
M->>A: Request normal restart
References: server/…/maintenance.service.ts• L25-75 server/…/maintenance-worker.service.ts• L67-87 server/…/maintenance-worker.service.ts• L242-360 server/…/maintenance-auth.guard.ts• L18-41 server/…/maintenance-worker.controller.ts• L67-136
Operational coordination
Background work must remain safe when more than one server process can see the same database and queue system. The server uses shared database locks for singleton work and reports when the queue worker is absent. Administrator endpoints then expose the state needed to pause, inspect, drain, or resume work.
Shared locks
A shared database lock gives one process ownership of work that must not run twice. The same coordination rule protects nightly jobs, plugin import, version checks, maintenance actions, and other system tasks. This is separate from queue concurrency: a queue controls parallel jobs, while a lock controls ownership across processes.
References: server/…/queue.service.ts• L44-63 server/…/workflow-execution.service.ts• L55-67 server/…/maintenance-worker.service.ts• L297-328 server/…/enum.ts• L977-995
Worker availability
The API does not execute background jobs itself. It checks the shared queue for a connected microservices worker and logs when no worker can process the waiting work. This gives operators a useful distinction between a full queue and a queue with no executor.
References: server/…/job.repository.ts• L102-135 server/…/queue.service.ts• L82-95
Cluster group analysis
Users in a cluster group share the result of face clustering. A group-level regenerate request enqueues one forceful facial-recognition job for the group, so the expensive analysis stays asynchronous and uses the same queue controls as other asset analysis.
This product behavior belongs with automated asset analysis, while the cluster group API owns access checks and the request to start the job.
References: server/…/cluster-group.service.ts• L75-82 server/…/cluster-group.controller.ts• L96-105
Queue controls
Administrators can read queue counts and job records, pause or resume eligible queues, and remove queued or failed jobs. The background-task queue cannot be paused because it carries general system work that the server depends on. These controls require administrator access and queue-specific permissions.
References: server/…/queue.controller.ts• L17-84 server/…/queue.service.ts• L146-189 server/…/queue.service.ts• L191-255
Change checklist
When these files or sections are changed, remember to consider these:
- When adding a job name, add one handler and assign it to a queue. Startup discovery rejects missing or duplicate handlers.
- When changing a queue name or job payload, update the producer, handler, queue-control DTOs, and any generated API contract that exposes the name or payload.
- When changing job follow-up behavior, check the event names and client notifications that mark an upload, edit, or analysis as ready.
- When changing nightly task settings, preserve the single-owner lock and the mapping from each setting to its queued job.
- When changing maintenance actions, keep stored maintenance state, restart events, progress updates, and the maintenance-only routes aligned.
- When changing worker roles, keep queue startup, API worker watching, and module registration consistent.
Important failure modes
| Trigger | Consequence | Guard |
|---|---|---|
| No microservices worker is connected | Queued background work waits and is not processed | The API checks worker presence and logs a warning. |
| A job has no discovered handler or has two handlers | The server fails during startup instead of accepting an ambiguous job graph | Job setup validates every job name and rejects duplicates. |
| An operator pauses the background-task queue | General system work could stop, so the operation is rejected | The queue service forbids pausing that queue. |
| Two processes start singleton work | Duplicate cleanup, imports, backups, or maintenance actions can conflict | Shared database locks allow one owner. |
| A workflow step fails | The workflow stops and is marked as failed when logging is enabled | The workflow runner catches step errors and records the failed step. |
| Maintenance restore fails | The server stays in maintenance mode and reports an error | The maintenance worker records the error in status and does not clear the mode. |
Key files
| File | Why to read it |
|---|---|
| server/…/app.module.ts | Shows how API, microservices, and maintenance roles compose the server. |
| server/…/job.repository.ts | Defines queue ownership, worker startup, handler discovery, enqueueing, and queue state. |
| server/…/job.service.ts | Shows the job event lifecycle and the main follow-up job chains. |
| server/…/queue.service.ts | Connects configuration and nightly schedules to queue operations. |
| server/…/queue.controller.ts | Defines administrator queue inspection and control boundaries. |
| server/…/workflow-execution.service.ts | Shows how asset events become workflow jobs and how steps read and write asset state. |
| server/…/maintenance.service.ts | Starts and ends maintenance mode from the normal server path. |
| server/…/maintenance-worker.service.ts | Runs protected maintenance actions and reports their progress. |
| server/…/cluster-group.service.ts | Shows how a group-level analysis request enters the background system. |