Knowledge Base updated: 2026-11-19
Knowledge Base Index
1. Orientation
ComfyUI is a Python application for graph-based generative media workflows. The highest-level flow is: the server accepts workflow prompts, validates a node graph, loads/patches models as needed, executes nodes through the prompt executor, and streams progress/results back over HTTP/WebSocket APIs.
Main parts:
- main.py / server.py — process startup, PromptServer, HTTP routes, WebSocket protocol, middleware, and API plumbing.
- execution.py /
comfy_execution/**— prompt graph validation, scheduling, caching, job lifecycle, and progress/media enrichment. - nodes.py,
comfy_extras/**,comfy_api_nodes/**— built-in, extra, and hosted/provider-backed node implementations. comfy/**— model loading, memory/device management, samplers, conditioning, encoders, and model architecture implementations.comfy_api/**— versioned public/internal API abstractions used by nodes and integrations.app/**and related asset directories — application services for settings, assets, users, custom nodes, models, cloud/deploy features, and persistence.comfy_config/**,utils/**, tests, and project config — configuration, CLI/options, utilities, scripts, and validation structure.
Critical state lives in:
- Filesystem model/input/output/custom-node trees — model discovery and generated media assets (
models/,input/,output/,custom_nodes/, plus path configuration in folder_paths.py). - Application database and migrations — app-level persisted entities and schema evolution (
app/database/**,alembic_db/**, alembic.ini). - Execution caches and job state — in-memory/runtime prompt execution state under
comfy_execution/**and execution.py.
2. Architecture diagrams
flowchart TD
User[Browser or API Client]
subgraph Server[Startup and API Server]
Main[main.py]
PromptServer[server.py PromptServer]
Protocol[protocol.py]
Middleware[middleware]
InternalAPI[api_server]
end
subgraph Execution[Prompt Execution]
Executor[execution.py]
Graph[comfy_execution graph and validation]
Jobs[comfy_execution jobs progress cache]
end
subgraph Nodes[Node Surfaces]
CoreNodes[nodes.py]
ExtraNodes[comfy_extras]
HostedNodes[comfy_api_nodes]
end
subgraph Models[Core ML Runtime]
Loading[model loading and management]
Sampling[sampling and conditioning]
Encoders[text image audio encoders]
LDM[ldm and specialized models]
end
subgraph AppServices[Application Services]
App[app services]
DB[app database and alembic]
Assets[input output models custom_nodes]
end
subgraph APIPlatform[Versioned API Platform]
ComfyAPI[comfy_api]
end
User --> PromptServer
Main --> PromptServer
PromptServer --> Middleware
PromptServer --> Protocol
PromptServer --> InternalAPI
PromptServer --> Executor
Executor --> Graph
Graph --> Jobs
Executor --> CoreNodes
Executor --> ExtraNodes
Executor --> HostedNodes
CoreNodes --> Loading
ExtraNodes --> Sampling
HostedNodes --> ComfyAPI
Loading --> Encoders
Loading --> LDM
Sampling --> LDM
App --> DB
App --> Assets
PromptServer --> App
Loading --> Assets
sequenceDiagram
participant Client
participant Server as server.py PromptServer
participant Validation as comfy_execution.validation
participant Queue as execution.py / jobs
participant Nodes as node registry
participant Models as comfy model runtime
Client->>Server: submit prompt graph
Server->>Validation: validate graph and inputs
Validation-->>Server: validated graph or errors
Server->>Queue: enqueue execution job
Queue->>Nodes: execute node graph
Nodes->>Models: load/patch/sample/encode as required
Models-->>Nodes: tensors, latents, media outputs
Nodes-->>Queue: node outputs and progress
Queue-->>Server: status, previews, results
Server-->>Client: HTTP/WebSocket updates
flowchart LR
Paths[folder_paths.py and configured dirs]
Detection[model_detection and supported_models]
Loaders[sd.py diffusers_load checkpoint_convert]
Patcher[model_patcher and patcher_extension]
Memory[model_management memory_management multigpu]
Encoders[CLIP text vision audio encoders]
Samplers[sample.py samplers k_diffusion]
Outputs[latents images audio video]
Paths --> Detection
Detection --> Loaders
Loaders --> Patcher
Loaders --> Encoders
Patcher --> Memory
Memory --> Samplers
Encoders --> Samplers
Samplers --> Outputs
flowchart TD
Registry[nodes.py node registry]
Core[Core built-in nodes]
Extras[comfy_extras nodes]
Hosted[comfy_api_nodes provider nodes]
API[comfy_api abstractions]
Execution[execution.py]
External[External provider APIs]
Registry --> Core
Registry --> Extras
Registry --> Hosted
Hosted --> API
Hosted --> External
Execution --> Registry
3. Glossary
- PromptServer — the server-side HTTP/WebSocket orchestration surface implemented primarily in server.py.
- Prompt graph — the submitted node graph validated and executed by execution.py and
comfy_execution/**. - Comfy API — the versioned API abstraction package under
comfy_api/**, distinct from the HTTP server routes. - Hosted API nodes — provider-backed nodes under
comfy_api_nodes/**that call external services through shared API utilities. - LDM — the collection of latent diffusion and specialized model architecture implementations under
comfy/ldm/**.
4. Codebase-wide invariants
- Core, extra, and hosted nodes are separated by path: built-ins live in nodes.py, optional local extensions live under
comfy_extras/**, and provider/API-backed nodes live undercomfy_api_nodes/**. - The prompt execution engine is split between the legacy/root entrypoint execution.py and supporting graph, validation, cache, job, progress, and enrichment modules under
comfy_execution/**. - Versioned API abstractions are isolated under
comfy_api/**; HTTP server routing and middleware are owned separately by server.py,api_server/**, andmiddleware/**. - Model assets and generated media are filesystem-oriented: representative roots include
models/,input/,output/,custom_nodes/, with path resolution centralized by folder_paths.py. - Test coverage is split between integration-style tests in
tests/**and unit-focused tests intests-unit/**, with shared test configuration in pytest.ini.
5. Routing table
-
middleware/**-> docs/server-startup-and-http-api.md -
api_server/routes/**-> docs/server-startup-and-http-api.md -
api_server/services/**-> docs/server-startup-and-http-api.md -
api_server/utils/**-> docs/server-startup-and-http-api.md -
any change to startup, HTTP routes, WebSocket messages, or server middleware -> docs/server-startup-and-http-api.md
-
comfy_execution/**-> docs/prompt-execution-graph.md -
any change to prompt validation, execution scheduling, node execution order, cache behavior, job cancellation, or progress reporting -> docs/prompt-execution-graph.md
-
latent_preview.py -> docs/core-node-registry-and-builtins.md
-
any change to core node registration, node class contracts, built-in nodes, or latent previews -> docs/core-node-registry-and-builtins.md
-
comfy/model_management.py -> docs/model-loading-and-management.md
-
comfy/model_detection.py -> docs/model-loading-and-management.md
-
comfy/supported_models.py -> docs/model-loading-and-management.md
-
comfy/supported_models_base.py -> docs/model-loading-and-management.md
-
comfy/model_patcher.py -> docs/model-loading-and-management.md
-
comfy/model_prefetch.py -> docs/model-loading-and-management.md
-
comfy/checkpoint_convert.py -> docs/model-loading-and-management.md
-
comfy/diffusers_convert.py -> docs/model-loading-and-management.md
-
comfy/diffusers_load.py -> docs/model-loading-and-management.md
-
comfy/memory_management.py -> docs/model-loading-and-management.md
-
comfy/system_memory.py -> docs/model-loading-and-management.md
-
comfy/pinned_memory.py -> docs/model-loading-and-management.md
-
comfy/anonymous_file.py -> docs/model-loading-and-management.md
-
any change to model discovery, checkpoint loading, diffusers conversion/loading, device placement, memory management, model patching, or prefetching -> docs/model-loading-and-management.md
-
comfy/samplers.py -> docs/sampling-diffusion-conditioning.md
-
comfy/sampler_helpers.py -> docs/sampling-diffusion-conditioning.md
-
comfy/model_sampling.py -> docs/sampling-diffusion-conditioning.md
-
comfy/controlnet.py -> docs/sampling-diffusion-conditioning.md
-
comfy/patcher_extension.py -> docs/sampling-diffusion-conditioning.md
-
comfy/extra_samplers/**-> docs/sampling-diffusion-conditioning.md -
comfy/k_diffusion/**-> docs/sampling-diffusion-conditioning.md -
comfy/latent_formats.py -> docs/sampling-diffusion-conditioning.md
-
comfy/context_windows.py -> docs/sampling-diffusion-conditioning.md
-
comfy/pixel_space_convert.py -> docs/sampling-diffusion-conditioning.md
-
any change to diffusion sampling, conditioning, ControlNet behavior, sampler algorithms, latent formats, or sampling hooks -> docs/sampling-diffusion-conditioning.md
-
comfy/clip_config*.json -> docs/text-image-audio-encoders.md
-
comfy/clip_vision_config*.json -> docs/text-image-audio-encoders.md
-
comfy/sd1_tokenizer/**-> docs/text-image-audio-encoders.md -
comfy/text_encoders/**-> docs/text-image-audio-encoders.md -
comfy/image_encoders/**-> docs/text-image-audio-encoders.md -
comfy/audio_encoders/**-> docs/text-image-audio-encoders.md -
any change to tokenizer, CLIP/text encoder, vision encoder, image encoder, or audio encoder behavior -> docs/text-image-audio-encoders.md
-
comfy/ldm/**-> docs/ldm-and-specialized-models.md -
comfy/cldm/**-> docs/ldm-and-specialized-models.md -
comfy/t2i_adapter/**-> docs/ldm-and-specialized-models.md -
comfy/taesd/**-> docs/ldm-and-specialized-models.md -
comfy/background_removal/**-> docs/ldm-and-specialized-models.md -
comfy/bg_removal_model.py -> docs/ldm-and-specialized-models.md
-
any change to specialized model architecture families, latent diffusion backbones, adapters, TAESD, or background-removal models -> docs/ldm-and-specialized-models.md
-
comfy_api/**-> docs/comfy-api-platform.md -
any change to versioned Comfy API abstractions, generated API stubs, feature flags, public/internal API helpers, or input/output/UI API types -> docs/comfy-api-platform.md
-
comfy_api_nodes/**-> docs/hosted-api-nodes.md -
any new or changed provider-backed hosted node, external provider client, hosted-node auth/input/output convention, or API-node utility -> docs/hosted-api-nodes.md
-
comfy_extras/**-> docs/extra-nodes-and-extensions.md -
any new or changed optional node, media-processing node, model-specific extra node, or extension under
comfy_extras-> docs/extra-nodes-and-extensions.md -
app/**-> docs/app-services-cloud-and-assets.md -
alembic_db/**-> docs/app-services-cloud-and-assets.md -
blueprints/**-> docs/app-services-cloud-and-assets.md -
custom_nodes/**-> docs/app-services-cloud-and-assets.md -
models/**-> docs/app-services-cloud-and-assets.md -
input/**-> docs/app-services-cloud-and-assets.md -
output/**-> docs/app-services-cloud-and-assets.md -
any change to app settings, users, assets, custom-node management, model management services, cloud deployment, governance, logging, database models, or migrations -> docs/app-services-cloud-and-assets.md
-
comfy_config/**-> docs/configuration-utilities-and-tooling.md -
comfy/cli_args.py -> docs/configuration-utilities-and-tooling.md
-
comfy/options.py -> docs/configuration-utilities-and-tooling.md
-
comfy/configurable.py -> docs/configuration-utilities-and-tooling.md
-
comfy/comfy_types/**-> docs/configuration-utilities-and-tooling.md -
comfy/utils.py -> docs/configuration-utilities-and-tooling.md
-
comfy/float.py -> docs/configuration-utilities-and-tooling.md
-
comfy/quant_ops.py -> docs/configuration-utilities-and-tooling.md
-
comfy/rmsnorm.py -> docs/configuration-utilities-and-tooling.md
-
comfy/nested_tensor.py -> docs/configuration-utilities-and-tooling.md
-
utils/**-> docs/configuration-utilities-and-tooling.md -
script_examples/**-> docs/configuration-utilities-and-tooling.md -
tests/**-> docs/configuration-utilities-and-tooling.md -
tests-unit/**-> docs/configuration-utilities-and-tooling.md -
pyproject.toml -> docs/configuration-utilities-and-tooling.md
-
requirements.txt -> docs/configuration-utilities-and-tooling.md
-
agent_requirements.txt -> docs/configuration-utilities-and-tooling.md
-
manager_requirements.txt -> docs/configuration-utilities-and-tooling.md
-
extra_model_paths.yaml.example-> docs/configuration-utilities-and-tooling.md -
any change to configuration parsing, CLI flags, shared utilities/types/math ops, install helpers, examples, tests, or project dependency/test configuration -> docs/configuration-utilities-and-tooling.md