Introducing a new way to review your PRs.

Read the announcement

comfy-org / comfyui

The most powerful and modular diffusion model GUI, api and backend with a graph/nodes interface. The fastest local inference engine in the world.

View on GitHub

13 documents · Updated Nov 19, 2026

docs / model-loading-and-management

Model Loading and Management

Overview

This area covers how ComfyUI resolves model files, detects their architectures, constructs model wrappers, patches weights, and moves weights between CPU, GPU, pinned host memory, disk-backed storage, and multiple GPUs. Route reviews here for changes under folder_paths.py, cuda_malloc.py, comfy/sd.py, comfy/model_management.py, comfy/model_detection.py, comfy/supported_models.py, comfy/supported_models_base.py, comfy/model_base.py, comfy/model_patcher.py, comfy/model_prefetch.py, comfy/checkpoint_convert.py, comfy/diffusers_convert.py, comfy/diffusers_load.py, comfy/memory_management.py, comfy/system_memory.py, comfy/multigpu.py, comfy/pinned_memory.py, comfy/storage.py, and comfy/anonymous_file.py.

Model discovery starts in folder_paths.py. At import time it derives base_path and models_dir from CLI options, then registers folder types such as checkpoints, loras, vae, text_encoders, diffusion_models, and diffusers in folder_names_and_paths. Nodes and loaders use get_filename_list() for cached recursive listings, get_full_path()/get_full_path_or_raise() for safe model-file lookup by folder type, and add_model_folder_path() when config or extensions add search roots. The same file also owns output/temp/input/user path helpers, but model routing depends mainly on folder type aliases (unet maps to diffusion_models, clip maps to text_encoders), extension filters, and the filename-list mtime cache.

Checkpoint loading is orchestrated by comfy/sd.py. load_checkpoint_guess_config() reads a checkpoint via comfy.utils.load_torch_file(), applies the small pre-detection conversion in comfy/checkpoint_convert.py for SenseNova-style merged checkpoints, and calls load_state_dict_guess_config(). That function finds the diffusion prefix with comfy/model_detection.py::unet_prefix_from_state_dict(), converts old quantized layouts when no custom ops are supplied, detects the model config with model_config_from_unet(), chooses UNet dtype/manual-cast settings through comfy/model_management.py, builds a model_base subclass through the detected config, wraps it in ModelPatcher or CoreModelPatcher, and loads weights through BaseModel.load_model_weights() in comfy/model_base.py. If the checkpoint contains VAE and text-encoder weights, load_state_dict_guess_config() also extracts them using prefixes declared by the detected model class in comfy/supported_models_base.py/comfy/supported_models.py, builds VAE and CLIP wrappers from comfy/sd.py, and returns (model_patcher, clip, vae, clipvision).

Standalone diffusion and diffusers loading share the same detection path. comfy/sd.py::load_diffusion_model() reads a UNet/diffusion-model state dict and delegates to load_diffusion_model_state_dict(), which accepts native Comfy/LD checkpoint prefixes, Diffusers MMDiT layouts via comfy/model_detection.py::convert_diffusers_mmdit(), and legacy Diffusers UNets via model_config_from_diffusers_unet() plus key mapping from comfy.utils.unet_to_diffusers(). comfy/diffusers_load.py::load_diffusers() is the folder-based entry point for Hugging Face Diffusers directories: it locates unet/, vae/, and text_encoder* weight files by conventional names, then calls comfy/sd.py loaders. comfy/diffusers_convert.py holds VAE and text-encoder key conversions, including Diffusers VAE block names and SD2/OpenCLIP QKV packing.

Architecture classification is split between structural detection and model declarations. comfy/model_detection.py::detect_unet_config() inspects state-dict keys and tensor shapes to build an architecture config for SD1/2/XL, SD3/MMDiT, Flux, Hunyuan, Wan, Cosmos, PixArt, audio, video, 3D, and many newer model families. model_config_from_unet_config() then walks comfy.supported_models.models and uses supported_models_base.BASE.matches() to choose a concrete config class. Each class in comfy/supported_models.py declares the stable traits for that family: unet_config match keys, latent format, sampling settings, memory-usage factor, supported inference dtypes, state-dict rewrites, and the model_base subclass to instantiate. comfy/model_base.py turns those declarations into executable diffusion models: BaseModel creates the underlying LDM/DiT module with the selected ops/dtype, owns latent format conversion, builds model-specific conditioning in extra_conds(), estimates inference memory, and saves checkpoints by recombining UNet/CLIP/VAE state dicts through the config class.

Text encoder and VAE loading are also centralized in comfy/sd.py. load_clip() reads one or more text-encoder files, preserves embedded tokenizer JSON metadata, and calls load_text_encoder_state_dicts(), whose detect_te_model() classifies CLIP, T5, LLaMA/Qwen/Gemma/Mistral/Qwen-VL, and other text encoders from keys and tensor shapes. The resulting CLIP object creates the tokenizer/model pair, chooses load/offload devices and dtype with model_management.text_encoder_*(), wraps the text encoder in a patcher, and calls model_management.load_models_gpu() before encoding. VAE.__init__() recognizes many VAE variants from state-dict signatures, normalizes Diffusers VAE keys when needed, sets latent/channel/upscale/downscale properties and memory estimators, then creates a patcher so encode/decode calls can be loaded/offloaded through the same lifecycle machinery.

All runnable model objects are exposed to execution as patchers. comfy/model_patcher.py::ModelPatcher owns model weight patches, object patches, attention/transformer patch hooks, LoRA/weight wrappers, injected forward hooks, callbacks, attachments, and additional nested models. Its clone() method shares the underlying module while copying patch metadata; deepclone_multigpu() uses cached_patcher_init factories registered by comfy/sd.py loaders to create fresh per-device model copies for MultiGPU rather than deep-copying already-patched modules. LoRA loading in comfy/sd.py::load_lora_for_models() and load_bypass_lora_for_models() maps LoRA keys to model/CLIP state dict keys, clones patchers, and either registers regular weight patches through add_patches() or installs bypass adapter injections.

Model lifecycle and device placement are controlled by comfy/model_management.py. It detects the active accelerator backend (CUDA/ROCm, XPU, NPU, MLU, MPS, CPU), computes VRAM state from CLI flags, chooses dtypes (unet_dtype(), text_encoder_dtype(), vae_dtype()), chooses initial/offload devices, enables attention backends, and maintains current_loaded_models. The main entry point is load_models_gpu(models, memory_required=...): it deduplicates requested patchers and nested patch models, unloads incompatible clones, estimates required memory per device, calls free_memory() to partially or fully offload other models, and then uses each LoadedModel.model_load() to call the patcher’s partially_load(). free_memory() and LoadedModel.model_unload() handle smart-memory eviction, partial low-VRAM unloads, pin eviction, cache emptying, and stale weakref cleanup.

ModelPatcher has two loading modes. The base ModelPatcher in comfy/model_patcher.py can fully patch/load modules to load_device or leave modules in low-VRAM mode with LowVramPatch callbacks attached to comfy_cast_weights modules; it backs up original weights, applies LoRA/hook patches on demand, pins offloaded weights when possible, and restores everything in unpatch_model()/detach(). The dynamic path (CoreModelPatcher resolves to the dynamic implementation when enabled) uses ModelPatcherDynamic, per-device dynamic_pins, and Aimdo VBAR virtual GPU memory. Dynamic loading keeps weights in host buffers/file-backed storage, faults them into VRAM on demand, tracks active/current-prompt pins, can provide writable model scopes, and participates in RAM/pin-pressure eviction through model_management.free_pins() and partially_unload_ram().

Prefetching and pinned-memory support make dynamic loading viable. comfy/model_prefetch.py builds per-forward prefetch queues when transformer_options["prefetch_dynamic_vbars"] is set by BaseModel._apply_model() for dynamic patchers. prefetch_queue_pop() pins the next module group with pin_modules(), coordinates offload/data streams from model_management.py, optionally records Aimdo malloc graphs and CUDA graphs, and cleans prefetched modules after use. comfy/pinned_memory.py allocates per-model host-buffer pins, registers them with CUDA host registration, balances pins by size/priority buckets, and can steal lower-priority pins under pressure. comfy/memory_management.py provides the low-level layout helpers used by dynamic/prefetch code: file-slice reads/writes for disk-backed tensors, byte-aligned gathered layouts for quantized tensors, VBAR-aligned size calculation, and a callback hook for external RAM-cache release.

Host and storage policy feed into the same memory decisions. comfy/system_memory.py wraps psutil with cgroup-aware total/available RAM, so container limits constrain pin and cache budgets. comfy/storage.py annotates loaded tensors with source paths and decides whether their storage is “fast disk” from CLI flags or Linux/Windows NVMe detection; patchers receive this via comfy.storage.state_dict_fast_disk() and dynamic pins use it to change pin/register behavior. comfy/anonymous_file.py creates memfd-backed or Windows sparse temporary files for anonymous tensor backing. cuda_malloc.py runs before importing torch in normal startup paths and sets PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync when CLI flags, torch CUDA version, and GPU blacklist allow it.

Multi-GPU support is additive on top of patchers. comfy/multigpu.py::create_multigpu_deepclones() clones the primary patcher, selects extra torch devices from model_management.get_all_torch_devices(), reuses eligible already-loaded multigpu clones when requested, or calls ModelPatcher.deepclone_multigpu() for fresh per-device copies. The clones are stored as additional_models["multigpu"]; GPUOptionsGroup.register() writes relative-speed options into model.model_options, and load_balance_devices() splits work across devices. MultiGPUThreadPool keeps one worker thread per extra device and calls model_prefetch.cleanup_malloc_graph() when workers exit.

Coupling

  • If a new model family or checkpoint key layout is added in comfy/model_detection.py::detect_unet_config(), also add or update the matching class in comfy/supported_models.py; otherwise model_config_from_unet_config() will log no match and comfy/sd.py::load_state_dict_guess_config() will reject the checkpoint even though detection found a config.
  • If a supported_models.py class changes vae_key_prefix, text_encoder_key_prefix, clip_vision_prefix, or its process_*_state_dict() methods, verify comfy/sd.py::load_state_dict_guess_config() and comfy/model_base.py::state_dict_for_saving() still round-trip the same keys; load and save paths use the same config object but in opposite directions.
  • If a loader in comfy/sd.py starts producing a patcher that can be selected onto another GPU/device, set cached_patcher_init on that patcher. comfy/model_patcher.py::deepclone_multigpu() relies on this factory to create clean multigpu clones, and will raise if it is missing.
  • If ModelPatcher clone semantics change in comfy/model_patcher.py, verify comfy/model_management.py::load_models_gpu() clone eviction and comfy/multigpu.py::create_multigpu_deepclones() still treat shared-module clones and deep multigpu clones differently; lifecycle code unloads same-module clones before loading a replacement.
  • If BaseModel._apply_model() or transformer modules change how transformer_options are passed, verify comfy/model_prefetch.py::make_prefetch_queue() still sees prefetch_dynamic_vbars for dynamic patchers; without that flag dynamic VBAR prefetching silently drops to synchronous faults.
  • If pin-budget logic changes in comfy/model_management.py, check comfy/pinned_memory.py, comfy/model_patcher.py::ModelPatcherDynamic, and comfy/model_prefetch.py; they share the same dynamic_pins tuple layout, TOTAL_PINNED_MEMORY accounting, and active/current-prompt eviction tiers.
  • If file-backed tensor metadata changes in comfy/memory_management.py, update both readers and writers: model_management.cast_to_gathered() reads tensor file slices into gathered buffers, pinned_memory.flush_pin() writes dirty pins back, and anonymous_file.py supplies the anonymous file handles these helpers expect.
  • If folder aliases or folder names change in folder_paths.py, check node loaders that pass folder type strings (checkpoints, vae, text_encoders, diffusion_models, diffusers, loras); the file intentionally maps only legacy unet and clip names, so new aliases must be added explicitly.
  • If dtype selection changes in comfy/model_management.py, verify comfy/sd.py::load_state_dict_guess_config(), load_diffusion_model_state_dict(), CLIP.__init__(), and VAE.__init__() still agree on supported dtypes and manual-cast behavior; model configs may limit supported_inference_dtypes, and CLIP/VAE have separate dtype paths.

Risks

  • comfy/model_detection.py::detect_unet_config() and comfy/supported_models.py are shape/key-signature driven. A new checkpoint whose keys overlap an earlier branch can be classified as the wrong model family, which then instantiates the wrong model_base class and produces missing/unexpected weights in BaseModel.load_model_weights().
  • comfy/model_patcher.py::ModelPatcherDynamic.register_load_device() must be called whenever an existing dynamic patcher is retargeted to a new load device. Its comment calls out Select{Model,CLIP,VAE}Device; missing registration causes partially_unload_ram() to read a nonexistent dynamic_pins[device] entry.
  • comfy/model_management.py global state (current_loaded_models, stream buffers, pinned-memory totals, dirty mmaps) is shared across all execution. If a new path loads/unloads models without going through load_models_gpu()/free_memory()/reset_cast_buffers(), stale patched weights, pinned registrations, or dirty mmap bounce state can persist into later prompts.
  • comfy/model_prefetch.py::cleanup_prefetched_modules() explicitly deletes _prefetch markers even when unpinning fails because stale markers make later casts read old transfer state. Any new prefetch error path must preserve that cleanup behavior or dynamic models can reuse invalid VRAM/host-buffer state.
  • comfy/sd.py::VAE.__init__() contains many architecture checks keyed by specific tensor names and shapes. Adding a VAE branch in the wrong order can shadow a more specific branch and set incorrect latent channels, temporal/spatial ratios, or memory estimators, breaking downstream encode/decode sizing.