Introducing a new way to review your PRs.

Read the announcement

comfy-org / comfyui

The most powerful and modular diffusion model GUI, api and backend with a graph/nodes interface. The fastest local inference engine in the world.

View on GitHub

13 documents · Updated Nov 19, 2026

docs / ldm-and-specialized-models

LDM and Specialized Models

Overview

This area contains ComfyUI's in-repo implementations of diffusion backbones, autoencoders, control networks, preview autoencoders, and a background-removal model. The core entry point is comfy/model_base.py: BaseModel.__init__ chooses an unet_model class and constructs it with the detected unet_config, selected comfy.ops operation class, device, and dtype. Classic latent-diffusion UNets use comfy/…/openaimodel.py; newer image/video/audio/3D families are separate packages under comfy/ldm/* and are still routed through BaseModel.apply_model so samplers, controls, wrappers, and LoRA machinery see one model interface.

The shared LDM primitives live under comfy/ldm/modules/ and comfy/ldm/models/. comfy/…/openaimodel.py defines the timestep-aware UNet stack (TimestepBlock, TimestepEmbedSequential, ResBlock, Upsample, Downsample, UNetModel) and has an important patch hook in forward_timestep_embed: transformer_options["patches"]["forward_timestep_embed_patch"] can replace layer execution, while transformer_options["transformer_index"] is incremented around SpatialTransformer and SpatialVideoTransformer calls. comfy/…/attention.py is the shared attention layer and backend switchboard: models call optimized_attention/optimized_attention_for_device, optionally with AttentionTensorContainer and per-module ComfyAttention, so xformers/flash/sage/comfy-kitchen backends and transformer_options["optimized_attention_override"] can be selected without changing each architecture. comfy/…/mmdit.py provides DiT/MMDiT building blocks (PatchEmbed, timestep/vector embedders, joint/dismantled blocks, positional embeddings) reused by SD3-like models and ControlNet variants. comfy/…/common_dit.py contains small but common DiT helpers such as patch-size padding and RMSNorm dispatch.

Autoencoder loading is concentrated in comfy/sd.py's VAE initialization. It identifies checkpoint families by state-dict keys and constructs classes from comfy/…/autoencoder.py and family-specific packages: legacy SD-style AutoencoderKL/AutoencodingEngine, 3D/video VAEs such as comfy/…/vae.py, comfy/…/vae.py, comfy/…/vae.py, comfy/…/vae2_2.py, comfy/ldm/hunyuan_video/*, plus audio/3D/specialized codecs such as comfy/ldm/mmaudio/vae/*, comfy/ldm/minimax/*, comfy/…/vae.py, and comfy/…/vae.py. The generic AutoencodingEngine in comfy/…/autoencoder.py instantiates encoder/decoder/regularizer objects from config strings through comfy/…/util.py, while AutoencodingEngineLegacy wraps the Stable Diffusion encoder/decoder with quant/post-quant convolutions and optional batched encode/decode.

The main architecture families under comfy/ldm/ are:

  • UNet and cascade families: comfy/…/openaimodel.py implements SD-era UNets, while comfy/…/stage_b.py and comfy/…/stage_c.py implement Stable Cascade stages with configurable sequences of convolution (C), timestep (T), attention (A), and feed-forward (F) blocks. These are selected by comfy/model_base.py and by VAE/control loading in comfy/sd.py and comfy/controlnet.py.
  • MMDiT/DiT image models: comfy/…/mmdit.py provides the SD3-style transformer base; packages such as comfy/ldm/pixart/, comfy/ldm/pixeldit/, comfy/ldm/lumina/, comfy/ldm/hydit/, comfy/ldm/aura/, comfy/ldm/hidream/, comfy/ldm/chroma/, comfy/ldm/qwen_image/, comfy/ldm/qwen_image21/, comfy/ldm/omnigen/, comfy/ldm/moss_image/, comfy/ldm/pan2/, and related image packages define model-specific patching, conditioning, and block layouts. comfy/model_base.py maps supported model detections to these classes and passes conditioning as c_crossattn, ADM/vector inputs, guidance, masks, or family-specific extras.
  • Flux-style dual-stream transformers: comfy/…/model.py implements a flow-matching transformer with image/text token streams, RoPE via EmbedND, double-stream and single-stream blocks from comfy/…/layers.py, guidance/vector embedders, reference-latent and multi-view helpers, and block replacement hooks through transformer_options["patches_replace"]["dit"]. comfy/model_base.py's Flux wrapper prepares concat/inpaint latents, cross-attention, guidance, reference latents, token masks, and UltraTex multi-view metadata before calling the model. comfy/ldm/flux3/, comfy/ldm/lens/, comfy/ldm/lightricks/, and several newer image packages follow the same general DiT/flow integration pattern.
  • Video DiT families: comfy/…/model.py adapts Flux/MMDiT ideas to video with token refiners, ByT5 mapping, optional vision/reference conditioning, and attention masks. comfy/…/model.py implements Wan video transformer variants with RoPE attention, text/image cross-attention classes, camera/VACE/HuMo/S2V variants, and many model-specific wrappers in sibling files such as comfy/…/ar_model.py, model_animate.py, model_animate2.py, model_vsr.py, and model_wandancer.py. comfy/…/model.py provides a general AdaLN-modulated video DiT using GeneralDITTransformerBlock, 3D positional embeddings, FPS conditioning, and image/video data-type handling. comfy/…/model.py implements CogVideoX's 3D transformer with text+video patch embedding, 3D sin/cos/RoPE position handling, AdaLN-zero blocks, and optional OFS conditioning surfaced by comfy/model_base.py.
  • Audio, music, 3D, detection, and utility models: Packages such as comfy/ldm/audio/, comfy/ldm/mmaudio/, comfy/ldm/minimax_music/, comfy/ldm/yue2/, comfy/ldm/ace/, comfy/ldm/trellis2/, comfy/ldm/triposplat/, comfy/ldm/hunyuan3d/, comfy/ldm/hunyuan3dv2_1/, comfy/ldm/sam3/, comfy/ldm/sam3d_body/, comfy/ldm/rt_detr/, comfy/ldm/raft/, comfy/ldm/moge/, and comfy/ldm/depth_anything_3/ are specialized backbones or codecs. They are still instantiated either by comfy/model_base.py for denoising-style models or by explicit loader branches in comfy/sd.py/nodes for codecs and auxiliary networks.

Control models are split between comfy/cldm/ and comfy/t2i_adapter/. comfy/…/cldm.py is the ControlNet implementation derived from the SD UNet: it mirrors input/middle block structure, encodes the hint through input_hint_block, emits zero-conv control tensors, and supports union ControlNet type embeddings using comfy/…/control_types.py. comfy/…/mmdit.py subclasses the shared MMDiT and adds per-joint-block linear control heads plus a separate pos_embed_input for the hint latent. comfy/…/dit_embedder.py implements SD3.5-style control embedding with dismantled transformer blocks and repeated control outputs. comfy/controlnet.py is the loader/orchestrator: it detects checkpoint formats, builds these CLDM classes with the same dtype/manual-cast operation selection as the main model, wraps them in ControlNet/ControlNetSD35, and routes fallback T2I-Adapter checkpoints into comfy/…/adapter.py. The T2I adapter returns dictionaries shaped like control residuals (input, optionally middle) after pixel-unshuffle/downsample stages; comfy/controlnet.py caches and merges those features during sampling.

Tiny preview/approximation autoencoders live in comfy/taesd/. comfy/…/taesd.py implements image TAESD encoders/decoders for 4-channel SD-style latents, 64-channel Qwen Image 2.1 RGBA 16x latents, and 128-channel Flux2-style latent reshaping. comfy/…/taehv.py implements temporal tiny autoencoders for HunyuanVideo/Wan/LTX/MiniMax-style latents using memory blocks plus temporal pooling/growing; it can run time steps in parallel or stream them with per-block memory. latent_preview.py uses TAESD decoder files from models/vae_approx when LatentPreviewMethod.TAESD is selected, and comfy/sd.py can also load TAESD checkpoints as full VAEs when it detects taesd_decoder.1.weight.

Background removal is an auxiliary model path under comfy/background_removal/ plus comfy/bg_removal_model.py. comfy_extras/nodes_bg_removal.py exposes LoadBackgroundRemovalModel and RemoveBackground nodes; the loader reads files from the background_removal model folder, comfy/bg_removal_model.py recognizes BiRefNet state dicts by bb.layers.1.blocks.0.attn.relative_position_index, loads comfy/…/birefnet.json, constructs comfy/…/birefnet.py's BiRefNet, and wraps it in CoreModelPatcher. BackgroundRemovalModel.encode_image preprocesses input with CLIP-style image normalization, runs one image at a time for batched inputs, bicubic-resizes the sigmoid output back to the source size, and returns a (B,H,W) foreground mask.

Routing / covers

  • Use this doc for comfy/ldm/** diffusion backbones, DiT/UNet primitives, family-specific VAEs/codecs, and specialized LDM-adjacent architectures.
  • Use this doc for comfy/cldm/** ControlNet model classes and comfy/t2i_adapter/** T2I/style adapter architectures; the higher-level loading/orchestration is in comfy/controlnet.py but depends on these covered classes.
  • Use this doc for comfy/taesd/** TAESD/TAEHV preview or tiny VAE implementations and their integration points in latent_preview.py and comfy/sd.py.
  • Use this doc for comfy/background_removal/** and comfy/bg_removal_model.py; the public nodes are in comfy_extras/nodes_bg_removal.py.

Coupling

  • If a model class under comfy/ldm/** changes its forward signature or expected condition names, update the corresponding wrapper in comfy/model_base.py; BaseModel._apply_model and family wrappers such as Flux, HunyuanVideo, WAN21, CosmosVideo, and CogVideoX are responsible for translating sampler/node inputs into the kwargs the architecture consumes.
  • If a VAE or codec under comfy/ldm/** or comfy/taesd/** changes latent channel count, temporal/spatial compression, output channels, padding rules, or encode/decode memory behavior, update the detection branch in comfy/sd.py; that file hard-codes state-dict keys, latent_channels, latent_dim, upscale/downscale formulas, process-in/out functions, and memory estimators for each covered autoencoder family.
  • If comfy/…/attention.py changes the optimized_attention, AttentionTensorContainer, or ComfyAttention contracts, verify Flux/Wan/HunyuanVideo/CogVideo/Cosmos/BiRefNet call sites. Many covered architectures pass containers or preferred_attention objects so backend overrides and low-precision attention policy can silently break if the wrapper protocol changes.
  • If transformer_options patch keys change, audit both comfy/…/openaimodel.py and DiT-family packages such as comfy/…/model.py and comfy/…/model.py; LoRA/control/custom-node integrations rely on keys like patches, patches_replace, attn1_patch, forward_timestep_embed_patch, block_index, and transformer_index being propagated consistently.
  • If ControlNet state-dict key detection changes in comfy/controlnet.py, update the covered architectures in comfy/…/cldm.py, comfy/…/mmdit.py, comfy/…/dit_embedder.py, and comfy/…/adapter.py together. The loader chooses architectures from checkpoint keys and then expects their output dictionaries/lists to match ControlBase.control_merge.
  • If adding a new background-removal architecture, update both BG_REMOVAL_MODELS and detection/config selection in comfy/bg_removal_model.py; comfy_extras/nodes_bg_removal.py only calls load() and will reject files that return None.

Risks

  • comfy/…/attention.py is shared by nearly every covered transformer and by comfy/…/birefnet.py. A backend-specific behavior change can affect image, video, control, and background-removal models at once; for example, Wan image cross-attention in comfy/…/model.py explicitly disables low-precision attention for a small image-attention path because Sage attention can produce NaNs there.
  • VAE family detection in comfy/sd.py is fragile because it is keyed on checkpoint tensor names and sometimes tensor shapes. If two covered autoencoders share keys such as decoder.conv_in.conv.weight or Wan decoder/head layouts, the wrong branch can instantiate a model with incorrect temporal compression or channel counts, causing latent shape mismatches later in sampling/decoding.
  • comfy/…/taehv.py's streaming mode keeps per-block temporal memory and trims frames according to t_upscale. If latent channel mappings or temporal down/upscale flags are changed without matching frames_to_trim and single-frame special cases, video previews/full tiny-VAE decoding can drop or duplicate frames.
  • comfy/bg_removal_model.py mutates the loaded state dict after load by popping keys not reported as unexpected. Code that reuses the same state dict object after calling load_background_removal_model can observe missing weights.