LDM and Specialized Models
Overview
This area contains ComfyUI's in-repo implementations of diffusion backbones, autoencoders, control networks, preview autoencoders, and a background-removal model. The core entry point is comfy/model_base.py: BaseModel.__init__ chooses an unet_model class and constructs it with the detected unet_config, selected comfy.ops operation class, device, and dtype. Classic latent-diffusion UNets use comfy/…/openaimodel.py; newer image/video/audio/3D families are separate packages under comfy/ldm/* and are still routed through BaseModel.apply_model so samplers, controls, wrappers, and LoRA machinery see one model interface.
The shared LDM primitives live under comfy/ldm/modules/ and comfy/ldm/models/. comfy/…/openaimodel.py defines the timestep-aware UNet stack (TimestepBlock, TimestepEmbedSequential, ResBlock, Upsample, Downsample, UNetModel) and has an important patch hook in forward_timestep_embed: transformer_options["patches"]["forward_timestep_embed_patch"] can replace layer execution, while transformer_options["transformer_index"] is incremented around SpatialTransformer and SpatialVideoTransformer calls. comfy/…/attention.py is the shared attention layer and backend switchboard: models call optimized_attention/optimized_attention_for_device, optionally with AttentionTensorContainer and per-module ComfyAttention, so xformers/flash/sage/comfy-kitchen backends and transformer_options["optimized_attention_override"] can be selected without changing each architecture. comfy/…/mmdit.py provides DiT/MMDiT building blocks (PatchEmbed, timestep/vector embedders, joint/dismantled blocks, positional embeddings) reused by SD3-like models and ControlNet variants. comfy/…/common_dit.py contains small but common DiT helpers such as patch-size padding and RMSNorm dispatch.
Autoencoder loading is concentrated in comfy/sd.py's VAE initialization. It identifies checkpoint families by state-dict keys and constructs classes from comfy/…/autoencoder.py and family-specific packages: legacy SD-style AutoencoderKL/AutoencodingEngine, 3D/video VAEs such as comfy/…/vae.py, comfy/…/vae.py, comfy/…/vae.py, comfy/…/vae2_2.py, comfy/ldm/hunyuan_video/*, plus audio/3D/specialized codecs such as comfy/ldm/mmaudio/vae/*, comfy/ldm/minimax/*, comfy/…/vae.py, and comfy/…/vae.py. The generic AutoencodingEngine in comfy/…/autoencoder.py instantiates encoder/decoder/regularizer objects from config strings through comfy/…/util.py, while AutoencodingEngineLegacy wraps the Stable Diffusion encoder/decoder with quant/post-quant convolutions and optional batched encode/decode.
The main architecture families under comfy/ldm/ are:
- UNet and cascade families: comfy/…/openaimodel.py implements SD-era UNets, while comfy/…/stage_b.py and comfy/…/stage_c.py implement Stable Cascade stages with configurable sequences of convolution (
C), timestep (T), attention (A), and feed-forward (F) blocks. These are selected by comfy/model_base.py and by VAE/control loading in comfy/sd.py and comfy/controlnet.py. - MMDiT/DiT image models: comfy/…/mmdit.py provides the SD3-style transformer base; packages such as
comfy/ldm/pixart/,comfy/ldm/pixeldit/,comfy/ldm/lumina/,comfy/ldm/hydit/,comfy/ldm/aura/,comfy/ldm/hidream/,comfy/ldm/chroma/,comfy/ldm/qwen_image/,comfy/ldm/qwen_image21/,comfy/ldm/omnigen/,comfy/ldm/moss_image/,comfy/ldm/pan2/, and related image packages define model-specific patching, conditioning, and block layouts. comfy/model_base.py maps supported model detections to these classes and passes conditioning asc_crossattn, ADM/vector inputs, guidance, masks, or family-specific extras. - Flux-style dual-stream transformers: comfy/…/model.py implements a flow-matching transformer with image/text token streams, RoPE via
EmbedND, double-stream and single-stream blocks from comfy/…/layers.py, guidance/vector embedders, reference-latent and multi-view helpers, and block replacement hooks throughtransformer_options["patches_replace"]["dit"]. comfy/model_base.py'sFluxwrapper prepares concat/inpaint latents, cross-attention, guidance, reference latents, token masks, and UltraTex multi-view metadata before calling the model.comfy/ldm/flux3/,comfy/ldm/lens/,comfy/ldm/lightricks/, and several newer image packages follow the same general DiT/flow integration pattern. - Video DiT families: comfy/…/model.py adapts Flux/MMDiT ideas to video with token refiners, ByT5 mapping, optional vision/reference conditioning, and attention masks. comfy/…/model.py implements Wan video transformer variants with RoPE attention, text/image cross-attention classes, camera/VACE/HuMo/S2V variants, and many model-specific wrappers in sibling files such as comfy/…/ar_model.py, model_animate.py, model_animate2.py, model_vsr.py, and model_wandancer.py. comfy/…/model.py provides a general AdaLN-modulated video DiT using
GeneralDITTransformerBlock, 3D positional embeddings, FPS conditioning, and image/video data-type handling. comfy/…/model.py implements CogVideoX's 3D transformer with text+video patch embedding, 3D sin/cos/RoPE position handling, AdaLN-zero blocks, and optional OFS conditioning surfaced by comfy/model_base.py. - Audio, music, 3D, detection, and utility models: Packages such as
comfy/ldm/audio/,comfy/ldm/mmaudio/,comfy/ldm/minimax_music/,comfy/ldm/yue2/,comfy/ldm/ace/,comfy/ldm/trellis2/,comfy/ldm/triposplat/,comfy/ldm/hunyuan3d/,comfy/ldm/hunyuan3dv2_1/,comfy/ldm/sam3/,comfy/ldm/sam3d_body/,comfy/ldm/rt_detr/,comfy/ldm/raft/,comfy/ldm/moge/, andcomfy/ldm/depth_anything_3/are specialized backbones or codecs. They are still instantiated either by comfy/model_base.py for denoising-style models or by explicit loader branches in comfy/sd.py/nodes for codecs and auxiliary networks.
Control models are split between comfy/cldm/ and comfy/t2i_adapter/. comfy/…/cldm.py is the ControlNet implementation derived from the SD UNet: it mirrors input/middle block structure, encodes the hint through input_hint_block, emits zero-conv control tensors, and supports union ControlNet type embeddings using comfy/…/control_types.py. comfy/…/mmdit.py subclasses the shared MMDiT and adds per-joint-block linear control heads plus a separate pos_embed_input for the hint latent. comfy/…/dit_embedder.py implements SD3.5-style control embedding with dismantled transformer blocks and repeated control outputs. comfy/controlnet.py is the loader/orchestrator: it detects checkpoint formats, builds these CLDM classes with the same dtype/manual-cast operation selection as the main model, wraps them in ControlNet/ControlNetSD35, and routes fallback T2I-Adapter checkpoints into comfy/…/adapter.py. The T2I adapter returns dictionaries shaped like control residuals (input, optionally middle) after pixel-unshuffle/downsample stages; comfy/controlnet.py caches and merges those features during sampling.
Tiny preview/approximation autoencoders live in comfy/taesd/. comfy/…/taesd.py implements image TAESD encoders/decoders for 4-channel SD-style latents, 64-channel Qwen Image 2.1 RGBA 16x latents, and 128-channel Flux2-style latent reshaping. comfy/…/taehv.py implements temporal tiny autoencoders for HunyuanVideo/Wan/LTX/MiniMax-style latents using memory blocks plus temporal pooling/growing; it can run time steps in parallel or stream them with per-block memory. latent_preview.py uses TAESD decoder files from models/vae_approx when LatentPreviewMethod.TAESD is selected, and comfy/sd.py can also load TAESD checkpoints as full VAEs when it detects taesd_decoder.1.weight.
Background removal is an auxiliary model path under comfy/background_removal/ plus comfy/bg_removal_model.py. comfy_extras/nodes_bg_removal.py exposes LoadBackgroundRemovalModel and RemoveBackground nodes; the loader reads files from the background_removal model folder, comfy/bg_removal_model.py recognizes BiRefNet state dicts by bb.layers.1.blocks.0.attn.relative_position_index, loads comfy/…/birefnet.json, constructs comfy/…/birefnet.py's BiRefNet, and wraps it in CoreModelPatcher. BackgroundRemovalModel.encode_image preprocesses input with CLIP-style image normalization, runs one image at a time for batched inputs, bicubic-resizes the sigmoid output back to the source size, and returns a (B,H,W) foreground mask.
Routing / covers
- Use this doc for
comfy/ldm/**diffusion backbones, DiT/UNet primitives, family-specific VAEs/codecs, and specialized LDM-adjacent architectures. - Use this doc for
comfy/cldm/**ControlNet model classes andcomfy/t2i_adapter/**T2I/style adapter architectures; the higher-level loading/orchestration is in comfy/controlnet.py but depends on these covered classes. - Use this doc for
comfy/taesd/**TAESD/TAEHV preview or tiny VAE implementations and their integration points in latent_preview.py and comfy/sd.py. - Use this doc for
comfy/background_removal/**and comfy/bg_removal_model.py; the public nodes are in comfy_extras/nodes_bg_removal.py.
Coupling
- If a model class under
comfy/ldm/**changes itsforwardsignature or expected condition names, update the corresponding wrapper in comfy/model_base.py;BaseModel._apply_modeland family wrappers such asFlux,HunyuanVideo,WAN21,CosmosVideo, andCogVideoXare responsible for translating sampler/node inputs into the kwargs the architecture consumes. - If a VAE or codec under
comfy/ldm/**orcomfy/taesd/**changes latent channel count, temporal/spatial compression, output channels, padding rules, or encode/decode memory behavior, update the detection branch in comfy/sd.py; that file hard-codes state-dict keys,latent_channels,latent_dim, upscale/downscale formulas, process-in/out functions, and memory estimators for each covered autoencoder family. - If comfy/…/attention.py changes the
optimized_attention,AttentionTensorContainer, orComfyAttentioncontracts, verify Flux/Wan/HunyuanVideo/CogVideo/Cosmos/BiRefNet call sites. Many covered architectures pass containers orpreferred_attentionobjects so backend overrides and low-precision attention policy can silently break if the wrapper protocol changes. - If
transformer_optionspatch keys change, audit both comfy/…/openaimodel.py and DiT-family packages such as comfy/…/model.py and comfy/…/model.py; LoRA/control/custom-node integrations rely on keys likepatches,patches_replace,attn1_patch,forward_timestep_embed_patch,block_index, andtransformer_indexbeing propagated consistently. - If ControlNet state-dict key detection changes in comfy/controlnet.py, update the covered architectures in comfy/…/cldm.py, comfy/…/mmdit.py, comfy/…/dit_embedder.py, and comfy/…/adapter.py together. The loader chooses architectures from checkpoint keys and then expects their output dictionaries/lists to match
ControlBase.control_merge. - If adding a new background-removal architecture, update both
BG_REMOVAL_MODELSand detection/config selection in comfy/bg_removal_model.py; comfy_extras/nodes_bg_removal.py only callsload()and will reject files that returnNone.
Risks
- comfy/…/attention.py is shared by nearly every covered transformer and by comfy/…/birefnet.py. A backend-specific behavior change can affect image, video, control, and background-removal models at once; for example, Wan image cross-attention in comfy/…/model.py explicitly disables low-precision attention for a small image-attention path because Sage attention can produce NaNs there.
- VAE family detection in comfy/sd.py is fragile because it is keyed on checkpoint tensor names and sometimes tensor shapes. If two covered autoencoders share keys such as
decoder.conv_in.conv.weightor Wan decoder/head layouts, the wrong branch can instantiate a model with incorrect temporal compression or channel counts, causing latent shape mismatches later in sampling/decoding. - comfy/…/taehv.py's streaming mode keeps per-block temporal memory and trims frames according to
t_upscale. If latent channel mappings or temporal down/upscale flags are changed without matchingframes_to_trimand single-frame special cases, video previews/full tiny-VAE decoding can drop or duplicate frames. - comfy/bg_removal_model.py mutates the loaded state dict after load by popping keys not reported as unexpected. Code that reuses the same state dict object after calling
load_background_removal_modelcan observe missing weights.