Introducing a new way to review your PRs.

Read the announcement

comfy-org / comfyui

The most powerful and modular diffusion model GUI, api and backend with a graph/nodes interface. The fastest local inference engine in the world.

View on GitHub

13 documents · Updated Nov 19, 2026

docs / text-image-audio-encoders

Text, Image, and Audio Encoders

Routing / Covers

Use this doc for changes under comfy/clip_model.py, comfy/clip_vision.py, comfy/clip_config*.json, comfy/clip_vision_config*.json, comfy/sd1_clip.py, comfy/sdxl_clip.py, comfy/sd1_tokenizer/**, comfy/text_encoders/**, comfy/image_encoders/**, comfy/audio_encoders/**, and comfy/audio.py. It covers CLIP/T5/Llama-family text encoders and tokenizers, CLIP/SigLIP/DINO vision encoders, multimodal token insertion for image/video prompts, and audio feature/transcription encoders.

Overview

The common text-conditioning contract is centered on comfy/sd1_clip.py. SDTokenizer turns prompt text into batches of (token, weight) pairs, including prompt-weight syntax such as (text:1.2), textual inversion references prefixed with embedding:, tokenizer override data from tokenizer_data, padding/min-length controls, and optional word IDs. SDClipModel receives those token batches through encode_token_weights(), calls process_tokens() to build token embeddings and attention masks, applies per-token weights by comparing against an empty prompt embedding, and returns conditioning tensors on comfy.model_management.intermediate_device(). SD1Tokenizer and SD1ClipModel wrap a single named encoder (normally clip_l) so higher layers can treat tokenization and encoding as keyed dictionaries rather than direct model calls.

comfy/clip_model.py implements the base CLIP architecture used by SD1/SDXL text encoders and by CLIP-like vision encoders. CLIPTextModel_ builds token and position embeddings, applies a causal attention mask, returns the requested hidden/intermediate layer, and pools at EOS or the last unmasked token. CLIPTextModel adds the projected pooled output. The same file also implements CLIPVision, CLIPVisionModelProjection, SigLIP/SigLIP2 image preprocessing, flexible SigLIP2 patch-resolution calculation, and the optional LlavaProjector used when a vision config declares projector_type: "llava3".

SDXL composes two CLIP text encoders in comfy/sdxl_clip.py: SDXLTokenizer tokenizes the same prompt for clip_l and clip_g; SDXLClipModel.encode_token_weights() encodes both, cuts sequences to a common token length, concatenates hidden states on the channel dimension, and returns clip_g pooled output. SDXLRefinerClipModel and StableCascadeClipModel reuse the same clip_g config in comfy/clip_config_bigg.json, but differ in selected hidden layer, pad token, and attention-mask behavior.

The newer text encoder families in comfy/text_encoders/** mostly subclass the SDTokenizer/SDClipModel contract instead of inventing a separate interface:

  • comfy/…/sd3_clip.py combines CLIP-L, CLIP-G, and T5-XXL. SD3Tokenizer returns l, g, and t5xxl token streams; SD3ClipModel.encode_token_weights() pads CLIP hidden states up to T5 width, concatenates CLIP and T5 sequences, returns a 2048-wide CLIP pooled vector, and can propagate the T5 attention mask in extra.
  • comfy/…/flux.py combines CLIP-L with T5-XXL for Flux, and also defines Llama/Qwen-style text encoders for Flux2/Klein/Flux3. Those models use chat templates and disable prompt weighting where the underlying tokenizer is not CLIP-like.
  • comfy/…/t5.py implements encoder-only T5/UMT5 with relative attention bias, RMS-style T5LayerNorm, gated or non-gated feed-forward blocks, and optional attention masks. Config JSON such as comfy/…/t5_config_xxl.json and comfy/…/umt5_config_base.json provides dimensions, vocab size, and gated-activation settings.
  • comfy/…/llama.py provides Llama/Gemma/Qwen transformer backbones, generation support, KV-cache variants, RoPE/MRoPE handling, optional LM heads, and multimodal Gemma projection. Wrappers in files such as comfy/…/qwen_image.py, comfy/…/qwen3vl.py, comfy/…/qwen35.py, comfy/…/lt.py, and comfy/…/lumina2.py adapt these backbones to the SDClipModel interface.
  • comfy/…/spiece_tokenizer.py wraps SentencePiece from either a model file or serialized bytes, with explicit special-token splitting. comfy/…/ace.py shows a nonstandard tokenizer/model pair: AceT5Tokenizer returns lyric BPE tokens and UMT5 prompt tokens, while AceT5Model.encode_token_weights() returns UMT5 conditioning plus conditioning_lyrics in extra.

Multimodal text prompts are inserted before the transformer, not appended afterward. SDClipModel.process_tokens() treats non-integer token entries as embedding descriptors; if the underlying transformer exposes preprocess_embed(), the descriptor is converted into one or more embeddings and spliced into the token embedding sequence, while embeds_info records index, size, type, and auxiliary data. Qwen/Gemma tokenizers mark placeholder tokens with descriptors: comfy/…/qwen_image.py replaces <|image_pad|> with {"type": "image", "data": ...}, comfy/…/qwen3vl.py does the same for one or more images, comfy/…/qwen35.py also supports video placeholders, and comfy/…/lt.py maps Gemma <image_soft_token> to resized images. The corresponding transformers (Qwen3VL.preprocess_embed() in comfy/…/qwen3vl.py, Qwen35.preprocess_embed() in comfy/…/qwen35.py, and Gemma classes in comfy/…/llama.py) run their vision towers, return inserted embeddings, and use embeds_info to build MRoPE position IDs or DeepStack visual injections during forward/generation.

Vision encoder loading lives in comfy/clip_vision.py. ClipVisionModel accepts a JSON config path or dict, selects an implementation from IMAGE_ENCODERS, creates a CoreModelPatcher, and exposes encode_image(). The encode path loads the model to the text-encoder device, preprocesses images with clip_preprocess() or siglip2_preprocess() from comfy/clip_model.py, asks the model for either the penultimate hidden state or all hidden states, and returns an Output object with last_hidden_state, image_embeds, penultimate_hidden_states, optional all_hidden_states, image_sizes, and optional mm_projected. load_clipvision_from_sd() detects CLIP-ViT, SigLIP/SigLIP2, DINOv2, and DINOv3 checkpoints by state-dict keys, chooses a config such as comfy/clip_vision_config_g.json, comfy/clip_vision_config_vitl_336_llava.json, comfy/…/dino2_large.json, or comfy/…/dino3_large.json, loads with relaxed state-dict matching, and extracts bundled NAF upsampler weights under the naf. prefix.

The image encoder implementations under comfy/image_encoders/** are used both through comfy/clip_vision.py and by multimodal text backbones. comfy/…/dino2.py implements DINOv2 patch embeddings, interpolated square position grids, layer-scale transformer blocks, and Depth Anything 3 extensions (get_intermediate_layers_da3()) for multi-view inputs with alternating local/global attention, camera-token injection, 2D RoPE, and reference-view reordering. comfy/…/dino3.py implements DINOv3 ViT with cls/register tokens, patch RoPE, gated/non-gated MLPs, and forward_features() for spatial feature maps. comfy/…/naf.py implements the optional NAF feature upsampler loaded by clip_vision.py: an image-guided convolutional query encoder plus pure-PyTorch neighborhood cross-attention for upsampling low-resolution feature maps.

Audio feature encoders are loaded through comfy/…/audio_encoders.py. load_audio_encoder_from_sd() normalizes known state-dict prefixes, detects wav2vec2 by encoder.layer_norm.bias, Whisper by model.encoder.embed_positions.weight, and SheetSage2 by MERT/decoder keys, then builds an AudioEncoderModel or SheetSage2AudioEncoder. AudioEncoderModel.encode_audio() loads through a CoreModelPatcher, resamples with comfy/audio.py, forwards the model, and returns encoded_audio, encoded_audio_all_layers, and audio_samples. SheetSage2AudioEncoder.generate_abc() uses the same loaded model to transcribe windows and convert SheetSage events to ABC notation.

comfy/audio.py is the DSP support layer for those encoders and nodes: it provides sinc resampling, Slaney mel filterbanks, STFT-based mel spectrograms, and biquad filters. comfy/…/whisper.py uses MelSpectrogram for Whisper-large-v3-style log-mel features before a convolutional/transformer audio encoder. comfy/…/wav2vec2.py implements wav2vec2 convolutional feature extraction, positional conv embeddings, and transformer layers. comfy/…/mert2.py implements the ConvNeXt/Conformer MERT-v2 encoder used inside comfy/…/sheetsage2.py; SheetSage2 mixes MERT layers with a learned layer_weight, projects to decoder memory, and autoregressively emits score tokens under PromptGrammarState constraints.

For model loading and sampling, the important boundary is the small surface exposed by these files: tokenizers return keyed token batches; text encoders implement encode_token_weights() and sometimes generate(); vision encoders expose encode_image(); audio encoders expose encode_audio() or generate_abc(). Samplers and conditioning nodes can change selected CLIP layers or execution device by calling set_clip_options()/reset_clip_options() on the wrappers in comfy/sd1_clip.py, comfy/sdxl_clip.py, and the comfy/text_encoders/** wrappers, without needing to know each encoder family’s internal model class.

Coupling

  • If SDTokenizer.tokenize_with_weights() or SDClipModel.process_tokens() in comfy/sd1_clip.py changes the token tuple format, update multimodal tokenizers in comfy/…/qwen_image.py, comfy/…/qwen3vl.py, comfy/…/qwen35.py, and comfy/…/lt.py: they rely on replacing a token tuple’s first element with an embed descriptor while preserving the remaining weight/word-id fields.
  • If attention-mask shape or meaning changes in comfy/sd1_clip.py, verify T5/SD3/Flux/Qwen wrappers that set enable_attention_masks=True and return_attention_masks=True, especially comfy/…/sd3_clip.py, comfy/…/flux.py, comfy/…/qwen_image.py, and comfy/…/qwen3vl.py; several downstream extra["attention_mask"] values are flattened or sliced rather than recomputed.
  • If comfy/clip_model.py changes CLIP/SigLIP output tuple ordering, update comfy/clip_vision.py and comfy/sd1_clip.py: ClipVisionModel.encode_image() assumes (last_hidden, intermediate, projected, mm_projected) for vision, while SDClipModel.forward() assumes text outputs contain hidden, intermediate, projected pooled, and unprojected pooled values.
  • If a vision config JSON changes model_type, image_size, patch_size, image_mean, or image_std, check comfy/clip_vision.py detection and preprocessing. siglip2_vision_model is routed to siglip2_preprocess() while other CLIP/SigLIP/DINO paths use clip_preprocess() with config-provided normalization.
  • If DINO checkpoint key names change, update load_clipvision_from_sd() in comfy/clip_vision.py; it selects DINOv2/DINOv3 configs from exact keys such as encoder.layer.23.layer_scale2.lambda1, layer.0.mlp.gate_proj.weight, and layer.23.attention.o_proj.bias.
  • If Qwen multimodal image/video preprocessing changes in comfy/…/qwen_vl.py, verify preprocess_embed() and position-ID construction in comfy/…/qwen3vl.py and comfy/…/qwen35.py; MRoPE position IDs and DeepStack visual masks are derived from embeds_info sizes returned by SDClipModel.process_tokens().
  • If comfy/audio.resample() changes output length rounding, verify comfy/…/audio_encoders.py, comfy/…/whisper.py, and comfy/…/sheetsage2.py; audio sample counts, Whisper’s fixed 30-second crop/pad, and SheetSage2’s time-window stitching assume stable sample lengths after resampling.
  • If FixedKV in comfy/…/llama.py changes fields or semantics, verify comfy/…/sheetsage2.py: its decoder reuses FixedKV for graphable autoregressive cache updates even though it is not a text encoder.

Risks

  • comfy/sd1_clip.py silently ignores multimodal embeddings whose final dimension does not match the token embedding dimension, padding the sequence with masked pad embeddings instead. If a tokenizer/model pair uses the wrong embedding_size or a vision projector emits a mismatched width, image/video conditioning is dropped while tokenization still succeeds.
  • comfy/clip_vision.py state-dict detection is key-based and returns None for unknown layouts. A checkpoint that renames CLIP/SigLIP/DINO layer keys without changing architecture will fail to load as a vision encoder rather than falling back to a config.
  • comfy/…/dino2.py has explicit assertions around interpolated position-grid size and a comment noting that height/width scale-factor order breaks non-square DA3 paths if swapped. Changing interpolation math can break multi-view/non-square feature extraction even when square CLIP-vision tests pass.
  • comfy/…/qwen3vl.py and comfy/…/qwen35.py rely on placeholder token IDs (151655 for Qwen image pads and vocab-derived image/video pad IDs for Qwen3.5). If tokenizer assets under comfy/text_encoders/qwen25_tokenizer/** or comfy/text_encoders/qwen35_tokenizer/** drift from those IDs, image/video embeddings will not be inserted.
  • comfy/audio.py implements _LFilter by moving tensors through SciPy on CPU. Biquad operations are differentiable only through the custom backward wrapper and will incur CPU transfers; using them in hot GPU paths can become a performance bottleneck.
  • comfy/…/sheetsage2.py constrains generated score tokens with PromptGrammarState. If ScoreTokenizer ranges or token meanings are changed without updating the grammar, transcription may produce invalid events or stop early despite the model logits being valid.