Text, Image, and Audio Encoders
Routing / Covers
Use this doc for changes under comfy/clip_model.py, comfy/clip_vision.py, comfy/clip_config*.json, comfy/clip_vision_config*.json, comfy/sd1_clip.py, comfy/sdxl_clip.py, comfy/sd1_tokenizer/**, comfy/text_encoders/**, comfy/image_encoders/**, comfy/audio_encoders/**, and comfy/audio.py. It covers CLIP/T5/Llama-family text encoders and tokenizers, CLIP/SigLIP/DINO vision encoders, multimodal token insertion for image/video prompts, and audio feature/transcription encoders.
Overview
The common text-conditioning contract is centered on comfy/sd1_clip.py. SDTokenizer turns prompt text into batches of (token, weight) pairs, including prompt-weight syntax such as (text:1.2), textual inversion references prefixed with embedding:, tokenizer override data from tokenizer_data, padding/min-length controls, and optional word IDs. SDClipModel receives those token batches through encode_token_weights(), calls process_tokens() to build token embeddings and attention masks, applies per-token weights by comparing against an empty prompt embedding, and returns conditioning tensors on comfy.model_management.intermediate_device(). SD1Tokenizer and SD1ClipModel wrap a single named encoder (normally clip_l) so higher layers can treat tokenization and encoding as keyed dictionaries rather than direct model calls.
comfy/clip_model.py implements the base CLIP architecture used by SD1/SDXL text encoders and by CLIP-like vision encoders. CLIPTextModel_ builds token and position embeddings, applies a causal attention mask, returns the requested hidden/intermediate layer, and pools at EOS or the last unmasked token. CLIPTextModel adds the projected pooled output. The same file also implements CLIPVision, CLIPVisionModelProjection, SigLIP/SigLIP2 image preprocessing, flexible SigLIP2 patch-resolution calculation, and the optional LlavaProjector used when a vision config declares projector_type: "llava3".
SDXL composes two CLIP text encoders in comfy/sdxl_clip.py: SDXLTokenizer tokenizes the same prompt for clip_l and clip_g; SDXLClipModel.encode_token_weights() encodes both, cuts sequences to a common token length, concatenates hidden states on the channel dimension, and returns clip_g pooled output. SDXLRefinerClipModel and StableCascadeClipModel reuse the same clip_g config in comfy/clip_config_bigg.json, but differ in selected hidden layer, pad token, and attention-mask behavior.
The newer text encoder families in comfy/text_encoders/** mostly subclass the SDTokenizer/SDClipModel contract instead of inventing a separate interface:
- comfy/…/sd3_clip.py combines CLIP-L, CLIP-G, and T5-XXL.
SD3Tokenizerreturnsl,g, andt5xxltoken streams;SD3ClipModel.encode_token_weights()pads CLIP hidden states up to T5 width, concatenates CLIP and T5 sequences, returns a 2048-wide CLIP pooled vector, and can propagate the T5 attention mask inextra. - comfy/…/flux.py combines CLIP-L with T5-XXL for Flux, and also defines Llama/Qwen-style text encoders for Flux2/Klein/Flux3. Those models use chat templates and disable prompt weighting where the underlying tokenizer is not CLIP-like.
- comfy/…/t5.py implements encoder-only T5/UMT5 with relative attention bias, RMS-style
T5LayerNorm, gated or non-gated feed-forward blocks, and optional attention masks. Config JSON such as comfy/…/t5_config_xxl.json and comfy/…/umt5_config_base.json provides dimensions, vocab size, and gated-activation settings. - comfy/…/llama.py provides Llama/Gemma/Qwen transformer backbones, generation support, KV-cache variants, RoPE/MRoPE handling, optional LM heads, and multimodal Gemma projection. Wrappers in files such as comfy/…/qwen_image.py, comfy/…/qwen3vl.py, comfy/…/qwen35.py, comfy/…/lt.py, and comfy/…/lumina2.py adapt these backbones to the
SDClipModelinterface. - comfy/…/spiece_tokenizer.py wraps SentencePiece from either a model file or serialized bytes, with explicit special-token splitting. comfy/…/ace.py shows a nonstandard tokenizer/model pair:
AceT5Tokenizerreturns lyric BPE tokens and UMT5 prompt tokens, whileAceT5Model.encode_token_weights()returns UMT5 conditioning plusconditioning_lyricsinextra.
Multimodal text prompts are inserted before the transformer, not appended afterward. SDClipModel.process_tokens() treats non-integer token entries as embedding descriptors; if the underlying transformer exposes preprocess_embed(), the descriptor is converted into one or more embeddings and spliced into the token embedding sequence, while embeds_info records index, size, type, and auxiliary data. Qwen/Gemma tokenizers mark placeholder tokens with descriptors: comfy/…/qwen_image.py replaces <|image_pad|> with {"type": "image", "data": ...}, comfy/…/qwen3vl.py does the same for one or more images, comfy/…/qwen35.py also supports video placeholders, and comfy/…/lt.py maps Gemma <image_soft_token> to resized images. The corresponding transformers (Qwen3VL.preprocess_embed() in comfy/…/qwen3vl.py, Qwen35.preprocess_embed() in comfy/…/qwen35.py, and Gemma classes in comfy/…/llama.py) run their vision towers, return inserted embeddings, and use embeds_info to build MRoPE position IDs or DeepStack visual injections during forward/generation.
Vision encoder loading lives in comfy/clip_vision.py. ClipVisionModel accepts a JSON config path or dict, selects an implementation from IMAGE_ENCODERS, creates a CoreModelPatcher, and exposes encode_image(). The encode path loads the model to the text-encoder device, preprocesses images with clip_preprocess() or siglip2_preprocess() from comfy/clip_model.py, asks the model for either the penultimate hidden state or all hidden states, and returns an Output object with last_hidden_state, image_embeds, penultimate_hidden_states, optional all_hidden_states, image_sizes, and optional mm_projected. load_clipvision_from_sd() detects CLIP-ViT, SigLIP/SigLIP2, DINOv2, and DINOv3 checkpoints by state-dict keys, chooses a config such as comfy/clip_vision_config_g.json, comfy/clip_vision_config_vitl_336_llava.json, comfy/…/dino2_large.json, or comfy/…/dino3_large.json, loads with relaxed state-dict matching, and extracts bundled NAF upsampler weights under the naf. prefix.
The image encoder implementations under comfy/image_encoders/** are used both through comfy/clip_vision.py and by multimodal text backbones. comfy/…/dino2.py implements DINOv2 patch embeddings, interpolated square position grids, layer-scale transformer blocks, and Depth Anything 3 extensions (get_intermediate_layers_da3()) for multi-view inputs with alternating local/global attention, camera-token injection, 2D RoPE, and reference-view reordering. comfy/…/dino3.py implements DINOv3 ViT with cls/register tokens, patch RoPE, gated/non-gated MLPs, and forward_features() for spatial feature maps. comfy/…/naf.py implements the optional NAF feature upsampler loaded by clip_vision.py: an image-guided convolutional query encoder plus pure-PyTorch neighborhood cross-attention for upsampling low-resolution feature maps.
Audio feature encoders are loaded through comfy/…/audio_encoders.py. load_audio_encoder_from_sd() normalizes known state-dict prefixes, detects wav2vec2 by encoder.layer_norm.bias, Whisper by model.encoder.embed_positions.weight, and SheetSage2 by MERT/decoder keys, then builds an AudioEncoderModel or SheetSage2AudioEncoder. AudioEncoderModel.encode_audio() loads through a CoreModelPatcher, resamples with comfy/audio.py, forwards the model, and returns encoded_audio, encoded_audio_all_layers, and audio_samples. SheetSage2AudioEncoder.generate_abc() uses the same loaded model to transcribe windows and convert SheetSage events to ABC notation.
comfy/audio.py is the DSP support layer for those encoders and nodes: it provides sinc resampling, Slaney mel filterbanks, STFT-based mel spectrograms, and biquad filters. comfy/…/whisper.py uses MelSpectrogram for Whisper-large-v3-style log-mel features before a convolutional/transformer audio encoder. comfy/…/wav2vec2.py implements wav2vec2 convolutional feature extraction, positional conv embeddings, and transformer layers. comfy/…/mert2.py implements the ConvNeXt/Conformer MERT-v2 encoder used inside comfy/…/sheetsage2.py; SheetSage2 mixes MERT layers with a learned layer_weight, projects to decoder memory, and autoregressively emits score tokens under PromptGrammarState constraints.
For model loading and sampling, the important boundary is the small surface exposed by these files: tokenizers return keyed token batches; text encoders implement encode_token_weights() and sometimes generate(); vision encoders expose encode_image(); audio encoders expose encode_audio() or generate_abc(). Samplers and conditioning nodes can change selected CLIP layers or execution device by calling set_clip_options()/reset_clip_options() on the wrappers in comfy/sd1_clip.py, comfy/sdxl_clip.py, and the comfy/text_encoders/** wrappers, without needing to know each encoder family’s internal model class.
Coupling
- If
SDTokenizer.tokenize_with_weights()orSDClipModel.process_tokens()in comfy/sd1_clip.py changes the token tuple format, update multimodal tokenizers in comfy/…/qwen_image.py, comfy/…/qwen3vl.py, comfy/…/qwen35.py, and comfy/…/lt.py: they rely on replacing a token tuple’s first element with an embed descriptor while preserving the remaining weight/word-id fields. - If attention-mask shape or meaning changes in comfy/sd1_clip.py, verify T5/SD3/Flux/Qwen wrappers that set
enable_attention_masks=Trueandreturn_attention_masks=True, especially comfy/…/sd3_clip.py, comfy/…/flux.py, comfy/…/qwen_image.py, and comfy/…/qwen3vl.py; several downstreamextra["attention_mask"]values are flattened or sliced rather than recomputed. - If comfy/clip_model.py changes CLIP/SigLIP output tuple ordering, update comfy/clip_vision.py and comfy/sd1_clip.py:
ClipVisionModel.encode_image()assumes(last_hidden, intermediate, projected, mm_projected)for vision, whileSDClipModel.forward()assumes text outputs contain hidden, intermediate, projected pooled, and unprojected pooled values. - If a vision config JSON changes
model_type,image_size,patch_size,image_mean, orimage_std, check comfy/clip_vision.py detection and preprocessing.siglip2_vision_modelis routed tosiglip2_preprocess()while other CLIP/SigLIP/DINO paths useclip_preprocess()with config-provided normalization. - If DINO checkpoint key names change, update
load_clipvision_from_sd()in comfy/clip_vision.py; it selects DINOv2/DINOv3 configs from exact keys such asencoder.layer.23.layer_scale2.lambda1,layer.0.mlp.gate_proj.weight, andlayer.23.attention.o_proj.bias. - If Qwen multimodal image/video preprocessing changes in comfy/…/qwen_vl.py, verify
preprocess_embed()and position-ID construction in comfy/…/qwen3vl.py and comfy/…/qwen35.py; MRoPE position IDs and DeepStack visual masks are derived fromembeds_infosizes returned bySDClipModel.process_tokens(). - If
comfy/audio.resample()changes output length rounding, verify comfy/…/audio_encoders.py, comfy/…/whisper.py, and comfy/…/sheetsage2.py; audio sample counts, Whisper’s fixed 30-second crop/pad, and SheetSage2’s time-window stitching assume stable sample lengths after resampling. - If
FixedKVin comfy/…/llama.py changes fields or semantics, verify comfy/…/sheetsage2.py: its decoder reusesFixedKVfor graphable autoregressive cache updates even though it is not a text encoder.
Risks
- comfy/sd1_clip.py silently ignores multimodal embeddings whose final dimension does not match the token embedding dimension, padding the sequence with masked pad embeddings instead. If a tokenizer/model pair uses the wrong
embedding_sizeor a vision projector emits a mismatched width, image/video conditioning is dropped while tokenization still succeeds. - comfy/clip_vision.py state-dict detection is key-based and returns
Nonefor unknown layouts. A checkpoint that renames CLIP/SigLIP/DINO layer keys without changing architecture will fail to load as a vision encoder rather than falling back to a config. - comfy/…/dino2.py has explicit assertions around interpolated position-grid size and a comment noting that height/width scale-factor order breaks non-square DA3 paths if swapped. Changing interpolation math can break multi-view/non-square feature extraction even when square CLIP-vision tests pass.
- comfy/…/qwen3vl.py and comfy/…/qwen35.py rely on placeholder token IDs (
151655for Qwen image pads and vocab-derived image/video pad IDs for Qwen3.5). If tokenizer assets undercomfy/text_encoders/qwen25_tokenizer/**orcomfy/text_encoders/qwen35_tokenizer/**drift from those IDs, image/video embeddings will not be inserted. - comfy/audio.py implements
_LFilterby moving tensors through SciPy on CPU. Biquad operations are differentiable only through the custom backward wrapper and will incur CPU transfers; using them in hot GPU paths can become a performance bottleneck. - comfy/…/sheetsage2.py constrains generated score tokens with
PromptGrammarState. IfScoreTokenizerranges or token meanings are changed without updating the grammar, transcription may produce invalid events or stop early despite the model logits being valid.