Introducing a new way to review your PRs.

Read the announcement

comfy-org / comfyui

The most powerful and modular diffusion model GUI, api and backend with a graph/nodes interface. The fastest local inference engine in the world.

View on GitHub

13 documents · Updated Nov 19, 2026

docs / sampling-diffusion-conditioning

Sampling, Diffusion, and Conditioning

Overview

This area owns ComfyUI's denoising orchestration: noise creation, sigma scheduling, sampler dispatch, conditioning batching, ControlNet/T2I injection, hook-based model patching, latent normalization, and context-window sampling.

Routing/covers:

The public sampling entrypoints in comfy/sample.py are thin adapters. prepare_noise() creates deterministic CPU noise from a seed and supports nested latents by generating one tensor per nested component before repacking as comfy.nested_tensor.NestedTensor. fix_empty_latent_channels() adapts all-zero latent placeholders to the loaded model's latent_format channel count, spatial/temporal downscale ratio, and latent dimensionality; for formats such as MiniMaxH3AV in comfy/latent_formats.py, fix_empty_latent() can turn a single empty latent into a nested video/audio latent. sample() constructs comfy.samplers.KSampler, while sample_custom() accepts a prebuilt sampler object and sigma tensor. Both return samples moved back to Comfy's intermediate device/dtype.

comfy/samplers.py is the core orchestrator. KSampler validates sampler/scheduler names, creates sigma schedules through calculate_sigmas(), handles denoise, start_step, last_step, and force_full_denoise, then delegates to sample(). Scheduler names map to local schedules (simple, normal, ddim_uniform, beta, linear_quadratic, kl_optimal) or k-diffusion schedules (karras, exponential) using the active model's model_sampling object. Sampler names map to KSAMPLER wrappers over functions in comfy/…/sampling.py, plus ddim as Euler with randomized inpaint noise and UniPC implementations from comfy/…/uni_pc.py.

CFGGuider in comfy/samplers.py owns one sampling run. set_conds() converts node-style condition tuples into dictionaries via comfy/sampler_helpers.py::convert_cond(), preserving model_conds, cross-attention, hook/control metadata, and a per-cond UUID. CFGGuider.sample() packs nested latents/noise with comfy.utils.pack_latents(), prepares denoise masks in the packed latent shape, clones model_options, registers hooks, loads models, then unpacks the result if the input was nested. inner_sample() normalizes non-empty latent inputs through BaseModel.process_latent_in(), calls process_conds(), stores sample_sigmas in transformer_options, executes the sampler through WrappersMP.SAMPLER_SAMPLE, and converts the output back through BaseModel.process_latent_out().

Conditioning is resolved before the sampler loop. process_conds() in comfy/samplers.py scales masks and percentage areas to latent dimensions, converts percent ranges into sigma bounds using model.model_sampling.percent_to_sigma(), asks model.extra_conds() to fill model-specific COND* entries, mirrors regional conditions across positive/negative sets when needed, initializes hook keyframes, pre-runs ControlNets, and optionally applies positive-side ControlNet/GLIGEN objects to unconditioned conds. comfy/conds.py defines the objects that survive this path: CONDRegular repeats tensors to batch size, CONDCrossAttn can concatenate prompt embeddings by repeating shorter token dimensions up to a limited LCM, CONDNoiseShape slices by regional area, and CONDConstant/CONDKVCache/CONDList handle non-batched or composite condition payloads.

Each model evaluation inside a sampler goes through sampling_function() and calc_cond_batch() in comfy/samplers.py. sampling_function() optionally skips the uncond pass when CFG is 1, supports sampler_calc_cond_batch_function, sampler_pre_cfg_function, custom sampler_cfg_function, and sampler_post_cfg_function hooks in model_options, then combines cond/uncond predictions. _calc_cond_batch() turns each cond dictionary into a cond_obj with area crop, mask multiplier, processed model_conds, ControlNet, GLIGEN patches, UUID, and hooks. Compatible conds are batched together only when their crop shape, control object, GLIGEN patch object, and condition sizes are concat-safe. Outputs are accumulated back into full latent space with mask/area weights. The multigpu path _calc_cond_batch_multigpu() duplicates the same grouping/aggregation while scheduling batches across model_options['multigpu_clones'] and using device-specific ControlNet clones.

Loaded-model setup lives in comfy/sampler_helpers.py. prepare_model_patcher() gathers hooks from all conds and chained ControlNets, merges persistent ModelPatcher wrappers/callbacks into model_options['transformer_options'], registers transformer-option hooks, additional-model hooks, and weight hooks, then records registered hooks for later filtering. prepare_sampling() is wrapper-extensible (WrappersMP.PREPARE_SAMPLING), clones multigpu controls as needed, gathers ControlNet/GLIGEN/additional models from conds and hooks, estimates memory using model.model.extra_conds_shapes(), loads the base model plus additional models with comfy.model_management.load_models_gpu(), and returns the real BaseModel. Cleanup calls both additional-model cleanup and ControlNet cleanup.

Control integration is centered in comfy/controlnet.py. ControlBase stores the original hint image, strength, timestep percent range, chained previous_controlnet, extra conditioning keys, optional extra_hooks, and per-device multigpu clones. ControlNet.get_control() recursively computes previous controls, checks the current sigma against the precomputed timestep range, resizes/encodes/preprocesses the hint to match x_noisy, applies latent-format conversion when required, broadcasts hints to the batched cond count, converts requested extra cond tensors, transforms sigma to the control model's timestep/input using the active model_sampling, runs the control model, and merges input/middle/output controls with strength. ControlLora builds a ControlNet from LoRA-style weights during pre_run(), and T2IAdapter caches adapter outputs instead of calling a UNet-style control model every step. Loader functions in the same file detect many checkpoint layouts and choose latent formats, compression ratios, extra cond keys, and mask concatenation behavior.

Patch and extension behavior is split between comfy/hooks.py and comfy/patcher_extension.py. Hooks are conditioning-level objects: WeightHook registers LoRA/model-as-LoRA patches on the ModelPatcher, TransformerOptionsHook merges wrappers/callbacks/patches into transformer_options, and AdditionalModelsHook contributes extra models to load. HookKeyframeGroup converts percent schedules to sigmas and updates per-step strengths during prepare_hook_patches_current_keyframe(). HookGroup deduplicates by hook identity, combines groups, filters registered hooks, and can derive clip scheduling ranges. patcher_extension.py provides the generic wrapper/callback registry and ordered WrapperExecutor; sampling uses wrappers around outer sample, prepare sampling, sampler sample, predict noise, calc-cond-batch, model apply, and diffusion model calls.

Sampler implementations are mostly in comfy/…/sampling.py. KSAMPLER.sample() wraps the guider with KSamplerX0Inpaint, applies the model's noise_scaling() to combine initial noise and latent image, calls the selected sample_* function, and applies inverse_noise_scaling() at the end. K-diffusion samplers repeatedly call the wrapper as model(x, sigma, **extra_args) and report callbacks containing x, step index, sigma/sigma_hat, and denoised prediction. The file includes deterministic ODE samplers, ancestral samplers, SDE samplers backed by BrownianTreeNoiseSampler, DDPM/LCM-style step samplers, CFG++ variants, and RF/CONST special cases using sigma_to_half_log_snr() and half_log_snr_to_sigma(). comfy/…/uni_pc.py adapts UniPC to Comfy's sigma convention by wrapping the model as an epsilon predictor and running a multistep predictor-corrector solver.

comfy/model_sampling.py defines the contract between a loaded model and the sampler. The small parameterization classes (EPS, V_PREDICTION, EDM, CONST, X0, image-to-image variants, COSMOS_RFLOW) implement calculate_input(), calculate_denoised(), noise_scaling(), and inverse_noise_scaling(). The schedule classes (ModelSamplingDiscrete, continuous EDM/V, flow/Flux/Cosmos/StableCascade variants, AV flow) provide sigma buffers, sigma(), timestep(), and percent_to_sigma(). These objects are stored on the loaded model and are consumed by schedulers, ControlNet timestep conversion, hook keyframe initialization, and sampler noise scaling.

Latent formats in comfy/latent_formats.py define how samples are normalized around the sampler and previewed. LatentFormat supplies scale_factor, latent channel/dimension counts, spatial and temporal downscale ratios, optional RGB preview factors, and process_in()/process_out(). Subclasses encode checkpoint-specific mean/std shifts, video/audio dimensionality, pixel-space formats, and special empty-latent repair. comfy/pixel_space_convert.py provides a fake VAE whose encode/decode are identity operations for pixel-space models represented as latents.

Long temporal or packed multimodal sampling can be routed through comfy/context_windows.py. A context handler is stored in model_options['context_handler']; calc_cond_batch() asks it whether to intercept the model call. IndexListContextHandler builds temporal windows, optionally applies FreeNoise before sampler execution through a WrappersMP.SAMPLER_SAMPLE wrapper, slices latents and matching condition tensors, handles packed multimodal shapes and guide frames, calls the normal cond-batch function per window, and fuses results using flat, pyramid, relative, or overlap-linear weights. It also installs a WrappersMP.PREPARE_SAMPLING wrapper so memory estimation can use per-window shapes for non-packed latents.

Coupling

  • If a sampler function is added or renamed in comfy/…/sampling.py or comfy/…/uni_pc.py, also update KSAMPLER_NAMES, SAMPLER_NAMES, and sampler_object() in comfy/samplers.py; unknown names silently fall back to the first sampler in KSampler.__init__().
  • If a scheduler is added in comfy/samplers.py, also update SCHEDULER_HANDLERS/SCHEDULER_NAMES and decide whether its handler consumes model_sampling or raw sigma_min/sigma_max; KSampler.calculate_sigmas() assumes this boolean contract.
  • If model_sampling semantics change in comfy/model_sampling.py, verify comfy/…/sampling.py, comfy/controlnet.py, and comfy/hooks.py: samplers use noise scaling and logSNR conversion, ControlNet converts sigma to control timesteps and model input, and hook keyframes use percent_to_sigma().
  • If condition dictionary keys change in nodes or model code, verify process_conds(), get_area_and_mult(), calc_cond_batch(), and comfy/conds.py; keys such as model_conds, area, mask, default, control, gligen, hooks, start_percent, and end_percent are interpreted structurally rather than through a schema.
  • If ControlNet chaining or extra hooks change in comfy/controlnet.py, verify comfy/sampler_helpers.py::get_hooks_from_cond() and preprocess_conds_hooks() in comfy/samplers.py; ControlNet extra_hooks are merged into cond hook groups before batching, and mismatched hook groups reduce batching efficiency or skip intended patches.
  • If wrapper/callback names change in comfy/patcher_extension.py, update the registration and lookup sites in comfy/samplers.py, comfy/sampler_helpers.py, and comfy/context_windows.py; wrapper dispatch is string-keyed and does not enforce coverage at registration time.
  • If packed/nested latent layout changes in comfy.utils or model code, verify CFGGuider.sample() in comfy/samplers.py, IndexListContextHandler in comfy/context_windows.py, and LatentFormat.fix_empty_latent() in comfy/latent_formats.py; packed callbacks, denoise masks, multimodal context slicing, and empty-latent repair all assume compatible shape metadata.
  • If a latent format changes channel count, downscale ratio, dimensions, or process-in/out math in comfy/latent_formats.py, verify comfy/sample.py::fix_empty_latent_channels(), ControlNet loaders in comfy/controlnet.py, and model preview/encode/decode paths; empty latents and ControlNet hint encoding depend on those class attributes.

Risks

  • comfy/samplers.py::_calc_cond_batch() and _calc_cond_batch_multigpu() intentionally duplicate the same hook grouping, memory-fit batching, and output accumulation logic. A fix made to only one path can make single-GPU and multigpu sampling diverge for masks, default conds, hooks, or ControlNet outputs.
  • comfy/context_windows.py::_prepare_sampling_wrapper() cannot clamp memory estimates for packed latents because latent_shapes are not attached yet. Large packed multimodal jobs may be over-estimated or still hit per-window VRAM assumptions differently from non-packed video latents.
  • comfy/samplers.py::_calc_cond_batch_multigpu() comments note async output copies can race on non-NVIDIA backends before aggregation. Extending multigpu support beyond current CUDA assumptions should add synchronization around the .to(output_device) copies.
  • comfy/controlnet.py::ControlNet.get_control() raises when a latent-format ControlNet needs a VAE but none was provided. Changes to ControlNet apply nodes or loader-selected latent_format values can turn what looks like a runtime conditioning option into a hard sampling failure.
  • comfy/patcher_extension.py::merge_nested_dicts() extends lists while recursively merging dictionaries. Reusing mutable model_options or transformer-option dictionaries across runs can duplicate wrappers/callbacks unless callers clone as CFGGuider.sample() and inner_sample() do.