Skip to content

Conversation

@Pfannkuchensack
Copy link
Collaborator

@Pfannkuchensack Pfannkuchensack commented Jan 16, 2026

Summary

Add initial support for FLUX 2 klein models architecture.

New Invocation Nodes

  • flux2_denoise: Denoising invocation for FLUX 2 models
  • flux2_klein_model_loader: Model loader for Klein architecture
  • flux2_klein_text_encoder: Text encoder using Qwen3-based encoding
  • flux2_vae_decode: VAE decoder for FLUX 2 output

Backend Support

  • New flux2 module with denoise and sampling utilities
  • Extended model manager configs for FLUX 2 model types
  • Updated model loaders to support Klein architecture
  • Added Qwen3 encoder configuration updates

Frontend Updates

  • Extended graph builder for FLUX 2 workflow support
  • Added FLUX 2 model types and configurations
  • Updated readiness checks for FLUX 2 models
  • UI components updated for new model type

Related Issues / Discussions

QA Instructions

  1. Install a FLUX 2 Kontext model (Klein architecture)
  2. Select the model in the UI
  3. Test text-to-image generation with the new model
  4. Verify step callbacks and progress display work correctly

Merge Plan

Standard merge after review.

Checklist

  • The PR has a short but descriptive title, suitable for a changelog
  • Tests added / updated (if applicable)
  • ❗Changes to a redux slice have a corresponding migration
  • Documentation added / updated (if applicable)
  • Updated What's New copy (if doing a release after this PR)

- Add new invocation nodes for FLUX 2:
  - flux2_denoise: Denoising invocation for FLUX 2
  - flux2_klein_model_loader: Model loader for Klein architecture
  - flux2_klein_text_encoder: Text encoder for Qwen3-based encoding
  - flux2_vae_decode: VAE decoder for FLUX 2

- Add backend support:
  - New flux2 module with denoise and sampling utilities
  - Extended model manager configs for FLUX 2 models
  - Updated model loaders for Klein architecture

- Update frontend:
  - Extended graph builder for FLUX 2 support
  - Added FLUX 2 model types and configurations
  - Updated readiness checks and UI components
@github-actions github-actions bot added python PRs that change python files invocations PRs that change invocations backend PRs that change backend files services PRs that change app services frontend PRs that change frontend files labels Jan 16, 2026
FLUX.2 VAE uses Batch Normalization in the patchified latent space
(128 channels). The decode must:
1. Patchify latents from (B, 32, H, W) to (B, 128, H/2, W/2)
2. Apply BN denormalization using running_mean/running_var
3. Unpatchify back to (B, 32, H, W) for VAE decode

Also fixed image normalization from [-1, 1] to [0, 255].

This fixes washed-out colors in generated FLUX.2 Klein images.
…ompatibility

- Add FLUX.2 transformer loader with BFL-to-diffusers weight conversion
- Fix AdaLayerNorm scale-shift swap for final_layer.adaLN_modulation weights
- Add VAE batch normalization handling for FLUX.2 latent normalization
- Add Qwen3 text encoder loader with ComfyUI FP8 quantization support
- Add frontend components for FLUX.2 Klein model selection
- Update configs and schema for FLUX.2 model types
@Pfannkuchensack Pfannkuchensack marked this pull request as ready for review January 17, 2026 02:59
- Add isFlux2Klein9BMainModelConfig and isNonCommercialMainModelConfig functions
- Update MainModelPicker and InitialStateMainModelPicker to show license icon
- Update license tooltip text to include FLUX.2 Klein 9B
@Pfannkuchensack Pfannkuchensack changed the title WIP: feat(flux2): add FLUX 2 Kontext model support feat(flux2): add FLUX 2 Kontext model support Jan 17, 2026
@Pfannkuchensack Pfannkuchensack changed the title feat(flux2): add FLUX 2 Kontext model support feat(flux2): add FLUX.2 klein model support Jan 17, 2026
@lstein
Copy link
Collaborator

lstein commented Jan 17, 2026

I tested with two models: the diffusers model at black-forest-labs/FLUX.2-klein-9B, and a quantized GGUF at https://huggingface.co/unsloth/FLUX.2-klein-4B-GGUF/resolve/main/flux-2-klein-4b-Q8_0.gguf

Initial observations:

  1. When using the diffusers model and the built-in text encoder from the main model, I get the error:
ValueError: Cannot use chat template functions because tokenizer.chat_template is not set and no template argument was passed! For information about writing templates and setting the tokenizer.chat_temp
late attribute, please see the documentation at https://huggingface.co/docs/transformers/main/en/chat_templating                   
  1. When using the diffusers model and clicking on the Qwen3 Encoder popup menu, I'm given the option to choose among a few previously-installed Z-Image Qwen3 Text Encoders (which may not be the desired behavior?) I choose Z-Image Qwen3 Text Encoder, but generation fails with
File "/home/lstein/Projects/InvokeAI/invokeai/backend/model_manager/load/model_cache/torch_module_autocast/custom_modules/custom_linear.py", line 82, in _autocast_forward                              
    return torch.nn.functional.linear(input, weight, bias)                                                                                                                                                
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                           
RuntimeError: mat1 and mat2 shapes cannot be multiplied (512x7680 and 12288x4096)
  1. The GGUF file is recognized as a Flux1 model (not Flux.2).

  2. In fact, there seems to be a regression with the Flux1 models as well. When using flux1 schnell or flux1 krea, I'm seeing this error on the front end: Failed to build graph: Node with id pos_cond_collect not found

@lstein
Copy link
Collaborator

lstein commented Jan 18, 2026

Here's a trivial fix for the error Cannot use chat template functions because tokenizer.chat_template is not set...

diff --git a/invokeai/backend/model_manager/util/select_hf_files.py b/invokeai/backend/model_manager/util/select_hf_files.py
index 5bfb2f1f29..615eb394ef 100644
--- a/invokeai/backend/model_manager/util/select_hf_files.py
+++ b/invokeai/backend/model_manager/util/select_hf_files.py
@@ -70,6 +70,7 @@ def filter_files(
                 "weights.pb",
                 "onnx_data",
                 "spiece.model",  # Added for `black-forest-labs/FLUX.1-schnell`.
+                "chat_template.jinja", # added for `black-forest-labs/FLUX.2-klein-9B`.
             )
         ):
             paths.append(file)

Backend:
- Add klein_4b/klein_9b variants for FLUX.2 Klein models
- Add qwen3_4b/qwen3_8b variants for Qwen3 encoder models
- Validate encoder variant matches Klein model (4B↔4B, 9B↔8B)
- Auto-detect Qwen3 variant from hidden_size during probing

Frontend:
- Show variant field for all model types in ModelView
- Filter Qwen3 encoder dropdown to only show compatible variants
- Update variant type definitions (zFlux2VariantType, zQwen3VariantType)
- Remove unused exports (isFluxDevMainModelConfig, isFlux2Klein9BMainModelConfig)
Distinguish between FLUX.2 Klein 9B (distilled) and Klein 9B Base (undistilled)
models by checking guidance_embeds in diffusers config or guidance_in keys in
safetensors. Klein 9B Base requires more steps but offers higher quality.
Backend changes:
- Update text encoder layers from [9,18,27] to (10,20,30) matching diffusers
- Use apply_chat_template with system message instead of manual formatting
- Change position IDs from ones to zeros to match diffusers implementation
- Add get_schedule_flux2() with empirical mu computation for proper schedule shifting
- Add txt_embed_scale parameter for Qwen3 embedding magnitude control
- Add shift_schedule toggle for base (28+ steps) vs distilled (4 steps) models
- Zero out guidance_embedder weights for Klein models without guidance_embeds

UI changes:
- Clear Klein VAE and Qwen3 encoder when switching away from flux2 base
- Clear Qwen3 encoder when switching between different Klein model variants
- Add toast notification informing user to select compatible encoder
@lstein
Copy link
Collaborator

lstein commented Jan 19, 2026

The recent commit is causing the previously-installed ZImage Qwen3 encoders to fail validation with warnings like this one:

[2026-01-19 08:13:24,169]::[InvokeAI]::WARNING --> Skipping invalid model config in the database with name Z-Image Qwen3 Text Encoder. Ignoring this model. ({"key":"04198228-6d55-4c14-8759-069ae2eb4672","hash":"blake3:85a...)
[2026-01-19 08:13:24,169]::[InvokeAI]::WARNING --> Validation error: 1 validation error for tagged-union[Main_Diffusers_SD1_Config,Main_Diffusers_SD2_Config,Main_Diffusers_SDXL_Config,Main_Diffusers_SDXLRefiner_Config,Main_Diffusers_SD3_Config,Main_Diffusers_FLUX_Config,Main_Diffusers_Flux2_Config,Main_Diffusers_CogView4_Config,Main_Diffusers_ZImage_Config,Main_Checkpoint_SD1_Config,Main_Checkpoint_SD2_Config,Main_Checkpoint_SDXL_Config,Main_Checkpoint_SDXLRefiner_Config,Main_Checkpoint_FLUX_Config,Main_Checkpoint_Flux2_Config,Main_Checkpoint_ZImage_Config,Main_BnBNF4_FLUX_Config,Main_GGUF_FLUX_Config,Main_GGUF_ZImage_Config,VAE_Checkpoint_SD1_Config,VAE_Checkpoint_SD2_Config,VAE_Checkpoint_SDXL_Config,VAE_Checkpoint_FLUX_Config,VAE_Checkpoint_Flux2_Config,VAE_Diffusers_SD1_Config,VAE_Diffusers_SDXL_Config,ControlNet_Checkpoint_SD1_Config,ControlNet_Checkpoint_SD2_Config,ControlNet_Checkpoint_SDXL_Config,ControlNet_Checkpoint_FLUX_Config,ControlNet_Checkpoint_ZImage_Config,ControlNet_Diffusers_SD1_Config,ControlNet_Diffusers_SD2_Config,ControlNet_Diffusers_SDXL_Config,ControlNet_Diffusers_FLUX_Config,LoRA_LyCORIS_SD1_Config,LoRA_LyCORIS_SD2_Config,LoRA_LyCORIS_SDXL_Config,LoRA_LyCORIS_FLUX_Config,LoRA_LyCORIS_ZImage_Config,LoRA_OMI_SDXL_Config,LoRA_OMI_FLUX_Config,LoRA_Diffusers_SD1_Config,LoRA_Diffusers_SD2_Config,LoRA_Diffusers_SDXL_Config,LoRA_Diffusers_FLUX_Config,LoRA_Diffusers_ZImage_Config,ControlLoRA_LyCORIS_FLUX_Config,T5Encoder_T5Encoder_Config,T5Encoder_BnBLLMint8_Config,Qwen3Encoder_Qwen3Encoder_Config,Qwen3Encoder_Checkpoint_Config,Qwen3Encoder_GGUF_Config,TI_File_SD1_Config,TI_File_SD2_Config,TI_File_SDXL_Config,TI_Folder_SD1_Config,TI_Folder_SD2_Config,TI_Folder_SDXL_Config,IPAdapter_InvokeAI_SD1_Config,IPAdapter_InvokeAI_SD2_Config,IPAdapter_InvokeAI_SDXL_Config,IPAdapter_Checkpoint_SD1_Config,IPAdapter_Checkpoint_SD2_Config,IPAdapter_Checkpoint_SDXL_Config,IPAdapter_Checkpoint_FLUX_Config,T2IAdapter_Diffusers_SD1_Config,T2IAdapter_Diffusers_SDXL_Config,Spandrel_Checkpoint_Config,CLIPEmbed_Diffusers_G_Config,CLIPEmbed_Diffusers_L_Config,CLIPVision_Diffusers_Config,SigLIP_Diffusers_Config,FLUXRedux_Checkpoint_Config,LlavaOnevision_Diffusers_Config,Unknown_Config]
`qwen3_encoder.qwen3_encoder.any`.variant
  Field required [type=missing, input_value={'key': '04198228-6d55-4c...ormat': 'qwen3_encoder'}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.12/v/missing

@lstein
Copy link
Collaborator

lstein commented Jan 19, 2026

4B is working for me, although I find the results pretty underwhelming relative to the hype.

However, 9B is producing bad images. I did fresh installs of the official diffusers versions from HuggingFace using the MM interface. Prompt was "photograph of ginger cat, sitting on the sofa in a modern sunlit apartment. A tag around the cat's neck reads "'Klein'". I used CFG 1, the Euler scheduler, and 10 steps.

4B with 10 steps:

image

9B with 10 steps:

image

4B with 8 steps

image

9B with 8 steps

image

4B with 4 steps

image

9B with 4 steps

image

And just for fun, 9B with 8 steps and CFG=4

image

@lstein
Copy link
Collaborator

lstein commented Jan 19, 2026

Another issue I just noticed. When I try to generate with Flux.1 (schnell or krea), I am getting "Node with id pos_cond_collect is not found."

- Configure scheduler with FLUX.2 Klein parameters from scheduler_config.json
  (use_dynamic_shifting=True, shift=3.0, time_shift_type="exponential")
- Pass mu parameter to scheduler.set_timesteps() for resolution-aware shifting
- Remove manual shift_schedule parameter (scheduler handles this automatically)
- Simplify get_schedule_flux2() to return linear sigmas only
- Remove txt_embed_scale parameter (no longer needed)

This matches the diffusers Flux2KleinPipeline behavior where the
FlowMatchEulerDiscreteScheduler applies dynamic timestep shifting
based on image resolution via the mu parameter.

Fixes 4-step distilled Klein 9B model quality issues.
The posCondCollect node was created with getPrefixedId() which generates
a random suffix (e.g., 'pos_cond_collect:abc123'), but g.getNode() was
called with the plain string 'pos_cond_collect', causing a node lookup
failure.

Fix by declaring posCondCollect as a module-scoped variable and
referencing it directly instead of using g.getNode().
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backend PRs that change backend files frontend PRs that change frontend files invocations PRs that change invocations python PRs that change python files services PRs that change app services

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants