[LoRA] Add support for Z-Image Base distilled loras

[asomoza]

Add support for the distilled loras here for the Base Z-Image model.

code:

import torch

from diffusers import ZImagePipeline


repo_id = "Tongyi-MAI/Z-Image"

pipe = ZImagePipeline.from_pretrained(repo_id, torch_dtype=torch.bfloat16)

pipe.load_lora_weights(
    "alibaba-pai/Z-Image-Fun-Lora-Distill",
    weight_name="Z-Image-Fun-Lora-Distill-8-Steps.safetensors",
    # weight_name="Z-Image-Fun-Lora-Distill-8-Steps-2602.safetensors",
    # weight_name="Z-Image-Fun-Lora-Distill-4-Steps-2602.safetensors",
    adapter_name="distill",
)
pipe.enable_model_cpu_offload()

pipe.set_adapters("distill", 0.8) # they recommend between 0.7-0.8

prompt = """stock photo of a fox wearing a beret and smock, standing in front of an easel, painting an abstract canvas. The canvas has splashes of vibrant colors, and near the fox's paw, inside the canvas there is also the text 'Diffusers' in wild, expressive strokes. The photo is taken from the side, with the fox in sharp focus while the colorful background of the studio fades softly, adding an artsy, playful tone."""
image = pipe(
    prompt,
    height=1024,
    width=1024,
    num_inference_steps=9,
    # num_inference_steps=5, # for the 4-step lora
    guidance_scale=1.0,
    generator=torch.Generator("cuda").manual_seed(42),
).images[0]

image.save("zimage-distilled.png")

Images:

8-Steps	8-Steps 2602	4-Steps 2602

@sayakpaul

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[sayakpaul]

Thanks. If you could address my comment on the support for existing LoRAs, that would be great!

[sayakpaul]

Good lord. Double _ 🥲

[sayakpaul]

Hope this does not break compatibility with existing LoRAs?