Skip to content

[Modular] loader related #13025

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
yiyixuxu wants to merge 4 commits into
base: main
Choose a base branch
from
+24 −6
Open

[Modular] loader related #13025

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
581a425
tag loader_id from Automodel
yiyixuxu Jan 24, 2026
1f57b17
style
yiyixuxu Jan 24, 2026
372222a
load_components by default only load components that are not already …
yiyixuxu Jan 24, 2026
e77f1c5
by default, skip loading the componeneets does not have the repo id
yiyixuxu Jan 24, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 9 additions & 2 deletions src/diffusers/models/auto_model.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@
from huggingface_hub.utils import validate_hf_hub_args

from ..configuration_utils import ConfigMixin
from ..utils import logging
from ..utils import DIFFUSERS_LOAD_ID_FIELDS, logging
from ..utils.dynamic_modules_utils import get_class_from_dynamic_module, resolve_trust_remote_code


Expand Down Expand Up @@ -220,4 +220,11 @@ def from_pretrained(cls, pretrained_model_or_path: Optional[Union[str, os.PathLi
raise ValueError(f"AutoModel can't find a model linked to {orig_class_name}.")

kwargs = {**load_config_kwargs, **kwargs}
return model_cls.from_pretrained(pretrained_model_or_path, **kwargs)
model = model_cls.from_pretrained(pretrained_model_or_path, **kwargs)

load_id_kwargs = {"pretrained_model_name_or_path": pretrained_model_or_path, **kwargs}
parts = [load_id_kwargs.get(field, "null") for field in DIFFUSERS_LOAD_ID_FIELDS]
load_id = "|".join("null" if p is None else p for p in parts)
model._diffusers_load_id = load_id

return model
2 changes: 2 additions & 0 deletions src/diffusers/modular_pipelines/modular_pipeline.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2142,6 +2142,8 @@ def load_components(self, names: Optional[Union[List[str], str]] = None, **kwarg
name
for name in self._component_specs.keys()
if self._component_specs[name].default_creation_method == "from_pretrained"
and self._component_specs[name].pretrained_model_name_or_path is not None
Copy link
Collaborator Author

@yiyixuxu yiyixuxu Jan 28, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

with this change, now, by default with pipeline.load_components():

  1. we don't load components that's already loaded in pipeline
  2. we don't load the components that do not have valid pretrained_model_name_or_path

for (1), currently, under the main, if we run script below, we will load the text_encoder, add it to the pipeline with update_components, and then reload again when you run load_components() - which is a bit unintuitive I think

in this PR, it will skip text_encoder instead of loading all components

pipe = ModularPipeline.from_pretrained()
text_encoder = AutoModel.from_pretrained(...)
pipe.update_components(text_encoder=text_encoder )
pipe.load_components(torch_dtype=torch.bfloat16)

for (2), in main, if you run this

from diffusers import ModularPipeline
pipe = ModularPipeline.from_pretrained("Qwen/Qwen-Image")
pipe.load_components(torch_dtype=torch.bfloat16)

you would get a confusing message like this


Failed to create component controlnet:
- Component spec: ComponentSpec(name='controlnet', type_hint=<class 'diffusers.models.controlnets.controlnet_qwenimage.QwenImageControlNetModel'>, description=None, config=None, pretrained_model_name_or_path=None, subfolder='', variant=None, revision=None, default_creation_method='from_pretrained', repo=None)
- load() called with kwargs: {'torch_dtype': torch.bfloat16}
If this component is not required for your workflow you can safely ignore this message.

Traceback:
Traceback (most recent call last):
  File "/fsx/yiyi/diffusers/src/diffusers/modular_pipelines/modular_pipeline.py", line 2173, in load_components
    components_to_register[name] = spec.load(**component_load_kwargs)
  File "/fsx/yiyi/diffusers/src/diffusers/modular_pipelines/modular_pipeline_utils.py", line 279, in load
    raise ValueError(
ValueError: `pretrained_model_name_or_path` info is required when using `load` method (you can directly set it in `pretrained_model_name_or_path` field of the ComponentSpec or pass it as an argument)

this is because controlnet is a component in qwen auto pipeline, but not included in Qwen/Qwen-Image; in this PR, we just skip it by default and not sending any message - I think it is more of an expected behavior

and getattr(self, name, None) is None
]
elif isinstance(names, str):
names = [names]
Expand Down
8 changes: 4 additions & 4 deletions src/diffusers/modular_pipelines/modular_pipeline_utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,14 +15,14 @@
import inspect
import re
from collections import OrderedDict
from dataclasses import dataclass, field, fields
from dataclasses import dataclass, field
from typing import Any, Dict, List, Literal, Optional, Type, Union

import torch

from ..configuration_utils import ConfigMixin, FrozenDict
from ..loaders.single_file_utils import _is_single_file_path_or_url
from ..utils import is_torch_available, logging
from ..utils import DIFFUSERS_LOAD_ID_FIELDS, is_torch_available, logging


if is_torch_available():
Expand Down Expand Up @@ -185,7 +185,7 @@ def loading_fields(cls) -> List[str]:
"""
Return the names of all loading‐related fields (i.e. those whose field.metadata["loading"] is True).
"""
return [f.name for f in fields(cls) if f.metadata.get("loading", False)]
return DIFFUSERS_LOAD_ID_FIELDS.copy()

@property
def load_id(self) -> str:
Expand All @@ -197,7 +197,7 @@ def load_id(self) -> str:
return "null"
parts = [getattr(self, k) for k in self.loading_fields()]
parts = ["null" if p is None else p for p in parts]
return "|".join(p for p in parts if p)
return "|".join(parts)

@classmethod
def decode_load_id(cls, load_id: str) -> Dict[str, Optional[str]]:
Expand Down
1 change: 1 addition & 0 deletions src/diffusers/utils/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@
DEFAULT_HF_PARALLEL_LOADING_WORKERS,
DEPRECATED_REVISION_ARGS,
DIFFUSERS_DYNAMIC_MODULE_NAME,
DIFFUSERS_LOAD_ID_FIELDS,
FLAX_WEIGHTS_NAME,
GGUF_FILE_EXTENSION,
HF_ENABLE_PARALLEL_LOADING,
Expand Down
8 changes: 8 additions & 0 deletions src/diffusers/utils/constants.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -73,3 +73,11 @@
ENCODE_ENDPOINT_SD_V1 = "https://qc6479g0aac6qwy9.us-east-1.aws.endpoints.huggingface.cloud/"
ENCODE_ENDPOINT_SD_XL = "https://xjqqhmyn62rog84g.us-east-1.aws.endpoints.huggingface.cloud/"
ENCODE_ENDPOINT_FLUX = "https://ptccx55jz97f9zgo.us-east-1.aws.endpoints.huggingface.cloud/"


DIFFUSERS_LOAD_ID_FIELDS = [
"pretrained_model_name_or_path",
"subfolder",
"variant",
"revision",
]
Loading