Skip to content

fix(loading): forward download_config to LocalEvaluationModuleFactory for local paths#760

Open
xodn348 wants to merge 1 commit into
huggingface:mainfrom
xodn348:fix/local-module-factory-download-config
Open

fix(loading): forward download_config to LocalEvaluationModuleFactory for local paths#760
xodn348 wants to merge 1 commit into
huggingface:mainfrom
xodn348:fix/local-module-factory-download-config

Conversation

@xodn348
Copy link
Copy Markdown

@xodn348 xodn348 commented May 18, 2026

Summary

evaluation_module_factory has two branches that resolve a local script path — one for a bare .py file and one for a directory containing a same-named script. In both cases the function constructs LocalEvaluationModuleFactory without forwarding the caller-supplied download_config. The factory then falls back to a fresh DownloadConfig(), silently discarding any settings the caller passed (e.g. local_files_only=True, a custom cache_dir, or use_etag=False). This makes it impossible to prevent outbound network requests when loading local evaluation modules in air-gapped or restricted-network environments, because the _download_additional_modules path inside the factory always receives default download behaviour regardless of what the caller specified.

The fix is a one-liner (×2): add download_config=download_config to each LocalEvaluationModuleFactory(...) call at lines 619–621 and 625–627 of src/evaluate/loading.py. The non-local (Hub, cached) branches already pass download_config correctly; this aligns the local branches with them.

Two regression tests are added to tests/test_load.py that verify download_config is forwarded for both the .py-file path and the directory path. They use mock.patch to intercept the factory constructor call and assert the download_config kwarg matches the caller-supplied object.

Issue

Fixes #709

Local verification

# Linters (black 22.12.0, isort, flake8) — matches CI quality job
$ cd /tmp/evaluate
$ python -m black --check --line-length 119 --target-version py36 src/evaluate/loading.py tests/test_load.py
All done! ✨ 🍰 ✨
2 files would be left unchanged.

$ isort --check-only src/evaluate/loading.py tests/test_load.py
(no output — all imports correctly sorted)

$ flake8 src/evaluate/loading.py tests/test_load.py
(no output — no issues)

ALL LINTERS PASS

# Unit tests (non-network, mirrors `python -m pytest -n 2 ./tests/` unit job)
$ PYTHONPATH=/tmp/evaluate/src python -m pytest tests/test_load.py -v \
    -k "not Hub and not hub and not remote"
============================= test session starts ==============================
platform linux -- Python 3.11.15, pytest-9.0.3, pluggy-1.6.0
collecting ... collected 9 items / 5 deselected / 4 selected

tests/test_load.py::ModuleFactoryTest::test_CachedMetricModuleFactory PASSED [ 25%]
tests/test_load.py::ModuleFactoryTest::test_LocalMetricModuleFactory PASSED [ 50%]
tests/test_load.py::ModuleFactoryTest::test_evaluation_module_factory_passes_download_config_local_dir PASSED [ 75%]
tests/test_load.py::ModuleFactoryTest::test_evaluation_module_factory_passes_download_config_local_script PASSED [100%]

======================= 4 passed, 5 deselected in 0.07s ========================
=== LOCAL_TEST_PASSED ===

Risk

The change is minimal — two download_config=download_config keyword arguments added to existing LocalEvaluationModuleFactory constructor calls. The parameter was already defined on the constructor with a safe default (DownloadConfig()), so passing it explicitly cannot break any existing behaviour: callers that previously got the default will now get exactly the same default (since the top-level evaluation_module_factory already initialises download_config = DownloadConfig(**download_kwargs) when none is supplied). The only callers affected are those who explicitly pass a non-default DownloadConfig to evaluation_module_factory with a local path — those callers previously had their config silently dropped; now it is respected.

… for local paths

Previously, both branches of evaluation_module_factory that resolve a local
.py file or directory omitted download_config when constructing
LocalEvaluationModuleFactory. This caused the factory to fall back to a
default DownloadConfig(), silently ignoring caller-supplied settings such as
local_files_only, cache_dir, or use_etag — making it impossible to prevent
outbound network access when loading local evaluation modules.

Fixes huggingface#709
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

DownloadConfig is not passed to LocalEvaluationModuleFactory when loading from local file, causing uncontrollable download behavior

1 participant