Skip to content

Add --load-tasks-multilingual and fix --custom-tasks for inspect backend#1199

Open
dzautner wants to merge 1 commit intohuggingface:mainfrom
LumiOpen:upstream-pr/inspect-load-multilingual
Open

Add --load-tasks-multilingual and fix --custom-tasks for inspect backend#1199
dzautner wants to merge 1 commit intohuggingface:mainfrom
LumiOpen:upstream-pr/inspect-load-multilingual

Conversation

@dzautner
Copy link
Copy Markdown

Summary

The inspect backend (lighteval eval) hardcodes custom_tasks=None and load_multilingual=False when constructing the Registry, making it impossible to use multilingual or custom tasks. All other backends (vllm, accelerate, sglang, endpoint) properly accept and pass through both parameters.

This PR:

  • Adds --load-tasks-multilingual CLI flag to the inspect backend, reusing the shared load_tasks_multilingual arg from cli_args.py (same pattern as all other backends)
  • Fixes custom_tasks being hardcoded to None — the parameter already exists in the function signature but was never forwarded to Registry

Changes (3 lines in main_inspect.py)

  1. Import the shared CLI arg definition
  2. Add the load_tasks_multilingual parameter to the eval() function
  3. Pass both custom_tasks and load_tasks_multilingual to Registry() instead of hardcoded values

Usage

lighteval eval "vllm/model" "mgsm:fi" --load-tasks-multilingual
lighteval eval "vllm/model" "my_task" --custom-tasks path/to/tasks.py

Test plan

  • Verified --load-tasks-multilingual flag appears in CLI help
  • Verified multilingual tasks load when flag is set (Registry finds tasks from multilingual/tasks/)
  • End-to-end eval run with maime25:fi (30 samples) completed successfully on AMD MI325X cluster using vLLM backend through inspect-ai
  • Ruff formatting passes

… backend

The inspect backend hardcoded load_multilingual=False and custom_tasks=None,
making it impossible to use multilingual tasks or custom task files. Wire
both CLI parameters through to the Registry, consistent with all other
backends (vllm, accelerate, sglang, endpoint, etc).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant