Skip to content

Fix Bug: _get_or_create_exp creates nested relative directories for file: URI file lock#2238

Open
2young-2simple-sometimes-naive wants to merge 1 commit into
microsoft:mainfrom
2young-2simple-sometimes-naive:patch-1
Open

Fix Bug: _get_or_create_exp creates nested relative directories for file: URI file lock#2238
2young-2simple-sometimes-naive wants to merge 1 commit into
microsoft:mainfrom
2young-2simple-sometimes-naive:patch-1

Conversation

@2young-2simple-sometimes-naive
Copy link
Copy Markdown
Contributor

File: qlib/workflow/expm.py, line 236

Severity: Medium — silently creates garbage directory trees under the working directory on every new
experiment.

Root Cause

When the MLflow tracking URI uses the file: scheme, e.g. file:/home/user/project/mlruns, urlparse
decomposes it as:

  • scheme = "file"
  • netloc = "" (empty, because there is no // authority component)
  • path = "/home/user/project/mlruns"

The lock path at line 236 is constructed as:

Path(os.path.join(pr.netloc, pr.path.lstrip("/"), "filelock"))

pr.path.lstrip("/") strips all leading slashes from the absolute path, turning
/home/user/project/mlruns into home/user/project/mlruns. Then os.path.join("",
"home/user/project/mlruns", "filelock") produces a relative path: home/user/project/mlruns/filelock.

When FileLock acquires this lock, it creates all parent directories relative to the process's current working directory, producing a nested copy of the full absolute path rooted at CWD (e.g., ./home/user/project/mlruns/).

Fix

One line change at expm.py:236:

Before:
with FileLock(Path(os.path.join(pr.netloc, pr.path.lstrip("/"), "filelock"))): # pylint: disable=E0110

After:
with FileLock(Path(pr.path) / "filelock"):

Path(pr.path) / "filelock" preserves the original absolute path from the parsed URI. It works correctly for all file: URI forms:

  • file:/absolute/path gives /absolute/path/filelock
  • file:///absolute/path gives /absolute/path/filelock
  • file:relative/path gives relative/path/filelock (unchanged — relative URIs already worked)

Reproduction

import qlib
from qlib.workflow import R

qlib.init(provider_uri="~/.qlib/qlib_data/cn_data", region="cn")

On first call with a new experiment name, this hits _get_or_create_exp

and creates the nested directory tree.

with R.start(experiment_name="test_experiment"):
pass

After running, check the working directory — you will find a nested path like
./home/user/project/mlruns/ mirroring the absolute path structure but rooted at CWD.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant