Store relative paths in the DB to make beets library portable#6460
Open
Store relative paths in the DB to make beets library portable#6460
Conversation
|
Thank you for the PR! The changelog has not been updated, so here is a friendly reminder to check if you need to add an entry. |
snejus
changed the title
snejus
force-pushed
the
use-relative-paths
branch
from
7fb98e0 to
68d0d21
Compare
❌ 4 Tests Failed:
View the top 3 failed test(s) by shortest run time
To view more test analytics, go to the Test Analytics Dashboard |
Contributor
There was a problem hiding this comment.
Pull request overview
PR make beets store item/album path in DB as library-relative, then expand back to absolute on read. Goal: portable DB when music folder move.
Changes:
- Add context var for active music dir, used by DB path encode/decode.
- Move path normalize/expand into
dbcoretypes + query layer, plus add migration to rewrite old rows. - Propagate context into thread pools/pipeline; update tests to expect absolute paths on public API.
Reviewed changes
Copilot reviewed 19 out of 19 changed files in this pull request and generated 9 comments.
Show a summary per file
| File | Description |
|---|---|
beets/context.py |
Add context var for music dir. |
beets/library/library.py |
Set music dir context early; register new migration. |
beets/dbcore/pathutils.py |
New helpers to normalize/expand paths for DB. |
beets/dbcore/types.py |
PathType encode/decode moved into DB type layer. |
beets/dbcore/query.py |
PathQuery now compares using DB-relative representation. |
beets/library/models.py |
Adjust non-PathQuery path field queries to match stored relative value. |
beets/library/migrations.py |
Add RelativePathMigration; add chunk size constant usage. |
beets/dbcore/db.py |
Add base migration chunk size constant. |
beets/util/pipeline.py |
Copy context into pipeline worker threads. |
beets/util/__init__.py |
Propagate context into par_map thread pool. |
beetsplug/replaygain.py |
Wrap pool work + callbacks in copied context. |
beetsplug/ipfs.py |
Create temp lib with /ipfs/ dir; store item paths relative to that root. |
beets/ui/__init__.py |
Override library DB path from BEETS_LIBRARY env var. |
test/test_library.py |
Update path expectations; assert raw stored relative paths. |
test/test_query.py |
Add query test for absolute input path. |
test/library/test_migrations.py |
Add migration test for absolute->relative rewrite. |
test/ui/test_ui.py |
Update ls -p output expectations to absolute paths. |
test/ui/commands/test_list.py |
Update $path output expectations to absolute paths. |
test/plugins/test_ipfs.py |
Normalize expected /ipfs/... path string. |
test/test_files.py |
Remove one remove/prune test (no replacement in this PR). |
snejus
force-pushed
the
use-relative-paths
branch
2 times, most recently
from
0e2a815 to
65f87ba
Compare
snejus
force-pushed
the
use-relative-paths
branch
2 times, most recently
from
4f52af6 to
c1b11aa
Compare
snejus
force-pushed
the
use-relative-paths
branch
from
4473908 to
8637db7
Compare
Comment on lines
+18
to
+27
| if not path or not os.path.isabs(path): | ||
| return path | ||
|
|
||
| music_dir = context.get_music_dir() | ||
| if not music_dir: | ||
| return path | ||
|
|
||
| if _is_same_path_or_child(path, music_dir): | ||
| return os.path.relpath(path, music_dir) | ||
|
|
There was a problem hiding this comment.
grug worry normalize_path_for_db do string prefix check before path normalized. If caller pass path like "/music/../other/file.mp3", helper think it child of music_dir and store "../other/file.mp3". Later expand make absolute outside library, break invariant "outside music dir stored as-is". Fix: normalize/abspath path (and music_dir) before _is_same_path_or_child/relpath, or use util.normpath on inputs inside normalize_path_for_db.
Suggested change
| if not path or not os.path.isabs(path): | |
| return path | |
| music_dir = context.get_music_dir() | |
| if not music_dir: | |
| return path | |
| if _is_same_path_or_child(path, music_dir): | |
| return os.path.relpath(path, music_dir) | |
| if not path: | |
| return path | |
| music_dir = context.get_music_dir() | |
| if not music_dir: | |
| return path | |
| # Normalize paths before checking directory relationship to avoid | |
| # misclassifying paths with ".." components as children of music_dir. | |
| normalized_path = util.normpath(path) | |
| normalized_music_dir = util.normpath(music_dir) | |
| if not os.path.isabs(normalized_path): | |
| return path | |
| if _is_same_path_or_child(normalized_path, normalized_music_dir): | |
| return os.path.relpath(normalized_path, normalized_music_dir) |
snejus
force-pushed
the
use-relative-paths
branch
5 times, most recently
from
318f2fd to
9a66310
Compare
Contributor
|
I would love to find some time in the following days to look into this in detail! Have you considered moving a library between operating systems yet? I could see that as one absolutly killer side effects for relative paths. Think portable external drive. If you havent thought about it yet, we might want to test that paths are normalized correctly between Windows and Posix. |
Member
Author
I hope you mean it's a positive side effect! Well, this new logic consistently uses Any ideas how could we possibly test it? |
Member
Author
|
I've just asked users to test this change - hopefully we have someone who can give it a go using these two filesystems. |
Convert item paths to relative on write and back to absolute on read, keeping the database free of hardcoded library directory. Fix tests to account for absolute path return values.
Move path relativization/expansion logic from Item._setitem/__getitem__ into dbcore layer (PathType.to_sql/from_sql and PathQuery), so all models benefit without per-model overrides. Propagate contextvars to pipeline and replaygain pool threads so the library root context variable is available during background processing.
Store paths relative to the music directory in the database instead of absolute paths. Add RelativePathMigration to handle existing absolute paths in `path` and `artpath` fields on startup. Also move `self.directory` assignment before `super().__init__()` so the migration can access it.
snejus
force-pushed
the
use-relative-paths
branch
from
9a66310 to
7f48736
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Migrate Item & Album Paths to Library-Relative Storage
Fixes: #133
Core Problem Solved
Before: Beets stored absolute file system paths in SQLite (e.g.
/home/user/Music/Artist/album/track.mp3). This made library databases non-portable — moving the music directory or sharing a database across machines broke all path references.After: Paths are stored relative to the music directory (e.g.
Artist/album/track.mp3), and expanded back to absolute paths transparently on read. The database is now portable.Architecture Changes
1. Context Variable for Music Directory (
beets/context.py)A new
contextvars.ContextVar(_music_dir_var) holds the active music directory, set once duringLibrary.__init__viacontext.set_music_dir(). This avoids passinglibrary.directorythrough every call stack.2. Path Relativization Moved to the DB Layer (
beets/dbcore/)Previously, path conversion lived in
Item._setitem/Item.__getitem__— model-specific overrides. It is now pushed down intoPathType.to_sql/PathType.from_sqlindbcore/types.py, through two helpers in the newbeets/dbcore/pathutils.py:normalize_path_for_db(path)expand_path_from_db(path)All models using
PathType(currentlyItemandAlbum) benefit automatically — no per-model overrides required.3.
PathQueryUpdated for Relative Storage (beets/dbcore/query.py)Queries like
path:/home/user/Music/Artistnow normalize the search term to its relative form before hitting the database, so SQL comparisons match stored values correctly. Bothcol_clause(SQL path) andmatch(in-memory path) usenormalize_path_for_db.4. One-Time Database Migration (
RelativePathMigration)Existing absolute paths in
pathandartpathcolumns are migrated on startup:self.directoryassignment was moved beforesuper().__init__()inLibrary.__init__so the migration can access the music dir when it runs.5. Context Propagation to Background Threads
The music dir context variable must be available in worker threads (pipeline stages, replaygain pool). Two propagation points were added:
beets/util/pipeline.py:Pipeline.run_parallel()snapshots the calling context withcontextvars.copy_context()and passes a per-thread copy to eachPipelineThread. Each stage coroutine is invoked viactx.run(...).beetsplug/replaygain.py: Pool workers and their callbacks are wrapped inctx.run(...)soexpand_path_from_dbworks correctly inside the process pool.beets/util/__init__.py:par_mapsimilarly propagates context into its thread pool workers.Data Flow: Read & Write Path
Key Invariants
item.pathalways returns an absolutebytespath.beetsplug/ipfs.pyfix)./) are skipped.