Save Apps in their own database table#373
Open
krokicki wants to merge 6 commits into
Open
Conversation
Three comments in apps/core.py claimed the poll loop dispatches `bjobs -u all` through a user's worker to fetch all jobs. The actual implementation passes an explicit list of cluster_job_ids and lets py-cluster-api run `bjobs` for just those IDs — cross-user visibility comes from LSF allowing ID-based queries across users, not from -u all. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
npm 10 warns "Unknown project config 'min-release-age'" because the key was renamed in npm 11. Using the new name silences the warning and is forward-compatible; npm 10 silently ignores it (the supply-chain floor only kicks in once we're on npm 11+). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This reverts commit dda7045.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Apps improvements
Move user apps from the manifest table to a dedicated DB table. Always use the cached manifest in the DB, only syncing from disk when they change. The main benefit is that this approach makes the Apps page load much faster.
This PR also contains some minor misc fixes to unrelated stuff like the devcontainer, and code comments.
Summary
user_preferencesto a dedicateduser_appstable, with each row caching the resolvedAppManifestso the hot path is a single SELECT instead of disk I/O on every request.get_or_load_manifest(cache-first read) andrefresh_cached_manifest(re-read from disk after clone/pull) so the DB cache stays in lockstep with the on-disk YAML.~/.gitconfiginto the devcontainer so git operations inside the container use the user's identity.bjobs -u all.Details
user_appstableNew Alembic migration
c4e8a7d92b15_add_user_apps_tableadds:user_apps(id, username, url, manifest_path, name, description, branch, manifest JSON, added_at, updated_at)(username, url, manifest_path)usernameCRUD helpers added to
database.py:list_user_apps,get_user_app,upsert_user_app,delete_user_app. The old JSON-blob-in-preferences path is removed from the apps endpoints.Manifest cache
get_or_load_manifestreads the cachedmanifestJSON from the row and validates it. On NULL or schema-validation failure, it falls back tofetch_app_manifest(disk) and writes the fresh value back. If no row exists (preview for not-yet-installed apps), it just reads from disk without inserting.refresh_cached_manifestis called after operations that mutate the on-disk YAML (e.g.submit_jobwithpull_latest=Trueagainst the manifest's own repo), keeping the cache consistent.GET /api/appsno longer hits disk per app — it returns cached manifests and lazily backfills any rows whose stored manifest is missing/stale.POST /api/apps/fetch-manifest,POST /api/apps,POST /api/apps/update, andDELETE /api/appsall go through the new table.Devcontainer
.devcontainer/devcontainer.jsonmounts${localEnv:HOME}/.gitconfigread-only into the container so commits/pulls inside the container use the host git identity.Polling comments
Updated comments in
apps/core.pyto accurately describe how the poll loop dispatches apollaction with explicitcluster_job_idsthrough one user's worker — LSF allows cross-user lookup by ID, so a single worker call covers all users' jobs. No behavior change.Other
FgButton.stories.tsx.minimum-release-agenpm config rename.@StephanPreibisch @allison-truhlar