Skip to content

Save Apps in their own database table#373

Open
krokicki wants to merge 6 commits into
mainfrom
apps-improvements
Open

Save Apps in their own database table#373
krokicki wants to merge 6 commits into
mainfrom
apps-improvements

Conversation

@krokicki
Copy link
Copy Markdown
Member

Apps improvements

Move user apps from the manifest table to a dedicated DB table. Always use the cached manifest in the DB, only syncing from disk when they change. The main benefit is that this approach makes the Apps page load much faster.

This PR also contains some minor misc fixes to unrelated stuff like the devcontainer, and code comments.

Summary

  • Move user-installed apps from a JSON blob in user_preferences to a dedicated user_apps table, with each row caching the resolved AppManifest so the hot path is a single SELECT instead of disk I/O on every request.
  • Add get_or_load_manifest (cache-first read) and refresh_cached_manifest (re-read from disk after clone/pull) so the DB cache stays in lockstep with the on-disk YAML.
  • Mount the host ~/.gitconfig into the devcontainer so git operations inside the container use the user's identity.
  • Fix misleading comments around job polling — py-cluster-api queries LSF by explicit job IDs, not bjobs -u all.

Details

user_apps table

New Alembic migration c4e8a7d92b15_add_user_apps_table adds:

  • user_apps(id, username, url, manifest_path, name, description, branch, manifest JSON, added_at, updated_at)
  • Unique constraint on (username, url, manifest_path)
  • Index on username

CRUD helpers added to database.py: list_user_apps, get_user_app, upsert_user_app, delete_user_app. The old JSON-blob-in-preferences path is removed from the apps endpoints.

Manifest cache

  • get_or_load_manifest reads the cached manifest JSON from the row and validates it. On NULL or schema-validation failure, it falls back to fetch_app_manifest (disk) and writes the fresh value back. If no row exists (preview for not-yet-installed apps), it just reads from disk without inserting.
  • refresh_cached_manifest is called after operations that mutate the on-disk YAML (e.g. submit_job with pull_latest=True against the manifest's own repo), keeping the cache consistent.
  • GET /api/apps no longer hits disk per app — it returns cached manifests and lazily backfills any rows whose stored manifest is missing/stale.
  • POST /api/apps/fetch-manifest, POST /api/apps, POST /api/apps/update, and DELETE /api/apps all go through the new table.

Devcontainer

.devcontainer/devcontainer.json mounts ${localEnv:HOME}/.gitconfig read-only into the container so commits/pulls inside the container use the host git identity.

Polling comments

Updated comments in apps/core.py to accurately describe how the poll loop dispatches a poll action with explicit cluster_job_ids through one user's worker — LSF allows cross-user lookup by ID, so a single worker call covers all users' jobs. No behavior change.

Other

  • Lint fix in FgButton.stories.tsx.
  • Reverted the earlier minimum-release-age npm config rename.

@StephanPreibisch @allison-truhlar

krokicki and others added 6 commits May 24, 2026 17:52
Three comments in apps/core.py claimed the poll loop dispatches
`bjobs -u all` through a user's worker to fetch all jobs. The actual
implementation passes an explicit list of cluster_job_ids and lets
py-cluster-api run `bjobs` for just those IDs — cross-user visibility
comes from LSF allowing ID-based queries across users, not from -u all.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
npm 10 warns "Unknown project config 'min-release-age'" because the key
was renamed in npm 11. Using the new name silences the warning and is
forward-compatible; npm 10 silently ignores it (the supply-chain floor
only kicks in once we're on npm 11+).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant