realm-server: HTTPS+HTTP/2 in local dev

.github/actions/init/action.yml

@@ -31,3 +31,20 @@ runs:
       if: ${{ steps.cache.outputs.cache-hit != 'true' }}
       shell: bash
       run: pnpm store prune
+
+    # Provision the mandatory dev cert so the realm-server can speak
+    # HTTPS+HTTP/2 — the only protocol it supports. Local devs get this
+    # via `mise run infra:ensure-dev-cert`; CI runs it explicitly here
+    # so every downstream job starts with the cert in
+    # ~/.local/share/boxel/dev-certs/ and `NODE_EXTRA_CA_CERTS` pointed
+    # at mkcert's root. See packages/realm-server/server.ts and
+    # mise-tasks/infra/ensure-dev-cert.
+    - name: Install mkcert
+      shell: bash
+      run: |
+        sudo apt-get update
+        sudo apt-get install -y mkcert libnss3-tools
+
+    - name: Provision dev TLS cert
+      shell: bash
+      run: mise run infra:ensure-dev-cert

.github/workflows/ci.yaml

@@ -980,7 +980,13 @@ jobs:
       - name: Start dev stack (icons + host-dist + base realm + prerenderer)
         run: |
           mise run test-services:matrix | tee -a /tmp/server.log &
-          timeout 600 bash -c 'until curl -sf http://localhost:4200 > /dev/null && curl -sf -H "Accept: application/vnd.api+json" http://localhost:4201/base/_readiness-check > /dev/null; do sleep 2; done'
+          # `-k` skips cert verification (the local realm-server speaks
+          # HTTPS with the mkcert leaf cert). `-w '%{http_code}'` + grep
+          # ensures we treat 3xx as still-not-ready instead of letting
+          # `-f` return 0 on the dispatcher's redirect response, which
+          # would race the tests ahead of the base realm finishing its
+          # initial index.
+          timeout 600 bash -c 'until curl -sk -o /dev/null -w "%{http_code}" http://localhost:4200/ | grep -qx 200 && curl -sk -o /dev/null -w "%{http_code}" -H "Accept: application/vnd.api+json" https://localhost:4201/base/_readiness-check | grep -qx 200; do sleep 2; done'
       - name: Run integration tests
         run: pnpm test:integration
         working-directory: packages/boxel-cli

QUICKSTART.md

@@ -2,10 +2,17 @@
 
 To build the entire repository and run the application, follow these steps:
 
-1. The 2 main system dependencies to install are:
-
+1. The system dependencies to install are:
    - [mise](https://mise.jdx.dev/getting-started.html)
    - [docker](https://docs.docker.com/get-docker/)
+   - [mkcert](https://github.com/FiloSottile/mkcert) — provisions the
+     local TLS cert the realm-server needs to speak HTTPS+HTTP/2 (local
+     dev has no HTTP fallback). Install with
+     `sudo apt install -y mkcert libnss3-tools` on Debian/Ubuntu or
+     `brew install mkcert nss` on macOS. After install, run
+     `mise run infra:ensure-dev-cert` once before the first
+     `mise run dev` / `pnpm start:all`; subsequent runs are a no-op. See
+     the repo-root [README](README.md#local-https-dev-access) for details.
 
 2. Clone the repo:
 
@@ -52,7 +59,7 @@ To build the entire repository and run the application, follow these steps:
    Note: Ensure that the realm-server is completely started by looking out for tor the test-realm indexing output.
 
    ```zsh
-   Realm http://localhost:4202/test/ has started ({
+   Realm https://localhost:4202/test/ has started ({
    "instancesIndexed": 8,
    "instanceErrors": 0,
    "moduleErrors": 0
@@ -76,19 +83,16 @@ To build the entire repository and run the application, follow these steps:
    Visit http://localhost:8080. Type in Username = "admin", Password: "password" Homeserver URL: http://localhost:8008
 
 10. Host App
-
-    - Visit http://localhost:4201/
+    - Visit http://localhost:4200/
     - Enter the registration flow and create a Boxel Account
     - When prompted for an authentication token, type in "dev-token"
 
 11. Validate email for login
-
     - Visit SMTP UI at http://localhost:5001/
     - Validate email
-    - Go back to Host http://localhost:4201/ and login
+    - Go back to Host http://localhost:4200/ and login
 
 12. Perform "Setup up Secure Payment Method" flow
-
     - More detailed steps can be found in our [README](README.md) Payment Setup section
 
 13. Run ai bot (Optional):

README.md

@@ -34,10 +34,10 @@ For a quickstart, see [here](./QUICKSTART.md)
 
 Two catalog realms run side by side. The **Cardstack Catalog** (served from [cardstack/boxel-catalog](https://github.com/cardstack/boxel-catalog)) is the source of truth for new development and the destination for community submissions. The **Legacy Catalog** (shipped from this monorepo) remains available during the deprecation window for content that hasn't been migrated upstream.
 
-| Realm                  | Source                                                                                  | URL path           | Purpose                                                                                                                          |
-| ---------------------- | --------------------------------------------------------------------------------------- | ------------------ | -------------------------------------------------------------------------------------------------------------------------------- |
-| **Cardstack Catalog**  | `packages/catalog` (clones [boxel-catalog](https://github.com/cardstack/boxel-catalog)) | `/catalog/`        | Official catalog. New listings and community submissions land here via `pr-listing-create` PRs to `cardstack/boxel-catalog`.     |
-| **Legacy Catalog**     | `packages/catalog-realm`                                                                | `/legacy-catalog/` | Historical catalog shipped from this repo. Kept visible in the workspace chooser while existing content migrates upstream.       |
+| Realm                 | Source                                                                                  | URL path           | Purpose                                                                                                                      |
+| --------------------- | --------------------------------------------------------------------------------------- | ------------------ | ---------------------------------------------------------------------------------------------------------------------------- |
+| **Cardstack Catalog** | `packages/catalog` (clones [boxel-catalog](https://github.com/cardstack/boxel-catalog)) | `/catalog/`        | Official catalog. New listings and community submissions land here via `pr-listing-create` PRs to `cardstack/boxel-catalog`. |
+| **Legacy Catalog**    | `packages/catalog-realm`                                                                | `/legacy-catalog/` | Historical catalog shipped from this repo. Kept visible in the workspace chooser while existing content migrates upstream.   |
 
 Both catalogs appear in the workspace chooser by default; the Cardstack Catalog is sorted first, Legacy Catalog last. Both are controlled by the same `SKIP_CATALOG` flag — setting `SKIP_CATALOG=true` skips setup and startup for both.
 
@@ -137,7 +137,7 @@ In order to run the realm server hosted app:
 1. `mise run services:host-build` to re-build the host app (this step can be omitted if you do not want host app re-builds)
 2. `mise run dev` to serve the base and experiments realms
 
-You can visit the URL of each realm server to view that realm's app. So for instance, the base realm's app is available at `http://localhost:4201/base` and the experiments realm's app is at `http://localhost:4201/experiments`.
+The recommended way to view a realm's app is the host vite dev server at `http://localhost:4200` — open it and navigate via the workspace chooser. The realm-server itself terminates HTTPS+HTTP/2 on `https://localhost:4201` (see "Local HTTPS dev access" below for the one-time cert setup), and the in-browser host on `:4200` makes its realm fetches over that https origin so the indexing path multiplexes per Chrome's HTTP/2 connection rules. Visiting `https://localhost:4201/<realm>` directly does work but will surface mixed-content warnings, because the host bundle and icons it loads are still served over plain HTTP on `:4200`/`:4206`.
 
 Live reloads are not available in this mode, however, if you use start the server with the environment variable `DISABLE_MODULE_CACHING=true` you can just refresh the page to grab the latest code changes if you are running rebuilds (step #1 and #2 above).
 
@@ -191,10 +191,10 @@ Here's what is spun up with `mise run dev`:
 
 | Port  | Description                                                                                   | `mise run dev` | `mise run services:realm-server-base` |
 | ----- | --------------------------------------------------------------------------------------------- | -------------- | ------------------------------------- |
-| :4201 | `/base` base realm                                                                            | ✅             | ✅                                    |
+| :4201 | `/base` base realm (HTTPS+HTTP/2; see "Local HTTPS dev access" below)                         | ✅             | ✅                                    |
 | :4201 | `/skills` skills realm                                                                        | ✅             | 🚫                                    |
 | :4201 | `/experiments` experiments realm                                                              | ✅             | 🚫                                    |
-| :4202 | `/test` host test realm, `/node-test` node test realm                                         | ✅             | 🚫                                    |
+| :4202 | `/test` host test realm, `/node-test` node test realm (HTTPS+HTTP/2)                          | ✅             | 🚫                                    |
 | :4205 | `/test` realm for matrix client tests (playwright controlled)                                 | 🚫             | 🚫                                    |
 | :4206 | Boxel icons server                                                                            | ✅             | 🚫                                    |
 | :4210 | Development Worker Manager (spins up 1 worker by default)                                     | ✅             | 🚫                                    |
@@ -207,9 +207,104 @@ Here's what is spun up with `mise run dev`:
 | :5435 | Postgres DB                                                                                   | ✅             | 🚫                                    |
 | :8008 | Matrix synapse server                                                                         | ✅             | 🚫                                    |
 
+#### Local HTTPS dev access
+
+##### Why
+
+Heavy aggregator cards (Cohort, dashboards) fan out 80+ federated-search
+requests per render. Chrome — including the headless Chromium that
+drives the prerender — caps any single origin at 6 concurrent HTTP/1.1
+connections, so the 80+ requests serialize. A single cohort render
+takes ~4–5 minutes under that ceiling. HTTP/2 multiplexes them over one
+connection and the same render finishes in seconds. Browsers only do
+HTTP/2 over TLS, so the local realm-server terminates a leaf cert.
+
+##### Setup
+
+```
+mise run infra:ensure-dev-cert
+```
+
+That task:
+
+1. Requires [`mkcert`](https://github.com/FiloSottile/mkcert)
+   (`sudo apt install -y mkcert libnss3-tools` on Debian/Ubuntu;
+   `brew install mkcert nss` on macOS). If it's missing, the task
+   prints these instructions and exits non-zero — local dev now
+   speaks HTTPS only and has no HTTP fallback.
+2. Attempts `mkcert -install` once (one-time sudo prompt) so mkcert's
+   root CA lands in your system trust store. Your normal browser then
+   silently accepts the cert. If you decline the sudo prompt the cert
+   is still generated, indexing keeps working, and you can run
+   `mkcert -install` later when convenient.
+3. Generates `~/.local/share/boxel/dev-certs/{localhost.pem,
+localhost-key.pem}`. Idempotent — re-runs are a no-op until the
+   cert is within 7 days of expiry.
+
+After provisioning, `mise run dev` (and `mise run dev-all`) automatically
+brings the realm-server up on `https://localhost:4201` (and
+`https://localhost:4202` for the test-realms server). Node clients
+(worker, scripts, prerender Node-side) trust the cert via
+`NODE_EXTRA_CA_CERTS` (pointed at mkcert's root by `env-vars.sh`); the
+prerender Chromium uses `--ignore-certificate-errors` for belt-and-
+suspenders coverage.
+
+##### Migration after pulling this change
+
+The canonical realm URLs are now `https://localhost:4201/…` and
+`https://localhost:4202/…`, so every row that was written under the old
+`http://localhost:42XX/…` canonical needs its URLs rewritten in place
+— PK columns, FK columns, JSONB documents (`pristine_doc`, `search_doc`,
+`error_doc`, `deps`, `value`, `headers`, etc.), and rendered HTML /
+markdown payloads. The repo ships an auto-run migration that handles
+all of it:
+
+- `pnpm migrate` (which `mise run dev` runs via `--migrateDB`) picks up
+  `packages/postgres/migrations/1779100257124_canonical-url-http-to-https.js`
+  on the next boot.
+- That migration walks `information_schema.columns`, finds every
+  text/varchar/jsonb column on every public table (skipping `modules`,
+  which the realm-server truncates on startup anyway), and runs an
+  in-place `REPLACE` for the two canonicals. WHERE-filtered so it only
+  touches rows that still contain the old URL — idempotent, and a
+  no-op in production (where the canonical URL is never `localhost`).
+
+After the migration runs, the realm-server boots normally on
+`https://localhost:4201/`. The same-port HTTP→HTTPS dispatcher catches
+any lingering `http://localhost:4201/…` requests (e.g. you typed it
+into a browser, or a card still has a stale URL in a rendered HTML
+attribute) and 301-redirects to the canonical https origin.
+
+Personal realm files under `realms/localhost_4201/**/*.json` may still
+have `id`/`relationships` URLs spelled `http://localhost:4201/…`. The
+indexer re-derives canonical URLs from the realm mount root, so those
+files index cleanly under the new canonical and the redirect handles
+runtime fetches; cleaning up the on-disk strings is optional (a
+`sed -i 's|http://localhost:4201|https://localhost:4201|g'` across the
+realm dir does it in one shot).
+
+##### Verify
+
+```
+curl -kI --http2 https://localhost:4201/_alive
+```
+
+Look for `HTTP/2 200`. The `mise run dev` log also confirms with
+`Realm server listening on port 4201 (https/h2)`.
+
+##### CI / hermetic test harness
+
+CI runs the same `mise run infra:ensure-dev-cert` step out of its init
+action (see `.github/actions/init/action.yml`) — mkcert is installed
+via apt and the cert is provisioned before any test job starts. So CI
+realm-servers boot HTTPS+HTTP/2 on `https://localhost:4201/…` exactly
+like local dev. Test harnesses that launch their own Chromium pass
+`--ignore-certificate-errors` so they don't need the system trust
+store; Node clients pick up the cert via `NODE_EXTRA_CA_CERTS`.
+
 #### Using `mise run services:realm-server`
 
-You can also use `mise run services:realm-server` if you want the functionality of `mise run dev`, but without running the test realms. This will enable you to open http://localhost:4201 and allow to select between the cards in the /base and /experiments realm. You must also make sure to run `mise run services:worker` in order to start the workers which are normally started in `mise run dev`.
+You can also use `mise run services:realm-server` if you want the functionality of `mise run dev`, but without running the test realms. Visit `http://localhost:4200` (the vite host) to navigate the workspace — the host bundle there fetches realm data over the realm-server's https origin on `:4201`. You must also make sure to run `mise run services:worker` in order to start the workers which are normally started in `mise run dev`.
 
 #### Indexing dashboard
 
@@ -610,12 +705,12 @@ BOXEL_ENVIRONMENT=parallel mise run services:ai-bot
 
 In environment mode, services are available at:
 
-| Service       | Hostname                                  |
-| ------------- | ----------------------------------------- |
-| Host app      | `http://host.<slug>.localhost`            |
-| Realm server  | `http://realm-server.<slug>.localhost`    |
-| Matrix        | `http://matrix.<slug>.localhost`          |
-| Icons         | `http://icons.<slug>.localhost`           |
+| Service      | Hostname                               |
+| ------------ | -------------------------------------- |
+| Host app     | `http://host.<slug>.localhost`         |
+| Realm server | `http://realm-server.<slug>.localhost` |
+| Matrix       | `http://matrix.<slug>.localhost`       |
+| Icons        | `http://icons.<slug>.localhost`        |
 
 Where `<slug>` is the lowercased, sanitized form of `BOXEL_ENVIRONMENT` (e.g., `feature/my-branch` becomes `feature-my-branch`).
 

docs/commands-in-headless-chrome.md

@@ -29,9 +29,9 @@ const event = {
   type: 'app.boxel.bot-trigger',
   content: {
     type: 'show-card',
-    realm: 'http://localhost:4201/experiments/',
+    realm: 'https://localhost:4201/experiments/',
     input: {
-      cardId: 'http://localhost:4201/experiments/Author/jane-doe',
+      cardId: 'https://localhost:4201/experiments/Author/jane-doe',
       format: 'isolated',
     },
   },
@@ -73,9 +73,9 @@ const event = {
   type: 'app.boxel.bot-trigger',
   content: {
     type: 'show-card',
-    realm: 'http://localhost:4201/experiments/',
+    realm: 'https://localhost:4201/experiments/',
     input: {
-      cardId: 'http://localhost:4201/experiments/Author/jane-doe',
+      cardId: 'https://localhost:4201/experiments/Author/jane-doe',
       format: 'isolated',
     },
   },
@@ -91,12 +91,12 @@ const event = {
 
 ```json
 {
-  "realmURL": "http://localhost:4201/experiments/",
+  "realmURL": "https://localhost:4201/experiments/",
   "realmUsername": "@alice:localhost",
   "runAs": "@alice:localhost",
   "command": "@cardstack/boxel-host/commands/show-card/default",
   "commandInput": {
-    "cardId": "http://localhost:4201/experiments/Author/jane-doe",
+    "cardId": "https://localhost:4201/experiments/Author/jane-doe",
     "format": "isolated"
   }
 }
@@ -125,7 +125,7 @@ type CommandRunnerRouteParams = {
 const requestId = '6f5508cf-0f10-44a8-a288-0f11f74c4f20';
 const command = '@cardstack/boxel-host/commands/show-card/default';
 const input = {
-  cardId: 'http://localhost:4201/experiments/Author/jane-doe',
+  cardId: 'https://localhost:4201/experiments/Author/jane-doe',
   format: 'isolated',
 };
 const nonce = '2';
@@ -170,7 +170,7 @@ localStorage.setItem(
   JSON.stringify({
     command: '@cardstack/boxel-host/commands/show-card/default',
     input: {
-      cardId: 'http://localhost:4201/experiments/Author/jane-doe',
+      cardId: 'https://localhost:4201/experiments/Author/jane-doe',
       format: 'isolated',
     },
     nonce,