DiamondLightSource · vredchenko · Apr 27, 2026 · Apr 21, 2026
diff --git a/docs/backend/api-server.md b/docs/backend/api-server.md
@@ -5,17 +5,17 @@ The core backend service providing HTTP API, database operations, and message qu
 ## Backend Operations
 
 ```bash
-# create env and launch service stack locally:
+# create env and launch service stack locally (from smartem-devtools checkout):
 ./scripts/k8s/dev-k8s.sh up
 
 # launch RabbitMQ worker (consumer)
 python -m smartem_backend.consumer              # ERROR level (default)
 python -m smartem_backend.consumer -v           # INFO level
 python -m smartem_backend.consumer -vv          # DEBUG level
 
-# simulating an system event:
-python -m smartem_backend.simulate_msg --help # to see a list of options
-./tools/simulate-messages.sh # run a simulation, triggering system events in sequence
+# simulating a system event (see tools.md for full usage):
+uv run python tools/external_message_simulator.py list-messages  # see available message types
+uv run python tools/external_message_simulator.py workflow-simulation --gridsquare-id "DEV_001"  # run a workflow
 
 # run HTTP API (recommended - uses proper module entry point):
 python -m smartem_backend.api_server

diff --git a/docs/backend/database.md b/docs/backend/database.md
@@ -35,8 +35,10 @@ This will:
 1. Create the Alembic version tracking table
 2. Apply migration 001: Create core SmartEM schema baseline (acquisition, grid, gridsquare, micrograph, foilhole, atlas, etc.)
 3. Apply migration 002: Add performance indexes for acquisition datetime queries
-4. Apply migration 003: Add prediction model tables
-5. Apply remaining migrations for quality metrics, training tables, and schema fixes
+4. Apply migration 003: Add SciFi robot prediction model tables
+5. Apply migration 004: Add quality metric and training tables
+6. Apply migration 005: Fix schema drift and sync indexes
+7. Apply migration 006: Add suggested acquisition index
 
 ### Running Migrations
 
@@ -232,7 +234,7 @@ python -m alembic revision --autogenerate -m "Sync models with database"
 
 **Missing baseline schema**: If you encounter errors like "relation 'gridsquare' does not exist" when running migrations:
 
-This indicates that migration 002 is trying to create indexes on tables that don't exist yet. This happens when the baseline schema migration is missing or not properly applied. The fix is to ensure the baseline migration (6e6302f1ccb6) is applied before migration 002.
+This indicates that migration 002 is trying to create indexes on tables that don't exist yet. This happens when the baseline schema migration is missing or not properly applied. The fix is to ensure migration 001 (`2025_09_18_1042-001_create_core_smartem_schema_baseline.py`) is applied before migration 002.
 
 ```bash
 # Solution: Run the complete migration sequence from scratch

diff --git a/docs/development/generate-docs.md b/docs/development/generate-docs.md
@@ -1,8 +1,44 @@
 # Generate Documentation
 
-## Documentation Generation
+## Overview
 
-```sh
-tox -e docs
-python -m http.server -d build/html
+Documentation lives as Markdown files in the `docs/` directory. During build, these are automatically converted to MDX
+and copied into the webui for rendering. The webui is deployed to GitHub Pages.
+
+## How it works
+
+1. **Source**: Markdown files in `docs/` (this is what you edit)
+2. **Sync**: `webui/scripts/generate-mdx-docs.ts` converts `.md` to `.mdx` and copies them into `webui/src/docs/` (gitignored)
+3. **Build**: The sync runs automatically as part of `npm run prebuild` and is watched during `npm run dev`
+4. **Deploy**: The `deploy-webui.yml` workflow builds and publishes to GitHub Pages on push to `main` when `docs/**` changes
+
+## Local preview
+
+```bash
+cd webui
+npm install
+npm run dev
 ```
+
+This starts the Vite dev server with live reload. Changes to `docs/*.md` files are picked up automatically.
+
+## Building for production
+
+```bash
+cd webui
+npm install
+npm run build
+```
+
+The built output goes to `webui/dist/`.
+
+## Navigation structure
+
+The `index.md` files in each `docs/` subdirectory contain `toctree` directives that define navigation order and
+grouping. The webui reads these during build to generate the sidebar navigation. When adding a new page, add its
+filename (without extension) to the relevant `index.md` toctree.
+
+## Deployment
+
+Deployment is automatic via the `deploy-webui.yml` GitHub Actions workflow. It triggers on pushes to `main` that
+modify files in `webui/`, `core/`, or `docs/`. Manual deployment is also available via `workflow_dispatch`.
diff --git a/docs/development/tools.md b/docs/development/tools.md
@@ -143,6 +143,57 @@ This tool is particularly useful for:
 
 Both tools work together to simulate the complete external data flow into the SmartEM system, enabling comprehensive testing without requiring actual microscopy equipment or external processing systems.
 
+## Database Tools
+
+### Schema Drift Detection
+
+Detects when SQLModel definitions have changed without corresponding Alembic migrations. Creates a temporary PostgreSQL database, applies existing migrations, then runs autogenerate to check for differences. Also used in CI via the `_schema_drift.yml` workflow.
+
+```bash
+# Requires a running PostgreSQL instance (e.g. from dev-k8s.sh up)
+uv run python tools/check_schema_drift.py
+```
+
+Exit code 0 means the schema is in sync; exit code 1 means drift was detected (or an error occurred).
+
+### Database Table Row Counts
+
+Connects to PostgreSQL and outputs row counts for every SmartEM table. Useful for verifying data after E2E tests or checking the state of a development database.
+
+```bash
+uv run python tools/db_table_totals.py
+```
+
+## API Documentation Generation
+
+### Generate API Docs
+
+Generates API documentation from OpenAPI specs into `docs/api/`. Processes two APIs:
+
+- **Athena API**: copies the original spec from `docs/athena-decision-service-api-spec.json` and adds a local mock server entry
+- **SmartEM API**: imports the FastAPI app and extracts the OpenAPI schema at runtime
+
+```bash
+uv run python tools/generate_api_docs.py
+```
+
+Output is written to `docs/api/athena/swagger.json` and `docs/api/smartem/swagger.json`.
+
+## Miscellaneous Tools
+
+### Create ISO Image
+
+Creates an ISO image from a source directory using `mkisofs` or `genisoimage`.
+
+```bash
+./tools/makeiso.sh <source_directory> [output.iso]
+
+# Example: create an ISO from a test dataset
+./tools/makeiso.sh ./tests/testdata/bi37708-28 dataset.iso
+```
+
+If the output filename is omitted, it defaults to `<source_directory>.iso`.
+
 ## Additional Development Commands
 
 ### Pre-commit Workflow

diff --git a/docs/glossary.md b/docs/glossary.md
@@ -19,9 +19,9 @@ collection strategies.
 
 ## ARIA
 
-Automated Real-time Image Analysis system developed by INSTRUCT-ERIC for real-time processing and quality assessment
-of cryo-electron microscopy data during acquisition. More information available at:
-https://instruct-eric.org/help/about-aria
+Central metadata repository for structural biology data from multiple facilities, managed by INSTRUCT-ERIC. ARIA
+collects and organises acquisition metadata, sample information, and processing results deposited from facilities via
+the FandanGO plugin framework. More information available at: https://instruct-eric.org/help/about-aria
 
 ## Atlas
 

diff --git a/docs/operations/index.md b/docs/operations/index.md
@@ -5,6 +5,8 @@ Cross-cutting operational documentation for deploying and running SmartEM in pro
 ```{toctree}
 :maxdepth: 1
 
+releasing
+publish-smartem-workspace-to-pypi
 kubernetes
 kubernetes-secrets
 containerization
@@ -16,6 +18,11 @@ logging
 
 ## Topics
 
+### Releases
+
+- [Release Procedure](releasing.md) - How to release smartem-decisions, smartem-epuplayer, and smartem-workspace
+- [PyPI Token Setup](publish-smartem-workspace-to-pypi.md) - First-time PyPI account and token configuration
+
 ### Kubernetes
 
 - [Kubernetes Deployment](kubernetes.md) - Deploying SmartEM to Kubernetes clusters

diff --git a/docs/operations/releasing.md b/docs/operations/releasing.md
@@ -0,0 +1,198 @@
+# Release Procedure
+
+How to release the three packages in the SmartEM ecosystem. All releases are tag-driven via GitHub Actions.
+
+## Release types
+
+| Type | Trigger | Purpose |
+|------|---------|---------|
+| **RC (Release Candidate)** | Automatic on push to `main` (when path filters match) | Pre-release for testing |
+| **Stable** | Push a version tag | Production release |
+| **Manual** | `workflow_dispatch` with `rc` or `stable` choice | Emergency or ad-hoc releases |
+
+## General flow
+
+1. Develop on a feature branch
+2. Merge PR to `main`
+3. CI automatically creates an RC release (if path filters match and no stable tag exists for the current version)
+4. When ready for stable: push a version tag
+5. CI runs tests, builds artifacts, publishes to PyPI/GHCR, and creates a GitHub Release
+
+## smartem-decisions
+
+**Repository:** [DiamondLightSource/smartem-decisions](https://github.com/DiamondLightSource/smartem-decisions)
+**Workflow:** `release-smartem-decisions.yml`
+**Versioning:** `setuptools_scm` (version derived from git tags automatically)
+**Tag prefix:** `smartem-decisions-v`
+
+### Path filters
+
+The workflow triggers on changes to:
+- `src/smartem_agent/**`
+- `src/smartem_common/**`
+- `src/smartem_backend/api_client.py`
+
+### Artifacts produced
+
+- Python wheel (PyPI, stable only)
+- Windows executable (`smartem-agent-windows-v{VERSION}.exe`)
+- Docker container image (GHCR, stable only)
+- GitHub Release with all artifacts
+
+### Releasing a stable version
+
+```bash
+# Version is determined by setuptools_scm from the tag
+git tag smartem-decisions-v1.2.0
+git push origin smartem-decisions-v1.2.0
+```
+
+CI will:
+1. Run tests on Linux and Windows
+2. Lint with ruff and pyright
+3. Build Python wheel and Windows exe
+4. Smoke test the Windows exe
+5. Publish wheel to PyPI (trusted publishing via OIDC)
+6. Build and push Docker image to GHCR (`ghcr.io/DiamondLightSource/smartem-decisions:{VERSION}`)
+7. Create a GitHub Release with release notes, wheel, and exe attached
+
+## smartem-epuplayer
+
+**Repository:** [DiamondLightSource/smartem-devtools](https://github.com/DiamondLightSource/smartem-devtools)
+**Workflow:** `release-smartem-epuplayer.yml`
+**Versioning:** Manual — set in both `pyproject.toml` and `smartem_epuplayer/__init__.py` (must match)
+**Tag prefix:** `epuplayer-v`
+**Package directory:** `packages/smartem-epuplayer/`
+
+### Path filters
+
+The workflow triggers on changes to:
+- `packages/smartem-epuplayer/**`
+
+### Artifacts produced
+
+- Python wheel (PyPI, stable only)
+- Windows executable (`epuplayer-windows-v{VERSION}.exe`)
+- GitHub Release with all artifacts
+
+### Bumping the version
+
+Before tagging, update the version in both places:
+
+```python
+# packages/smartem-epuplayer/pyproject.toml
+version = "0.3.0"
+
+# packages/smartem-epuplayer/smartem_epuplayer/__init__.py
+__version__ = "0.3.0"
+```
+
+The CI will fail if these don't match.
+
+### Releasing a stable version
+
+```bash
+git tag epuplayer-v0.3.0
+git push origin epuplayer-v0.3.0
+```
+
+## smartem-workspace
+
+**Repository:** [DiamondLightSource/smartem-devtools](https://github.com/DiamondLightSource/smartem-devtools)
+**Workflow:** `release-smartem-workspace.yml`
+**Versioning:** Manual — set in both `pyproject.toml` and `smartem_workspace/__init__.py` (must match)
+**Tag prefix:** `smartem-workspace-v`
+**Package directory:** `packages/smartem-workspace/`
+
+### Path filters
+
+The workflow triggers on changes to:
+- `packages/smartem-workspace/**`
+
+### Artifacts produced
+
+- Python wheel (PyPI, stable only)
+- GitHub Release
+
+### Bumping the version
+
+Same pattern as epuplayer — update both files:
+
+```python
+# packages/smartem-workspace/pyproject.toml
+version = "0.4.0"
+
+# packages/smartem-workspace/smartem_workspace/__init__.py
+__version__ = "0.4.0"
+```
+
+### Releasing a stable version
+
+```bash
+git tag smartem-workspace-v0.4.0
+git push origin smartem-workspace-v0.4.0
+```
+
+### uvx cache gotcha
+
+`smartem-workspace` is commonly run via `uvx`. After publishing a new version, users may still get the old version due
+to uvx caching. They need to either:
+
+```bash
+# Force refresh the cache
+uvx --refresh smartem-workspace --help
+
+# Or pin to latest explicitly
+uvx smartem-workspace@latest --help
+```
+
+## Manual / emergency releases
+
+All three workflows support `workflow_dispatch`. Go to the Actions tab of the relevant repository, select the release
+workflow, click "Run workflow", and choose `rc` or `stable`.
+
+This is useful when:
+- An RC wasn't triggered automatically (e.g. path filters didn't match)
+- You need to re-release after a CI fix
+- You want a stable release without pushing a tag first
+
+## Verifying a release
+
+### GitHub Releases
+
+Check the Releases page of the relevant repository. Both RC and stable releases appear there (RCs are marked as
+pre-release).
+
+### PyPI (stable only)
+
+```bash
+# smartem-decisions
+pip install smartem-decisions==1.2.0
+
+# smartem-epuplayer
+pip install smartem-epuplayer==0.3.0
+
+# smartem-workspace
+pip install smartem-workspace==0.4.0
+```
+
+### Docker (smartem-decisions stable only)
+
+```bash
+docker pull ghcr.io/diamondlightsource/smartem-decisions:1.2.0
+```
+
+### Windows executable
+
+Download from the GitHub Release assets and run:
+
+```bash
+smartem-agent-windows-v1.2.0.exe --help
+epuplayer-windows-v0.3.0.exe --help
+```
+
+## First-time PyPI setup
+
+If you're setting up PyPI publishing for the first time (new package or new repository), see
+[PyPI Token Setup](publish-smartem-workspace-to-pypi.md) for account creation, token generation, and GitHub Secrets
+configuration.