Rich text

.github/workflows/run_tests.yml

@@ -33,7 +33,7 @@ jobs:
         with:
           python-version: ${{ matrix.python }}
 
-      - name: Install libvips
+      - name: Install system dependencies
         if: runner.os == 'Linux'
         run: |
           sudo apt-get install -y libvips-dev

AGENTS.md

@@ -10,6 +10,8 @@ Before suggesting removal or simplification of existing configuration (editable
 
 After creating or updating a python file, run `uv run ruff check --fix ${file} 2>/dev/null || true` to fix any linting errors.
 
+The docstrings are written in Markdown (not reStructuredText).
+
 ## Testing
 
 Always run `uv run pytest` as the test runner command. Do not use `pytest` directly or any other test runner unless explicitly told otherwise.

LEXICON.md

@@ -0,0 +1,91 @@
+# Lexicon
+
+Project-wide glossary. Each term has one precise meaning here — when
+writing internals, docs, or commit messages, use these.
+
+The lexicon grows on demand: only terms whose meaning could otherwise
+drift (because they overlap with everyday English, or because different
+parts of the codebase might call the same thing different names) get an
+entry.
+
+
+## Attachment
+
+A row in the `attachment` table that records the metadata of a stored
+file (filename, content type, byte size, the service that holds the
+bytes) plus two lifecycle columns:
+
+- **source** — where the upload came from. `"direct"` for files
+  submitted as part of a form; `"rich_text"` for files uploaded by the
+  editor before the surrounding form is submitted. Open to extension
+  for other upload mechanisms (API, email import, etc.).
+- **pending** — `True` when the upload exists but no parent record has
+  confirmed ownership yet. Flips to `False` when a `HasRichText` save
+  finds the attachment referenced in a body. Rows that stay pending
+  past a grace period are purged by a periodic sweep.
+
+The `Attachment` model lives in the storage addon. Other models
+reference it through a regular `ForeignKeyField` (form-submitted
+uploads) or by ID inside a Rich Text Document (editor-embedded
+uploads).
+
+
+## Rich Text Document
+
+The AST-shaped JSON value stored by a Rich Text Field. A tree of nodes
+following the ProseMirror schema, with `{type, attrs?, content?,
+marks?, text?}` shapes.
+
+At runtime, the value is a `RichTextDocument` Python object:
+
+- `__html__()` renders the document to HTML (used implicitly by Jinja
+  whenever the document is interpolated into a template).
+- `__str__()` renders to plain text — suitable for search indices,
+  email previews, OG meta tags.
+- `.attachments` is the list of `Attachment` rows referenced by the
+  document, pre-fetched in a single batched query.
+
+
+## Rich Text Field
+
+A Peewee field whose column type is JSON and whose Python value is a
+Rich Text Document.
+
+Built by the `make_rich_text_field(parent_cls)` factory so the backing
+column type can be swapped (`proper.models.JSONField` by default) to
+`playhouse.postgres_ext.JSONField` for Postgres-native `jsonb`, or any
+other JSON-shaped Peewee field.
+
+The model field intentionally does **no** validation: validation
+belongs at form boundaries, not at the persistence layer. Models that
+declare one or more Rich Text Fields should also mix in `HasRichText`,
+which adds the `save`/`delete_instance` hooks that keep referenced
+attachments in sync with the document.
+
+
+## Attachment Embed
+
+An `attachment`-type node inside a Rich Text Document, referencing an
+`Attachment` row by UUID. The on-the-wire shape is:
+
+```json
+{
+  "type": "attachment",
+  "attrs": {
+    "id": "<uuid>",
+    "alt": "...",
+    "caption": "...",
+    "url": "...",
+    "content_type": "..."
+  }
+}
+```
+
+`alt` and `caption` are per-embed user-editable text — the same
+`Attachment` can be embedded in two different documents with different
+captions.
+
+`url` and `content_type` are render hints the editor uses to draw the
+embed live without an extra network round-trip. The server-side
+renderer ignores them and uses the resolved `Attachment` row instead,
+so they can safely drift.

Makefile

@@ -19,9 +19,17 @@ lintfix:
 coverage:
 	uv run pytest --cov-config=pyproject.toml --cov-report html --cov proper src/proper tests
 
+.PHONY: vendor-hotwire
+vendor-hotwire:
+	uv run python bin/vendor-hotwire.py
+
+.PHONY: vendor-lexxy
+vendor-lexxy:
+	uv run python bin/vendor-lexxy.py
+
 .PHONY: docs
 docs:
-	uv run python docs/docs.py
+	cd docs && uv run python docs.py
 
 .PHONY: docs-build
 docs-build:

README.md

@@ -11,7 +11,7 @@ Full human-shaped docs at [properproject.org/docs](https://properproject.org/doc
 
 ## For the press release
 
-> Proper is the Python web framework I built for myself after years in the trade, once I understood that hard conventions do not lock you in — they free you to focus on what's important.
+> Proper is the Python web framework I built for myself after years in the trade, once I understood that hard conventions do not lock you in - they free you to focus on what's important.
 
 (It also works for the back cover of my future biography by Walter Isaacson).
 

bin/vendor-hotwire.py

@@ -0,0 +1,68 @@
+#!/usr/bin/env python3
+"""Download pinned Stimulus and Turbo bundles and save them as the
+vendored JS files shipped with newly-created Proper apps.
+
+Run when bumping the pinned versions:
+
+```bash
+uv run python bin/vendor-hotwire.py
+# or
+make vendor-hotwire
+```
+
+The output goes to:
+
+    blueprint/[[app_name]]/assets/js/vendor/
+
+Files are then committed to the Proper repo so newly-created apps
+work without a runtime CDN dependency.
+
+The destination filenames (`stimulus.js`, `turbo.js`) match the
+default `IMPORT_MAP` entries in
+`blueprint/[[app_name]]/config/import_map.tt.py`.
+"""
+import urllib.request
+from pathlib import Path
+
+
+# Each package gets a (url_template, output_filename) pair. We pull the
+# pre-built single-file bundles that Hotwire ships in their npm packages
+# directly - these are documented entry points in the Hotwire docs and
+# are guaranteed to be self-contained, with no external imports the
+# browser would chase back to a CDN.
+PACKAGES = {
+    "@hotwired/stimulus": {
+        "version": "3.2.2",
+        "url": "https://unpkg.com/@hotwired/stimulus@{version}/dist/stimulus.js",
+        "filename": "stimulus.js",
+    },
+    "@hotwired/turbo": {
+        "version": "8.0.23",
+        "url": "https://unpkg.com/@hotwired/turbo@{version}/dist/turbo.es2017-esm.js",
+        "filename": "turbo.js",
+    },
+}
+
+OUTPUT_DIR = (
+    Path(__file__).resolve().parent.parent
+    / "blueprint" / "[[app_name]]"
+    / "assets" / "js" / "vendor"
+)
+
+
+def main() -> None:
+    OUTPUT_DIR.mkdir(parents=True, exist_ok=True)
+    for pkg, info in PACKAGES.items():
+        url = info["url"].format(version=info["version"])
+        dest = OUTPUT_DIR / info["filename"]
+        print(f"  → {pkg}@{info['version']}")
+        print(f"    from: {url}")
+        print(f"    to:   {dest}")
+        with urllib.request.urlopen(url) as response:  # noqa: S310
+            content = response.read()
+        dest.write_bytes(content)
+    print(f"\nWrote {len(PACKAGES)} files to {OUTPUT_DIR}")
+
+
+if __name__ == "__main__":
+    main()

bin/vendor-lexxy.py

@@ -0,0 +1,123 @@
+#!/usr/bin/env python3
+"""Download a pinned Lexxy bundle (JS + CSS) and save it to the
+vendored JS/CSS directories shipped with newly-created Proper apps.
+
+Run when bumping the pinned version:
+
+```bash
+uv run python bin/vendor-lexxy.py
+# or
+make vendor-lexxy
+```
+
+The JS output is the *bundled* ESM build from esm.sh - a single
+self-contained file with all peer dependencies (lexical, dompurify,
+@rails/activestorage) inlined. That avoids having to vendor and
+import-map each peer individually, which would otherwise be ~10
+files for Lexical alone.
+
+Destinations (mirrored to both the general app blueprint and the
+rich_text addon blueprint):
+
+    blueprint/[[app_name]]/assets/js/vendor/lexxy.js
+    blueprint/[[app_name]]/assets/css/vendor/lexxy.css
+"""
+import re
+import urllib.request
+from pathlib import Path
+
+
+VERSION = "0.9.12-beta"
+
+JS_URL = f"https://esm.sh/@37signals/lexxy@{VERSION}/es2022/lexxy.bundle.mjs"
+HELPERS_URL = f"https://esm.sh/@37signals/lexxy@{VERSION}/es2022/helpers.mjs"
+
+CSS_SOURCES = (
+    "lexxy-variables.css",
+    "lexxy-content.css",
+    "lexxy-editor.css",
+)
+CSS_BASE = f"https://cdn.jsdelivr.net/npm/@37signals/lexxy@{VERSION}/dist/stylesheets"
+
+REPO_ROOT = Path(__file__).resolve().parent.parent
+TARGET = REPO_ROOT / "src" / "proper" / "blueprints" / "rich_text" / "[[app_name]]" / "assets"
+
+custom_css = {
+  "lexxy-editor.css": b"""
+:where(lexxy-toolbar) {
+  font-size: 0.9em;
+}
+""",
+
+"lexxy-content.css": b"""
+:where(lexxy-editor) .attachment__caption textarea {
+  margin: 0;
+}
+""",
+}
+
+
+def _download(url: str) -> bytes:
+    print(f"    from: {url}")
+    with urllib.request.urlopen(url) as response:  # noqa: S310
+        return response.read()
+
+
+def _write_to_target(rel_path: str, content: bytes) -> None:
+    dest = TARGET / rel_path
+    dest.parent.mkdir(parents=True, exist_ok=True)
+    dest.write_bytes(content)
+    print(f"    to:   {dest}")
+
+
+def _rewrite_paths_to_esm_sh(js_bytes: bytes) -> bytes:
+    """Rewrite absolute paths in the Lexxy bundle to point at esm.sh.
+
+    esm.sh's `lexxy.bundle.mjs` keeps `import` statements referring to
+    paths like `/lexical@^0.44.0/Lexical.prod?target=es2022` - relative
+    to esm.sh's origin. Served from our app, those resolve to 404s.
+
+    We rewrite them to absolute `https://esm.sh/...` URLs so the browser
+    fetches peer deps directly from esm.sh. This introduces a runtime
+    CDN dependency for those deps but avoids the 30+ files we'd need to
+    vendor (lexical core + 12 lexical packages + dompurify + helpers,
+    each with `.dev` and `.prod` variants).
+    """
+    # Rewrite quoted absolute paths starting with `/` followed by a
+    # package name (`@scope/pkg@` or `pkg@`).
+    js = re.sub(rb'"(/(?:@[^/"]+/)?[^/"]+@[^"]+)"', rb'"https://esm.sh\1"', js_bytes)
+    # `./helpers.mjs` is Lexxy's sibling file; we vendor it too.
+    js = js.replace(b'"./helpers.mjs"', b'"./lexxy-helpers.js"')
+    return js
+
+
+def _strip_source_map_reference(js_bytes: bytes) -> bytes:
+    """The source map reference comment at the end of the bundle is not
+    relevant to us and just adds noise, so we strip it out."""
+    return re.sub(rb"^//# sourceMappingURL=.*$", b"", js_bytes, flags=re.MULTILINE)
+
+
+def main() -> None:
+    print(f"  → @37signals/lexxy@{VERSION} (JS bundle)")
+    js_content = _download(JS_URL)
+    js_content = _rewrite_paths_to_esm_sh(js_content)
+    js_content = _strip_source_map_reference(js_content)
+    _write_to_target("js/vendor/lexxy.js", js_content)
+
+    print(f"  → @37signals/lexxy@{VERSION} (helpers)")
+    helpers_content = _download(HELPERS_URL)
+    helpers_content = _rewrite_paths_to_esm_sh(helpers_content)
+    helpers_content = _strip_source_map_reference(helpers_content)
+    _write_to_target("js/vendor/lexxy-helpers.js", helpers_content)
+
+    print(f"  → @37signals/lexxy@{VERSION} (CSS)")
+    for name in CSS_SOURCES:
+        css_content = _download(f"{CSS_BASE}/{name}")
+        if name in custom_css:
+            css_content += custom_css[name]
+        _write_to_target(f"css/{name}", css_content)
+
+    print(f"\nVendored Lexxy {VERSION} into blueprint.\n\n")
+
+if __name__ == "__main__":
+    main()

blueprint/AGENTS.md

@@ -6,12 +6,12 @@ Deep reference docs are bundled in the `proper` skill.
 
 ## Key Conventions
 
-- **Everything is CRUD** — controllers map to RESTful resource actions (index, new, create, show, edit, update, delete)
-- **Singular names** — resource/model/controller names are always singular (e.g., `Post`, not `Posts`)
-- **Boot order matters** — `__init__.py` imports: main → router → controllers → models → tasks. New controllers/models must be imported in their respective `__init__.py`
-- **`form.save()` returns unsaved** — it returns a model instance that hasn't been persisted yet; call `instance.save()` to write to DB
-- **Controllers are sync** — despite ASGI, controller methods are regular sync Python
-- **`current` context** — `current.request`, `current.user`, `current.auth_session` available in controllers and templates
+- **Everything is CRUD** - controllers map to RESTful resource actions (index, new, create, show, edit, update, delete)
+- **Singular names** - resource/model/controller names are always singular (e.g., `Post`, not `Posts`)
+- **Boot order matters** - `__init__.py` imports: main → router → controllers → models → tasks. New controllers/models must be imported in their respective `__init__.py`
+- **`form.save()` returns unsaved** - it returns a model instance that hasn't been persisted yet; call `instance.save()` to write to DB
+- **Controllers are sync** - despite ASGI, controller methods are regular sync Python
+- **`current` context** - `current.request`, `current.user`, `current.auth_session` available in controllers and templates
 
 ## Common Commands
 
@@ -29,12 +29,12 @@ uv run proper run                                # Start dev server
 | -------------------- | ---------------------------------- | ---------------------------------- |
 | Controller           | `controllers/{name}_controller.py` | `controllers/__init__.py`          |
 | Model                | `models/{name}.py`                 | `models/__init__.py`               |
-| Form                 | `forms/{name}.py`                  | —                                  |
-| Views                | `views/{name}/*.jx`       | —                                  |
+| Form                 | `forms/{name}.py`                  | -                                  |
+| Views                | `views/{name}/*.jx`       | -                                  |
 | Concern (controller) | `controllers/concerns/{name}.py`   | controller that uses it            |
 | Concern (model)      | `models/concerns/{name}.py`        | model that uses it                 |
-| Task                 | `tasks/{name}.py`                  | —                                  |
-| Email                | `emails/{name}.py` + `views/emails/{name}.jx` | —                    |
+| Task                 | `tasks/{name}.py`                  | -                                  |
+| Email                | `emails/{name}.py` + `views/emails/{name}.jx` | -                    |
 | Config               | `config/{name}.py`                 | `config/__init__.py`               |
 
 ## General Guidelines