Add sdata.pl.annotate() — interactive region selection via anywidget

Productionises the Sandbox.ipynb prototype as a user-facing method on
PlotAccessor. Public surface is a single function:

    sdata.pl.annotate(coordinate_system, element, *, persist=True) -> None

Both args are required positional. The function validates that the image
element is registered in the given coordinate system, renders it to a PNG,
constructs an internal _InteractiveSession with anywidget-driven drawing
tools (rectangle / polygon / lasso), and displays the widget. Drawn shapes
are written into sdata.shapes[name] on click of the Save button; the
optional "Write to disk" button persists via sdata.write_element.

Module layout (src/spatialdata_plot/pl/interactive/):
  - _canvas.py             DrawCanvas anywidget class
  - static/draw_canvas.js  ESM module read from disk by anywidget (HMR-friendly)
  - _render.py             render_to_png: sdata.pl → PNG + ax extent
  - _commit.py             pixel-coord shape → CS-coord shapely Polygon → ShapesModel
  - _persist.py            commit_to_memory + persist_to_disk (collision policy)
  - _session.py            _InteractiveSession orchestrating the widget

The new optional extra `interactive` (anywidget, ipykernel, ipywidgets)
gates this feature behind a clear ImportError when missing:

    pip install 'spatialdata-plot[interactive]'

The prototype iteration explored ipympl (rejected: PNG-over-websocket
latency unusable over SSH) and plotly's FigureWidget (rejected: client-
side relayout events don't sync back to Python in VSCode-Remote, plus
plotly 6's anywidget-backed FigureWidget broke the comm path entirely).
The custom anywidget approach was the only architecture that worked
reliably over SSH while staying responsive.

Drawing UX:
  - Tools: rect (drag), polygon (click + snap-close), lasso (drag freehand)
  - Wheel zoom, shift-drag pan, alt-click shape to delete
  - Ctrl+Z undo, R/P/L tool shortcuts, F fit view, Enter close polygon
  - Multi-shape bundling: each Save commits all canvas shapes as one
    ShapesModel with multiple rows under a single name

Tests cover the unit surface (pixel→CS conversion, ShapesModel transform
registration, render-to-PNG correctness, commit/persist policy, widget
smoke).

Spec at plans/interactive-selection.md updated to document the
architectural pivot from the original ipympl approach.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Author: timtreis

plans/interactive-selection.md

@@ -16,13 +16,29 @@ SLURM compute node. No napari, no desktop GUI.
 - Selector shapes in v0: rectangle, polygon (click vertices), lasso (freehand).
 - Scale handling: auto-downsample on the fly. Pyramid-aware when available;
   `dask.coarsen` fallback when not.
-- Layers beneath the selector in v0: images only. Selector attaches to the
-  `Axes` returned by the existing `sdata.pl.render_images().pl.show()` pipeline
-  — we reuse the existing canvas, no duplicate render path.
-- Backend: `%matplotlib widget` (ipympl) + `matplotlib.widgets.{Rectangle,
-  Polygon,Lasso}Selector`. Pure server-side render, PNG frames over websocket.
+- Layers in v0: images only. The image is rendered once via the existing
+  `sdata.pl.render_images().pl.show()` pipeline into a matplotlib figure,
+  exported to PNG, and laid under a client-side drawing canvas.
+- Backend: **custom anywidget** with HTML5/SVG drawing tools (rectangle,
+  polygon, freehand-lasso). All drawing happens in the browser; shape
+  geometry is reported back to Python via traitlet sync. Image is sent
+  once as a base64 data URL; mouse moves never round-trip the kernel.
   No bokeh/datashader.
 
+### Why anywidget, not ipympl or plotly
+
+The original spec called for `%matplotlib widget` (ipympl). The prototype
+revealed two showstoppers over SSH:
+1. **ipympl streams PNG frames per mouse-move** over websocket — every drag
+   incurs SSH round-trip latency, making freehand drawing unusable.
+2. **plotly's `FigureWidget`** has broken two-way shape sync in
+   VSCode-Remote-SSH (regardless of plotly 5 vs 6 — different bugs each).
+
+A small (~250-line) anywidget with traitlet-synced shape geometry was the
+only architecture that worked reliably in VSCode-Remote and produced
+responsive drawing. The image render still uses sdata-plot's matplotlib
+pipeline; we just don't drive interaction through it.
+
 ## Resolved questions (locked 2026-05-21, task #1)
 
 - **Q1 — Channel/contrast widgets**: **No live widgets in v0.** `channel=` and
@@ -43,42 +59,59 @@ SLURM compute node. No napari, no desktop GUI.
 import spatialdata_plot  # registers .pl
 
 session = sdata.pl.interactive(
-    element="he_image",
-    coordinate_system="global",
-    channel=[0, 1, 2],            # optional
-    clims=(0, 30000),             # optional
-    selector="polygon",           # 'rectangle' | 'polygon' | 'lasso'
-    name="tumor_region",
-    overwrite=False,
-    persist=True,
-    max_render_pixels=2_000_000,
+    coordinate_system=None,   # optional pre-selection; None = let user pick in UI
+    element=None,             # optional pre-selection; None = let user pick in UI
+    persist=True,             # show "Write to disk" button (False = memory only)
 )
-session.show()                    # returns the ipympl Figure
-# user draws on canvas, double-click / release to commit
-sdata["tumor_region"]             # ShapesModel
+session.show()                # renders the ipywidgets controls + draw canvas
+
+# User picks CS + image, clicks Render, draws shapes, names + Saves each set.
+# Each Save adds an entry to sdata.shapes (memory). Write to disk persists
+# the most recent commit via sdata.write_element.
+
+sdata["tumor_region"]         # ShapesModel
 sub = sdata.query.polygon(sdata, sdata["tumor_region"])
 ```
 
+Removed kwargs vs original spec:
+- `selector=` — UI has a tool toggle (rect/polygon/lasso); no need to bind one
+  selector at construction (Q3 resolution).
+- `name=` — typed in the UI before each Save (Q4 resolution).
+- `channel=`, `clims=` — deferred to v1 (Q1 resolution).
+- `max_render_pixels=` — render is fixed at `figsize=(7,7), dpi=120` ≈ 840×840
+  PNG; pyramid-aware downsampling deferred to v1.
+- `overwrite=` — collision handling is automatic: same name → append UTC
+  timestamp.
+
 ## Module layout
 
 ```
 src/spatialdata_plot/pl/interactive/
-  __init__.py        # exports InteractiveSession
-  _session.py        # InteractiveSession class, public entrypoint
-  _render.py         # thin wrapper around existing render_images
-  _downsample.py     # pyramid-aware scale picker; in-memory coarsen
-  _selectors.py      # RectangleAdapter, PolygonAdapter, LassoAdapter
-  _commit.py         # vertices → CS-correct shapely → ShapesModel
-  _persist.py        # write_element + overwrite/timestamp policy
+  __init__.py        # exports interactive, InteractiveSession, DrawCanvas
+  _session.py        # InteractiveSession class — ipywidgets controls
+  _canvas.py         # DrawCanvas anywidget + traitlets
+  _render.py         # render_to_png helper (sdata.pl → PNG + extent)
+  _commit.py         # pixel-shape → CS-correct shapely Polygon → ShapesModel
+  _persist.py        # write_element + collision/timestamp policy
+  static/
+    draw_canvas.js   # the ESM module; _esm = Path(...) reads at import
 
 tests/test_interactive/
-  test_commit.py
-  test_downsample.py
-  test_selectors_headless.py
+  test_commit.py     # pixel→CS conversion + ShapesModel correctness
+  test_render.py     # render_to_png returns valid PNG + extent
+  test_persist.py    # collision/timestamp policy
+  test_canvas.py     # smoke: instantiate widget, check traitlet defaults
 ```
 
-`sdata.pl.interactive(...)` becomes a method on `PlotAccessor` in
-`src/spatialdata_plot/_accessor.py`, returning an `InteractiveSession`.
+`sdata.pl.interactive(...)` is a method on `PlotAccessor` in
+`src/spatialdata_plot/_accessor.py`. It constructs an `InteractiveSession`
+and returns it; `session.show()` displays the controls + draw canvas.
+
+Dropped from the original spec:
+- `_downsample.py` — pyramid-aware downsampling deferred to v1; v0 renders
+  at a fixed dpi (`figsize=(7,7), dpi=120`).
+- `_selectors.py` — matplotlib selectors are replaced by the anywidget; the
+  three drawing tools (rect/polygon/lasso) live in `static/draw_canvas.js`.
 
 ## Coordinate-system rules (highest-risk surface)
 
@@ -92,25 +125,31 @@ tests/test_interactive/
 
 Avoids the classic double-applied-transform bug.
 
-## Downsampling
+## Rendering
+
+`_render.render_to_png(sdata, element, coordinate_system) -> (png_bytes, image_w, image_h, xlim, ylim)`
 
-`_downsample.pick_scale(image, bbox, max_pixels) -> (level_or_factor, array)`
+- Uses `sdata.pl.render_images(element=...).pl.show(coordinate_systems=..., ax=...)`.
+- Axes fills the figure (`ax.add_axes([0,0,1,1])`, `set_axis_off()`) so PNG pixel
+  coordinates map exactly to data coordinates via `xlim`/`ylim`.
+- Fixed at `figsize=(7,7)` × `dpi=120` ≈ 840×840 PNG for v0. Pyramid-aware
+  downsampling deferred to v1.
+- 3D / z-stacks: refused by `render_images` itself (commit 3ebefe1) — we
+  propagate that error.
 
-- `MultiscaleSpatialImage`: walk scales coarse→fine, pick finest within budget.
-- Single-scale: `dask.array.coarsen` with integer factor, warn once.
-- Static extent in v0. Auto-redraw on `xlim_changed` is v1.
-- Default `max_render_pixels ≈ 2M` (~1500×1500), tuned for ipympl PNG over SSH.
+## Drawing tools (in `static/draw_canvas.js`)
 
-## Selector adapters
+| kind        | gesture                                        | commit trigger                                |
+|-------------|------------------------------------------------|-----------------------------------------------|
+| rectangle   | left-drag corner → corner                      | mouse release                                 |
+| polygon     | click each vertex                              | snap-to-first-vertex (within 10 px) or Enter  |
+| lasso       | left-drag freehand                             | mouse release                                 |
 
-| kind        | matplotlib class      | commit trigger                |
-|-------------|-----------------------|-------------------------------|
-| rectangle   | `RectangleSelector`   | mouse release                 |
-| polygon     | `PolygonSelector`     | close (double-click / enter)  |
-| lasso       | `LassoSelector`       | mouse release                 |
+Plus client-side: wheel-zoom, shift-drag-pan, alt-click-shape-to-delete,
+hover-highlight, Ctrl+Z undo, Delete clear, R/P/L tool shortcuts, F fit.
 
-Lasso vertices simplified via `shapely.simplify(tolerance=0.5px)` before
-persist.
+Lasso vertices are simplified server-side via `shapely.simplify(tolerance=0.5)`
+in `_commit` before persisting.
 
 ## Persistence policy
 
@@ -133,22 +172,27 @@ persist.
 
 ## Test strategy
 
-- Unit: `_commit` (synthetic vertices → ShapesModel correctness).
-- Unit: `_downsample` (scale picker correctness on synthetic arrays).
-- Headless: `_selectors` via programmatic `_press`/`_onmove`/`_release`.
-- NO visual tests in v0. CI does not need a live canvas.
-- Manual checklist in PR description for the canvas itself.
+- Unit: `_commit` (synthetic pixel-coord shapes → CS-coord ShapesModel correctness).
+- Unit: `_render` (returns valid PNG bytes + extent matching the axis limits).
+- Unit: `_persist` (collision-rename + timestamp policy).
+- Smoke: `_canvas` (instantiate `DrawCanvas`, check traitlet defaults).
+- NO visual / live-canvas tests in v0 — the JS widget can't be driven from Python.
+  Manual checklist in PR description covers the canvas behaviour.
 
 ## Dependencies
 
-`[project.dependencies]`:
+Exposed as `[project.optional-dependencies].interactive` so the feature is
+opt-in (`pip install spatialdata-plot[interactive]`). Mirrors the pixi
+`interactive` dep-group.
 
-- `ipympl` (NEW)
-- `ipywidgets` (NEW or pin existing transitive)
-- `shapely` (already transitive via geopandas)
-- `geopandas` (already transitive via spatialdata)
+- `anywidget` (NEW) — the widget framework.
+- `ipywidgets` (NEW or pin existing transitive) — for the controls VBox.
+- `ipykernel` — needed by anywidget for comm channel.
+- `shapely`, `geopandas` — already transitive via spatialdata.
 
-Only `ipympl` is genuinely new.
+`ipympl` and `plotly` are NOT runtime deps of the new architecture (we tried
+both and rejected them). They remain in the prototype/pixi feature only for
+historical comparison and may be dropped from the interactive feature later.
 
 ## v1 roadmap (after v0 ships)
 

pyproject.toml

@@ -31,6 +31,11 @@ dependencies = [
   "scikit-learn",
   "spatialdata>=0.3",
 ]
+optional-dependencies.interactive = [
+  "anywidget",
+  "ipykernel",
+  "ipywidgets",
+]
 urls.Documentation = "https://spatialdata.scverse.org/projects/plot/en/latest/index.html"
 urls.Home-page = "https://github.com/scverse/spatialdata-plot.git"
 urls.Source = "https://github.com/scverse/spatialdata-plot.git"
@@ -61,11 +66,11 @@ doc = [
   "sphinxcontrib-katex",
   "sphinxext-opengraph",
 ]
-interactive = [
-  "anywidget",
-  "ipykernel",
+interactive-extras = [
+  # Prototype-only helpers used by Sandbox.ipynb. The published runtime extra
+  # is [project.optional-dependencies].interactive above (anywidget/ipykernel/
+  # ipywidgets only) — these are kept here for the dev-interactive-py313 env.
   "ipympl",
-  "ipywidgets",
   # pinned to 5.x: plotly 6's anywidget-backed FigureWidget doesn't relay
   # client-side draw events back to Python, so layout.shapes never syncs.
   "plotly>=5.20,<6",
@@ -106,6 +111,12 @@ python = ">=3.11"
 [tool.pixi.pypi-dependencies]
 spatialdata-plot = { path = ".", editable = true }
 
+# When the `interactive` feature is active, install the package with the
+# `interactive` PyPI extra (anywidget, ipykernel, ipywidgets) so the pixi
+# env mirrors what `pip install spatialdata-plot[interactive]` would give.
+[tool.pixi.feature.interactive.pypi-dependencies]
+spatialdata-plot = { path = ".", editable = true, extras = [ "interactive" ] }
+
 [tool.pixi.tasks]
 format = "ruff format ."
 kernel-install = 'python -m ipykernel install --user --name pixi-dev --display-name "sdata-plot (dev)"'
@@ -129,7 +140,7 @@ default = { features = [ "py313" ], solve-group = "py313" }
 # 3.11 lane (for gh-actions)
 dev-py311 = { features = [ "dev", "test", "py311" ], solve-group = "py311" }
 dev-py313 = { features = [ "dev", "test", "py313" ], solve-group = "py313" }
-dev-interactive-py313 = { features = [ "dev", "test", "interactive", "py313" ], solve-group = "py313" }
+dev-interactive-py313 = { features = [ "dev", "test", "interactive", "interactive-extras", "py313" ], solve-group = "py313" }
 docs-py311 = { features = [ "doc", "py311" ], solve-group = "py311" }
 docs-py313 = { features = [ "doc", "py313" ], solve-group = "py313" }
 test-py313 = { features = [ "test", "py313" ], solve-group = "py313" }

src/spatialdata_plot/pl/basic.py

@@ -171,6 +171,74 @@ def _copy(
 
         return sdata
 
+    def annotate(
+        self,
+        coordinate_system: str,
+        element: str,
+        *,
+        persist: bool = True,
+    ) -> None:
+        """Draw and save regions interactively on an image element.
+
+        Renders the image element in the given coordinate system as a
+        client-side drawing canvas (rectangle / polygon / lasso tools).
+        Drawn shapes are saved into ``sdata.shapes`` under a user-typed name
+        on click of the *Save* button — each save creates one ShapesModel
+        with one row per drawn shape, registered with an ``Identity``
+        transformation in the chosen coordinate system.
+
+        Requires the ``interactive`` extra: ``pip install 'spatialdata-plot[interactive]'``.
+
+        Parameters
+        ----------
+        coordinate_system :
+            Coordinate system to render and resolve drawn shapes against.
+            Drawn polygons are stored with an ``Identity`` transformation
+            in this CS.
+        element :
+            Name of the image element to render.
+        persist :
+            If ``True`` (default), show a *Write to disk* button that calls
+            :meth:`SpatialData.write_element` for the most recent save.
+            Set to ``False`` to limit the session to in-memory commits.
+
+        Returns
+        -------
+        None
+            Displays the widget in the current notebook cell. Drawn and
+            saved shapes appear in ``sdata.shapes``; inspect them there.
+
+        Raises
+        ------
+        ValueError
+            If ``coordinate_system`` is unknown, ``element`` is unknown,
+            or ``element`` is not registered in ``coordinate_system``.
+        ImportError
+            If the ``interactive`` extra is not installed.
+
+        Examples
+        --------
+        >>> import spatialdata_plot  # noqa: F401  registers .pl
+        >>> sdata.pl.annotate("global", "he_image")
+        >>> # ... user draws and clicks Save with name "tumor" ...
+        >>> sdata.shapes["tumor"]
+        """
+        try:
+            from spatialdata_plot.pl.interactive._session import _InteractiveSession
+        except ImportError as exc:
+            raise ImportError(
+                "sdata.pl.annotate() requires the `interactive` extra. "
+                "Install with: pip install 'spatialdata-plot[interactive]'"
+            ) from exc
+
+        session = _InteractiveSession(
+            self._sdata,
+            coordinate_system=coordinate_system,
+            element=element,
+            persist=persist,
+        )
+        session.show()
+
     @_deprecation_alias(elements="element", version="0.3.0")
     def render_shapes(
         self,

src/spatialdata_plot/pl/interactive/__init__.py

@@ -0,0 +1,10 @@
+"""Interactive region selection on a SpatialData image.
+
+Use via :meth:`spatialdata_plot.pl.basic.PlotAccessor.annotate`:
+
+>>> import spatialdata_plot  # noqa: F401  registers .pl
+>>> sdata.pl.annotate("global", "he_image")
+"""
+from __future__ import annotations
+
+__all__: list[str] = []

src/spatialdata_plot/pl/interactive/_canvas.py

@@ -0,0 +1,49 @@
+"""anywidget wrapping a client-side SVG drawing canvas."""
+from __future__ import annotations
+
+from pathlib import Path
+
+import anywidget
+import traitlets
+
+_ESM_PATH = Path(__file__).parent / "static" / "draw_canvas.js"
+
+
+class DrawCanvas(anywidget.AnyWidget):
+    """Client-side SVG drawing surface for interactive region selection.
+
+    The image (PNG data URL) is shown as a CSS-transformed background; an
+    overlay SVG catches mouse events and emits committed shapes in image-
+    pixel coordinates via the ``shapes`` traitlet.
+
+    Convert the pixel-coord shapes to data/CS coordinates with
+    :func:`spatialdata_plot.pl.interactive._commit.pixel_shape_to_polygon`.
+
+    Traitlets
+    ---------
+    image_url
+        ``data:image/png;base64,...`` for the rendered image.
+    image_width, image_height
+        Pixel dimensions of the PNG (used to set the SVG ``viewBox``).
+    tool
+        ``"rectangle"``, ``"polygon"``, or ``"lasso"``.
+    shapes
+        List of ``{"type": "rect"|"polygon", "verts": [[x, y], ...]}`` in
+        image-pixel coordinates. JS pushes to this on commit.
+    clear_trigger, close_poly_trigger, undo_trigger, fit_trigger
+        Integer counters. Increment from Python to invoke the corresponding
+        JS action; JS observers are stateless w.r.t. the value, only the
+        change event matters.
+    """
+
+    _esm = _ESM_PATH
+
+    image_url = traitlets.Unicode("").tag(sync=True)
+    image_width = traitlets.Int(720).tag(sync=True)
+    image_height = traitlets.Int(720).tag(sync=True)
+    tool = traitlets.Unicode("rectangle").tag(sync=True)
+    shapes = traitlets.List([]).tag(sync=True)
+    clear_trigger = traitlets.Int(0).tag(sync=True)
+    close_poly_trigger = traitlets.Int(0).tag(sync=True)
+    undo_trigger = traitlets.Int(0).tag(sync=True)
+    fit_trigger = traitlets.Int(0).tag(sync=True)