Skip to content

Refactor API: Writer-centric rendering and Python module restructure #90

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
cpsievert wants to merge 7 commits into from
Closed

Refactor API: Writer-centric rendering and Python module restructure #90

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
7cb225e
Refactor Rust API: Writer::render(), Reader::execute_sql(), unregister()
cpsievert Jan 30, 2026
9d52240
Restructure Python bindings with cleaner module hierarchy
cpsievert Jan 30, 2026
3fa8574
Update Python tests for new API structure
cpsievert Jan 30, 2026
cfa3114
Update documentation for new API structure
cpsievert Jan 30, 2026
471602e
Fix formatting (cargo fmt)
cpsievert Jan 30, 2026
3bafd5f
Fix REST API for NoVisualise error and clippy warnings
cpsievert Jan 30, 2026
84091f6
Add Reader ABC for custom reader implementations
cpsievert Jan 30, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
145 changes: 63 additions & 82 deletions CLAUDE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -165,7 +165,7 @@ let prepared = ggsql::prepare(

// Render to Vega-Lite JSON
let writer = VegaLiteWriter::new();
let json = prepared.render(&writer)?;
let json = writer.render(&prepared)?;
```

### Core Functions
Expand Down Expand Up @@ -869,13 +869,12 @@ When running in Positron IDE, the extension provides enhanced functionality:
**Features**:

- PyO3-based Rust bindings compiled to a native Python extension
- Two-stage API mirroring the Rust API: `prepare()` → `render()`
- DuckDB reader with DataFrame registration
- Custom Python reader support: any object with `execute(sql) -> DataFrame` method
- Works with any narwhals-compatible DataFrame (polars, pandas, etc.)
- LazyFrames are collected automatically
- Returns native `altair.Chart` objects via `render_altair()` convenience function
- Two-stage API: `reader.execute()` → `writer.render()`
- DuckDB reader with inline DataFrame registration via `execute(query, data_dict)`
- Automatic table cleanup after query execution
- Returns native `altair.Chart` objects via `writer.render_chart()`
- Query validation and introspection (SQL, layer queries, stat queries)
- `NoVisualiseError` exception for queries without VISUALISE clause

**Installation**:

Expand All @@ -892,40 +891,24 @@ maturin develop
import ggsql
import polars as pl

# Create reader and register data
reader = ggsql.DuckDBReader("duckdb://memory")
df = pl.DataFrame({"x": [1, 2, 3], "y": [10, 20, 30]})
reader.register("data", df)

# Prepare visualization
prepared = ggsql.prepare(
# Execute with inline data registration (auto-registers and unregisters)
reader = ggsql.DuckDBReader("duckdb://memory")
spec = reader.execute(
"SELECT * FROM data VISUALISE x, y DRAW point",
reader
{"data": df}
)

# Inspect metadata
print(f"Rows: {prepared.metadata()['rows']}")
print(f"Columns: {prepared.metadata()['columns']}")
print(f"SQL: {prepared.sql()}")
print(f"Rows: {spec.metadata()['rows']}")
print(f"Columns: {spec.metadata()['columns']}")
print(f"SQL: {spec.sql()}")

# Render to Vega-Lite JSON
# Render to Vega-Lite JSON or Altair chart
writer = ggsql.VegaLiteWriter()
json_output = prepared.render(writer)
```

**Convenience Function** (`render_altair`):

For quick visualizations without explicit reader setup:

```python
import ggsql
import polars as pl

df = pl.DataFrame({"x": [1, 2, 3], "y": [10, 20, 30]})

# Render DataFrame to Altair chart in one call
chart = ggsql.render_altair(df, "VISUALISE x, y DRAW point")
chart.display() # In Jupyter
json_output = writer.render(spec)
chart = writer.render_chart(spec)
```

**Query Validation**:
Expand All @@ -941,73 +924,71 @@ print(f"SQL portion: {validated.sql()}")
print(f"Errors: {validated.errors()}")
```

**Classes**:

| Class | Description |
| -------------------------- | -------------------------------------------- |
| `DuckDBReader(connection)` | Database reader with DataFrame registration |
| `VegaLiteWriter()` | Vega-Lite JSON output writer |
| `Validated` | Result of `validate()` with query inspection |
| `Prepared` | Result of `prepare()`, ready for rendering |
**Handling Plain SQL**:

**Functions**:
```python
try:
spec = reader.execute("SELECT * FROM data", {"data": df})
except ggsql.NoVisualiseError:
# Use execute_sql() for queries without VISUALISE
result_df = reader.execute_sql("SELECT * FROM data")
```

| Function | Description |
| ------------------------ | ------------------------------------------------- |
| `validate(query)` | Syntax/semantic validation with query inspection |
| `prepare(query, reader)` | Full preparation (reader can be native or custom) |
| `render_altair(df, viz)` | Convenience: render DataFrame to Altair chart |
**Classes**:

**Prepared Object Methods**:
| Class | Description |
| -------------------------- | --------------------------------------------------- |
| `DuckDBReader(connection)` | Database reader with DataFrame registration |
| `VegaLiteWriter()` | Vega-Lite JSON output writer with render methods |
| `Validated` | Result of `validate()` with query inspection |
| `Prepared` | Result of `reader.execute()`, ready for rendering |
| `NoVisualiseError` | Exception for queries without VISUALISE clause |

| Method | Description |
| ---------------- | -------------------------------------------- |
| `render(writer)` | Generate Vega-Lite JSON |
| `metadata()` | Get rows, columns, layer_count |
| `sql()` | Get the SQL portion |
| `visual()` | Get the VISUALISE portion |
| `layer_count()` | Number of DRAW layers |
| `data()` | Get the main DataFrame |
| `layer_data(i)` | Get layer-specific DataFrame (if filtered) |
| `stat_data(i)` | Get stat transform DataFrame (if applicable) |
| `layer_sql(i)` | Get layer filter SQL (if applicable) |
| `stat_sql(i)` | Get stat transform SQL (if applicable) |
| `warnings()` | Get validation warnings |
**Functions**:

**Custom Python Readers**:
| Function | Description |
| ----------------- | ------------------------------------------------ |
| `validate(query)` | Syntax/semantic validation with query inspection |

Any Python object with an `execute(sql: str) -> polars.DataFrame` method can be used as a reader:
**DuckDBReader Methods**:

```python
import ggsql
import polars as pl
| Method | Description |
| ---------------------------- | -------------------------------------------------------- |
| `execute(query, data=None)` | Execute ggsql query with optional data dict registration |
| `execute_sql(sql)` | Execute plain SQL, return DataFrame |
| `register(name, df)` | Manually register DataFrame as table |
| `unregister(name)` | Unregister table (fails silently if not found) |

class MyReader:
"""Custom reader that returns static data."""
**VegaLiteWriter Methods**:

def execute(self, sql: str) -> pl.DataFrame:
return pl.DataFrame({"x": [1, 2, 3], "y": [10, 20, 30]})
| Method | Description |
| ------------------------- | ------------------------------------ |
| `render(spec)` | Render to Vega-Lite JSON string |
| `render_chart(spec)` | Render to Altair chart object |

# Use custom reader with prepare()
reader = MyReader()
prepared = ggsql.prepare(
"SELECT * FROM data VISUALISE x, y DRAW point",
reader
)
```
**Prepared Object Methods**:

Optional methods for custom readers:
| Method | Description |
| --------------- | -------------------------------------------- |
| `metadata()` | Get rows, columns, layer_count |
| `sql()` | Get the SQL portion |
| `visual()` | Get the VISUALISE portion |
| `layer_count()` | Number of DRAW layers |
| `data()` | Get the main DataFrame |
| `layer_data(i)` | Get layer-specific DataFrame (if filtered) |
| `stat_data(i)` | Get stat transform DataFrame (if applicable) |
| `layer_sql(i)` | Get layer filter SQL (if applicable) |
| `stat_sql(i)` | Get stat transform SQL (if applicable) |
| `warnings()` | Get validation warnings |

- `supports_register() -> bool` - Return `True` if registration is supported
- `register(name: str, df: polars.DataFrame) -> None` - Register a DataFrame as a table
**Type Stubs**:

Native readers (e.g., `DuckDBReader`) use an optimized fast path, while custom Python readers are automatically bridged via IPC serialization.
The Python package includes manually maintained type stubs (`ggsql-python/python/ggsql/_ggsql.pyi`) that provide IDE support and type checking for the native Rust extension. When making API changes to `ggsql-python/src/lib.rs`, always update the corresponding stubs to keep them in sync. The stubs include detailed docstrings that appear in IDE tooltips, so they provide significant value beyond just type information.

**Dependencies**:

- Python >= 3.10
- altair >= 5.0
- narwhals >= 2.15
- polars >= 1.0

---
Expand Down
6 changes: 3 additions & 3 deletions ggsql-jupyter/src/executor.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ use ggsql::{
prepare,
reader::{DuckDBReader, Reader},
validate,
writer::VegaLiteWriter,
writer::{VegaLiteWriter, Writer},
};
use polars::frame::DataFrame;

Expand Down Expand Up @@ -60,7 +60,7 @@ impl QueryExecutor {
// 2. Check if there's a visualization
if !validated.has_visual() {
// Pure SQL query - execute directly and return DataFrame
let df = self.reader.execute(code)?;
let df = self.reader.execute_sql(code)?;
tracing::info!(
"Pure SQL executed: {} rows, {} cols",
df.height(),
Expand All @@ -79,7 +79,7 @@ impl QueryExecutor {
);

// 4. Render to Vega-Lite
let vega_json = prepared.render(&self.writer)?;
let vega_json = self.writer.render(&prepared)?;

tracing::debug!("Generated Vega-Lite spec: {} chars", vega_json.len());

Expand Down
Loading