Skip to content

Commit 5be5ffe

Browse files
committed
feat: fork sqlc-gen-go as iodesystems/sqlc-go-codegen-metaquery
Extends sqlc-gen-go with per-query metadata emission and a runtime CTE builder so queries can be dynamically wrapped with filters, ordering, pagination, and aggregations without modifying the original SQL. Scope: - Rename module to github.com/iodesystems/sqlc-go-codegen; rename plugin binary to sqlc-go-codegen-metaquery. - Emit per-query `MetaFoo` metadata vars alongside sqlc's generated query code (new internal/metaquery.go + internal/templates/metaqueryFile.tmpl, wired via internal/gen.go). - `metaquery` runtime pkg: Query/Column/Arg/Table types, Builder with Where/WhereExpr/Having/OrderBy/GroupBy/Select/Agg/Paginate, and Build/BuildCount that wrap the original SQL as a CTE with original placeholders preserved. - Typed safety layer: typed `WrapFoo(args...)` constructors, typed `FooCols` column refs (TextCol/IntCol/FloatCol/BoolCol/TimeCol/ BytesCol/AnyCol) with op-appropriate methods, Op constants, and a ValidateFilter for JSON-driven flows. - `metaquery/mqpgx` submodule: pgx/v5 adapter with Run (untyped Result), Scan[T] (typed TypedResult[T] with shape validation via Validate[T]), ScanOne[T], and automatic total-count population. - `emit_metaquery` option (ladder: off|meta|wrap|cols, default cols) with per-query `-- metaquery: <level>` override. - Bidirectional Filter/OrderBy/PageRequest types so HTTP handlers can unmarshal from request bodies straight into the builder and marshal the applied state back via Meta. - Fix internal/result.go buildStructs to populate DBName and Column on model-struct fields (needed for metadata). End-to-end: - `internal/endtoend/testdata/metaquery_authors/` — authors fixture with real sqlc generate + unit tests that exercise every surface. - `examples/pgx/` — docker-compose pg17, migrations, sqlc.yaml, a demo CLI (seed/list/search/counts) outputting JSON envelopes, Makefile, and README explaining the emit-level ladder. Tests pass across all four modules (plugin, metaquery, mqpgx, fixture, example). Live demo verified against pg17.
1 parent d0bacc9 commit 5be5ffe

51 files changed

Lines changed: 4156 additions & 140 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

.idea/.gitignore

Lines changed: 10 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

Makefile

Lines changed: 8 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,18 +1,20 @@
11
.PHONY: build test
22

3+
BIN_NAME = sqlc-go-codegen-metaquery
4+
35
build:
46
go build ./...
57

6-
test: bin/sqlc-gen-go.wasm
8+
test: bin/$(BIN_NAME).wasm
79
go test ./...
810

9-
all: bin/sqlc-gen-go bin/sqlc-gen-go.wasm
11+
all: bin/$(BIN_NAME) bin/$(BIN_NAME).wasm
1012

11-
bin/sqlc-gen-go: bin go.mod go.sum $(wildcard **/*.go)
12-
cd plugin && go build -o ../bin/sqlc-gen-go ./main.go
13+
bin/$(BIN_NAME): bin go.mod go.sum $(wildcard **/*.go)
14+
cd plugin && go build -o ../bin/$(BIN_NAME) ./main.go
1315

14-
bin/sqlc-gen-go.wasm: bin/sqlc-gen-go
15-
cd plugin && GOOS=wasip1 GOARCH=wasm go build -o ../bin/sqlc-gen-go.wasm main.go
16+
bin/$(BIN_NAME).wasm: bin/$(BIN_NAME)
17+
cd plugin && GOOS=wasip1 GOARCH=wasm go build -o ../bin/$(BIN_NAME).wasm main.go
1618

1719
bin:
1820
mkdir -p bin

README.md

Lines changed: 142 additions & 117 deletions
Original file line numberDiff line numberDiff line change
@@ -1,157 +1,182 @@
1-
# sqlc-gen-go
1+
# sqlc-go-codegen-metaquery
2+
3+
A fork of [`sqlc-gen-go`](https://github.com/sqlc-dev/sqlc-gen-go) that emits
4+
per-query **metadata** alongside the usual sqlc output, plus a small runtime
5+
that wraps any query as a CTE so you can compose filters, ordering,
6+
pagination, and aggregations dynamically — without ever modifying the
7+
original SQL.
8+
9+
Built for **Go + Postgres + pgx/v5 + sqlc**. Drop-in compatible with stock
10+
sqlc-gen-go: the existing `q.ListAuthors(ctx)` methods are unchanged; the
11+
metaquery surface is purely additive.
12+
13+
```go
14+
// Your sqlc query:
15+
// -- name: ListAuthors :many
16+
// SELECT id, name, bio FROM authors ORDER BY name;
17+
18+
// Wrap it, add filters/pagination, run it — all at runtime:
19+
res, _ := mqpgx.Scan[db.Author](ctx, conn,
20+
db.WrapListAuthors().
21+
ApplyFilter(db.ListAuthorsCols.Name.ILike("%ada%")).
22+
ApplyOrder(db.ListAuthorsCols.CreatedAt.Desc()).
23+
ApplyPagination(metaquery.PageRequest{Page: 0, Size: 20, Total: true}))
24+
25+
// res.Data is []db.Author
26+
// res.Meta has the applied Filter/OrderBy/Pagination + column metadata,
27+
// ready to serialize as an HTTP response body.
28+
```
29+
30+
See [`examples/pgx/`](examples/pgx/) for a full runnable demo (docker-compose
31+
pg17, migrations, seed, JSON-output CLI).
32+
33+
## Why use this
234

3-
> [!IMPORTANT]
4-
> This repository is read-only. It contains a working Go codegen plugin extracted from https://github.com/sqlc-dev/sqlc which you can fork and modify to meet your needs.
35+
Stock sqlc is great for static queries but painful once you need dynamic
36+
filtering, pagination, or admin UIs on top. Typical workarounds:
37+
- Write N variants of a query — explodes quickly
38+
- Drop to `database/sql` + a query builder — loses sqlc's type safety
39+
- Build your own metadata indirection — what this project is
540

6-
See [Building from source](#building-from-source) and [Migrating from sqlc's built-in Go codegen](#migrating-from-sqlcs-built-in-go-codegen) if you want to use a modified fork in your project.
41+
This fork keeps sqlc's compile-time query validation and generated types,
42+
adds a typed wrapper per query, and gives you a runtime API that's
43+
JSON-round-trippable for HTTP endpoints and admin consoles.
744

8-
## Usage
45+
## Installation
46+
47+
This repo is a standalone fork, not registered as a GitHub fork. Install
48+
from source:
49+
50+
```sh
51+
git clone https://github.com/iodesystems/sqlc-go-codegen-metaquery.git
52+
cd sqlc-go-codegen-metaquery
53+
make all # produces bin/sqlc-go-codegen-metaquery (+ .wasm)
54+
```
55+
56+
Point sqlc at the binary:
957

1058
```yaml
59+
# sqlc.yaml
1160
version: '2'
1261
plugins:
13-
- name: golang
14-
wasm:
15-
url: https://downloads.sqlc.dev/plugin/sqlc-gen-go_1.6.0.wasm
16-
sha256: 3e54bb8ba8911e939d0f67ddccb710f79799e72e0c51516586414b3f3be12cfc
62+
- name: metaquery
63+
process:
64+
cmd: /path/to/sqlc-go-codegen-metaquery
1765
sql:
1866
- schema: schema.sql
1967
queries: query.sql
2068
engine: postgresql
2169
codegen:
22-
- plugin: golang
70+
- plugin: metaquery
2371
out: db
2472
options:
2573
package: db
2674
sql_package: pgx/v5
75+
emit_db_tags: true
76+
emit_json_tags: true
77+
emit_metaquery: cols # off | meta | wrap | cols (default: cols)
2778
```
2879
29-
## Building from source
80+
Then `sqlc generate` produces the usual `db/query.sql.go` + `db/models.go` +
81+
a new `db/query.sql.metaquery.go` carrying the per-query metadata and typed
82+
helpers.
3083

31-
Assuming you have the Go toolchain set up, from the project root you can simply `make all`.
84+
In your Go code, import the runtime:
3285

33-
```sh
34-
make all
86+
```go
87+
import (
88+
"github.com/iodesystems/sqlc-go-codegen-metaquery/metaquery"
89+
"github.com/iodesystems/sqlc-go-codegen-metaquery/metaquery/mqpgx"
90+
)
3591
```
3692

37-
This will produce a standalone binary and a WASM blob in the `bin` directory.
38-
They don't depend on each other, they're just two different plugin styles. You can
39-
use either with sqlc, but we recommend WASM and all of the configuration examples
40-
here assume you're using a WASM plugin.
93+
The `mqpgx` adapter lives in a sibling Go module so users of the core
94+
`metaquery` package with a different driver aren't forced to pull in pgx.
4195

42-
To use a local WASM build with sqlc, just update your configuration with a `file://`
43-
URL pointing at the WASM blob in your `bin` directory:
96+
## What it emits
4497

45-
```yaml
46-
plugins:
47-
- name: golang
48-
wasm:
49-
url: file:///path/to/bin/sqlc-gen-go.wasm
50-
sha256: ""
51-
```
98+
For each query sqlc processes, three symbols (at the default `cols` level):
5299

53-
As-of sqlc v1.24.0 the `sha256` is optional, but without it sqlc won't cache your
54-
module internally which will impact performance.
100+
| Symbol | Purpose |
101+
| --- | --- |
102+
| `MetaListAuthors metaquery.Query` | Runtime-readable metadata: the SQL text, columns (Name, GoType, DBType, NotNull, Table, …), args, source file, etc. JSON-serializable. |
103+
| `WrapListAuthors(args...) *metaquery.Builder` | Typed constructor that binds the original query's positional args at compile time and returns a Builder ready for filter/order/pagination/aggregation methods. |
104+
| `ListAuthorsCols struct{...}` | Typed column references. `ListAuthorsCols.Name` is a `TextCol`; `ListAuthorsCols.ID` is an `IntCol`. Each exposes ops appropriate to its type (`ILike(string)` on text, `Gt(int64)`, `Between(int64, int64)` on int, etc.). Column names and op/value types are compile-time checked. |
55105

56-
## Migrating from sqlc's built-in Go codegen
106+
Six column kinds ship by default (Text/Int/Float/Bool/Time/Bytes), with
107+
`AnyCol` as the escape hatch for arrays, enums, pgtype.Numeric, UUIDs, etc.
57108

58-
We’ve worked hard to make switching to sqlc-gen-go as seamless as possible. Let’s say you’re generating Go code today using a sqlc.yaml configuration that looks something like this:
109+
## Emission levels
59110

60-
```yaml
61-
version: 2
62-
sql:
63-
- schema: "query.sql"
64-
queries: "query.sql"
65-
engine: "postgresql"
66-
gen:
67-
go:
68-
package: "db"
69-
out: "db"
70-
emit_json_tags: true
71-
emit_pointers_for_null_types: true
72-
query_parameter_limit: 5
73-
overrides:
74-
- column: "authors.id"
75-
go_type: "your/package.SomeType"
76-
rename:
77-
foo: "bar"
78-
```
111+
You pay only for what you use. Set globally via `emit_metaquery` or per-query
112+
with `-- metaquery: <level>`:
79113

80-
To use the sqlc-gen-go WASM plugin for Go codegen, your config will instead look something like this:
114+
| Level | Emits | Typical use |
115+
| --- | --- | --- |
116+
| `off` | nothing | Query is only called via the regular sqlc method; no dynamic wrapping needed |
117+
| `meta` | `Meta<Name>` only | You want runtime introspection (schema export, generic API handler) without the builder surface |
118+
| `wrap` | `+ Wrap<Name>(args...)` | Typed wrappers but filter against column names as strings |
119+
| `cols` *(default)* | `+ <Name>Cols` | Full Tier 2 — typed wrappers + typed column refs |
81120

82-
```yaml
83-
version: 2
84-
plugins:
85-
- name: golang
86-
wasm:
87-
url: https://downloads.sqlc.dev/plugin/sqlc-gen-go_1.6.0.wasm
88-
sha256: 3e54bb8ba8911e939d0f67ddccb710f79799e72e0c51516586414b3f3be12cfc
89-
sql:
90-
- schema: "query.sql"
91-
queries: "query.sql"
92-
engine: "postgresql"
93-
codegen:
94-
- plugin: golang
95-
out: "db"
96-
options:
97-
package: "db"
98-
emit_json_tags: true
99-
emit_pointers_for_null_types: true
100-
query_parameter_limit: 5
101-
overrides:
102-
- column: "authors.id"
103-
go_type: "your/package.SomeType"
104-
rename:
105-
foo: "bar"
121+
Per-query override example:
122+
123+
```sql
124+
-- name: TruncateAll :exec
125+
-- metaquery: off
126+
-- (Only called from seed/migration code; no builder needed.)
127+
TRUNCATE authors RESTART IDENTITY CASCADE;
106128
```
107129

108-
The differences are:
109-
* An additional top-level `plugins` list with an entry for the Go codegen WASM plugin. If you’ve built the plugin from source you’ll want to use a `file://` URL. The `sha256` field is required, but will be optional in the upcoming sqlc v1.24.0 release.
110-
* Within the `sql` block, rather than `gen` with `go` nested beneath you’ll have a `codegen` list with an entry referencing the plugin name from the top-level `plugins` list. All options from the current `go` configuration block move as-is into the `options` block within `codegen`. The only special case is `out`, which moves up a level into the `codegen` configuration itself.
130+
## Safety surface
111131

112-
### Global overrides and renames
132+
| Failure mode | Caught | How |
133+
| --- | --- | --- |
134+
| Wrong wrapper arg type | compile time | `WrapGetAuthor("not an int")` won't compile |
135+
| Unknown column in `.Where`/`.OrderBy`/etc | compile time (via typed cols) or pre-query (via whitelist) | `ListAuthorsCols.Typo` → compile error; `.Where("typo",...)` → Build-time error |
136+
| Wrong op for column type | compile time (typed cols) or pre-query (ValidateFilter) | `IntCol` has no `.ILike`; `Filter{Op: OpILike}` on an int column → `op "ILIKE" not valid for column "id" (int64/int)` |
137+
| Wrong value type | compile time (typed cols) or pre-query (ValidateFilter) | `IntCol.Eq` takes `int64`; JSON-driven filters with string values for int columns are rejected before any query runs |
138+
| Scan-struct shape drift | pre-query | `Validate[T]` reflects on T and diffs against `b.OutputColumns()` |
139+
| Malformed raw SQL in `WhereExpr`/`Agg` | query time | Passed through to Postgres verbatim; caller owns safety |
113140

114-
If you have global overrides or renames configured, you’ll need to move those to the new top-level `options` field. Replace the existing `go` field name with the name you gave your plugin in the `plugins` list. We’ve used `"golang"` in this example.
141+
The validator is also callable as a library function (`metaquery.ValidateFilter(q, f)`)
142+
so JSON-driven HTTP handlers can fail fast on bad client input, using the
143+
same rules the builder applies internally.
115144

116-
If your existing configuration looks like this:
145+
## Relationship to upstream
117146

118-
```yaml
119-
version: "2"
120-
overrides:
121-
go:
122-
rename:
123-
id: "Identifier"
124-
overrides:
125-
- db_type: "timestamptz"
126-
nullable: true
127-
engine: "postgresql"
128-
go_type:
129-
import: "gopkg.in/guregu/null.v4"
130-
package: "null"
131-
type: "Time"
132-
...
147+
This is a clean fork of
148+
[`github.com/sqlc-dev/sqlc-gen-go`](https://github.com/sqlc-dev/sqlc-gen-go),
149+
not a registered GitHub fork. Upstream is tracked as a git remote:
150+
151+
```sh
152+
git remote -v
153+
# origin git@github.com:IodeSystems/sqlc-go-codegen-metaquery.git
154+
# upstream https://github.com/sqlc-dev/sqlc-gen-go.git
133155
```
134156

135-
Then your updated configuration would look something like this:
157+
To sync upstream updates:
136158

137-
```yaml
138-
version: "2"
139-
plugins:
140-
- name: golang
141-
wasm:
142-
url: https://downloads.sqlc.dev/plugin/sqlc-gen-go_1.6.0.wasm
143-
sha256: 3e54bb8ba8911e939d0f67ddccb710f79799e72e0c51516586414b3f3be12cfc
144-
options:
145-
golang:
146-
rename:
147-
id: "Identifier"
148-
overrides:
149-
- db_type: "timestamptz"
150-
nullable: true
151-
engine: "postgresql"
152-
go_type:
153-
import: "gopkg.in/guregu/null.v4"
154-
package: "null"
155-
type: "Time"
156-
...
159+
```sh
160+
git fetch upstream
161+
git merge upstream/main # resolve conflicts in the three touched files
162+
# (internal/gen.go, internal/result.go, go.mod + Makefile)
163+
# import-path conflicts resolve mechanically with sed.
157164
```
165+
166+
## Caveats / non-goals
167+
168+
- **Postgres + pgx/v5 only.** MySQL and SQLite aren't blocked by the runtime
169+
design but aren't implemented; the `mqpgx` adapter is pgx-specific.
170+
- **No WASM distribution yet.** Build the binary from source; if you need
171+
WASM, `make all` produces one in `bin/`.
172+
- **No released versions yet.** Pin a commit sha or `main` for now.
173+
- **The builder wraps, never rewrites.** A filter references the *output*
174+
columns of the original query. If you need to filter on a column the
175+
query doesn't project, either widen the query or use `WhereExpr`.
176+
- **`Sum`/`Avg`/`Min`/`Max` aggregates project the source column's type
177+
without null-safety** — aggregates over empty groups return NULL. Use
178+
`Agg("x", "coalesce(sum(y), 0)", "int64")` as the escape hatch.
179+
180+
## License
181+
182+
MIT — same as upstream. See [LICENSE](LICENSE).

examples/pgx/Makefile

Lines changed: 55 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,55 @@
1+
.PHONY: up down wait plugin generate build seed list list-typed counts demo clean
2+
3+
DSN ?= postgres://demo:demo@localhost:5544/demo?sslmode=disable
4+
SQLC ?= $(shell command -v sqlc 2>/dev/null || echo $${HOME}/go/bin/sqlc)
5+
PLUGIN := ../../bin/sqlc-go-codegen-metaquery
6+
DEMO := ./bin/demo
7+
8+
up:
9+
docker compose up -d
10+
11+
down:
12+
docker compose down -v
13+
14+
wait:
15+
@printf "waiting for pg..."
16+
@until docker compose exec -T db pg_isready -U demo -d demo >/dev/null 2>&1; do \
17+
printf "."; sleep 1; \
18+
done; echo " ready"
19+
20+
plugin:
21+
$(MAKE) -C ../.. bin/sqlc-go-codegen-metaquery
22+
23+
generate: plugin
24+
$(SQLC) generate
25+
26+
build:
27+
mkdir -p bin
28+
go build -o $(DEMO) .
29+
30+
seed: build
31+
$(DEMO) seed --db '$(DSN)'
32+
33+
list: build
34+
$(DEMO) list --db '$(DSN)' --size 10
35+
36+
list-typed: build
37+
$(DEMO) list --typed --db '$(DSN)' --size 10
38+
39+
counts: build
40+
$(DEMO) counts --db '$(DSN)'
41+
42+
# Full walk-through: start db, generate code, seed, and show each subcommand.
43+
demo: up wait generate build seed
44+
@echo '---- list (untyped, page 1 of 2) ----'
45+
$(DEMO) list --db '$(DSN)' --size 3 \
46+
--where '[{"column":"name","op":"ILIKE","value":"%a%"}]' \
47+
--order '[{"column":"name","dir":"ASC"}]'
48+
@echo '---- list --typed ----'
49+
$(DEMO) list --typed --db '$(DSN)' --size 3
50+
@echo '---- counts (group by author_id) ----'
51+
$(DEMO) counts --db '$(DSN)' \
52+
--order '[{"column":"author_id","dir":"ASC"}]'
53+
54+
clean:
55+
rm -rf bin

0 commit comments

Comments
 (0)