|
1 | | -# sqlc-gen-go |
| 1 | +# sqlc-go-codegen-metaquery |
| 2 | + |
| 3 | +A fork of [`sqlc-gen-go`](https://github.com/sqlc-dev/sqlc-gen-go) that emits |
| 4 | +per-query **metadata** alongside the usual sqlc output, plus a small runtime |
| 5 | +that wraps any query as a CTE so you can compose filters, ordering, |
| 6 | +pagination, and aggregations dynamically — without ever modifying the |
| 7 | +original SQL. |
| 8 | + |
| 9 | +Built for **Go + Postgres + pgx/v5 + sqlc**. Drop-in compatible with stock |
| 10 | +sqlc-gen-go: the existing `q.ListAuthors(ctx)` methods are unchanged; the |
| 11 | +metaquery surface is purely additive. |
| 12 | + |
| 13 | +```go |
| 14 | +// Your sqlc query: |
| 15 | +// -- name: ListAuthors :many |
| 16 | +// SELECT id, name, bio FROM authors ORDER BY name; |
| 17 | + |
| 18 | +// Wrap it, add filters/pagination, run it — all at runtime: |
| 19 | +res, _ := mqpgx.Scan[db.Author](ctx, conn, |
| 20 | + db.WrapListAuthors(). |
| 21 | + ApplyFilter(db.ListAuthorsCols.Name.ILike("%ada%")). |
| 22 | + ApplyOrder(db.ListAuthorsCols.CreatedAt.Desc()). |
| 23 | + ApplyPagination(metaquery.PageRequest{Page: 0, Size: 20, Total: true})) |
| 24 | + |
| 25 | +// res.Data is []db.Author |
| 26 | +// res.Meta has the applied Filter/OrderBy/Pagination + column metadata, |
| 27 | +// ready to serialize as an HTTP response body. |
| 28 | +``` |
| 29 | + |
| 30 | +See [`examples/pgx/`](examples/pgx/) for a full runnable demo (docker-compose |
| 31 | +pg17, migrations, seed, JSON-output CLI). |
| 32 | + |
| 33 | +## Why use this |
2 | 34 |
|
3 | | -> [!IMPORTANT] |
4 | | -> This repository is read-only. It contains a working Go codegen plugin extracted from https://github.com/sqlc-dev/sqlc which you can fork and modify to meet your needs. |
| 35 | +Stock sqlc is great for static queries but painful once you need dynamic |
| 36 | +filtering, pagination, or admin UIs on top. Typical workarounds: |
| 37 | +- Write N variants of a query — explodes quickly |
| 38 | +- Drop to `database/sql` + a query builder — loses sqlc's type safety |
| 39 | +- Build your own metadata indirection — what this project is |
5 | 40 |
|
6 | | -See [Building from source](#building-from-source) and [Migrating from sqlc's built-in Go codegen](#migrating-from-sqlcs-built-in-go-codegen) if you want to use a modified fork in your project. |
| 41 | +This fork keeps sqlc's compile-time query validation and generated types, |
| 42 | +adds a typed wrapper per query, and gives you a runtime API that's |
| 43 | +JSON-round-trippable for HTTP endpoints and admin consoles. |
7 | 44 |
|
8 | | -## Usage |
| 45 | +## Installation |
| 46 | + |
| 47 | +This repo is a standalone fork, not registered as a GitHub fork. Install |
| 48 | +from source: |
| 49 | + |
| 50 | +```sh |
| 51 | +git clone https://github.com/iodesystems/sqlc-go-codegen-metaquery.git |
| 52 | +cd sqlc-go-codegen-metaquery |
| 53 | +make all # produces bin/sqlc-go-codegen-metaquery (+ .wasm) |
| 54 | +``` |
| 55 | + |
| 56 | +Point sqlc at the binary: |
9 | 57 |
|
10 | 58 | ```yaml |
| 59 | +# sqlc.yaml |
11 | 60 | version: '2' |
12 | 61 | plugins: |
13 | | -- name: golang |
14 | | - wasm: |
15 | | - url: https://downloads.sqlc.dev/plugin/sqlc-gen-go_1.6.0.wasm |
16 | | - sha256: 3e54bb8ba8911e939d0f67ddccb710f79799e72e0c51516586414b3f3be12cfc |
| 62 | +- name: metaquery |
| 63 | + process: |
| 64 | + cmd: /path/to/sqlc-go-codegen-metaquery |
17 | 65 | sql: |
18 | 66 | - schema: schema.sql |
19 | 67 | queries: query.sql |
20 | 68 | engine: postgresql |
21 | 69 | codegen: |
22 | | - - plugin: golang |
| 70 | + - plugin: metaquery |
23 | 71 | out: db |
24 | 72 | options: |
25 | 73 | package: db |
26 | 74 | sql_package: pgx/v5 |
| 75 | + emit_db_tags: true |
| 76 | + emit_json_tags: true |
| 77 | + emit_metaquery: cols # off | meta | wrap | cols (default: cols) |
27 | 78 | ``` |
28 | 79 |
|
29 | | -## Building from source |
| 80 | +Then `sqlc generate` produces the usual `db/query.sql.go` + `db/models.go` + |
| 81 | +a new `db/query.sql.metaquery.go` carrying the per-query metadata and typed |
| 82 | +helpers. |
30 | 83 |
|
31 | | -Assuming you have the Go toolchain set up, from the project root you can simply `make all`. |
| 84 | +In your Go code, import the runtime: |
32 | 85 |
|
33 | | -```sh |
34 | | -make all |
| 86 | +```go |
| 87 | +import ( |
| 88 | + "github.com/iodesystems/sqlc-go-codegen-metaquery/metaquery" |
| 89 | + "github.com/iodesystems/sqlc-go-codegen-metaquery/metaquery/mqpgx" |
| 90 | +) |
35 | 91 | ``` |
36 | 92 |
|
37 | | -This will produce a standalone binary and a WASM blob in the `bin` directory. |
38 | | -They don't depend on each other, they're just two different plugin styles. You can |
39 | | -use either with sqlc, but we recommend WASM and all of the configuration examples |
40 | | -here assume you're using a WASM plugin. |
| 93 | +The `mqpgx` adapter lives in a sibling Go module so users of the core |
| 94 | +`metaquery` package with a different driver aren't forced to pull in pgx. |
41 | 95 |
|
42 | | -To use a local WASM build with sqlc, just update your configuration with a `file://` |
43 | | -URL pointing at the WASM blob in your `bin` directory: |
| 96 | +## What it emits |
44 | 97 |
|
45 | | -```yaml |
46 | | -plugins: |
47 | | -- name: golang |
48 | | - wasm: |
49 | | - url: file:///path/to/bin/sqlc-gen-go.wasm |
50 | | - sha256: "" |
51 | | -``` |
| 98 | +For each query sqlc processes, three symbols (at the default `cols` level): |
52 | 99 |
|
53 | | -As-of sqlc v1.24.0 the `sha256` is optional, but without it sqlc won't cache your |
54 | | -module internally which will impact performance. |
| 100 | +| Symbol | Purpose | |
| 101 | +| --- | --- | |
| 102 | +| `MetaListAuthors metaquery.Query` | Runtime-readable metadata: the SQL text, columns (Name, GoType, DBType, NotNull, Table, …), args, source file, etc. JSON-serializable. | |
| 103 | +| `WrapListAuthors(args...) *metaquery.Builder` | Typed constructor that binds the original query's positional args at compile time and returns a Builder ready for filter/order/pagination/aggregation methods. | |
| 104 | +| `ListAuthorsCols struct{...}` | Typed column references. `ListAuthorsCols.Name` is a `TextCol`; `ListAuthorsCols.ID` is an `IntCol`. Each exposes ops appropriate to its type (`ILike(string)` on text, `Gt(int64)`, `Between(int64, int64)` on int, etc.). Column names and op/value types are compile-time checked. | |
55 | 105 |
|
56 | | -## Migrating from sqlc's built-in Go codegen |
| 106 | +Six column kinds ship by default (Text/Int/Float/Bool/Time/Bytes), with |
| 107 | +`AnyCol` as the escape hatch for arrays, enums, pgtype.Numeric, UUIDs, etc. |
57 | 108 |
|
58 | | -We’ve worked hard to make switching to sqlc-gen-go as seamless as possible. Let’s say you’re generating Go code today using a sqlc.yaml configuration that looks something like this: |
| 109 | +## Emission levels |
59 | 110 |
|
60 | | -```yaml |
61 | | -version: 2 |
62 | | -sql: |
63 | | -- schema: "query.sql" |
64 | | - queries: "query.sql" |
65 | | - engine: "postgresql" |
66 | | - gen: |
67 | | - go: |
68 | | - package: "db" |
69 | | - out: "db" |
70 | | - emit_json_tags: true |
71 | | - emit_pointers_for_null_types: true |
72 | | - query_parameter_limit: 5 |
73 | | - overrides: |
74 | | - - column: "authors.id" |
75 | | - go_type: "your/package.SomeType" |
76 | | - rename: |
77 | | - foo: "bar" |
78 | | -``` |
| 111 | +You pay only for what you use. Set globally via `emit_metaquery` or per-query |
| 112 | +with `-- metaquery: <level>`: |
79 | 113 |
|
80 | | -To use the sqlc-gen-go WASM plugin for Go codegen, your config will instead look something like this: |
| 114 | +| Level | Emits | Typical use | |
| 115 | +| --- | --- | --- | |
| 116 | +| `off` | nothing | Query is only called via the regular sqlc method; no dynamic wrapping needed | |
| 117 | +| `meta` | `Meta<Name>` only | You want runtime introspection (schema export, generic API handler) without the builder surface | |
| 118 | +| `wrap` | `+ Wrap<Name>(args...)` | Typed wrappers but filter against column names as strings | |
| 119 | +| `cols` *(default)* | `+ <Name>Cols` | Full Tier 2 — typed wrappers + typed column refs | |
81 | 120 |
|
82 | | -```yaml |
83 | | -version: 2 |
84 | | -plugins: |
85 | | -- name: golang |
86 | | - wasm: |
87 | | - url: https://downloads.sqlc.dev/plugin/sqlc-gen-go_1.6.0.wasm |
88 | | - sha256: 3e54bb8ba8911e939d0f67ddccb710f79799e72e0c51516586414b3f3be12cfc |
89 | | -sql: |
90 | | -- schema: "query.sql" |
91 | | - queries: "query.sql" |
92 | | - engine: "postgresql" |
93 | | - codegen: |
94 | | - - plugin: golang |
95 | | - out: "db" |
96 | | - options: |
97 | | - package: "db" |
98 | | - emit_json_tags: true |
99 | | - emit_pointers_for_null_types: true |
100 | | - query_parameter_limit: 5 |
101 | | - overrides: |
102 | | - - column: "authors.id" |
103 | | - go_type: "your/package.SomeType" |
104 | | - rename: |
105 | | - foo: "bar" |
| 121 | +Per-query override example: |
| 122 | + |
| 123 | +```sql |
| 124 | +-- name: TruncateAll :exec |
| 125 | +-- metaquery: off |
| 126 | +-- (Only called from seed/migration code; no builder needed.) |
| 127 | +TRUNCATE authors RESTART IDENTITY CASCADE; |
106 | 128 | ``` |
107 | 129 |
|
108 | | -The differences are: |
109 | | -* An additional top-level `plugins` list with an entry for the Go codegen WASM plugin. If you’ve built the plugin from source you’ll want to use a `file://` URL. The `sha256` field is required, but will be optional in the upcoming sqlc v1.24.0 release. |
110 | | -* Within the `sql` block, rather than `gen` with `go` nested beneath you’ll have a `codegen` list with an entry referencing the plugin name from the top-level `plugins` list. All options from the current `go` configuration block move as-is into the `options` block within `codegen`. The only special case is `out`, which moves up a level into the `codegen` configuration itself. |
| 130 | +## Safety surface |
111 | 131 |
|
112 | | -### Global overrides and renames |
| 132 | +| Failure mode | Caught | How | |
| 133 | +| --- | --- | --- | |
| 134 | +| Wrong wrapper arg type | compile time | `WrapGetAuthor("not an int")` won't compile | |
| 135 | +| Unknown column in `.Where`/`.OrderBy`/etc | compile time (via typed cols) or pre-query (via whitelist) | `ListAuthorsCols.Typo` → compile error; `.Where("typo",...)` → Build-time error | |
| 136 | +| Wrong op for column type | compile time (typed cols) or pre-query (ValidateFilter) | `IntCol` has no `.ILike`; `Filter{Op: OpILike}` on an int column → `op "ILIKE" not valid for column "id" (int64/int)` | |
| 137 | +| Wrong value type | compile time (typed cols) or pre-query (ValidateFilter) | `IntCol.Eq` takes `int64`; JSON-driven filters with string values for int columns are rejected before any query runs | |
| 138 | +| Scan-struct shape drift | pre-query | `Validate[T]` reflects on T and diffs against `b.OutputColumns()` | |
| 139 | +| Malformed raw SQL in `WhereExpr`/`Agg` | query time | Passed through to Postgres verbatim; caller owns safety | |
113 | 140 |
|
114 | | -If you have global overrides or renames configured, you’ll need to move those to the new top-level `options` field. Replace the existing `go` field name with the name you gave your plugin in the `plugins` list. We’ve used `"golang"` in this example. |
| 141 | +The validator is also callable as a library function (`metaquery.ValidateFilter(q, f)`) |
| 142 | +so JSON-driven HTTP handlers can fail fast on bad client input, using the |
| 143 | +same rules the builder applies internally. |
115 | 144 |
|
116 | | -If your existing configuration looks like this: |
| 145 | +## Relationship to upstream |
117 | 146 |
|
118 | | -```yaml |
119 | | -version: "2" |
120 | | -overrides: |
121 | | - go: |
122 | | - rename: |
123 | | - id: "Identifier" |
124 | | - overrides: |
125 | | - - db_type: "timestamptz" |
126 | | - nullable: true |
127 | | - engine: "postgresql" |
128 | | - go_type: |
129 | | - import: "gopkg.in/guregu/null.v4" |
130 | | - package: "null" |
131 | | - type: "Time" |
132 | | -... |
| 147 | +This is a clean fork of |
| 148 | +[`github.com/sqlc-dev/sqlc-gen-go`](https://github.com/sqlc-dev/sqlc-gen-go), |
| 149 | +not a registered GitHub fork. Upstream is tracked as a git remote: |
| 150 | + |
| 151 | +```sh |
| 152 | +git remote -v |
| 153 | +# origin git@github.com:IodeSystems/sqlc-go-codegen-metaquery.git |
| 154 | +# upstream https://github.com/sqlc-dev/sqlc-gen-go.git |
133 | 155 | ``` |
134 | 156 |
|
135 | | -Then your updated configuration would look something like this: |
| 157 | +To sync upstream updates: |
136 | 158 |
|
137 | | -```yaml |
138 | | -version: "2" |
139 | | -plugins: |
140 | | -- name: golang |
141 | | - wasm: |
142 | | - url: https://downloads.sqlc.dev/plugin/sqlc-gen-go_1.6.0.wasm |
143 | | - sha256: 3e54bb8ba8911e939d0f67ddccb710f79799e72e0c51516586414b3f3be12cfc |
144 | | -options: |
145 | | - golang: |
146 | | - rename: |
147 | | - id: "Identifier" |
148 | | - overrides: |
149 | | - - db_type: "timestamptz" |
150 | | - nullable: true |
151 | | - engine: "postgresql" |
152 | | - go_type: |
153 | | - import: "gopkg.in/guregu/null.v4" |
154 | | - package: "null" |
155 | | - type: "Time" |
156 | | -... |
| 159 | +```sh |
| 160 | +git fetch upstream |
| 161 | +git merge upstream/main # resolve conflicts in the three touched files |
| 162 | + # (internal/gen.go, internal/result.go, go.mod + Makefile) |
| 163 | + # import-path conflicts resolve mechanically with sed. |
157 | 164 | ``` |
| 165 | + |
| 166 | +## Caveats / non-goals |
| 167 | + |
| 168 | +- **Postgres + pgx/v5 only.** MySQL and SQLite aren't blocked by the runtime |
| 169 | + design but aren't implemented; the `mqpgx` adapter is pgx-specific. |
| 170 | +- **No WASM distribution yet.** Build the binary from source; if you need |
| 171 | + WASM, `make all` produces one in `bin/`. |
| 172 | +- **No released versions yet.** Pin a commit sha or `main` for now. |
| 173 | +- **The builder wraps, never rewrites.** A filter references the *output* |
| 174 | + columns of the original query. If you need to filter on a column the |
| 175 | + query doesn't project, either widen the query or use `WhereExpr`. |
| 176 | +- **`Sum`/`Avg`/`Min`/`Max` aggregates project the source column's type |
| 177 | + without null-safety** — aggregates over empty groups return NULL. Use |
| 178 | + `Agg("x", "coalesce(sum(y), 0)", "int64")` as the escape hatch. |
| 179 | + |
| 180 | +## License |
| 181 | + |
| 182 | +MIT — same as upstream. See [LICENSE](LICENSE). |
0 commit comments