change ParseResponse - returns multiple statements from single call

Author: asmyasnikov

docs/guides/engine-plugins.md

@@ -16,7 +16,7 @@ Data returned by the engine plugin (SQL text, parameters, columns) is passed thr
 
 An engine plugin is an external process that implements one RPC:
 
-- **Parse** — accepts the query text and either schema SQL or connection parameters, and returns processed SQL, parameter list, and result columns.
+- **Parse** — accepts the **entire contents** of one query file (e.g. `query.sql`) and either schema SQL or connection parameters; returns **one Statement per query block** in that file (each with sql, parameters, columns, and name/cmd).
 
 Process plugins (e.g. written in Go) talk to sqlc over **stdin/stdout** using **Protocol Buffers**. The protocol is defined in `protos/engine/engine.proto`.
 
@@ -102,40 +102,54 @@ The engine API exposes only **Parse**. There are no separate methods for catalog
 
 ### 2. Parse
 
+sqlc calls Parse **once per query file** (e.g. once for `query.sql`). The plugin receives the full file contents and returns one **Statement** per query block in that file. sqlc then passes each statement to the codegen plugin as a separate query.
+
 **Request**
 
-- `sql` — The query text to parse.
+- `sql` — The **entire contents** of one query file (all query blocks, with `-- name: X :one`-style comments).
 - `schema_source` — One of:
-  - `schema_sql`: schema as in a schema.sql file (used for schema-based parsing).
+  - `schema_sql`: full schema as in schema.sql (for schema-based parsing).
   - `connection_params`: DSN and options for database-only mode.
 
 **Response**
 
-- `sql` — Processed query text. Often the same as input; with a schema you may expand `*` into explicit columns.
-- `parameters` — List of parameters (position/name, type, nullable, array, etc.).
-- `columns` — List of result columns (name, type, nullable, table/schema if known).
+Return `statements`: one `Statement` per query block. Each `Statement` has:
+
+- `name` — Query name (from `-- name: GetUser` etc.).
+- `cmd` — Command/type: use the `Cmd` enum (`engine.Cmd_CMD_ONE`, `engine.Cmd_CMD_MANY`, `engine.Cmd_CMD_EXEC`, etc.). See `protos/engine/engine.proto` for the full list.
+- `sql` — Processed SQL for that block (as-is or with `*` expanded using schema).
+- `parameters` — Parameters for this statement.
+- `columns` — Result columns (names, types, nullability, etc.) for this statement.
+
+The engine package provides helpers (optional) to split `query.sql` and parse `"-- name: X :cmd"` lines in the same way as the built-in engines:
 
-Example handler:
+- `engine.CommentSyntax` — Which comment styles to accept (`Dash`, `SlashStar`, `Hash`).
+- `engine.ParseNameAndCmd(line, syntax)` — Parses a single line like `"-- name: ListAuthors :many"` → `(name, cmd, ok)`. `cmd` is `engine.Cmd`.
+- `engine.QueryBlocks(content, syntax)` — Splits file content into `[]engine.QueryBlock` (each has `Name`, `Cmd`, `SQL`).
+- `engine.StatementMeta(name, cmd, sql)` — Builds a `*engine.Statement` with name/cmd/sql set; you add parameters and columns.
+
+Example handler using helpers:
 
 ```go
 func handleParse(req *engine.ParseRequest) (*engine.ParseResponse, error) {
-    sql := req.GetSql()
+    queryFileContent := req.GetSql()
+    syntax := engine.CommentSyntax{Dash: true, SlashStar: true, Hash: true}
 
     var schema *SchemaInfo
     if s := req.GetSchemaSql(); s != "" {
         schema = parseSchema(s)
     }
     // Or use req.GetConnectionParams() for database-only mode.
 
-    parameters := extractParameters(sql)
-    columns := extractColumns(sql, schema)
-    processedSQL := processSQL(sql, schema) // e.g. expand SELECT *
-
-    return &engine.ParseResponse{
-        Sql:        processedSQL,
-        Parameters: parameters,
-        Columns:    columns,
-    }, nil
+    blocks, _ := engine.QueryBlocks(queryFileContent, syntax)
+    var statements []*engine.Statement
+    for _, b := range blocks {
+        st := engine.StatementMeta(b.Name, b.Cmd, processSQL(b.SQL, schema))
+        st.Parameters = extractParameters(b.SQL)
+        st.Columns = extractColumns(b.SQL, schema)
+        statements = append(statements, st)
+    }
+    return &engine.ParseResponse{Statements: statements}, nil
 }
 ```
 
@@ -164,28 +178,29 @@ Invocation:
 sqlc-engine-mydb parse   # stdin: ParseRequest, stdout: ParseResponse
 ```
 
-The definition lives in `engine/engine.proto` (and generated Go in `pkg/engine`).
+The definition lives in `protos/engine/engine.proto` (generated Go in `pkg/engine`). After editing the proto, run `make proto-engine-plugin` to regenerate the Go code.
 
 ## Example
 
 The protocol and Go SDK are in this repository: `protos/engine/engine.proto` and `pkg/engine/` (including `sdk.go` with `engine.Run` and `engine.Handler`). Use them to build a binary that implements the Parse RPC; register it under `engines` in sqlc.yaml as shown above.
 
 ## Architecture
 
-For each `sql[]` block, `sqlc generate` branches on the configured engine: built-in (postgresql, mysql, sqlite) use the compiler and catalog; any engine listed under `engines:` in sqlc.yaml uses the plugin path (no compiler, schema + queries go to the plugin's Parse RPC, then output goes to codegen).
+For each `sql[]` block, `sqlc generate` branches on the configured engine: built-in (postgresql, mysql, sqlite) use the compiler and catalog; any engine listed under `engines:` in sqlc.yaml uses the plugin path (no compiler). For the plugin path, sqlc calls Parse **once per query file**, sending the full file contents and schema (or connection params). The plugin returns **N statements** (one per query block); sqlc passes each statement to codegen as a separate query.
 
 ```
 ┌─────────────────────────────────────────────────────────────────┐
-│  sqlc generate                                                  │
-│  1. Read sqlc.yaml, find engine for this sql block              │
-│  2. If plugin engine: call plugin parse (sql + schema_sql etc.)  │
-│  3. Use returned sql, parameters, columns in codegen             │
+│  sqlc generate (plugin engine)                                  │
+│  1. Per query file: one Parse(schema_sql|connection_params,      │
+│     full query file content)                                    │
+│  2. ParseResponse.statements = one Statement per query block     │
+│  3. Each statement → one codegen query (N helpers)               │
 └─────────────────────────────────────────────────────────────────┘
 
     sqlc                          sqlc-engine-mydb
       │──── spawn, args: ["parse"] ──────────────────────────────► │
-      │──── stdin: ParseRequest{sql, schema_sql|connection_params} ► │
-      │◄─── stdout: ParseResponse{sql, parameters, columns} ─────── │
+      │──── stdin: ParseRequest{sql=full query.sql, schema_sql|…}  ► │
+      │◄─── stdout: ParseResponse{statements: [stmt1, stmt2, …]} ── │
 ```
 
 ## See also

internal/cmd/plugin_engine.go

@@ -17,7 +17,6 @@ import (
 	"github.com/sqlc-dev/sqlc/internal/config"
 	"github.com/sqlc-dev/sqlc/internal/metadata"
 	"github.com/sqlc-dev/sqlc/internal/multierr"
-	"github.com/sqlc-dev/sqlc/internal/source"
 	"github.com/sqlc-dev/sqlc/internal/sql/catalog"
 	"github.com/sqlc-dev/sqlc/internal/sql/sqlpath"
 	"google.golang.org/protobuf/proto"
@@ -86,9 +85,6 @@ func (r *engineProcessRunner) parseRequest(ctx context.Context, req *pb.ParseReq
 	return resp, nil
 }
 
-// defaultCommentSyntax is used when parsing query names from plugin-engine query files.
-var defaultCommentSyntax = metadata.CommentSyntax(source.CommentSyntax{Dash: true, SlashStar: true, Hash: false})
-
 // runPluginQuerySet runs the plugin-engine path: schema and queries are sent to the
 // engine plugin via ParseRequest; the responses are turned into compiler.Result and
 // passed to ProcessResult. No AST or compiler parsing is used.
@@ -177,26 +173,24 @@ func runPluginQuerySet(ctx context.Context, rp resultProcessor, name, dir string
 			continue
 		}
 		queryContent := string(blob)
-		blocks, err := metadata.QueryBlocks(queryContent, defaultCommentSyntax)
+		resp, err := parseFn(queryContent)
 		if err != nil {
 			merr.Add(filename, queryContent, 0, err)
 			continue
 		}
-		for _, b := range blocks {
-			resp, err := parseFn(b.SQL)
-			if err != nil {
-				merr.Add(filename, queryContent, 0, err)
-				continue
-			}
-			q := pluginResponseToCompilerQuery(b.Name, b.Cmd, filepath.Base(filename), resp)
+		baseName := filepath.Base(filename)
+		stmts := resp.GetStatements()
+		for _, st := range stmts {
+			q := statementToCompilerQuery(st, baseName)
 			if q == nil {
 				continue
 			}
-			if _, exists := set[b.Name]; exists {
-				merr.Add(filename, queryContent, 0, fmt.Errorf("duplicate query name: %s", b.Name))
+			qName := st.GetName()
+			if _, exists := set[qName]; exists {
+				merr.Add(filename, queryContent, 0, fmt.Errorf("duplicate query name: %s", qName))
 				continue
 			}
-			set[b.Name] = struct{}{}
+			set[qName] = struct{}{}
 			queries = append(queries, q)
 		}
 	}
@@ -236,14 +230,17 @@ func loadSchemaSQL(schemaPaths []string, readFile func(string) ([]byte, error))
 	return strings.Join(parts, "\n"), nil
 }
 
-func pluginResponseToCompilerQuery(name, cmd, filename string, resp *pb.ParseResponse) *compiler.Query {
-	sqlTrimmed := strings.TrimSpace(resp.GetSql())
+// statementToCompilerQuery converts one engine.Statement from the plugin into a compiler.Query.
+func statementToCompilerQuery(st *pb.Statement, filename string) *compiler.Query {
+	if st == nil {
+		return nil
+	}
+	sqlTrimmed := strings.TrimSpace(st.GetSql())
 	if sqlTrimmed == "" {
 		return nil
 	}
-
 	var params []compiler.Parameter
-	for _, p := range resp.GetParameters() {
+	for _, p := range st.GetParameters() {
 		col := &compiler.Column{
 			DataType:  p.GetDataType(),
 			NotNull:   !p.GetNullable(),
@@ -256,9 +253,8 @@ func pluginResponseToCompilerQuery(name, cmd, filename string, resp *pb.ParseRes
 		}
 		params = append(params, compiler.Parameter{Number: pos, Column: col})
 	}
-
 	var columns []*compiler.Column
-	for _, c := range resp.GetColumns() {
+	for _, c := range st.GetColumns() {
 		columns = append(columns, &compiler.Column{
 			Name:      c.GetName(),
 			DataType:  c.GetDataType(),
@@ -267,12 +263,11 @@ func pluginResponseToCompilerQuery(name, cmd, filename string, resp *pb.ParseRes
 			ArrayDims: int(c.GetArrayDims()),
 		})
 	}
-
 	return &compiler.Query{
 		SQL: sqlTrimmed,
 		Metadata: metadata.Metadata{
-			Name:     name,
-			Cmd:      cmd,
+			Name:     st.GetName(),
+			Cmd:      pb.CmdToString(st.GetCmd()),
 			Filename: filename,
 		},
 		Params:  params,

internal/cmd/plugin_engine_test.go

@@ -106,9 +106,10 @@ func TestPluginPipeline_FullPipeline(t *testing.T) {
 		engineRecord.QuerySQL = querySQL
 		engineRecord.CalledWith = append(engineRecord.CalledWith, struct{ SchemaSQL, QuerySQL string }{schemaSQL, querySQL})
 		return &pb.ParseResponse{
-			Sql:        engineRecord.ReturnedSQL,
-			Parameters: engineRecord.ReturnedParams,
-			Columns:    engineRecord.ReturnedCols,
+			Statements: []*pb.Statement{{
+				Name: "GetUser", Cmd: pb.Cmd_CMD_ONE, Sql: engineRecord.ReturnedSQL,
+				Parameters: engineRecord.ReturnedParams, Columns: engineRecord.ReturnedCols,
+			}},
 		}, nil
 	}
 
@@ -277,28 +278,30 @@ func TestPluginPipeline_WithoutOverride_UsesPluginPackage(t *testing.T) {
 	}
 }
 
-// TestPluginPipeline_NBlocksNCalls verifies that N sqlc blocks in query.sql yield N plugin Parse calls,
-// and each call receives the schema (or connection params in databaseOnly mode).
+// TestPluginPipeline_NBlocksNCalls verifies that one Parse call receives full query.sql and schema,
+// and the plugin returns N statements (one per query block); sqlc then generates N helpers.
 func TestPluginPipeline_NBlocksNCalls(t *testing.T) {
 	const (
 		schemaContent = "CREATE TABLE users (id INT, name TEXT);"
 		block1        = "-- name: GetUser :one\nSELECT id, name FROM users WHERE id = $1"
 		block2        = "-- name: ListUsers :many\nSELECT id, name FROM users ORDER BY id"
 	)
 	queryContent := block1 + "\n\n" + block2
-	// QueryBlocks slices from " name: " line to the next " name: " (exclusive), so first block includes "\n\n".
-	expectedBlock1 := block1 + "\n\n"
-	expectedBlock2 := block2
 
 	engineRecord := &engineMockRecord{
-		ReturnedSQL: "SELECT id, name FROM users WHERE id = $1",
 		ReturnedParams: []*pb.Parameter{{Position: 1, DataType: "int", Nullable: false}},
 		ReturnedCols:   []*pb.Column{{Name: "id", DataType: "int", Nullable: false}, {Name: "name", DataType: "text", Nullable: false}},
 	}
+	var codegenReq *plugin.GenerateRequest
 	pluginParse := func(schemaSQL, querySQL string) (*pb.ParseResponse, error) {
 		engineRecord.Calls++
 		engineRecord.CalledWith = append(engineRecord.CalledWith, struct{ SchemaSQL, QuerySQL string }{schemaSQL, querySQL})
-		return &pb.ParseResponse{Sql: querySQL, Parameters: engineRecord.ReturnedParams, Columns: engineRecord.ReturnedCols}, nil
+		return &pb.ParseResponse{
+			Statements: []*pb.Statement{
+				{Name: "GetUser", Cmd: pb.Cmd_CMD_ONE, Sql: "SELECT id, name FROM users WHERE id = $1", Parameters: engineRecord.ReturnedParams, Columns: engineRecord.ReturnedCols},
+				{Name: "ListUsers", Cmd: pb.Cmd_CMD_MANY, Sql: "SELECT id, name FROM users ORDER BY id", Parameters: nil, Columns: engineRecord.ReturnedCols},
+			},
+		}, nil
 	}
 	conf, _ := config.ParseConfig(strings.NewReader(testPluginPipelineConfig))
 	inputs := &sourceFiles{
@@ -309,25 +312,29 @@ func TestPluginPipeline_NBlocksNCalls(t *testing.T) {
 	debug.ProcessPlugins = true
 	o := &Options{
 		Env: Env{Debug: debug}, Stderr: &bytes.Buffer{}, PluginParseFunc: pluginParse,
-		CodegenHandlerOverride: ext.HandleFunc(func(_ context.Context, req *plugin.GenerateRequest) (*plugin.GenerateResponse, error) { return &plugin.GenerateResponse{}, nil }),
+		CodegenHandlerOverride: ext.HandleFunc(func(_ context.Context, req *plugin.GenerateRequest) (*plugin.GenerateResponse, error) {
+			codegenReq = req
+			return &plugin.GenerateResponse{}, nil
+		}),
 	}
 	_, err := generate(context.Background(), inputs, o)
 	if err != nil {
 		t.Fatalf("generate failed: %v", err)
 	}
-	if n := len(engineRecord.CalledWith); n != 2 {
-		t.Errorf("expected 2 Parse calls (2 blocks), got %d", n)
+	if engineRecord.Calls != 1 {
+		t.Errorf("expected 1 Parse call (full query.sql), got %d", engineRecord.Calls)
 	}
-	for i, call := range engineRecord.CalledWith {
-		if call.SchemaSQL != schemaContent {
-			t.Errorf("Parse call %d: every call must receive schema; got schemaSQL %q", i+1, call.SchemaSQL)
-		}
+	if len(engineRecord.CalledWith) != 1 {
+		t.Fatalf("expected 1 CalledWith, got %d", len(engineRecord.CalledWith))
 	}
-	if len(engineRecord.CalledWith) >= 1 && engineRecord.CalledWith[0].QuerySQL != expectedBlock1 {
-		t.Errorf("Parse call 1: query must be first block; got %q", engineRecord.CalledWith[0].QuerySQL)
+	if engineRecord.CalledWith[0].SchemaSQL != schemaContent {
+		t.Errorf("Parse must receive schema; got %q", engineRecord.CalledWith[0].SchemaSQL)
 	}
-	if len(engineRecord.CalledWith) >= 2 && engineRecord.CalledWith[1].QuerySQL != expectedBlock2 {
-		t.Errorf("Parse call 2: query must be second block; got %q", engineRecord.CalledWith[1].QuerySQL)
+	if engineRecord.CalledWith[0].QuerySQL != queryContent {
+		t.Errorf("Parse must receive full query.sql; got %q", engineRecord.CalledWith[0].QuerySQL)
+	}
+	if codegenReq == nil || len(codegenReq.Queries) != 2 {
+		t.Errorf("codegen must receive 2 queries (from 2 statements); got %d", len(codegenReq.Queries))
 	}
 }
 
@@ -364,7 +371,9 @@ func TestPluginPipeline_DatabaseOnly_ReceivesNoSchema(t *testing.T) {
 	pluginParse := func(schemaSQL, querySQL string) (*pb.ParseResponse, error) {
 		engineRecord.Calls++
 		engineRecord.CalledWith = append(engineRecord.CalledWith, struct{ SchemaSQL, QuerySQL string }{schemaSQL, querySQL})
-		return &pb.ParseResponse{Sql: querySQL, Parameters: nil, Columns: engineRecord.ReturnedCols}, nil
+		return &pb.ParseResponse{
+			Statements: []*pb.Statement{{Name: "GetOne", Cmd: pb.Cmd_CMD_ONE, Sql: "SELECT 1", Parameters: nil, Columns: engineRecord.ReturnedCols}},
+		}, nil
 	}
 	conf, err := config.ParseConfig(strings.NewReader(testPluginPipelineConfigDatabaseOnly))
 	if err != nil {