[FEATURE] Reduce boilerplate for adding new PPL commands and functions

[dai-chen]

Is your feature request related to a problem?

Adding new PPL commands or functions requires extensive boilerplate code across multiple layers of the codebase, making it error-prone and time-consuming. Developers must manually:

1. Update ANTLR grammar files (lexer and parser)
2. Create AST node classes and implement AST/RelNode visitor pattern methods
3. Register functions in multiple locations (PPLBuiltinOperators, PPLFuncImpTable, BuiltinFunctionName)
4. Create unit tests that validate the translation pipeline and Spark SQL generated
5. Create multiple integration test classes (IT, Yaml IT, ExplainIT, AnonymousIT, CrossClusterIT)
6. Remember engine-specific setup (e.g., enableCalcite() for V3-only functions)
7. Update documentation following the latest doc structure

This repetitive process leads to:

• Human errors: Easy to miss required steps or files - some are subtle even if reference similar PRs.
• Inconsistency: Different developers may structure code differently.
• Slow onboarding: New contributors face a steep learning curve.
• Maintenance burden: Changes to architecture require updating many files.

What solution would you like?

Below are several ideas that address the problem from different levels (tooling, specs, and architecture):

1. Interactive scaffolding tool: Add a Gradle task that generates the required files and code insertions from a few prompts, with a strict “expected files changed” checklist to prevent missed steps (human or AI) and copy-paste drift.

2. Spec-driven code generation: Define a YAML/Markdown spec as the single source of truth for commands/functions (metadata, signatures, engine support, tests/docs) and generate the repetitive glue at build time—AI-friendly by design.

3. Better abstractions: Introduce a small DSL for AST → RelNode translation and shared test base classes/mixins to standardize setup. This reduces long-term maintenance and makes both scaffolding and codegen simpler and safer.

What alternatives have you considered?

• Better documentation only: Document the process better, but doesn't prevent human errors
• Copy-paste from examples: Current approach, leads to inconsistencies and forgotten steps
• Code review checklists: Helpful but reactive - catches errors after they're made

Do you have any additional context?

• This issue was motivated by where backport PR failure below due to missing enableCalcite(). However, more broadly, it highlights a recurring friction point in PPL development: adding new PPL features requires too much scattered, easy-to-miss boilerplate, and we should improve the end-to-end developer workflow to make contributions safer and more consistent.
  • [Backport 2.19-dev] issue #4514 tonumber function as part of roadmap #4287  #4947
  • [Backport 2.19-dev] Feature addtotals and addcoltotals #4959
• Success metrics may include:
  • Time to add new PPL command/function reduced by X%+
  • New contributors can onboard and ship first PR within X days
  • Consistent code structure across all commands/functions

[Swiddis]

I've been thinking about this too. A lot of the boilerplate in parsing seems avoidable.

Ideally, we should be defining the grammar independently of individual commands, and have a shared system for registering new command syntaxes that bundles all of the core validation. This simplifies the AST parsing logic lets us centralize all of our syntax validation, and we can then focus on just the actual logic of the commands. Most languages don't register every standard library function directly in their lexer, they leave this to a separate resolution/binding step.

For example, even if you remember to update all the right places, many commands define grammars with fixed field order (such as the current topCommand). Other commands (e.g. spathCommand) accept in arbitrary order. It's also error-prone because we keep reserving keywords and then breaking keywordsCanBeId

Instead, consider the scenario where the grammar is just:

command
  : commandName namedCommandParameter* commandArguments*

namedCommandParmeter
  : keyword parameterValue
// ...

We write a parser wrapper that can fetch arbitrary specific parameters. Then registering all of the commands can be done with a system resembling:

commands.register(
  Command.named("top")
         .engine(CALCITE)
         .namedArgument("number", INTEGER_LITERAL)
         .namedArgument("countfield", FIELD_NAME)
         // ...
         .fieldList() // <-- these command-specific grammar points get implemented as part of the main parser layer.
                      // Most are reusable. For fieldList, we can reuse this between `fields`, `top`, `rare`...
         .by() // <-- Reusable between top/rare and stats, probably more
         .invokes(TopComand::evaluate)
);

I think we can do this approach fairly easily for most commands. There's probably a couple with very weird grammar, we can handle those individually as they come up.

At a command level, this isn't much longer than specifying the syntax in the grammar. But then we can frontload all of the parsing/validation/anonymization to this layer, and commands then only need to write the implementation. This would at least work for the commands with the typical "parameters, arguments, maybe custom extra stuff (like by)" syntax.

From the description, this type of approach can theoretically cover points 1, 2, 3, 4, most of 5, and 6. I don't think 7 can reasonably be automated since each command has its own functionality. It also covers a few unmentioned points:

8. Implementing anonymizer logic
9. Making sure the grammar is as flexible as it can be (see the above top vs spath field ordering issue)
10. Making sure new command names/parameter names aren't accidentally blocked for use as identifiers
11. (Future) Providing error messages for syntax misuse that are better than the current antlr unexpected token error
12. (Future) emitting metrics for individual command usage

[Swiddis]

I think we can do this approach fairly easily for most commands. There's probably a couple with very weird grammar, we can handle those individually as they come up.

I did a quick scan through the existing commands and broke them up by how easy it'd be to write a generic parser for them. Tier 1 being "just collect the arguments in a Map" and tier 4 being impractical to handle without special-case grammar.

Tier 1: Perfect Fit (10 commands)

Commands that already match commandName namedArgs* fields?:

Command	Syntax	Notes
reverse	REVERSE	No parameters at all
head	HEAD number? (FROM from)?	Two optional positional args
expand	EXPAND field (AS alias)?	Single field + optional alias
flatten	FLATTEN field (AS aliases)?	Single field + optional aliases
dedup	DEDUP number? fields (KEEPEMPTY=bool)? (CONSECUTIVE=bool)?	Already uses ArgumentFactory
fillnull	Multiple variants but could normalize to named args	Could standardize
kmeans	KMEANS (CENTROIDS=n)? (ITERATIONS=n)? (DISTANCE_TYPE=s)?	Pure named args
ad	AD (param=value)*	Pure named args
ml	ML (param=value)*	Generic named args
addtotals	ADDTOTALS fields? options*	Named options

Tier 2: Good Fit with Extensions (15 commands)

Commands that need special clause support (BY, OVER, AS, SORT):

Command	Syntax	Notes
rare/top	TOP number? options* fields (BY group)?	Add .byClause() support
stats	STATS args* aggTerms* (BY group | span)?	Add aggregation term support
eventstats	EVENTSTATS args* aggTerms* (BY group)?	Same as stats
streamstats	STREAMSTATS args* aggTerms* (BY group)?	Same as stats
sort	SORT count? sortFields	Need sortField type (prefix/suffix/auto)
fields/table	FIELDS (+|-)? fieldList	Simple prefix operator
rename	RENAME field AS field (COMMA field AS field)*	Pair list pattern
replace	REPLACE pattern WITH replacement IN fields	Special pattern
bin	BIN field (SPAN=x)? options* (AS alias)?	Named options + alias
grok	GROK sourceField pattern	Two positional args
parse	PARSE sourceField pattern	Two positional args
spath	SPATH (INPUT=f)? (OUTPUT=f)? (PATH=p)?	Pure named args, with an optional positional arg
regex	REGEX field EQUAL pattern	Operator + pattern
rex	REX FIELD=f options* pattern options*	Named + pattern
trendline	TRENDLINE (SORT field)? clauses*	Special clause syntax

Overall doable. We need to extend the registration API with:

• .byClause() / .statsByClause()
• .aggregationTerms()
• .sortClause()
• .pairList()
• .pattern()

Tier 3: Complex but Manageable (10 commands)

Commands with complex nested structures but could be abstracted:

Command	Syntax	Notes
chart	CHART options* agg (OVER row)? (BY col)? options*	OVER/BY semantics different from stats
timechart	TIMECHART params* agg (BY field)? params*	Similar to chart
eval	EVAL field = expression (COMMA field = expression)*	Full expression parsing needed
patterns	PATTERNS field (BY group)? options* params*	Many specialized options
lookup	LOOKUP table mappings ((APPEND|REPLACE) outputs)?	Mapping pair semantics
addcoltotals	ADDCOLTOTALS fields? options*	Similar to addtotals
appendcol	APPENDCOL (OVERRIDE=bool)? [subpipeline]	Nested pipeline in brackets
append	APPEND [subpipeline]	Nested pipeline in brackets
appendpipe	APPENDPIPE [subpipeline]	Nested pipeline in brackets
where	WHERE logicalExpression	Full expression parsing

Would need:

• Expression parsing hooks (for eval, where)
• Subpipeline support (for append commands)
• Complex clause combinations (chart/timechart)

Tier 4: Major Outliers (5 commands)

Commands with highly specialized syntax that don't fit the pattern of other commands:

Command	Why It's an Outlier
search	• Full boolean expression language • Field comparisons with operators • Nested expressions with AND/OR/NOT • IN lists • Time modifiers • Can appear before FROM clause • Complex search term types
join	• Multiple join type keywords (INNER, LEFT, RIGHT, etc.) • Side aliases (LEFT=l, RIGHT=r) • Join hints with dot notation (left.hint=value) • Join criteria with ON/WHERE • Options (OVERWRITE, TYPE, MAX) • Complex syntax variations
multisearch	• Multiple nested subsearches • Each subsearch is a full search + pipeline • Syntax: MULTISEARCH ([search | cmds]+)
describe	• Special command for metadata • Different execution path
show	• Special command for system info • Different execution path

These commands are fundamentally different:

• search: Has its own expression grammar (searchExpression rule with ~10 sub-rules)
• join: Has 7+ different syntax patterns depending on join type
• multisearch: Recursively contains other commands
• describe/show: Administrative commands with different semantics

[dai-chen]

Hi @Swiddis The generic parser looks interesting! In your top command example, could you give more details what's behind it? Do we represent the new command logic in current AST->RelNode pipeline or by SQL directly? Or just give executable code in your TopCommand:evaluate?

[Swiddis]

The idea is that evaluate will receive some sort of data structure that contains all of its parameters, pre-parsed & validated, and it'll get assembled in planning. So Top might be working on something like {"showcount": true, "countfield": "count", "number": 5} and need to convert that to RelNode. The parsing boilerplate is all abstracted away, and the map structure more accurately maps how devs (or at least I) actually want to implement commands ("just add this parameter & let me write the relnode logic").

In a perfect world this would be a TopParameters-style class that's reflected/determined by the parser, but I'm not sure how feasible that is. It might need to be some sort of Map. Ideally there's a single source of truth that combines "what data the RelNode generator works with" and "what data the parser is fetching."

[krisfreedain]

Catch All Triage - 1 2