Evals TUI tree traversal#2100
Open
miguelg719 wants to merge 1 commit intomainfrom
Open
Conversation
|
miguelg719
force-pushed
the
miguelgonzalez/stg-1880-evals-nail-tui-onboarding-experience-and-command-discovery
branch
from
bacbe10 to
fef0768
Compare
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
2 issues found across 7 files
Confidence score: 3/5
- There is a concrete regression risk in
packages/evals/tui/commandTree.ts: for inputs likeevals act, the fallback forwards tokens including the"evals"sigil to the run handler, which can misparse arguments and break expected command handling. - A second medium-severity concern is in
packages/evals/tui/commandTree.ts, where user-facing failures are thrown as genericErrorvalues and echoed with raw token interpolation, increasing the chance of confusing or unsanitized REPL error output. - Given one high-confidence parsing bug (7/10) plus one additional medium issue (6/10), this is not merge-blocking by itself but carries meaningful user-impact risk if shipped without a fix.
- Pay close attention to
packages/evals/tui/commandTree.ts- command fallback token forwarding and REPL error sanitization need verification.
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="packages/evals/tui/commandTree.ts">
<violation number="1" location="packages/evals/tui/commandTree.ts:242">
P2: Custom agent: **Exception and error message sanitization**
User-facing command errors are thrown as generic `Error` and echoed to the REPL with raw token interpolation, rather than using sanitized typed errors.</violation>
<violation number="2" location="packages/evals/tui/commandTree.ts:275">
P1: When the `evals` sigil is present and the token after it is unknown (e.g., `evals act`), the root fallback passes the full `tokens` array — including the leading `"evals"` — to the run handler. This causes `parseRunArgs` to misinterpret `"evals"` as a target/flag. Strip the sigil before delegating.</violation>
</file>
Architecture diagram
sequenceDiagram
participant Shell as Shell/argv
participant CLI as cli.ts
participant REPL as repl.ts
participant CmdTree as commandTree.ts
participant Tok as tokenize.ts
participant Handler as Command Handlers
participant Disc as discovery.ts
participant Run as run command
Note over Shell,Run: NEW: Unified command tree traversal
alt argv mode (shell)
Shell->>CLI: node evals [args...]
CLI->>CmdTree: buildCommandTree()
CmdTree-->>CLI: root CommandNode
CLI->>CLI: tokenizeArgv(args) — split on >
CLI->>CmdTree: dispatch(root, tokens, ctx)
else REPL mode (interactive)
Shell->>REPL: node evals (no args)
REPL->>REPL: printBanner()
REPL->>CmdTree: buildCommandTree()
CmdTree-->>REPL: root CommandNode
loop Each input line
REPL->>Tok: tokenize(line) — split on whitespace or >
Tok-->>REPL: tokens[]
REPL->>CmdTree: dispatch(root, tokens, ctx)
end
end
Note over CmdTree: Resolution logic (resolveCommand)
CmdTree->>CmdTree: Check first token for meta commands (..,help,exit,clear)
alt Meta command
CmdTree->>REPL: Pop context / print help / exit
else Leading "evals" sigil
CmdTree->>CmdTree: Strip sigil, resolve from root
else Relative path (REPL context)
CmdTree->>CmdTree: walkPath(contextPath) → current node
CmdTree->>CmdTree: matchPath(current, tokens) → deepest match
alt Full match
CmdTree->>CmdTree: Check for --help/help/-h on first arg
opt Help requested
CmdTree->>Handler: node.printHelp(absolutePath)
end
CmdTree->>Handler: node.handler(args, ctx)
alt REPL mode + node has children + no args
CmdTree->>REPL: setContextPath(absolutePath)
REPL->>REPL: renderPrompt(contextPath)
end
else Partial match at root (unknown token)
CmdTree->>Run: fallback: run.handler(tokens, ctx)
Run->>Disc: discoverTasks(getRuntimeTasksRoot())
Disc-->>Run: registry
Run->>Run: execute run with registry
else Partial match at depth (unknown)
CmdTree-->>REPL: throw Error (unknown subcommand)
end
end
Note over REPL: Context-aware prompt
REPL->>REPL: renderPrompt(contextPath) → "evals > config > core > "
Note over REPL: Esc key handling (CHANGED)
REPL->>REPL: On Esc press
alt abortRef.current is null (idle)
alt contextPath.length > 0
REPL->>REPL: pop context level
REPL->>REPL: setPrompt(renderPrompt(contextPath))
end
else abortRef.current exists (run in flight)
REPL->>REPL: Cooperative/aggressive abort logic
REPL->>REPL: abortRef.current.abort(reason)
end
Note over CLI: Root fallback for unknown tokens
CmdTree->>Ctx: unknown at root → call run handler with all tokens
Ctx->>CmdTree: runNode.handler(tokens, ctx)
Note over CmdTree: Registry lazy loading
CmdTree->>REPL: getRegistry()
REPL->>Disc: discoverTasks(resolvedTasksRoot)
Disc-->>REPL: TaskRegistry
REPL->>REPL: cache in ctx.setRegistry()
Note over Shell,Run: Banner changed to mention .. context navigation
Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review, or fix all with cubic.
| if (result.context.length === 0) { | ||
| const runNode = findChild(root, "run"); | ||
| if (runNode?.handler) { | ||
| await runNode.handler(tokens, ctx); |
Contributor
There was a problem hiding this comment.
P1: When the evals sigil is present and the token after it is unknown (e.g., evals act), the root fallback passes the full tokens array — including the leading "evals" — to the run handler. This causes parseRunArgs to misinterpret "evals" as a target/flag. Strip the sigil before delegating.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/evals/tui/commandTree.ts, line 275:
<comment>When the `evals` sigil is present and the token after it is unknown (e.g., `evals act`), the root fallback passes the full `tokens` array — including the leading `"evals"` — to the run handler. This causes `parseRunArgs` to misinterpret `"evals"` as a target/flag. Strip the sigil before delegating.</comment>
<file context>
@@ -0,0 +1,660 @@
+ if (result.context.length === 0) {
+ const runNode = findChild(root, "run");
+ if (runNode?.handler) {
+ await runNode.handler(tokens, ctx);
+ return;
+ }
</file context>
| @@ -0,0 +1,660 @@ | |||
| /** | |||
Contributor
There was a problem hiding this comment.
P2: Custom agent: Exception and error message sanitization
User-facing command errors are thrown as generic Error and echoed to the REPL with raw token interpolation, rather than using sanitized typed errors.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/evals/tui/commandTree.ts, line 242:
<comment>User-facing command errors are thrown as generic `Error` and echoed to the REPL with raw token interpolation, rather than using sanitized typed errors.</comment>
<file context>
@@ -0,0 +1,660 @@
+ if (result.node.handler) {
+ await result.node.handler(result.args, ctx);
+ } else if (result.args.length > 0) {
+ throw new Error(
+ `Unknown subcommand "${result.args[0]}" in ${pretty(
+ result.absolutePath,
</file context>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
why
what changed
test plan
Summary by cubic
Introduces a unified command tree for the Evals TUI/CLI with context-aware traversal, consistent parsing, and clearer help routing. Supports Linear STG-1880 by improving onboarding and command discovery with
>navigation, anevalssigil for root jumps, and contextual help.New Features
buildCommandTreepowers both REPL and argv dispatch.>works as a path separator in REPL and argv; quoted>is preserved in REPL, while argv re-splits any arg containing>...goes up one level; bareevalsjumps to root or strips a leading sigil; invoking a node with children auto-enters that context in REPL; prompt shows the current path; banner updated.--helpafter a leaf calls that node’s help;helpshows help for the current context.run(e.g.,evals act); unknown at depth errors for clarity.Refactors
cli.tsandrepl.tsnow resolve and execute viadispatchfrom the shared command tree; addedtokenizeArgvfor argv>splitting andrenderPromptfor context-aware prompts.tui/tokenize.ts; addedtui/commandTree.tswith resolution, prompt rendering, and argv tokenization.>chaining), argv sigil handling, and CLI edge cases.Written for commit fef0768. Summary will update on new commits.