Skip to content

agent-friendly conversion CLIs#274

Draft
kkozik-amplify wants to merge 11 commits intorelease/8.xfrom
cli-file-globbing
Draft

agent-friendly conversion CLIs#274
kkozik-amplify wants to merge 11 commits intorelease/8.xfrom
cli-file-globbing

Conversation

@kkozik-amplify
Copy link
Copy Markdown
Collaborator

@kkozik-amplify kkozik-amplify commented Mar 11, 2026 •
edited
Loading

Summary

Make hcl2tojson and jsontohcl2 agent-friendly by applying the conventions established in hq.

Shared infrastructure (cli/helpers.py)

  • _expand_file_args() moved from hq.py — tool-side glob expansion for all three CLIs
  • _error() structured error formatter (JSON or plain text to stderr)
  • _collect_files() directory walker with configurable extensions
  • quiet parameter on all conversion helpers
  • Shared exit code constants

hcl2tojson — new features

  • Structured errors — no more Python tracebacks; clean messages to stderr
  • Exit codes — 0 (success), 1 (partial/skipped), 2 (all unparsable), 4 (I/O error)
  • Directory → stdout — hcl2tojson dir/ streams NDJSON without requiring -o
  • --ndjson — one JSON object per line with __file__ provenance for multi-file
  • --compact — single-line JSON; auto-compact when stdout is not a TTY
  • --only / --exclude — block type filtering (e.g. --only resource,module)
  • --fields — field projection (e.g. --fields cpu,memory)
  • --quiet / -q — suppress stderr progress output
  • Stdin-first defaults — no args = read stdin (like jq)
  • Glob expansion — 'modules/**/*.tf' works even when quoted

jsontohcl2 — new features

  • Structured errors + exit codes — 0/1 (JSON parse)/2 (bad structure)/4 (I/O)/5 (diff)
  • --diff ORIGINAL — unified diff preview against an existing .tf file
  • --semantic-diff ORIGINAL — structural diff that ignores formatting (alignment, commas, comments, array layout); uses diff_dicts() from hcl2/query/diff.py
  • --diff-json — machine-readable JSON output for both diff modes
  • --dry-run — convert and print without writing files
  • --fragment — generate attribute snippets from JSON fragments
  • --quiet / -q, stdin-first defaults, glob expansion

Bug fixes

  • --fields list leak — leaf lists (e.g. regions, tags) whose keys were not in the field set leaked through; now only recurses into structural containers (dicts and lists-of-dicts)
  • Non-dict JSON rejection — jsontohcl2 --dry-run and normal mode silently accepted arrays/scalars with garbled output; load_python() now raises TypeError (exit 2)
  • --fragment string format — documented the inner-quote convention prominently in CLI help, epilog, and docs

Convention alignment

  • All three CLIs share glob expansion, structured errors, --quiet, stdin-first
  • Help text includes examples epilog and exit code documentation
  • CLAUDE.md and docs/01_getting_started.md updated

Caused by #49

Test plan

  • 1330 tests pass (196 CLI tests including 13 new integration tests)
  • pre-commit: black, mypy, pylint all pass
  • --fields no longer leaks leaf lists (unit + integration tests)
  • Non-dict JSON rejected with exit 2 (unit + integration tests)
  • --semantic-diff exits 0 for round-trips across all golden fixtures
  • --semantic-diff detects value changes with exit 5
  • --semantic-diff ignores formatting-only differences
  • --diff-json produces valid structured JSON output
  • Manual: hcl2tojson dir/ streams NDJSON to stdout with __file__
  • Manual: hcl2tojson --ndjson --only resource 'modules/**/*.tf'
  • Manual: echo 'x = 1' | hcl2tojson works without -
  • Manual: jsontohcl2 --semantic-diff original.tf modified.json shows only value changes

🤖 Generated with Claude Code

@kkozik-amplify @claude
CLI: accept multiple files and replace positional OUT_PATH with -o fl…
e768460 
…ag (#49)

Both hcl2tojson and jsontohcl2 now accept multiple positional PATH
arguments. Output is specified via -o/--output instead of a positional
arg. Shell glob expansion (e.g. hcl2tojson *.tf -o out/) works naturally.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@kkozik-amplify kkozik-amplify marked this pull request as ready for review March 11, 2026 16:32
@kkozik-amplify kkozik-amplify requested a review from a team as a code owner March 11, 2026 16:32
kkozik-amplify and others added 2 commits March 31, 2026 17:11
@kkozik-amplify
Merge branch 'release/8.x' into cli-file-globbing
a9d0cdb
@kkozik-amplify @claude
agent-friendly conversion CLIs (#49)
9d0de0d 
Make hcl2tojson and jsontohcl2 agent-friendly by adding structured error
handling, distinct exit codes, NDJSON streaming, glob expansion, and
convention alignment with hq. Key additions: --ndjson with __file__
provenance, --compact with TTY auto-detection, --only/--exclude block
filtering, --fields projection, --diff/--dry-run/--fragment for
jsontohcl2, --quiet mode, stdin-first defaults, and shared infrastructure
in cli/helpers.py.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@kkozik-amplify kkozik-amplify changed the title CLI: accept multiple files and replace positional OUT_PATH with -o flag Phase 2: agent-friendly conversion CLIs Mar 31, 2026
@kkozik-amplify kkozik-amplify changed the title Phase 2: agent-friendly conversion CLIs agent-friendly conversion CLIs Mar 31, 2026
kkozik-amplify and others added 5 commits April 1, 2026 12:58
@kkozik-amplify @claude
fix review issues: --ndjson stdin, --fragment semantics, --diff exit …
ca36822 
…code, path collisions

- Fix --ndjson crash when reading from stdin (open("-") → sys.stdin)
- Make --fragment strip __is_block__ markers so block-structured JSON
  becomes flat attributes, differentiating it from --dry-run
- Use dedicated EXIT_DIFF=5 for --diff mode instead of overloading
  EXIT_PARTIAL=1 (JSON parse error)
- Preserve relative path structure in _convert_multiple_files to avoid
  basename collisions (dir1/main.tf + dir2/main.tf → out/dir1/ + out/dir2/)
- Fix _project_fields list filter to preserve falsy scalars (0, False, "")
- Fix --compact help text (was incorrectly described as alias for --json-indent 0)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@kkozik-amplify @claude
error on multi-file/directory stdout without --ndjson or -o
d4cef28 
Concatenated pretty-printed JSON documents are neither valid JSON nor
valid NDJSON. Instead of silently producing unparseable output, require
the user to explicitly choose --ndjson (streaming) or -o <dir> (files).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@kkozik-amplify @claude
fix CLI review findings: error handling, exit codes, and robustness
b0b85ec 
Warnings (bugs):
- jsontohcl2 dir/ without -o now errors instead of crashing with RuntimeError
- _convert_multiple_files normalizes paths to absolute before commonpath
- Non-NDJSON multi-file -s correctly exits 1 on partial failures
- _convert_directory uses os.makedirs for nested output paths

Info (improvements):
- --diff/--dry-run/--fragment are now mutually exclusive in jsontohcl2
- NDJSON mode emits structured JSON errors to stderr
- stdout output is buffered when skip=True to prevent partial writes

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@kkozik-amplify @claude
fix PR review findings: dead except clause, fragment validation, test…
849f471 
… gaps

- Replace dead `except JSON_SKIPPABLE` with explicit `except UnicodeDecodeError`
  since JSONDecodeError is already caught above (json_to_hcl.py)
- Validate --fragment input is a dict, not an array or scalar
- Warn when --ndjson --json-indent are combined (indent is ignored)
- Add 8 tests covering: fragment __is_block__ stripping, fragment non-dict
  rejection, structure error exit code 2, --diff edge cases (nonexistent
  original, stdin), --ndjson stdin, --ndjson all-fail-with-skip, --exclude
  with multiple comma-separated types

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@kkozik-amplify @claude
fix review findings: stale docs, stdin skip, commonpath edge case
a77c908 
- Fix CLAUDE.md and docs claiming `hcl2tojson dir/` works without --ndjson
- Route stdin through _convert_single_file so --skip flag is honored
- Guard _convert_multiple_files commonpath against identical file paths
- Replace os.mkdir with os.makedirs in _convert_directory
- Remove redundant sorted() in _collect_files
- Reject stdin (-) in multi-file mode with clear error message
- Add 7 tests covering stdin+skip, multi-file+stdin, duplicate paths

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
kkozik-amplify and others added 2 commits April 1, 2026 16:07
@kkozik-amplify @claude
fix review findings: exit codes, stdin -o, compact separators, SIGPIPE
30dba61 
- Single-file skip now exits 1 (EXIT_PARTIAL) instead of silent 0
- stdin honors -o output path instead of silently ignoring it
- _stream_ndjson tracks IO vs parse errors for correct all-fail exit code
- NDJSON skips empty records after --only/--exclude filtering
- --compact uses separators=(",",":") for truly compact JSON; NDJSON too
- Add _install_sigpipe_handler() for clean pipe-to-head behavior
- Remove dead _convert_stdin (replaced by _convert_single_stream)
- Merge redundant stdin/isfile branches in both CLIs

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@kkozik-amplify @claude
add subprocess-based CLI integration tests for hcl2tojson and jsontohcl2
e1b8819 
50 tests across 12 classes verify end-to-end behavior that unit tests
cannot: real exit codes, stdout/stderr separation, stdin piping, pipe
composition (hcl2tojson | jsontohcl2 round-trips against golden fixtures),
NDJSON structured errors, basename collision handling, stdout buffering
with skip, stdin-to-output-file, compact separator correctness, NDJSON
IO vs parse error exit codes, and TTY vs pipe default behavior.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@kkozik-amplify kkozik-amplify marked this pull request as draft April 1, 2026 16:59
@kkozik-amplify @claude
fix --fields list leak, reject non-dict JSON input, add --semantic-di…
ef933fe 
…ff mode

--fields incorrectly preserved leaf lists (e.g. regions, tags) whose keys
were not in the field set, defeating the purpose of token-reduction for
agent pipelines. Now only recurses into structural containers (dicts and
lists-of-dicts for block hierarchy).

jsontohcl2 --dry-run and normal mode silently accepted non-dict JSON
(arrays, scalars) producing garbled output with exit 0. load_python()
now raises TypeError, caught by the existing handler to exit 2.

New --semantic-diff ORIGINAL mode compares parsed HCL dict against JSON
dict using diff_dicts(), completely ignoring formatting differences
(alignment, trailing commas, comments, array layout). --diff-json flag
outputs structured JSON for machine consumption.

Documented --fragment inner-quote string convention in CLI help, epilog,
and docs to prevent the common first-use mistake of passing plain JSON
strings.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@rddimon rddimon rddimon approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@kkozik-amplify @rddimon