config file schema & validation

[boneskull]

• fix: quiet mode and TS loading
• fix(core): implement --iterations flag to control benchmark iteration count
• feat(core): add --limit-by flag with smart defaults for benchmark limiting
• docs(readme): update readme
• chore: fix invalid config in examples, etc.
• feat: config file schema and validation

[Copilot]

Pull Request Overview

This PR implements config file schema validation with Zod and adds the --iterations and --limit-by flags to control benchmark execution behavior. It fixes issues with quiet mode when using the JSON reporter, updates TypeScript file loading to use cosmiconfig-typescript-loader, and corrects invalid configuration properties across examples and test files.

Key changes:

• Added Zod schema validation for configuration files with JSON Schema generation
• Implemented --limit-by flag with smart defaults (iterations/time/any/all modes)
• Fixed quiet mode to allow JSON output to stdout while suppressing other console output
• Replaced tsx with cosmiconfig-typescript-loader for TypeScript config/benchmark loading

Reviewed Changes

Copilot reviewed 23 out of 25 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
test/integration/symlink-install.test.ts	Added --reporters json flag to test command for explicit reporter configuration
test/integration/reporters.test.ts	Simplified JSON reporter test assertions to check key structure paths
test/integration/quiet-mode.test.ts	Updated quiet mode tests to expect JSON output on stdout (not empty)
test/integration/limit-by.test.ts	New comprehensive test suite for --limit-by flag behavior and smart defaults
test/integration/iterations.test.ts	New test suite verifying --iterations flag controls sample count correctly
test/integration/configuration.test.ts	Renamed output config property to outputDir throughout tests
src/types/core.ts	Added limitBy field to ModestBenchConfig interface with documentation
src/types/cli.ts	Added limitBy field to RunCommandArgs interface
src/reporters/json.ts	Exported JsonOutput interface for test type safety
src/core/loader.ts	Switched from tsx to cosmiconfig-typescript-loader for TypeScript file loading
src/core/engine.ts	Implemented limitBy logic to control tinybench time/iteration parameters
src/config/schema.ts	New file defining Zod schemas for configuration validation
src/config/manager.ts	Integrated Zod validation, smart defaults for limitBy, and cache-busting for JS modules
src/cli/index.ts	Added --limit-by option and renamed output to outputDir
src/cli/commands/run.ts	Fixed quiet mode logic to allow JSON stdout output
src/cli/commands/history.ts	Renamed output option to outputDir
scripts/generate-schema.ts	New script to generate JSON Schema from Zod schemas
package.json	Added build:schema step, moved tsx to devDependencies, added zod and cosmiconfig-typescript-loader
modestbench.config.json	Removed invalid concurrent property
examples/modestbench.config.json	Removed invalid concurrent and historyLimit properties
examples/README.md	Updated examples to use warmup as number and removed invalid config properties
cspell.json	Added "limitby" to dictionary
README.md	Extensive documentation updates for new features and usage patterns

[Copilot]

The regex pattern [[{] uses an unnecessary nested bracket. This should be [\\[{] or [\[{] to properly match either an opening square bracket or opening curly brace.

Suggested change

	/benchmarks\s*:\s*[[{]/.test(content) ||
	/benchmarks\s*:\s*[\[{]/.test(content) ||

[Copilot]

Pull Request Overview

Copilot reviewed 16 out of 18 changed files in this pull request and generated no new comments.

[Copilot]

Pull Request Overview

Copilot reviewed 16 out of 18 changed files in this pull request and generated no new comments.

[Copilot]

Pull Request Overview

Copilot reviewed 16 out of 18 changed files in this pull request and generated no new comments.