Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
79 changes: 79 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,10 @@ See the [examples folder](./examples/) for more common usage examples.
- [`new Suite([options])`](#new-suiteoptions)
- [`suite.add(name[, options], fn)`](#suiteaddname-options-fn)
- [`suite.run()`](#suiterun)
- [Dead Code Elimination Detection](#dead-code-elimination-detection)
- [How It Works](#how-it-works)
- [Configuration](#configuration)
- [When DCE Warnings Appear](#when-dce-warnings-appear)
- [Plugins](#plugins)
- [Plugin Methods](#plugin-methods)
- [Example Plugin](#example-plugin)
Expand Down Expand Up @@ -131,7 +135,10 @@ A `Suite` manages and executes benchmark functions. It provides two methods: `ad
* `useWorkers` {boolean} Whether to run benchmarks in worker threads. **Default:** `false`.
* `plugins` {Array} Array of plugin instances to use.
* `repeatSuite` {number} Number of times to repeat each benchmark. Automatically set to `30` when `ttest: true`. **Default:** `1`.
* `plugins` {Array} Array of plugin instances to use. **Default:** `[V8NeverOptimizePlugin]`.
* `minSamples` {number} Minimum number of samples per round for all benchmarks in the suite. Can be overridden per benchmark. **Default:** `10` samples.
* `detectDeadCodeElimination` {boolean} Enable dead code elimination detection. When enabled, default plugins are disabled to allow V8 optimizations. **Default:** `false`.
* `dceThreshold` {number} Threshold multiplier for DCE detection. Benchmarks faster than baseline × threshold will trigger warnings. **Default:** `10`.

If no `reporter` is provided, results are printed to the console.

Expand Down Expand Up @@ -179,6 +186,78 @@ Using delete property x 5,853,505 ops/sec (10 runs sampled) min..max=(169ns ...

Runs all added benchmarks and returns the results.

## Dead Code Elimination Detection

**bench-node** includes optional detection of dead code elimination (DCE) to help identify benchmarks that may be producing inaccurate results. When the JIT compiler optimizes away your benchmark code, it can run nearly as fast as an empty function, leading to misleading performance measurements.

**Important:** DCE detection is **opt-in**. When enabled, the `V8NeverOptimizePlugin` is automatically disabled to allow V8 optimizations to occur naturally. This helps catch benchmarks that would be optimized away in real-world scenarios.

### How It Works

When enabled, bench-node measures a baseline (empty function) performance before running your benchmarks. After each benchmark completes, it compares the timing against this baseline. If a benchmark runs suspiciously fast (less than 10× slower than the baseline by default), a warning is emitted.

### Example Warning Output

```
⚠️ Dead Code Elimination Warnings:
The following benchmarks may have been optimized away by the JIT compiler:

• array creation
Benchmark: 3.98ns/iter
Baseline: 0.77ns/iter
Ratio: 5.18x of baseline
Suggestion: Ensure the result is used or assign to a variable

ℹ️ These benchmarks are running nearly as fast as an empty function,
which suggests the JIT may have eliminated the actual work.
```

### Configuration

```js
const { Suite, V8NeverOptimizePlugin } = require('bench-node');

// Enable DCE detection (disables V8NeverOptimizePlugin automatically)
const suite = new Suite({
detectDeadCodeElimination: true
});

// Enable DCE detection with custom threshold (default is 10x)
const strictSuite = new Suite({
detectDeadCodeElimination: true,
dceThreshold: 20 // Only warn if < 20x slower than baseline
});

// Use both DCE detection AND prevent optimization
// (helpful for educational purposes to see warnings even when using %NeverOptimizeFunction)
const educationalSuite = new Suite({
plugins: [new V8NeverOptimizePlugin()],
detectDeadCodeElimination: true
});
```

### When DCE Warnings Appear

Common scenarios that trigger warnings:

```js
// ❌ Result not used - will be optimized away
suite.add('computation', () => {
const result = Math.sqrt(144);
// result is never used
});

// ✅ Result is used - less likely to be optimized
suite.add('computation', () => {
const result = Math.sqrt(144);
if (result !== 12) throw new Error('Unexpected');
});
```

**Note:** DCE detection only works in `'ops'` benchmark mode and when not using worker threads. It is automatically disabled for `'time'` mode and worker-based benchmarks.

See [examples/dce-detection/](./examples/dce-detection/) for more examples.

## Plugins

Plugins extend the functionality of the benchmark module.
Expand Down
46 changes: 46 additions & 0 deletions examples/dce-detection/example.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
const { Suite } = require("../../lib/index");

// Enable DCE detection to catch benchmarks that may be optimized away
const suite = new Suite({
detectDeadCodeElimination: true,
});

// Example 1: Likely to trigger DCE warning - result not used
suite.add("simple addition (likely DCE)", () => {
const result = 1 + 1;
// result is never used - JIT might optimize this away
});

// Example 2: Result is used - should not trigger warning
suite.add("simple addition (used)", () => {
const result = 1 + 1;
if (result !== 2) throw new Error("Unexpected result");
});

// Example 3: Array creation without use - likely DCE
suite.add("array creation (likely DCE)", () => {
const arr = new Array(100);
// arr is never accessed - might be optimized away
});

// Example 4: Array creation with access - should not trigger warning
suite.add("array creation (accessed)", () => {
const arr = new Array(100);
arr[0] = 1; // Using the array
});

// Example 5: Object creation without use - likely DCE
suite.add("object creation (likely DCE)", () => {
const obj = { x: 1, y: 2, z: 3 };
// obj is never accessed
});

// Example 6: More realistic computation
suite.add("string operations", () => {
const str1 = "hello";
const str2 = "world";
const result = str1 + " " + str2;
if (!result.includes("hello")) throw new Error("Missing hello");
});

suite.run();
17 changes: 17 additions & 0 deletions examples/dce-detection/with-dce-disabled.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
const { Suite } = require("../../lib/index");

// Default behavior - DCE detection is disabled, V8NeverOptimizePlugin is used
// You don't need to set detectDeadCodeElimination: false explicitly
const suite = new Suite();

// These benchmarks will run with V8NeverOptimizePlugin, so they'll be slower
// but more deterministic and won't be optimized away
suite.add("simple addition", () => {
const result = 1 + 1;
});

suite.add("array creation", () => {
const arr = new Array(100);
});

suite.run();
42 changes: 42 additions & 0 deletions examples/dce-detection/without-never-optimize.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
const { Suite } = require("../../lib/index");

// Enable DCE detection - this automatically disables V8NeverOptimizePlugin
// In this mode, V8 optimizations occur naturally and DCE warnings help identify issues
const suite = new Suite({
detectDeadCodeElimination: true,
});

// Example 1: Likely to trigger DCE warning - result not used
suite.add("simple addition (likely DCE)", () => {
const result = 1 + 1;
// result is never used - JIT will optimize this away
});

// Example 2: Result is used - should not trigger warning
suite.add("simple addition (used)", () => {
const result = 1 + 1;
if (result !== 2) throw new Error("Unexpected result");
});

// Example 3: Array creation without use - likely DCE
suite.add("array creation (likely DCE)", () => {
const arr = new Array(100);
// arr is never accessed - will be optimized away
});

// Example 4: Array creation with access - should not trigger warning
suite.add("array creation (accessed)", () => {
const arr = new Array(100);
arr[0] = 1; // Using the array
});

// Example 5: More realistic computation that takes time
suite.add("actual work", () => {
let sum = 0;
for (let i = 0; i < 100; i++) {
sum += Math.sqrt(i);
}
if (sum < 0) throw new Error("Impossible");
});

suite.run();
27 changes: 27 additions & 0 deletions index.d.ts
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,8 @@ export declare namespace BenchNode {
repeatSuite?: number; // Number of times to repeat each benchmark (default: 1, or 30 when ttest is enabled)
ttest?: boolean; // Enable t-test mode for statistical significance (auto-sets repeatSuite=30)
reporterOptions?: ReporterOptions;
detectDeadCodeElimination?: boolean; // Enable DCE detection, default: false
dceThreshold?: number; // DCE detection threshold multiplier, default: 10
}

interface BenchmarkOptions {
Expand Down Expand Up @@ -142,6 +144,29 @@ export declare namespace BenchNode {
getResult(benchmarkName: string): PluginResult;
toString(): string;
}

class DeadCodeEliminationDetectionPlugin implements Plugin {
constructor(options?: { threshold?: number });
isSupported(): boolean;
setBaseline(timePerOp: number): void;
onCompleteBenchmark(
result: OnCompleteBenchmarkResult,
bench: Benchmark,
): void;
getWarning(
benchmarkName: string,
): { timePerOp: number; baselineTime: number; ratio: number } | undefined;
getAllWarnings(): Array<{
name: string;
timePerOp: number;
baselineTime: number;
ratio: number;
}>;
hasWarning(benchmarkName: string): boolean;
emitWarnings(): void;
toString(): string;
reset(): void;
}
}

export declare const textReport: BenchNode.ReporterFunction;
Expand Down Expand Up @@ -215,3 +240,5 @@ export declare function compareBenchmarks(
sample2: number[],
alpha?: number,
): TTest.CompareBenchmarksResult;

export declare class DeadCodeEliminationDetectionPlugin extends BenchNode.DeadCodeEliminationDetectionPlugin {}
73 changes: 72 additions & 1 deletion lib/index.js
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ const {
V8GetOptimizationStatus,
V8OptimizeOnNextCallPlugin,
MemoryPlugin,
DeadCodeEliminationDetectionPlugin,
} = require("./plugins");
const {
validateFunction,
Expand Down Expand Up @@ -122,6 +123,7 @@ class Suite {
#minSamples;
#repeatSuite;
#ttest;
#dceDetector;

constructor(options = {}) {
this.#benchmarks = [];
Expand All @@ -140,11 +142,27 @@ class Suite {

this.#useWorkers = options.useWorkers || false;

// DCE detection is opt-in to avoid breaking changes
const dceEnabled = options.detectDeadCodeElimination === true;
if (dceEnabled) {
this.#dceDetector = new DeadCodeEliminationDetectionPlugin(
options.dceThreshold ? { threshold: options.dceThreshold } : {},
);
}

// Plugin setup: If DCE detection is enabled, default to no plugins (allow optimization)
// Otherwise, use V8NeverOptimizePlugin as the default
if (options?.plugins) {
validateArray(options.plugins, "plugin");
validatePlugins(options.plugins);
this.#plugins = options.plugins;
} else if (dceEnabled) {
// DCE detection requires optimization to be enabled, so no default plugins
this.#plugins = [];
} else {
// Default behavior - use V8NeverOptimizePlugin
this.#plugins = [new V8NeverOptimizePlugin()];
}
this.#plugins = options?.plugins || [new V8NeverOptimizePlugin()];

this.#benchmarkMode = options.benchmarkMode || "ops";
validateBenchmarkMode(this.#benchmarkMode, "options.benchmarkMode");
Expand Down Expand Up @@ -231,6 +249,15 @@ class Suite {
throwIfNoNativesSyntax();
const results = new Array(this.#benchmarks.length);

// Measure baseline for DCE detection (only in ops mode, not in worker mode)
if (
this.#dceDetector &&
!this.#useWorkers &&
this.#benchmarkMode === "ops"
) {
await this.#measureBaseline();
}

// It doesn't make sense to warmup a fresh new instance of Worker.
// TODO: support warmup directly in the Worker.
if (!this.#useWorkers) {
Expand All @@ -250,6 +277,15 @@ class Suite {

for (let i = 0; i < this.#benchmarks.length; ++i) {
const benchmark = this.#benchmarks[i];

// Add DCE detector to benchmark plugins if enabled
if (this.#dceDetector && this.#benchmarkMode === "ops") {
const originalPlugins = benchmark.plugins;
benchmark.plugins = [...benchmark.plugins, this.#dceDetector];
// Regenerate function string with new plugins
benchmark.fnStr = createFnString(benchmark);
}

// Warmup is calculated to reduce noise/bias on the results
const initialIterations = await getInitialIterations(benchmark);
debugBench(
Expand Down Expand Up @@ -280,9 +316,43 @@ class Suite {
this.#reporter(results, this.#reporterOptions);
}

// Emit DCE warnings after reporting
if (this.#dceDetector) {
this.#dceDetector.emitWarnings();
}

return results;
}

async #measureBaseline() {
debugBench("Measuring baseline for DCE detection...");

// Create a minimal baseline benchmark (empty function)
const baselineBench = new Benchmark(
"__baseline__",
() => {},
0.01, // minTime
0.05, // maxTime
this.#plugins,
1, // repeatSuite
10, // minSamples
);

const initialIterations = await getInitialIterations(baselineBench);
const result = await runBenchmark(
baselineBench,
initialIterations,
"ops",
1,
10,
);

const baselineTimePerOp = (1 / result.opsSec) * 1e9; // Convert to ns
debugBench(`DCE baseline: ${timer.format(baselineTimePerOp)}/iter`);

this.#dceDetector.setBaseline(baselineTimePerOp);
}

async runWorkerBenchmark(benchmark, initialIterations) {
return new Promise((resolve, reject) => {
const workerPath = path.resolve(__dirname, "./worker-runner.js");
Expand Down Expand Up @@ -319,6 +389,7 @@ module.exports = {
V8GetOptimizationStatus,
V8OptimizeOnNextCallPlugin,
MemoryPlugin,
DeadCodeEliminationDetectionPlugin,
chartReport,
textReport,
prettyReport,
Expand Down
4 changes: 4 additions & 0 deletions lib/plugins.js
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,9 @@ const { V8NeverOptimizePlugin } = require("./plugins/v8-never-opt");

const { V8GetOptimizationStatus } = require("./plugins/v8-print-status");
const { MemoryPlugin } = require("./plugins/memory");
const {
DeadCodeEliminationDetectionPlugin,
} = require("./plugins/dce-detection");

const { validateFunction, validateArray } = require("./validators");

Expand Down Expand Up @@ -48,5 +51,6 @@ module.exports = {
V8NeverOptimizePlugin,
V8GetOptimizationStatus,
V8OptimizeOnNextCallPlugin,
DeadCodeEliminationDetectionPlugin,
validatePlugins,
};
Loading
Loading