--test-name-pattern needing to come before filenames is hostile to npm scripts

[domenic]

What is the problem this feature will solve?

It is common practice to set up npm scripts for testing. E.g.

{
  "test": "node --test tests/*.js more-tests/*.js"
}

However, this cannot be combined with --test-name-pattern. Attempting to do so, e.g.

npm test -- --test-name-pattern="my pattern"

will not work, because this gets translated to

node --test tests/*.js more-tests/*.js --test-name-pattern="my pattern"

which, I believe, ends up passing --test-name-pattern="my pattern" as an argument to these test files, instead of passing it as an argument to the test runner. The correct invocation is

node --test-name-pattern="my pattern" --test tests/*.js more-tests/*.js

but this is impossible to do via npm scripts, it seems. (See alternatives considered.)

What is the feature you are proposing to solve the problem?

I don't know what a good solution to this would be. Some possible ideas:

• Special-case command line processing such that when --test is present, node grabs the --test-name-pattern argument for itself instead of passing it to scripts?

• Introduce a new binary, e.g. node_test, which processes command-line arguments in such a way? I believe this is how most test runners behave.

• Introduce a file-based customization of the test runner, including which tests to run, so that I don't have to pass the test filenames as arguments to the test runner in a way that causes this problem?

• Improve npm scripts to support a better method of passing arguments in the middle of the script? (See below.)

What alternatives have you considered?

I investigated how to get npm scripts to substitute in arguments you pass to npm run into the script command, so that the translation becomes the correct one. This is a well-studied problem, and the following two Stack Overflow posts have the best answers, as far as I can tell:

• https://stackoverflow.com/q/11580961/3191
• https://stackoverflow.com/q/37869773/3191

None of them seem very satisfactory, unfortunately. In particular, if you want something that works cross-platform, you basically have to write a wrapper script.

As an alternative, I could continue using other test runners, which support npm scripts better.

[benjamingr]

@nodejs/test_runner @MoLow wdyt?

FWIW:

Introduce a file-based customization of the test runner, including which tests to run, so that I don't have to pass the test filenames as arguments to the test runner in a way that causes this problem?

You can do this using run, additionally you can pass parameters through the NODE_OPTIONS environment variable as a workaround.

[jacob-ebey]

I run into this all the time and if I could remove the script test:target for this use-case that would be phenomenal. Every single other test runner worth while accounts for this and has become part of my workflow that breaks in these setups.

[avivkeller]

you can pass parameters through the NODE_OPTIONS environment variable as a workaround.

--test-name-pattern and --test-skip-pattern aren't allowed in NODE_OPTIONS, would you like me to open a PR to change that?

➜  ~ node -v          
v22.1.0
➜  ~ NODE_OPTIONS="--test-name-pattern" node
node: --test-name-pattern is not allowed in NODE_OPTIONS
➜  ~ NODE_OPTIONS="--test-name-pattern=\"foo\"" node
node: --test-name-pattern= is not allowed in NODE_OPTIONS

[cjihrig]

If anyone wants to pick this up, it would involve updating the test runner's parseCommandLine() function to also consider values in process.argv. I don't think it would be difficult to implement technically, but there are a few caveats to consider:

• The current behavior of only considering process.execArgv is how all Node core CLI flags work. Since this would be introducing an inconsistency, I'm not sure how well it would be received.
• Node's C++ layer will continue to be unaware of any flags passed in process.argv. I think --test, --experimental-test-coverage, and --experimental-test-isolation are currently the only test runner flags used in the C++ layer. Another inconsistency to be aware of.
• I don't think we would want to extend this to non-test runner flags. For example, people might expect --require or --import in process.argv to work, but they wouldn't.
• I'm not sure if we would only want to do this when the CLI (--test) is being used, or any time node:test is used.

My feeling is that doing this might introduce more issues than it solves, but I wouldn't block anyone from doing it (others people might though).

[avivkeller]

FWIW:
Currently, (without any changes) there are a few ways to achieve this goal (as @benjamingr previously mentioned):

1. run() - You can use the run function to specify patterns
2. NODE_OPTIONS - As of cli: allow --test-[name/skip]-pattern in NODE_OPTIONS #53001, you can use environment variables to specify patterns.

So I'm not sure if it's worth checking a whole new set of arguments when there are other ways to achieve this.

[jacksonthall22]

I'll throw in my 2 cents—I'm only just learning about node's testing suite, but it is definitely misleading that both --test and --test-name-pattern "do the testing". Just wasted about 2 hours realizing I could not easily solve OP's same problem before landing here. Seems like --test should be responsible for doing the testing, with the next arg being the path/pattern, and then just have --test-name-pattern=.../--test-skip-pattern=.../etc. act like modifiers on --test's default behavior. I would not expect those flags to actually "execute" the tests on their own (and hence they should not require a path/pattern arg after them). That would allow for an ideal scenario where both/many modifiers could be used (not even sure if that's possible now?):

package.json

{
  "scripts": {
    "build": "node --test tests/*.js"
  }
}

and use it like this:

npm build -- --test-name-pattern=@foo  --test-skip-pattern=@bar

No idea what the technical/practical implications of this change would be though since I'm new here, just seems like a more logical interface.

[krutoo]

Same issue but run() is not suitable for me because I need to --import setup file before tests run