FEATURE: generate which doesn't overwrite existing

[andrzejnovak]

I have a minor hassle when adding new tests and running

 --mpl-generate-path=tests/baseline

I overwrite all existing baseline images in my repo. I then have to sort through which are added, because usually the regenerated old tests are still the same visually, but are not identical hash-wise.

I would propose adding a

 --mpl-generate-missing=tests/baseline

Which would only generate new figures if the baseline is mising.

[ConorMacBride]

Instead of adding a specific option for missing baseline images, I'm thinking we could have a more general --mpl-generate=<option>[,<option>...] that accepts any number of the following options:

• images: Create or replace all baseline images
• hashes: Replace all hash library files
• nothing, just --mpl-generate: Equivalent to images,hashes (or only those that are configured)
• missing-images: Create all missing baseline images
• missing-hashes: Add all missing hashes to existing hash library files, or create the hash library file if it doesn't exist
• missing: Equivalent to missing-images,missing-hashes (or only those that are configured)
• failing-images: Replace all failing baseline images
• failing-hashes: Replace all failing hashes in existing hash library files
• failing: Equivalent to failing-images,failing-hashes (or only those that are configured)

This option would just create or replace files at the location they are configured to be for testing, which would be ideal for most users who store images and hashes in a git repo. We could keep --mpl-generate-path=<path> and --mpl-generate-hash-library=<path> to still allow generating to a different location, with those options setting a default --mpl-generate=images and --mpl-generate=hashes respectively to preserve backwards compatibility.

I'm keen to keep all the configuration options as simple as possible, so we'd need to see if there's a strong need for this among users before starting any implementation. It feels like it should be a common pain point, but I don't have the data.

[andrzejnovak]

Yeah this sounds good. Particularly the generate failing options!