Possible failure in automatic idempotency cleaner test AutomaticIdempotency.toml

[dlambrig]

When a transaction uses automatic idempotency, the client assigns a unique idempotency ID. These are stored in special key space. They can be deleted once they are no longer needed to support the retry semantics:

• A cleaner process runs asynchronously to eventually delete old idempotency ID entries.
• When automatic idempotency is enabled, the client will delete it after it receives the commit acknowledgement.

tests/fast/AutomaticIdempotency.toml puts load on the automatic idempotency machinery under fault injection, and verifies that idempotency records are preserved long enough to support retries, and then eventually cleaned safely.

It has been noted that the test could fail if deleting an idempotency id through the "automatic idempotency" codepath triggers the "cleaner cleaned too far" assertion.

The problem is rare. An attempted repro was done by forcing the test to use automatic idempotency (by default it only enables it 10% of the time), and running the test in Joshua 500K times, this did not show up.