feat(dreyfus): change how indexer processes are linked to Clouseau

[pgj]

Dreyfus and Clouseau maintain a shared LRU cache of actively managed indexes, which may fall apart if the rate of eviction is getting too high.

Such situations may happen when there are too many indexes to manage in a relatively short period of time so that Clouseau is forced to switch between them very rapidly and therefore it becomes a subject of thrashing. Thrashing may lead to process termination messages of indexers being closed (by the unexpected eviction) not necessarily processed in time and the subsequent search requests on the Dreyfus side will start failing via observing noproc errors due to the backing indexer is gone. This immediately translates to an HTTP 500 errors, which is visible to the user.

Maintaining a shared state joined by Erlang process identifiers as keys is an inherently fragile idea due to their volatile nature when one of the sides is emulated in Java. That is why we shall find a more stable key, the path of the index file which is less prone to change.

Indexes would be still managed by dedicated indexers processes as before, but the communication to them would happen solely through a third-party process on the Clouseau side. This process would then keep track of the states and availability of each indexer and handle the message forwarding for them properly.

The change is implemented in a backward compatible way by putting the new-style communication method behind a toggle. This would make it possible for Dreyfus to work with Clouseau instances that have not been upgraded to support this protocol.

Testing recommendations

The existing set of tests should be able to cover this change. More specifically, if one runs the standard Search tests with Clouseau configuration, everything should just keep working as before.

./configure --dev --with-clouseau
make eunit apps=dreyfus
make elixir-search
make mango-test

TODO: Talk about reconfiguring Dreyfus to start utilizing the forwarding interface. Albeit the involved tests suites should be about the same. There shall be no semantic differences.

Related Issues or Pull Requests

This change would be served best with cloudant-labs/clouseau#149.

Checklist

• This is my own work, I did not use AI, LLM's or similar technology (but vanilla Emacs)
• Code is written and works correctly
• Changes are covered by tests
• Any new configurable parameters are documented in rel/overlay/etc/default.ini