Dqlite returns node join success before the join was successful in the distributed context #799
Description
When we add a new node via a Dqlite clients request, Dqlite is returning before the cluster membership state is fully committed. Our client has no way of knowing whether the cluster membership addition was successful or not. The first red arrow marks the point we currently return at. In the distributed sense our add node operation would only be completed when we have confirmation from the followers that the join was successfully committed (second red arrow).
@just-now's diagram illustrates the issue:
Related to this issue is the Dqlite cluster.yaml only being synced every second: canonical/go-dqlite#388.