Self healing open

[cjen1-msft]

This PR is the reification of: #7003

The idea is that if a service has fully crashed, it should be able to heal itself so long as it isn't too damaged.
Specifically, the restarting nodes should gossip the knowledge they have locally to try and elect the replica with the best local state.

The result is that after the self-healing-open one replica is chosen to recover and open, while all others restart to then join it.

The protocol can be roughly surmised as:

• Start up.
• Gossip your state (claimed length of ledger) and authenticate other replicas attestation and identities.
• Once you have heard from everyone you expect to hear from, vote for the node with the longest ledger (ties broken by identity).
• If you receive votes from a majority of your expected cluster, transition_to_open and broadcast IAmOpen to the other nodes.
• If you receive IAmOpen from a trusted node, restart and join it.

This still requires the submission of ledger recovery shares, however if local sealing is available those can be used instead.

UPDATE:
To be specific on when and where things happen.

• After public ledger recovery, during create rpc it will reset the self-healing-open state and once that state is committed start several timers, one to resend gossips/votes/iamopen messages, and one to advance the failover state-machine
• All messages include an attestation that is validated on receipt, and the handlers all make a change to the KV state and then try to advance the protocol.
• Finally once the protocol is complete either:
  • The chosen replica runs transition-to-open and on the next timer tick stops the timers.
  • The others shutdown, to later be restarted by the host's orchestrator where they will try and join the network, landing on the chosen replica.

[achamayou]

I think we want this to take place over the same rpc interface we use for snapshot download, joining etc, which is probably not the primary as that tends to be exposed publicly. Perhaps that's a configuration option? Perhaps we need to name that interface.

[cjen1-msft]

I've updated this such that all self-healing-open messages pass through the same interface, which in theory should be on a private vnet etc, and is set in the config.
In general having a concept of an internal interface (one used for snapshots, joining, ledgers etc) would be useful (as we can centrally manage it), but that is probably a bigger discussion we should have in the new year.

[achamayou]

why not?

[cjen1-msft]

Afaicr symmetry can interact badly with the eventually properties.