First Chunk of Graceful Shutdown: TCP to Neighbors

[dnwiebe]

Graceful Shutdown is a big epic. We can get it started here with only a small chunk of its eventual capabilities, leaving most of the shutdown as ungraceful as it already is. This card is an child of the [https://github.com/MASQ-Project/Node/issues/406] spike. Check out the comments on that card for a complete picture of this process.

Tasks

• Currently, UiShutdownRequest is passed from the UiGateway to the Neighborhood, and the Neighborhood handles it and terminates the process. Modify this so that the UiGateway recognizes the UiShutdownRequest and handles the process termination.
• Modify the UiGateway so that when it receives a UiShutdownRequest, it sets an alarm (perhaps queues up a message for itself to be sent in five seconds) and sends a shutdown command as an Actor message to the StreamHandlerPool (the Actor, not the one in ProxyClient). Keep in mind that eventually the UiGateway will need to send this message to every Actor.
• Modify the StreamHandlerPool so that when it receives a shutdown command, it does the following (this is the list of tasks mentioned in [https://github.com/Graceful Shutdown Spike Node#406] minus two elements that will be added as Graceful Shutdown is filled out):
  1. Shut down all TCP streams. The easy way to do this will be to traverse StreamHandlerPool::stream_writers and send RemoveStreamMsgs to the StreamHandlerPool (or, more conveniently, just call StreamHandlerPool::handle_remove_stream_msg()) for each one. However, each time a RemoveStreamMsg is processed, the Neighborhood will be directed to remove a half-neighborship and broadcast Standard Gossip. This will mean we'll need a delay to make sure all the Gossip is sent out, and a separate Standard Gossip message for each removed stream will cause a brief Gossip storm on the network. The right way to do it is to invent a new WithdrawFromNetworkMsg that can be sent directly to the Neighborhood without routing through the Dispatcher that, upon receipt, will cause removal of all outgoing half-neighborships, and then trigger one round of Standard Gossip to all immediate neighbors showing those removals.
  2. Listen for an okay-I'm-done Actor message from the Neighborhood to signal that the Standard Gossip is on its way. (This okay-I'm-done message will be very common in the complete Graceful Shutdown: it should be designed to be generic. It should contain enough information so that the receiver can tell which Actor is claiming to be done.)
  3. Send an identical okay-I'm-done message to the UiGateway.
• Modify the UiGateway so that when it gets either the okay-I'm-done message from the StreamHandlerPool or the previously-set alarm goes off, it kills the process.