Handle disconnects

[GDYendell]

The current model is when an AttributeIO update or a scan raises an exception, it stops and does not attempt to reconnect. The process will need to restarted once connection is re-established. This was done to prevent it from creating masses of log output.

What behaviour do we want?

• Just keep sending requests and logging errors
• Add some logic to AttributeIO to only log errors once and keep trying, but more quietly
• Stop the application and let K8s restart until it works
  • How do we decide when to stop, when just one update loop stops? When all stop?
• Something else?

Thoughts @coretl, @gilesknap?

[coretl]

Ideas after discussing in person:

• K8s schedules the pod immediatelyas soon as the resource exists (e.g. USB device present, maybe ping device)
• When FastCS starts, the transports that support it (e.g. not softioc) should present a state of "Connecting"
• FastCS connects, introspects and populates transport, presenting a state of "Connected"
• We give an environment variable "FASTCS_STOP_ON_ERROR=YES"
• If any update loop errors, then check the environment variable, if YES then stop all polling and present a state of "Error", if NO then log the error and continue polling
• If any update loop raises DisconnectedError, then stop all polling, presenting a state of "Disconnected"
• "Disconnected" or "Error" state should be propagated to all PV severity in the transport

The remaining question is what to do in "Error" or "Disconnected" state. Options are:

1. Nothing, let the user restart the process
2. Exit the application so k8s restart the process
3. Put a liveness probe in so k8s restarts the process
4. Call controller.connect again until the device comes back, then exit the process only if introspection data is different

Thoughts?

[Tom-Willemsen]

For what it's worth, for our limited use of FastCS at ISIS (not running in k8s), we'd ideally like the behaviour to broadly mirror the behaviour of a "standard" streamdevice IOC, i.e. PVs going into INVALID alarm and reconnecting when the connection is available again, without killing the whole process.

This is what we currently do in a downstream fastcs driver to reconnect after connection loss - we also catch exceptions in update loops before they reach FastCS. Currently with that approach PVs don't go into INVALID alarm so user doesn't really get feedback that connection is dead.

[GDYendell]

This looks like a nice example. Not all devices will have an explicit ping command, but they probably have a benign command that can be used instead, or we can fallback on any scan task failing meaning a disconnect (potentially with a threshold within duration check). But, there should be a standard reconnect method for drivers to implement to test the connection and restart the scan tasks. I have an initial version of this locally, but for now I am just testing with a reconnect method in the interactive shell, which works quite nicely.