Prevent zombie Kafka source on low watermark failures

[patrickwwbutler]

We saw an issue where a source failed because the kafka low watermarks were beyond the resume_upper for that source, indicating there was data compacted away that we would never get back. However, because the source emitted an error at the timestamp of the snapshot, and then exited, implicitly dropping its capabilities, the remap operator believed that all messages up until the input upper (high watermarks of the kafka partitions) had been completed, and advanced the resume_uppers to the high watermarks. This is normal procedure when emitting definite errors that indicate the source is irreparably broken.

However, at this point the kafka replication operator emitted a HealthStatusUpdate with halting = true. This meant that the healthcheck operator would then issue a suspend and restart command to the storage controller. When restarting, because the resume uppers had advanced, the source was able to continue reading from the previous frontier, and the status appeared healthy until querying the source table, when you would get the low watermark error.

Instead, we should emit a HealthStatusUpdate::stalled which will tell the healthcheck operator not to issue a restart command, and the source will stay visibly stalled rather than appear healthy.