image-creator being a machine-tool to be used by other systems as well as it being task-oriented and sensible to exofailures (downloads), it should be able to report its status (where it is in the state machine), what it's doing and overall progress in a machine-readable format.
As a crucial SPOF, it should also allow post-mortem investigations.
Progress report and History, although regularly mixed for simplicity, respond to different needs and constraint:
- History is to be used by humans, so text format is preferred
- Progress Report is to be used by machines so JSON is preferred
- Textual Progress Report should also be possible
Options for it includes stdout/stderr output, output to file or socket, file or TCP socket to read or query status.
Let's keep in mind that it will mostly be used from within a container (I suppose).
At this moment, my proposal would be:
- Textual history + progress to stdout.
- JSON live progress to a specified file.
stdout is very easy to integrate and work-with. Kind of how it's always being used, so history can easily be read/fetched/archived.
Progress would be dynamic, using CR character so that it doesn't pollute history (not visible in docker logs).
Using a specified file for machine-readable progress has several advantages:
- easy to consume
- passive, synchronous. no need to query or wait for answer
- it's atomic: you just read the whole file instead of wondering where you picked the stream at or how much you should be reading from the socket.
- Previous point is super useful when parsing JSON
- can be set to a RAMfs (
/dev/shm for instance) so you're not hitting IOs
- easy to bind and reuse when using containers
It's important that long-lasting tasks report both to the history and periodically on the progress to avoid the common ambiguity of the last running task: is it still running or is the process stalled?
image-creatorbeing a machine-tool to be used by other systems as well as it being task-oriented and sensible to exofailures (downloads), it should be able to report its status (where it is in the state machine), what it's doing and overall progress in a machine-readable format.As a crucial SPOF, it should also allow post-mortem investigations.
Progress report and History, although regularly mixed for simplicity, respond to different needs and constraint:
Options for it includes stdout/stderr output, output to file or socket, file or TCP socket to read or query status.
Let's keep in mind that it will mostly be used from within a container (I suppose).
At this moment, my proposal would be:
stdoutis very easy to integrate and work-with. Kind of how it's always being used, so history can easily be read/fetched/archived.Progress would be dynamic, using CR character so that it doesn't pollute history (not visible in docker logs).
Using a specified file for machine-readable progress has several advantages:
/dev/shmfor instance) so you're not hitting IOsIt's important that long-lasting tasks report both to the history and periodically on the progress to avoid the common ambiguity of the last running task: is it still running or is the process stalled?