Job States

[blobcode]

📖 Description
We need to enhance the scheduler so it can track job lifecycle states and know whether a job has been assigned to a supervisor. This will ensure visibility into job progress and enable proper rescheduling or error handling when failures occur.

✅ Objectives

• Add Job State Tracking in the scheduler
• Maintain job metadata (state, assigned supervisor, timestamps)
• Persist state transitions in Redis
• Define and implement the following states:
  • Scheduled / Waiting → job created but not yet assigned
  • In Progress → assigned to a supervisor, currently executing
  • Success → job completed successfully, results available
  • Error → job failed due to worker-side error (recoverable or reportable)
  • Failure → worker terminated unexpectedly, job must be rescheduled
• Implement State Transitions
  • Scheduled → In Progress (when a supervisor accepts the job)
  • In Progress → Success (job finishes successfully)
  • In Progress → Error (worker reports error)
  • In Progress → Failure (supervisor crashes or loses heartbeat)
  • Error/Failure → Scheduled (if retried)
• Integrate with existing scheduler logic
• Ensure jobs in non-terminal states are properly retried/rescheduled
• Prevent duplicate job assignment
• Testing & Documentation

🎯 Acceptance Criteria

• Scheduler tracks job state and assigned supervisor
• State transitions follow defined rules
• Completed jobs have results retrievable by state = Success
• Failed/error jobs are logged and eligible for rescheduling if configured
• Unit and integration tests are passing
• Documentation updated for developers and operators