Exit code tracking and 127 handling

[Matios102]

#74

This pull request updates the worker's test case evaluation to improve error reporting, especially around process exit codes, and refines the message formats in the documentation and code. The changes clarify the meaning of exit codes in test results, add more detailed error messages, and enhance test coverage for edge cases like command-not-found errors. The README has also been updated to better document the message structure and processing flow.

Test case evaluation and error reporting:

• Added explicit handling for exit code 127 (command not found), which may indicate memory limit issues, with a new error message and status in verifier.go and corresponding constant in constants.go [1] [2] [3].
• Included the ExitCode field in all test result objects for more granular error reporting, both in the code and in the example response structures in the README [1] [2] [3] [4] [5] [6] [7] [8] [9].
• Updated error messages in the README to clarify the meaning of exit codes and to distinguish between different failure types, such as time limit, memory limit, and non-zero exit codes [1] [2].

Testing improvements:

• Extended unit tests in verifier_test.go to cover cases for command-not-found exit code, output comparison failures, and to assert correct handling of the new ExitCode field [1] [2] [3] [4] [5].

Documentation updates:

• Overhauled the README's message structure and processing flow to match the new fields and error handling, including more detailed payload examples and clearer descriptions of each processing step [1] [2] [3] [4] [5] [6] [7].

[Copilot]

Pull request overview

This pull request enhances exit code tracking and error reporting in the worker's test case evaluation system. The key improvement is the addition of explicit handling for exit code 127, which may indicate memory limit issues due to failed shared library loading. The PR also adds the ExitCode field to all test results for more granular diagnostics and updates the README with more comprehensive examples and explanations of the message structure.

Key changes:

• Added ExitCode field to TestResult struct for detailed error tracking across all test outcomes
• Implemented special handling for exit code 127 (command not found) with a dedicated error message suggesting possible memory limit issues
• Enhanced test coverage with new test cases for exit code 127 and output comparison failures
• Updated README documentation with complete message examples showing exit codes and clarified error message formats

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
pkg/solution/solution.go	Added ExitCode field to TestResult struct to track process exit codes
pkg/constants/constants.go	Added ExitCodeCommandNotFound constant and TestCaseMessagePossiblyExceededMemory error message template
internal/stages/verifier/verifier.go	Implemented exit code 127 handling and populated ExitCode field in all test result branches
internal/stages/verifier/verifier_test.go	Added tests for command not found scenario and output comparison failures, with exit code validation
README.md	Updated message structure examples to include exit codes and clarified error message formats for different failure types

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.