feat: include run stdout/stderr in output (#31)

* feat: include run stdout/stderr in output

* fix README

Author: andre15silva

README.md

@@ -68,9 +68,41 @@ gitbug-java checkout BID WORK_DIR [--fixed]
 
 4. Run Actions
 ```bash
-gitbug-java run WORK_DIR
+gitbug-java run WORKDIR [--act_cache_dir=ACT_CACHE_DIR | --timeout=TIMEOUT]
 ```
 
+A verbose mode is also available with the option `-v` or `--verbose`.
+
+## Obtain parsed test execution results
+
+The parsed test execution results are stored, after executing the `gitbug-java run` command, under `${WORKDIR}/.gitbug-java/test-results.json`
+The file includes the following information:
+```json
+{
+    "expected_tests": num_expected_executed_tests,
+    "executed_tests": num_executed_tests,
+    "skipped_tests": num_skipped_tests,
+    "passing_tests": num_executed_tests - num_failed_tests,
+    "failing_tests": num_failed_tests,
+    "unexpected_tests": list(unexpected_tests),
+    "missing_tests": list(missing_tests),
+    "failed_tests": [
+        {"classname": test.classname, "name": test.name}
+        for test in failed_tests
+    ],
+    "run_outputs": [
+        {
+            "workflow_name": run.workflow_name,
+            "stdout": run.stdout,
+            "stderr": run.stderr,
+        }
+        for run in runs
+    ],
+}
+```
+
+Note: Our output includes information from the entire GitHub Action run, including the stack-trace from the test run but also the output from other steps in the executed workflows. This is different from benchmarks such as Defects4J that provide only the test execution stack-trace segregated from other outputs. Currently, we do not support extracting only the test execution stack-trace.
+
 ## Citation
 
 If you use GitBug-Java in your research work, please cite [GitBug-Java: A Reproducible Benchmark of Recent Java Bugs](https://arxiv.org/pdf/2402.02961.pdf)

gitbug-java

@@ -176,13 +176,13 @@ class GitBugJavaCli(object):
     def run(
         self,
         workdir: str,
-        output: str = "report",
         act_cache_dir: Optional[str] = f"{get_project_root()}/act-cache",
         timeout: int = 0,
     ) -> bool:
         """
         Run the bug checked-out in workdir
         """
+        output = Path(workdir, ".gitbug-java")
         if not Path(output).exists():
             os.makedirs(output)
 

gitbug/bug.py

@@ -276,7 +276,7 @@ def flat_failed_tests(runs):
                 for unexpected_test in unexpected_tests:
                     print(f"- {unexpected_test}")
 
-        output_path = os.path.join(output, f"{self.bid}.json")
+        output_path = os.path.join(output, f"test-results.json")
         with open(output_path, "w") as f:
             json.dump(
                 {
@@ -291,6 +291,14 @@ def flat_failed_tests(runs):
                         {"classname": test.classname, "name": test.name}
                         for test in failed_tests
                     ],
+                    "run_outputs": [
+                        {
+                            "workflow_name": run.workflow_name,
+                            "stdout": run.stdout,
+                            "stderr": run.stderr,
+                        }
+                        for run in runs
+                    ],
                 },
                 f,
             )