Skip to content

Commit dd816d7

Browse files
authored
feat: include run stdout/stderr in output (#31)
* feat: include run stdout/stderr in output * fix README
1 parent 6211c27 commit dd816d7

File tree

3 files changed

+43
-3
lines changed

3 files changed

+43
-3
lines changed

README.md

Lines changed: 33 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -68,9 +68,41 @@ gitbug-java checkout BID WORK_DIR [--fixed]
6868

6969
4. Run Actions
7070
```bash
71-
gitbug-java run WORK_DIR
71+
gitbug-java run WORKDIR [--act_cache_dir=ACT_CACHE_DIR | --timeout=TIMEOUT]
7272
```
7373

74+
A verbose mode is also available with the option `-v` or `--verbose`.
75+
76+
## Obtain parsed test execution results
77+
78+
The parsed test execution results are stored, after executing the `gitbug-java run` command, under `${WORKDIR}/.gitbug-java/test-results.json`
79+
The file includes the following information:
80+
```json
81+
{
82+
"expected_tests": num_expected_executed_tests,
83+
"executed_tests": num_executed_tests,
84+
"skipped_tests": num_skipped_tests,
85+
"passing_tests": num_executed_tests - num_failed_tests,
86+
"failing_tests": num_failed_tests,
87+
"unexpected_tests": list(unexpected_tests),
88+
"missing_tests": list(missing_tests),
89+
"failed_tests": [
90+
{"classname": test.classname, "name": test.name}
91+
for test in failed_tests
92+
],
93+
"run_outputs": [
94+
{
95+
"workflow_name": run.workflow_name,
96+
"stdout": run.stdout,
97+
"stderr": run.stderr,
98+
}
99+
for run in runs
100+
],
101+
}
102+
```
103+
104+
Note: Our output includes information from the entire GitHub Action run, including the stack-trace from the test run but also the output from other steps in the executed workflows. This is different from benchmarks such as Defects4J that provide only the test execution stack-trace segregated from other outputs. Currently, we do not support extracting only the test execution stack-trace.
105+
74106
## Citation
75107
76108
If you use GitBug-Java in your research work, please cite [GitBug-Java: A Reproducible Benchmark of Recent Java Bugs](https://arxiv.org/pdf/2402.02961.pdf)

gitbug-java

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -176,13 +176,13 @@ class GitBugJavaCli(object):
176176
def run(
177177
self,
178178
workdir: str,
179-
output: str = "report",
180179
act_cache_dir: Optional[str] = f"{get_project_root()}/act-cache",
181180
timeout: int = 0,
182181
) -> bool:
183182
"""
184183
Run the bug checked-out in workdir
185184
"""
185+
output = Path(workdir, ".gitbug-java")
186186
if not Path(output).exists():
187187
os.makedirs(output)
188188

gitbug/bug.py

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -276,7 +276,7 @@ def flat_failed_tests(runs):
276276
for unexpected_test in unexpected_tests:
277277
print(f"- {unexpected_test}")
278278

279-
output_path = os.path.join(output, f"{self.bid}.json")
279+
output_path = os.path.join(output, f"test-results.json")
280280
with open(output_path, "w") as f:
281281
json.dump(
282282
{
@@ -291,6 +291,14 @@ def flat_failed_tests(runs):
291291
{"classname": test.classname, "name": test.name}
292292
for test in failed_tests
293293
],
294+
"run_outputs": [
295+
{
296+
"workflow_name": run.workflow_name,
297+
"stdout": run.stdout,
298+
"stderr": run.stderr,
299+
}
300+
for run in runs
301+
],
294302
},
295303
f,
296304
)

0 commit comments

Comments
 (0)