Skip to content

Retry instrumented tests for a maximum of 3 times because of CI flakiness #654

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
ggtlvkma356 wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Retry instrumented tests for a maximum of 3 times because of CI flakiness #654

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
47 changes: 21 additions & 26 deletions .github/workflows/build-and-test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,32 +34,27 @@ jobs:
cache: npm
- run: npm ci
- run: npm test
instrumented-tests-1:
uses: ./.github/workflows/instrumented-tests-run.yml
instrumented-tests-2:
needs: instrumented-tests-1
if: always() && needs.instrumented-tests-1.outputs.test-outcome == 'failure'
uses: ./.github/workflows/instrumented-tests-run.yml
instrumented-tests-3:
needs: instrumented-tests-2
if: always() && needs.instrumented-tests-2.outputs.test-outcome == 'failure'
uses: ./.github/workflows/instrumented-tests-run.yml
instrumented-tests:
if: always()
needs: [instrumented-tests-1, instrumented-tests-2, instrumented-tests-3]
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
persist-credentials: false
- name: Enable KVM
run: |
echo 'KERNEL=="kvm", GROUP="kvm", MODE="0666", OPTIONS+="static_node=kvm"' | sudo tee /etc/udev/rules.d/99-kvm4all.rules
sudo udevadm control --reload-rules
sudo udevadm trigger --name-match=kvm
- uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
with:
cache: npm
- name: Set up JDK 21
uses: actions/setup-java@be666c2fcd27ec809703dec50e508c2fdc7f6654 # v5.2.0
with:
distribution: 'temurin'
java-version: 21
cache: gradle
- name: Run instrumented tests
uses: reactivecircus/android-emulator-runner@e89f39f1abbbd05b1113a29cf4db69e7540cae5a # v2.37.0
with:
api-level: 36
target: google_apis
arch: x86_64
emulator-options: -no-window -gpu swiftshader_indirect -noaudio -no-boot-anim -camera-back none
disable-animations: true
script: ./gradlew connectedCheck --no-daemon
- run: |
if [[ "${{ needs.instrumented-tests-1.outputs.test-outcome }}" == "success" || \
"${{ needs.instrumented-tests-2.outputs.test-outcome }}" == "success" || \
"${{ needs.instrumented-tests-3.outputs.test-outcome }}" == "success" ]]; then
echo "Instrumented tests passed"
exit 0
fi
echo "All 3 attempts failed"
exit 1
51 changes: 51 additions & 0 deletions .github/workflows/instrumented-tests-run.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
name: Instrumented tests run

on:
workflow_call:
outputs:
test-outcome:
value: ${{ jobs.test.outputs.test-outcome }}

permissions:
contents: read

jobs:
test:
runs-on: ubuntu-latest
timeout-minutes: 30
outputs:
test-outcome: ${{ steps.tests.outcome }}
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
persist-credentials: false
- name: Enable KVM
run: |
echo 'KERNEL=="kvm", GROUP="kvm", MODE="0666", OPTIONS+="static_node=kvm"' | sudo tee /etc/udev/rules.d/99-kvm4all.rules
sudo udevadm control --reload-rules
sudo udevadm trigger --name-match=kvm
- uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
with:
cache: npm
- name: Set up JDK 21
uses: actions/setup-java@be666c2fcd27ec809703dec50e508c2fdc7f6654 # v5.2.0
with:
distribution: 'temurin'
java-version: 21
cache: gradle
- name: Run instrumented tests
id: tests
continue-on-error: true
uses: reactivecircus/android-emulator-runner@e89f39f1abbbd05b1113a29cf4db69e7540cae5a # v2.37.0
with:
api-level: 36
target: google_apis
arch: x86_64
ram-size: 2048M
heap-size: 512M
disk-size: 6000M
emulator-options: -no-window -gpu swiftshader_indirect -noaudio -no-boot-anim -camera-back none -no-snapshot
disable-animations: true
script: |
adb shell am wait-for-broadcast-idle
./gradlew connectedCheck --no-daemon
Copy link
Copy Markdown
Member

@inthewaves inthewaves May 15, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe it could be possible to write a script that would only retry the tests that failed, it doesn't seem like failures always render the emulator unusable

e.g. https://github.com/GrapheneOS/PdfViewer/actions/runs/25924498009/job/76201971415?pr=654 appeared to be able to run all 76 tests and only failed 2 tests. At least locally, connectedCheck puts tests results in app/build/outputs/androidTest-results/connected/<variant>/<device-id>/TEST-*.xml

There's also test-result.pb, but I'm not sure if the .proto for it is known (https://issuetracker.google.com/issues/337538011)

There's a plugin but I don't think it works for Android UI tests like these: gradle/test-retry-gradle-plugin#289

Copy link
Copy Markdown
Member

@inthewaves inthewaves May 15, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

.proto is probably in Java classes from https://mvnrepository.com/artifact/com.google.testing.platform/core-proto/0.0.9-alpha04 (Apache 2.0 library), not really usable as is

Copy link
Copy Markdown
Member

@inthewaves inthewaves May 15, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Example: protoc --proto_path=$PWD --decode=google.testing.platform.proto.api.core.TestSuiteResult test_suite_result.proto < test-result.pb

test_suite_result.proto (or per-file version from https://github.com/inthewaves/com.google.testing.platform-core-proto):

syntax = "proto3";

package google.testing.platform.proto.api.core;

enum ArtifactType {
  ARTIFACT_TYPE_UNSPECIFIED = 0;
  EXECUTABLE = 1;
  ANDROID_APK = 2;
  TEST_DATA = 3;
  ANDROID_APP_BUNDLE = 4;
  ANDROID_APK_SET = 5;
  REMOTE_DATA = 6;
}

enum TestStatus {
  TEST_STATUS_UNSPECIFIED = 0;
  STARTED = 1;
  FAILED = 2;
  PASSED = 3;
  IGNORED = 4;
  ERROR = 5;
  ABORTED = 6;
  CANCELLED = 7;
  SKIPPED = 8;
  IN_PROGRESS = 9;
}

enum Type {
  DEFAULT = 0;
  BOOL = 1;
  CLASS = 2;
  DOUBLE = 3;
  ENUM = 4;
  INTEGER = 5;
  LONG = 6;
  STRING = 7;
  METHOD = 8;
  FIELD = 9;
  FLOAT = 20;
  CHAR = 12;
  SHORT = 13;
  BYTE = 14;
  NULL = 15;
  ANNOTATION = 16;
}

message TestSuiteResult {
  TestSuiteMetaData test_suite_meta_data = 1;
  TestStatus test_status = 2;
  repeated TestResult test_result = 3;
  PlatformError platform_error = 4;
  repeated Artifact output_artifact = 5;
  repeated Issue issue = 6;
}

message DeviceId {
  string id = 1;
  string friendly_name = 2;
}

message Annotation {
  string class_name = 1;
  repeated AnnotationValue annotation_value = 2;
}

message AnnotationValue {
  string field_name = 1;
  repeated string field_value = 2;
  Type field_type = 3;
  bool is_array = 4;
  repeated Annotation field_annotation_value = 6;
}

message Artifact {
  Label label = 1;
  Path source_path = 2;
  Path destination_path = 3;
  ArtifactType type = 4;
  string checksum = 5;
  string mime_type = 6;
  Any handling = 7;
}

message Error {
  string error_message = 1;
  string error_type = 2;
  string stack_trace = 3;
}

message ErrorDetail {
  ErrorSummary summary = 1;
  ErrorDetail cause = 2;
  repeated ErrorDetail suppressed = 3;
}

message ErrorSummary {
  Label namespace = 1;
  int32 error_code = 2;
  string error_name = 3;
  string error_classification = 4;
  string error_message = 5;
  string class_name = 6;
  string stack_trace = 7;
}

message Issue {
  enum Severity {
    SEVERITY_UNSPECIFIED = 0;
    INFO = 1;
    SUGGESTION = 2;
    WARNING = 3;
    SEVERE = 4;
  }
  Label namespace = 1;
  Severity severity = 2;
  int32 code = 3;
  string name = 4;
  string message = 5;
}

message Label {
  string label = 1;
  string namespace = 2;
}

message Path {
  string path = 1;
}

message PlatformError {
  repeated ErrorDetail errors = 2;
}

message TestCase {
  string test_class = 1;
  string test_package = 2;
  string test_method = 3;
  repeated Annotation test_class_annotation = 4;
  repeated Annotation test_method_annotation = 5;
  Timestamp start_time = 6;
  Timestamp end_time = 7;
  DeviceId device = 8;
}

message TestResult {
  message TestDetailsEntry {
    string key = 1;
    string value = 2;
  }
  TestCase test_case = 1;
  TestStatus test_status = 2;
  repeated TestDetailsEntry details = 5;
  Error error = 3;
  repeated Artifact output_artifact = 4;
}

message TestSuiteMetaData {
  string test_suite_name = 1;
  int32 scheduled_test_case_count = 2;
  DeviceId device = 3;
}

message Timestamp {
  int64 seconds = 1;
  int32 nanos = 2;
}

message Any {
  string type_url = 1;
  bytes value = 2;
}

Loading