Retry instrumented tests for a maximum of 3 times because of CI flakiness

[ggtlvkma356]

No description provided.

[thestinger]

Is this a GitHub issue? I don't mind just triggering it to run again.

[ggtlvkma356]

It seems to be because of the free runners are quite small so the emulators are not running in the most stable state. I have tested with multiple runs and I get random failed tests.

According to this, retrying is actually recommended. We can manually re-run them but I think it is better just make it automatic.

[inthewaves]

Maybe it could be possible to write a script that would only retry the tests that failed, it doesn't seem like failures always render the emulator unusable

e.g. https://github.com/GrapheneOS/PdfViewer/actions/runs/25924498009/job/76201971415?pr=654 appeared to be able to run all 76 tests and only failed 2 tests. At least locally, connectedCheck puts tests results in app/build/outputs/androidTest-results/connected/<variant>/<device-id>/TEST-*.xml

There's also test-result.pb, but I'm not sure if the .proto for it is known (https://issuetracker.google.com/issues/337538011)

There's a plugin but I don't think it works for Android UI tests like these: gradle/test-retry-gradle-plugin#289

[inthewaves]

.proto is probably in Java classes from https://mvnrepository.com/artifact/com.google.testing.platform/core-proto/0.0.9-alpha04 (Apache 2.0 library), not really usable as is

[inthewaves]

Example: protoc --proto_path=$PWD --decode=google.testing.platform.proto.api.core.TestSuiteResult test_suite_result.proto < test-result.pb

test_suite_result.proto (or per-file version from https://github.com/inthewaves/com.google.testing.platform-core-proto):

syntax = "proto3";

package google.testing.platform.proto.api.core;

enum ArtifactType {
  ARTIFACT_TYPE_UNSPECIFIED = 0;
  EXECUTABLE = 1;
  ANDROID_APK = 2;
  TEST_DATA = 3;
  ANDROID_APP_BUNDLE = 4;
  ANDROID_APK_SET = 5;
  REMOTE_DATA = 6;
}

enum TestStatus {
  TEST_STATUS_UNSPECIFIED = 0;
  STARTED = 1;
  FAILED = 2;
  PASSED = 3;
  IGNORED = 4;
  ERROR = 5;
  ABORTED = 6;
  CANCELLED = 7;
  SKIPPED = 8;
  IN_PROGRESS = 9;
}

enum Type {
  DEFAULT = 0;
  BOOL = 1;
  CLASS = 2;
  DOUBLE = 3;
  ENUM = 4;
  INTEGER = 5;
  LONG = 6;
  STRING = 7;
  METHOD = 8;
  FIELD = 9;
  FLOAT = 20;
  CHAR = 12;
  SHORT = 13;
  BYTE = 14;
  NULL = 15;
  ANNOTATION = 16;
}

message TestSuiteResult {
  TestSuiteMetaData test_suite_meta_data = 1;
  TestStatus test_status = 2;
  repeated TestResult test_result = 3;
  PlatformError platform_error = 4;
  repeated Artifact output_artifact = 5;
  repeated Issue issue = 6;
}

message DeviceId {
  string id = 1;
  string friendly_name = 2;
}

message Annotation {
  string class_name = 1;
  repeated AnnotationValue annotation_value = 2;
}

message AnnotationValue {
  string field_name = 1;
  repeated string field_value = 2;
  Type field_type = 3;
  bool is_array = 4;
  repeated Annotation field_annotation_value = 6;
}

message Artifact {
  Label label = 1;
  Path source_path = 2;
  Path destination_path = 3;
  ArtifactType type = 4;
  string checksum = 5;
  string mime_type = 6;
  Any handling = 7;
}

message Error {
  string error_message = 1;
  string error_type = 2;
  string stack_trace = 3;
}

message ErrorDetail {
  ErrorSummary summary = 1;
  ErrorDetail cause = 2;
  repeated ErrorDetail suppressed = 3;
}

message ErrorSummary {
  Label namespace = 1;
  int32 error_code = 2;
  string error_name = 3;
  string error_classification = 4;
  string error_message = 5;
  string class_name = 6;
  string stack_trace = 7;
}

message Issue {
  enum Severity {
    SEVERITY_UNSPECIFIED = 0;
    INFO = 1;
    SUGGESTION = 2;
    WARNING = 3;
    SEVERE = 4;
  }
  Label namespace = 1;
  Severity severity = 2;
  int32 code = 3;
  string name = 4;
  string message = 5;
}

message Label {
  string label = 1;
  string namespace = 2;
}

message Path {
  string path = 1;
}

message PlatformError {
  repeated ErrorDetail errors = 2;
}

message TestCase {
  string test_class = 1;
  string test_package = 2;
  string test_method = 3;
  repeated Annotation test_class_annotation = 4;
  repeated Annotation test_method_annotation = 5;
  Timestamp start_time = 6;
  Timestamp end_time = 7;
  DeviceId device = 8;
}

message TestResult {
  message TestDetailsEntry {
    string key = 1;
    string value = 2;
  }
  TestCase test_case = 1;
  TestStatus test_status = 2;
  repeated TestDetailsEntry details = 5;
  Error error = 3;
  repeated Artifact output_artifact = 4;
}

message TestSuiteMetaData {
  string test_suite_name = 1;
  int32 scheduled_test_case_count = 2;
  DeviceId device = 3;
}

message Timestamp {
  int64 seconds = 1;
  int32 nanos = 2;
}

message Any {
  string type_url = 1;
  bytes value = 2;
}

[inthewaves]

After some discussion I think this is fine. We could use parallel runners in the future if running time gets too long to detect genuine test failures