Any way to improve stability? (InstallExceptions, Process crashing etc.)

[growse]

I'm finding that a large percentage of my espresso test runs end in some sort of emulator crash or failure, causing the job to fail. This usually ends up in me effectively hitting the "re-run job" button until the stars align and the test suite runs to completion. The test suite is reliable enough when run locally, it's just on github actions that it fails.

Is this a problem that others are seeing, or is it the result of something that I'm doing wrong / badly within my project? Do I just have to accept that the emulator on github runners is hilariously unstable?

One output example (https://github.com/owntracks/android/runs/1897406606?check_suite_focus=true):

Run reactivecircus/android-emulator-runner@v2
  with:
    api-level: 29
    working-directory: project
    script: adb emu geo fix 0 51
  ./gradlew createDebugCoverageReport --no-build-cache
  
    profile: pixel_3a
    target: google_apis
    disable-animations: true
    arch: x86_64
    avd-name: test
    emulator-options: -no-window -gpu swiftshader_indirect -no-snapshot -noaudio -no-boot-anim
  env:
    JAVA_HOME_11.0.10_x64: /Users/runner/hostedtoolcache/jdk/11.0.10/x64
    JAVA_HOME: /Users/runner/hostedtoolcache/jdk/11.0.10/x64
    JAVA_HOME_11_0_10_X64: /Users/runner/hostedtoolcache/jdk/11.0.10/x64
API level: 29
target: google_apis
CPU architecture: x86_64
Hardware profile: pixel_3a
SD card path or size: 
AVD name: test
emulator options: -no-window -gpu swiftshader_indirect -no-snapshot -noaudio -no-boot-anim
disable animations: true
custom working directory: project
Script:
adb emu geo fix 0 51
./gradlew createDebugCoverageReport --no-build-cache
Installing latest build tools, platform tools, and platform.
...
 Emulator booted.
/Users/runner/Library/Android/sdk/platform-tools/adb shell input keyevent 82
emulator: INFO: boot completed
emulator: INFO: boot time 114955 ms
emulator: Increasing screen off timeout, logcat buffer size to 2M.
emulator: Revoking microphone permissions for Google App.
Disabling animations.
...
> Task :app:mergeDebugAndroidTestJavaResource
> Task :app:dexBuilderDebugAndroidTest
> Task :app:mergeLibDexDebugAndroidTest
> Task :app:mergeProjectDexDebugAndroidTest
> Task :app:mergeExtDexDebugAndroidTest
> Task :app:packageDebugAndroidTest
emulator: ### WARNING: /etc/localtime does not point to zoneinfo-compatible timezone name

> Task :app:connectedDebugAndroidTest
Unable to install /Users/runner/work/android/android/project/app/build/outputs/apk/debug/app-debug.apk
com.android.ddmlib.InstallException
	at com.android.ddmlib.internal.DeviceImpl.installRemotePackage(DeviceImpl.java:1230)
	at com.android.ddmlib.internal.DeviceImpl.installPackage(DeviceImpl.java:1050)
	at com.android.ddmlib.internal.DeviceImpl.installPackage(DeviceImpl.java:1026)
	at com.android.ddmlib.internal.DeviceImpl.installPackage(DeviceImpl.java:1015)
	at com.android.build.gradle.internal.testing.ConnectedDevice.installPackage(ConnectedDevice.java:130)
	at com.android.build.gradle.internal.testing.SimpleTestRunnable.run(SimpleTestRunnable.java:134)
	at com.android.ide.common.workers.ExecutorServiceAdapter$submit$submission$1.run(ExecutorServiceAdapter.kt:68)
	at java.base/java.util.concurrent.ForkJoinTask$AdaptedRunnableAction.exec(ForkJoinTask.java:1407)
	at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
	at java.base/java.util.concurrent.ForkJoinTask.tryExternalHelp(ForkJoinTask.java:381)
	at java.base/java.util.concurrent.ForkJoinTask.externalInterruptibleAwaitDone(ForkJoinTask.java:351)
	at java.base/java.util.concurrent.ForkJoinTask.get(ForkJoinTask.java:1004)
	at com.android.ide.common.workers.ExecutorServiceAdapter.await(ExecutorServiceAdapter.kt:102)
	at com.android.build.gradle.internal.testing.BaseTestRunner.runTests(BaseTestRunner.java:193)
	at com.android.build.gradle.internal.tasks.DeviceProviderInstrumentTestTask.lambda$doTaskAction$2(DeviceProviderInstrumentTestTask.java:206)
	at com.android.builder.testing.api.DeviceProvider.use(DeviceProvider.java:53)
	at com.android.build.gradle.internal.tasks.DeviceProviderInstrumentTestTask.doTaskAction(DeviceProviderInstrumentTestTask.java:194)
	at com.android.build.gradle.internal.tasks.NonIncrementalTask$taskAction$$inlined$recordTaskAction$1.invoke(AndroidVariantTask.kt:74)
	at 
<...enormous stacktrace snipped...> 
org.gradle.launcher.daemon.server.exec.RequestStopIfSingleUsedDaemon.execute(RequestStopIfSingleUsedDaemon.java:35)
	at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:104)
	at org.gradle.launcher.daemon.server.exec.ForwardClientInput$2.create(ForwardClientInput.java:78)
	at org.gradle.launcher.daemon.server.exec.ForwardClientInput$2.create(ForwardClientInput.java:75)
	at org.gradle.util.Swapper.swap(Swapper.java:38)
	at org.gradle.launcher.daemon.server.exec.ForwardClientInput.execute(ForwardClientInput.java:75)
	at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:104)
	at org.gradle.launcher.daemon.server.exec.LogAndCheckHealth.execute(LogAndCheckHealth.java:55)
	at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:104)
	at org.gradle.launcher.daemon.server.exec.LogToClient.doBuild(LogToClient.java:63)
	at org.gradle.launcher.daemon.server.exec.BuildCommandOnly.execute(BuildCommandOnly.java:37)
	at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:104)
	at org.gradle.launcher.daemon.server.exec.EstablishBuildEnvironment.doBuild(EstablishBuildEnvironment.java:84)
	at org.gradle.launcher.daemon.server.exec.BuildCommandOnly.execute(BuildCommandOnly.java:37)
	at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:104)
	at org.gradle.launcher.daemon.server.exec.StartBuildOrRespondWithBusy$1.run(StartBuildOrRespondWithBusy.java:52)
	at org.gradle.launcher.daemon.server.DaemonStateCoordinator$1.run(DaemonStateCoordinator.java:297)
	at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
	at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:48)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:56)
	at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: com.android.ddmlib.ShellCommandUnresponsiveException
	at com.android.ddmlib.AdbHelper.executeRemoteCommand(AdbHelper.java:598)
	at com.android.ddmlib.AdbHelper.executeRemoteCommand(AdbHelper.java:381)
	at com.android.ddmlib.internal.DeviceImpl.executeShellCommand(DeviceImpl.java:711)
	at com.android.ddmlib.internal.DeviceImpl.installRemotePackage(DeviceImpl.java:1221)
	... 250 more

com.android.build.gradle.internal.testing.ConnectedDevice > runTests[test(AVD) - 10] FAILED 
	com.android.builder.testing.api.DeviceException: com.android.ddmlib.InstallException
		at com.android.build.gradle.internal.testing.ConnectedDevice.installPackage(ConnectedDevice.java:136)

Here's another example where the process crashes (https://github.com/owntracks/android/runs/1805228040?check_suite_focus=true)

> Task :app:kaptGenerateStubsDebugAndroidTestKotlin
> Task :app:kaptDebugAndroidTestKotlin
> Task :app:compileDebugAndroidTestKotlin
> Task :app:compileDebugAndroidTestJavaWithJavac

> Task :app:transformClassesWithObjectBoxAndroidTransformForDebugAndroidTest
Transformed 0 entities and copied 13 classes in 9 ms

> Task :app:mergeDebugAndroidTestJavaResource
> Task :app:dexBuilderDebugAndroidTest
> Task :app:mergeLibDexDebugAndroidTest
> Task :app:mergeProjectDexDebugAndroidTest
> Task :app:mergeExtDexDebugAndroidTest
> Task :app:packageDebugAndroidTest
> Task :app:connectedDebugAndroidTest
emulator: ### WARNING: /etc/localtime does not point to zoneinfo-compatible timezone name

Starting 27 tests on test(AVD) - 10

org.owntracks.android.e2e.ContactActivityTests > testClickingOnContactLoadsContactOnMap[test(AVD) - 10] FAILED 
Test failed to run to completion. Reason: 'Instrumentation run failed due to 'Process crashed.''. Check device logcat for details
Tests on test(AVD) - 10 failed: Instrumentation run failed due to 'Process crashed.'

> Task :app:connectedDebugAndroidTest FAILED

From my workflow:

jobs:
  test:
    name: Build & Test
    runs-on: macos-latest
    steps:
    - uses: actions/checkout@v2
    - name: set up JDK 11
      uses: actions/setup-java@v1
      with:
        java-version: 11
    - uses: actions/cache@v2
      with:
        path: |
          ~/.gradle/caches
          ~/.gradle/wrapper
        key: ${{ runner.os }}-gradle-${{ github.repository }}-${{ hashFiles('**/*.gradle*') }}
    - name: Build with Gradle
      run: ./gradlew assembleDebug
      working-directory: project
      timeout-minutes: 10
    - name: Run unit tests
      run: ./gradlew check jacocoTestDebugUnitTestReport
      working-directory: project
      timeout-minutes: 10
    - name: Test with Gradle on the emulator
      uses: reactivecircus/android-emulator-runner@v2
      with:
        api-level: 29
        working-directory: project
        script: |
            adb emu geo fix 0 51
            ./gradlew createDebugCoverageReport --no-build-cache
        profile: pixel_3a
        target: google_apis
        disable-animations: true
        arch: x86_64

[ychescale9]

I've also been getting all kinds of issues with newer version of the system images recently, we could be caused by updated version of the emulator binary / system images. There's still no way to ping these SDK components to a specific version, so the action always downloads the latest stable version.

I've also seen occasional intermittent issues with macos VMs which we unfortunately have no control either.

Have you tried running with a lower API level?

[AfzalivE]

Haven't tested the following very thoroughly but not downloading the new cmdline-tools and just using the tools already present in the macos image seems to be more stable. Reason:

config.ini for the AVD created with the latest cmdline tools seems to have many more options and possibly more points of failure, or more variables that require tweaking.

The only reason I found this out is because our Azure pipeline was using the built-in avdmanager (26.1.1) and was much more stable than the same tests running on GitHub Actions with the avdmanager (3.0?) in cmdline-tools.

Here's our fork if you're curious about it but I wouldn't suggest using our fork since release/v2 is where I'm experimenting with things.

https://github.com/Doist/android-emulator-runner

[ychescale9]

Thanks for sharing your findings.

The action has been using cmdline-tools 3.0 since November last year so I'd be surprised if it's the cause of the recent regression.

The main rationale for using the new cmdline-tools is being able to build with Java 9+ (new AGP now requires Java 11+).

I haven't had a chance to look into this so please do share any insights your find from your experiment!

BTW cmdline-tools 3.0 is installed in the latest macOS VM so we don't need to download it either way.

[AfzalivE]

One thing I noticed after you mentioned that cmdline-tools had Java 9+ support is that the SDK tools are deprecated and haven't been updated since Sept 2017! So I decided to dig deeper into how the config.ini created from tools/avdmanager was different from the one created by avdmanager from cmdline-tools.

cmdline3.0-config.ini.txt

sdk-config.ini.txt

Seem very different, right? The interesting part to me was hw.cpu.ncore=1 in the cmdline-3.0 config.ini.

The default value for this property is 2 in AOSP. So I used the cmdline-tools and just appended hw.cpu.ncore=2 to the config.ini. Lo and behold! Some issues I was seeing a lot (e.g. data loading until timeout and instrumentation crashes): Gone! (or at least reduced drastically).

[ychescale9]

@AfzalivE thanks for digging into this!

Is it worth firing a bug on the issue tracker to confirm this is the safe to do and get more context around this significant change?

And please feel free to send a PR if you think this is worth trying to improve instability.

[AfzalivE]

Is it worth firing a bug on the issue tracker to confirm this is the safe to do and get more context around this significant change?

Maybe, created it. Found why it's happening too: https://cs.android.com/android-studio/platform/tools/base/+/mirror-goog-studio-master-dev:sdklib/src/main/java/com/android/sdklib/internal/avd/EmulatedProperties.java;l=46?q=RECOMMENDED_NUMBER_OF_CORES

Since the macOS VM has 3 cores, the resulting number is 1. Maybe the issue will get closed as "working as intended" but in my experience, this is stabler than 1 in the GitHub-hosted runner.

I'll send a PR your way, also added a few more options in our fork that I'll send upstream but I'll keep them in separate PRs so you can accept/reject them individually if you like.

[ychescale9]

Wow great work! Looking forward to the PRs :)

[growse]

So from the looks of things, stability is now much better than it was before!

Thanks for everyone's work on this :)