Using Patrol on Firebase Test Lab

[Giuspepe]

This package looks very promising and easy to use - thanks for open sourcing it!

Are there plans to make patrol compatible with running integration tests on Firebase Test Lab?

If I understand correctly, patrol can be used on local machines by running patrol drive, but I haven't found any instructions on whether it's possible to use patrol on CI/CD as well. I'm thinking of something similar to the integration_test package which makes it possible to run integration tests on CI/CD with Firebase Test Lab (see the following instructions: https://docs.flutter.dev/testing/integration-tests#testing-on-firebase-test-lab).

[bartekpacia]

Hi @Giuspepe! Congrats on opening the first external issue in this repository! 🥳

Are there plans to make Patrol compatible with running integration tests on Firebase Test Lab?

We definitely want to make Patrol run on as many CI providers as possible. I don't have much experience using Firebase Test Lab, so I quickly skimmed through its docs and from what I see, they let you upload the APK of the app under test (app-debug.apk) and the instrumentation APK (app-debug-androidTest.apk). Unfortunately it looks like they don't offer a possibility to run an executable, like patrol, so it's definitely gonna require some research. Thanks for that link to the docs, it's helpful.

Until now, we've made Patrol run integration tests on GitHub Actions. Check out the run-integration-tests.yaml workflow to see an example of this.

[pawlowskim]

Hello,

@bartekpacia could you give some hint how did you integrate patrol tests on your CI?

1. What CI are you using?
2. Are you using real devices or simulators to run tests?
3. How do you handle failed tests?

[bartekpacia]

Hi @pawlowskim 👋

What CI are you using?

GitHub Actions. You can see the workflow files in the .github/workflows folder in this repository.

Are you using real devices or simulators to run tests?

Currently, we only use Android emulators.

We want to run tests on iOS simulators and physical iOS devices as well, but that'd require us to use some device farm. We didn't have time for this yet, but it's definitely on our radar.

How do you handle failed tests?

Short answer: we don't. We run a single test and if it fails, patrol exits with a non-zero exit code, which fails the job, and makes the workflow run red. It's a very, very basic form of error reporting, I admit.

I can assure you though that will focus on this area in the near future. If you have any ideas or suggestions on how we should approach this topic (or just some random, useful thoughts on the topic), share them with us :)

PS Another problem we're facing in GitHub Actions is the flakiness of connecting to the VM service running on the device. See #272 to learn more.

[JulianBissekkou]

I would like to understand why we need to call patrol drive and why this is not a standard flutter integration test.
I assume that this is because of the AutomatorServer that needs to be started, but I would love to get a more detailed explanation of the problem 🤔

Maybe you can guide me to some of the files that are important to fully understand this problem.
We are working with firebase test lab and I might be able to give some input on the problem and find a potential fix 👍🏽

Maybe we can also document how patrol works internally in #173 :)

[bartekpacia]

I would like to understand why we need to call patrol drive and why this is not a standard flutter integration test.
I assume that this is because of the AutomatorServer that needs to be started

Exactly. We need to run the native automation server, which enables interaction with native UI, and there's no way to do this by using only flutter test.

Maybe we can also document how patrol works internally in #173 :)

Good idea.

[JulianBissekkou]

@bartekpacia
I just looked into the sources and I was wondering if we can solve this issue by starting the AutomationServer in a custom test driver that is registered on the native side.

For example:

/*imports*/
-@RunWith(FlutterTestRunner.class)
+@RunWith(PatrolTestRunner.class)
public class MainActivityTest {
    @Rule
    public ActivityTestRule<MainActivity> rule = new ActivityTestRule<>(MainActivity.class, true, false);
}

This might be a naiv impl for starting the server.

class PatrolTestRunner extends FlutterTestRunner {

    public PatrolTestRunner(Class<?> testClass) {
        super(testClass);
    }

    @Override
    public void run(RunNotifier notifier) {
        startPatrolServer();
        super.run(notifier);
        stopPatrolServer();
    }
}

We can also move this into a Github discussion if you like. Or we can have a quick chat/call 👍🏽

[bartekpacia]

@JulianBissekkou This looks very interesting – thank you! I'll share this with the team and get back to you.

I think there's no need for a separate Discussion – this issue is the right place for this.

[JulianBissekkou]

On iOS this might be a little tricky because we can't just subclass FLTIntegrationTestRunner.
(Maybe I am missing something)
I created an Issue in the flutter repository: flutter/flutter#113519

[bartekpacia]

@JulianBissekkou, we're working on this in #661 :)

[bartekpacia]

I'm happy to say that basic support is almost ready 🎉 All that's left is writing good docs (and maybe squashing some bugs) :)

See also this.

[neiljaywarner]

@bartekpacia very exciting

[bartekpacia]

Look no further! Here are the docs!

Please report issues in case there are problems :) But Firebase Test Lab is now officially supported.

[JulianBissekkou]

Really cool! Good job guys 💪🏽
Is the AutomationServer now started in PATROL_INTEGRATION_TEST_IOS_RUNNER and PatrolTestRule ?

[bartekpacia]

Yes!

[JulianBissekkou]

Just got the time to integrate patrol into my app. When running the tests from the IDE we still have to use patrol drive, right?
By default the IDEs will execute test using flutter test. This will also be fixed in #661, right?

[bartekpacia]

Oops, forgot to respond because :) Sorry!

When running the tests from the IDE we still have to use patrol drive, right?

Currently, you can use patrol drive (to run the tests with flutter_driver), or you can use the new patrol test run the tests natively: with Gradle on Android and Xcode/xcodebuild on iOS. But patrol drive is deprecated and we plan to remove it in 1.0.

By default the IDEs will execute test using flutter test. This will also be fixed in #661, right?

flutter test by default runs widget/unit tests under test/, so it shouldn't be a problem.

But flutter test integration_test won't work with Patrol's native automation feature. You have to use patrol test.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar problem, please file a new issue. Make sure to follow the template and provide all the information necessary to reproduce the issue.