Add screenshot testing

[Guardiola31337]

We're gonna start working on adding screenshot testing to the SDK Test App.
Espresso doesn't find native views to assert on.
This will let us to add test coverage to some use cases (e.g. Add a marker to a map).

There are some options to add screenshot testing in Android:

• screenshot-tests-for-android
  • Already played with the framework
• test-screenshots
  • Works well if you already have firebase testing
  • Works with Instrumentation and Espresso
  • Allows you to compare between different devices easily

Any thoughts?

cc/ @mapbox/android

[zugaldia]

@Guardiola31337 Thank you for starting this effort, it's very helpful.

Considering we're already using Firebase, any reasons/advantages to not use their tools?

[Guardiola31337]

@zugaldia I haven't used test-screenshots from Firebase before. Because of that, I started with screenshot-tests-for-android from Facebook in the first place. But, definitely I have to try the Firebase solution, because screenshot-tests-for-android doesn't support TextureView/SurfaceView. Was working on adding support to TextureView with no luck 😿

All the findings are explained in the Add TextureView/SurfaceView support ticket.

Next step would be to check test-screenshots from Firebase and see if there is the same issue.

[zugaldia]

@Guardiola31337 sounds like a plan 👍 . I also subscribed to facebook/screenshot-tests-for-android#55 and see if we get some insights there.

[Guardiola31337]

test-screenshots from Firebase seems to have same issue as screenshot-tests-for-android from Facebook.

I wrote a test which loads a simple map, adds a marker and takes a screenshot.
This time (as you can see below) I'm getting a screenshot with the part of the map in black.
Screenshot:

The test is working properly as you can see in the following video:

We could report the issue to Google (asking them if they are going to support this in the near future) or we could spend more time adding it ourselves to screenshot-tests-for-android.

What do you think? Any alternative?

Cc/ @mapbox/android

[zugaldia]

Good research @Guardiola31337 .

We could report the issue to Google (asking them if they are going to support this in the near future)

Is Firebase's tool open source like Facebook's?

or we could spend more time adding it ourselves to screenshot-tests-for-android.

It looks like fixing one (say Facebook because it's open source) will help fixing the other, and therefore getting us more options to test.

[Guardiola31337]

Is Firebase's tool open source like Facebook's?

No, it's not. Actually, to make it work, you have to include a private library called cloudtestingscreenshotter_lib in the project. It's all explained in the documentation.

It looks like fixing one (say Facebook because it's open source) will help fixing the other, and therefore getting us more options to test.

Definitely, it seems a similar issue on taking screenshots of maps.

In order to check that it's not only a problem with maps from Mapbox, I wrote a test using GMaps and I got the same results
Screenshot:

cc/ @zugaldia

[zugaldia]

@Guardiola31337 In that case, let's favor working on the open source option.

[Guardiola31337]

Follow up answer 👉 screenshot-tests-for-android#55 and PR 👉 screenshot-tests-for-android#71 which unblocks Mapbox screenshot testing issues.

At the moment, it only works when using render texture mode, i.e. TextureViews.

[tobrun]

the way forward for this issue is using MapSnapshotter from #9748,
an initial POC can be found in #10045.