Remote Control API

[CWBudde]

We're currently using scrcpy to perform a series of QA tests. We use it to stream the screen content in one direction and user inputs in the other direction. While this works well, the tasks are quite repetive and so we started writing some automations for this.

Something like: If (screen shows x) then click(x, y).

To do so we have written an external software that copies the screen content from scrcpy and eventually send events like mouse clicks to it. While this works more or less sufficient for our needs, it still lacks some features. For example, we must keep the scrcpy window shown (i.e. not minimized) and the user must keep the mouse still while the automation is in progress. Also, we haven't found a way to input multi touch gestures.

It would be very handy if the scrcpy comes as library (or with a library) or exports that would allow a programmatic extraction of data and supplying input data to it.

Any thoughts about it? It shouldn't be that complicated and I'm willing to help, but at the moment I have no idea how much work this will be in the end or whether this is already possible in some way.

[rom1v]

Also, we haven't found a way to input multi touch gestures.

This is not implemented yet (see #59). The main reason is that I have currently no multi-touch computer to develop it. I will probably try to get one some day, but I miss time.

It would be very handy if the scrcpy comes as library (or with a library) or exports that would allow a programmatic extraction of data and supplying input data to it.

Yes, such a split would be better. But I did not think about the details yet.

Currently, it uses SDL for everything, but if it's a lib (that could for example be used in a Qt app), it should not depend on SDL anymore (only the app using the lib would use SDL, or Qt, or whatever). But the lib needs to create threads, so it will need a threading library (SDL abstracts that, Qt abstracts that, but if it's in a lib that does not depend on SDL or Qt, then we need something)…

EDIT: also, one drawback, if it's a lib, the interface need to be somewhat stable, so it will be more difficult to add features/change things.

To do so we have written an external software that copies the screen content from scrcpy and eventually send events like mouse clicks to it. While this works more or less sufficient for our needs, it still lacks some features. For example, we must keep the scrcpy window shown (i.e. not minimized) and the user must keep the mouse still while the automation is in progress.

For now, you could fork scrcpy locally, and call controller_push_event() manually to push the key/mouse events you want. That's not ideal, but currently, it's not a lib.

[Gokkulnath]

Hi @rom1v

I am also in great need of this feature. Is it implemented already ? it would be a great addition to automate Testing of Android apps.