Skip to content

This issue was moved to a discussion.

You can continue the conversation there. Go to discussion →

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Android App? #189

Closed
RaghavRao opened this issue Dec 1, 2018 · 47 comments
Closed

Android App? #189

RaghavRao opened this issue Dec 1, 2018 · 47 comments
Labels
enhancement New feature or request

Comments

@RaghavRao
Copy link

There seems to be an Android client for Synergy floating around (https://github.com/symless/synergy-android-7). I tried connecting to Barrier (and Synergy 1.10 Pro before I switched to Barrier) and I get this:
[2018-12-01T07:18:05] NOTE: accepted client connection
but nothing more.

This is probably too time-intensive to incorporate but would be nice to have if it's easy to integrate into the project linked above.

Thanks for making an awesome product!
@AdrianKoshka AdrianKoshka added the enhancement New feature or request label Dec 12, 2018
@BitFros7y
Copy link

BitFros7y commented Apr 5, 2019 •
edited
Loading

Any progress on this? It would be very useful if you are developing and/or testing on live android device...

@AdrianKoshka
Copy link

If somebody wants to step up and contribute, I'm more than happy to have the Android support, but as it stands, we don't really have enough developers to justify this.

@RaghavRao
Copy link
Author

I'm willing to take a crack at it, if you can give me pointers.

@AdrianKoshka
Copy link

AdrianKoshka commented Apr 9, 2019 via email •
edited
Loading

@dadosch
Copy link

dadosch commented Jun 24, 2019

I built the app locally (had to do some updates to gradle etc). I get the following:

2019-06-24 15:11:32.497 2134-15547/org.synergy W/System.err: org.synergy.io.msgs.InvalidMessageException
2019-06-24 15:11:32.498 2134-15547/org.synergy W/System.err:     at org.synergy.io.msgs.HelloMessage.<init>(HelloMessage.java:60)
2019-06-24 15:11:32.499 2134-15547/org.synergy W/System.err:     at org.synergy.client.Client.handleHello(Client.java:195)
2019-06-24 15:11:32.500 2134-15547/org.synergy W/System.err:     at org.synergy.client.Client.access$300(Client.java:48)
2019-06-24 15:11:32.501 2134-15547/org.synergy W/System.err:     at org.synergy.client.Client$4.run(Client.java:237)
2019-06-24 15:11:32.501 2134-15547/org.synergy W/System.err:     at org.synergy.base.EventQueue.dispatchEvent(EventQueue.java:135)
2019-06-24 15:11:32.503 2134-15547/org.synergy W/System.err:     at org.synergy.Synergy$MainLoopThread.run(Synergy.java:64)
2019-06-24 15:11:32.504 2134-15547/org.synergy I/Synergy: Event grabbed:org.synergy.base.Log:153
2019-06-24 15:11:32.506 2134-15547/org.synergy I/Synergy: Event:SOCKET_CONNECTED:org.synergy.net.TCPSocket@a2cb1a8:org.synergy.base.Log:153
2019-06-24 15:11:32.509 2134-15547/org.synergy I/Synergy: Event grabbed:org.synergy.base.Log:153
2019-06-24 15:11:32.510 2134-15547/org.synergy I/Synergy: Event:STREAM_INPUT_READY:org.synergy.net.TCPSocket@a2cb1a8:org.synergy.base.Log:153
2019-06-24 15:11:33.519 2134-15547/org.synergy W/System.err: org.synergy.io.msgs.InvalidMessageException
2019-06-24 15:11:33.522 2134-15547/org.synergy W/System.err:     at org.synergy.io.msgs.HelloMessage.<init>(HelloMessage.java:60)
2019-06-24 15:11:33.523 2134-15547/org.synergy W/System.err:     at org.synergy.client.Client.handleHello(Client.java:195)
2019-06-24 15:11:33.524 2134-15547/org.synergy W/System.err:     at org.synergy.client.Client.access$300(Client.java:48)
2019-06-24 15:11:33.525 2134-15547/org.synergy W/System.err:     at org.synergy.client.Client$4.run(Client.java:237)
2019-06-24 15:11:33.526 2134-15547/org.synergy W/System.err:     at org.synergy.base.EventQueue.dispatchEvent(EventQueue.java:135)
2019-06-24 15:11:33.527 2134-15547/org.synergy W/System.err:     at org.synergy.Synergy$MainLoopThread.run(Synergy.java:64)
2019-06-24 15:11:33.529 2134-15547/org.synergy I/Synergy: Event grabbed:org.synergy.base.Log:153
2019-06-24 15:11:33.530 2134-15547/org.synergy I/Synergy: Event:SOCKET_CONNECTED:org.synergy.net.TCPSocket@a1d23a7:org.synergy.base.Log:153
2019-06-24 15:11:33.534 2134-15547/org.synergy I/Synergy: Event grabbed:org.synergy.base.Log:153
2019-06-24 15:11:33.536 2134-15547/org.synergy I/Synergy: Event:STREAM_INPUT_READY:org.synergy.net.TCPSocket@a1d23a7:org.synergy.base.Log:153
2019-06-24 15:11:34.540 2134-15547/org.synergy W/System.err: org.synergy.io.msgs.InvalidMessageException
2019-06-24 15:11:34.545 2134-15547/org.synergy W/System.err:     at org.synergy.io.msgs.HelloMessage.<init>(HelloMessage.java:60)
2019-06-24 15:11:34.546 2134-15547/org.synergy W/System.err:     at org.synergy.client.Client.handleHello(Client.java:195)
2019-06-24 15:11:34.546 2134-15547/org.synergy W/System.err:     at org.synergy.client.Client.access$300(Client.java:48)
2019-06-24 15:11:34.547 2134-15547/org.synergy W/System.err:     at org.synergy.client.Client$4.run(Client.java:237)
2019-06-24 15:11:34.548 2134-15547/org.synergy W/System.err:     at org.synergy.base.EventQueue.dispatchEvent(EventQueue.java:135)
2019-06-24 15:11:34.548 2134-15547/org.synergy W/System.err:     at org.synergy.Synergy$MainLoopThread.run(Synergy.java:64)
2019-06-24 15:11:34.550 2134-15547/org.synergy I/Synergy: Event grabbed:org.synergy.base.Log:153
2019-06-24 15:11:34.551 2134-15547/org.synergy I/Synergy: Event:SOCKET_CONNECTED:org.synergy.net.TCPSocket@c5800f2:org.synergy.base.Log:153
2019-06-24 15:11:34.553 2134-15547/org.synergy I/Synergy: Event grabbed:org.synergy.base.Log:153
2019-06-24 15:11:34.554 2134-15547/org.synergy I/Synergy: Event:STREAM_INPUT_READY:org.synergy.net.TCPSocket@c5800f2:org.synergy.base.Log:153
2019-06-24 15:11:34.557 2134-15547/org.synergy W/System.err: org.synergy.io.msgs.InvalidMessageException
2019-06-24 15:11:34.560 2134-15547/org.synergy W/System.err:     at org.synergy.io.msgs.HelloMessage.<init>(HelloMessage.java:60)
2019-06-24 15:11:34.563 2134-15547/org.synergy W/System.err:     at org.synergy.client.Client.handleHello(Client.java:195)
2019-06-24 15:11:34.565 2134-15547/org.synergy W/System.err:     at org.synergy.client.Client.access$300(Client.java:48)
2019-06-24 15:11:34.566 2134-15547/org.synergy W/System.err:     at org.synergy.client.Client$4.run(Client.java:237)
2019-06-24 15:11:34.567 2134-15547/org.synergy W/System.err:     at org.synergy.base.EventQueue.dispatchEvent(EventQueue.java:135)
2019-06-24 15:11:34.568 2134-15547/org.synergy W/System.err:     at org.synergy.Synergy$MainLoopThread.run(Synergy.java:64)
2019-06-24 15:12:01.870 2134-2134/org.synergy W/org.synergy: type=1400 audit(0.0:27): avc: denied { write } for name="uinput" dev="tmpfs" ino=12413 scontext=u:r:untrusted_app:s0:c512,c768 tcontext=u:object_r:uhid_device:s0 tclass=chr_file permissive=0
2019-06-24 15:12:01.906 2134-15547/org.synergy I/Synergy: Event grabbed:org.synergy.base.Log:153
2019-06-24 15:12:01.907 2134-15547/org.synergy I/Synergy: Event:SOCKET_CONNECTED:org.synergy.net.TCPSocket@39adc6:org.synergy.base.Log:153
2019-06-24 15:12:01.908 2134-15547/org.synergy I/Synergy: Event grabbed:org.synergy.base.Log:153
2019-06-24 15:12:01.909 2134-15547/org.synergy I/Synergy: Event:STREAM_INPUT_READY:org.synergy.net.TCPSocket@39adc6:org.synergy.base.Log:153

@nathaneltitane
Copy link

@Genome36 might be able to help

@RaghavRao
Copy link
Author

Only tangentially related, but I got it to build on Linux on Dex on Samsung, so it works in a Linux Container (not the main phone though).

I have also read that you need root to control an actual pointer? But i have seen "Wireless Mouse" applications that send it over wifi so I know it's possible.

@mirh
Copy link

mirh commented Sep 24, 2019

You don't need root.

@RaghavRao
Copy link
Author

RaghavRao commented Sep 24, 2019 via email

@dadosch
Copy link

dadosch commented Sep 24, 2019 •
edited
Loading

It should be here https://github.com/dadosch/synergy-android-7/

Please note that I think (but I don't remember exactly) that I was able to fix the message I wrote earlier (InvalidMessageException) with dadosch/synergy-android-7@2f05f23

However, when I tested the app connecting worked fine, but the mouse emulation part didn't (I think it had something to do with this file https://github.com/dadosch/synergy-android-7/blob/master/app/src/main/jni/synergy-jni.c not working properly)

(But the mouse not working might also be because of my LineageOS not running some commands properly)

@dadosch
Copy link

dadosch commented Sep 24, 2019 •
edited
Loading

For testing, you need to run Barrier without "--enable-crypto".

Current state: No mouse shown on android phone, but events should be successfully received.
Keypress events succesfully received, but no keypress done. (I think you need to implement a soft keyboard here)

@dadosch
Copy link

dadosch commented Sep 25, 2019 •
edited
Loading

scrcpy https://github.com/Genymobile/scrcpy/blob/master/DEVELOP.md#input-events-injection also does injecting key and mouse events. Note that they handle special keys like umlauts or fancy French characters quite well. (But I don't know wether it works without adb, but no root should be required) EDIT: I think the key part IS that it works via adb so it has the permissions to install the server with appropriate permissions. However, no cursor is shown on the device when controlled via scrcpy.

If it would be possible to use their way to emulate keypresses (even if that would mean a slightly more complex installation process than simply installing a APK), I think I would prefer their way over a root-only route.

Steps needed: (IMHO)

  • find out how scrcpy can inject key/touch events
  • their "server" (that part that runs on the android phone) needs to be forked/modified to understand the barrier/synergy protocol
  • a script that a user can run easily from their computer in order to get the server running (via ADB, possibly reconnect adb over tcp so USB is only needed on every reconnect) on their phone which receives the key/touch events.
  • find out how to show a proper cursor with the simulated touch events

@mirh I tried out that app . While moving the "cursor" (basically a large image of a curser) with the python script worked, clicking didn't work.

A different approach would be to use a bluetooth adapter on the computer to act as HID device (a bit like https://github.com/xenogenesi/hidclient, but only used when the mouse would be leaving the computer's screen), but then you would need to configure the resolution of the phone screen on the computer I suppose (Since HID mice send only relative data, don't they?) This would be universal and could be used not only for phones.

@mirh
Copy link

mirh commented Sep 26, 2019

While moving the "cursor" (basically a large image of a curser) with the python script worked, clicking didn't work.

AFAIU the accessibility service can only interact with "accessible objects".
And I guess like it would have its security sense to just allow so much....

Root+adb (which can also wireless btw) seem the two most proper routes

@mister2d
Copy link

Is there any traction on an Android application. It can be REALLY useful when I use Samsung Dex.

@shymega
Copy link

shymega commented May 19, 2020

Not officially. We have other pressing bugs and issues to catch up on, before even Android app development can be considered.. apologies.

@Kanteran
Copy link

I will leave this here as reference: scrcpy

@shymega shymega mentioned this issue May 24, 2020
Closed
@ThibautVerron
Copy link

DeskDock can be used as a bridge between barrier and android.

@baconicsynergy
Copy link

Just chiming in, I would absolutely love Android support. Once other pressing matters have been attended to and settled, perhaps this could be a possibility 🦖 Love the work you all do

@haivala
Copy link

haivala commented Apr 21, 2021

Just thought of this. Would love to use barrier client on my android tv to connect my linux barrier server. I already use the client in gpu passthrough windows and work laptop

@dondell
Copy link

dondell commented Apr 22, 2021

I'm an Android Developer. Can someone point me how can I contribute here? I am using this app and I want to support the android development.

@dadosch
Copy link

dadosch commented May 23, 2021 •
edited
Loading

KDE connect for android added mouse input recently, it works through the accessibility services. I think this can be a starting point: https://github.com/KDE/kdeconnect-android/tree/master/src/org/kde/kdeconnect/Plugins/MouseReceiverPlugin

For keyboard input, they use a softkeyboard which has to be selected in order to work.

But after all, no root is required with those methods so this would be really great.

@Kanteran
Copy link

Kanteran commented May 23, 2021 via email

@jswhisperer
Copy link

came here to say this exact thing #189
I reckon everything but the networking is done? but of course I'm not an expert.
keyboard, clipboard bi-directional, mouse, bi-directional, media controls all work

@jswhisperer
Copy link

might have made some progress #1184

@dadosch
Copy link

dadosch commented Jun 3, 2021

came here to say this exact thing #189
I reckon everything but the networking is done? but of course I'm not an expert.
keyboard, clipboard bi-directional, mouse, bi-directional, media controls all work

Networking is not the issue, the barrier protocol is nearly identical to synergy. Two years ago, I did the necessary change here: https://github.com/dadosch/synergy-android-7

It was just a small adjustement in the packets. However, encryption is not implemented.

As you can see, I removed the old injection code a few days ago. What is now needed is to implement the cursor movement and key input using the accessibility service, just like kdeconnect did. Then it would be possible to control an android phone using barrier, but not controlling a pc running barrier using the phone. (But I think the main use case is the first one)

@jswhisperer
Copy link

Ah ok so forwarding the x11 over ssh to something like :DISPLAY0 is irrelevant, you've basically taken care of the data transfer with a small protocol shim, as I understand.

The work is in implementing the accessibility features, allowing cursors movement.

Agreed I only want to use my desktop mouse on computer, vice versa can use vnc viewer.

Security is important but I think it's fairly safe to assume a LAN will not allow port ranges outside 80 etc to access from outside by default, would be difficult to hack.

@jswhisperer
Copy link

Thanks for the response! I wish I knew more, not that this is mission critical, just a fun idea.

@jayesh83
Copy link

If somebody wants to step up and contribute, I'm more than happy to have the Android support, but as it stands, we don't really have enough developers to justify this.

Willing to work on the app for barrier

@jayesh83
Copy link

came here to say this exact thing #189
I reckon everything but the networking is done? but of course I'm not an expert.
keyboard, clipboard bi-directional, mouse, bi-directional, media controls all work

Networking is not the issue, the barrier protocol is nearly identical to synergy. Two years ago, I did the necessary change here: https://github.com/dadosch/synergy-android-7

It was just a small adjustement in the packets. However, encryption is not implemented.

As you can see, I removed the old injection code a few days ago. What is now needed is to implement the cursor movement and key input using the accessibility service, just like kdeconnect did. Then it would be possible to control an android phone using barrier, but not controlling a pc running barrier using the phone. (But I think the main use case is the first one)

I have worked on a project in which I did touch injections using Accessibility service in android. Willing to have a discussion on building app for barrier to share mouse and keyboard

@jswhisperer
Copy link

I don't have any java experience or I'd jump on board with you @jayesh83
maybe @dadosch has some starting points for you if you have the time? :P

@RaghavRao
Copy link
Author

RaghavRao commented Jun 12, 2021 via email

@jayesh83
Copy link

jayesh83 commented Jun 12, 2021 •
edited
Loading

@gregpalaci @dadosch @RaghavRao Alright guys then. No issues let's give it a shot. I'm having my final year college exam from 15 till 16. Two days. After that I can devote myself to this project.

@dadosch
Copy link

dadosch commented Jun 12, 2021

Nice :) I'm quite busy at the moment, but will contribute whenever I find the time. I see two possibilities: Use the base from synergy (https://github.com/dadosch/synergy-android-7), which would mean we can focus on functionality, but there is a ton of legacy and even unused code in there. Also most stuff hasn't been touched in years and Android has changed quite a lot since then. The alternative would be to start from scratch with everything and at most only copy single files (if the license matches of course)

@jayesh83
Copy link

Yup you're right @dadosch We'll start from scratch as android has changed alot from then. though we'll use any code required from synergy-android-7. but we must start from scratch.

@shymega
Copy link

shymega commented Jun 13, 2021 via email

@jayesh83
Copy link

Sure

@shymega shymega mentioned this issue Jun 13, 2021
Closed
@jayesh83
Copy link

Somebody create empty android studio project and push here so that any one willing to work can clone and starting working on their on part. Just to initiate barrier_android work I had create an empty android studio project and tried pushing on opening PR to my fork but couldn't do "can't push no history match" some error was showing (refer ss for error) so anyone have idea about initiating android work by creating new branch here would help.

Screenshot from 2021-06-17 23-29-52

@jayesh83
Copy link

Pointers to work for barrier android
#1. First lets work on mouse clicks and movement

  1. Show mouse pointer in android when mouse/trackpad moves
  2. Inject mouse click event in phone.

::Implementation::

  1. We have to know when to show mouse pointer (currently in desktop user moves the mouse out of screen edge and mouse pointer goes to another pc) in same way we need to know when mouse pointer leaves desktop/phone edge and comes to android.
    Then listen for incoming pointer movement msgs from pc to phone and draw each msg in android client to depict mouse movement. msg could be of format (move x y) we'll parse that in android and draw pointer or move pointer from last location to this new location.
  2. To perform touch/click injection we need to use Accessibility service in android. As android's security has grown immensely I have done complete RnD on remote touch injection & got to know none of the methods works except Accessibility Service.

As I have worked on same kind of remote touch injection project I can work on touch injection part. Once I have a msg (ACTION_DOWN x y) I can perform touch leave this part on me. but first we need a way to get this message (click message) from server.

@dadosch
Copy link

dadosch commented Jun 17, 2021

@jayesh83 I think it would make much sense to do the android development in a separate repo, with issues and a discussion tab. This way, we can develop without much friction, but it can still be merged here as a separate branch later if needed. I created https://github.com/dadosch/barrier-android, so feel free to open a PR towards that :)

@jayesh83
Copy link

Oh yes it would absolutely make sense. We'll initiate work here now

@capellan2000
Copy link

DeskDock can be used as a bridge between barrier and android.

How is this possible? I have not found any information or tutorial that explains this.

@WillForan
Copy link

WillForan commented Jun 21, 2021 •
edited
Loading

@capellan2000 in DeskDockServer_1.2.2.jar's res/strings

50:SETTINGS_SYNERGY_DESCRIPTION_KEY: "You can integrate DeskDock into an existing Synergy environment. This effectively disables DeskDock's own system input hooks and arrangement implementation. The DeskDockServer application acts as a broker between the DeskDock app on the Android device and the Synergy server.";
51:SETTINGS_SYNERGY_ENABLED_KEY: "Use Synergy/Barrier Integration";

@capellan2000
Copy link

Now I understand. Thanks for answering this request!

@piovanId
Copy link

piovanId commented Aug 18, 2021 •
edited
Loading

DeskDock can be used as a bridge between barrier and android.

While this is true, this solution is still pretty inconsistent, since the mouse almost never shows on my screen while using deskdock (almost never since it randomly appeared once, only for disappearing after I rebooted my PC) and it's really unusable without the cursor

@piovanId
Copy link

so, I was fidgeting with the app after some good time I had it installed and it didn't show the cursor, and I found out that the app or android automatically switches the accessibility settings for the app that regard the cursor off, and if you manually turn it on it works! hope this can be helpful

@CrisSherban
Copy link

so, I was fidgeting with the app after some good time I had it installed and it didn't show the cursor, and I found out that the app or android automatically switches the accessibility settings for the app that regard the cursor off, and if you manually turn it on it works! hope this can be helpful

How did you manage to turn it on manually? I was also able to connect with the synergy app and use my android tablet as a client, but the cursor does not show even if the log on the server tells that the cursor has moved to the client's screen.

@shymega
Copy link

shymega commented Oct 3, 2021

Moving this is to a discussion, better suited to IMHO. I will also create a tracker issue for progress - recommendation is to use discussion for design, and tracker issue for development progress.

@debauchee debauchee locked and limited conversation to collaborators Oct 3, 2021
@shymega shymega closed this as completed Oct 3, 2021

This issue was moved to a discussion.

You can continue the conversation there. Go to discussion →

Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests