Enable (or unblock) Web APIs for interacting with devices such as WebUSB, Web Serial & Web Bluetooth

[beriberikix]

Hello! We're in the hardware/IoT space and for our use case we need to interact with physical devices on a users machine for programming & debugging. One straightforward approach is to leverage W3C APIs for WebUSB, Web Serial & Web Bluetooth that are currently available in Chromium-based browser. As I understand it, VSCode by default blocks these APIs even if the browser supports them (see microsoft/vscode#116761.) So if 116761 is correct, the request is to allow the appropriate permissions on the iFrame.

I originally posted this #3463 but @csweichel suggested that this would be a better place to raise it :)

An alternative we're considering is creating a desktop agent like the Local Companion but using the browser APIs would be preferred for performance reasons.

/cc @0Grit @alvarowolfx

[akosyakov]

Please use gitpod-io/gitpod issues for gitpod issue, this issue tracker for OpenVSCode Server.

Also notice that fixing VS Code issues it is not is in scope of OpenVSCode Server, but only provide compatible server.

microsoft/vscode#116761 is the best place

[akosyakov]

We could try to contribute to VS Code to resolve it. Adding to prod inbox to decide.

[0Grit]

microsoft/vscode#117786

[akosyakov]

/schedule

We agreed to try to move microsoft/vscode#117786 forward.

[roboquat]

@akosyakov: Issue scheduled in the IDE team (WIP: 0)

In response to this:

/schedule

We agreed to try to move microsoft/vscode#117786 forward.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[akosyakov]

Unfortunately after team discussions we realised that we don't have capacity for upstream issues and have to focus more on bug fixes and robust deployments. I removed it from the scheduled for now.

[0Grit]

Totally get it.

[akosyakov]

VS Code supports web extensions which are running in browser web worker: https://github.com/microsoft/vscode-docs/blob/vnext/api/extension-guides/web-extensions.md
Web worker has access to serial api: https://developer.mozilla.org/en-US/docs/Web/API/WorkerNavigator/serial
For instance this extension seems to work in such way already - they simple open another browser page

Updated: won't work, see #5814 (comment)

[akosyakov]

I tend to close it, since there is a work around and it is upstream issue.

[titimoby]

I know this issue was about WebUSB but what about something linked to local companion?
Something like "USB over IP" as they say at https://shadow.tech/fr/
I use one of their "remote Windows box" and this feature allows audio, micro controllers...

[akosyakov]

I know this issue was about WebUSB but what about something linked to local companion?

The local companion can only tunnel TCP/IP. If you have tools which can do serial over TCP/IP then it should work. There are no plans currently to add serial tunnelling. You can file an issue for us to research.

[beriberikix]

VS Code supports web extensions which are running in browser web worker

Does Gitpod support the complete set of web extension functionality? That would be great! Than indeed this is a workaround

serial over TCP/IP

Actually, USB/IP is based on TCP. SLIP is a modified protocol AFAIK. Using USB/IP is something we're actively investigating. If anyone else is interested, we'd love to chat!

[0Grit]

@Yatekii

[thegecko]

VS Code supports web extensions which are running in browser web worker

Although these APIs are mostly available, I believe the request* methods for requesting access to devices are explicitly disabled in web workers and these can only be invoked in the main thread.

e.g. you can't do navigator.usb.requestDevice({ ... }); which means you can't initially connect a device. This needs to be enabled in VSCode such as microsoft/vscode#117786

However, once a device is authorised, a web worker can get access to it using the navigator.usb.getDevices() method.

[0Grit]

@vitprajzler

[akosyakov]

@thegecko Thanks for checking. I just realised that it probably won't work from web worker even if we did a request from main thread or webviews, since we run webviews and web extensions on separate origins. I would expect that browsers bind requested devices per origin. So fixing the upstream issue is only way to go.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[beriberikix]

VSCode recently introduced a way to allow WebUSB from webviews - see microsoft/vscode#152310 & https://github.com/thegecko/webusb-web-extension. Does this carry through to Gitpod as well?

[LeoDog896]

I'm trying to connect a camera to gitpod for a robot project -- having these types of APIs exposed can really help!