Support for Coded PHY (Bluetooth 5.0 feature) when scanning.

[pvvx]

• bleak version:
  all
• Python version:
  all
• Operating System:
  Linux, Android
• BlueZ version (bluetoothctl -v) in case of Linux:
  bluetoothctl: 5.66

Description

There is no advertising reception and the ability to connect to BLE LongRange (Coded PHY).

What I Did

Bugs in Android and Linux with scanning with different PHY format have long been fixed.
The specification 'Bluetooth 5.0' since 2016 opens the work of BLE advertising in two PHY modes - LE Coded or 1M on the main channels for extended advertising.

home-assistant/core#88277 (comment)

Logs

Switch the adapter to work with CodedPHY:
hcitool -i hci0 cmd 08 31 03 04 04

Further work in 'Bleak' is limited to 'Coded PHY' format only.

Information about adapter capabilities:

# btmgmt phy
Supported phys: BR1M1SLOT BR1M3SLOT BR1M5SLOT EDR2M1SLOT EDR2M3SLOT EDR2M5SLOT EDR3M1SLOT EDR3M3SLOT EDR3M5SLOT LE1MTX LE1MRX LE2MTX LE2MRX LECODEDTX LECODEDRX
Configurable phys: BR1M3SLOT BR1M5SLOT EDR2M1SLOT EDR2M3SLOT EDR2M5SLOT EDR3M1SLOT EDR3M3SLOT EDR3M5SLOT LE2MTX LE2MRX LECODEDTX LECODEDRX
Selected phys: BR1M1SLOT BR1M3SLOT BR1M5SLOT EDR2M1SLOT EDR2M3SLOT EDR2M5SLOT EDR3M1SLOT EDR3M3SLOT EDR3M5SLOT LECODEDTX LECODEDRX

# hciconfig -a
hci0:   Type: Primary  Bus: USB
        BD Address: 8C:88:2B:20:8B:42  ACL MTU: 1021:6  SCO MTU: 255:12
        UP RUNNING
        RX bytes:2020 acl:0 sco:0 events:178 errors:0
        TX bytes:23965 acl:0 sco:0 commands:178 errors:0
        Features: 0xff 0xff 0xff 0xfe 0xdb 0xfd 0x7b 0x87
        Packet type: DM1 DM3 DM5 DH1 DH3 DH5 HV1 HV2 HV3
        Link policy: RSWITCH HOLD SNIFF PARK
        Link mode: SLAVE ACCEPT
        Name: 'nanopineoplus2'
        Class: 0x000000
        Service Classes: Unspecified
        Device Class: Miscellaneous,
        HCI Version: 5.1 (0xa)  Revision: 0x999
        LMP Version: 5.1 (0xa)  Subversion: 0x646b
        Manufacturer: Realtek Semiconductor Corporation (93)

The adapter's capabilities allow it to accept 1M and Coded PHY.

What are the commands to make 'Bleak' work according to the Bluetooth 5.0 specification?

[dlech]

There are not any operating systems APIs to request using coded phy that I am aware of.

[pvvx]

https://developer.android.com/reference/android/bluetooth/BluetoothAdapter#isLeCodedPhySupported() ...
https://developer.android.com/reference/android/bluetooth/BluetoothDevice#PHY_LE_CODED
https://developer.android.com/reference/android/bluetooth/le/ScanSettings.Builder#setPhy(int)

[dlech]

What about Windows, Mac and Linux?

[pvvx]

New versions of bluez work.
One method is to switch the USB-BT5.0+ adapter to Coded PHY mode.

For example, you can take:
Low-cost Realtek RTL8761 BT adapter (current Linux firmware on RTL8761 supports BT5.1)

USB ID 0bda:8771 Realtek Semiconductor Corp. Bluetooth radio

For Home Assistan to work with 'LE Long Range' in "Bluetooth" integration based on "bleak", it is advisable to install the latest version of bluez and add it to /lib/systemd/system/bluetooth.service
ExecStart=/usr/local/libexec/bluetooth/bluetoothd --experimental

Before launching Home Assistant, you need to switch the USB-BT5.0+ adapter to work in CodedPHY:
hcitool -i hci1 cmd 08 31 03 04 04

# btmon -i 1
Bluetooth monitor ver 5.66
= Note: Linux version 5.15.89-sunxi64 (aarch64)                                                                0.654717
= Note: Bluetooth subsystem version 2.22                                                                       0.654746
= New Index: 8C:88:2B:00:E7:EA (Primary,USB,hci1)                                                              0.654754
= Open Index: 8C:88:2B:00:E7:EA                                                                                0.654757
= Index Info: 8C:88:2B:00:E7:EA (Realtek Semiconductor Corporation)                                            0.654761
= New Index: 00:1A:7D:DA:71:13 (Primary,USB,hci0)                                                              0.654764
= Open Index: 00:1A:7D:DA:71:13                                                                                0.654766
= Index Info: 00:1A:7D:DA:71:13 (Cambridge Silicon Radio)                                                      0.654768
@ RAW Open: hass (privileged) version 2.22                                                            {0x0002} 0.654777
@ MGMT Open: bluetoothd (privileged) version 1.21                                                     {0x0001} 0.654786
> HCI Event: LE Meta Event (0x3e) plen 59                                                                   #1 2.325655
      LE Extended Advertising Report (0x0d)
        Num reports: 1
        Entry 0
          Event type: 0x0001
            Props: 0x0001
              Connectable
            Data status: Complete
          Address type: Public (0x00)
          Address: A4:C1:38:0B:5E:ED (Telink Semiconductor (Taipei) Co. Ltd.)
          Primary PHY: LE Coded
          Secondary PHY: LE Coded
          SID: 0x00
          TX power: 127 dBm
          RSSI: -56 dBm (0xc8)
          Periodic advertising interval: 0.00 msec (0x0000)
          Direct address type: Public (0x00)
          Direct address: 00:00:00:00:00:00 (OUI 00-00-00)
          Data length: 0x21
        02 01 06 11 16 1c 18 02 00 8d 23 02 8a 07 03 03  ..........#.....
        e3 0e 02 01 56 0b 09 41 54 43 5f 30 42 35 45 45  ....V..ATC_0B5EE
        44                                               D
        Flags: 0x06
          LE General Discoverable Mode
          BR/EDR Not Supported
        Service Data: User Data (0x181c)
          Data: 02008d23028a070303e30e020156
        Name (complete): ATC_0B5EED
@ MGMT Event: Device Found (0x0012) plen 47                                                           {0x0001} 2.325754
        LE Address: A4:C1:38:0B:5E:ED (Telink Semiconductor (Taipei) Co. Ltd.)
        RSSI: -56 dBm (0xc8)
        Flags: 0x00000000
        Data length: 33
        Flags: 0x06
          LE General Discoverable Mode
          BR/EDR Not Supported
        Service Data: User Data (0x181c)
          Data: 02008d23028a070303e30e020156
        Name (complete): ATC_0B5EED
> HCI Event: LE Meta Event (0x3e) plen 59                                                                  #2 12.826263
      LE Extended Advertising Report (0x0d)
        Num reports: 1
        Entry 0
          Event type: 0x0001
            Props: 0x0001
              Connectable
            Data status: Complete
          Address type: Public (0x00)
          Address: A4:C1:38:0B:5E:ED (Telink Semiconductor (Taipei) Co. Ltd.)
          Primary PHY: LE Coded
          Secondary PHY: LE Coded
          SID: 0x00
          TX power: 127 dBm
          RSSI: -54 dBm (0xca)
          Periodic advertising interval: 0.00 msec (0x0000)
          Direct address type: Public (0x00)
          Direct address: 00:00:00:00:00:00 (OUI 00-00-00)
          Data length: 0x21
        02 01 06 11 16 1c 18 02 00 8e 23 02 8a 07 03 03  ..........#.....
        e3 0e 02 01 56 0b 09 41 54 43 5f 30 42 35 45 45  ....V..ATC_0B5EE
        44                                               D
        Flags: 0x06
          LE General Discoverable Mode
          BR/EDR Not Supported
        Service Data: User Data (0x181c)
          Data: 02008e23028a070303e30e020156
        Name (complete): ATC_0B5EED
@ MGMT Event: Device Found (0x0012) plen 47                                                          {0x0001} 12.826337
        LE Address: A4:C1:38:0B:5E:ED (Telink Semiconductor (Taipei) Co. Ltd.)
        RSSI: -54 dBm (0xca)
        Flags: 0x00000000
        Data length: 33
        Flags: 0x06
          LE General Discoverable Mode
          BR/EDR Not Supported
        Service Data: User Data (0x181c)
          Data: 02008e23028a070303e30e020156
        Name (complete): ATC_0B5EED

To get a BLE 5.0 scan on such an adapter (1M and Сoded PHY), switching the scan to different PHY variants is required.

---

Mac doesn't interest me at all because of the many limitations in BLE. There are Bluetooth SIG standards, but Macs have their own :)

BLE scanning does not work well on Windows initially...

[pvvx]

http://www.bluez.org/
Bluetooth Management API
/doc/mgmt-api.txt

Set PHY Configuration Command
=============================

	Command Code:		0x0045
	Controller Index:	<controller id>
	Command Parameters: 	Selected_PHYs (4 Octet)
	Return Parameters:

	This command is used to set the default PHY to the controller.

	This will be stored and used for all the subsequent scanning
	and connection initiation.

	The list of supported PHYs can be retrieved via the
	Get PHY Configuration command. Selecting unsupported or
	deselecting default PHYs will result in an Invalid Parameter
	error.

	This can be called at any point to change the Selected PHYs.

	Refer Get PHY Configuration command for PHYs parameter.

	This command generates a Command Complete event on success
	or a Command Status event on failure.

	Possible errors:	Invalid Parameters
				Invalid Index

[DienoX]

@pvvx
Thank you for your efforts on LE CODED.
I hope that it will be possible to properly implement Bluetooth 5 in bleak. Although in support for Android and Linux.
I'm keeping my fingers crossed for that.

So far I am very happy with the implementation of your software when using it with Linux.

[DienoX]

Why was it opened?

[dlech]

This is a feature that has been requested by multiple users. I don't know if it is currently technically possible, but maybe some day.

[pvvx]

Starting with the bluetooth 5.0 standard from 2016, it is required to set secondary and primary PHY for scanning and advertising.
The current Bleak does not have support for Bluetooth 5.0 yet.
https://techcrunch.com/2023/05/31/hubble-network-is-building-out-a-space-based-bluetooth-network-to-connect-over-billion-devices/

[pvvx]

How are things going with the support of "Periodic Advertising with Responses (PAwR)", "Encrypted Advertising Data" and so on?

[Sorien]

any news on this?

[inteltryb93]

Seriously, how hasn't this been tackled yet? It's like waiting for the next ice age! We've got all the specs laid out, bugs squashed left and right, and yet, we're still twiddling our thumbs here without BLE LongRange support in Bleak? Come on, folks! Android and Linux have sorted their end ages ago, and Bluetooth 5.0 has been out since the dinosaurs roamed the Earth! We've got the adapter practically waving its hands saying, "I'm ready, use me!" So, what gives? Can we please get some action on this?

[pvvx]

Bleak believes that there is no need for people to use new smartphones and USB3.0. Use Nokia 3310 and USB 1.1. :)

[wenstarner]

Hi @dlech, are there any plans to implement Bluetooth 5.0+ in bleak? I'm trying to investigate support for LE/long-range.

Just wondering if you know of any updates about this or might know how to find out, if BT 5.0+ is indeed now working in bluez?

Cheers

[DienoX]

I'm curious about this too

[dlech]

I don't think anything much has changed. Windows and Android look like they might provide some APIs for enabling Coded PHY explicitly, but Mac and BlueZ don't (BlueZ can handle it on the low-level management interface to the kernel, but we don't use that, we use the D-bus API which doesn't have any APIs for this).

[pvvx]

So Bleak is almost ten years out of date, just like Linux?

[wenstarner]

BlueZ can handle it on the low-level management interface to the kernel, but we don't use that, we use the D-bus API which doesn't have any APIs for this).

That might be the practical outcome, but from what @dlech has said, it seems like the challenge is to make this possible in D-bus first. If so, feedback needs to be provided to the D-bus team. @pvvx the productive way forward is to isolate the issue. It seems like the target for feedback should be the D-bus team.

@dlech, would you say that is correct? Also, aside from the possibility of this being made possible via D-bus, is there any prospect of having bleak using BlueZ' own kernel interface for this?

[pvvx]

Are you sure Bluez supports connection with LE Long Range devices?
Receiving BLE ads in LE Long Range does not mean that the connection will be processed. Bluez Core version is 4.2. This is the version from ten years ago.

LE Long Range advertising is received by the adapter itself, independently of Bluez. The only difference is in the initialization of the adapter - specifying the modes in which it operates.
And the connection in LR requires a different approach than it is currently implemented. Bleak simply resets the adapter to legacy mode.

[pvvx]

is there any prospect of having bleak using BlueZ' own kernel interface for this?

Everyone has long switched to alternative APIs.

[DienoX]

What are the alternative interfaces?

[pvvx]

What are the alternative interfaces?

For simple options - opening a socket to work directly with the adapter via HCI.

Have to use alternative bluetooth stack. There are already several of them.
And this is inevitable, since Bluez uses the most non-optimal algorithm for connecting to BLE devices.

---

To work in Bluez requires rewriting all existing Bluez command formats used in the API, adding primary and secondary PHY types. No one will do this.

[DienoX]

What do you recommend instead of Bluez?

[wenstarner]

What do you recommend instead of Bluez?

Yes, @pvvx, what do you recommend? I think the priority in order to be productive is to make a suggestion that the bleak team could consider. What you're saying is an interesting description of the problem—but in order to solve it, what would you recommend to the bleak team?

You mention alternative bluetooth stacks... do you recommend any?

[pvvx]

I don't use Linux much because it is very outdated and I can't recommend anything optimal.

For BLE in Linux, you need to start by reworking the BT adapter drivers in the kernel...

[wenstarner]

Okay, well that doesn't seem to help the issue.

So it seems like the options are:

1. an alternative Bluetooth stack (how likely this is, who knows);
2. working directly with the kernel (very unlikely?); or
3. going via D-bus and hoping the D-bus team implements an API.

@DienoX, perhaps we can get in touch with the D-bus team to inquire.

[pvvx]

A simple example of receiving LE Long Range advertising can be made from https://github.com/frawau/aioblescan.
If you change the line in aioblescan.py

class HCI_Cmd_LE_Set_Extended_Scan_Params(HCI_Command):
…
phys=4,

https://github.com/frawau/aioblescan/blob/master/aioblescan/aioblescan.py#L1185

[DienoX]

I suggest migrating to Fluoride.
https://chromeos.dev/en/posts/androids-bluetooth-stack-fluoride-comes-to-chromeos

It has a lot of users on Android so it gets a lot of support and is well supported by other devices.

Recently Chromeos switched to it.

[wenstarner]

I suggest migrating to Fluoride. https://chromeos.dev/en/posts/androids-bluetooth-stack-fluoride-comes-to-chromeos

It has a lot of users on Android so it gets a lot of support and is well supported by other devices.

Recently Chromeos switched to it.

And it looks like it is very actively maintained.

@dlech, would something like this at all be a possibility?

[DienoX]

And it has supported Bluetooth Long Range for several years.

[dlech]

is there any prospect of having bleak using BlueZ' own kernel interface for this?

No, accessing this interface requires elevated permissions. And going behind the back of bluetoothd can cause problems if both are trying to interact with the same thing.

would something like this at all be a possibility?

It does look like Floride can run on Linux, but someone would need to package it to make it easy for people to install. Then we would also need to develop a new backend in Bleak for it. So technical possibility is high, but likelihood of someone actually doing the work, not so much.