Add BlueZ in CHIP

[yunhanw-google]

-- Add BlueZ in Third-party for CHIP Linux device layer BLE manager integration

fixes #741

[CLAassistant]

All committers have signed the CLA.

[turon]

Please add initial GN support as well, or at least let's discuss the plan for it with @mspang.

[mspang]

Please add initial GN support as well, or at least let's discuss the plan for it with @mspang.

Can you provide some context on why bluez needs to be managed? Is it possible to have two copies of bluez running at the same time on the developer's system?

[andy31415]

Seems ok, however needs a build fix.

[rwalker-apple]

woble->chipoble?

[yunhanw-google]

Please add initial GN support as well, or at least let's discuss the plan for it with @mspang.

Can you provide some context on why bluez needs to be managed? Is it possible to have two copies of bluez running at the same time on the developer's system?

In the past several year, since bluez5.44, the bluez BLE dbus APIs starts to become ready for public experimental use, and experience continuous changes on API behavior and parameters, bugs are introduced in recent years. When engineers, including me, add fix in bluez upstream, there is only sanity and manual test to verify if this fix work, and then those fix easily break other features in lack of comprehensive unit and integration test. Different ubuntu, debian, embedded linux, chrome os I have worked before are using different version of BlueZ, for example, chrome os is still using bluez5.44, debian(tesing/experimental) are using 5.50, other different ubuntu OS are using various bluez version. Currently our qa teams proves 5.47 and 5.48 are pretty stable on BLE APIs, for example, 5.45,5.46 have several ble stability bugs we found. Therefore we lock down the commit in 5.48 in Weave. We would also plan BlueZ 5.50 or latest after enough tests have been done.

BlueZ is using bluetoothd daemon, currently in this pr, it is compiling in linux device layer, but not running. When testing it, developers usually kill the system's bluetoothd process, and run the compiled bluetoothd during testing. In weave, we have automated this test procedure for weave ble integration test via virtual btvirt, the bluez bt emulator generated from BlueZ, note os system itself is not including this btvirt tool, and these weave over BlueZ tests are running during presubmit and postsubmit CI. Usually in weave, at default, these ble tests are at default disabled. developer locally don't need run these weave over BlueZ test, when developer touched these codes, they need to run the automate test.

That's why we need to compile blueZ. It is not possible to have two BlueZ bluetoothd running at the same time in same host now.

Btw, in project cirque, we also are exploring how to bring up multiple bluetoothd across multiple docker in same host for BlueZ BLE virtualization in docker, which would need some additional changes in blueZ upstream.

[yunhanw-google]

Seems ok, however needs a build fix.

Hi, Andrei

Yes, I am fixing them!

Best wishes
Yunhan

[mspang]

Please add initial GN support as well, or at least let's discuss the plan for it with @mspang.

Can you provide some context on why bluez needs to be managed? Is it possible to have two copies of bluez running at the same time on the developer's system?

In the past several year, since bluez5.44, the bluez BLE dbus APIs starts to become ready for public experimental use, and experience continuous changes on API behavior and parameters, bugs are introduced in recent years. When engineers, including me, add fix in bluez upstream, there is only sanity and manual test to verify if this fix work, and then those fix easily break other features in lack of comprehensive unit and integration test. Different ubuntu, debian, embedded linux, chrome os I have worked before are using different version of BlueZ, for example, chrome os is still using bluez5.44, debian(tesing/experimental) are using 5.50, other different ubuntu OS are using various bluez version. Currently our qa teams proves 5.47 and 5.48 are pretty stable on BLE APIs, for example, 5.45,5.46 have several ble stability bugs we found. Therefore we lock down the commit in 5.48 in Weave. We would also plan BlueZ 5.50 or latest after enough tests have been done.

BlueZ is using bluetoothd daemon, currently in this pr, it is compiling in linux device layer, but not running. When testing it, developers usually kill the system's bluetoothd process, and run the compiled bluetoothd during testing.

Ok, that's what I feared, you can't really solve this problem by simply building a new bluez inside CHIP; running it interferes with the system install.

We might consider just providing instructions on installing a backport and just asserting that the version is OK. It sounds like Debian testing/gLinux are new enough, and CrOS we can fix ourselves..

In weave, we have automated this test procedure for weave ble integration test via virtual btvirt, the bluez bt emulator generated from BlueZ, note os system itself is not including this btvirt tool, and these weave over BlueZ tests are running during presubmit and postsubmit CI. Usually in weave, at default, these ble tests are at default disabled. developer locally don't need run these weave over BlueZ test, when developer touched these codes, they need to run the automate test.

That's why we need to compile blueZ. It is not possible to have two BlueZ bluetoothd running at the same time in same host now.

Btw, in project cirque, we also are exploring how to bring up multiple bluetoothd across multiple docker in same host for BlueZ BLE virtualization in docker, which would need some additional changes in blueZ upstream.

[yunhanw-google]

"Ok, that's what I feared, you can't really solve this problem by simply building a new bluez inside CHIP; running it interferes with the system install.

We might consider just providing instructions on installing a backport and just asserting that the version is OK. It sounds like Debian testing/gLinux are new enough, and CrOS we can fix ourselves..

It will not interfere with the system install, it just compiled it inside CHIP device layer automatically, and not install in system path. As I mentioned, if you don't run ble related test, you don't need touch it, which would not impact you. If you run it, Chip BlueZ support would automatically help you. Besides, if you compile chip in docker, and run bluetoothd in docker, it would not impact/interfere your system blueZ. The docker installation/validation/test would be recommended way in presubmit and local validation to provide the unified/controllable CHIP environment for developer/tester for the IOT world.

In github CHIP docker, there is no bluez inside, there is no interference at all from that point currently.

Debian testing/glinux's 5.50 is not working as we expect, we need to lock down specific commit.

We also added bluez in weave, from internal or external partners, we don't hear the related complains on the bluetoothd interference issue. From the experience, developer/tester bring up woble test, and run it smoothly.

[mspang]

"Ok, that's what I feared, you can't really solve this problem by simply building a new bluez inside CHIP; running it interferes with the system install.

We might consider just providing instructions on installing a backport and just asserting that the version is OK. It sounds like Debian testing/gLinux are new enough, and CrOS we can fix ourselves..

It will not interfere with the system install, it just compiled it inside CHIP device layer automatically, and not install in system path. As I mentioned, if you don't run ble related test, you don't need touch it, which would not impact you. If you run it, Chip BlueZ support would automatically help you. Besides, if you compile chip in docker, and run bluetoothd in docker, it would not impact/interfere your system blueZ. The docker installation/validation/test would be recommended way in presubmit and local validation to provide the unified/controllable CHIP environment for developer/tester for the IOT world.

In github CHIP docker, there is no bluez inside, there is no interference at all from that point currently.

Debian testing/glinux's 5.50 is not working as we expect, we need to lock down specific commit. CROS has stopped bluez support, and reject all upgrade or fix from bluez 5.44.

Does it work with bluez's HEAD? If not, we have a future problem in addition to the problems you've listed already.

We also added bluez in weave, from internal or external partners, we don't hear the related complains on the bluetoothd interference issue. From the experience, developer/tester bring up woble test, and run it smoothly.

I thought you said you had to stop the OS daemon to run the local one? That's the interference I was referring to.

[yunhanw-google]

Per offline chat with @mspang, we agree to build bluez in chip, I would try to work with @mspang for gn integration, i will open another ticket there.

Resolved

[woody-apple]

@jelderton @saurabhst @BroderickCarlin ?

[github-actions]

Size increase report for "gn_nrf-example-build"

File	Section	File	VM

Full report output

BLOAT REPORT

Files found only in the build output:
    report.csv

[github-actions]

Size increase report for "gn_linux-example-build"

File	Section	File	VM

Full report output

BLOAT REPORT

Files found only in the build output:
    report.csv

[woody-apple]

@saurabhst @jelderton ?

[github-actions]

Size increase report for "nrf-example-build"

File	Section	File	VM

Full report output

BLOAT REPORT

Files found only in the build output:
    report.csv

Comparing ./master_artifact/chip-nrf52840-lock-example.out and ./pull_artifact/chip-nrf52840-lock-example.out:

sections,vmsize,filesize
.debug_str,0,28
.debug_macro,0,6
[Unmapped],0,-2

[github-actions]

Size increase report for "linux-example-build"

File	Section	File	VM

Full report output

BLOAT REPORT

Files found only in the build output:
    report.csv

Comparing ./master_artifact/chip-standalone-demo.out and ./pull_artifact/chip-standalone-demo.out:

sections,vmsize,filesize
.debug_str,0,28
.debug_macro,0,6
[Unmapped],0,-2

[github-actions]

Size increase report for "esp32-example-build"

File	Section	File	VM

Full report output

BLOAT REPORT

Files found only in the build output:
    report.csv

Comparing ./master_artifact/chip-wifi-echo.elf and ./pull_artifact/chip-wifi-echo.elf:

sections,vmsize,filesize