Support home brew DIY sensors #548
Comments
|
In case it's interesting, I've put my Arduino IDE ESP32 code for the pulse counter in github too now: |
|
Hi! |
|
There are some important developments, we have plans to move ble_monitor to HA core, in which we will make adding sensors more flexible (e.g. via a json file). So, please allow us some time to make this switch, and we will try to take this request into account as well. |
Congratulations @Ernst79 - a significant recognition of all the hard work you've been putting into this - well done! |
|
What hardware did you use for your sensors @nikfo953? I used an ESP32 board, so I could have used the ESPHome firmware and Home Assistant plugin to send the data back from sensor to Raspberry Pi using WiFi (my ESP32 is permanently plugged in rather than battery powered). I just chose to use Bluetooth because I thought it might save a bit of energy and be an interesting challenge. Not sure my ESP32 code works very well though - the Bluetooth advertising seems to be continuous rather than turn off some of the time if there's nothing new to tell...and then work (and it being installed under the cabinets in my kitchen) got in the way of playing with it any more! |
|
@scrambledleek, my sensor is light (lux), temp and humid sensors, that is semi self-developed. It uses the Nordic nrf-chip for BLE comms. I haven't designed the sensor myself, but I have access to the protocol. |
|
Okay @nikfo953, if you're keen to try out using the ble_monitor to receive your sensor data before it gets integrated into core Home Assistant, we may be able to extend/adjust what I did for my pulse counters a little. Adding support for lux, temp and %RH readings in ... If you can get your sensor advertisement data to fit that format (described in the same file, or in my ESP32_BLEPulseCounter repository code), and turn logging on, you should then be able to see your data being received in the HA log. HA probably won't create the sensor entities for you at this stage. To handle that some changes might be needed in That change would break my pulse counting homebrew sensor, so wouldn't make it into the main branch of my git repo, but would be the quickest way for you get get the sensor entities showing up in HA as a test. |
|
Thx @scrambledleek! |
|
I started experimenting with the iBeacon protocol, as this is a standard BLE example on how to use BLE on an Espressif ESP32 in Arduino IDE. @dmamontov's work allows to send my own data (without setting up connections on wifi/zigbee/normal bluetooth) to HA on a raspberry pi just by adapting the above example and changing continuously the major/minor data. I've adapted it already not to sleep (ESP32 stopped running after some hours), just created a loop, and I've added a watchdog timer. The iBeacon protocol is very simple: a UUID (128 bit) that can be freely generated/"allocated" on https://www.uuidgenerator.net/ and two 16-bit values, so any sensor can be given a unique identifier (like IPv6) and broadcast 2 sensor values. |
|
there is altbeacon as an alternative |
|
The altbeacon is really to be used only as a beacon, it has a 20-byte identifier, but no real data fields.
I don't know if the BLE stack automatically adds a CRC value to discard wrong received packages, otherwise a CRC check would also be valuable. |
|
I'll leave it here as an idea: As BLE format I suggest to use CayenneLPP it's very simple, contains well defined data (from LwM2M standard) and used in areas required low bandwidth and low power consumption like Lora WAN and NB-IoT. HA sensors should be generated on the fly according data type of received values, this will allow to avoid the need to transfer the description of the sensors by the device. In case when user will need to exclude or disable by default some types of sensors from a specific device or globally, data type ids of this sensors can be specified in the configuration (yaml or GUI). |
|
I like the idea from @myhomeiot, so I've created a new issue where I keep track of the progress. The format shouldn't be that hard to implement, but the auto-generating of sensors will require some adjustments. See #689 |
|
I was thinking that we can also use the official
e.g. 05162A1964 would than decode to Requirements of the value byte(s) can be found here E.g. for battery, it's this
The main advantage I see is that we stick to the official BLE |
|
@Ernst79 The data will take a bit more space compared to the CayenneLPP, but it looks great and the most important that this data packages will be standard and can be interpreted by other systems. I doesn't found the usual button pressed/released characteristic, but I think it can be emulated with some existing in specification characteristic. |
|
BLE monitor 7.7.0 adds initial testing support for your own Do It Yourself (DIY) sensors. The BLE format (called If you need a different sensor type than the currently implemented ones, let me know! Important note |
0x2A6D | pressure | uint32 (4 bytes) | 0.001 | 07166D2A78091000 | 1051.0 ?
Digital signals? Switches, Keys, Relay, Counts, Times (UTC) ... ? 0x2A56 The Digital characteristic is used to expose and change the state of an IO Modules digital signals. 0x2A4D The Report characteristic is used to exchange data between a HID Device and a HID Host. How to encrypt data? |
|
Are (BT5.0) LE Advertising Extensions supported? This packet could contain up to 254 bytes payload. |
Yes. Before starting the scan, the adapter supported features are checked, and LE extended scan is launched if supported. |
Pressure sensor in Home Assistant is in Encryption of data is a good idea, I'll also add the suggested sensors you mentioned. |
|
Digital State Bits: 0x2A56 Count (24 bits) 0x2AEB |
|
Count sensors are both added in 7.9.1-beta. I'm first going to focus on "automatic" sensor creation now, as the manually modification of |
|
The modified LYWSD03MMC has the following data to transmit:
To transfer these values, at least 9 UUID identifiers are required. If transmitted in parts, this requires a reduction in the transmission period, which excludes the use in stand-alone devices and is a pollution of the radio air. Typical encryption in BLE requires 8 additional bytes (analog mija). Typical, because the BLE device already contains procedures or hardware devices to support AES CCM encryption. |
|
@pvvx I understand the issue, but I'm not sure we can put all the data in one advertisements. Xiaomi sensors normally also send only one or two properties in one advertisements and just send multiple advertisements, all with one or two properties. e.g. Battery data is almost always send in a separate data packet, often at a totally different interval (e.g. once per 10 minutes). The length of Xiaomi MiBeacon data is similar, also 1 length byte, 2 bytes UUID identifiers and 1 or more bytes for the actual data. Perhaps you can split up the data that you want to send very often, and data that you don't want to send that often (like battery voltage, battery level in %). I think you have two choices, one is to extend your own ATC format, but I guess you run into the same issue of limited length. The other is that we extend the HA_BLE format with the missing events, which should not be a problem. Regarding encryption, I agree that encryption should be an option, but I have no experience adding encryption, so it will take me a lot of time to investigate and to add this. Moreover, next month, I'm going to move, so will have limited time to do a lot of stuff. For now, I have the following priority list
Regarding the two analog voltage inputs, I have to think how we can have multiple sensors of the same type ( |
|
The largest power consumption of a BLE device occurs during transmission. |
In version 5, only one property per advertisements. According to the technical characteristics of modern SoCs with BLE and other conditions, there is such a dependence: Duplication of advertising is necessary, because. some use ESP as a receiver. This SoC has a lot of losses in BLE, especially when working with WiFi at the same time. |
|
Ok, I understand. But what do you propose to make it more efficient than? One thing I can think of is changing the I don't see an easy way to support a wide variety of sensor types without the need of an UUID to identify which data it is. I guess with two advertisements, you can send all the data. |
|
Format option: As recommended by BLE, a typical advertisement contains 'BLE Discovery Mode flags' - this is 3 bytes. Data Structure Format: SizeAndType, DataID, [byte0[..byte N]] SizeAndType:
'DataID' may describe a variable name from a user installation and/or generic device classes. For the user data header, you must select 2 generic UUIDs. In the encrypted data, the last 8 bytes of the message mean: 32 bits - the counter and 32 bits "Message integrity check (MIC)". "Bindkey" is used - by analogy with the mija format. In this format, it is possible to transfer four 16-bit values simultaneously and in encrypted form. |
|
Chrome Bluetooth API does not return MAC address in java script.
Then for MAC DataID is not transmitted. |
|
OK, I like your proposal. I also have a an issue with the GATT specifications, which seem to miss quite some types. e.g. there is nothing that even looks like an open/close sensor (either a door sensor or a switch sensor). Can you show me an example how an data packet looks like. I'm struggling a bit with the SizeAndType byte. How do I get the length from this byte, e.g. when I have two dataIDs, e.g. temperature and humidity? |
|
Assume that the following DataID values are defined:
And set UUID16 for unencrypted data = 0x1234 and humidity 50.55%
|
If you do not feel sorry for the DID numbers, then it is possible to describe individual events without data:
But it is not recommended to do so...
If the data is not declared, then the default value is always taken - for example = 0 The problem remains in the Chrome API for javascript with ad encryption. |
|
Ok, I will change this in HA BLE according to your proposal. Better to do it right now, than changing it later. Strange that the Chrome API can’t find the MAC with scanning, the MAC is always in the first part of an BLE advertisement. But I’m not familiar with the API. |
For security reasons, the user selects a device in a separate menu, and the API provides access only to the data of the selected device. And in the device selection request, you must describe all the UUIDs that you will work with. |
|
@pvvx. I made a very first draft. I need to add more sensors, add some Error handling, examples, tests, etc. but the first draft is in the new-HA_BLE branch. It isn't beta ready yet, but if you want to try, overwrite the Format is explained here (for now) https://github.com/custom-components/ble_monitor/blob/new-HA_BLE/docs/ha_ble.md |
|
Average consumption at 3.3V Xiaomi LYWSD03MMC B1.4 (default settings): "temperature", "humidity", "battery", "voltage", "rssi" - worked ok.
|
|
Removing the device helped. At the moment there is no "switch" and "opening". |
|
I have added the There are two ways to solve this.
|
There are no issue here. The current optional variant was created specifically for the test. All this is not yet included in the version for users. For the release version, there will be a completely different algorithm. The current version of 'TelinkMiFlasher.html' and options for new betas are already built in and working: In the "counter" mode, the counter is transmitted only by "report interval" and when the count overflows 16 bits (device cannot store more than 16 counter bits across hard/soft resets. But the counter size is 32 bits.). |
|
Great. I'm a bit worried about the 4 GB per day, you showed. Not sure where this is coming from. I have now added the possibility to specify a MAC in the payload. You can even use a different MAC, to make some "virtual" device. One question for you. I have now specified a fixed "factor" per measurement type, to get a sufficient number of digits. Is this "workable" for you? Or should we add the possibility to overrule the factor? |
This is normal "HA" behavior with default settings. The actual size of the overwrite is larger, because disk devices rewrite not one sector of 512 bytes, but a block of 16 or more kilobytes.
Let's look at the requests as the format evolves. |
|
The first release with "HA_BLE" in version 3.7 is ready. Log (temperature and humidity = ADC1 and ADC2 for test): |
|
Excellent. I have started working on the encryption now. Will use the same type as you have in the ATC firmware. |
|
Encryption... My suggestion:
Package type:
The ATC and PVVX encryption example uses an 8-bit counter to achieve maximum power savings (minimum data packet size) and low cryptographic strength. The use of AES:CCM was taken for the sake of compatibility. PS: If you do not provide fake theoretical security that is not used in a real process, then there will be many people who want to declare their literacy after reading documentation on cryptography... |
|
Ok, will use that. This encryption stuff is new for me. |
|
@pvvx I start to understand it a bit (the encoding/decoding). I also managed to add the decryption (locally) for my own test, but I can't figure out where the Should the 4 bytes counter be part of the import struct
import binascii
from Cryptodome.Cipher import AES
def parse_value(hexvalue):
vlength = len(hexvalue)
# print("vlength:", vlength, "hexvalue", hexvalue.hex(), "typecode", typecode)
if vlength >= 3:
temp = round(int.from_bytes(hexvalue[2:4], "little", signed=False) * 0.01, 2)
humi = round(int.from_bytes(hexvalue[6:8], "little", signed=False) * 0.01, 2)
print("Temperature:", temp, "Humidity:", humi)
return 1
print("MsgLength:", vlength, "HexValue:", hexvalue.hex())
return None
def decrypt_payload(payload, key, nonce):
mic = payload[-4:] # mic
cipherpayload = payload[:-4] # EncodeData
print("Nonce: %s" % nonce.hex())
print("CryptData: %s" % cipherpayload.hex(), "Mic: %s" % mic.hex())
cipher = AES.new(key, AES.MODE_CCM, nonce=nonce, mac_len=4)
cipher.update(b"\x11")
data = None
try:
data = cipher.decrypt_and_verify(cipherpayload, mic)
except ValueError as error:
print("Decryption failed: %s" % error)
return None
print("DecryptData:", data.hex())
print()
if parse_value(data) != None:
return 1
print('??')
return None
def decrypt_aes_ccm(key, mac, data):
print("MAC:", mac.hex(), "Binkey:", key.hex())
print()
adslength = len(data)
if adslength > 8 and data[0] <= adslength and data[0] > 7 and data[1] == 0x16 and data[2] == 0x1C and data[3] == 0x18:
pkt = data[:data[0]+1]
# nonce: mac[6] + head[4] + cnt[1]
nonce = b"".join([mac, pkt[:4]])
return decrypt_payload(pkt[4:], key, nonce)
else:
print("Error: format packet!")
return None
#=============================
# main()
#=============================
def main():
print()
print("====== Test encode -----------------------------------------")
temp = 25.06
humi = 50.55
print("Temperature:", temp, "Humidity:", humi)
print()
data = bytes(bytearray.fromhex('2302CA090303BF13'))
mac = binascii.unhexlify('5448E68F80A5') # MAC
binkey = binascii.unhexlify('231d39c1d7cc1ab1aee224cd096db932')
print("MAC:", mac.hex(), "Binkey:", binkey.hex())
adshead = struct.pack(">BBH", len(data) + 7, 0x16, 0x1C18) # ad struct head: len, id, uuid16
beacon_nonce = b"".join([mac, adshead])
cipher = AES.new(binkey, AES.MODE_CCM, nonce=beacon_nonce, mac_len=4)
cipher.update(b"\x11")
ciphertext, mic = cipher.encrypt_and_digest(data)
print("Data:", data.hex(), adshead.hex())
print("Nonce:", beacon_nonce.hex())
print("CryptData:", ciphertext.hex(), "Mic:", mic.hex())
adstruct = b"".join([adshead, ciphertext, mic])
print()
print("AdStruct:", adstruct.hex())
print()
print("====== Test decode -----------------------------------------")
decrypt_aes_ccm(binkey, mac, adstruct);
if __name__ == '__main__':
main() |
|
At the entry we have:
Ad Structure:
Head_UUID16:
or decrypted_data = HA-BLE Data Array You can accept the condition that 'count_id' should only increase in the positive direction. If in 5 seconds there was a change in 'count_id' by more than 100 units - a warning "attack - manipulation of encrypted data" and/or message is not accepted. If 'count_id' does not differ from the previous one for more than 16 data receptions or within an hour - a similar warning "Attack - manipulation of encrypted data" and/or message is not accepted. If 'count_id' differs negatively (starts counting from a new value), then the message is rejected and only subsequent messages that meet the conditions described above are allowed. :) |
|
Added encryption in the HA_BLE-new branch. Below the script to encrypt the message (and decrypt). Will add that to the docs later, after some cleaning. import struct
import binascii
from Cryptodome.Cipher import AES
def parse_value(hexvalue):
vlength = len(hexvalue)
# print("vlength:", vlength, "hexvalue", hexvalue.hex(), "typecode", typecode)
if vlength >= 3:
temp = round(int.from_bytes(hexvalue[2:4], "little", signed=False) * 0.01, 2)
humi = round(int.from_bytes(hexvalue[6:8], "little", signed=False) * 0.01, 2)
print("Temperature:", temp, "Humidity:", humi)
return 1
print("MsgLength:", vlength, "HexValue:", hexvalue.hex())
return None
def decrypt_payload(payload, mic, key, nonce):
print("Nonce: %s" % nonce.hex())
print("CryptData: %s" % payload.hex(), "Mic: %s" % mic.hex())
cipher = AES.new(key, AES.MODE_CCM, nonce=nonce, mac_len=4)
cipher.update(b"\x11")
try:
data = cipher.decrypt_and_verify(payload, mic)
except ValueError as error:
print("Decryption failed: %s" % error)
return None
print("DecryptData:", data.hex())
print()
if parse_value(data) != None:
return 1
print('??')
return None
def decrypt_aes_ccm(key, mac, data):
print("MAC:", mac.hex(), "Bindkey:", key.hex())
print()
adslength = len(data)
if adslength > 15 and data[0] == 0x1E and data[1] == 0x18:
pkt = data[:data[0] + 1]
uuid = pkt[0:2]
encrypted_data = pkt[2:-8]
count_id = pkt[-8:-4]
mic = pkt[-4:]
# nonce: mac [6], uuid16 [2], count_id [4] # 6+2+4 = 12 bytes
nonce = b"".join([mac, uuid, count_id])
return decrypt_payload(encrypted_data, mic, key, nonce)
else:
print("Error: format packet!")
return None
# =============================
# main()
# =============================
def main():
print()
print("====== Test encode -----------------------------------------")
temp = 25.06
humi = 50.55
print("Temperature:", temp, "Humidity:", humi)
print()
data = bytes(bytearray.fromhex('2302CA090303BF13')) # HA BLE data
count_id = bytes(bytearray.fromhex('00112233')) # count id
mac = binascii.unhexlify('5448E68F80A5') # MAC
uuid16 = b"\x1E\x18"
bindkey = binascii.unhexlify('231d39c1d7cc1ab1aee224cd096db932')
print("MAC:", mac.hex(), "Binkey:", bindkey.hex())
nonce = b"".join([mac, uuid16, count_id]) # 6+2+4 = 12 bytes
cipher = AES.new(bindkey, AES.MODE_CCM, nonce=nonce, mac_len=4)
cipher.update(b"\x11")
ciphertext, mic = cipher.encrypt_and_digest(data)
print("Data:", data.hex())
print("Nonce:", nonce.hex())
print("CryptData:", ciphertext.hex(), "Mic:", mic.hex())
adstruct = b"".join([uuid16, ciphertext, count_id, mic])
print()
print("AdStruct:", adstruct.hex())
print()
print("====== Test decode -----------------------------------------")
decrypt_aes_ccm(bindkey, mac, adstruct)
if __name__ == '__main__':
main() |
|
|
|
Fixed the packet id. It is taking the packet id
|
|
@pvvx It took a while, but I have release a beta, 8.0.1-beta, which supports automatically adding of sensors in HA, based on the sensor types it receives (only for HA BLE). In the future, I probably am going to move the other sensors to the same system (other sensors are now added based on a predefined sensor list per sensortype), but that's for later. There are a few TODO's, but it seems to be working.
|
Transmission with auto-switching in 3 formats at once (atc1441, pvvx, mijia) is only in the old version. Starting with version 3.7, the "All" mode has been replaced by "HA-BLE". Those. one of 4 formats is used without encryption or with encryption enabled. Additional options such as "counter", "switch", "open/close" are fully demonstrated in the "HA-BLE" protocol, and in others - by capabilities.
*mijia-20e1 - Vendor-defined attributes 0x20E1, size 4, uint32 |
|
The new HA BLE format has been released as final now (8.1.0), so I'm closing this issue. I will create a new issue for the restore state functionality, that isn't working yet for HA BLE sensors. @pvvx Many thanks for you tips to improve the format, much appreciated. |
Request by @scrambledleek to create support for home brew BLE sensors. BLE monitor should:
The text was updated successfully, but these errors were encountered: