[mqtt] Faulty display of an linked item using Units of Measurement.

[JueBag]

Setup

A device reports a temperature ( measured in °C) via mqtt as a plain number 16.0.
A channel is setup on that value using a JSONPATH transformation and a Unit of Measurement setting of °C.
A linked Number:Temperature item would display 16,0 °C.

Expected Behavior

Using a State Description pattern on the item of %.1f °F the displayed value would be converted to Fahrenheit. 60,8 °F.

Current Behavior

Initially the converted value is displayed correctly, however after the channel receives an update the displayed value changes back to the Degree value keeping the Fahrenheit sign 16,0 °F.
This change is correlating with this following events.log entry.

2021-03-14 14:00:32.259 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'TestTemperature' changed from 16.0 °C to 16.0 °F

Discussed on the forum

[openhab-bot]

This issue has been mentioned on openHAB Community. There might be relevant details there:

https://community.openhab.org/t/mqtt-unit-of-measurement-faulty-behaviour/118946/17

[openhab-bot]

This issue has been mentioned on openHAB Community. There might be relevant details there:

https://community.openhab.org/t/format-percent/119180/8

[Rossko57]

The short version of the problem description is that the binding is posting just a number via the channel, and forgetting to append the given unit.

[J-N-K]

Debatable if this is a bug or an enhancement.

Looking at the code it seems that the unit is only honored for outgoing values (the command to the channel is converted to the unit in the channel config, so if you configure °C and send °F, the value is converted to °C and then send to the broker). The unit isn't considered for incoming values (i.e. values received from MQTT, they are always posted as DecimalType). IIRC it was delibaretly done that way because it would require a bigger refactoring to fully enable UoM in MQTT.

[JueBag]

In view of what you discribed the display of the "Degree value with the Fahrenheit sign" would be explained, but what about the initially displayed converted (Fahrenheit) value?
At least that is rising questions, being it "bug" or not.
Additionally I'm seeing the item display the raw number only if the channel.is not setup with Units of Measurement °C. Using such a setting the item (with no State description pattern) shows this unit. That is not inline with your explanation.

[J-N-K]

No, that's explainable.

What happens if you send a DecimalType is that the Item tries to determine the unit from the state description:

https://github.com/openhab/openhab-core/blob/4bf156ffc0fff8d76b103d4813958d88b0ca7f26/bundles/org.openhab.core/src/main/java/org/openhab/core/library/items/NumberItem.java#L109

So if the state description reports °F, the new value is set as °F.

Before the first update the state description is probably not set (not sure about that) and the default unit (°C) is used, so the value set for the item is 16°C. .If you set the state desciption the display changes (correctly) to 60 °F. With the next update the state description is °F and the number 16 is send as 16 °F.

[JueBag]

I'm sorry but the actual code is above my humble java knowledge!
However I don't see a default unit (of °C) being used at all, since when setting the channel to °F (which is wrong) the item does display this unit. So no default unit is used but the one sent from the channel setting. Removing the unit results in the display of a number without dimension. What is odd is the log-entry when removing the °F, it reads:
[openhab.event.ItemStateChangedEvent ] - Item 'TestTemperature' changed from 15.3 °F to 15.3 °C
So there is a default unit, but it is not used for displaying??

[J-N-K]

I guess you could set something like %.1f %unit% as state description to display the default unit. I have next-to-zero knowledge about UI components. New states for a Number-item are processed in the following way

1. check if new state is DecimalType, if not, go to 7
2. check if dimension is null (that is the case for a plain Number item), if so go to 5
3. try to get unit: do we have a state description? if so, get that unit and go to 6
4. check if we have a unitProvider (that's probably the system default unit) and get the unit from there and go to 6
5. set state without unit and exit
6. use the numeric value and the unit, set the state as QuantityType and exit
7. check if QuantityType, convert unit if necessary, set the state as QuantityType and exit
8. check if other acceptable type (e.g. UnDefType), if so, set state and exit
9. fail

[Rossko57]

However I don't see a default unit (of °C) being used at all,

Yes, you did. In your original forum post,before applying pattern -

Linking it without a State Description Pattern gives a display of : 16°C.

The binding did not supply °C, it was picked up by the framework as the system default unit appropriate for a Number:Temperature type Item, What does "default unit" mean? If you post a number with NO units (as the binding currently does) then it assumes you meant °C

That's all working as designed - except for the binding not appending the unit you told it.

I can't believe the MQTT channel unit parameter was only intended for outgoing. That's really pointless, as its easily done with a "format before sending" in any case.
Let's call it "partially implemented" and finish the job?

[JueBag]

@Rossko57
In the forum post I had the channel set to use °C, so it could have been the default or the setting for Unit of Measurement.
See my post above where I use the (false unit) °F instead on the channel which got displayed by the item.
@J-N-K
Yes, using the State description pattern of %.1f %unit% does display °C if the channel has no Unit of Measurement set.

Looking at the procedure I think the problem is between step 6 and 7. IMHO the " convert if necessary " should be applied.

[J-N-K]

@Rossko57 No, it doesn't "only append" the unit. It converts the value of the item to that unit before sending. So if the item's state is 60.8 °F, the value is converted to 16°C before sending.

@JueBag Step 6 is a final step, we never get to step 7. Values received from MQTT are send as DecimalType and conversion only takes place for QuantityType.

To both of you: I fully agree that this is an unexpected behaviour and not clearly documented. And the correct solution would be to add the missing code. But as so said: that could be not full straight forward.

[Rossko57]

it doesn't "only append" the unit. It converts the value of the item to that unit before sending. So if the item's state is 60.8 °F, the value is converted to 16°C before sending.

Sure. But we're talking about incoming data here. In this case, numeric only. "We" the user/configurer knows that represents F or whatever, and wishes to add that info to the incoming data. In other words, add " °F" to the numeric to create a Quantity, ready for use updating a Number:Temperature Item.

[J-N-K]

Fully agreed. The problem is that we would need to assign the correct „accepted item type“ dynamically depending on the selected unit, i.e. we would need to change the channel-type when configuration changes.

That can be done but finding the correct dimension (like temperature, pressure) from the given unit is not trivial.

Just posting a QuantityType will probably work but could end up with unsupported unit errors.

[JueBag]

I "see" that it not only appends the unit.
Looking closely at data while doing the change to the Unit of Measurement setting °F I figured that the item state did not only get the Fahrenheit sign, before the next StateUpdate the value was also converted to Fahrenheit! After the update the unconverted value with the F-sign is displayed. So it initialy converts and afterwards only appends.

For me the problem boils down to:
What is the difference of the information that is held by an item (1) to the one it gets from an state update(2) and what causes this difference?

In case of (1) the set of a State Description pattern does trigger a complete conversion to the desired unit.
In case of (2) the already set State Description pattern does only change the unit sign without doing the conversion.

Looking at the linked code I'd say in case of (1) the state is recognized as a QunatityType ,in case of (2) it isn't.

[Rossko57]

Just posting a QuantityType will probably work but could end up with unsupported unit errors.

I believe this works fine in recent Modbus binding, with just a number type channel linked to Number:type of user's choice. The binding creates a Quantity from raw numeric data and user-designated "unit" parameter.
It is of course possible to misconfigure and try putting "22 °C" or "14 bananas" into a Number:Length Item, but this is after all an "advanced" manually configured usage and not expected to be idiot proof?

I do not know if Modbus specifically allows all Number:xxx Item types as "accepted", or just takes advantage of lax policing when creating links.

EDIT -duh, Opus has already linked his plain number type MQTT channel to a Number:Temperature Item, so I guess there is no practical problem here.

[Rossko57]

What is the difference of the information that is held by an item (1) to the one it gets from an state update(2) and what causes this difference?

I'm not sure what's troubling you here.

Let's work through it.
You got an Item Number:Temperature, it starts life as NULL.
Along comes the binding and posts "16" to it.
But it's a temperature type, it must have a unit, so the OH framework intervenes and looks for a unit - there's no Item pattern option, but there is a system default, so it uses that.
There's no conversion because we started with just-a-number, there are no units to convert from.
The post of "16" ends up as Item state "16 °C"

Next you come along and set Item option pattern "°F".
The Item still contains state "16 °C"
But the UI uses the pattern immediately, conversion is done from °C to °F, that's the real purpose of the pattern, and so you see "65 °F" at this time.

Later. along comes the binding with another update, it's still just "16"
Again, OH framework looks for a unit - but this time there is a pattern, so it uses that.
There's still no conversion because we still started with just-a-number,
This time update "16" ends up as Item state "16 °F"

The UI still uses the pattern, does the no-operation conversion °F to °F, and displays "16 °F"

You might now come along and post "273 K" via the API. That's fine, it's a valid temperature.
OH framework dutifully looks for a unit - still there is a pattern °F, so it uses that.
But this time the update has come with units, so a conversion is applied K to°F
Update "273 K" will give Item state "32 °F"

[Rossko57]

Thinks; if you want to prove a point here, you can make a state transformation on the MQTT channel that emulates what we'd like/expect the unit= parameter to do. Just a JS script that appends string " °F" to the raw data number (as string)

[JueBag]

Sometimes one doesn't see the wood because of all those trees!
Thanks for this ( niigthly) explanation.

[openhab-bot]

This issue has been mentioned on openHAB Community. There might be relevant details there:

https://community.openhab.org/t/oh-3-1-0-m3-item-with-actual-state-suddenly-listed-as-undefined-in-location-card/120144/9

[openhab-bot]

This issue has been mentioned on openHAB Community. There might be relevant details there:

https://community.openhab.org/t/mqtt-binding-and-units-of-measurement/127026/3

[rkoshak]

Next you come along and set Item option pattern "°F".
The Item still contains state "16 °C"
But the UI uses the pattern immediately, conversion is done from °C to °F, that's the real purpose of the pattern, and so you see "65 °F" at this time.

I know I'm late to the party here but I just ran into exactly this problem. Based on what I'm seeing this isn't want is actually happening and it makes me wonder if there isn't some issue in openhab-core too.

I'm receiving a number. I know this number is °C. I set "°C" as the unit in the Channel config.

So far so good. I see in events.log that the Item is changing and using "°C".

Now here is the tricky part. I want to see "°F" in the UIs so I edit the State Description Pattern to be "%.0f °F".

What I'd expect to happen is that in the UI I now see the value converted to °F. And I in fact to see that. However, I also see that OH, on it's own and in response to adding the State Description Pattern also changed the state of the Item. At this point the Item is no longer storing "20 °C", it's actually storing "68 °F". This can be seen in the cart as well.

(Sorry, I don't know how to make it smaller).

That chart fragment starts at everything being °C. We start out at 18 °C. The Channel is configured to use °C for the units.

Then I change only the State Description of the Item to use °F and you see the line spike up to 65ish and there was an Item changed event in the logs showing this as well. It's not just the way the Item is displayed that changed, the Item's state changed.

The next value received from the device then gets applied to the Item. Despite the fact that the unit in the Channel is °C, it treats the value as if it were °F without doing any conversion at all. The cart drops to 18 °F which is clearly wrong.

The next dip you see was me running an experiment. I changed the State Description Pattern to use °C. The Item then changed to around -6 °C which is the °C equivalent to 18 °F. Again, it's not just the way it displayed, the Item actually changed to this state based on events.log.

Finally I removed the unit from the State Description Pattern and left it as °C in the Channel and things started to work, all be it as °C.

Something is changing the state of the Items when the State Description Pattern is changed which doesn't seem like it should. State Description should change how it's displayed, it shouldn't change how it's stored, right?

Also, the units that the binding should be providing (°C in this case) is ignored or overridden by the State Description Pattern.

The only way I'm able to take in that raw °C number and convert it to a usable °F is to not use the unit parameter of the Channel config and instead use the following JS transform with a State Description of "%.0f °F".

(function(i) {
  var QuantityType = Java.type("org.openhab.core.library.types.QuantityType");
  return new QuantityType(i +" °C").toUnit("°F").floatValue();
})(input)

Therefore I propose, at a minimum, the description of what the unit parameter on the Channel config does be changed to remove the implication that it can be used in any way to convert the incoming units to a different unit.

And if someone agrees that there seems to be something outside of the MQTT binding also at play here I'll file an issue in core too.

[Rossko57]

I don't think it works like that, Rich.
Try interpreting with assumptions -
MQTT binding posts no units (so defaults get invoked)
Changing pattern does not change Item state.
The chart isn't showing Item state, it's showing persisted data.

Persistence is another story ...
It can't store units.
I'm not entirely clear if it converts to default upon persisting, or just saves the numeric part as-is. ( I think your chart demonstrates it converts to (current) default before storing. )
It definitely applies the (current) default when retrieving data.
So either way, if you change default (by pattern) you will get different historic data back, without changing the database.

Store 20C, throw away unit and store 20.
Change default.
Retrieve 20, apply default ... chart 20F

The glitch in your chart I think confirms convert-before-store theory.
Finer detail -
Store 20C, default C, throw away unit and store 20.
Change default to F.
Store 20C, convert to default F, throw away unit and store 68.
Get update to 20F, (because of MQTT problem)
Store 20F, default F, throw away unit and store 20.

Retrieve data for charting, apply default F ...
chart 20F , 68F, 20F

[rkoshak]

Here is the complete set of tests I've run thus far. I can't get the ItemChangeEvent to happen any more so that must have been an MQTT message triggered when not expected or something like that. But I think it does absolutely confirm that the MQTT Channel Config units property is being passed to the Item somehow.

Thing config:

UID: mqtt:topic:broker:temp-test
label: MQTT Temperature Test Thing
thingTypeUID: mqtt:topic
configuration: {}
bridgeUID: mqtt:broker:broker
channels:
  - id: degc
    channelTypeUID: mqtt:number
    label: Degrees C
    description: ""
    configuration:
      stateTopic: test/temp
      unit: °C
  - id: degf
    channelTypeUID: mqtt:number
    label: Degrees F
    description: ""
    configuration:
      stateTopic: test/temp
      unit: °F
  - id: nounits
    channelTypeUID: mqtt:number
    label: No Units
    description: ""
    configuration:
      stateTopic: test/temp

Item Config:

Each Channel is linked to a Number:Temperature. The name corresponds to the channel it's linked to

System Config:

Regional settings has Imperial checked which means the default units is °F

Tests:

Test 1: Send 18 to test/temp, review logs and Item states

2021-10-03 21:33:13.563 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'MQTTTemperatureTestThing_DegreesF' changed from NULL to 18 °F                                                                                           │
2021-10-03 21:33:13.563 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'MQTTTemperatureTestThing_NoUnits' changed from NULL to 18 °F                                                                                            │
2021-10-03 21:33:13.563 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'MQTTTemperatureTestThing_DegreesC' changed from NULL to 18 °C

• DegreesF shows 18 °F in MainUI
• NoUnits shows 18 in MainUI, notice there are no units shown despite events.log indicating that the Item changed to 18 °F.
• DegreesC shows 18 °C in MainUI, proving that the binding, at least as of OH 3.2 M2, does pass the units in some way.

Querying Persistence through the API shows:

• DegreesF = 18
• NoUnits = 18
• DegreesC = 18

Conclusion:

The units from the binding has to be passed to OH and it has to be taking precedence over the system default. That's the only way DegreesC could be 18 °C. There is no State Description metadata for any of these Items. If the units configured on the Channel were not being passed to the Item, the DegreesC Item would show the Item changing to °F but as can be seen in this log statement it clearly is °C. The only place where °C is configured anywhere at this point is the MQTT DegreesC Channel.

You can see that the system default does apply when there are no units configured on the Thing's Channel with the NoUnits Item. However, the unit only appears in events.log, not in MainUI without a State Description Pattern.

Test 2: With Items initialized to 18, change state description for all three Items to "%.0f °F"

• Changing the State Description on NoUnits elicits no change. This is expected because the Item is already °F.
• Changing the State Description on DegreesF elicits no change. This is expected because the Item is already °F.
• Changing the State Description on DegreesC elicits no change in the Item's state. This is unexpected and counter to my prior observations so something else must have been going on.

MainUI now shows:

• DegreesF = 18 °F
• NoUints = 18 °F
• DegreesC = 18 °F, this is surprising, it should have converted it to something like 65. This is also counter to my previous observations. But there might be a refresh problem or in MainUI because...

Persistence now shows:

• DegreesF = 18
• NoUnits = 18
• DegreesC = 64.4

So the value is converted somewhere and the converted value is what is stored in the DB.

Interestingly I didn't see the ItemStateChangedEvent this time like I did last week. So something must have happened before when I ran my adhoc tests at the same time that I changed the State Description to generate a changed event.

I publish 19 to the topic.

2021-10-04 07:30:06.500 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'MQTTTemperatureTestThing_NoUnits' changed from 18 °F to 19 °F
2021-10-04 07:30:06.500 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'MQTTTemperatureTestThing_DegreesF' changed from 18 °F to 19 °F
2021-10-04 07:30:06.500 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'MQTTTemperatureTestThing_DegreesC' changed from 18 °C to 19 °F

Notice the interesting behavior for DegreesC. It changed from °C to °F. The number still was not updated to a °F number though, we still have the 19 as the actual state. Looking in MainUI I now see 19 °F for all three Items. Persistence now shows 19 for all three Items.

Conclusion:

The units setting on the MQTT Channel is only used if there isn't a State Description, but it is in fact used at other times. But with the current behavior it is impossible to receive a value of one unit and convert it to another compatible set of units even for display.

Test 3: Same as Test 2 except using %.0f °C and starting from the results of Test 2.

• Changing the State Description for DegreesF to use °C results in the Item showing -7 °C. The weirdness above can be attributed to a failure to refresh MainUI. Again there was not an ItemChangedEvent so that is looking like a red herring.
• Changing the State Description for NoUnits to use °C results in the Item showing -7 °C.
• Changing the State Description for DegreesC to use °C results in the Item showing -7 °C.

Persistence shows:

• DegreesF = -7.22
• NoUnits = -7.22
• DegreesC = -7.22

I publish 20 to the topic.

2021-10-04 07:52:34.764 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'MQTTTemperatureTestThing_NoUnits' changed from 19 °F to 20 °C
2021-10-04 07:52:34.765 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'MQTTTemperatureTestThing_DegreesF' changed from 19 °F to 20 °C
2021-10-04 07:52:34.765 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'MQTTTemperatureTestThing_DegreesC' changed from 19 °F to 20 °C

The same observations as above. The Item changes to the raw number with °C without any conversion.

Test 4: Returning to no State Description

• Removing the State Description from DegreesF shows in MainUI the Item changes to 68 °F.
• Removing the State Description from NoUnits shows in MainUI the Item changes to 20. No conversion takes place and the default units are not picked up, just as in Test 1.
• Removing the State Description from DegreesC shows in MainUI the Item remains 20 °C.

Persistence shows:

• DegreesF = 68
• NoUnits = 68, this is really odd. The value was converted to °F to store in the database but not for what is shown in MainUI.
• DegreesC = 20

Publishing 21 results in

2021-10-04 08:02:10.232 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'MQTTTemperatureTestThing_DegreesF' changed from 20 °C to 21 °F
2021-10-04 08:02:10.233 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'MQTTTemperatureTestThing_NoUnits' changed from 20 °C to 21 °F
2021-10-04 08:02:10.259 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'MQTTTemperatureTestThing_DegreesC' changed from 20 °C to 21 °C

This is consistent with the other tests above. The number is never converted and the units used for the actual state of the Item is taken from Channel config or, when the Channel config doesn't have one defined the system default.

Test 5: Set the state description for an non-MQTT sensor

I don't have any temperature sensors that are reporting via any other binding except MQTT. I tried to find one I could play with. All the units my Things support are Dimensionless (which is less than useful in this particular case) or Time which only has the one unit so I can't try converting to another. This test will have to wait.

Overall Conclusions

• The units parameters is somehow passed to OH and that unit, when defined, is used instead of the system default.
• The only time the system default units are used is when there units is not defined in the Channel nor in the State Description.
• The State Description units will always take precedence.

Because the State Description always takes precedence, it is impossible to receive a value as °C but see it and use it as °F using the built in UoM features, which we already know. One has to use a transformation to convert the incoming value to use the units you want.

Persistence stores the value using the units defined in the State Description.

[openhab-bot]

This issue has been mentioned on openHAB Community. There might be relevant details there:

https://community.openhab.org/t/mqtt-binding-and-units-of-measurement/127026/14

[Rossko57]

There is "something funny" going on here, like the channel passing a Number state (which happens to have unit text appended) instead of a Quantity state (which has the magical conversion properties).
Only after the Item updated to new state would the usual conversions apply.

You can emulate that kind of behaviour in rules, I do not know if something similar can happen as channel updates.

There might be further enlightment or confusion repeating the exercise with say Number:Power type, which unlike Number:Temperature has no system default units to back up absence of declared units. There's less moving parts and it should be more predictable.

[openhab-bot]

This issue has been mentioned on openHAB Community. There might be relevant details there:

https://community.openhab.org/t/uom-input-in-ui/127690/10

[openhab-bot]

This issue has been mentioned on openHAB Community. There might be relevant details there:

https://community.openhab.org/t/convert-wh-to-kwh-divide-by-1000/130107/5