Disallow nullness check in read filter

[dmehra]

InfluxDB appears to not support nullness checks. If I write a point that has value2 and another point that doesn't, then read influx value2 = null returns an empty set.

juttle> emit -points [{ value: 0.01, value2: 25, host: 'www123', name: 'cpu' }] | write influx -db 'dmtest'
juttle> emit -points [{ value: 0.02, host: 'www123', name: 'cpu' }] | write influx -db 'dmtest'
juttle> read influx -db 'dmtest' -from :0: value2 = null
└───────────┴───────────┘

Influx also confuses nulls and zeroes, so if I write another point with value2 set to 0, the nullness check would bring it up:

juttle> emit -points [{ value: 0.03, value2: 0, host: 'www123', name: 'cpu' }] | write influx -db 'dmtest'
juttle> read influx -db 'dmtest' -from :0: value2 = null
┌────────────────────────────────────┬──────────┬──────────┬──────────┬──────────┬──────────┐
│ time                               │ name     │ value    │ host     │ name2    │ value2   │
├────────────────────────────────────┼──────────┼──────────┼──────────┼──────────┼──────────┤
│ 2016-02-11T20:21:42.299Z           │ cpu      │ 0.03     │ www123   │          │ 0        │
└────────────────────────────────────┴──────────┴──────────┴──────────┴──────────┴──────────┘

The not-nullness check is also misinterpreted, as it will only show points with non-zero value:

juttle> read influx -db 'dmtest' -from :0: value2 != null
┌────────────────────────────────────┬──────────┬──────────┬──────────┬───────────┬──────────┐
│ time                               │ name     │ value    │ host     │ name2     │ value2   │
├────────────────────────────────────┼──────────┼──────────┼──────────┼───────────┼──────────┤
│ 2016-02-11T20:18:11.761Z           │ cpu      │ 0.01     │ www123   │           │ 25       │
├────────────────────────────────────┼──────────┼──────────┼──────────┼───────────┼──────────┤

That was for value fields. The nullness check is also completely broken for tag fields. If I write a point that has name3 while other points do not, both the check for null and for not-null return empty sets:

juttle> emit -points [{ value: 0.05, host: 'www123', name: 'cpu', name3: 'special' }] | write influx -db 'dmtest'
juttle> read influx -db 'dmtest' -from :0: name3 != null
└───────────┴───────────┘
juttle> read influx -db 'dmtest' -from :0: name3 = null
└───────────┴───────────┘

Given this behavior, the juttle adapter should just error out with a message like "Unsupported by InfluxDB" if the user attempts a null/not-null check in the read filter.

[dmehra]

On second thoughts, I'm not entirely sure this is unsupported.

What is definitely unsupported in InfluxDB is storing null values, it was disallowed in this commit because "The TSM engine does not handle null values." (tsm is the main, soon to be the only, storage option for Influx).

However, the nullness check was brought up in issue#454 and closed as "obsoleted by v0.9" meaning what, that it should work now?

[dmehra]

I think I found how one is supposed to check for nullness of tags in idiomatic influx query language, that would be with a regex! Source (look for The WHERE Clause)

Instead of name3 != null you do this:

juttle> read influx -db 'dmtest' -from :0: name3 =~ /.*/
┌────────────────────────────────────┬──────────┬──────────┬──────────┬──────────┬───────────┬──────────┐
│ time                               │ name     │ value    │ host     │ name2    │ name3     │ value2   │
├────────────────────────────────────┼──────────┼──────────┼──────────┼──────────┼───────────┼──────────┤
│ 2016-02-11T21:01:17.657Z           │ cpu      │ 0.05     │ www123   │          │ special   │          │
└────────────────────────────────────┴──────────┴──────────┴──────────┴──────────┴───────────┴──────────┘

And instead of name3 = null you do this:

juttle> read influx -db 'dmtest' -from :0: name3 !~ /.*/
┌────────────────────────────────────┬──────────┬──────────┬──────────┬───────────┬──────────┬──────────┐
│ time                               │ name     │ value    │ host     │ name2     │ name3    │ value2   │
├────────────────────────────────────┼──────────┼──────────┼──────────┼───────────┼──────────┼──────────┤
│ 2016-02-11T20:18:11.761Z           │ cpu      │ 0.01     │ www123   │           │          │ 25       │
├────────────────────────────────────┼──────────┼──────────┼──────────┼───────────┼──────────┼──────────┤
│ 2016-02-11T20:18:25.031Z           │ cpu      │ 0.02     │ www123   │           │          │          │
└────────────────────────────────────┴──────────┴──────────┴──────────┴───────────┴──────────┴──────────┘

[dmehra]

@bkutil see if there is an issue we should file against InfluxDB here. It seems that for tags they just went with the surprising regex approach, but for field values i still think it's just broken, the way they treat nulls same as zeroes.

And while I briefly thought we could map name = null to the regex check, I already realized we cannot since we have no way of distinguishing a tag from a field value in the read filter.

Still seems like the short term solution is to disallow null checks in the juttle read filter.