KSQL should not filter out stream rows with null key

[big-andy-coates]

It makes sense that tables ignore any records in their changelog logs (or sources) would null keys. A table requires a primary key, and hence a null is invalid.

However, streams have no such restriction. Yet, ksql will filter out rows where the key is null, e.g. the following fails:

{
      "name": "partition by with null value",
      "statements": [
        "CREATE STREAM TEST (ROWKEY BIGINT KEY, ID bigint) with (kafka_topic='test_topic', value_format = 'json');",
        "CREATE STREAM REPARTITIONED AS select * from TEST partition by ID;"
      ],
      "inputs": [
        {"topic": "test_topic", "key": 0, "value": null},
        {"topic": "test_topic", "key": 0, "value": {}},
        {"topic": "test_topic", "key": 0, "value": {"ID": 10}}
      ],

      "outputs": [
        {"topic": "REPARTITIONED", "key":  null, "value": {}},
        {"topic": "REPARTITIONED", "key":  null, "value": {}}
        {"topic": "REPARTITIONED", "key": 10, "value": {"ID": 10}}
      ]
    },
    

To get it to pass one must remove the output rows with null keys:

{
      "name": "partition by with null value",
      "statements": [
        "CREATE STREAM TEST (ROWKEY BIGINT KEY, ID bigint) with (kafka_topic='test_topic', value_format = 'json');",
        "CREATE STREAM REPARTITIONED AS select * from TEST partition by ID;"
      ],
      "inputs": [
        {"topic": "test_topic", "key": 0, "value": null},
        {"topic": "test_topic", "key": 0, "value": {}},
        {"topic": "test_topic", "key": 0, "value": {"ID": 10}}
      ],

      "outputs": [
        {"topic": "REPARTITIONED", "key": 10, "value": {"ID": 10}}
      ]
    },
    

[blueedgenick]

Yikes! This seems like a pretty serious bug - can you tell if this had always been the case or it was a regression somewhere along the line?

…

On Tue, Mar 10, 2020, 7:38 AM Andy Coates ***@***.***> wrote: It makes sense that tables ignore any records in their changelog logs (or sources) would null keys. A table requires a primary key, and hence a null is invalid. However, streams have no such restriction. Yet, ksql will filter out rows where the key is null, e.g. the following fails: { "name": "partition by with null value", "statements": [ "CREATE STREAM TEST (ROWKEY BIGINT KEY, ID bigint) with (kafka_topic='test_topic', value_format = 'json');", "CREATE STREAM REPARTITIONED AS select * from TEST partition by ID;" ], "inputs": [ {"topic": "test_topic", "key": 0, "value": null}, {"topic": "test_topic", "key": 0, "value": {}}, {"topic": "test_topic", "key": 0, "value": {"ID": 10}} ], "outputs": [ {"topic": "REPARTITIONED", "key": null, "value": {}}, {"topic": "REPARTITIONED", "key": null, "value": {}} {"topic": "REPARTITIONED", "key": 10, "value": {"ID": 10}} ] }, To get it to pass one must remove the output rows with null keys: { "name": "partition by with null value", "statements": [ "CREATE STREAM TEST (ROWKEY BIGINT KEY, ID bigint) with (kafka_topic='test_topic', value_format = 'json');", "CREATE STREAM REPARTITIONED AS select * from TEST partition by ID;" ], "inputs": [ {"topic": "test_topic", "key": 0, "value": null}, {"topic": "test_topic", "key": 0, "value": {}}, {"topic": "test_topic", "key": 0, "value": {"ID": 10}} ], "outputs": [ {"topic": "REPARTITIONED", "key": 10, "value": {"ID": 10}} ] }, — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#4747?email_source=notifications&email_token=ABCXJIHXXS2P5HAS7KBSZ4LRGZGG5A5CNFSM4LFANXRKYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4IT5FLWA>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABCXJIC4G5ADDRQV46444STRGZGG5ANCNFSM4LFANXRA> .

[mjsax]

That is an interesting case. Note that a query like this would be "non-deterministic" with regard the output topic partitioning, because for the null-key record a random partition would be selected.

Hence, I am wondering if this might be an issue or not?

[big-andy-coates]

Yikes! This seems like a pretty serious bug - can you tell if this had always been the case or it was a regression somewhere along the line?

I think this has always been the case.

[big-andy-coates]

That is an interesting case. Note that a query like this would be "non-deterministic" with regard the output topic partitioning, because for the null-key record a random partition would be selected.

Hence, I am wondering if this might be an issue or not?

The non-determinism may well be an issue. If it is, then I'm guessing we'll need to introduce a SQL NULL object of some kind, which will serialize to something other than null bytes. This would ensure consistent partitioning.

However, I feel the introduction of a NULL object is potentially best left to a different ticket. Initially, we can just ensure we're not filtering out these rows.