/query endpoint doesn't respond sporadically if more than one client tries to run a query.

[maksymilian-gasztych]

Describe the bug
/query endpoint doesn't return any response around one in ten times if if there are at least 2 simultaneous clients querying it.
If there are much more simultaneous clients it even stops responding completely.

I also created this ticket, but I managed to get some more details that I included here, so closing that one

To Reproduce

1. Create any stream or table
   CREATE STREAM test3 (id INT KEY,one DECIMAl(5,5),two DECIMAL(5,2))WITH (kafka_topic='test', partitions=2, value_format='json');
2. query using curl

 curl -X "POST" "http://localhost:28088/query" \
-H "Accept: application/vnd.ksql.v1+json" \
-d $'{
"ksql": "select one,two,one/two from test3 emit changes;",
"streamsProperties": {}
}'

3. Open ksql CLI and try to run the query around 10 times, the cli will hang unresponsive untli you kill the curl request.
   select one,two,one/two from test3 emit changes;

Expected behavior
/query endpoint works for more than one client without any issues

Actual behaviour
No response is returned

Additional context
Verified on 0.14.0 and 0.15.0
Just in case attaching docker settings I'm using

confluentinc/ksqldb-server:0.15.0 settings:

      KSQL_KSQL_SERVICE_ID: "cluster"
      KSQL_LISTENERS: "http://0.0.0.0:8088"
      KSQL_BOOTSTRAP_SERVERS: "broker:9092"
      KSQL_KSQL_SCHEMA_REGISTRY_URL: "http://schema-registry:8081"
      KSQL_KSQL_LOGGING_PROCESSING_STREAM_AUTO_CREATE: "true"
      KSQL_KSQL_LOGGING_PROCESSING_TOPIC_AUTO_CREATE: "true"
      KSQL_CONNECT_GROUP_ID: "ksql-connect-cluster"
      KSQL_CONNECT_BOOTSTRAP_SERVERS: "broker:9092"
      KSQL_CONNECT_KEY_CONVERTER: "org.apache.kafka.connect.storage.StringConverter"
      KSQL_CONNECT_VALUE_CONVERTER: "io.confluent.connect.avro.AvroConverter"
      KSQL_CONNECT_KEY_CONVERTER_SCHEMA_REGISTRY_URL: "http://schema-registry:8081"
      KSQL_CONNECT_VALUE_CONVERTER_SCHEMA_REGISTRY_URL: "http://schema-registry:8081"
      KSQL_CONNECT_CONFIG_STORAGE_TOPIC: "ksql-connect-configs"
      KSQL_CONNECT_OFFSET_STORAGE_TOPIC: "ksql-connect-offsets"
      KSQL_CONNECT_STATUS_STORAGE_TOPIC: "ksql-connect-statuses"
      KSQL_CONNECT_CONFIG_STORAGE_REPLICATION_FACTOR: 1
      KSQL_CONNECT_OFFSET_STORAGE_REPLICATION_FACTOR: 1
      KSQL_CONNECT_STATUS_STORAGE_REPLICATION_FACTOR: 1
      KSQL_CONNECT_REST_ADVERTISED_HOST_NAME: "ksqldb-server"

confluentinc/cp-schema-registry:6.1.0 settings:

      SCHEMA_REGISTRY_HOST_NAME: schema-registry
      SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS: 'broker:9092'
      SCHEMA_REGISTRY_LISTENERS: "http://0.0.0.0:8081"
      SCHEMA_REGISTRY_GROUP_ID: "cp-schema-registry-cluster"
      SCHEMA_REGISTRY_SCHEMA_COMPATIBILITY_LEVEL: "forward"

confluentinc/cp-enterprise-kafka:6.0.0 settings:

      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker:9092,PLAINTEXT_HOST://localhost:29092
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1

[agavra]

Thanks for the details @maksymilian-gasztych! Just noting to the team that I was able to reproduce it locally - seems like some fascinating race condition.

[maksymilian-gasztych]

Glad that you were able to reproduce it too. Hopefully it's an easy fix, or that you can implement tombstone support for /query-strem endpoint, because it's a very big issue. I'm honestly surprised that noone has yet encountered it.

[agavra]

0.19 timeline to understand exactly what the impact is, and hopefully fix it as well

[vvcephei]

Ok, I've got a fix here: #7420

[vvcephei]

This should be fixed in 0.18 via #7420