feat: expose JMX metric that classifies an error state

[agavra]

Description

This PR introduces JMX metrics that drill query errors down into more specific categories: UNKNOWN, USER and SYSTEM (open to better names) when an uncaught error is raised from KafkaStreams.

Review Guide

There's a lot in this PR, I can split it up if it helps the review but it's not that many lines of code so I figured it's digestible with a little guide:

• QueryError a throwable paired with a classification of that error (see above)
• ErrorStateListener extends StateListener to add an additional method that reacts to QueryError when its raised
• QueryErrorClassifier classifies exceptions; it also supports chaining so we can easily introduce other ones
• MissingTopicClassifier will check if any required topics are missing, and if so classifies the error as USER
• I moved registering the uncaught exceptions into the QueryMetadata where I have more information than in the KafkaStreamsBuilder
• The rest is wiring

Testing done

Caused a failure by deleting a necessary topic (foo) and then deleting the kafka streams local dir to trigger a faster error. In the logs:

[2020-05-15 10:47:35,201] ERROR Unhandled exception caught in streams thread _confluent-ksql-def
ault_query_CTAS_COUNTS_0-8d6e554e-a19b-4c68-abe7-eca385fc9630-StreamThread-3. (io.confluent.ksql
.engine.KsqlEngine:43)
org.apache.kafka.streams.errors.ProcessorStateException: task directory [/tmp/kafka-streams/_con
fluent-ksql-default_query_CTAS_COUNTS_0/0_0] doesn't exist and couldn't be created
...
[2020-05-15 10:47:35,880] WARN Query _confluent-ksql-default_query_CTAS_COUNTS_0 requires topic Aggregate-GroupBy-repartition which cannot be found. (io.confluent.ksql.query.MissingTopicClassifier:54)

I used jconsole to make sure that the JMX MBean metrics were properly set. Here's a picture of it below:

Reviewer checklist

• Ensure docs are updated if necessary. (eg. if a user visible feature is being added or changed).
• Ensure relevant issues are linked (description should include text like "Fixes #")

[rodesai]

This LGTM. How would we implement more complex classifiers? Is the idea to add more ways of combining classifiers (for example, add an or method)?

[rodesai]

meant to omit this?

[agavra]

oops I thought I removed that! thanks

[rodesai]

this is currently not used right?

[agavra]

yeah it's not used yet - I'm planning on introducing a regex classifier so that you can chain them (e.g. use MissingTopic.and(SomeRegexClassifier).and(AnotherRegexClassifier)) this isn't really a boolean "and" but rather a "andThen" - I can rename it