Raise exception during tests for OK service checks sent with messages

[yzhan289]

What does this PR do?

This PR is Part 2 of 2. See Part 1: #9888

Service checks with status OK should no longer report any messages. This PR adds a check in the aggregator to throw an exception for future integrations.

This PR will be merged in once the first part PR is merged in.

Motivation

Additional Notes

Review checklist (to be filled by reviewers)

• Feature or bugfix MUST have appropriate tests (unit, integration, e2e)
• PR title must be written as a CHANGELOG entry (see why)
• Files changes must correspond to the primary purpose of the PR as described in the title (small unrelated changes should have their own PR)
• PR must have changelog/ and integration/ labels attached

[codecov]

Codecov Report

Merging #9898 (3b38fdf) into master (fdf5936) will increase coverage by 0.02%.
The diff coverage is 33.33%.

Flag	Coverage Δ
active_directory	100.00% <ø> (ø)
activemq_xml	82.31% <ø> (ø)
aerospike	86.97% <ø> (+0.36%)	⬆️
airflow	90.00% <ø> (ø)
amazon_msk	87.94% <ø> (ø)
ambari	86.87% <ø> (ø)
apache	95.43% <ø> (ø)
aspdotnet	93.87% <ø> (ø)
avi_vantage	92.07% <ø> (ø)
azure_iot_edge	82.01% <ø> (ø)
btrfs	82.91% <ø> (ø)
cacti	83.95% <ø> (ø)
cassandra_nodetool	94.19% <ø> (ø)
ceph	91.02% <ø> (ø)
cilium	85.84% <ø> (+1.88%)	⬆️
cisco_aci	95.88% <ø> (ø)
clickhouse	96.65% <ø> (ø)
cloud_foundry_api	95.98% <ø> (+0.12%)	⬆️
cockroachdb	97.18% <ø> (ø)
consul	91.74% <ø> (ø)
coredns	96.36% <ø> (ø)
couch	95.19% <ø> (+0.24%)	⬆️
couchbase	81.45% <ø> (ø)
crio	100.00% <ø> (ø)
datadog_checks_base	89.58% <33.33%> (+0.33%)	⬆️
datadog_checks_dev	79.71% <ø> (ø)
datadog_checks_downloader	80.64% <ø> (ø)
datadog_cluster_agent	97.50% <ø> (ø)
directory	94.70% <ø> (ø)
disk	91.15% <ø> (-0.49%)	⬇️
dns_check	94.00% <ø> (ø)
dotnetclr	100.00% <ø> (ø)
druid	97.70% <ø> (ø)
ecs_fargate	77.65% <ø> (ø)
eks_fargate	94.05% <ø> (ø)
elastic	88.54% <ø> (ø)
envoy	93.77% <ø> (+0.25%)	⬆️
etcd	93.09% <ø> (ø)
exchange_server	100.00% <ø> (ø)
external_dns	100.00% <ø> (ø)
fluentd	94.77% <ø> (ø)
gearmand	77.27% <ø> (+1.29%)	⬆️
gitlab	89.94% <ø> (ø)
gitlab_runner	90.32% <ø> (ø)
glusterfs	80.09% <ø> (+0.92%)	⬆️
go_expvar	91.95% <ø> (ø)
gunicorn	93.60% <ø> (+0.75%)	⬆️
haproxy	95.22% <ø> (+0.17%)	⬆️
harbor	91.58% <ø> (ø)
hazelcast	92.39% <ø> (ø)
hdfs_datanode	90.00% <ø> (ø)
hdfs_namenode	87.94% <ø> (ø)
http_check	90.00% <ø> (+1.81%)	⬆️
ibm_db2	93.65% <ø> (ø)
ibm_mq	89.92% <ø> (ø)
ibm_was	97.44% <ø> (ø)
iis	93.01% <ø> (ø)
istio	78.91% <ø> (+0.63%)	⬆️
kafka_consumer	80.95% <ø> (ø)
kong	92.21% <ø> (ø)
kube_apiserver_metrics	97.35% <ø> (ø)
kube_controller_manager	97.14% <ø> (ø)
kube_dns	98.85% <ø> (ø)
kube_metrics_server	100.00% <ø> (ø)
kube_proxy	100.00% <ø> (ø)
kube_scheduler	96.41% <ø> (ø)
kubelet	89.47% <ø> (ø)
kubernetes_state	89.64% <ø> (ø)
kyototycoon	85.96% <ø> (ø)
lighttpd	83.64% <ø> (ø)
linkerd	85.14% <ø> (+1.14%)	⬆️
linux_proc_extras	96.22% <ø> (ø)
mapr	82.62% <ø> (ø)
mapreduce	82.27% <ø> (+0.45%)	⬆️
marathon	83.12% <ø> (ø)
marklogic	95.33% <ø> (ø)
mcache	93.37% <ø> (ø)
mesos_master	90.68% <ø> (ø)
mesos_slave	93.63% <ø> (ø)
mongo	94.74% <ø> (+0.28%)	⬆️
mysql	85.51% <ø> (+0.14%)	⬆️
nagios	89.53% <ø> (ø)
network	77.76% <ø> (+1.00%)	⬆️
nfsstat	95.20% <ø> (ø)
nginx	95.11% <ø> (+0.93%)	⬆️
nginx_ingress_controller	98.30% <ø> (ø)
openldap	96.33% <ø> (ø)
openmetrics	97.14% <ø> (ø)
openstack	51.30% <ø> (ø)
openstack_controller	90.31% <ø> (ø)
oracle	92.44% <ø> (+0.60%)	⬆️
pdh_check	97.77% <ø> (ø)
pgbouncer	90.27% <ø> (ø)
php_fpm	90.04% <ø> (+0.43%)	⬆️
postfix	88.04% <ø> (ø)
postgres	92.67% <ø> (+0.39%)	⬆️
powerdns_recursor	95.93% <ø> (ø)
process	85.20% <ø> (+0.28%)	⬆️
prometheus	94.17% <ø> (ø)
proxysql	99.62% <ø> (ø)
rabbitmq	93.74% <ø> (ø)
redisdb	87.28% <ø> (ø)
rethinkdb	97.93% <ø> (ø)
riak	99.22% <ø> (ø)
riakcs	93.61% <ø> (ø)
sap_hana	93.04% <ø> (ø)
scylla	100.00% <ø> (ø)
snmp	91.18% <ø> (+0.04%)	⬆️
snowflake	94.28% <ø> (+0.57%)	⬆️
sonarqube	95.69% <ø> (ø)
spark	93.46% <ø> (-0.18%)	⬇️
sqlserver	82.06% <ø> (ø)
squid	100.00% <ø> (ø)
ssh_check	91.58% <ø> (ø)
statsd	87.36% <ø> (+1.05%)	⬆️
supervisord	92.30% <ø> (ø)
system_core	91.04% <ø> (ø)
system_swap	98.30% <ø> (ø)
tcp_check	88.83% <ø> (ø)
teamcity	80.00% <ø> (ø)
tls	97.04% <ø> (+0.87%)	⬆️
tokumx	58.40% <ø> (?)
twemproxy	78.33% <ø> (ø)
twistlock	80.80% <ø> (ø)
varnish	84.57% <ø> (+0.24%)	⬆️
vault	94.91% <ø> (+0.53%)	⬆️
vertica	92.33% <ø> (ø)
voltdb	96.81% <ø> (ø)
vsphere	89.77% <ø> (+0.08%)	⬆️
win32_event_log	86.03% <ø> (+0.28%)	⬆️
windows_service	95.83% <ø> (ø)
wmi_check	92.91% <ø> (ø)
yarn	90.30% <ø> (ø)
zk	84.46% <ø> (+0.25%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

[djmitche]

Will this cause custom checks that still produce a message to fail?

[yzhan289]

@djmitche This will only throw exceptions in testing, so it won't affect anything in a live setting.

[djmitche]

Got it, thanks!

[yzhan289]

Merging in changes in Part 1 to rerun CI.