Emitting Kafka row lag

[Tang8330]

Motivation

Today, we are emitting ingestion lag by diffing the timestamp within the Kafka or Pub/Sub message against the current timestamp.

However, that is the ingestion age, not how long it will take to get to queue 0, where the time to queue 0 is an output from:

1. Rate of change of the row lag:
   • How many rows are we processing
   • How many rows are we adding
2. How large is the backlog (row lag)

Visually, it'll look something like this:

Today, the ingestion age will just tell you how old is the message you are processing. With this PR, we will soon have the ability to display time to queue 0 within our Analytics Portal.

Changes

1. We are calculating high watermark by diffing the current message offset along with the Kafka partition's high watermark
2. We are sending 50% of this data to Datadog such that it will only add minimal latency to data processing time