Requests that end up with a HystrixBadRequestException are not visible in Dashboard traffic

[pparth]

Hello,

I used HystrixBadRequestException in order to differentiate between the 4XX http errors and the 5XX ones, when wrapping an http client using a Hystrix command. I can see in the Dashboard that these requests do not count as Exceptions. But to my surprise, i also see that they do not count at all! So, we have this weird thing where the requests are properly monitored in the Pool traffic, but there is not traffic in the relevant Command. If we accept that these are client-side errors, then i would assume that they should be countered as Succeeded. Is this a bug or maybe i am missing something important in the whole rationale?

[benjchristensen]

Interesting, that would indeed be a bug.

I agree it could count as a success but I'm concerned it will confuse people. On the flip side, having another metric to track and display is not desirable.

Does anyone else have an opinion?

Shall I proceed to make these requests count against the 'Success' metric?

[pparth]

Well, IMHO, confusion is avoided only if we add another "greeny-like" metric to track. As this out of the question due to the fixes that need to be done in the stream structure and dashboard, i think that not measuring these requests is a little more confusing than adding them to Success.

[pparth]

Or at least, not add them to Success metric but add them to request rates and latency percentiles as traffic.

[benjchristensen]

The request rates are derived from Success+Timeout+ other metrics, so there isn't another top-level counter.

I don't like the idea of another "green" metric.

I'm leaning towards counting them as 'Success' but also adding another metric for detailed inspection, similar to how countFallbacks and countExceptionsThrown exist, but aren't used to represent traffic numbers.

@daveray and @abersnaze What do you think should be done here?

[daveray]

I like making it a separate detail metric. I'm already confused enough when I look at the dashboard without another green thing there.

[benjchristensen]

I agree with the sentiment, but that wouldn't solve the issue of requests not showing up in the overall rate of traffic. In other words, the RPS could be 0 but there could actually be BadRequest traffic through it without having this metric involved in the main display.

[pparth]

Anything new on that?

[benjchristensen]

Over a year old ... putting in 1.4.x in case anyone wants to pick this up.

[mattrjacobs]

#13 consider creating a separate metric for HystrixBadRequestExceptions. I will consider this issue to track the consumption of this metric, so will therefore mark it in the Dashboard milestone

[mattrjacobs]

I just added a metric to track HystrixBadRequestExceptions. If anyone is interested in adding it to the dashboard, please submit a PR.

[mattrjacobs]

The change in #776 adds the existing bad request metric to the dashboard (in the lower left-hand corner with the color of lightSeaGreen). No semantics of command operation were touched

[yoorick]

Hello,

I have a question, why the added badRequests counter still doesn't count towards totalRequestCount.
Was this an intentional decision? If yes, what's the reason behind that?

Today, in case of any significant amount of Bad Requests, you get misleading numbers in the metrics stream like that:

{
  "errorPercentage": 50,
  "errorCount": 16,
  "requestCount": 32,
  "rollingCountBadRequests": 30,
  "rollingCountExceptionsThrown": 30,
  "rollingCountFailure": 16,
  "rollingCountSuccess": 16,
  ...
}

You can see that real request count in this example is 62. (And real errorPercentage is either ~25% or ~75%, depending on how to treat bad requests, but not 50%).