[autothrottle] path optimal throttle calculations #293

jamiealquiza · 2020-02-27T00:38:03Z

Overview

Currently, autothrottle mimics the out of the box reassign-partitions throttle parameter in that a single value is assigned to both the inbound and outbound throttle rates for all brokers participating in a reassignment. The single rate value is determined by observing metrics data at each check interval and making a calculation based on the most loaded broker.

Internally, Kafka doesn't actually require that the inbound and outbound rates for a given broker are the same, nor does it require rates among brokers to be the same value. This PR creates a graph of the reassignment and observes broker utilization according to role; a "follower" in a replication is receiving data and a "leader" is sending data. A per-path rate is calculated according to the configured instance type capacity limit, an observation of current network utilization (accounting for any previously set throttles), and a ratio (separately configurable for inbound vs outbound flow) of free network capacity that's eligible for replication use. For each path, the rate is updated if it's more than n (configurable) distance from any previously set rates.

tl;dr this ensure that every replication path individually runs as fast as possible while adaptively maintaining capacity for consumers. Closes #81.

In Action

2020/02/27 22:28:12 Autothrottle Running
2020/02/27 22:28:13 Admin API: localhost:8080
2020/02/27 22:28:13 Topics with ongoing reassignments: [test0]
2020/02/27 22:28:13 Source brokers participating in replication: [1037 1039]
2020/02/27 22:28:13 Destination brokers participating in replication: [1033 1041]
2020/02/27 22:28:14 Replication throttle rate for broker 1037 [leader] (based on a 90% max free capacity utilization): 139.83MB/s
2020/02/27 22:28:14 Updated throttle on broker 1037 [leader]
2020/02/27 22:28:15 Replication throttle rate for broker 1039 [leader] (based on a 90% max free capacity utilization): 147.24MB/s
2020/02/27 22:28:15 Updated throttle on broker 1039 [leader]
2020/02/27 22:28:15 Replication throttle rate for broker 1041 [follower] (based on a 90% max free capacity utilization): 179.75MB/s
2020/02/27 22:28:15 Updated throttle on broker 1041 [follower]
2020/02/27 22:28:15 Replication throttle rate for broker 1033 [follower] (based on a 90% max free capacity utilization): 181.88MB/s
2020/02/27 22:28:15 Updated throttle on broker 1033 [follower]
2020/02/27 22:28:28 Topics with ongoing reassignments: [test0]
2020/02/27 22:28:28 Source brokers participating in replication: [1037 1039]
2020/02/27 22:28:28 Destination brokers participating in replication: [1033 1041]
2020/02/27 22:28:28 Replication throttle rate for broker 1039 [leader] (based on a 90% max free capacity utilization): 225.00MB/s
2020/02/27 22:28:28 Updated throttle on broker 1039 [leader]
2020/02/27 22:28:28 Replication throttle rate for broker 1041 [follower] (based on a 90% max free capacity utilization): 225.00MB/s
2020/02/27 22:28:28 Updated throttle on broker 1041 [follower]
2020/02/27 22:28:29 Replication throttle rate for broker 1033 [follower] (based on a 90% max free capacity utilization): 225.00MB/s
2020/02/27 22:28:29 Updated throttle on broker 1033 [follower]
2020/02/27 22:28:29 Replication throttle rate for broker 1037 [leader] (based on a 90% max free capacity utilization): 225.00MB/s
2020/02/27 22:28:29 Updated throttle on broker 1037 [leader]
2020/02/27 22:28:43 Topics with ongoing reassignments: [test0]
2020/02/27 22:28:43 Source brokers participating in replication: [1037]
2020/02/27 22:28:43 Destination brokers participating in replication: [1041]
2020/02/27 22:28:43 Replication throttle rate for broker 1037 [leader] (based on a 90% max free capacity utilization): 225.00MB/s
2020/02/27 22:28:43 Proposed throttle is within 0.00% of the previous throttle (below 10.00% threshold), skipping throttle update for broker 1037
2020/02/27 22:28:43 Replication throttle rate for broker 1041 [follower] (based on a 90% max free capacity utilization): 225.00MB/s
2020/02/27 22:28:43 Proposed throttle is within 0.00% of the previous throttle (below 10.00% threshold), skipping throttle update for broker 1041

cmd/autothrottle/throttles.go

scanterog · 2020-03-02T19:37:17Z

cmd/autothrottle/main.go

 	v := flag.Bool("version", false, "version")
 	flag.StringVar(&Config.APIKey, "api-key", "", "Datadog API key")
 	flag.StringVar(&Config.AppKey, "app-key", "", "Datadog app key")
 	flag.StringVar(&Config.NetworkTXQuery, "net-tx-query", "avg:system.net.bytes_sent{service:kafka} by {host}", "Datadog query for broker outbound bandwidth by host")
+	flag.StringVar(&Config.NetworkRXQuery, "net-rx-query", "avg:system.net.bytes_rcvd{service:kafka} by {host}", "Datadog query for broker outbound bandwidth by host")


Suggested change

flag.StringVar(&Config.NetworkRXQuery, "net-rx-query", "avg:system.net.bytes_rcvd{service:kafka} by {host}", "Datadog query for broker outbound bandwidth by host")

flag.StringVar(&Config.NetworkRXQuery, "net-rx-query", "avg:system.net.bytes_rcvd{service:kafka} by {host}", "Datadog query for broker inbound bandwidth by host")

scanterog · 2020-03-02T20:02:22Z

kafkametrics/datadog/datadog.go

-	// BrokerIDTag is the host tag name
-	// for Kafka broker IDs.
+	// NetworkRXQuery is a query string that should return the inbound
+	// network metrics by house for the reference Kafka brokers.


nit: s/house/host

scanterog · 2020-03-02T20:23:38Z

cmd/autothrottle/limits.go

-	case c.Maximum <= 0 || c.Maximum > 100:
-		return nil, errors.New("maximum must be > 0 and < 100")
+	case c.SourceMaximum <= 0 || c.SourceMaximum > 100:
+		return nil, errors.New("source maximum must be > 0 and < 100")


nit: "source maximum must be > 0 and <= 100" or in case we want < 100, set c.SourceMaximum >= 100. Same for dest.

scanterog · 2020-03-02T20:50:21Z

cmd/autothrottle/main.go

+		if replicatingNow.isSubSetOf(replicatingPreviously) {
+			throttleMeta.DisableTopicUpdates()
+		} else {
+			throttleMeta.DisableTopicUpdates()


Suggested change

throttleMeta.DisableTopicUpdates()

throttleMeta.EnableTopicUpdates()

scanterog

jamiealquiza added 30 commits February 18, 2020 18:04

applies broker throttle before topic throttle

8c3e0e0

updates kafkametrics Broker fields

266e12b

mergeBrokerList, updateBroker funcs

52ed875

datadog metrics handler fetches net rx metrics

9965938

file rename

da1b260

drops createNetTXQuery

e30277e

TestMergeBrokerLists

15e6fe2

reorganized tests

c71eb44

brokersFromSeries conditional metric type population

655401e

metric list merging

3ea8e20

adds network-rx-query flag

c19a20d

replicationCapacityForBroker, throttleByRole types

9e309b7

max-tx-rate, max-rx-rate flags

7e1de46

storeLeaderCapacity, storeFollowerCapacity

cf9da11

UpdateKafkaConfig returns []bool

771e170

applyBrokerThrottles rewrite using replicationCapacityByBroker

d287e0a

drops repCapacityByMetrics

e96ff6a

updated UpdateKafkaConfig tests

5e5713f

Dockerfile go version

0123b65

existing tests pass

6f6e426

moved flag parsing

62d6f00

drops headroom method, maximum param

195d319

TestIncompleteBrokerMetrics

d119ed1

corrected replicationHeadroom

d283c48

TestReplicationHeadroom

829428f

drops maxSrcNetTX

fb3310b

mockBrokerMetrics

d83242e

comments

baed03e

fixes reassignment

271f2b3

GetTopicStateISR

55bb547

jamiealquiza added 3 commits February 26, 2020 17:20

error

d9b0a8c

brokerReplicationCapacities

7100873

TestBrokerReplicationCapacities

ff679b1

jamiealquiza commented Feb 27, 2020

View reviewed changes

cmd/autothrottle/throttles.go Outdated Show resolved Hide resolved

jamiealquiza commented Feb 27, 2020

View reviewed changes

cmd/autothrottle/throttles.go Outdated Show resolved Hide resolved

passes broker max rates config

eac6935

jamiealquiza force-pushed the jamie/autothrottle branch from 2328e96 to eac6935 Compare February 27, 2020 20:14

jamiealquiza added 6 commits February 27, 2020 14:39

mergeBrokerLists merges brokers keyed by Host

ad0fde3

revised previous throttle tracking

a31e9fd

updated README

6203701

applyTopicThrottles returns events

f793331

applyBrokerThrottles returns events

c524c64

autothrottle writes per-broker rates in dd event

2b87e5e

jamiealquiza changed the title ~~[TESTING] [autothrottle] per-path throttle calculations~~ [TESTING] [autothrottle] path optimal throttle calculations Feb 28, 2020

jamiealquiza added 2 commits February 28, 2020 13:15

conditional topic throttled replicas updates

3a6b6c2

autothrottle skips repeat topic throttled replica updates

ef0014b

jamiealquiza force-pushed the jamie/autothrottle branch from da76cfd to ef0014b Compare March 2, 2020 20:27

scanterog reviewed Mar 2, 2020

View reviewed changes

fixes/cleanup

3107638

jamiealquiza changed the title ~~[TESTING] [autothrottle] path optimal throttle calculations~~ [autothrottle] path optimal throttle calculations Mar 4, 2020

scanterog approved these changes Mar 4, 2020

View reviewed changes

jamiealquiza merged commit 896a115 into master Mar 4, 2020

jamiealquiza deleted the jamie/autothrottle branch March 4, 2020 18:46

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[autothrottle] path optimal throttle calculations #293

[autothrottle] path optimal throttle calculations #293

jamiealquiza commented Feb 27, 2020 •

edited

Loading

scanterog Mar 2, 2020

scanterog Mar 2, 2020

scanterog Mar 2, 2020

scanterog Mar 2, 2020

scanterog left a comment

	flag.StringVar(&Config.NetworkRXQuery, "net-rx-query", "avg:system.net.bytes_rcvd{service:kafka} by {host}", "Datadog query for broker outbound bandwidth by host")
	flag.StringVar(&Config.NetworkRXQuery, "net-rx-query", "avg:system.net.bytes_rcvd{service:kafka} by {host}", "Datadog query for broker inbound bandwidth by host")

	throttleMeta.DisableTopicUpdates()
	throttleMeta.EnableTopicUpdates()

[autothrottle] path optimal throttle calculations #293

[autothrottle] path optimal throttle calculations #293

Conversation

jamiealquiza commented Feb 27, 2020 • edited Loading

Overview

In Action

scanterog Mar 2, 2020

Choose a reason for hiding this comment

scanterog Mar 2, 2020

Choose a reason for hiding this comment

scanterog Mar 2, 2020

Choose a reason for hiding this comment

scanterog Mar 2, 2020

Choose a reason for hiding this comment

scanterog left a comment

Choose a reason for hiding this comment

jamiealquiza commented Feb 27, 2020 •

edited

Loading