race condition with gossiped silences causes crash #982

iksaif · 2017-09-08T13:11:03Z

Looks like our alertmanager is crashing when handling 1000s of silences

time="2017-09-08T12:38:05Z" level=info msg="running maintenance" component=silences source="silence.go:232" 
time="2017-09-08T12:38:05Z" level=info msg="running maintenance" component=nflog source="nflog.go:286" 
time="2017-09-08T12:38:05Z" level=info msg="maintenance done" component=nflog duration=137.709µs source="nflog.go:303" 
time="2017-09-08T12:38:05Z" level=info msg="maintenance done" component=silences duration=84.744µs source="silence.go:249" 
fatal error: concurrent map iteration and map write

goroutine 1608 [running]:
runtime.throw(0xa07f7b, 0x26)
	/usr/local/go/src/runtime/panic.go:596 +0x95 fp=0xc4201e8c20 sp=0xc4201e8c00
runtime.mapiternext(0xc4201e8d70)
	/usr/local/go/src/runtime/hashmap.go:737 +0x7ee fp=0xc4201e8cd0 sp=0xc4201e8c20
github.com/prometheus/alertmanager/silence.gossipData.Encode(0xc424e69c20, 0xc82580, 0xc424e69c20, 0xc4213f37c0)
	/go/src/github.com/prometheus/alertmanager/silence/silence.go:756 +0x261 fp=0xc4201e8e48 sp=0xc4201e8cd0
github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh.(*gossipSender).deliver(0xc4233224b0, 0xc420893380, 0xc4201e8ef0, 0x1, 0x0)
	/go/src/github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh/gossip.go:144 +0x8c fp=0xc4201e8eb8 sp=0xc4201e8e48
github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh.(*gossipSender).run(0xc4233224b0, 0xc420893380, 0xc424dacae0, 0xc424dacb40)
	/go/src/github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh/gossip.go:105 +0x101 fp=0xc4201e8fc0 sp=0xc4201e8eb8
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc4201e8fc8 sp=0xc4201e8fc0
created by github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh.newGossipSender
	/go/src/github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh/gossip.go:94 +0x193

goroutine 1 [chan receive, 15 minutes]:
main.main()
	/go/src/github.com/prometheus/alertmanager/cmd/alertmanager/main.go:333 +0x1b9c

goroutine 5 [syscall, 15 minutes]:
os/signal.signal_recv(0x0)
	/usr/local/go/src/runtime/sigqueue.go:116 +0x104
os/signal.loop()
	/usr/local/go/src/os/signal/signal_unix.go:22 +0x22
created by os/signal.init.1
	/usr/local/go/src/os/signal/signal_unix.go:28 +0x41

goroutine 21 [select]:
github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh.(*localPeer).actorLoop(0xc42016ef30, 0xc42017c360)
	/go/src/github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh/local_peer.go:141 +0x120
created by github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh.newLocalPeer
	/go/src/github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh/local_peer.go:31 +0x12c

goroutine 22 [select]:
github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh.(*routes).run(0xc42017a880, 0xc42017c480, 0xc42017c4e0, 0xc42017c540)
	/go/src/github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh/routes.go:177 +0x1a3
created by github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh.newRoutes
	/go/src/github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh/routes.go:44 +0x453

goroutine 23 [select, 1 minutes]:
github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh.(*connectionMaker).queryLoop(0xc420180460, 0xc42017c5a0)
	/go/src/github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh/connection_maker.go:226 +0x15c
created by github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh.newConnectionMaker
	/go/src/github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh/connection_maker.go:75 +0x21a

goroutine 29 [select]:
github.com/prometheus/alertmanager/nflog.(*nlog).run(0xc42017aa80)
	/go/src/github.com/prometheus/alertmanager/nflog/nflog.go:308 +0x2f8
created by github.com/prometheus/alertmanager/nflog.New
	/go/src/github.com/prometheus/alertmanager/nflog/nflog.go:267 +0x35d

goroutine 51 [select]:
github.com/prometheus/alertmanager/silence.(*Silences).Maintenance(0xc4201752c0, 0xd18c2e2800, 0xc42013b110, 0xd, 0xc42017c600)
	/go/src/github.com/prometheus/alertmanager/silence/silence.go:254 +0x2e5
main.main.func3(0xc4201752c0, 0xc420172410, 0xc42017c600, 0xc420172a30)
	/go/src/github.com/prometheus/alertmanager/cmd/alertmanager/main.go:178 +0xb1
created by main.main
	/go/src/github.com/prometheus/alertmanager/cmd/alertmanager/main.go:180 +0xf62

goroutine 52 [IO wait, 2 minutes]:
net.runtime_pollWait(0x7ff41e7c6ee0, 0x72, 0x0)
	/usr/local/go/src/runtime/netpoll.go:164 +0x59
net.(*pollDesc).wait(0xc420180a78, 0x72, 0x0, 0xc421db8320)
	/usr/local/go/src/net/fd_poll_runtime.go:75 +0x38
net.(*pollDesc).waitRead(0xc420180a78, 0xffffffffffffffff, 0x0)
	/usr/local/go/src/net/fd_poll_runtime.go:80 +0x34
net.(*netFD).accept(0xc420180a10, 0x0, 0xc7ea00, 0xc421db8320)
	/usr/local/go/src/net/fd_unix.go:430 +0x1e5
net.(*TCPListener).accept(0xc420176078, 0xc4203eef68, 0x85ef7f, 0xc42016f110)
	/usr/local/go/src/net/tcpsock_posix.go:136 +0x2e
net.(*TCPListener).AcceptTCP(0xc420176078, 0xc421628010, 0x0, 0x0)
	/usr/local/go/src/net/tcpsock.go:215 +0x49
github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh.(*Router).listenTCP.func1(0xc420176078, 0xc4201860e0)
	/go/src/github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh/router.go:113 +0x59
created by github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh.(*Router).listenTCP
	/go/src/github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh/router.go:121 +0x16a

goroutine 56 [select, 15 minutes]:
github.com/prometheus/alertmanager/provider/mem.(*Alerts).runGC(0xc420175720)
	/go/src/github.com/prometheus/alertmanager/provider/mem/mem.go:54 +0x259
created by github.com/prometheus/alertmanager/provider/mem.NewAlerts
	/go/src/github.com/prometheus/alertmanager/provider/mem/mem.go:47 +0x15f

goroutine 114 [select]:
github.com/prometheus/alertmanager/inhibit.(*Inhibitor).runGC(0xc4202cb020)
	/go/src/github.com/prometheus/alertmanager/inhibit/inhibit.go:55 +0x160
created by github.com/prometheus/alertmanager/inhibit.(*Inhibitor).Run
	/go/src/github.com/prometheus/alertmanager/inhibit/inhibit.go:72 +0xbf

goroutine 229 [IO wait]:
net.runtime_pollWait(0x7ff41e7c6a60, 0x72, 0x0)
	/usr/local/go/src/runtime/netpoll.go:164 +0x59
net.(*pollDesc).wait(0xc4202ab028, 0x72, 0x0, 0xc425151f40)
	/usr/local/go/src/net/fd_poll_runtime.go:75 +0x38
net.(*pollDesc).waitRead(0xc4202ab028, 0xffffffffffffffff, 0x0)
	/usr/local/go/src/net/fd_poll_runtime.go:80 +0x34
net.(*netFD).accept(0xc4202aafc0, 0x0, 0xc7ea00, 0xc425151f40)
	/usr/local/go/src/net/fd_unix.go:430 +0x1e5
net.(*TCPListener).accept(0xc420506000, 0xc4207cbfc0, 0x942500, 0xffffffffffffffff)
	/usr/local/go/src/net/tcpsock_posix.go:136 +0x2e
net.(*TCPListener).AcceptTCP(0xc420506000, 0xc4201e2e58, 0xc4201e2e60, 0xc4201e2e50)
	/usr/local/go/src/net/tcpsock.go:215 +0x49
net/http.tcpKeepAliveListener.Accept(0xc420506000, 0xa15a78, 0xc4207cbf40, 0xc84f00, 0xc42016ed50)
	/usr/local/go/src/net/http/server.go:3044 +0x2f
net/http.(*Server).Serve(0xc4201ac000, 0xc84840, 0xc420506000, 0x0, 0x0)
	/usr/local/go/src/net/http/server.go:2643 +0x228
net/http.(*Server).ListenAndServe(0xc4201ac000, 0xc4201ac000, 0x70e701)
	/usr/local/go/src/net/http/server.go:2585 +0xb0
net/http.ListenAndServe(0x7ffde51d23ba, 0x6, 0xc7db40, 0xc4203ce1a0, 0xc4203c67b8, 0x7231b9)
	/usr/local/go/src/net/http/server.go:2787 +0x7f
main.listen(0x7ffde51d23ba, 0x6, 0xc4203ce1a0)
	/go/src/github.com/prometheus/alertmanager/cmd/alertmanager/main.go:437 +0x4f
created by main.main
	/go/src/github.com/prometheus/alertmanager/cmd/alertmanager/main.go:309 +0x198b

goroutine 238 [select]:
github.com/prometheus/alertmanager/dispatch.(*aggrGroup).run(0xc4211e59e0, 0xc42113bf20)
	/go/src/github.com/prometheus/alertmanager/dispatch/dispatch.go:344 +0xb5d
created by github.com/prometheus/alertmanager/dispatch.(*Dispatcher).processAlert
	/go/src/github.com/prometheus/alertmanager/dispatch/dispatch.go:265 +0x32c

goroutine 98 [chan receive, 15 minutes]:
github.com/prometheus/alertmanager/provider/mem.(*Alerts).Subscribe.func1(0xc420175720, 0x0, 0xc42035a240, 0xd85c18, 0x0, 0x0, 0xc42035a2a0)
	/go/src/github.com/prometheus/alertmanager/provider/mem/mem.go:114 +0x19b
created by github.com/prometheus/alertmanager/provider/mem.(*Alerts).Subscribe
	/go/src/github.com/prometheus/alertmanager/provider/mem/mem.go:115 +0x1aa

goroutine 59 [select]:
github.com/prometheus/alertmanager/dispatch.(*Dispatcher).run(0xc420189440, 0xc83680, 0xc4202a8fe0)
	/go/src/github.com/prometheus/alertmanager/dispatch/dispatch.go:174 +0x6d5
github.com/prometheus/alertmanager/dispatch.(*Dispatcher).Run(0xc420189440)
	/go/src/github.com/prometheus/alertmanager/dispatch/dispatch.go:68 +0x158
created by main.main.func7
	/go/src/github.com/prometheus/alertmanager/cmd/alertmanager/main.go:279 +0x54a

goroutine 60 [select]:
github.com/prometheus/alertmanager/inhibit.(*Inhibitor).Run(0xc4202cb020)
	/go/src/github.com/prometheus/alertmanager/inhibit/inhibit.go:78 +0x352
created by main.main.func7
	/go/src/github.com/prometheus/alertmanager/cmd/alertmanager/main.go:280 +0x572

goroutine 115 [chan receive, 15 minutes]:
github.com/prometheus/alertmanager/provider/mem.(*Alerts).Subscribe.func1(0xc420175720, 0x1, 0xc420368420, 0xd85c18, 0x0, 0x0, 0xc420368480)
	/go/src/github.com/prometheus/alertmanager/provider/mem/mem.go:114 +0x19b
created by github.com/prometheus/alertmanager/provider/mem.(*Alerts).Subscribe
	/go/src/github.com/prometheus/alertmanager/provider/mem/mem.go:115 +0x1aa

goroutine 230 [select, 15 minutes, locked to thread]:
runtime.gopark(0xa16078, 0x0, 0x9f2b8f, 0x6, 0x18, 0x2)
	/usr/local/go/src/runtime/proc.go:271 +0x13a
runtime.selectgoImpl(0xc4203eff50, 0x0, 0x18)
	/usr/local/go/src/runtime/select.go:423 +0x1364
runtime.selectgo(0xc4203eff50)
	/usr/local/go/src/runtime/select.go:238 +0x1c
runtime.ensureSigM.func1()
	/usr/local/go/src/runtime/signal_unix.go:434 +0x2dd
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:2197 +0x1

goroutine 231 [select, 15 minutes]:
main.main.func8(0xc42017cba0, 0xc42017cb40, 0xc4202cb1a0, 0xc420188480)
	/go/src/github.com/prometheus/alertmanager/cmd/alertmanager/main.go:322 +0x108
created by main.main
	/go/src/github.com/prometheus/alertmanager/cmd/alertmanager/main.go:328 +0x1b65

goroutine 12601 [select]:
github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh.(*LocalConnection).actorLoop(0xc4218d4160, 0xc42635c480, 0xc42635c4e0, 0xc7dcc0, 0xc4210f7810)
	/go/src/github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh/connection.go:366 +0x3ec
github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh.(*LocalConnection).run(0xc4218d4160, 0xc42635c480, 0xc42635c4e0, 0xc42635c540, 0x1)
	/go/src/github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh/connection.go:268 +0x6fc
created by github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh.startLocalConnection
	/go/src/github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh/connection.go:102 +0x330

goroutine 10598 [select, 1 minutes]:
github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh.(*gossipSender).run(0xc4230f09b0, 0xc4217094a0, 0xc421709ce0, 0xc421709d40)
	/go/src/github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh/gossip.go:101 +0x21f
created by github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh.newGossipSender
	/go/src/github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh/gossip.go:94 +0x193

goroutine 116 [select]:
github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh.(*LocalConnection).actorLoop(0xc420926420, 0xc420f44780, 0xc420f447e0, 0xc7dcc0, 0xc4209fca10)
	/go/src/github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh/connection.go:366 +0x3ec
github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh.(*LocalConnection).run(0xc420926420, 0xc420f44780, 0xc420f447e0, 0xc420f44840, 0x1)
	/go/src/github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh/connection.go:268 +0x6fc
created by github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh.startLocalConnection
	/go/src/github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh/connection.go:102 +0x330

goroutine 335 [select]:
github.com/prometheus/alertmanager/dispatch.(*aggrGroup).run(0xc4216c9560, 0xc4216ad090)
	/go/src/github.com/prometheus/alertmanager/dispatch/dispatch.go:344 +0xb5d
created by github.com/prometheus/alertmanager/dispatch.(*Dispatcher).processAlert
	/go/src/github.com/prometheus/alertmanager/dispatch/dispatch.go:265 +0x32c

goroutine 47 [select]:
github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh.(*gossipSender).run(0xc421e62230, 0xc420f44840, 0xc421a196e0, 0xc421a19740)
	/go/src/github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh/gossip.go:101 +0x21f
created by github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh.newGossipSender
	/go/src/github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh/gossip.go:94 +0x193

goroutine 48 [select]:
github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh.(*gossipSender).run(0xc421e622d0, 0xc420f44840, 0xc421a197a0, 0xc421a19800)
	/go/src/github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh/gossip.go:101 +0x21f
created by github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh.newGossipSender
	/go/src/github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh/gossip.go:94 +0x193

goroutine 321 [select, 2 minutes]:
github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh.(*gossipSender).run(0xc421e62320, 0xc420f44840, 0xc421a19860, 0xc421a198c0)
	/go/src/github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh/gossip.go:101 +0x21f
created by github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh.newGossipSender
	/go/src/github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh/gossip.go:94 +0x193

goroutine 322 [runnable]:
encoding/gob.(*Decoder).getDecEnginePtr(0xc423192700, 0x6, 0xc420178fc0, 0xc8aee0, 0x911040, 0x9)
	/usr/local/go/src/encoding/gob/decode.go:1136 +0x2b6
encoding/gob.(*Decoder).decodeValue(0xc423192700, 0x6, 0x911040, 0xc424cdf450, 0x16)
	/usr/local/go/src/encoding/gob/decode.go:1187 +0xeb
encoding/gob.(*Decoder).DecodeValue(0xc423192700, 0x911040, 0xc424cdf450, 0x16, 0x0, 0x0)
	/usr/local/go/src/encoding/gob/decoder.go:210 +0x147
encoding/gob.(*Decoder).Decode(0xc423192700, 0x911040, 0xc424cdf450, 0xc424cdf440, 0x9)
	/usr/local/go/src/encoding/gob/decoder.go:185 +0x16d
github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh.(*Router).handleGossip(0xc4201860e0, 0x4, 0xc422da6241, 0xb7, 0xb7, 0xbf5b546, 0xd694a0)
	/go/src/github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh/router.go:176 +0x27f
github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh.(*LocalConnection).handleProtocolMsg(0xc420926420, 0xc422da6204, 0xc422da6241, 0xb7, 0xb7, 0x0, 0x0)
	/go/src/github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh/connection.go:454 +0x18f
github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh.(*LocalConnection).receiveTCP(0xc420926420, 0xc7dcc0, 0xc4209fca10)
	/go/src/github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh/connection.go:441 +0x100
created by github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh.(*LocalConnection).run
	/go/src/github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh/connection.go:259 +0x6d1

goroutine 336 [select]:
github.com/prometheus/alertmanager/dispatch.(*aggrGroup).run(0xc4216c9710, 0xc4216ad150)
	/go/src/github.com/prometheus/alertmanager/dispatch/dispatch.go:344 +0xb5d
created by github.com/prometheus/alertmanager/dispatch.(*Dispatcher).processAlert
	/go/src/github.com/prometheus/alertmanager/dispatch/dispatch.go:265 +0x32c

goroutine 401 [select]:
github.com/prometheus/alertmanager/dispatch.(*aggrGroup).run(0xc4216c98c0, 0xc4216ad210)
)
	/go/src/github.com/prometheus/alertmanager/dispatch/dispatch.go:344 +0xb5d
created by github.com/prometheus/alertmanager/dispatch.(*Dispatcher).processAlert
	/go/src/github.com/prometheus/alertmanager/dispatch/dispatch.go:265 +0x32c

goroutine 331 [select]:
github.com/prometheus/alertmanager/dispatch.(*aggrGroup).run(0xc4216c8ea0, 0xc4216acd90)
	/go/src/github.com/prometheus/alertmanager/dispatch/dispatch.go:344 +0xb5d
created by github.com/prometheus/alertmanager/dispatch.(*Dispatcher).processAlert
	/go/src/github.com/prometheus/alertmanager/dispatch/dispatch.go:265 +0x32c

goroutine 1578 [select]:
github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh.(*LocalConnection).actorLoop(0xc4246b2000, 0xc4208932c0, 0xc420893320, 0xc7dcc0, 0xc4213f2f10)
	/go/src/github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh/connection.go:366 +0x3ec
github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh.(*LocalConnection).run(0xc4246b2000, 0xc4208932c0, 0xc420893320, 0xc420893380, 0x1)
	/go/src/github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh/connection.go:268 +0x6fc
created by github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh.startLocalConnection
	/go/src/github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh/connection.go:102 +0x330

goroutine 287 [select]:
github.com/prometheus/alertmanager/dispatch.(*aggrGroup).run(0xc4211d6000, 0xc4207dc230)
	/go/src/github.com/prometheus/alertmanager/dispatch/dispatch.go:344 +0xb5d
created by github.com/prometheus/alertmanager/dispatch.(*Dispatcher).processAlert
	/go/src/github.com/prometheus/alertmanager/dispatch/dispatch.go:265 +0x32c

goroutine 334 [select]:
github.com/prometheus/alertmanager/dispatch.(*aggrGroup).run(0xc4216c93b0, 0xc4216acfd0)
	/go/src/github.com/prometheus/alertmanager/dispatch/dispatch.go:344 +0xb5d
created by github.com/prometheus/alertmanager/dispatch.(*Dispatcher).processAlert
	/go/src/github.com/prometheus/alertmanager/dispatch/dispatch.go:265 +0x32c

goroutine 357 [select]:
github.com/prometheus/alertmanager/dispatch.(*aggrGroup).run(0xc42204f7a0, 0xc421e8a6a0)
	/go/src/github.com/prometheus/alertmanager/dispatch/dispatch.go:344 +0xb5d
created by github.com/prometheus/alertmanager/dispatch.(*Dispatcher).processAlert
	/go/src/github.com/prometheus/alertmanager/dispatch/dispatch.go:265 +0x32c

goroutine 332 [select]:
github.com/prometheus/alertmanager/dispatch.(*aggrGroup).run(0xc4216c9050, 0xc4216ace50)
	/go/src/github.com/prometheus/alertmanager/dispatch/dispatch.go:344 +0xb5d
created by github.com/prometheus/alertmanager/dispatch.(*Dispatcher).processAlert
	/go/src/github.com/prometheus/alertmanager/dispatch/dispatch.go:265 +0x32c

goroutine 333 [select]:
github.com/prometheus/alertmanager/dispatch.(*aggrGroup).run(0xc4216c9200, 0xc4216acf10)
	/go/src/github.com/prometheus/alertmanager/dispatch/dispatch.go:344 +0xb5d
created by github.com/prometheus/alertmanager/dispatch.(*Dispatcher).processAlert
	/go/src/github.com/prometheus/alertmanager/dispatch/dispatch.go:265 +0x32c

goroutine 358 [select]:
github.com/prometheus/alertmanager/dispatch.(*aggrGroup).run(0xc42204f950, 0xc421e8a760)
	/go/src/github.com/prometheus/alertmanager/dispatch/dispatch.go:344 +0xb5d
created by github.com/prometheus/alertmanager/dispatch.(*Dispatcher).processAlert
	/go/src/github.com/prometheus/alertmanager/dispatch/dispatch.go:265 +0x32c

goroutine 359 [select]:
github.com/prometheus/alertmanager/dispatch.(*aggrGroup).run(0xc42204fa70, 0xc421e8a820)
	/go/src/github.com/prometheus/alertmanager/dispatch/dispatch.go:344 +0xb5d
created by github.com/prometheus/alertmanager/dispatch.(*Dispatcher).processAlert
	/go/src/github.com/prometheus/alertmanager/dispatch/dispatch.go:265 +0x32c

goroutine 360 [select]:
github.com/prometheus/alertmanager/dispatch.(*aggrGroup).run(0xc42204fb90, 0xc421e8a8e0)
	/go/src/github.com/prometheus/alertmanager/dispatch/dispatch.go:344 +0xb5d
created by github.com/prometheus/alertmanager/dispatch.(*Dispatcher).processAlert
	/go/src/github.com/prometheus/alertmanager/dispatch/dispatch.go:265 +0x32c

goroutine 361 [select]:
github.com/prometheus/alertmanager/dispatch.(*aggrGroup).run(0xc42204fd40, 0xc421e8a9a0)
	/go/src/github.com/prometheus/alertmanager/dispatch/dispatch.go:344 +0xb5d
created by github.com/prometheus/alertmanager/dispatch.(*Dispatcher).processAlert
	/go/src/github.com/prometheus/alertmanager/dispatch/dispatch.go:265 +0x32c

goroutine 362 [select]:
github.com/prometheus/alertmanager/dispatch.(*aggrGroup).run(0xc42204fef0, 0xc421e8aa60)
	/go/src/github.com/prometheus/alertmanager/dispatch/dispatch.go:344 +0xb5d
created by github.com/prometheus/alertmanager/dispatch.(*Dispatcher).processAlert
	/go/src/github.com/prometheus/alertmanager/dispatch/dispatch.go:265 +0x32c

goroutine 363 [select]:
github.com/prometheus/alertmanager/dispatch.(*aggrGroup).run(0xc42169c120, 0xc421e8ab20)
	/go/src/github.com/prometheus/alertmanager/dispatch/dispatch.go:344 +0xb5d
created by github.com/prometheus/alertmanager/dispatch.(*Dispatcher).processAlert
	/go/src/github.com/prometheus/alertmanager/dispatch/dispatch.go:265 +0x32c

goroutine 364 [select]:
github.com/prometheus/alertmanager/dispatch.(*aggrGroup).run(0xc42169c2d0, 0xc421e8abe0)
	/go/src/github.com/prometheus/alertmanager/dispatch/dispatch.go:344 +0xb5d
created by github.com/prometheus/alertmanager/dispatch.(*Dispatcher).processAlert
	/go/src/github.com/prometheus/alertmanager/dispatch/dispatch.go:265 +0x32c

goroutine 365 [select]:
github.com/prometheus/alertmanager/dispatch.(*aggrGroup).run(0xc42169c480, 0xc421e8aca0)
	/go/src/github.com/prometheus/alertmanager/dispatch/dispatch.go:344 +0xb5d
created by github.com/prometheus/alertmanager/dispatch.(*Dispatcher).processAlert
	/go/src/github.com/prometheus/alertmanager/dispatch/dispatch.go:265 +0x32c

goroutine 366 [select]:
github.com/prometheus/alertmanager/dispatch.(*aggrGroup).run(0xc42169c630, 0xc421e8ad60)
	/go/src/github.com/prometheus/alertmanager/dispatch/dispatch.go:344 +0xb5d
created by github.com/prometheus/alertmanager/dispatch.(*Dispatcher).processAlert
	/go/src/github.com/prometheus/alertmanager/dispatch/dispatch.go:265 +0x32c

goroutine 367 [select]:
github.com/prometheus/alertmanager/dispatch.(*aggrGroup).run(0xc42169c7e0, 0xc421e8ae20)
	/go/src/github.com/prometheus/alertmanager/dispatch/dispatch.go:344 +0xb5d
created by github.com/prometheus/alertmanager/dispatch.(*Dispatcher).processAlert
	/go/src/github.com/prometheus/alertmanager/dispatch/dispatch.go:265 +0x32c

goroutine 368 [select]:
github.com/prometheus/alertmanager/dispatch.(*aggrGroup).run(0xc42169c990, 0xc421e8aee0)
	/go/src/github.com/prometheus/alertmanager/dispatch/dispatch.go:344 +0xb5d
created by github.com/prometheus/alertmanager/dispatch.(*Dispatcher).processAlert
	/go/src/github.com/prometheus/alertmanager/dispatch/dispatch.go:265 +0x32c

goroutine 385 [select]:
github.com/prometheus/alertmanager/dispatch.(*aggrGroup).run(0xc42169cb40, 0xc421e8b040)
	/go/src/github.com/prometheus/alertmanager/dispatch/dispatch.go:344 +0xb5d
created by github.com/prometheus/alertmanager/dispatch.(*Dispatcher).processAlert
	/go/src/github.com/prometheus/alertmanager/dispatch/dispatch.go:265 +0x32c

goroutine 12584 [select, 1 minutes]:
github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh.(*gossipSender).run(0xc424d08eb0, 0xc42635c540, 0xc420909200, 0xc420909260)
	/go/src/github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh/gossip.go:101 +0x21f
created by github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh.newGossipSender
	/go/src/github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh/gossip.go:94 +0x193

goroutine 402 [select]:
github.com/prometheus/alertmanager/dispatch.(*aggrGroup).run(0xc4216c9a70, 0xc4216ad2d0)
	/go/src/github.com/prometheus/alertmanager/dispatch/dispatch.go:344 +0xb5d
created by github.com/prometheus/alertmanager/dispatch.(*Dispatcher).processAlert
	/go/src/github.com/prometheus/alertmanager/dispatch/dispatch.go:265 +0x32c

goroutine 403 [select]:
github.com/prometheus/alertmanager/dispatch.(*aggrGroup).run(0xc4216c9c20, 0xc4216ad390)
	/go/src/github.com/prometheus/alertmanager/dispatch/dispatch.go:344 +0xb5d
created by github.com/prometheus/alertmanager/dispatch.(*Dispatcher).processAlert
	/go/src/github.com/prometheus/alertmanager/dispatch/dispatch.go:265 +0x32c

goroutine 1609 [IO wait]:
net.runtime_pollWait(0x7ff41e7c6520, 0x72, 0x11)
	/usr/local/go/src/runtime/netpoll.go:164 +0x59
net.(*pollDesc).wait(0xc4252a24c8, 0x72, 0xc7ff80, 0xc7b410)
	/usr/local/go/src/net/fd_poll_runtime.go:75 +0x38
net.(*pollDesc).waitRead(0xc4252a24c8, 0xc4233aa004, 0x4)
	/usr/local/go/src/net/fd_poll_runtime.go:80 +0x34
net.(*netFD).Read(0xc4252a2460, 0xc4233aa004, 0x4, 0x4, 0x0, 0xc7ff80, 0xc7b410)
	/usr/local/go/src/net/fd_unix.go:250 +0x1b7
net.(*conn).Read(0xc422d54288, 0xc4233aa004, 0x4, 0x4, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:181 +0x70
io.ReadAtLeast(0x7ff41e4d2970, 0xc422d54288, 0xc4233aa004, 0x4, 0x4, 0x4, 0xd694a0, 0x72, 0x0)
	/usr/local/go/src/io/io.go:307 +0xa9
io.ReadFull(0x7ff41e4d2970, 0xc422d54288, 0xc4233aa004, 0x4, 0x4, 0x4, 0x0, 0xc42204bf30)
	/usr/local/go/src/io/io.go:325 +0x58
github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh.(*lengthPrefixTCPReceiver).Receive(0xc4213f2f10, 0x0, 0x0, 0x0, 0x0, 0x0)
	/go/src/github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh/protocol_crypto.go:168 +0x96
github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh.(*LocalConnection).receiveTCP(0xc4246b2000, 0xc7dcc0, 0xc4213f2f10)
	/go/src/github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh/connection.go:434 +0x56
created by github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh.(*LocalConnection).run
	/go/src/github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh/connection.go:259 +0x6d1

goroutine 506 [IO wait]:
net.runtime_pollWait(0x7ff41e7c6820, 0x72, 0x12)
	/usr/local/go/src/runtime/netpoll.go:164 +0x59
net.(*pollDesc).wait(0xc420146d88, 0x72, 0xc7ff80, 0xc7b410)
	/usr/local/go/src/net/fd_poll_runtime.go:75 +0x38
net.(*pollDesc).waitRead(0xc420146d88, 0xc420e1d000, 0x1000)
	/usr/local/go/src/net/fd_poll_runtime.go:80 +0x34
net.(*netFD).Read(0xc420146d20, 0xc420e1d000, 0x1000, 0x1000, 0x0, 0xc7ff80, 0xc7b410)
	/usr/local/go/src/net/fd_unix.go:250 +0x1b7
net.(*conn).Read(0xc421c6c178, 0xc420e1d000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:181 +0x70
net/http.(*connReader).Read(0xc4216d7700, 0xc420e1d000, 0x1000, 0x1000, 0x0, 0x2, 0x0)
	/usr/local/go/src/net/http/server.go:754 +0x140
bufio.(*Reader).fill(0xc4220ec060)
	/usr/local/go/src/bufio/bufio.go:97 +0x117
bufio.(*Reader).ReadSlice(0xc4220ec060, 0xa, 0x2, 0x0, 0x55e2f7, 0xc4218540eb, 0xc7b038)
	/usr/local/go/src/bufio/bufio.go:338 +0xbb
bufio.(*Reader).ReadLine(0xc4220ec060, 0xc420016700, 0x100, 0xf8, 0x9e3600, 0xa, 0x460742)
	/usr/local/go/src/bufio/bufio.go:367 +0x37
net/textproto.(*Reader).readLineSlice(0xc423140060, 0xc421ba3ac8, 0xc421ba3ac8, 0x40fdb8, 0x100, 0x9e3600)
	/usr/local/go/src/net/textproto/reader.go:55 +0x5f
net/textproto.(*Reader).ReadLine(0xc423140060, 0xc420016700, 0x72, 0x8000000000000000, 0x0)
	/usr/local/go/src/net/textproto/reader.go:36 +0x2f
net/http.readRequest(0xc4220ec060, 0x0, 0xc420016700, 0x0, 0x0)
	/usr/local/go/src/net/http/request.go:918 +0xa5
net/http.(*conn).readRequest(0xc42518f4a0, 0xc84e40, 0xc4216d76c0, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/http/server.go:934 +0x213
net/http.(*conn).serve(0xc42518f4a0, 0xc84e40, 0xc4216d76c0)
	/usr/local/go/src/net/http/server.go:1763 +0x49a
created by net/http.(*Server).Serve
	/usr/local/go/src/net/http/server.go:2668 +0x2ce

goroutine 25641 [select]:
github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh.(*gossipSender).run(0xc42348eaf0, 0xc421dba240, 0xc425b60360, 0xc425b603c0)
	/go/src/github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh/gossip.go:101 +0x21f
created by github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh.newGossipSender
	/go/src/github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh/gossip.go:94 +0x193

goroutine 11444 [select, 4 minutes]:
github.com/prometheus/alertmanager/dispatch.(*aggrGroup).run(0xc4207b4510, 0xc420c69330)
	/go/src/github.com/prometheus/alertmanager/dispatch/dispatch.go:344 +0xb5d
created by github.com/prometheus/alertmanager/dispatch.(*Dispatcher).processAlert
	/go/src/github.com/prometheus/alertmanager/dispatch/dispatch.go:265 +0x32c

goroutine 664 [select]:
github.com/prometheus/alertmanager/dispatch.(*aggrGroup).run(0xc4211d62d0, 0xc4246c0350)
	/go/src/github.com/prometheus/alertmanager/dispatch/dispatch.go:344 +0xb5d
created by github.com/prometheus/alertmanager/dispatch.(*Dispatcher).processAlert
	/go/src/github.com/prometheus/alertmanager/dispatch/dispatch.go:265 +0x32c

goroutine 1393 [select, 4 minutes]:
github.com/prometheus/alertmanager/dispatch.(*aggrGroup).run(0xc4204c86c0, 0xc421164f50)
	/go/src/github.com/prometheus/alertmanager/dispatch/dispatch.go:344 +0xb5d
created by github.com/prometheus/alertmanager/dispatch.(*Dispatcher).processAlert
	/go/src/github.com/prometheus/alertmanager/dispatch/dispatch.go:265 +0x32c

goroutine 1453 [select, 4 minutes]:
github.com/prometheus/alertmanager/dispatch.(*aggrGroup).run(0xc4207e45a0, 0xc4216ac4c0)
	/go/src/github.com/prometheus/alertmanager/dispatch/dispatch.go:344 +0xb5d
created by github.com/prometheus/alertmanager/dispatch.(*Dispatcher).processAlert
	/go/src/github.com/prometheus/alertmanager/dispatch/dispatch.go:265 +0x32c

goroutine 901 [select]:
github.com/prometheus/alertmanager/dispatch.(*aggrGroup).run(0xc4211d6990, 0xc4246c1580)
	/go/src/github.com/prometheus/alertmanager/dispatch/dispatch.go:344 +0xb5d
created by github.com/prometheus/alertmanager/dispatch.(*Dispatcher).processAlert
	/go/src/github.com/prometheus/alertmanager/dispatch/dispatch.go:265 +0x32c

goroutine 822 [select]:
github.com/prometheus/alertmanager/dispatch.(*aggrGroup).run(0xc4216c81b0, 0xc4233aa2a0)
	/go/src/github.com/prometheus/alertmanager/dispatch/dispatch.go:344 +0xb5d
created by github.com/prometheus/alertmanager/dispatch.(*Dispatcher).processAlert
	/go/src/github.com/prometheus/alertmanager/dispatch/dispatch.go:265 +0x32c

goroutine 1365 [select, 4 minutes]:
github.com/prometheus/alertmanager/dispatch.(*aggrGroup).run(0xc4207b4b40, 0xc421091cb0)
	/go/src/github.com/prometheus/alertmanager/dispatch/dispatch.go:344 +0xb5d
created by github.com/prometheus/alertmanager/dispatch.(*Dispatcher).processAlert
	/go/src/github.com/prometheus/alertmanager/dispatch/dispatch.go:265 +0x32c

goroutine 915 [select]:
github.com/prometheus/alertmanager/dispatch.(*aggrGroup).run(0xc4200fe990, 0xc425158160)
	/go/src/github.com/prometheus/alertmanager/dispatch/dispatch.go:344 +0xb5d
created by github.com/prometheus/alertmanager/dispatch.(*Dispatcher).processAlert
	/go/src/github.com/prometheus/alertmanager/dispatch/dispatch.go:265 +0x32c

goroutine 126 [select, 4 minutes]:
github.com/prometheus/alertmanager/dispatch.(*aggrGroup).run(0xc4207aec60, 0xc421164970)
	/go/src/github.com/prometheus/alertmanager/dispatch/dispatch.go:344 +0xb5d
created by github.com/prometheus/alertmanager/dispatch.(*Dispatcher).processAlert
	/go/src/github.com/prometheus/alertmanager/dispatch/dispatch.go:265 +0x32c

goroutine 125 [select, 4 minutes]:
github.com/prometheus/alertmanager/dispatch.(*aggrGroup).run(0xc4207ae510, 0xc421164460)
	/go/src/github.com/prometheus/alertmanager/dispatch/dispatch.go:344 +0xb5d
created by github.com/prometheus/alertmanager/dispatch.(*Dispatcher).processAlert
	/go/src/github.com/prometheus/alertmanager/dispatch/dispatch.go:265 +0x32c

goroutine 128 [select, 4 minutes]:
github.com/prometheus/alertmanager/dispatch.(*aggrGroup).run(0xc4204c8090, 0xc421164e40)
	/go/src/github.com/prometheus/alertmanager/dispatch/dispatch.go:344 +0xb5d
created by github.com/prometheus/alertmanager/dispatch.(*Dispatcher).processAlert
	/go/src/github.com/prometheus/alertmanager/dispatch/dispatch.go:265 +0x32c

goroutine 669 [select]:
github.com/prometheus/alertmanager/dispatch.(*aggrGroup).run(0xc4211d6480, 0xc4246c0e60)
	/go/src/github.com/prometheus/alertmanager/dispatch/dispatch.go:344 +0xb5d
created by github.com/prometheus/alertmanager/dispatch.(*Dispatcher).processAlert
	/go/src/github.com/prometheus/alertmanager/dispatch/dispatch.go:265 +0x32c

goroutine 1394 [select, 4 minutes]:
github.com/prometheus/alertmanager/dispatch.(*aggrGroup).run(0xc4204c8cf0, 0xc421165070)
	/go/src/github.com/prometheus/alertmanager/dispatch/dispatch.go:344 +0xb5d
created by github.com/prometheus/alertmanager/dispatch.(*Dispatcher).processAlert
	/go/src/github.com/prometheus/alertmanager/dispatch/dispatch.go:265 +0x32c

goroutine 626 [select, 4 minutes]:
github.com/prometheus/alertmanager/dispatch.(*aggrGroup).run(0xc4203d2480, 0xc42029b320)
	/go/src/github.com/prometheus/alertmanager/dispatch/dispatch.go:344 +0xb5d
created by github.com/prometheus/alertmanager/dispatch.(*Dispatcher).processAlert
	/go/src/github.com/prometheus/alertmanager/dispatch/dispatch.go:265 +0x32c

goroutine 911 [select, 4 minutes]:
github.com/prometheus/alertmanager/dispatch.(*aggrGroup).run(0xc4207a8120, 0xc4210902c0)
	/go/src/github.com/prometheus/alertmanager/dispatch/dispatch.go:344 +0xb5d
created by github.com/prometheus/alertmanager/dispatch.(*Dispatcher).processAlert
	/go/src/github.com/prometheus/alertmanager/dispatch/dispatch.go:265 +0x32c

goroutine 1607 [select]:
github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh.(*gossipSender).run(0xc423322460, 0xc420893380, 0xc424daca20, 0xc424daca80)
	/go/src/github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh/gossip.go:101 +0x21f
created by github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh.newGossipSender
	/go/src/github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh/gossip.go:94 +0x193

goroutine 127 [select, 4 minutes]:
github.com/prometheus/alertmanager/dispatch.(*aggrGroup).run(0xc4207afb90, 0xc421164d90)
	/go/src/github.com/prometheus/alertmanager/dispatch/dispatch.go:344 +0xb5d
created by github.com/prometheus/alertmanager/dispatch.(*Dispatcher).processAlert
	/go/src/github.com/prometheus/alertmanager/dispatch/dispatch.go:265 +0x32c

goroutine 1397 [select, 4 minutes]:
github.com/prometheus/alertmanager/dispatch.(*aggrGroup).run(0xc42079c360, 0xc421165960)
	/go/src/github.com/prometheus/alertmanager/dispatch/dispatch.go:344 +0xb5d
created by github.com/prometheus/alertmanager/dispatch.(*Dispatcher).processAlert
	/go/src/github.com/prometheus/alertmanager/dispatch/dispatch.go:265 +0x32c

goroutine 1606 [select]:
github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh.(*gossipSender).run(0xc423322410, 0xc420893380, 0xc424dac960, 0xc424dac9c0)
	/go/src/github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh/gossip.go:101 +0x21f
created by github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh.newGossipSender
	/go/src/github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh/gossip.go:94 +0x193

goroutine 1401 [select, 4 minutes]:
github.com/prometheus/alertmanager/dispatch.(*aggrGroup).run(0xc420188120, 0xc421165f00)
	/go/src/github.com/prometheus/alertmanager/dispatch/dispatch.go:344 +0xb5d
created by github.com/prometheus/alertmanager/dispatch.(*Dispatcher).processAlert
	/go/src/github.com/prometheus/alertmanager/dispatch/dispatch.go:265 +0x32c

goroutine 1400 [select, 4 minutes]:
github.com/prometheus/alertmanager/dispatch.(*aggrGroup).run(0xc4203b5560, 0xc421165d80)
	/go/src/github.com/prometheus/alertmanager/dispatch/dispatch.go:344 +0xb5d
created by github.com/prometheus/alertmanager/dispatch.(*Dispatcher).processAlert
	/go/src/github.com/prometheus/alertmanager/dispatch/dispatch.go:265 +0x32c

goroutine 1396 [select, 4 minutes]:
github.com/prometheus/alertmanager/dispatch.(*aggrGroup).run(0xc4204c9cb0, 0xc4211658b0)
	/go/src/github.com/prometheus/alertmanager/dispatch/dispatch.go:344 +0xb5d
created by github.com/prometheus/alertmanager/dispatch.(*Dispatcher).processAlert
	/go/src/github.com/prometheus/alertmanager/dispatch/dispatch.go:265 +0x32c

goroutine 1395 [select, 4 minutes]:
github.com/prometheus/alertmanager/dispatch.(*aggrGroup).run(0xc4204c9560, 0xc4211652c0)
	/go/src/github.com/prometheus/alertmanager/dispatch/dispatch.go:344 +0xb5d
created by github.com/prometheus/alertmanager/dispatch.(*Dispatcher).processAlert
	/go/src/github.com/prometheus/alertmanager/dispatch/dispatch.go:265 +0x32c

goroutine 1398 [select, 4 minutes]:
github.com/prometheus/alertmanager/dispatch.(*aggrGroup).run(0xc42079cd80, 0xc421165a10)
	/go/src/github.com/prometheus/alertmanager/dispatch/dispatch.go:344 +0xb5d
created by github.com/prometheus/alertmanager/dispatch.(*Dispatcher).processAlert
	/go/src/github.com/prometheus/alertmanager/dispatch/dispatch.go:265 +0x32c

goroutine 10599 [select]:
github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh.(*gossipSender).run(0xc4230f0a00, 0xc4217094a0, 0xc421709da0, 0xc421709e00)
	/go/src/github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh/gossip.go:101 +0x21f
created by github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh.newGossipSender
	/go/src/github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh/gossip.go:94 +0x193

goroutine 1399 [select, 4 minutes]:
github.com/prometheus/alertmanager/dispatch.(*aggrGroup).run(0xc4203b4090, 0xc421165c60)
	/go/src/github.com/prometheus/alertmanager/dispatch/dispatch.go:344 +0xb5d
created by github.com/prometheus/alertmanager/dispatch.(*Dispatcher).processAlert
	/go/src/github.com/prometheus/alertmanager/dispatch/dispatch.go:265 +0x32c

goroutine 12585 [select]:
github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh.(*gossipSender).run(0xc424d08f00, 0xc42635c540, 0xc4209092c0, 0xc420909320)
	/go/src/github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh/gossip.go:101 +0x21f
created by github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh.newGossipSender
	/go/src/github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh/gossip.go:94 +0x193

goroutine 10595 [select]:
github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh.(*LocalConnection).actorLoop(0xc424e0e370, 0xc4217093e0, 0xc421709440, 0xc7dcc0, 0xc4210d8010)
	/go/src/github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh/connection.go:366 +0x3ec
github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh.(*LocalConnection).run(0xc424e0e370, 0xc4217093e0, 0xc421709440, 0xc4217094a0, 0x1)
	/go/src/github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh/connection.go:268 +0x6fc
created by github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh.startLocalConnection
	/go/src/github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh/connection.go:102 +0x330

goroutine 25440 [select]:
github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh.(*LocalConnection).actorLoop(0xc421dfc000, 0xc421dba180, 0xc421dba1e0, 0xc7dcc0, 0xc420bc0b50)
	/go/src/github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh/connection.go:366 +0x3ec
github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh.(*LocalConnection).run(0xc421dfc000, 0xc421dba180, 0xc421dba1e0, 0xc421dba240, 0x1)
	/go/src/github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh/connection.go:268 +0x6fc
created by github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh.startLocalConnection
	/go/src/github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh/connection.go:102 +0x330

goroutine 1258 [select, 4 minutes]:
github.com/prometheus/alertmanager/dispatch.(*aggrGroup).run(0xc4207a4510, 0xc4210d8440)
	/go/src/github.com/prometheus/alertmanager/dispatch/dispatch.go:344 +0xb5d
created by github.com/prometheus/alertmanager/dispatch.(*Dispatcher).processAlert
	/go/src/github.com/prometheus/alertmanager/dispatch/dispatch.go:265 +0x32c

goroutine 1295 [select, 4 minutes]:
github.com/prometheus/alertmanager/dispatch.(*aggrGroup).run(0xc4207a1050, 0xc4210f6e90)
	/go/src/github.com/prometheus/alertmanager/dispatch/dispatch.go:344 +0xb5d
created by github.com/prometheus/alertmanager/dispatch.(*Dispatcher).processAlert
	/go/src/github.com/prometheus/alertmanager/dispatch/dispatch.go:265 +0x32c

goroutine 10601 [IO wait]:
net.runtime_pollWait(0x7ff41e7c65e0, 0x72, 0x13)
	/usr/local/go/src/runtime/netpoll.go:164 +0x59
net.(*pollDesc).wait(0xc4211c75d8, 0x72, 0xc7ff80, 0xc7b410)
	/usr/local/go/src/net/fd_poll_runtime.go:75 +0x38
net.(*pollDesc).waitRead(0xc4211c75d8, 0xc420c2a034, 0x4)
	/usr/local/go/src/net/fd_poll_runtime.go:80 +0x34
net.(*netFD).Read(0xc4211c7570, 0xc420c2a034, 0x4, 0x4, 0x0, 0xc7ff80, 0xc7b410)
	/usr/local/go/src/net/fd_unix.go:250 +0x1b7
net.(*conn).Read(0xc4209a9448, 0xc420c2a034, 0x4, 0x4, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:181 +0x70
io.ReadAtLeast(0x7ff41e4d2970, 0xc4209a9448, 0xc420c2a034, 0x4, 0x4, 0x4, 0xd694a0, 0x72, 0x0)
	/usr/local/go/src/io/io.go:307 +0xa9
io.ReadFull(0x7ff41e4d2970, 0xc4209a9448, 0xc420c2a034, 0x4, 0x4, 0x4, 0x0, 0xc42183ff30)
	/usr/local/go/src/io/io.go:325 +0x58
github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh.(*lengthPrefixTCPReceiver).Receive(0xc4210d8010, 0x0, 0x0, 0x0, 0x0, 0x0)
	/go/src/github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh/protocol_crypto.go:168 +0x96
github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh.(*LocalConnection).receiveTCP(0xc424e0e370, 0xc7dcc0, 0xc4210d8010)
	/go/src/github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh/connection.go:434 +0x56
created by github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh.(*LocalConnection).run
	/go/src/github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh/connection.go:259 +0x6d1

goroutine 20174 [select, 4 minutes]:
github.com/prometheus/alertmanager/dispatch.(*aggrGroup).run(0xc4208197a0, 0xc420c3b890)
	/go/src/github.com/prometheus/alertmanager/dispatch/dispatch.go:344 +0xb5d
created by github.com/prometheus/alertmanager/dispatch.(*Dispatcher).processAlert
	/go/src/github.com/prometheus/alertmanager/dispatch/dispatch.go:265 +0x32c

goroutine 10600 [runnable]:
github.com/prometheus/alertmanager/silence.gossipData.Encode(0xc424e69c20, 0xc82580, 0xc424e69c20, 0xc4233ab960)
	/go/src/github.com/prometheus/alertmanager/silence/silence.go:756 +0x9b
github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh.(*gossipSender).deliver(0xc4230f0a50, 0xc4217094a0, 0xc42118eef0, 0x1, 0x0)
	/go/src/github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh/gossip.go:144 +0x8c
github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh.(*gossipSender).run(0xc4230f0a50, 0xc4217094a0, 0xc421709e60, 0xc421709ec0)
	/go/src/github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh/gossip.go:105 +0x101
created by github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh.newGossipSender
	/go/src/github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh/gossip.go:94 +0x193

goroutine 3969 [select, 3 minutes]:
github.com/prometheus/alertmanager/dispatch.(*aggrGroup).run(0xc4220bc240, 0xc420c4b1d0)
	/go/src/github.com/prometheus/alertmanager/dispatch/dispatch.go:344 +0xb5d
created by github.com/prometheus/alertmanager/dispatch.(*Dispatcher).processAlert
	/go/src/github.com/prometheus/alertmanager/dispatch/dispatch.go:265 +0x32c

goroutine 12586 [select, 2 minutes]:
github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh.(*gossipSender).run(0xc424d08fa0, 0xc42635c540, 0xc420909380, 0xc4209093e0)
	/go/src/github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh/gossip.go:101 +0x21f
created by github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh.newGossipSender
	/go/src/github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh/gossip.go:94 +0x193

goroutine 27158 [select, 1 minutes]:
github.com/prometheus/alertmanager/dispatch.(*aggrGroup).run(0xc4211fc990, 0xc420eca1a0)
	/go/src/github.com/prometheus/alertmanager/dispatch/dispatch.go:344 +0xb5d
created by github.com/prometheus/alertmanager/dispatch.(*Dispatcher).processAlert
	/go/src/github.com/prometheus/alertmanager/dispatch/dispatch.go:265 +0x32c

goroutine 25642 [IO wait]:
net.runtime_pollWait(0x7ff41e7c60a0, 0x72, 0xc)
	/usr/local/go/src/runtime/netpoll.go:164 +0x59
net.(*pollDesc).wait(0xc4211f4688, 0x72, 0xc7ff80, 0xc7b410)
	/usr/local/go/src/net/fd_poll_runtime.go:75 +0x38
net.(*pollDesc).waitRead(0xc4211f4688, 0xc4266dc004, 0x4)
	/usr/local/go/src/net/fd_poll_runtime.go:80 +0x34
net.(*netFD).Read(0xc4211f4620, 0xc4266dc004, 0x4, 0x4, 0x0, 0xc7ff80, 0xc7b410)
	/usr/local/go/src/net/fd_unix.go:250 +0x1b7
net.(*conn).Read(0xc421628010, 0xc4266dc004, 0x4, 0x4, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:181 +0x70
io.ReadAtLeast(0x7ff41e4d2970, 0xc421628010, 0xc4266dc004, 0x4, 0x4, 0x4, 0xd694a0, 0x72, 0x0)
	/usr/local/go/src/io/io.go:307 +0xa9
io.ReadFull(0x7ff41e4d2970, 0xc421628010, 0xc4266dc004, 0x4, 0x4, 0x4, 0x0, 0xc421840f30)
	/usr/local/go/src/io/io.go:325 +0x58
github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh.(*lengthPrefixTCPReceiver).Receive(0xc420bc0b50, 0x0, 0x0, 0x0, 0x0, 0x0)
	/go/src/github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh/protocol_crypto.go:168 +0x96
github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh.(*LocalConnection).receiveTCP(0xc421dfc000, 0xc7dcc0, 0xc420bc0b50)
	/go/src/github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh/connection.go:434 +0x56
created by github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh.(*LocalConnection).run
	/go/src/github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh/connection.go:259 +0x6d1

goroutine 12587 [IO wait]:
net.runtime_pollWait(0x7ff41e7c66a0, 0x72, 0x19)
	/usr/local/go/src/runtime/netpoll.go:164 +0x59
net.(*pollDesc).wait(0xc4252a2fb8, 0x72, 0xc7ff80, 0xc7b410)
	/usr/local/go/src/net/fd_poll_runtime.go:75 +0x38
net.(*pollDesc).waitRead(0xc4252a2fb8, 0xc420c2a234, 0x4)
	/usr/local/go/src/net/fd_poll_runtime.go:80 +0x34
net.(*netFD).Read(0xc4252a2f50, 0xc420c2a234, 0x4, 0x4, 0x0, 0xc7ff80, 0xc7b410)
	/usr/local/go/src/net/fd_unix.go:250 +0x1b7
net.(*conn).Read(0xc42187e918, 0xc420c2a234, 0x4, 0x4, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:181 +0x70
io.ReadAtLeast(0x7ff41e4d2970, 0xc42187e918, 0xc420c2a234, 0x4, 0x4, 0x4, 0xd694a0, 0x72, 0x0)
	/usr/local/go/src/io/io.go:307 +0xa9
io.ReadFull(0x7ff41e4d2970, 0xc42187e918, 0xc420c2a234, 0x4, 0x4, 0x4, 0x0, 0xc42026ef30)
	/usr/local/go/src/io/io.go:325 +0x58
github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh.(*lengthPrefixTCPReceiver).Receive(0xc4210f7810, 0x0, 0x0, 0x0, 0x0, 0x0)
	/go/src/github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh/protocol_crypto.go:168 +0x96
github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh.(*LocalConnection).receiveTCP(0xc4218d4160, 0xc7dcc0, 0xc4210f7810)
	/go/src/github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh/connection.go:434 +0x56
created by github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh.(*LocalConnection).run
	/go/src/github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh/connection.go:259 +0x6d1

goroutine 25640 [select]:
github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh.(*gossipSender).run(0xc42348eaa0, 0xc421dba240, 0xc425b602a0, 0xc425b60300)
	/go/src/github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh/gossip.go:101 +0x21f
created by github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh.newGossipSender
	/go/src/github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh/gossip.go:94 +0x193

goroutine 25639 [select, 1 minutes]:
github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh.(*gossipSender).run(0xc42348ea50, 0xc421dba240, 0xc425b601e0, 0xc425b60240)
	/go/src/github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh/gossip.go:101 +0x21f
created by github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh.newGossipSender
	/go/src/github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh/gossip.go:94 +0x193
2017/09/08 12:38:32.218455 [DEBUG] (cli) receiving signal "child exited"
2017/09/08 12:38:32.218475 [INFO] (runner) child process died
Consul Template returned errors:
child process died with exit code 2

The text was updated successfully, but these errors were encountered:

stuartnelson3 · 2017-09-08T13:31:33Z

The crash reason is listed: fatal error: concurrent map iteration and map write.

It looks like it's something in our gossip implementation:

github.com/prometheus/alertmanager/silence.gossipData.Encode(0xc424e69c20, 0xc82580, 0xc424e69c20, 0xc4213f37c0)
	/go/src/github.com/prometheus/alertmanager/silence/silence.go:756 +0x261 fp=0xc4201e8e48 sp=0xc4201e8cd0
github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh.(*gossipSender).deliver(0xc4233224b0, 0xc420893380, 0xc4201e8ef0, 0x1, 0x0)
	/go/src/github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh/gossip.go:144 +0x8c fp=0xc4201e8eb8 sp=0xc4201e8e48
github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh.(*gossipSender).run(0xc4233224b0, 0xc420893380, 0xc424dacae0, 0xc424dacb40)
	/go/src/github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh/gossip.go:105 +0x101 fp=0xc4201e8fc0 sp=0xc4201e8eb8
runtime.goexit()

From the stack trace, it looks like un-coordinated access to a hash map.

stuartnelson3 · 2017-09-08T13:36:08Z

After taking a quick look, turning type gossipData map[string]*pb.MeshSilence into a struct that has an embedded sync.RWMutex and then making use of that when reading/writing should solve the problem.

I'm surprised this hasn't surfaced until now.

iksaif · 2017-09-08T13:39:23Z

Yes, I wonder if this is something in mesh or in the alertmanager implementation (which sounds simple).

github.com/prometheus/alertmanager/silence.gossipData.Encode(0xc424e69c20, 0xc82580, 0xc424e69c20, 0xc4213f37c0)
github.com/prometheus/alertmanager/silence.gossipData.Encode(0xc424e69c20, 0xc82580, 0xc424e69c20, 0xc4233ab960)

So if I read it correctly, Encode is called on the same gossipData twice. As far as I understood, this should be called only after s.gossip.GossipBroadcast(st). Sounds weird that this would call concurrent code on gossipData .

https://github.com/weaveworks/mesh/blob/50329b4166c9c7d4024470996fa55956902d7815/gossip.go#L55 doesn't seem to say anything but the examples clearly do that : https://github.com/weaveworks/mesh/blob/50329b4166c9c7d4024470996fa55956902d7815/examples/increment-only-counter/state.go#L58

stuartnelson3 · 2017-09-08T13:42:11Z

It's definitely in our implementation

iksaif · 2017-09-08T13:48:38Z

If nobody is currently on a fix, I'll take this one.

stuartnelson3 · 2017-09-08T14:26:09Z

Sounds good!

This should fix silence/silence.go prometheus#982

stuartnelson3 · 2017-09-10T12:31:41Z

@fabxc @brancz Running make test with -race (https://github.com/prometheus/alertmanager/blob/master/Makefile#L33) reveals many data races with the codebase.

Is Alertmanager receiving any attention from CoreOS in the near future? It would be great to go through and clean these up, but I'm not aware of the time necessary for the fixes. Perhaps @iksaif could expand his work in #984 :)

This should fix silence/silence.go prometheus#982

brancz · 2017-09-13T08:58:15Z

@stuartnelson3 we actually have a some time planned regarding Alertmanager robustness, so we should get to this relatively soon. I'm hoping to get started next week.

This should fix silence/silence.go #982

iksaif · 2017-09-14T18:17:20Z

Note: the patch may introduce a deadlock, I'll try to diagnose that tomorrow. It's not very common though.

iksaif · 2017-09-15T07:25:44Z

Yes .. we have a deadlock:

goroutine profile: total 2414
1146 @ 0x42c9da 0x42cabe 0x403dcd 0x403add 0x714a05 0x4599d1
#	0x714a04	github.com/prometheus/alertmanager/vendor/github.com/prometheus/client_golang/prometheus.computeApproximateRequestSize+0x154	/go/src/github.com/prometheus/alertmanager/vendor/github.com/prometheus/client_golang/prometheus/http.go:316

879 @ 0x42c9da 0x42cabe 0x43d151 0x43ce34 0x45ff3d 0x878ffa 0x89b2dd 0x89d948 0x89d245 0x722e8a 0x73b8cd 0x73509d 0x73b59c 0x670702 0x66ca72 0x4599d1
#	0x43ce33	sync.runtime_SemacquireMutex+0x33													/usr/local/go/src/runtime/sema.go:62
#	0x45ff3c	sync.(*Mutex).Lock+0x9c															/usr/local/go/src/sync/mutex.go:87
#	0x878ff9	github.com/prometheus/alertmanager/silence.(*Silences).Set+0x69										/go/src/github.com/prometheus/alertmanager/silence/silence.go:381
#	0x89b2dc	github.com/prometheus/alertmanager/api.(*API).setSilence+0xac										/go/src/github.com/prometheus/alertmanager/api/api.go:499
#	0x89d947	github.com/prometheus/alertmanager/api.(*API).(github.com/prometheus/alertmanager/api.setSilence)-fm+0x47				/go/src/github.com/prometheus/alertmanager/api/api.go:131
#	0x89d244	github.com/prometheus/alertmanager/api.(*API).Register.func1.1+0x64									/go/src/github.com/prometheus/alertmanager/api/api.go:111
#	0x722e89	github.com/prometheus/alertmanager/vendor/github.com/prometheus/client_golang/prometheus.InstrumentHandlerFuncWithOpts.func1+0x269	/go/src/github.com/prometheus/alertmanager/vendor/github.com/prometheus/client_golang/prometheus/http.go:287
#	0x73b8cc	github.com/prometheus/alertmanager/vendor/github.com/prometheus/common/route.(*Router).handle.func1+0x31c				/go/src/github.com/prometheus/alertmanager/vendor/github.com/prometheus/common/route/route.go:50
#	0x73509c	github.com/prometheus/alertmanager/vendor/github.com/julienschmidt/httprouter.(*Router).ServeHTTP+0x75c					/go/src/github.com/prometheus/alertmanager/vendor/github.com/julienschmidt/httprouter/router.go:299
#	0x73b59b	github.com/prometheus/alertmanager/vendor/github.com/prometheus/common/route.(*Router).ServeHTTP+0x4b					/go/src/github.com/prometheus/alertmanager/vendor/github.com/prometheus/common/route/route.go:88
#	0x670701	net/http.serverHandler.ServeHTTP+0x91													/usr/local/go/src/net/http/server.go:2568
#	0x66ca71	net/http.(*conn).serve+0x611														/usr/local/go/src/net/http/server.go:1825

266 @ 0x42c9da 0x42cabe 0x43d151 0x43ce34 0x45ff3d 0x87a37d 0x87db35 0x87a175 0x89b9bf 0x89d8e8 0x89d245 0x722e8a 0x73b8cd 0x73509d 0x73b59c 0x670702 0x66ca72 0x4599d1
#	0x43ce33	sync.runtime_SemacquireMutex+0x33													/usr/local/go/src/runtime/sema.go:62
#	0x45ff3c	sync.(*Mutex).Lock+0x9c															/usr/local/go/src/sync/mutex.go:87
#	0x87a37c	github.com/prometheus/alertmanager/silence.(*Silences).query+0x8c									/go/src/github.com/prometheus/alertmanager/silence/silence.go:593
#	0x87db34	github.com/prometheus/alertmanager/silence.(*Silences).Query.func1+0x114								/go/src/github.com/prometheus/alertmanager/silence/silence.go:578
#	0x87a174	github.com/prometheus/alertmanager/silence.(*Silences).Query+0xa4									/go/src/github.com/prometheus/alertmanager/silence/silence.go:579
#	0x89b9be	github.com/prometheus/alertmanager/api.(*API).listSilences+0x5e										/go/src/github.com/prometheus/alertmanager/api/api.go:549
#	0x89d8e7	github.com/prometheus/alertmanager/api.(*API).(github.com/prometheus/alertmanager/api.listSilences)-fm+0x47				/go/src/github.com/prometheus/alertmanager/api/api.go:130
#	0x89d244	github.com/prometheus/alertmanager/api.(*API).Register.func1.1+0x64									/go/src/github.com/prometheus/alertmanager/api/api.go:111
#	0x722e89	github.com/prometheus/alertmanager/vendor/github.com/prometheus/client_golang/prometheus.InstrumentHandlerFuncWithOpts.func1+0x269	/go/src/github.com/prometheus/alertmanager/vendor/github.com/prometheus/client_golang/prometheus/http.go:287
#	0x73b8cc	github.com/prometheus/alertmanager/vendor/github.com/prometheus/common/route.(*Router).handle.func1+0x31c				/go/src/github.com/prometheus/alertmanager/vendor/github.com/prometheus/common/route/route.go:50
#	0x73509c	github.com/prometheus/alertmanager/vendor/github.com/julienschmidt/httprouter.(*Router).ServeHTTP+0x75c					/go/src/github.com/prometheus/alertmanager/vendor/github.com/julienschmidt/httprouter/router.go:299
#	0x73b59b	github.com/prometheus/alertmanager/vendor/github.com/prometheus/common/route.(*Router).ServeHTTP+0x4b					/go/src/github.com/prometheus/alertmanager/vendor/github.com/prometheus/common/route/route.go:88
#	0x670701	net/http.serverHandler.ServeHTTP+0x91													/usr/local/go/src/net/http/server.go:2568
#	0x66ca71	net/http.(*conn).serve+0x611														/usr/local/go/src/net/http/server.go:1825

74 @ 0x42c9da 0x42cabe 0x43d151 0x43ce34 0x45ff3d 0x87a37d 0x87db35 0x87a175 0x88c7c9 0x88c015 0x88fcad 0x88bdbe 0x896776 0x896908 0x894e6f 0x8943ac 0x4599d1
#	0x43ce33	sync.runtime_SemacquireMutex+0x33							/usr/local/go/src/runtime/sema.go:62
#	0x45ff3c	sync.(*Mutex).Lock+0x9c									/usr/local/go/src/sync/mutex.go:87
#	0x87a37c	github.com/prometheus/alertmanager/silence.(*Silences).query+0x8c			/go/src/github.com/prometheus/alertmanager/silence/silence.go:593
#	0x87db34	github.com/prometheus/alertmanager/silence.(*Silences).Query.func1+0x114		/go/src/github.com/prometheus/alertmanager/silence/silence.go:578
#	0x87a174	github.com/prometheus/alertmanager/silence.(*Silences).Query+0xa4			/go/src/github.com/prometheus/alertmanager/silence/silence.go:579
#	0x88c7c8	github.com/prometheus/alertmanager/notify.(*SilenceStage).Exec+0x1a8			/go/src/github.com/prometheus/alertmanager/notify/notify.go:352
#	0x88c014	github.com/prometheus/alertmanager/notify.MultiStage.Exec+0xa4				/go/src/github.com/prometheus/alertmanager/notify/notify.go:259
#	0x88fcac	github.com/prometheus/alertmanager/notify.(*MultiStage).Exec+0xac			<autogenerated>:6
#	0x88bdbd	github.com/prometheus/alertmanager/notify.RoutingStage.Exec+0xdd			/go/src/github.com/prometheus/alertmanager/notify/notify.go:245
#	0x896775	github.com/prometheus/alertmanager/dispatch.(*Dispatcher).processAlert.func1+0x85	/go/src/github.com/prometheus/alertmanager/dispatch/dispatch.go:260
#	0x896907	github.com/prometheus/alertmanager/dispatch.(*aggrGroup).run.func1+0x57			/go/src/github.com/prometheus/alertmanager/dispatch/dispatch.go:368
#	0x894e6e	github.com/prometheus/alertmanager/dispatch.(*aggrGroup).flush+0x47e			/go/src/github.com/prometheus/alertmanager/dispatch/dispatch.go:428
#	0x8943ab	github.com/prometheus/alertmanager/dispatch.(*aggrGroup).run+0x88b			/go/src/github.com/prometheus/alertmanager/dispatch/dispatch.go:369

13 @ 0x42c9da 0x43bc64 0x43a8cc 0x849aff 0x4599d1
#	0x849afe	github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh.(*gossipSender).run+0x21e	/go/src/github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh/gossip.go:101

5 @ 0x42c9da 0x43bc64 0x43a8cc 0x8466ac 0x844c2c 0x4599d1
#	0x8466ab	github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh.(*LocalConnection).actorLoop+0x3eb	/go/src/github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh/connection.go:366
#	0x844c2b	github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh.(*LocalConnection).run+0x6fb	/go/src/github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh/connection.go:268

3 @ 0x42c9da 0x42cabe 0x43d151 0x43ce34 0x45ff3d 0x87b223 0x84ad2d 0x85a3a2 0x846fdf 0x846e00 0x4599d1
#	0x43ce33	sync.runtime_SemacquireMutex+0x33										/usr/local/go/src/runtime/sema.go:62
#	0x45ff3c	sync.(*Mutex).Lock+0x9c												/usr/local/go/src/sync/mutex.go:87
#	0x87b222	github.com/prometheus/alertmanager/silence.gossiper.OnGossip+0xa2						/go/src/github.com/prometheus/alertmanager/silence/silence.go:702
#	0x84ad2c	github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh.(*gossipChannel).deliver+0xbc		/go/src/github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh/gossip_channel.go:66
#	0x85a3a1	github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh.(*Router).handleGossip+0x491		/go/src/github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh/router.go:190
#	0x846fde	github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh.(*LocalConnection).handleProtocolMsg+0x18e	/go/src/github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh/connection.go:454
#	0x846dff	github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh.(*LocalConnection).receiveTCP+0xff		/go/src/github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh/connection.go:441

2 @ 0x42c9da 0x427af7 0x427139 0x4e7bf8 0x4e7c64 0x4e93b7 0x4fc460 0x666c28 0x4599d1
#	0x427138	net.runtime_pollWait+0x58			/usr/local/go/src/runtime/netpoll.go:164
#	0x4e7bf7	net.(*pollDesc).wait+0x37			/usr/local/go/src/net/fd_poll_runtime.go:75
#	0x4e7c63	net.(*pollDesc).waitRead+0x33			/usr/local/go/src/net/fd_poll_runtime.go:80
#	0x4e93b6	net.(*netFD).Read+0x1b6				/usr/local/go/src/net/fd_unix.go:250
#	0x4fc45f	net.(*conn).Read+0x6f				/usr/local/go/src/net/net.go:181
#	0x666c27	net/http.(*connReader).backgroundRead+0x57	/usr/local/go/src/net/http/server.go:656

2 @ 0x42c9da 0x42cabe 0x404e31 0x404a65 0x89f20b 0x4599d1
#	0x89f20a	github.com/prometheus/alertmanager/provider/mem.(*Alerts).Subscribe.func1+0x19a	/go/src/github.com/prometheus/alertmanager/provider/mem/mem.go:114

2 @ 0x42c9da 0x42cabe 0x43d151 0x43cd54 0x460f2e 0x87c074 0x84a0d9 0x84b628 0x84ad89 0x85a3a2 0x846fdf 0x846e00 0x4599d1
#	0x43cd53	sync.runtime_Semacquire+0x33											/usr/local/go/src/runtime/sema.go:47
#	0x460f2d	sync.(*RWMutex).Lock+0x6d											/usr/local/go/src/sync/rwmutex.go:91
#	0x87c073	github.com/prometheus/alertmanager/silence.(*gossipData).Merge+0xd3						/go/src/github.com/prometheus/alertmanager/silence/silence.go:810
#	0x84a0d8	github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh.(*gossipSender).Send+0xe8			/go/src/github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh/gossip.go:183
#	0x84b627	github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh.(*gossipChannel).relay+0x157		/go/src/github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh/gossip_channel.go:117
#	0x84ad88	github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh.(*gossipChannel).deliver+0x118		/go/src/github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh/gossip_channel.go:70
#	0x85a3a1	github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh.(*Router).handleGossip+0x491		/go/src/github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh/router.go:190
#	0x846fde	github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh.(*LocalConnection).handleProtocolMsg+0x18e	/go/src/github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh/connection.go:454
#	0x846dff	github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh.(*LocalConnection).receiveTCP+0xff		/go/src/github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh/connection.go:441

2 @ 0x42c9da 0x42cabe 0x43d151 0x43ce34 0x45ff3d 0x849d82 0x849b85 0x8499e1 0x4599d1
#	0x43ce33	sync.runtime_SemacquireMutex+0x33									/usr/local/go/src/runtime/sema.go:62
#	0x45ff3c	sync.(*Mutex).Lock+0x9c											/usr/local/go/src/sync/mutex.go:87
#	0x849d81	github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh.(*gossipSender).pick+0x71		/go/src/github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh/gossip.go:154
#	0x849b84	github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh.(*gossipSender).deliver+0x64	/go/src/github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh/gossip.go:140
#	0x8499e0	github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh.(*gossipSender).run+0x100		/go/src/github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh/gossip.go:105

2 @ 0x42c9da 0x43bc64 0x43a8cc 0x73a342 0x4599d1
#	0x73a341	github.com/prometheus/alertmanager/vendor/golang.org/x/net/context.propagateCancel.func1+0x171	/go/src/github.com/prometheus/alertmanager/vendor/golang.org/x/net/context/context.go:244

1 @ 0x40e412 0x4408d4 0x6990f2 0x4599d1
#	0x4408d3	os/signal.signal_recv+0x103	/usr/local/go/src/runtime/sigqueue.go:116
#	0x6990f1	os/signal.loop+0x21		/usr/local/go/src/os/signal/signal_unix.go:22

1 @ 0x42c9da 0x427af7 0x427139 0x4e7bf8 0x4e7c64 0x4e93b7 0x4fc460 0x667120 0x55c0a7 0x55d02b 0x55d217 0x5ebd0f 0x5ebb0f 0x661115 0x6684f3 0x66c8fa 0x4599d1
#	0x427138	net.runtime_pollWait+0x58			/usr/local/go/src/runtime/netpoll.go:164
#	0x4e7bf7	net.(*pollDesc).wait+0x37			/usr/local/go/src/net/fd_poll_runtime.go:75
#	0x4e7c63	net.(*pollDesc).waitRead+0x33			/usr/local/go/src/net/fd_poll_runtime.go:80
#	0x4e93b6	net.(*netFD).Read+0x1b6				/usr/local/go/src/net/fd_unix.go:250
#	0x4fc45f	net.(*conn).Read+0x6f				/usr/local/go/src/net/net.go:181
#	0x66711f	net/http.(*connReader).Read+0x13f		/usr/local/go/src/net/http/server.go:754
#	0x55c0a6	bufio.(*Reader).fill+0x116			/usr/local/go/src/bufio/bufio.go:97
#	0x55d02a	bufio.(*Reader).ReadSlice+0xba			/usr/local/go/src/bufio/bufio.go:338
#	0x55d216	bufio.(*Reader).ReadLine+0x36			/usr/local/go/src/bufio/bufio.go:367
#	0x5ebd0e	net/textproto.(*Reader).readLineSlice+0x5e	/usr/local/go/src/net/textproto/reader.go:55
#	0x5ebb0e	net/textproto.(*Reader).ReadLine+0x2e		/usr/local/go/src/net/textproto/reader.go:36
#	0x661114	net/http.readRequest+0xa4			/usr/local/go/src/net/http/request.go:918
#	0x6684f2	net/http.(*conn).readRequest+0x212		/usr/local/go/src/net/http/server.go:934
#	0x66c8f9	net/http.(*conn).serve+0x499			/usr/local/go/src/net/http/server.go:1763

1 @ 0x42c9da 0x427af7 0x427139 0x4e7bf8 0x4e7c64 0x4eabd5 0x5077de 0x505a49 0x671e1f 0x670b68 0x670850 0x67155f 0x8c740f 0x4599d1
#	0x427138	net.runtime_pollWait+0x58			/usr/local/go/src/runtime/netpoll.go:164
#	0x4e7bf7	net.(*pollDesc).wait+0x37			/usr/local/go/src/net/fd_poll_runtime.go:75
#	0x4e7c63	net.(*pollDesc).waitRead+0x33			/usr/local/go/src/net/fd_poll_runtime.go:80
#	0x4eabd4	net.(*netFD).accept+0x1e4			/usr/local/go/src/net/fd_unix.go:430
#	0x5077dd	net.(*TCPListener).accept+0x2d			/usr/local/go/src/net/tcpsock_posix.go:136
#	0x505a48	net.(*TCPListener).AcceptTCP+0x48		/usr/local/go/src/net/tcpsock.go:215
#	0x671e1e	net/http.tcpKeepAliveListener.Accept+0x2e	/usr/local/go/src/net/http/server.go:3044
#	0x670b67	net/http.(*Server).Serve+0x227			/usr/local/go/src/net/http/server.go:2643
#	0x67084f	net/http.(*Server).ListenAndServe+0xaf		/usr/local/go/src/net/http/server.go:2585
#	0x67155e	net/http.ListenAndServe+0x7e			/usr/local/go/src/net/http/server.go:2787
#	0x8c740e	main.listen+0x4e				/go/src/github.com/prometheus/alertmanager/cmd/alertmanager/main.go:437

1 @ 0x42c9da 0x427af7 0x427139 0x4e7bf8 0x4e7c64 0x4eabd5 0x5077de 0x505a49 0x860d89 0x4599d1
#	0x427138	net.runtime_pollWait+0x58										/usr/local/go/src/runtime/netpoll.go:164
#	0x4e7bf7	net.(*pollDesc).wait+0x37										/usr/local/go/src/net/fd_poll_runtime.go:75
#	0x4e7c63	net.(*pollDesc).waitRead+0x33										/usr/local/go/src/net/fd_poll_runtime.go:80
#	0x4eabd4	net.(*netFD).accept+0x1e4										/usr/local/go/src/net/fd_unix.go:430
#	0x5077dd	net.(*TCPListener).accept+0x2d										/usr/local/go/src/net/tcpsock_posix.go:136
#	0x505a48	net.(*TCPListener).AcceptTCP+0x48									/usr/local/go/src/net/tcpsock.go:215
#	0x860d88	github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh.(*Router).listenTCP.func1+0x58	/go/src/github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh/router.go:113

1 @ 0x42c9da 0x42cabe 0x404e31 0x404a65 0x8c5afc 0x42c58a 0x4599d1
#	0x8c5afb	main.main+0x1b9b	/go/src/github.com/prometheus/alertmanager/cmd/alertmanager/main.go:333
#	0x42c589	runtime.main+0x209	/usr/local/go/src/runtime/proc.go:185

1 @ 0x42c9da 0x42cabe 0x43d151 0x43ce34 0x45ff3d 0x84a1b1 0x84b495 0x84b00f 0x878e5a 0x879164 0x89b2dd 0x89d948 0x89d245 0x722e8a 0x73b8cd 0x73509d 0x73b59c 0x670702 0x66ca72 0x4599d1
#	0x43ce33	sync.runtime_SemacquireMutex+0x33													/usr/local/go/src/runtime/sema.go:62
#	0x45ff3c	sync.(*Mutex).Lock+0x9c															/usr/local/go/src/sync/mutex.go:87
#	0x84a1b0	github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh.(*gossipSender).Broadcast+0x30					/go/src/github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh/gossip.go:190
#	0x84b494	github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh.(*gossipChannel).relayBroadcast+0x164				/go/src/github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh/gossip_channel.go:110
#	0x84b00e	github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh.(*gossipChannel).GossipBroadcast+0x4e				/go/src/github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh/gossip_channel.go:83
#	0x878e59	github.com/prometheus/alertmanager/silence.(*Silences).setSilence+0x269									/go/src/github.com/prometheus/alertmanager/silence/silence.go:373
#	0x879163	github.com/prometheus/alertmanager/silence.(*Silences).Set+0x1d3									/go/src/github.com/prometheus/alertmanager/silence/silence.go:408
#	0x89b2dc	github.com/prometheus/alertmanager/api.(*API).setSilence+0xac										/go/src/github.com/prometheus/alertmanager/api/api.go:499
#	0x89d947	github.com/prometheus/alertmanager/api.(*API).(github.com/prometheus/alertmanager/api.setSilence)-fm+0x47				/go/src/github.com/prometheus/alertmanager/api/api.go:131
#	0x89d244	github.com/prometheus/alertmanager/api.(*API).Register.func1.1+0x64									/go/src/github.com/prometheus/alertmanager/api/api.go:111
#	0x722e89	github.com/prometheus/alertmanager/vendor/github.com/prometheus/client_golang/prometheus.InstrumentHandlerFuncWithOpts.func1+0x269	/go/src/github.com/prometheus/alertmanager/vendor/github.com/prometheus/client_golang/prometheus/http.go:287
#	0x73b8cc	github.com/prometheus/alertmanager/vendor/github.com/prometheus/common/route.(*Router).handle.func1+0x31c				/go/src/github.com/prometheus/alertmanager/vendor/github.com/prometheus/common/route/route.go:50
#	0x73509c	github.com/prometheus/alertmanager/vendor/github.com/julienschmidt/httprouter.(*Router).ServeHTTP+0x75c					/go/src/github.com/prometheus/alertmanager/vendor/github.com/julienschmidt/httprouter/router.go:299
#	0x73b59b	github.com/prometheus/alertmanager/vendor/github.com/prometheus/common/route.(*Router).ServeHTTP+0x4b					/go/src/github.com/prometheus/alertmanager/vendor/github.com/prometheus/common/route/route.go:88
#	0x670701	net/http.serverHandler.ServeHTTP+0x91													/usr/local/go/src/net/http/server.go:2568
#	0x66ca71	net/http.(*conn).serve+0x611														/usr/local/go/src/net/http/server.go:1825

1 @ 0x42c9da 0x42cabe 0x43d151 0x43ce34 0x45ff3d 0x8781ce 0x87d03f 0x877d4c 0x8c7dc1 0x4599d1
#	0x43ce33	sync.runtime_SemacquireMutex+0x33						/usr/local/go/src/runtime/sema.go:62
#	0x45ff3c	sync.(*Mutex).Lock+0x9c								/usr/local/go/src/sync/mutex.go:87
#	0x8781cd	github.com/prometheus/alertmanager/silence.(*Silences).GC+0xdd			/go/src/github.com/prometheus/alertmanager/silence/silence.go:281
#	0x87d03e	github.com/prometheus/alertmanager/silence.(*Silences).Maintenance.func1+0x2ee	/go/src/github.com/prometheus/alertmanager/silence/silence.go:235
#	0x877d4b	github.com/prometheus/alertmanager/silence.(*Silences).Maintenance+0x19b	/go/src/github.com/prometheus/alertmanager/silence/silence.go:258
#	0x8c7dc0	main.main.func3+0xb0								/go/src/github.com/prometheus/alertmanager/cmd/alertmanager/main.go:178

1 @ 0x42c9da 0x42cabe 0x43d151 0x43ce34 0x45ff3d 0x87b0fc 0x85a6f7 0x84cd90 0x4599d1
#	0x43ce33	sync.runtime_SemacquireMutex+0x33									/usr/local/go/src/runtime/sema.go:62
#	0x45ff3c	sync.(*Mutex).Lock+0x9c											/usr/local/go/src/sync/mutex.go:87
#	0x87b0fb	github.com/prometheus/alertmanager/silence.gossiper.Gossip+0x4b						/go/src/github.com/prometheus/alertmanager/silence/silence.go:690
#	0x85a6f6	github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh.(*Router).sendAllGossip+0xa6	/go/src/github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh/router.go:198
#	0x84cd8f	github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh.(*localPeer).actorLoop+0x10f	/go/src/github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh/local_peer.go:145

1 @ 0x42c9da 0x43bc64 0x43a8cc 0x45660d 0x4599d1
#	0x42c9d9	runtime.gopark+0x139		/usr/local/go/src/runtime/proc.go:271
#	0x43bc63	runtime.selectgoImpl+0x1363	/usr/local/go/src/runtime/select.go:423
#	0x43a8cb	runtime.selectgo+0x1b		/usr/local/go/src/runtime/select.go:238
#	0x45660c	runtime.ensureSigM.func1+0x2dc	/usr/local/go/src/runtime/signal_unix.go:434

1 @ 0x42c9da 0x43bc64 0x43a8cc 0x7c9380 0x4599d1
#	0x7c937f	github.com/prometheus/alertmanager/inhibit.(*Inhibitor).runGC+0x15f	/go/src/github.com/prometheus/alertmanager/inhibit/inhibit.go:55

1 @ 0x42c9da 0x43bc64 0x43a8cc 0x7c96e2 0x4599d1
#	0x7c96e1	github.com/prometheus/alertmanager/inhibit.(*Inhibitor).Run+0x351	/go/src/github.com/prometheus/alertmanager/inhibit/inhibit.go:78

1 @ 0x42c9da 0x43bc64 0x43a8cc 0x84868c 0x4599d1
#	0x84868b	github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh.(*connectionMaker).queryLoop+0x15b	/go/src/github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh/connection_maker.go:226

1 @ 0x42c9da 0x43bc64 0x43a8cc 0x85c893 0x4599d1
#	0x85c892	github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh.(*routes).run+0x1a2	/go/src/github.com/prometheus/alertmanager/vendor/github.com/weaveworks/mesh/routes.go:177

1 @ 0x42c9da 0x43bc64 0x43a8cc 0x8668c8 0x4599d1
#	0x8668c7	github.com/prometheus/alertmanager/nflog.(*nlog).run+0x2f7	/go/src/github.com/prometheus/alertmanager/nflog/nflog.go:308

1 @ 0x42c9da 0x43bc64 0x43a8cc 0x892e95 0x8919c8 0x4599d1
#	0x892e94	github.com/prometheus/alertmanager/dispatch.(*Dispatcher).run+0x6d4	/go/src/github.com/prometheus/alertmanager/dispatch/dispatch.go:174
#	0x8919c7	github.com/prometheus/alertmanager/dispatch.(*Dispatcher).Run+0x157	/go/src/github.com/prometheus/alertmanager/dispatch/dispatch.go:68

1 @ 0x42c9da 0x43bc64 0x43a8cc 0x89e359 0x4599d1
#	0x89e358	github.com/prometheus/alertmanager/provider/mem.(*Alerts).runGC+0x258	/go/src/github.com/prometheus/alertmanager/provider/mem/mem.go:54

1 @ 0x42c9da 0x43bc64 0x43a8cc 0x8c8908 0x4599d1
#	0x8c8907	main.main.func8+0x107	/go/src/github.com/prometheus/alertmanager/cmd/alertmanager/main.go:322

1 @ 0x8b9972 0x8b9770 0x8b5de5 0x8bd171 0x8bd483 0x66df04 0x66f340 0x8c35f8 0x73b8cd 0x73509d 0x73b59c 0x670702 0x66ca72 0x4599d1
#	0x8b9971	runtime/pprof.writeRuntimeProfile+0xa1										/usr/local/go/src/runtime/pprof/pprof.go:632
#	0x8b976f	runtime/pprof.writeGoroutine+0x9f										/usr/local/go/src/runtime/pprof/pprof.go:594
#	0x8b5de4	runtime/pprof.(*Profile).WriteTo+0x3b4										/usr/local/go/src/runtime/pprof/pprof.go:302
#	0x8bd170	net/http/pprof.handler.ServeHTTP+0x1d0										/usr/local/go/src/net/http/pprof/pprof.go:209
#	0x8bd482	net/http/pprof.Index+0x1e2											/usr/local/go/src/net/http/pprof/pprof.go:221
#	0x66df03	net/http.HandlerFunc.ServeHTTP+0x43										/usr/local/go/src/net/http/server.go:1942
#	0x66f33f	net/http.(*ServeMux).ServeHTTP+0x12f										/usr/local/go/src/net/http/server.go:2238
#	0x8c35f7	net/http.(*ServeMux).ServeHTTP-fm+0x47										/go/src/github.com/prometheus/alertmanager/ui/web.go:78
#	0x73b8cc	github.com/prometheus/alertmanager/vendor/github.com/prometheus/common/route.(*Router).handle.func1+0x31c	/go/src/github.com/prometheus/alertmanager/vendor/github.com/prometheus/common/route/route.go:50
#	0x73509c	github.com/prometheus/alertmanager/vendor/github.com/julienschmidt/httprouter.(*Router).ServeHTTP+0x75c		/go/src/github.com/prometheus/alertmanager/vendor/github.com/julienschmidt/httprouter/router.go:299
#	0x73b59b	github.com/prometheus/alertmanager/vendor/github.com/prometheus/common/route.(*Router).ServeHTTP+0x4b		/go/src/github.com/prometheus/alertmanager/vendor/github.com/prometheus/common/route/route.go:88
#	0x670701	net/http.serverHandler.ServeHTTP+0x91										/usr/local/go/src/net/http/server.go:2568
#	0x66ca71	net/http.(*conn).serve+0x611											/usr/local/go/src/net/http/server.go:1825

Calling gossip.GossipBroadcast() will cause a deadlock if there is a currently executing OnBroadcast* function. See prometheus#982

* silences: avoid deadlock Calling gossip.GossipBroadcast() will cause a deadlock if there is a currently executing OnBroadcast* function. See #982 * silence_test: better unit test to detect deadlocks

…tion to 1.16 version (prometheus#982) Add additional example of how to save old metrics Signed-off-by: Ivan Kiselev <ivan@messagebird.com>

mxinden added area/usability component/silences kind/bug priority/Pmaybe labels Sep 8, 2017

stuartnelson3 added the low hanging fruit label Sep 8, 2017

iksaif pushed a commit to criteo-forks/alertmanager that referenced this issue Sep 8, 2017

[wip] Add a mutex to silences.go:gossipData

c7a81f6

This should fix silence/silence.go prometheus#982

iksaif pushed a commit to iksaif/alertmanager that referenced this issue Sep 8, 2017

[wip] Add a mutex to silences.go:gossipData

1301a90

This should fix silence/silence.go prometheus#982

iksaif pushed a commit to iksaif/alertmanager that referenced this issue Sep 8, 2017

[wip] Add a mutex to silences.go:gossipData

8d99903

This should fix silence/silence.go prometheus#982

iksaif mentioned this issue Sep 8, 2017

Add a mutex to silences.go:gossipData #984

Merged

iksaif pushed a commit to iksaif/alertmanager that referenced this issue Sep 8, 2017

Add a mutex to silences.go:gossipData

1866bcd

This should fix silence/silence.go prometheus#982

iksaif pushed a commit to iksaif/alertmanager that referenced this issue Sep 8, 2017

Add a mutex to silences.go:gossipData

4773101

This should fix silence/silence.go prometheus#982

stuartnelson3 changed the title ~~alertmanager crashing (probably because of number of silences)~~ race condition with gossiped silences causes crash Sep 10, 2017

iksaif pushed a commit to iksaif/alertmanager that referenced this issue Sep 12, 2017

Add a mutex to silences.go:gossipData

79ac86e

This should fix silence/silence.go prometheus#982

iksaif pushed a commit to criteo-forks/alertmanager that referenced this issue Sep 12, 2017

Add a mutex to silences.go:gossipData

dce78eb

This should fix silence/silence.go prometheus#982

iksaif pushed a commit to criteo-forks/alertmanager that referenced this issue Sep 12, 2017

Add a mutex to silences.go:gossipData

eb5c6f8

This should fix silence/silence.go prometheus#982

stuartnelson3 pushed a commit that referenced this issue Sep 13, 2017

Add a mutex to silences.go:gossipData (#984)

869a038

This should fix silence/silence.go #982

iksaif pushed a commit to iksaif/alertmanager that referenced this issue Sep 15, 2017

silences: avoid deadlock

243d6bc

Calling gossip.GossipBroadcast() will cause a deadlock if there is a currently executing OnBroadcast* function. See prometheus#982

iksaif pushed a commit to iksaif/alertmanager that referenced this issue Sep 15, 2017

silences: avoid deadlock

28830e6

Calling gossip.GossipBroadcast() will cause a deadlock if there is a currently executing OnBroadcast* function. See prometheus#982

iksaif mentioned this issue Sep 15, 2017

silences: avoid deadlock #995

Merged

iksaif pushed a commit to iksaif/alertmanager that referenced this issue Sep 15, 2017

silences: avoid deadlock

b00a48c

Calling gossip.GossipBroadcast() will cause a deadlock if there is a currently executing OnBroadcast* function. See prometheus#982

iksaif pushed a commit to criteo-forks/alertmanager that referenced this issue Sep 15, 2017

silences: avoid deadlock

1dddc93

Calling gossip.GossipBroadcast() will cause a deadlock if there is a currently executing OnBroadcast* function. See prometheus#982

iksaif closed this as completed Jan 5, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

race condition with gossiped silences causes crash #982

race condition with gossiped silences causes crash #982

iksaif commented Sep 8, 2017

stuartnelson3 commented Sep 8, 2017

stuartnelson3 commented Sep 8, 2017

iksaif commented Sep 8, 2017

stuartnelson3 commented Sep 8, 2017

iksaif commented Sep 8, 2017

stuartnelson3 commented Sep 8, 2017

stuartnelson3 commented Sep 10, 2017

brancz commented Sep 13, 2017 •

edited

Loading

iksaif commented Sep 14, 2017

iksaif commented Sep 15, 2017

race condition with gossiped silences causes crash #982

race condition with gossiped silences causes crash #982

Comments

iksaif commented Sep 8, 2017

stuartnelson3 commented Sep 8, 2017

stuartnelson3 commented Sep 8, 2017

iksaif commented Sep 8, 2017

stuartnelson3 commented Sep 8, 2017

iksaif commented Sep 8, 2017

stuartnelson3 commented Sep 8, 2017

stuartnelson3 commented Sep 10, 2017

brancz commented Sep 13, 2017 • edited Loading

iksaif commented Sep 14, 2017

iksaif commented Sep 15, 2017

brancz commented Sep 13, 2017 •

edited

Loading