Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[0.9.2] Panic immediately after restarting due to write timeouts #3468

Closed
jhorwit2 opened this issue Jul 26, 2015 · 4 comments
Closed

[0.9.2] Panic immediately after restarting due to write timeouts #3468

jhorwit2 opened this issue Jul 26, 2015 · 4 comments
Labels
area/continuous queries area/writes

Comments

@jhorwit2
Copy link
Contributor

I noticed my writes started to timeout at 5 seconds, so I attempted to restart influx to allow it to catch up; however, upon restart it immediately paniced.

[run] 2015/07/26 13:41:34 time limit reached, initializing hard shutdown
[admin] 2015/07/26 13:41:35 listening on HTTP: 0.0.0.0:8083
[httpd] 2015/07/26 13:41:35 authentication enabled: false
[httpd] 2015/07/26 13:41:35 listening on HTTP: 0.0.0.0:8086
2015/07/26 13:41:35 InfluxDB starting, version 0.9.2, commit 6c0a91f775f9fc0e625d17ffa04a3fe86945ba09
2015/07/26 13:41:35 GOMAXPROCS set to 16
[run] 2015/07/26 13:41:35 listening for signals
2015/07/26 13:41:35 Sending anonymous usage statistics to m.influxdb.com
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x188 pc=0x70e64a]

goroutine 98 [running]:
github.com/boltdb/bolt.(*DB).beginRWTx(0x0, 0x0, 0x0, 0x0)
    /root/go/src/github.com/boltdb/bolt/db.go:458 +0x3a
github.com/boltdb/bolt.(*DB).Begin(0x0, 0xc20818d701, 0x0, 0x0, 0x0)
    /root/go/src/github.com/boltdb/bolt/db.go:413 +0x41
github.com/boltdb/bolt.(*DB).Update(0x0, 0xc20805be68, 0x0, 0x0)
    /root/go/src/github.com/boltdb/bolt/db.go:531 +0x42
github.com/influxdb/influxdb/tsdb.(*Shard).WritePoints(0xc208198420, 0xc2082fe000, 0x12c, 0x200, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/shard.go:311 +0x555
github.com/influxdb/influxdb/tsdb.(*Store).WriteToShard(0xc208046660, 0x20, 0xc2082fe000, 0x12c, 0x200, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/store.go:281 +0xf7
github.com/influxdb/influxdb/cluster.func·002(0x20, 0x1, 0xc2082fe000, 0x12c, 0x200)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:241 +0xfc
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:271 +0x30b

goroutine 1 [chan receive]:
main.(*Main).Run(0xc20808fa80, 0xc20800a010, 0x4, 0x4, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/cmd/influxd/main.go:86 +0x73b
main.main()
    /root/go/src/github.com/influxdb/influxdb/cmd/influxd/main.go:38 +0xdc

goroutine 6 [syscall]:
os/signal.loop()
    /root/.gvm/gos/go1.4.2/src/os/signal/signal_unix.go:21 +0x1f
created by os/signal.init·1
    /root/.gvm/gos/go1.4.2/src/os/signal/signal_unix.go:27 +0x35

goroutine 17 [syscall, locked to thread]:
runtime.goexit()
    /root/.gvm/gos/go1.4.2/src/runtime/asm_amd64.s:2232 +0x1

goroutine 8 [IO wait]:
net.(*pollDesc).Wait(0xc208010450, 0x72, 0x0, 0x0)
    /root/.gvm/gos/go1.4.2/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc208010450, 0x0, 0x0)
    /root/.gvm/gos/go1.4.2/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).accept(0xc2080103f0, 0x0, 0x7ff7026d6cc8, 0xc2080fc000)
    /root/.gvm/gos/go1.4.2/src/net/fd_unix.go:419 +0x40b
net.(*TCPListener).AcceptTCP(0xc20803ade0, 0x0, 0x0, 0x0)
    /root/.gvm/gos/go1.4.2/src/net/tcpsock_posix.go:234 +0x4e
net.(*TCPListener).Accept(0xc20803ade0, 0x0, 0x0, 0x0, 0x0)
    /root/.gvm/gos/go1.4.2/src/net/tcpsock_posix.go:244 +0x4c
github.com/influxdb/influxdb/tcp.(*Mux).Serve(0xc2080c1980, 0x7ff7026dd308, 0xc20803ade0, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tcp/mux.go:45 +0x57
created by github.com/influxdb/influxdb/cmd/influxd/run.func·003
    /root/go/src/github.com/influxdb/influxdb/cmd/influxd/run/server.go:298 +0x7a8

goroutine 9 [chan receive]:
github.com/influxdb/influxdb/tcp.(*listener).Accept(0xc20803ade8, 0x0, 0x0, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tcp/mux.go:117 +0x80
github.com/influxdb/influxdb/meta.(*raftLayer).Accept(0xc2080c19b0, 0x0, 0x0, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/meta/store.go:1855 +0x64
github.com/hashicorp/raft.(*NetworkTransport).listen(0xc2080104d0)
    /root/go/src/github.com/hashicorp/raft/net_transport.go:346 +0x50
created by github.com/hashicorp/raft.NewNetworkTransport
    /root/go/src/github.com/hashicorp/raft/net_transport.go:138 +0x2c4

goroutine 10 [select]:
github.com/hashicorp/raft.(*Raft).leaderLoop(0xc2080be1c0)
    /root/go/src/github.com/hashicorp/raft/raft.go:807 +0xc7c
github.com/hashicorp/raft.(*Raft).runLeader(0xc2080be1c0)
    /root/go/src/github.com/hashicorp/raft/raft.go:779 +0x74e
github.com/hashicorp/raft.(*Raft).run(0xc2080be1c0)
    /root/go/src/github.com/hashicorp/raft/raft.go:576 +0xc9
github.com/hashicorp/raft.*Raft.(github.com/hashicorp/raft.run)·fm()
    /root/go/src/github.com/hashicorp/raft/raft.go:241 +0x27
github.com/hashicorp/raft.func·011()
    /root/go/src/github.com/hashicorp/raft/state.go:152 +0x51
created by github.com/hashicorp/raft.(*raftState).goFunc
    /root/go/src/github.com/hashicorp/raft/state.go:153 +0xe3

goroutine 11 [select]:
github.com/hashicorp/raft.(*Raft).runFSM(0xc2080be1c0)
    /root/go/src/github.com/hashicorp/raft/raft.go:490 +0xf74
github.com/hashicorp/raft.*Raft.(github.com/hashicorp/raft.runFSM)·fm()
    /root/go/src/github.com/hashicorp/raft/raft.go:242 +0x27
github.com/hashicorp/raft.func·011()
    /root/go/src/github.com/hashicorp/raft/state.go:152 +0x51
created by github.com/hashicorp/raft.(*raftState).goFunc
    /root/go/src/github.com/hashicorp/raft/state.go:153 +0xe3

goroutine 12 [select]:
github.com/hashicorp/raft.(*Raft).runSnapshots(0xc2080be1c0)
    /root/go/src/github.com/hashicorp/raft/raft.go:1594 +0x466
github.com/hashicorp/raft.*Raft.(github.com/hashicorp/raft.runSnapshots)·fm()
    /root/go/src/github.com/hashicorp/raft/raft.go:243 +0x27
github.com/hashicorp/raft.func·011()
    /root/go/src/github.com/hashicorp/raft/state.go:152 +0x51
created by github.com/hashicorp/raft.(*raftState).goFunc
    /root/go/src/github.com/hashicorp/raft/state.go:153 +0xe3

goroutine 13 [chan receive]:
github.com/influxdb/influxdb/tcp.(*listener).Accept(0xc20803adf0, 0x0, 0x0, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tcp/mux.go:117 +0x80
github.com/influxdb/influxdb/meta.(*Store).serveExecListener(0xc20800c280)
    /root/go/src/github.com/influxdb/influxdb/meta/store.go:470 +0x8f
created by github.com/influxdb/influxdb/meta.(*Store).Open
    /root/go/src/github.com/influxdb/influxdb/meta/store.go:205 +0x1c2

goroutine 14 [select]:
github.com/influxdb/influxdb/cmd/influxd/run.(*Server).monitorErrorChan(0xc2080aa0e0, 0xc2080465a0)
    /root/go/src/github.com/influxdb/influxdb/cmd/influxd/run/server.go:421 +0x14a
created by github.com/influxdb/influxdb/cmd/influxd/run.func·003
    /root/go/src/github.com/influxdb/influxdb/cmd/influxd/run/server.go:304 +0x919

goroutine 34 [select]:
github.com/influxdb/influxdb/tsdb.(*Shard).autoflusher(0xc208136630, 0xc2081d6b40)
    /root/go/src/github.com/influxdb/influxdb/tsdb/shard.go:468 +0x39d
created by github.com/influxdb/influxdb/tsdb.func·010
    /root/go/src/github.com/influxdb/influxdb/tsdb/shard.go:149 +0x611

goroutine 35 [select]:
github.com/influxdb/influxdb/tsdb.(*Shard).autoflusher(0xc208032e70, 0xc2081d6d80)
    /root/go/src/github.com/influxdb/influxdb/tsdb/shard.go:468 +0x39d
created by github.com/influxdb/influxdb/tsdb.func·010
    /root/go/src/github.com/influxdb/influxdb/tsdb/shard.go:149 +0x611

goroutine 36 [select]:
github.com/influxdb/influxdb/tsdb.(*Shard).autoflusher(0xc208032f20, 0xc2081d6f60)
    /root/go/src/github.com/influxdb/influxdb/tsdb/shard.go:468 +0x39d
created by github.com/influxdb/influxdb/tsdb.func·010
    /root/go/src/github.com/influxdb/influxdb/tsdb/shard.go:149 +0x611

goroutine 37 [select]:
github.com/influxdb/influxdb/tsdb.(*Shard).autoflusher(0xc208033080, 0xc2081d71a0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/shard.go:468 +0x39d
created by github.com/influxdb/influxdb/tsdb.func·010
    /root/go/src/github.com/influxdb/influxdb/tsdb/shard.go:149 +0x611

goroutine 38 [select]:
github.com/influxdb/influxdb/services/hh.(*Service).retryWrites(0xc20804d360)
    /root/go/src/github.com/influxdb/influxdb/services/hh/service.go:98 +0x2e7
created by github.com/influxdb/influxdb/services/hh.(*Service).Open
    /root/go/src/github.com/influxdb/influxdb/services/hh/service.go:62 +0xef

goroutine 39 [select]:
github.com/influxdb/influxdb/services/hh.(*Service).expireWrites(0xc20804d360)
    /root/go/src/github.com/influxdb/influxdb/services/hh/service.go:116 +0x2a4
created by github.com/influxdb/influxdb/services/hh.(*Service).Open
    /root/go/src/github.com/influxdb/influxdb/services/hh/service.go:63 +0x109

goroutine 40 [chan receive]:
github.com/influxdb/influxdb/tcp.(*listener).Accept(0xc20803adf8, 0x0, 0x0, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tcp/mux.go:117 +0x80
github.com/influxdb/influxdb/cluster.(*Service).serve(0xc208010310)
    /root/go/src/github.com/influxdb/influxdb/cluster/service.go:79 +0x102
created by github.com/influxdb/influxdb/cluster.(*Service).Open
    /root/go/src/github.com/influxdb/influxdb/cluster/service.go:56 +0x67

goroutine 41 [select]:
github.com/influxdb/influxdb/services/precreator.(*Service).runPrecreation(0xc20800ae60)
    /root/go/src/github.com/influxdb/influxdb/services/precreator/service.go:72 +0x407
created by github.com/influxdb/influxdb/services/precreator.(*Service).Open
    /root/go/src/github.com/influxdb/influxdb/services/precreator/service.go:50 +0xcb

goroutine 42 [chan receive]:
github.com/influxdb/influxdb/tcp.(*listener).Accept(0xc20803ae00, 0x0, 0x0, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tcp/mux.go:117 +0x80
github.com/influxdb/influxdb/services/snapshotter.(*Service).serve(0xc20800af50)
    /root/go/src/github.com/influxdb/influxdb/services/snapshotter/service.go:68 +0xad
created by github.com/influxdb/influxdb/services/snapshotter.(*Service).Open
    /root/go/src/github.com/influxdb/influxdb/services/snapshotter/service.go:46 +0x62

goroutine 43 [IO wait]:
net.(*pollDesc).Wait(0xc2081aa450, 0x72, 0x0, 0x0)
    /root/.gvm/gos/go1.4.2/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc2081aa450, 0x0, 0x0)
    /root/.gvm/gos/go1.4.2/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).accept(0xc2081aa3f0, 0x0, 0x7ff7026d6cc8, 0xc20818c5c8)
    /root/.gvm/gos/go1.4.2/src/net/fd_unix.go:419 +0x40b
net.(*TCPListener).AcceptTCP(0xc208126178, 0xc20813c380, 0x0, 0x0)
    /root/.gvm/gos/go1.4.2/src/net/tcpsock_posix.go:234 +0x4e
net.(*TCPListener).Accept(0xc208126178, 0x0, 0x0, 0x0, 0x0)
    /root/.gvm/gos/go1.4.2/src/net/tcpsock_posix.go:244 +0x4c
net/http.(*Server).Serve(0xc2081b8720, 0x7ff7026dd308, 0xc208126178, 0x0, 0x0)
    /root/.gvm/gos/go1.4.2/src/net/http/server.go:1728 +0x92
net/http.Serve(0x7ff7026dd308, 0xc208126178, 0x7ff7026fe6b8, 0xc20818c6a0, 0x0, 0x0)
    /root/.gvm/gos/go1.4.2/src/net/http/server.go:1606 +0xa1
github.com/influxdb/influxdb/services/admin.(*Service).serve(0xc20800aff0)
    /root/go/src/github.com/influxdb/influxdb/services/admin/service.go:105 +0x147
created by github.com/influxdb/influxdb/services/admin.(*Service).Open
    /root/go/src/github.com/influxdb/influxdb/services/admin/service.go:68 +0x431

goroutine 44 [select]:
github.com/influxdb/influxdb/services/continuous_querier.(*Service).backgroundLoop(0xc208010380)
    /root/go/src/github.com/influxdb/influxdb/services/continuous_querier/service.go:151 +0x301
created by github.com/influxdb/influxdb/services/continuous_querier.(*Service).Open
    /root/go/src/github.com/influxdb/influxdb/services/continuous_querier/service.go:87 +0x1ed

goroutine 45 [IO wait]:
net.(*pollDesc).Wait(0xc2081aa4c0, 0x72, 0x0, 0x0)
    /root/.gvm/gos/go1.4.2/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc2081aa4c0, 0x0, 0x0)
    /root/.gvm/gos/go1.4.2/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).accept(0xc2081aa460, 0x0, 0x7ff7026d6cc8, 0xc208030070)
    /root/.gvm/gos/go1.4.2/src/net/fd_unix.go:419 +0x40b
net.(*TCPListener).AcceptTCP(0xc208126188, 0xc20805edc8, 0x0, 0x0)
    /root/.gvm/gos/go1.4.2/src/net/tcpsock_posix.go:234 +0x4e
net.(*TCPListener).Accept(0xc208126188, 0x0, 0x0, 0x0, 0x0)
    /root/.gvm/gos/go1.4.2/src/net/tcpsock_posix.go:244 +0x4c
net/http.(*Server).Serve(0xc2081233e0, 0x7ff7026dd308, 0xc208126188, 0x0, 0x0)
    /root/.gvm/gos/go1.4.2/src/net/http/server.go:1728 +0x92
net/http.Serve(0x7ff7026dd308, 0xc208126188, 0x7ff7026fe618, 0xc2080acc00, 0x0, 0x0)
    /root/.gvm/gos/go1.4.2/src/net/http/server.go:1606 +0xa1
github.com/influxdb/influxdb/services/httpd.(*Service).serve(0xc20800b130)
    /root/go/src/github.com/influxdb/influxdb/services/httpd/service.go:107 +0x79
created by github.com/influxdb/influxdb/services/httpd.(*Service).Open
    /root/go/src/github.com/influxdb/influxdb/services/httpd/service.go:75 +0x590

goroutine 46 [select]:
github.com/influxdb/influxdb/services/retention.(*Service).deleteShardGroups(0xc2080469c0)
    /root/go/src/github.com/influxdb/influxdb/services/retention/service.go:67 +0x433
created by github.com/influxdb/influxdb/services/retention.(*Service).Open
    /root/go/src/github.com/influxdb/influxdb/services/retention/service.go:44 +0x67

goroutine 47 [select]:
github.com/influxdb/influxdb/services/retention.(*Service).deleteShards(0xc2080469c0)
    /root/go/src/github.com/influxdb/influxdb/services/retention/service.go:101 +0x821
created by github.com/influxdb/influxdb/services/retention.(*Service).Open
    /root/go/src/github.com/influxdb/influxdb/services/retention/service.go:45 +0x81

goroutine 48 [chan receive]:
github.com/influxdb/influxdb/cmd/influxd/run.(*Server).startServerReporting(0xc2080aa0e0)
    /root/go/src/github.com/influxdb/influxdb/cmd/influxd/run/server.go:372 +0x18e
created by github.com/influxdb/influxdb/cmd/influxd/run.func·003
    /root/go/src/github.com/influxdb/influxdb/cmd/influxd/run/server.go:327 +0xdbe

goroutine 49 [select]:
github.com/influxdb/influxdb/cmd/influxd/run.(*Command).monitorServerErrors(0xc208011b90)
    /root/go/src/github.com/influxdb/influxdb/cmd/influxd/run/command.go:125 +0x2da
created by github.com/influxdb/influxdb/cmd/influxd/run.(*Command).Run
    /root/go/src/github.com/influxdb/influxdb/cmd/influxd/run/command.go:107 +0xbb4

goroutine 52 [runnable]:
github.com/influxdb/influxdb/tsdb.(*FieldCodec).EncodeFields(0xc2083d9f20, 0xc2082a2ba0, 0x0, 0x0, 0x0, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/shard.go:841 +0xe3
github.com/influxdb/influxdb/tsdb.(*Shard).WritePoints(0xc208198420, 0xc208312000, 0x12c, 0x200, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/shard.go:254 +0x459
github.com/influxdb/influxdb/tsdb.(*Store).WriteToShard(0xc208046660, 0x20, 0xc208312000, 0x12c, 0x200, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/store.go:281 +0xf7
github.com/influxdb/influxdb/cluster.func·002(0x20, 0x1, 0xc208312000, 0x12c, 0x200)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:241 +0xfc
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:271 +0x30b

goroutine 67 [select]:
github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard(0xc2080102a0, 0xc2080e3200, 0xc20811f60f, 0xb, 0xc208030140, 0x7, 0x1, 0xc2082fe000, 0x12c, 0x200, ...)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:278 +0x913
github.com/influxdb/influxdb/cluster.func·001(0xc2080e3200, 0xc20811f60f, 0xb, 0xc208030140, 0x7, 0xc2082fe000, 0x12c, 0x200)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:205 +0xad
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).WritePoints
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:206 +0x447

goroutine 16 [IO wait]:
net.(*pollDesc).Wait(0xc208010290, 0x72, 0x0, 0x0)
    /root/.gvm/gos/go1.4.2/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc208010290, 0x0, 0x0)
    /root/.gvm/gos/go1.4.2/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).Read(0xc208010230, 0xc2082f0000, 0x1000, 0x1000, 0x0, 0x7ff7026d6cc8, 0xc20818c6f8)
    /root/.gvm/gos/go1.4.2/src/net/fd_unix.go:242 +0x40f
net.(*conn).Read(0xc20803a108, 0xc2082f0000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /root/.gvm/gos/go1.4.2/src/net/net.go:121 +0xdc
net/http.noteEOFReader.Read(0x7ff7026fe798, 0xc20803a108, 0xc208136738, 0xc2082f0000, 0x1000, 0x1000, 0xc208251540, 0x0, 0x0)
    /root/.gvm/gos/go1.4.2/src/net/http/transport.go:1270 +0x6e
net/http.(*noteEOFReader).Read(0xc208152f40, 0xc2082f0000, 0x1000, 0x1000, 0xc208014400, 0x0, 0x0)
    <autogenerated>:125 +0xd4
bufio.(*Reader).fill(0xc208047ec0)
    /root/.gvm/gos/go1.4.2/src/bufio/bufio.go:97 +0x1ce
bufio.(*Reader).Peek(0xc208047ec0, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
    /root/.gvm/gos/go1.4.2/src/bufio/bufio.go:132 +0xf0
net/http.(*persistConn).readLoop(0xc2081366e0)
    /root/.gvm/gos/go1.4.2/src/net/http/transport.go:842 +0xa4
created by net/http.(*Transport).dialConn
    /root/.gvm/gos/go1.4.2/src/net/http/transport.go:660 +0xc9f

goroutine 66 [select]:
net/http.(*persistConn).writeLoop(0xc2081366e0)
    /root/.gvm/gos/go1.4.2/src/net/http/transport.go:945 +0x41d
created by net/http.(*Transport).dialConn
    /root/.gvm/gos/go1.4.2/src/net/http/transport.go:661 +0xcbc

goroutine 19 [select]:
github.com/influxdb/influxdb/cluster.(*PointsWriter).WritePoints(0xc2080102a0, 0xc20817a100, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:210 +0x634
github.com/influxdb/influxdb/services/httpd.(*Handler).serveWriteLine(0xc2080acc00, 0x7ff7026fec20, 0xc208153060, 0xc20814cea0, 0xc208252000, 0x1ff74, 0x3fe00, 0x0)
    /root/go/src/github.com/influxdb/influxdb/services/httpd/handler.go:568 +0xdb9
github.com/influxdb/influxdb/services/httpd.(*Handler).serveWrite(0xc2080acc00, 0x7ff7026fec20, 0xc208153060, 0xc20814cea0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/services/httpd/handler.go:418 +0x69d
github.com/influxdb/influxdb/services/httpd.*Handler.(github.com/influxdb/influxdb/services/httpd.serveWrite)·fm(0x7ff7026fec20, 0xc208153060, 0xc20814cea0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/services/httpd/handler.go:107 +0x4f
github.com/influxdb/influxdb/services/httpd.func·001(0x7ff7026fec20, 0xc208153060, 0xc20814cea0)
    /root/go/src/github.com/influxdb/influxdb/services/httpd/handler.go:696 +0x78
net/http.HandlerFunc.ServeHTTP(0xc2080ee3a0, 0x7ff7026fec20, 0xc208153060, 0xc20814cea0)
    /root/.gvm/gos/go1.4.2/src/net/http/server.go:1265 +0x41
github.com/influxdb/influxdb/services/httpd.func·002(0x7ff7026feb48, 0xc208153000, 0xc20814cea0)
    /root/go/src/github.com/influxdb/influxdb/services/httpd/handler.go:754 +0x288
net/http.HandlerFunc.ServeHTTP(0xc2080e8b10, 0x7ff7026feb48, 0xc208153000, 0xc20814cea0)
    /root/.gvm/gos/go1.4.2/src/net/http/server.go:1265 +0x41
github.com/influxdb/influxdb/services/httpd.func·003(0x7ff7026feb48, 0xc208153000, 0xc20814cea0)
    /root/go/src/github.com/influxdb/influxdb/services/httpd/handler.go:763 +0xba
net/http.HandlerFunc.ServeHTTP(0xc2080ee3c0, 0x7ff7026feb48, 0xc208153000, 0xc20814cea0)
    /root/.gvm/gos/go1.4.2/src/net/http/server.go:1265 +0x41
github.com/influxdb/influxdb/services/httpd.func·004(0x7ff7026feb48, 0xc208153000, 0xc20814cea0)
    /root/go/src/github.com/influxdb/influxdb/services/httpd/handler.go:796 +0x31a
net/http.HandlerFunc.ServeHTTP(0xc2080e8b40, 0x7ff7026feb48, 0xc208153000, 0xc20814cea0)
    /root/.gvm/gos/go1.4.2/src/net/http/server.go:1265 +0x41
github.com/influxdb/influxdb/services/httpd.func·005(0x7ff7026feb48, 0xc208153000, 0xc20814cea0)
    /root/go/src/github.com/influxdb/influxdb/services/httpd/handler.go:806 +0x179
net/http.HandlerFunc.ServeHTTP(0xc2080e8b60, 0x7ff7026feb48, 0xc208153000, 0xc20814cea0)
    /root/.gvm/gos/go1.4.2/src/net/http/server.go:1265 +0x41
github.com/influxdb/influxdb/services/httpd.func·006(0x7ff7026feb48, 0xc208152fe0, 0xc20814cea0)
    /root/go/src/github.com/influxdb/influxdb/services/httpd/handler.go:814 +0x123
net/http.HandlerFunc.ServeHTTP(0xc2080ee3e0, 0x7ff7026feb48, 0xc208152fe0, 0xc20814cea0)
    /root/.gvm/gos/go1.4.2/src/net/http/server.go:1265 +0x41
github.com/influxdb/influxdb/services/httpd.func·007(0x7ff7026feb10, 0xc208316000, 0xc20814cea0)
    /root/go/src/github.com/influxdb/influxdb/services/httpd/handler.go:824 +0x12c
net/http.HandlerFunc.ServeHTTP(0xc2080ee400, 0x7ff7026feb10, 0xc208316000, 0xc20814cea0)
    /root/.gvm/gos/go1.4.2/src/net/http/server.go:1265 +0x41
github.com/bmizerany/pat.(*PatternServeMux).ServeHTTP(0xc20803ace8, 0x7ff7026feb10, 0xc208316000, 0xc20814cea0)
    /root/go/src/github.com/bmizerany/pat/mux.go:109 +0x21c
github.com/influxdb/influxdb/services/httpd.(*Handler).ServeHTTP(0xc2080acc00, 0x7ff7026feb10, 0xc208316000, 0xc20814cea0)
    /root/go/src/github.com/influxdb/influxdb/services/httpd/handler.go:175 +0x256
net/http.serverHandler.ServeHTTP(0xc2081233e0, 0x7ff7026feb10, 0xc208316000, 0xc20814cea0)
    /root/.gvm/gos/go1.4.2/src/net/http/server.go:1703 +0x19a
net/http.(*conn).serve(0xc208306000)
    /root/.gvm/gos/go1.4.2/src/net/http/server.go:1204 +0xb57
created by net/http.(*Server).Serve
    /root/.gvm/gos/go1.4.2/src/net/http/server.go:1751 +0x35e

goroutine 82 [select]:
github.com/influxdb/influxdb/cluster.(*PointsWriter).WritePoints(0xc2080102a0, 0xc208120000, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:210 +0x634
github.com/influxdb/influxdb/services/httpd.(*Handler).serveWriteLine(0xc2080acc00, 0x7ff7026fec20, 0xc2082b6160, 0xc208034750, 0xc20837a000, 0x2046a, 0x3fe00, 0x0)
    /root/go/src/github.com/influxdb/influxdb/services/httpd/handler.go:568 +0xdb9
github.com/influxdb/influxdb/services/httpd.(*Handler).serveWrite(0xc2080acc00, 0x7ff7026fec20, 0xc2082b6160, 0xc208034750, 0x0)
    /root/go/src/github.com/influxdb/influxdb/services/httpd/handler.go:418 +0x69d
github.com/influxdb/influxdb/services/httpd.*Handler.(github.com/influxdb/influxdb/services/httpd.serveWrite)·fm(0x7ff7026fec20, 0xc2082b6160, 0xc208034750, 0x0)
    /root/go/src/github.com/influxdb/influxdb/services/httpd/handler.go:107 +0x4f
github.com/influxdb/influxdb/services/httpd.func·001(0x7ff7026fec20, 0xc2082b6160, 0xc208034750)
    /root/go/src/github.com/influxdb/influxdb/services/httpd/handler.go:696 +0x78
net/http.HandlerFunc.ServeHTTP(0xc2080ee3a0, 0x7ff7026fec20, 0xc2082b6160, 0xc208034750)
    /root/.gvm/gos/go1.4.2/src/net/http/server.go:1265 +0x41
github.com/influxdb/influxdb/services/httpd.func·002(0x7ff7026feb48, 0xc2082b6100, 0xc208034750)
    /root/go/src/github.com/influxdb/influxdb/services/httpd/handler.go:754 +0x288
net/http.HandlerFunc.ServeHTTP(0xc2080e8b10, 0x7ff7026feb48, 0xc2082b6100, 0xc208034750)
    /root/.gvm/gos/go1.4.2/src/net/http/server.go:1265 +0x41
github.com/influxdb/influxdb/services/httpd.func·003(0x7ff7026feb48, 0xc2082b6100, 0xc208034750)
    /root/go/src/github.com/influxdb/influxdb/services/httpd/handler.go:763 +0xba
net/http.HandlerFunc.ServeHTTP(0xc2080ee3c0, 0x7ff7026feb48, 0xc2082b6100, 0xc208034750)
    /root/.gvm/gos/go1.4.2/src/net/http/server.go:1265 +0x41
github.com/influxdb/influxdb/services/httpd.func·004(0x7ff7026feb48, 0xc2082b6100, 0xc208034750)
    /root/go/src/github.com/influxdb/influxdb/services/httpd/handler.go:796 +0x31a
net/http.HandlerFunc.ServeHTTP(0xc2080e8b40, 0x7ff7026feb48, 0xc2082b6100, 0xc208034750)
    /root/.gvm/gos/go1.4.2/src/net/http/server.go:1265 +0x41
github.com/influxdb/influxdb/services/httpd.func·005(0x7ff7026feb48, 0xc2082b6100, 0xc208034750)
    /root/go/src/github.com/influxdb/influxdb/services/httpd/handler.go:806 +0x179
net/http.HandlerFunc.ServeHTTP(0xc2080e8b60, 0x7ff7026feb48, 0xc2082b6100, 0xc208034750)
    /root/.gvm/gos/go1.4.2/src/net/http/server.go:1265 +0x41
github.com/influxdb/influxdb/services/httpd.func·006(0x7ff7026feb48, 0xc2082b60e0, 0xc208034750)
    /root/go/src/github.com/influxdb/influxdb/services/httpd/handler.go:814 +0x123
net/http.HandlerFunc.ServeHTTP(0xc2080ee3e0, 0x7ff7026feb48, 0xc2082b60e0, 0xc208034750)
    /root/.gvm/gos/go1.4.2/src/net/http/server.go:1265 +0x41
github.com/influxdb/influxdb/services/httpd.func·007(0x7ff7026feb10, 0xc20804c0a0, 0xc208034750)
    /root/go/src/github.com/influxdb/influxdb/services/httpd/handler.go:824 +0x12c
net/http.HandlerFunc.ServeHTTP(0xc2080ee400, 0x7ff7026feb10, 0xc20804c0a0, 0xc208034750)
    /root/.gvm/gos/go1.4.2/src/net/http/server.go:1265 +0x41
github.com/bmizerany/pat.(*PatternServeMux).ServeHTTP(0xc20803ace8, 0x7ff7026feb10, 0xc20804c0a0, 0xc208034750)
    /root/go/src/github.com/bmizerany/pat/mux.go:109 +0x21c
github.com/influxdb/influxdb/services/httpd.(*Handler).ServeHTTP(0xc2080acc00, 0x7ff7026feb10, 0xc20804c0a0, 0xc208034750)
    /root/go/src/github.com/influxdb/influxdb/services/httpd/handler.go:175 +0x256
net/http.serverHandler.ServeHTTP(0xc2081233e0, 0x7ff7026feb10, 0xc20804c0a0, 0xc208034750)
    /root/.gvm/gos/go1.4.2/src/net/http/server.go:1703 +0x19a
net/http.(*conn).serve(0xc20804c000)
    /root/.gvm/gos/go1.4.2/src/net/http/server.go:1204 +0xb57
created by net/http.(*Server).Serve
    /root/.gvm/gos/go1.4.2/src/net/http/server.go:1751 +0x35```
@toddboom
Copy link
Contributor

@jhorwit2 what did your write payloads look like? how many points per batch, cardinality, etc? how long were you writing data before you saw the timeouts?

@jhorwit2
Copy link
Contributor Author

Payloads very, but I have been sending the data for over a week now on 0.9.1 without issues. I upgraded this morning around 9am and response times peaked around 1:30.

I have 4 servers sending anywhere between 200-500 points in each write and they each average about 1 write per second, so 800-2k points per second getting sent to influx. Tags are sorted. Points are sorted in asc by timestamp.

The fields have a lot of cardinality, but the tags have very low cardinality (under 100).

Example payload of one endpoint

300lo,browser=Chrome,edge=false,releaseVersion=v2.2.1-wp,valid=true PARAMS.pub="stuff",browserFamily="IE",PARAMS.dp="stuff",PARAMS.rev="v2.2.1-wp",operatingSystem="Windows",PARAMS.fp="stuff",dp_valid="true",operatingSystemVersion="7",browserVersion="11",uid_valid="true",uvc_valid="true",device="device",PARAMS.gen="100",cookie="stuff" 1437485321799

As far as fields go, every request for each measurement has the same fields with some value (whether blank or not)

@jhorwit2
Copy link
Contributor Author

I keep getting this and it appears to be tied to a CQ query timeout, which causes cascading timeouts across the application (#3469). I have noticed since the upgrade to 0.9.2, the restart hasn't been as graceful.

I always get this no matter if I stop/start or restart with or without errors.

sudo /etc/init.d/influxdb restart
influxd process was stopped [ OK ]
Starting the process influxd [ OK ]
influxd process failed to start [ FAILED ]

The server will start fine though.

@jhorwit2 jhorwit2 mentioned this issue Jul 30, 2015
Closed
@beckettsean
Copy link
Contributor

Should be solved with 0.9.3. Please reopen if that isn't the case.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/continuous queries area/writes
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@toddboom @jhorwit2 @beckettsean