SNMP TRAP causes panic: runtime error: index out of range

[SanderWegter]

Relevant telegraf.conf:

[[inputs.snmp_trap]]

System info:

Ubuntu 18.04.3 LTS
Telegraf version 1.14.0 - Docker
Docker version 19.03.5, build 633a0ea838
docker-compose version 1.23.2, build 1110ad01

Docker

telegraf:
    image: telegraf
    restart: always
    volumes:
      - /opt/data/monitoring/telegraf/config:/etc/telegraf
      - /var/run/docker.sock:/var/run/docker.sock
      - /usr/share/snmp/mibs:/usr/share/snmp/mibs
      - /opt/data/monitoring/telegraf/scripts:/scripts
    ports:
      - 161:161
    networks:
      influx:
      mgmt-vlan:
        ipv4_address: "X.X.X.X"
    depends_on:
      - influxdb
    command: -config /etc/telegraf/telegraf.conf -config-directory /etc/telegraf/telegraf.d

Steps to reproduce:

Sending an SNMP trap from a Huawei (S5720) switch causes telegraf to restart

Trapbuffer:

#Apr 15 2020 13:00:53+01:00 HOSTNAME L2IFPPI/4/MACHASHCONFLICTALARM:OID 1.3.6.1.4.1.2011.5.25.315.3.6 A hash conflict occurs in MAC addresses.(IfIndex=30, MacAddr=[aa.bb.cc.dd.ee.ff (hex)], VLAN=81, VsiName=, InterfaceName=XGE0/0/1).

Telegraf log

2020-04-15T12:00:53.970798104Z panic: runtime error: index out of range [6] with length 6
2020-04-15T12:00:53.970835615Z 
2020-04-15T12:00:53.971069258Z goroutine 32 [running]:
2020-04-15T12:00:53.971749109Z github.com/soniah/gosnmp.parseLength(0xc001a7d086, 0x6, 0xf7a, 0x10000000040e9a6, 0x0)
2020-04-15T12:00:53.972186461Z 	/go/pkg/mod/github.com/soniah/gosnmp@v1.22.0/helper.go:501 +0x8e
2020-04-15T12:00:53.973367938Z github.com/soniah/gosnmp.(*GoSNMP).decodeValue(0x44bebe0, 0xc001a7d086, 0x6, 0xf7a, 0x297d7f1, 0x5, 0x1, 0x0, 0xc0009b3ba8)
2020-04-15T12:00:53.973794938Z 	/go/pkg/mod/github.com/soniah/gosnmp@v1.22.0/helper.go:172 +0x1648
2020-04-15T12:00:53.974653098Z github.com/soniah/gosnmp.(*GoSNMP).decodeValue(0x44bebe0, 0xc001a7d084, 0x47, 0xf7c, 0x297d7f1, 0x5, 0xc0015332a0, 0x1b, 0x0)
2020-04-15T12:00:53.974666183Z 	/go/pkg/mod/github.com/soniah/gosnmp@v1.22.0/helper.go:175 +0x16c4
2020-04-15T12:00:53.974673731Z github.com/soniah/gosnmp.(*GoSNMP).unmarshalVBL(0x44bebe0, 0xc001a7d01c, 0xaf, 0xfe4, 0xc000c8ab60, 0x22e2860, 0x463a180)
2020-04-15T12:00:53.974681325Z 	/go/pkg/mod/github.com/soniah/gosnmp@v1.22.0/marshal.go:986 +0x377
2020-04-15T12:00:53.974688316Z github.com/soniah/gosnmp.(*GoSNMP).unmarshalResponse(0x44bebe0, 0xc001a7d00f, 0xbc, 0xff1, 0xc000c8ab60, 0xe0, 0x2904cc0)
2020-04-15T12:00:53.974695523Z 	/go/pkg/mod/github.com/soniah/gosnmp@v1.22.0/marshal.go:871 +0x34e
2020-04-15T12:00:53.974702365Z github.com/soniah/gosnmp.(*GoSNMP).unmarshalPayload(0x44bebe0, 0xc001a7d000, 0xcb, 0x1000, 0xf, 0xc000c8ab60, 0x0, 0x0)
2020-04-15T12:00:53.974709607Z 	/go/pkg/mod/github.com/soniah/gosnmp@v1.22.0/marshal.go:791 +0x9b
2020-04-15T12:00:53.974716595Z github.com/soniah/gosnmp.(*GoSNMP).UnmarshalTrap(0x44bebe0, 0xc001a7d000, 0xcb, 0x1000, 0xcb)
2020-04-15T12:00:53.974738567Z 	/go/pkg/mod/github.com/soniah/gosnmp@v1.22.0/trap.go:228 +0x31c
2020-04-15T12:00:53.974745798Z github.com/soniah/gosnmp.(*TrapListener).Listen(0xc000314080, 0x2988e5d, 0x4, 0x0, 0x0)
2020-04-15T12:00:53.974752475Z 	/go/pkg/mod/github.com/soniah/gosnmp@v1.22.0/trap.go:187 +0x2f9
2020-04-15T12:00:53.974758786Z github.com/influxdata/telegraf/plugins/inputs/snmp_trap.(*SnmpTrap).Start.func1(0xc00057c000, 0x2988e5d, 0x4)
2020-04-15T12:00:53.974765217Z 	/go/src/github.com/influxdata/telegraf/plugins/inputs/snmp_trap/snmp_trap.go:133 +0x43
2020-04-15T12:00:53.974771468Z created by github.com/influxdata/telegraf/plugins/inputs/snmp_trap.(*SnmpTrap).Start
2020-04-15T12:00:53.974777654Z 	/go/src/github.com/influxdata/telegraf/plugins/inputs/snmp_trap/snmp_trap.go:132 +0x317
2020-04-15T12:00:57.017425019Z 2020-04-15T12:00:57Z I! Starting Telegraf 1.14.0

Expected behavior:

No crash

Actual behavior:

Crashes and restarts the telegraf process

Additional info:

Telegraf is running in a Docker container configured with ipvlan

Edit:

It seems to crash whenever there is a MACHASHCONFLICT with one of the mac addresses starting with 44.d4

Crashed at:

2020-04-15T12:00:53.970798104Z 
2020-04-15T11:57:50.905715908Z
2020-04-15T11:38:31.627745654Z 
2020-04-15T11:35:28.610391850Z
2020-04-15T11:19:12.613270979Z

Trapbuffer: (Shortened the message to show just the mac addresses, these are all unique)

#Apr 15 2020 13:00:53+01:00 MacAddr=[44.d4.37.aa

#Apr 15 2020 12:57:50+01:00 MacAddr=[44.d4.37.bb

#Apr 15 2020 12:38:31+01:00 MacAddr=[44.d4.37.cc
#Apr 15 2020 12:38:31+01:00 MacAddr=[44.d4.37.dd
#Apr 15 2020 12:38:31+01:00 MacAddr=[44.d4.37.ee
#Apr 15 2020 12:38:31+01:00 MacAddr=[44.d4.37.ff
#Apr 15 2020 12:38:31+01:00 MacAddr=[44.d4.37.00
#Apr 15 2020 12:38:31+01:00 MacAddr=[44.d4.37.11
#Apr 15 2020 12:38:31+01:00 MacAddr=[44.d4.37.22
#Apr 15 2020 12:38:31+01:00 MacAddr=[44.d4.37.33
#Apr 15 2020 12:38:31+01:00 MacAddr=[44.d4.37.44
#Apr 15 2020 12:38:31+01:00 MacAddr=[44.d4.37.55

#Apr 15 2020 12:35:28+01:00 MacAddr=[44.d4.37.66
#Apr 15 2020 12:35:28+01:00 MacAddr=[44.d4.37.77
#Apr 15 2020 12:35:28+01:00 MacAddr=[44.d4.37.88
#Apr 15 2020 12:35:28+01:00 MacAddr=[44.d4.37.ba
#Apr 15 2020 12:35:28+01:00 MacAddr=[44.d4.37.ab
#Apr 15 2020 12:35:28+01:00 MacAddr=[44.d4.37.cd
#Apr 15 2020 12:35:28+01:00 MacAddr=[44.d4.37.ef

#Apr 15 2020 12:19:12+01:00 MacAddr=[44.d4.37.12
#Apr 15 2020 12:19:12+01:00 MacAddr=[44.d4.37.34
#Apr 15 2020 12:19:12+01:00 MacAddr=[44.d4.37.56

#Apr 15 2020 12:11:04+01:00 MacAddr=[44.d4.37.7a

[SanderWegter]

Updated Telegraf to 1.14.5, still the same issue.

[reimda]

This may be fixed now that #8588 is merged. Could you retest?

[DustyArmstrong]

I have the same issue as here, however my issue appears to be a segmentation violation.

panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1e5249d]

goroutine 9 [running]: github.com/soniah/gosnmp.(*TrapListener).Close(0xc000225d60)
/go/pkg/mod/github.com/soniah/gosnmp@v1.25.0/trap.go:131 +0x3d github.com/influxdata/telegraf/plugins/inputs/snmp_trap.(*SnmpTrap).Stop(0xc000328380) /go/src/github.com/influxdata/telegraf/plugins/inputs/snmp_trap/snmp_trap.go:251 +0x36 github.com/influxdata/telegraf/agent.stopServiceInputs(0xc00004e680, 0xa, 0x10) /go/src/github.com/influxdata/telegraf/agent/agent.go:445 +0x82
github.com/influxdata/telegraf/agent.(*Agent).testRunInputs(0xc0001366d0, 0x3490f80, 0xc000220080, 0x0, 0xc00033ba60, 0x0, 0xc0002157b8) /go/src/github.com/influxdata/telegraf/agent/agent.go:434 +0x1b7 github.com/influxdata/telegraf/agent.(*Agent).test.func4(0xc00036e8f4, 0xc0001366d0, 0x3490f80, 0xc000220080, 0x0, 0xc00033ba60)
/go/src/github.com/influxdata/telegraf/agent/agent.go:977 +0x8b created by github.com/influxdata/telegraf/agent.(*Agent).test /go/src/github.com/influxdata/telegraf/agent/agent.go:975 +0x352

I can create a separate issue for this if preferred. Running version 1.17.2 (Docker).

[SanderWegter]

I haven't encountered this in a while. Currently running Telegraf 1.17.3. We'll keep an eye on it but for now this issue can be closed.