Node instability since 0.5.6

[mishmosh]

Compiling reports of node instability here.

[mishmosh]

0.5.6 has proven to be unstable for both storage and retrieval bots -- after the crashes last night and corrupted db, we only have 1 retrieval bot online and 0 storage bots. Corrupted dbs are resyncing and it will take a long time, and will probably crash and be corrupted again.

0.5.1 had no issues for storage bots, but memory leak issues for retrieval. Though it never had any panic crashed like 0.5.6, so in general it's more stable.

ERROR: initializing node: starting node: could not build arguments for function "github.com/filecoin-project/lotus/node/modules/lp2p".StartListening.func1 (/go/lotus/node/modules/lp2p/addrs.go:98): failed to build host.Host: could not build arguments for function "reflect".makeFuncStub (/usr/local/go/src/reflect/asm_amd64.s:12): failed to build lp2p.BaseIpfsRouting: could not build arguments for function "reflect".makeFuncStub (/usr/local/go/src/reflect/asm_amd64.s:12): failed to build dtypes.MetadataDS: received non-nil error from function "reflect".makeFuncStub (/usr/local/go/src/reflect/asm_amd64.s:12): opening datastore /metadata: leveldb/journal: block/chunk corrupted: missing chunk part (0 bytes) [file=340058.log]

Also seeing a lot of memory errors:

2020-09-02T23:42:03.817Z	ERROR	evtsm	go-statemachine@v0.0.0-20200813232949-df9b130df370/machine.go:83	Executing event planner failed: write /root/.lotus/datastore/metadata/340058.log: cannot allocate memory

[mishmosh]

Seems to be resolved by 0.6.1.