Tune PebbleDB Options

[alexanderbez]

We currently use default opts for PebbleDB as a SS backend in store v2.

Benchmarks show that reads are very fast but writes are pretty awful in terms of performance.

Even though SS should be read optimized, writes should not be a huge bottleneck. There are a handful of levers we should explore tuning, e.g. MemTableSize, to ensure writes are as efficient as possible.

For reference, these are the PebbleDB levers CockroachDB tunes: https://github.com/cockroachdb/cockroach/blob/master/pkg/storage/pebble.go#L602

[alexanderbez]

cc @Kbhat1

[tac0turtle]

Are you able to post the benchmarks here?

[alexanderbez]

#17504 :)

[tac0turtle]

Are the results posted anywhere?

[alexanderbez]

Are the results posted anywhere?

They're in the PR @tac0turtle: https://pastebin.com/raw/q5LhTcRy

[tac0turtle]

my bad was viewing on mobile

[alexanderbez]

@Kbhat1 from Sei is helping us benchmark further and find a possible set of ideal opts.