ledger-tool: stream output of accounts subcommand

[t-nelson]

Problem

ledger-tool accounts buffers all accounts before printing them to console. with the number of accounts on some of the public cluster, this consumes an unreasonable amount of memory, requiring absurd hardware to do what should be a fairly trivial task

Summary of Changes

stream the output while we scan instead of buffering

[t-nelson]

Note to self: if backporting, need #28501, #28502 and #28533

[t-nelson]

seems to work. former is master, latter this PR

test:

./cargo run --release --bin solana-ledger-tool -- --ledger ${LEDGER_DIR} accounts --halt-at-slot ${SNAPSHOT_SLOT} --encoding base64 --output json > /dev/null

where SNAPSHOT_SLOT is the slot height at which the most recent snapshot in LEDGER_DIR was taken. snapshot was from mb, slot 155512607.

[t-nelson]

@jeffwashington would you expect such memory growth (~20GB, 16-36GB) during a full accounts scan under normal operation? i reran the above tests with an empty scan_func and observed the same growth as the second run (with this PR)

gonna try again with two scan passes and see how that behaves

[t-nelson]

gonna try again with two scan passes and see how that behaves

flat on the second run... must be some cache priming 🤷‍♂️

[jeffwashington]

try this metric:
SELECT mean("count_in_mem") AS "mean_count_in_mem" FROM ":db:"."autogen"."accounts_index"
it is possible/likely that the entire accounts index is being loaded into memory during this scan.

[CriesofCarrots]

Haven't take it for a spin yet, but lgtm!
Not that this wold ever come up, but I assume that json encoding with zero loadable accounts would print some version of []?

[t-nelson]

Not that this wold ever come up, but I assume that json encoding with zero loadable accounts would print some version of []?

yep! creating the serializer_seq then .end()ing it does the wrapping

[steviez]

LGTM