Optimize WAL redo speed

[knizhnik]

No description provided.

[hlinnaka]

This has been sitting for a while, let's try to move this forward:

• Does the pollfds change really make a difference? In a quick test on my laptop now, I don't see any measurable difference, but there's a lot of noise...
• Can you rebase and open a separate PR for the postgres changes, please? They're hard to comment on here and they're independent from the pollfds change so they can be considered separately.

[antons-antons]

somewhat offtopic, but what's the best way to have a zenith repo PR that has Postgres changes as well?

also, I think instead of teaching buffer cache of consumer differences, we should introduce an explicit "local buffers only" mode to avoid special case of am_wal_redo_postgres

[knizhnik]

This has been sitting for a while, let's try to move this forward:

* Does the pollfds change really make a difference? In a quick test on my laptop now, I don't see any measurable difference, but there's a lot of noise...

The same for me: I didn't measured any noticeable difference. I just leave this change mostly for because of "common sense": if it is possible to do something once, why repeat it multiple times? Especially if it doesn't somehow complicate the code...

* Can you rebase and open a separate PR for the postgres changes, please? They're hard to comment on here and they're independent from the pollfds change so they can be considered separately.

Will do.

[knizhnik]

also, I think instead of teaching buffer cache of consumer differences, we should introduce an explicit "local buffers only" mode to avoid special case of am_wal_redo_postgres

I prefer to minimize number of changes we have made in core.
Solution with local buffer requires changes just in few places, while introducing yet another buffer cache mode will require much more changes.

[knizhnik]

Can you rebase and open a separate PR for the postgres changes, please? They're hard to comment on here and they're independent from the pollfds change so they can be considered separately.

There was already PR for postgres-only changes:

neondatabase/postgres#144

It doesn't require rebase.

[LizardWizzard]

As discussed moving to draft.