write page headers on the first xlog segment

[antons-antons]

This is not meant for production (yet?)

with this change pg_waldump will be able to dump log records starting with the first record written by compute.

gotchas

1. start LSN has to be passed by the user (there's no automagic skipping to the first log record)
2. unmodified pg_waldump needs .partial to be removed from the segment name
3. The code is not meant to be universal, this is a hack. If we were to implement the solution properly we need a little better input validation and flexibility on which parameters we record (e.g. tli is fixed, a bunch of numbers are hardcoded)

[hlinnaka]

I'm a little nervous of making a WAL segment that looks valid, but isn't. I'm afraid that it will accidentally get streamed over to safekeepers or elsewhere, and overwrite or take the place of actual valid WAL.

Maybe I'm just paranoid. I agree it would be very nice if pg_waldump worked on our WAL files. But it's something to consider. How valid does the canned block look like? Would xlogreader.c accept it? Would our safekeeper code accept it?

[antons-antons]

I'm a little nervous of making a WAL segment that looks valid, but isn't. I'm afraid that it will accidentally get streamed over to safekeepers or elsewhere, and overwrite or take the place of actual valid WAL.

The original intent is to enable this mode optionally (e.g. for desktop debugging); this is happening on the Safekeepers themselves

Maybe I'm just paranoid. I agree it would be very nice if pg_waldump worked on our WAL files. But it's something to consider. How valid does the canned block look like? Would xlogreader.c accept it? Would our safekeeper code accept it?

The page header is in XLog[Long]PageHeaderData format, so reading the canned page works just fine (pg_waldump runs the validation that Postgres was reader code does)
given this is happening on the Safekeepers on offsets below first write LSN the concern is invocation of this function on the segments other than the first segment (this is actually tricky as current init-db gives us LSN (0/163cff8) that falls into segment 000000010000000000000001 but this doesn't mean it will always do this; currently it's hardcoded to be only on segment 1)
Additionally there're no page headers between first segment page and first page with data (which means pg_waldump has to get start LSN)

this code change is mostly a conversation starter, I'm happy with morphing it into a working solution. The motivation is to aid debugging as it could be painful without ability to read transaction log

[arssher]

You shouldn't do that at truncate_wal, which is called whenever uncommitted tail of log is truncated due to changed term -- it'd make that fake record direct to the truncation point instead of the first valid record. write_canned_page must be done by the very first write to the first segment.

Also not sure why you call write_canned_page twice, once should be enough. Regarding the whole safety of the approach -- it should be ok, as currently no one ever reads that (zeros before first record in the first segment) part of WAL. At least safekeeper remembers LSN of the first record and starts there. But it would be nice to add some description how it works and safety checks that first valid LSN is after that crafted record and within the first segment.

[antons-antons]

write_canned_page must be done by the very first write to the first segment.

I couldn't find a correct place to do that without introducing a state variable, I may need some guidance here

Also not sure why you call write_canned_page twice, once should be enough.

This is due to the way XLog is read and validated, first page in the segment should always be present regardless of the reading position (header contains block size and segment size "needed" for reading the rest of the data), then the page we going to start dumping log records from needs to have a header as well

[arssher]

I couldn't find a correct place to do that without introducing a state variable, I may need some guidance here

I think you do need a state variable =(

[antons-antons]

retired in favor of a change to pg_waldump neondatabase/postgres#145