Write Buffer

Required reading: Data Consistency

The write buffer is a logical extension of the WAL file. Since StormDB requires its data on file to align perfectly, we flush this write buffer only when it's full or a timeout occurs.

In the event of a timeout, if the number of records pending a write are not a multiple of 128, then the last record is simply repeated until this requirement is satisfied. This repetition is to ensure that the offset calculation is predictable based on the data consistency strategy.

The write buffer is required to optimise write throughput to the underlying filesystem.

Reading from the write buffer

Any absolute address which overflows from the WAL file on disk, is assumed to be contained within the write buffer. As a result, reads for records which haven't been flushed to disk will be read from the in-memory write buffer.