Add a DynamicBuffer for faster VFS implementations #258
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Currently, the InMemoryFileSystem uses a Uint8List buffer per file. When resizing the buffer, it creates a new one and copies the content over.
While copying the bytes is fairly fast for small buffers, doing that for every written page for a 10MB+ file size adds significant overhead. Basically, creating a database with size n takes
O(n^2)
time, which becomes very noticeable with large databases. This affects both IndexedDbFileSystem and InMemoryFileSystem, since IndexedDbFileSystem uses InMemoryFileSystem internally. In some of my tests, creating a 30MB database takes around 120s with either VFS.This introduces a DynamicBuffer class that handles resizing the buffer. Instead of resizing on every write, it doubles the size every time new capacity is needed. This changes it to
O(n)
to create a database of size n. In my tests, the same database takes 5s to create with InMemoryFileSystem, and 10s with IndexedDbFileSystem.Another approach would be to use the JS ArrayBuffer.resize together with
maxByteLength
, since ArrayBuffers are used internally for ByteBuffer with dart2js anyway. However, it may not be supported by all browsers, and would need a different approach for dart2wasm.In my testing this appears to work well with both IndexedDbFileSystem and InMemoryFileSystem. It would still be good to review the IndexedDbFileSystem implementation specifically - I'm not sure whether the change could introduce some edge case, especially around
previousContent
.