You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently, we store no information about the state of an upload on the client. That means on resumption, the only information about the state of the upload is what's on Storacha, which doesn't even know this data is part of a single upload until the very end.
That means we must:
Scan through files to pass to uploadDirectory
Rerun all IPLD-ificaiton of flat files from scratch
Recalculate all hashes including pieceCIDs even for parts we've already uploaded (can't determine if it's already uploaded or put on filecoin without the hash)
Almost surely we do a few things twice accidentally in the upload
While we can vaguely resume small uploads this way, this gets untenable to for large uploads, especially terabyte size were a simple network outage causing an interruption is a non-trivial possibility.
It seems like we need a domain object to describe an in progress upload, that tracks state of an upload, either as a local object or as something we store and modify on Storacha. Generally, I would keep this as a local object for now.
In terms of functionality, once we have an ID and lookup for in progress upload state, we should print it on the CLI as a reference to use to resume an upload.
The text was updated successfully, but these errors were encountered:
What
Currently, we store no information about the state of an upload on the client. That means on resumption, the only information about the state of the upload is what's on Storacha, which doesn't even know this data is part of a single upload until the very end.
That means we must:
While we can vaguely resume small uploads this way, this gets untenable to for large uploads, especially terabyte size were a simple network outage causing an interruption is a non-trivial possibility.
It seems like we need a domain object to describe an in progress upload, that tracks state of an upload, either as a local object or as something we store and modify on Storacha. Generally, I would keep this as a local object for now.
In terms of functionality, once we have an ID and lookup for in progress upload state, we should print it on the CLI as a reference to use to resume an upload.
The text was updated successfully, but these errors were encountered: