Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Resumable uploads #169

Open
hannahhoward opened this issue Feb 20, 2025 · 0 comments
Open

Resumable uploads #169

hannahhoward opened this issue Feb 20, 2025 · 0 comments
Assignees

Comments

@hannahhoward
Copy link
Member

hannahhoward commented Feb 20, 2025

What

Currently, we store no information about the state of an upload on the client. That means on resumption, the only information about the state of the upload is what's on Storacha, which doesn't even know this data is part of a single upload until the very end.

That means we must:

  1. Scan through files to pass to uploadDirectory
  2. Rerun all IPLD-ificaiton of flat files from scratch
  3. Recalculate all hashes including pieceCIDs even for parts we've already uploaded (can't determine if it's already uploaded or put on filecoin without the hash)
  4. Almost surely we do a few things twice accidentally in the upload

While we can vaguely resume small uploads this way, this gets untenable to for large uploads, especially terabyte size were a simple network outage causing an interruption is a non-trivial possibility.

It seems like we need a domain object to describe an in progress upload, that tracks state of an upload, either as a local object or as something we store and modify on Storacha. Generally, I would keep this as a local object for now.

In terms of functionality, once we have an ID and lookup for in progress upload state, we should print it on the CLI as a reference to use to resume an upload.

@hannahhoward hannahhoward self-assigned this Feb 20, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant