Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Drop lockedCandidatesForBatch and mark chunks as 'busy' on preparing as a fix for #3341 #3342

Merged
merged 3 commits into from
Dec 7, 2021
Merged

Drop lockedCandidatesForBatch and mark chunks as 'busy' on preparing as a fix for #3341 #3342

merged 3 commits into from
Dec 7, 2021

Conversation

yaegor
Copy link
Contributor

@yaegor yaegor commented Dec 2, 2021

this.lockedCandidatesForBatch code in #uploadParts definitely is the cause of the bug with unresumable uploads after pause (#3341).
Not sure if this.lockedCandidatesForBatch is at all necessary, so proposing to drop it.

@aduh95
Copy link
Contributor

aduh95 commented Dec 3, 2021

For reference, this was added in #3056 (see #3056 (comment)).

Removing a public property is technically a breaking change (this should have been a private property really), maybe we should add an option to ignore it so it's not breaking anymore?

The fact that this doesn't break any test is promising though.

@yaegor yaegor mentioned this pull request Dec 3, 2021
Closed
@Murderlon
Copy link
Member

Thanks for investigating this @yaegor.

@martin-brennan as you worked on this, do you know if we can safely remove this?

@yaegor
Mark upload chunk as busy just before preparing upload part to ensure…
cdc0c5a 
… proper selection of the next chunks to upload

This ensures that the chunk will not be selected as next upload candidate until it is aborted.
@yaegor yaegor changed the title Drop lockedCandidatesForBatch as a fix for #3341 Drop lockedCandidatesForBatch and mark chunks as 'bus' on preparing as a fix for #3341 Dec 6, 2021
@yaegor yaegor changed the title Drop lockedCandidatesForBatch and mark chunks as 'bus' on preparing as a fix for #3341 Drop lockedCandidatesForBatch and mark chunks as 'busy' on preparing as a fix for #3341 Dec 6, 2021
@martin-brennan
Copy link
Contributor

@Murderlon I will take a look today and refresh my memory, though as aduh95 said if it's not breaking any tests it's probably safe.

@martin-brennan
Copy link
Contributor

Oh just realised @yaegor made a change to fix this by setting the candidate chunks to busy. I think the proposed fix here should work fine and we can drop this.lockedCandidatesForBatch 👍

Just cross-referencing the OP's comment from #3341 too:

I've looked into the code more and lockedCandidatesForBatch cannot be just dropped as in #3342 as that can result in duplicate parts uploads. Seems like chunk's busy state can be set to true before preparing chunk for upload and since busy state is properly reset on upload errors, that should work.

@Murderlon
Copy link
Member

Thanks @yaegor and @martin-brennan

@Murderlon Murderlon linked an issue Dec 7, 2021 that may be closed by this pull request
S3 Multipart: Paused uploads hang indefinitely on resume #3341
Closed
@Murderlon Murderlon merged commit a6270bf into transloadit:main Dec 7, 2021
@github-actions github-actions bot mentioned this pull request Dec 7, 2021
Merged
github-actions bot added a commit that referenced this pull request Dec 7, 2021
@github-actions
Release: uppy@2.3.0 (#3348)
2af711c 
| Package                   | Version | Package                   | Version |
| ------------------------- | ------- | ------------------------- | ------- |
| @uppy/angular             |   0.2.6 | @uppy/locales             |   2.0.4 |
| @uppy/audio               |   0.2.0 | @uppy/onedrive            |   2.0.5 |
| @uppy/aws-s3              |   2.0.6 | @uppy/provider-views      |   2.0.6 |
| @uppy/aws-s3-multipart    |   2.2.0 | @uppy/react               |   2.1.2 |
| @uppy/box                 |   1.0.5 | @uppy/screen-capture      |   2.0.5 |
| @uppy/companion           |   3.1.2 | @uppy/status-bar          |   2.1.2 |
| @uppy/companion-client    |   2.0.4 | @uppy/store-default       |   2.0.3 |
| @uppy/core                |   2.1.3 | @uppy/thumbnail-generator |   2.0.6 |
| @uppy/dashboard           |   2.1.2 | @uppy/transloadit         |   2.0.5 |
| @uppy/drag-drop           |   2.0.5 | @uppy/tus                 |   2.1.2 |
| @uppy/dropbox             |   2.0.5 | @uppy/url                 |   2.0.5 |
| @uppy/facebook            |   2.0.5 | @uppy/utils               |   4.0.4 |
| @uppy/file-input          |   2.0.5 | @uppy/webcam              |   2.0.5 |
| @uppy/golden-retriever    |   2.0.6 | @uppy/xhr-upload          |   2.0.6 |
| @uppy/google-drive        |   2.0.5 | @uppy/zoom                |   1.0.5 |
| @uppy/image-editor        |   1.1.0 | @uppy/robodog             |   2.1.4 |
| @uppy/informer            |   2.0.5 | uppy                      |   2.3.0 |
| @uppy/instagram           |   2.0.5 |                           |         |

- meta: add release automations (Antoine du Hamel / #3304)
- @uppy/dashboard: Save meta fields when opening the image editor (Merlijn Vos / #3339)
- @uppy/aws-s3-multipart: Drop `lockedCandidatesForBatch` and mark chunks as busy when preparing (Yegor Yarko / #3342)
- @uppy/webcam: fix broken links in `webcam.md` (Antoine du Hamel / #3346)
- @uppy/audio: new @uppy/audio plugin for recording with microphone (Artur Paikin / #2976)
- build: force use of `@babel/plugin-proposal-optional-chaining` (Antoine du Hamel / #3335)
- @uppy/companion: fix deploy Yarn version (Antoine du Hamel / #3327)
- @uppy/companion: upgrade aws-sdk (Mikael Finstad / #3334)
- @uppy/core: disable loose transpilation for legacy bundle (Antoine du Hamel / #3329)
- @uppy/angular: examples: update `angular-example` to Angular v13 (Antoine du Hamel / #3325)
- meta: Update BACKLOG.md (Artur Paikin, Merlijn Vos)
- meta: Add disableLocalFiles to options summary (Steve Barker / #3323)
- meta: Create SECURITY.md (Ziding Zhang / #3052)
- @uppy/image-editor: Pass croppedCanvasOptions to getCroppedCanvas (Mohamed Boudra / #3320)
- meta: finish `master`->`main` job (Mikael Finstad / #3315)
- website: update documents that were out of date (Antoine du Hamel / #3317)
- @uppy/status-bar: Status bar error state improvements (Merlijn Vos / #3299)
- doc: Fix typo in `docs/drag-drop.md` (Ash Allen / #3319)
- website: Update /support and docs about Transloadit-hosted Companion (Artur Paikin / #3243)
- @uppy/aws-s3,@uppy/box,@uppy/core,@uppy/dashboard,@uppy/drag-drop,@uppy/dropbox,@uppy/facebook,@uppy/file-input,@uppy/google-drive,@uppy/image-editor,@uppy/instagram,@uppy/locales,@uppy/onedrive,@uppy/screen-capture,@uppy/status-bar,@uppy/thumbnail-generator,@uppy/transloadit,@uppy/url,@uppy/webcam,@uppy/xhr-upload,@uppy/zoom: Refactor locale scripts & generate types and docs (Merlijn Vos / #3276)
- @uppy/companion: Remove references of incorrect `options` argument for `companion.socket` (Mikael Finstad / #3307)
- @uppy/companion: Upgrade linting to 2.0.0-0 (Kevin van Zonneveld / #3280)
@snyk-bot snyk-bot mentioned this pull request Dec 8, 2021
Open
HeavenFox pushed a commit to docsend/uppy that referenced this pull request Jun 27, 2023
@github-actions
Release: uppy@2.3.0 (transloadit#3348)
b1ce382 
| Package                   | Version | Package                   | Version |
| ------------------------- | ------- | ------------------------- | ------- |
| @uppy/angular             |   0.2.6 | @uppy/locales             |   2.0.4 |
| @uppy/audio               |   0.2.0 | @uppy/onedrive            |   2.0.5 |
| @uppy/aws-s3              |   2.0.6 | @uppy/provider-views      |   2.0.6 |
| @uppy/aws-s3-multipart    |   2.2.0 | @uppy/react               |   2.1.2 |
| @uppy/box                 |   1.0.5 | @uppy/screen-capture      |   2.0.5 |
| @uppy/companion           |   3.1.2 | @uppy/status-bar          |   2.1.2 |
| @uppy/companion-client    |   2.0.4 | @uppy/store-default       |   2.0.3 |
| @uppy/core                |   2.1.3 | @uppy/thumbnail-generator |   2.0.6 |
| @uppy/dashboard           |   2.1.2 | @uppy/transloadit         |   2.0.5 |
| @uppy/drag-drop           |   2.0.5 | @uppy/tus                 |   2.1.2 |
| @uppy/dropbox             |   2.0.5 | @uppy/url                 |   2.0.5 |
| @uppy/facebook            |   2.0.5 | @uppy/utils               |   4.0.4 |
| @uppy/file-input          |   2.0.5 | @uppy/webcam              |   2.0.5 |
| @uppy/golden-retriever    |   2.0.6 | @uppy/xhr-upload          |   2.0.6 |
| @uppy/google-drive        |   2.0.5 | @uppy/zoom                |   1.0.5 |
| @uppy/image-editor        |   1.1.0 | @uppy/robodog             |   2.1.4 |
| @uppy/informer            |   2.0.5 | uppy                      |   2.3.0 |
| @uppy/instagram           |   2.0.5 |                           |         |

- meta: add release automations (Antoine du Hamel / transloadit#3304)
- @uppy/dashboard: Save meta fields when opening the image editor (Merlijn Vos / transloadit#3339)
- @uppy/aws-s3-multipart: Drop `lockedCandidatesForBatch` and mark chunks as busy when preparing (Yegor Yarko / transloadit#3342)
- @uppy/webcam: fix broken links in `webcam.md` (Antoine du Hamel / transloadit#3346)
- @uppy/audio: new @uppy/audio plugin for recording with microphone (Artur Paikin / transloadit#2976)
- build: force use of `@babel/plugin-proposal-optional-chaining` (Antoine du Hamel / transloadit#3335)
- @uppy/companion: fix deploy Yarn version (Antoine du Hamel / transloadit#3327)
- @uppy/companion: upgrade aws-sdk (Mikael Finstad / transloadit#3334)
- @uppy/core: disable loose transpilation for legacy bundle (Antoine du Hamel / transloadit#3329)
- @uppy/angular: examples: update `angular-example` to Angular v13 (Antoine du Hamel / transloadit#3325)
- meta: Update BACKLOG.md (Artur Paikin, Merlijn Vos)
- meta: Add disableLocalFiles to options summary (Steve Barker / transloadit#3323)
- meta: Create SECURITY.md (Ziding Zhang / transloadit#3052)
- @uppy/image-editor: Pass croppedCanvasOptions to getCroppedCanvas (Mohamed Boudra / transloadit#3320)
- meta: finish `master`->`main` job (Mikael Finstad / transloadit#3315)
- website: update documents that were out of date (Antoine du Hamel / transloadit#3317)
- @uppy/status-bar: Status bar error state improvements (Merlijn Vos / transloadit#3299)
- doc: Fix typo in `docs/drag-drop.md` (Ash Allen / transloadit#3319)
- website: Update /support and docs about Transloadit-hosted Companion (Artur Paikin / transloadit#3243)
- @uppy/aws-s3,@uppy/box,@uppy/core,@uppy/dashboard,@uppy/drag-drop,@uppy/dropbox,@uppy/facebook,@uppy/file-input,@uppy/google-drive,@uppy/image-editor,@uppy/instagram,@uppy/locales,@uppy/onedrive,@uppy/screen-capture,@uppy/status-bar,@uppy/thumbnail-generator,@uppy/transloadit,@uppy/url,@uppy/webcam,@uppy/xhr-upload,@uppy/zoom: Refactor locale scripts & generate types and docs (Merlijn Vos / transloadit#3276)
- @uppy/companion: Remove references of incorrect `options` argument for `companion.socket` (Mikael Finstad / transloadit#3307)
- @uppy/companion: Upgrade linting to 2.0.0-0 (Kevin van Zonneveld / transloadit#3280)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Murderlon Murderlon Murderlon approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

S3 Multipart: Paused uploads hang indefinitely on resume
4 participants
@yaegor @aduh95 @Murderlon @martin-brennan