Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

@uppy/companion: increase max limits for remote file list operations #4417

Merged
merged 4 commits into from
Apr 18, 2023
Merged

@uppy/companion: increase max limits for remote file list operations #4417

merged 4 commits into from
Apr 18, 2023

Conversation

mifi
Copy link
Contributor

@mifi mifi commented Apr 15, 2023 •
edited
Loading

...because adding 10k files still takes about 5 minutes (to paginate all files), see discussion here: #4389

advantages:

  • adding huge folder as much faster
  • listing files is faster when scrolling (when a lot of files), because less /list API calls

disadvantages:

  • it takes longer to show the user the first list (when a lot of files) and more data will be used (e.g. mobile)

TODO

  • All other providers?

@Murderlon Murderlon changed the title increase max limits for remote file list operations @uppy/companion: increase max limits for remote file list operations Apr 17, 2023
@arturi
Copy link
Contributor

arturi commented Apr 18, 2023

API limits to pageSize: 100, because we have too many fields: https://stackoverflow.com/questions/42592125/list-request-page-size-being-ignored

If we can reduce the number of fields, maybe we can get more:

const DRIVE_FILE_FIELDS = 'kind,id,imageMediaMetadata,name,mimeType,ownedByMe,permissions(role,emailAddress),size,modifiedTime,iconLink,thumbnailLink,teamDriveId,videoMediaMetadata,shortcutDetails(targetId,targetMimeType),parents'

@arturi arturi merged commit 80a8275 into main Apr 18, 2023
@arturi arturi deleted the companion-list-max branch April 18, 2023 17:04
@github-actions github-actions bot mentioned this pull request Apr 18, 2023
Merged
github-actions bot added a commit that referenced this pull request Apr 18, 2023
@github-actions
Release: uppy@3.8.0 (#4418)
57a71e1 
| Package                   | Version | Package                   | Version |
| ------------------------- | ------- | ------------------------- | ------- |
| @uppy/angular             |   0.5.2 | @uppy/progress-bar        |   3.0.2 |
| @uppy/audio               |   1.1.1 | @uppy/provider-views      |   3.3.0 |
| @uppy/aws-s3              |   3.1.0 | @uppy/react               |   3.1.2 |
| @uppy/aws-s3-multipart    |   3.2.0 | @uppy/react-native        |   0.5.1 |
| @uppy/box                 |   2.1.1 | @uppy/redux-dev-tools     |   3.0.2 |
| @uppy/companion           |   4.5.0 | @uppy/remote-sources      |   1.0.3 |
| @uppy/companion-client    |   3.1.3 | @uppy/screen-capture      |   3.1.1 |
| @uppy/compressor          |   1.0.2 | @uppy/status-bar          |   3.1.1 |
| @uppy/core                |   3.2.0 | @uppy/store-default       |   3.0.3 |
| @uppy/dashboard           |   3.4.0 | @uppy/store-redux         |   3.0.3 |
| @uppy/drag-drop           |   3.0.2 | @uppy/svelte              |   3.0.2 |
| @uppy/dropbox             |   3.1.1 | @uppy/thumbnail-generator |   3.0.3 |
| @uppy/facebook            |   3.1.1 | @uppy/transloadit         |   3.1.3 |
| @uppy/file-input          |   3.0.2 | @uppy/tus                 |   3.1.0 |
| @uppy/form                |   3.0.2 | @uppy/unsplash            |   3.2.1 |
| @uppy/golden-retriever    |   3.0.3 | @uppy/url                 |   3.3.1 |
| @uppy/google-drive        |   3.1.1 | @uppy/utils               |   5.3.0 |
| @uppy/image-editor        |   2.1.2 | @uppy/vue                 |   1.0.2 |
| @uppy/informer            |   3.0.2 | @uppy/webcam              |   3.3.1 |
| @uppy/instagram           |   3.1.1 | @uppy/xhr-upload          |   3.2.0 |
| @uppy/locales             |   3.2.0 | @uppy/zoom                |   2.1.1 |
| @uppy/onedrive            |   3.1.1 | uppy                      |   3.8.0 |

- @uppy/companion: increase max limits for remote file list operations (Mikael Finstad / #4417)
- @uppy/xhr-upload: fix type in README.md (Top Master / #4416)
- @uppy/core: improve performance of validating & uploading files (Mikael Finstad / #4402)
- @uppy/provider-views: Concurrent file listing (Mikael Finstad / #4401)
- @uppy/core,@uppy/locales,@uppy/provider-views: User feedback adding recursive folders take 2 (Mikael Finstad / #4399)
- @uppy/dashboard: Single File Mode: fix layout and make optional (Artur Paikin / #4374)
- @uppy/informer: add a check in `TransitionGroup` when component is null (Juan Belej / #4410)
- meta: Fix logos in all the readmes (Artur Paikin / #4407)
- meta: fix logo in readme (Kid / #4403)
@H4NG-MAN H4NG-MAN mentioned this pull request May 12, 2023
Open
HeavenFox pushed a commit to docsend/uppy that referenced this pull request Jun 27, 2023
@mifi @arturi
@uppy/companion: increase max limits for remote file list operations (t…
86b6a06 
…ransloadit#4417)

* increase max limits for list operations

* fix broken test

* pageSize won't work until we get rid of permissions field

* Revert "fix broken test"

This reverts commit 70daa70.

---------

Co-authored-by: Artur Paikin <artur@arturpaikin.com>
HeavenFox pushed a commit to docsend/uppy that referenced this pull request Jun 27, 2023
@github-actions
Release: uppy@3.8.0 (transloadit#4418)
00f6d47 
| Package                   | Version | Package                   | Version |
| ------------------------- | ------- | ------------------------- | ------- |
| @uppy/angular             |   0.5.2 | @uppy/progress-bar        |   3.0.2 |
| @uppy/audio               |   1.1.1 | @uppy/provider-views      |   3.3.0 |
| @uppy/aws-s3              |   3.1.0 | @uppy/react               |   3.1.2 |
| @uppy/aws-s3-multipart    |   3.2.0 | @uppy/react-native        |   0.5.1 |
| @uppy/box                 |   2.1.1 | @uppy/redux-dev-tools     |   3.0.2 |
| @uppy/companion           |   4.5.0 | @uppy/remote-sources      |   1.0.3 |
| @uppy/companion-client    |   3.1.3 | @uppy/screen-capture      |   3.1.1 |
| @uppy/compressor          |   1.0.2 | @uppy/status-bar          |   3.1.1 |
| @uppy/core                |   3.2.0 | @uppy/store-default       |   3.0.3 |
| @uppy/dashboard           |   3.4.0 | @uppy/store-redux         |   3.0.3 |
| @uppy/drag-drop           |   3.0.2 | @uppy/svelte              |   3.0.2 |
| @uppy/dropbox             |   3.1.1 | @uppy/thumbnail-generator |   3.0.3 |
| @uppy/facebook            |   3.1.1 | @uppy/transloadit         |   3.1.3 |
| @uppy/file-input          |   3.0.2 | @uppy/tus                 |   3.1.0 |
| @uppy/form                |   3.0.2 | @uppy/unsplash            |   3.2.1 |
| @uppy/golden-retriever    |   3.0.3 | @uppy/url                 |   3.3.1 |
| @uppy/google-drive        |   3.1.1 | @uppy/utils               |   5.3.0 |
| @uppy/image-editor        |   2.1.2 | @uppy/vue                 |   1.0.2 |
| @uppy/informer            |   3.0.2 | @uppy/webcam              |   3.3.1 |
| @uppy/instagram           |   3.1.1 | @uppy/xhr-upload          |   3.2.0 |
| @uppy/locales             |   3.2.0 | @uppy/zoom                |   2.1.1 |
| @uppy/onedrive            |   3.1.1 | uppy                      |   3.8.0 |

- @uppy/companion: increase max limits for remote file list operations (Mikael Finstad / transloadit#4417)
- @uppy/xhr-upload: fix type in README.md (Top Master / transloadit#4416)
- @uppy/core: improve performance of validating & uploading files (Mikael Finstad / transloadit#4402)
- @uppy/provider-views: Concurrent file listing (Mikael Finstad / transloadit#4401)
- @uppy/core,@uppy/locales,@uppy/provider-views: User feedback adding recursive folders take 2 (Mikael Finstad / transloadit#4399)
- @uppy/dashboard: Single File Mode: fix layout and make optional (Artur Paikin / transloadit#4374)
- @uppy/informer: add a check in `TransitionGroup` when component is null (Juan Belej / transloadit#4410)
- meta: Fix logos in all the readmes (Artur Paikin / transloadit#4407)
- meta: fix logo in readme (Kid / transloadit#4403)
This was referenced Nov 27, 2023
Open
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Murderlon Murderlon Murderlon approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@mifi @arturi @Murderlon