Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP: Compress extensions for S3 #4753

Closed
wants to merge 2 commits into from
Closed

Conversation

bayandin
Copy link
Member

@bayandin bayandin commented Jul 18, 2023

Problem

We want to upload compressed extensions + have an index file for them.

Summary of changes

Checklist before requesting a review

  • I have performed a self-review of my code.
  • If it is a core feature, I have added thorough tests.
  • Do we need to implement analytics? if so did you add the relevant metrics to the dashboard?
  • If this PR requires public announcement, mark it with /release-notes label and add several sentences in this section.

Checklist before merging

  • Do not forget to reformat commit message to not include the above checklist

@github-actions
Copy link

github-actions bot commented Jul 18, 2023

1040 tests run: 988 passed, 0 failed, 52 skipped (full report)


Flaky tests (2)

Postgres 15

  • test_remote_storage_upload_queue_retries[local_fs]: debug
  • test_remote_timeline_client_calls_started_metric[local_fs]: release

@bayandin bayandin force-pushed the bayandin/new-extension-format branch 3 times, most recently from 40f9b5a to f846758 Compare July 19, 2023 11:07
@bayandin bayandin changed the title WIP: Tar-gzip extension on S3 WIP: Compress extensions for S3 Jul 19, 2023
@awestover awestover mentioned this pull request Jul 20, 2023
@bayandin bayandin force-pushed the bayandin/new-extension-format branch 2 times, most recently from fb79a98 to f846758 Compare July 20, 2023 19:55
@awestover awestover force-pushed the bayandin/new-extension-format branch from f846758 to 1b6e13a Compare July 21, 2023 15:08
@awestover
Copy link
Contributor

awestover commented Jul 21, 2023

Note: there is a slight discrepancy between the layout of files created here and the assumed layout in rust code.

Rust assumes a layout like this:

5615610098 // this is an extension build number
├── v14
│   ├── extensions
│   │   ├── anon.tar.zst
│   │   └── embedding.tar.zst
│   └── ext_index.json
└── v15
    ├── extensions
    │   ├── anon.tar.zst
    │   └── embedding.tar.zst
    └── ext_index.json
5615261079
├── v14
│   ├── extensions
│   │   └── anon.tar.zst
│   └── ext_index.json
└── v15
    ├── extensions
    │   └── anon.tar.zst
    └── ext_index.json

currently, this PR creates files like this:

5615610098 // this is an extension build number
├── v14
│   ├── extensions
│   │   ├── anon/anon.tar.zst
│   │   └── embedding/embedding.tar.zst
│   └── ext_index.json
└── v15
    ├── extensions
    │   ├── anon/anon.tar.zst
    │   └── embedding/embedding.tar.zst
    └── ext_index.json
5615261079
├── v14
│   ├── extensions
│   │   └── anon/anon.tar.zst
│   └── ext_index.json
└── v15
    ├── extensions
    │   └── anon/anon.tar.zst
    └── ext_index.json

I think the layout assumed in rust is nicer, but not sure if you were planning on having multiple archive files in each of these subdirectories?
It's an easy python or rust fix too if you'd rather keep this layout

@bayandin
Copy link
Member Author

Merged in #4764

@bayandin bayandin closed this Jul 27, 2023
@bayandin bayandin deleted the bayandin/new-extension-format branch July 27, 2023 10:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants