Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RepoSack::update_and_load_repos: First load repositories from cache #245

Conversation

jrohel
Copy link
Contributor

@jrohel jrohel commented Jan 23, 2023

Optimizing resource usage in RepoSack::update_and_load_repo.

The order of uploading repositories to the sack has been changed.

  • The system repository is still loaded first.
  • Then repositories with valid data in the cache (the cache contains data and it is unexpired or the repository uses ONLY_CACHE or LAZY SynStrategy) are sent to thread_sack_loader.
  • The following are repositories with expired data that have been found to match the source.
  • Finally, while thread_sack_loader is uploading prepared repositories to the sack, the remaining repositories (repositories with invalid or empty cache) are downloaded.

Downloading and importing repository keys has been changed:

  • First, only repositories that do not have a key verification problem are prepared for uploading to the bag.
  • Then keys are downloaded and imported for repositories with bad keys.
  • These repositories are then verified again and processed further if successful.

@m-blaha m-blaha self-assigned this Jan 24, 2023
@jrohel jrohel force-pushed the feature/improve_update_and_load_repos branch from 9f7b7d6 to 8aa6def Compare February 13, 2023 08:30
@jrohel jrohel force-pushed the feature/improve_update_and_load_repos branch 3 times, most recently from 499871a to dbb6480 Compare March 14, 2023 20:54
jrohel added 7 commits March 27, 2023 10:38
Work optimization. The order in which the repositories are loaded into
the sack is changed. Repositories with a valid cache are sent to
the `thread_sack_loader` before downloading other repositories.
Removes the automatic download and import of keys from
the repo::RepoDownloader::perform method.
It is now automatically done in the repo::RepoSack::update_and_load_repos
method.

The advantage is optimization. First, all repositories that do not have
a problem with key verification are prepared for uploading to the sack.
Then keys are downloaded/imported for repositories with bad keys. These
repositories are then verified again and, if successful, further processed.
It is now possible to call `repo::RepoSack::update_and_load_repos` with
download and import key support disabled.
… keys

Uses `FileDownloader` class to download repository keys instead
of `download_url` method from `RepoDownloader`.

A repository independent librepo handler is now used and multiple key
files can be downloaded in parallel.
Removes downloading/copying of local files to temporary files.
Repositories keys are imported from local files directly from their
original location. Then remote keys files are downloaded and imported.
@jrohel jrohel force-pushed the feature/improve_update_and_load_repos branch from dbb6480 to 3c63ce2 Compare March 27, 2023 08:55
Copy link
Member

@m-blaha m-blaha left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you, LGTM.

@m-blaha m-blaha merged commit ea187f9 into rpm-software-management:main Mar 29, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants