Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Option "disable_multithreading", repo::RepoSack ctors and repo::Repo::fetch_metadata method private, fix/adjust unit tests #273

Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
RepoSack::update_and_load_repos: Added single-threaded mode
Single-threaded mode is used when multi-threaded mode is disabled
(configuration option "disable_multithreading").
  • Loading branch information
jrohel committed Mar 7, 2023
commit af7f480dfedde3771d5a070877b51459a7041ef0
16 changes: 14 additions & 2 deletions include/libdnf/repo/repo_sack.hpp
Original file line number Diff line number Diff line change
Expand Up @@ -112,17 +112,26 @@ class RepoSack : public sack::Sack<Repo> {
/// @param load_system Whether to load the system repository
void update_and_load_enabled_repos(bool load_system);

/// Downloads (if necessary) repository metadata and loads them in parallel.
/// Downloads (if necessary) repository metadata and loads them.
///
/// It works in parallel unless multithreading is disabled with
/// the "disable_multithreading" configuration option.
///
/// Multi-threaded version:
/// Launches a thread that picks repos from a queue and loads them into
/// memory (calling their `load()` method). Then iterates over `repos`,
/// potentially downloads fresh metadata (by calling the
/// `download_metadata()` method) and then queues them for loading. This
/// speeds up the process by loading repos into memory while others are being
/// downloaded.
///
/// Single-threaded version:
/// Iterates over `repos`. Potentially downloads fresh metadata
/// (by calling the `download_metadata()` method) and then loads them into memory
/// (calling their `load()` method).
///
/// @param repos The repositories to update and load
void update_and_load_repos(libdnf::repo::RepoQuery & repos);
void update_and_load_repos(const libdnf::repo::RepoQuery & repos);

RepoSackWeakPtr get_weak_ptr() { return RepoSackWeakPtr(this, &sack_guard); }

Expand All @@ -146,6 +155,9 @@ class RepoSack : public sack::Sack<Repo> {

void internalize_repos();

void update_and_load_repos_singlethreaded(const libdnf::repo::RepoQuery & repos);
void update_and_load_repos_multithreaded(const libdnf::repo::RepoQuery & repos);

BaseWeakPtr base;

repo::Repo * system_repo{nullptr};
Expand Down
43 changes: 42 additions & 1 deletion libdnf/repo/repo_sack.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -190,7 +190,39 @@ RepoWeakPtr RepoSack::get_system_repo() {
}


void RepoSack::update_and_load_repos(libdnf::repo::RepoQuery & repos) {
void RepoSack::update_and_load_repos_singlethreaded(const libdnf::repo::RepoQuery & repos) {
// If the input RepoQuery contains a system repo, we load the system repo first.
for (auto & repo : repos) {
if (repo->get_type() == libdnf::repo::Repo::Type::SYSTEM) {
repo->load();
break;
}
}

// Fetch and load available repositories.
for (auto & repo : repos) {
if (repo->get_type() != libdnf::repo::Repo::Type::AVAILABLE) {
continue;
}
try {
repo->fetch_metadata();
repo->load();
} catch (const RepoDownloadError & e) {
if (!repo->get_config().skip_if_unavailable().get_value()) {
throw;
}
base->get_logger()->warning(
"Error loading repo \"{}\" (skipping due to \"skip_if_unavailable=true\"): {}",
repo->get_id(),
e.what()); // TODO(lukash) we should print nested exceptions
}
}

base->get_rpm_package_sack()->load_config_excludes_includes();
}


void RepoSack::update_and_load_repos_multithreaded(const libdnf::repo::RepoQuery & repos) {
std::atomic<bool> except_in_main_thread{false}; // set to true if an exception occurred in the main thread
std::exception_ptr except_ptr; // for pass exception from thread_sack_loader to main thread,
// a default-constructed std::exception_ptr is a null pointer
Expand Down Expand Up @@ -300,6 +332,15 @@ void RepoSack::update_and_load_repos(libdnf::repo::RepoQuery & repos) {
}


void RepoSack::update_and_load_repos(const libdnf::repo::RepoQuery & repos) {
if (base->get_config().disable_multithreading().get_value()) {
update_and_load_repos_singlethreaded(repos);
} else {
update_and_load_repos_multithreaded(repos);
}
}


void RepoSack::update_and_load_enabled_repos(bool load_system) {
if (load_system) {
// create the system repository if it does not exist
Expand Down