Change default AMG settings #1291

MarcelKoch · 2023-02-23T17:37:11Z

Based on the discussion in #1264 this PR changes some of the default settings of the AMG. The overall goal it to make it possible that our default AMG (no extra settings except the coarsening and stopping criterion) helps to reduce the necessary number of iterations adequately.

I've made each change in the defaults its own commit, with a brief description of my reasoning.

TODO:

update the documentation of the example
evaluate if we should also default the coarsening and/or stopping criterion
- no to both of them.

MarcelKoch · 2023-02-23T17:51:17Z

There seem to be some incompatibilities between the initial_guess_mode and other settings of the multigrid. I'm not sure exactly which settings that touches on. But I've experienced that the mode provided can lead to no convergence at all. I think this needs to be investigated and then we need to warn the user if some incompatible settings are used.

MarcelKoch · 2023-02-24T10:32:09Z

While working on this, I come to realize that the default parameters are much better realized with the factory improvements mentioned in #1162. For example using

using create_factory_t = std::function<std::shared_ptr<LinOpFactory>(std::shared_ptr<const Executor>)>;
std::vector<create_factory_t> pre_smoother;

would allow defining the defaults directly in the parameter declaration:

std::vector<create_factory_t> pre_smoother{[](std::shared_ptr<const Executor> exec) {
  return Ir::build().with_solver(Jacobi::build().on(exec)).on(exec);
};

Perhaps with some helper functionality/change in factory it could be really concise.

include/ginkgo/core/solver/multigrid.hpp

examples/multigrid-preconditioned-solver-customized/build.sh

examples/multigrid-preconditioned-solver-customized/doc/intro.dox

examples/mixed-multigrid-preconditioned-solver/doc/intro.dox

examples/mixed-multigrid-preconditioned-solver/build.sh

examples/mixed-multigrid-preconditioned-solver/CMakeLists.txt

yhmtsai · 2023-03-07T10:30:26Z

core/solver/multigrid.cpp

+            // default coarse grid solver, direct LU
+            // TODO: maybe remove fixed index type
+            auto gen_default_solver = [&] {
+                return experimental::solver::Direct<value_type, int32>::build()


Could we use the experimental class in another public class?

I don't see why not. The default initialization is internal anyway.

AFAIR, the experimental feature can be turned off somehow?
if it is not the case, we can use it.

The experimental features can't be turned off. Only the experimental MPI stuff is disabled if MPI is not available.

examples/ir-ilu-preconditioned-solver/doc/builds-on

...es/multigrid-preconditioned-solver-customized/multigrid-preconditioned-solver-customized.cpp

fritzgoebel

LGTM

MarcelKoch · 2023-03-14T08:10:20Z

rebase!

codecov · 2023-03-14T20:37:51Z

Codecov Report

Patch coverage: 95.52% and project coverage change: -0.46 ⚠️

Comparison is base (012d3aa) 90.81% compared to head (f41e680) 90.36%.

❗ Current head f41e680 differs from pull request most recent head fc83cc1. Consider uploading reports for the commit fc83cc1 to get more accurate results

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #1291      +/-   ##
===========================================
- Coverage    90.81%   90.36%   -0.46%     
===========================================
  Files          570      570              
  Lines        48580    48595      +15     
===========================================
- Hits         44118    43911     -207     
- Misses        4462     4684     +222

Impacted Files	Coverage Δ
core/test/solver/multigrid.cpp	`89.60% <94.44%> (+0.32%)`	⬆️
core/solver/multigrid.cpp	`89.16% <95.34%> (+0.88%)`	⬆️
core/factorization/lu.cpp	`98.14% <100.00%> (ø)`
include/ginkgo/core/solver/multigrid.hpp	`100.00% <100.00%> (ø)`
reference/test/solver/multigrid_kernels.cpp	`92.78% <100.00%> (+0.03%)`	⬆️

... and 14 files with indirect coverage changes

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

Before: identity Now: GMRES with Jacobi, solved to machine precision*10 and max. number of iterations = size(matrix) Rational: The coarse grid solve usually should be exact (up to machine precision). With GMRES we should be able to achieve that for the largest set of matrices. Since we would expect the matrices to be very small, a scalar Jacobi preconditioner should be sufficient, compared to alternatives like ILU.

Before: nothing/identity Now: IR using scalar Jacobi with the default number of smoother steps and 0.9 relaxation factor Rational: Jacobi can be applied to most matrices and it would do something sensible. Alternatively ILU with exact factorization might be a choice.

Before: provided Now: zero Rational: The mode `provided` does not work at all with the current defaults. It leads to no convergence. By changing to mode zero that is fixed

this renames the old example to better show its topic

also changes default value to empty list

Co-authored-by: Fritz Goebel <fritz.goebel@kit.edu>

- example doc Co-authored-by: Fritz Goebel <fritz.goebel@kit.edu>

ginkgo-bot · 2023-03-16T11:30:41Z

Note: This PR changes the Ginkgo ABI:

Functions changes summary: 2 Removed, 0 Changed (138 filtered out), 460 Added functions
Variables changes summary: 0 Removed, 0 Changed, 0 Added variable

For details check the full ABI diff under Artifacts here

sonarqubecloud · 2023-03-16T23:15:00Z

SonarCloud Quality Gate failed.

0 Bugs
0 Vulnerabilities
0 Security Hotspots
12 Code Smells

22.0% Coverage
36.1% Duplication

Release 1.6.0 of Ginkgo. The Ginkgo team is proud to announce the new Ginkgo minor release 1.6.0. This release brings new features such as: - Several building blocks for GPU-resident sparse direct solvers like symbolic and numerical LU and Cholesky factorization, ..., - A distributed Schwarz preconditioner, - New FGMRES and GCR solvers, - Distributed benchmarks for the SpMV operation, solvers, ... - Support for non-default streams in the CUDA and HIP backends, - Mixed precision support for the CSR SpMV, - A new profiling logger which integrates with NVTX, ROCTX, TAU and VTune to provide internal Ginkgo knowledge to most HPC profilers! and much more. If you face an issue, please first check our [known issues page](https://github.com/ginkgo-project/ginkgo/wiki/Known-Issues) and the [open issues list](https://github.com/ginkgo-project/ginkgo/issues) and if you do not find a solution, feel free to [open a new issue](https://github.com/ginkgo-project/ginkgo/issues/new/choose) or ask a question using the [github discussions](https://github.com/ginkgo-project/ginkgo/discussions). Supported systems and requirements: + For all platforms, CMake 3.13+ + C++14 compliant compiler + Linux and macOS + GCC: 5.5+ + clang: 3.9+ + Intel compiler: 2018+ + Apple Clang: 14.0 is tested. Earlier versions might also work. + NVHPC: 22.7+ + Cray Compiler: 14.0.1+ + CUDA module: CUDA 9.2+ or NVHPC 22.7+ + HIP module: ROCm 4.5+ + DPC++ module: Intel OneAPI 2021.3+ with oneMKL and oneDPL. Set the CXX compiler to `dpcpp`. + Windows + MinGW: GCC 5.5+ + Microsoft Visual Studio: VS 2019+ + CUDA module: CUDA 9.2+, Microsoft Visual Studio + OpenMP module: MinGW. ### Version Support Changes + ROCm 4.0+ -> 4.5+ after [#1303](#1303) + Removed Cygwin pipeline and support [#1283](#1283) ### Interface Changes + Due to internal changes, `ConcreteExecutor::run` will now always throw if the corresponding module for the `ConcreteExecutor` is not build [#1234](#1234) + The constructor of `experimental::distributed::Vector` was changed to only accept local vectors as `std::unique_ptr` [#1284](#1284) + The default parameters for the `solver::MultiGrid` were improved. In particular, the smoother defaults to one iteration of `Ir` with `Jacobi` preconditioner, and the coarse grid solver uses the new direct solver with LU factorization. [#1291](#1291) [#1327](#1327) + The `iteration_complete` event gained a more expressive overload with additional parameters, the old overloads were deprecated. [#1288](#1288) [#1327](#1327) ### Deprecations + Deprecated less expressive `iteration_complete` event. Users are advised to now implement the function `void iteration_complete(const LinOp* solver, const LinOp* b, const LinOp* x, const size_type& it, const LinOp* r, const LinOp* tau, const LinOp* implicit_tau_sq, const array<stopping_status>* status, bool stopped)` [#1288](#1288) ### Added Features + A distributed Schwarz preconditioner. [#1248](#1248) + A GCR solver [#1239](#1239) + Flexible Gmres solver [#1244](#1244) + Enable Gmres solver for distributed matrices and vectors [#1201](#1201) + An example that uses Kokkos to assemble the system matrix [#1216](#1216) + A symbolic LU factorization allowing the `gko::experimental::factorization::Lu` and `gko::experimental::solver::Direct` classes to be used for matrices with non-symmetric sparsity pattern [#1210](#1210) + A numerical Cholesky factorization [#1215](#1215) + Symbolic factorizations in host-side operations are now wrapped in a host-side `Operation` to make their execution visible to loggers. This means that profiling loggers and benchmarks are no longer missing a separate entry for their runtime [#1232](#1232) + Symbolic factorization benchmark [#1302](#1302) + The `ProfilerHook` logger allows annotating the Ginkgo execution (apply, operations, ...) for profiling frameworks like NVTX, ROCTX and TAU. [#1055](#1055) + `ProfilerHook::created_(nested_)summary` allows the generation of a lightweight runtime profile over all Ginkgo functions written to a user-defined stream [#1270](#1270) for both host and device timing functionality [#1313](#1313) + It is now possible to enable host buffers for MPI communications at runtime even if the compile option `GINKGO_FORCE_GPU_AWARE_MPI` is set. [#1228](#1228) + A stencil matrices generator (5-pt, 7-pt, 9-pt, and 27-pt) for benchmarks [#1204](#1204) + Distributed benchmarks (multi-vector blas, SpMV, solver) [#1204](#1204) + Benchmarks for CSR sorting and lookup [#1219](#1219) + A timer for MPI benchmarks that reports the longest time [#1217](#1217) + A `timer_method=min|max|average|median` flag for benchmark timing summary [#1294](#1294) + Support for non-default streams in CUDA and HIP executors [#1236](#1236) + METIS integration for nested dissection reordering [#1296](#1296) + SuiteSparse AMD integration for fillin-reducing reordering [#1328](#1328) + Csr mixed-precision SpMV support [#1319](#1319) + A `with_loggers` function for all `Factory` parameters [#1337](#1337) ### Improvements + Improve naming of kernel operations for loggers [#1277](#1277) + Annotate solver iterations in `ProfilerHook` [#1290](#1290) + Allow using the profiler hooks and inline input strings in benchmarks [#1342](#1342) + Allow passing smart pointers in place of raw pointers to most matrix functions. This means that things like `vec->compute_norm2(x.get())` or `vec->compute_norm2(lend(x))` can be simplified to `vec->compute_norm2(x)` [#1279](#1279) [#1261](#1261) + Catch overflows in prefix sum operations, which makes Ginkgo's operations much less likely to crash. This also improves the performance of the prefix sum kernel [#1303](#1303) + Make the installed GinkgoConfig.cmake file relocatable and follow more best practices [#1325](#1325) ### Fixes + Fix OpenMPI version check [#1200](#1200) + Fix the mpi cxx type binding by c binding [#1306](#1306) + Fix runtime failures for one-sided MPI wrapper functions observed on some OpenMPI versions [#1249](#1249) + Disable thread pinning with GPU executors due to poor performance [#1230](#1230) + Fix hwloc version detection [#1266](#1266) + Fix PAPI detection in non-implicit include directories [#1268](#1268) + Fix PAPI support for newer PAPI versions: [#1321](#1321) + Fix pkg-config file generation for library paths outside prefix [#1271](#1271) + Fix various build failures with ROCm 5.4, CUDA 12, and OneAPI 6 [#1214](#1214), [#1235](#1235), [#1251](#1251) + Fix incorrect read for skew-symmetric MatrixMarket files with explicit diagonal entries [#1272](#1272) + Fix handling of missing diagonal entries in symbolic factorizations [#1263](#1263) + Fix segmentation fault in benchmark matrix construction [#1299](#1299) + Fix the stencil matrix creation for benchmarking [#1305](#1305) + Fix the additional residual check in IR [#1307](#1307) + Fix the cuSPARSE CSR SpMM issue on single strided vector when cuda >= 11.6 [#1322](#1322) [#1331](#1331) + Fix Isai generation for large sparsity powers [#1327](#1327) + Fix Ginkgo compilation and test with NVHPC >= 22.7 [#1331](#1331) + Fix Ginkgo compilation of 32 bit binaries with MSVC [#1349](#1349)

MarcelKoch added the 1:ST:ready-for-review This PR is ready for review label Feb 23, 2023

MarcelKoch self-assigned this Feb 23, 2023

ginkgo-bot added reg:build This is related to the build system. mod:core This is related to the core module. reg:example This is related to the examples. type:solver This is related to the solvers labels Feb 23, 2023

fritzgoebel requested changes Mar 3, 2023

View reviewed changes

yhmtsai reviewed Mar 7, 2023

View reviewed changes

examples/ir-ilu-preconditioned-solver/doc/builds-on Outdated Show resolved Hide resolved

...es/multigrid-preconditioned-solver-customized/multigrid-preconditioned-solver-customized.cpp Outdated Show resolved Hide resolved

MarcelKoch requested review from fritzgoebel and yhmtsai March 7, 2023 12:47

yhmtsai approved these changes Mar 8, 2023

View reviewed changes

fritzgoebel approved these changes Mar 9, 2023

View reviewed changes

MarcelKoch added 1:ST:ready-to-merge This PR is ready to merge. and removed 1:ST:ready-for-review This PR is ready for review labels Mar 13, 2023

ginkgo-bot force-pushed the default-amg-settings branch from 9acfdcd to f41e680 Compare March 14, 2023 08:10

MarcelKoch force-pushed the default-amg-settings branch from f41e680 to 6639481 Compare March 16, 2023 10:42

MarcelKoch added 9 commits March 16, 2023 12:15

changes default initial guess

aeed492

Before: provided Now: zero Rational: The mode `provided` does not work at all with the current defaults. It leads to no convergence. By changing to mode zero that is fixed

adds example to show that default setup works

2f844ca

this renames the old example to better show its topic

add simpler example with customized multigrid

952d87f

document default coarse grid solver

5981c2c

document default smoother

4eee629

restore smoother generation if smoother list is empty

34038fd

use default smoother if list is empty

b28a12d

also changes default value to empty list

MarcelKoch and others added 6 commits March 16, 2023 12:15

updates multigrid tests

5ce0f51

converts to CSR within LU if necessary

76df14f

updates coarse grid solver to direct LU

bb0ec98

Co-authored-by: Fritz Goebel <fritz.goebel@kit.edu>

fixes reference tests

1420bed

review updates:

70a506f

- example doc Co-authored-by: Fritz Goebel <fritz.goebel@kit.edu>

fixes default parameter documentation

fc83cc1

MarcelKoch force-pushed the default-amg-settings branch from 6639481 to fc83cc1 Compare March 16, 2023 11:15

MarcelKoch merged commit 8b4c322 into develop Mar 17, 2023

MarcelKoch deleted the default-amg-settings branch March 17, 2023 08:32

MarcelKoch mentioned this pull request Mar 20, 2023

AMG defaults #1264

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Change default AMG settings #1291

Change default AMG settings #1291

MarcelKoch commented Feb 23, 2023 •

edited

Loading

MarcelKoch commented Feb 23, 2023

MarcelKoch commented Feb 24, 2023

yhmtsai Mar 7, 2023

MarcelKoch Mar 7, 2023

yhmtsai Mar 7, 2023

MarcelKoch Mar 7, 2023

fritzgoebel left a comment

MarcelKoch commented Mar 14, 2023

codecov bot commented Mar 14, 2023 •

edited

Loading

ginkgo-bot commented Mar 16, 2023

sonarqubecloud bot commented Mar 16, 2023

Change default AMG settings #1291

Change default AMG settings #1291

Conversation

MarcelKoch commented Feb 23, 2023 • edited Loading

MarcelKoch commented Feb 23, 2023

MarcelKoch commented Feb 24, 2023

yhmtsai Mar 7, 2023

Choose a reason for hiding this comment

MarcelKoch Mar 7, 2023

Choose a reason for hiding this comment

yhmtsai Mar 7, 2023

Choose a reason for hiding this comment

MarcelKoch Mar 7, 2023

Choose a reason for hiding this comment

fritzgoebel left a comment

Choose a reason for hiding this comment

MarcelKoch commented Mar 14, 2023

codecov bot commented Mar 14, 2023 • edited Loading

Codecov Report

ginkgo-bot commented Mar 16, 2023

sonarqubecloud bot commented Mar 16, 2023

MarcelKoch commented Feb 23, 2023 •

edited

Loading

codecov bot commented Mar 14, 2023 •

edited

Loading