Wrap SuiteSparse AMD #1328

upsj · 2023-04-26T11:24:12Z

This adds some files from SuiteSparse implementing the symmetric AMD reordering algorithm as a stop-gap measure until we have our own AMD implementation.
It also implements a fallback Csr sorting algorithm for CUDA/HIP 64 bit indices using Thrust to make a handful of tests pass.

fritzgoebel · 2023-05-02T12:17:53Z

If this should be usable with the ScaledReordered interface, it would need to be a ReorderingBase. I'd be open to change the approach to ScaledReordered, but I think we should be consistent with reordering implementations and the interface.

core/reorder/amd.cpp

core/test/reorder/amd.cpp

yhmtsai

do we need to extract the amd reordering file into our repo?

third_party/SuiteSparse/wrapper64.cpp

third_party/SuiteSparse/wrapper32.cpp

core/CMakeLists.txt

core/test/reorder/amd.cpp

yhmtsai · 2023-05-22T13:28:08Z

core/test/reorder/amd.cpp

+            int fillin_permuted =
+                factorized_permuted_mtx->get_num_stored_elements() -
+                permuted_mtx->get_num_stored_elements();
+            ASSERT_LE(fillin_permuted, fillin_mtx - this->fillin_reduction);


Is fillin_reduction also an answer here?

I don't follow entirely - fillin_reduction stores how much the reordering should reduce the fill-in at least

yhmtsai · 2023-05-22T13:28:42Z

core/test/reorder/amd.cpp

+                     this->mtx->get_num_stored_elements();
+    int fillin_permuted = factorized_permuted_mtx->get_num_stored_elements() -
+                          permuted_mtx->get_num_stored_elements();
+    ASSERT_LE(fillin_permuted, fillin_mtx * 2 / 5);


where's 2/5 from?

the reordering leads to a 60% fill-in reduction, I wanted to avoid hard-coding the fill-in

where's 60% from?
It's from the property or just from this testing case.

just the specific test case

core/reorder/amd.cpp

yhmtsai

do we need to extract the amd reordering file into our repo?

upsj · 2023-05-23T08:42:46Z

IMO Ginkgo already has a lot of different build configurations to keep track of, and with AMD being rather small, and to still enable building Ginkgo without internet access, I would prefer building the code every time. In the long term, we will probably replace it by our own AMD implementation

yhmtsai · 2023-05-23T13:39:18Z

I think building ginkgo without internet access is a different thing and should use a different way from putting all codes in this repo. For example, when we release or merge pr into develop, there's an action or compressed files for the installation without internet access except for those GPL packages.
Putting code directly in our repo is hard to track the package version.
also replacing it with own AMD implementation should not be different no matter which way we go, right?

From the license in the third party, some packages contain the GPL license. Will the license file be also under GPL?

upsj · 2023-05-29T08:03:23Z

@yhmtsai all our dependencies are licensed under BSD 3-clause. The AMD implementation in Suitesparse has proper version numbers, and has only changed superficially since 2013, so I believe we don't need to worry about having to track upstream changes. Vendoring CAS was also done to make builds of libginkgo (excluding tests etc) possible from airgapped systems. I don't think we should give up that property for effectively 3 source code files from a large repository.

Technically a license file itself is under copyright owned by the author, but since all licenses listed in there require reproduction of the license text in derivative works, we are good. We could instead only include the AMD/Doc/LICENSE.txt though, which would slim down the PR a bit.

yhmtsai

as the decision in the meeting. LGTM

- remove unused licenses - format files - clarify workspace size computation Co-authored-by: Fritz Göbel <fritz.goebel@kit.edu> Co-authored-by: Yuhsiang M. Tsai <yhmtsai@gmail.com>

fritzgoebel

LGTM

sonarqubecloud · 2023-05-31T11:36:30Z

SonarCloud Quality Gate failed.

0 Bugs
0 Vulnerabilities
0 Security Hotspots
197 Code Smells

91.0% Coverage
0.0% Duplication

Release 1.6.0 of Ginkgo. The Ginkgo team is proud to announce the new Ginkgo minor release 1.6.0. This release brings new features such as: - Several building blocks for GPU-resident sparse direct solvers like symbolic and numerical LU and Cholesky factorization, ..., - A distributed Schwarz preconditioner, - New FGMRES and GCR solvers, - Distributed benchmarks for the SpMV operation, solvers, ... - Support for non-default streams in the CUDA and HIP backends, - Mixed precision support for the CSR SpMV, - A new profiling logger which integrates with NVTX, ROCTX, TAU and VTune to provide internal Ginkgo knowledge to most HPC profilers! and much more. If you face an issue, please first check our [known issues page](https://github.com/ginkgo-project/ginkgo/wiki/Known-Issues) and the [open issues list](https://github.com/ginkgo-project/ginkgo/issues) and if you do not find a solution, feel free to [open a new issue](https://github.com/ginkgo-project/ginkgo/issues/new/choose) or ask a question using the [github discussions](https://github.com/ginkgo-project/ginkgo/discussions). Supported systems and requirements: + For all platforms, CMake 3.13+ + C++14 compliant compiler + Linux and macOS + GCC: 5.5+ + clang: 3.9+ + Intel compiler: 2018+ + Apple Clang: 14.0 is tested. Earlier versions might also work. + NVHPC: 22.7+ + Cray Compiler: 14.0.1+ + CUDA module: CUDA 9.2+ or NVHPC 22.7+ + HIP module: ROCm 4.5+ + DPC++ module: Intel OneAPI 2021.3+ with oneMKL and oneDPL. Set the CXX compiler to `dpcpp`. + Windows + MinGW: GCC 5.5+ + Microsoft Visual Studio: VS 2019+ + CUDA module: CUDA 9.2+, Microsoft Visual Studio + OpenMP module: MinGW. ### Version Support Changes + ROCm 4.0+ -> 4.5+ after [#1303](#1303) + Removed Cygwin pipeline and support [#1283](#1283) ### Interface Changes + Due to internal changes, `ConcreteExecutor::run` will now always throw if the corresponding module for the `ConcreteExecutor` is not build [#1234](#1234) + The constructor of `experimental::distributed::Vector` was changed to only accept local vectors as `std::unique_ptr` [#1284](#1284) + The default parameters for the `solver::MultiGrid` were improved. In particular, the smoother defaults to one iteration of `Ir` with `Jacobi` preconditioner, and the coarse grid solver uses the new direct solver with LU factorization. [#1291](#1291) [#1327](#1327) + The `iteration_complete` event gained a more expressive overload with additional parameters, the old overloads were deprecated. [#1288](#1288) [#1327](#1327) ### Deprecations + Deprecated less expressive `iteration_complete` event. Users are advised to now implement the function `void iteration_complete(const LinOp* solver, const LinOp* b, const LinOp* x, const size_type& it, const LinOp* r, const LinOp* tau, const LinOp* implicit_tau_sq, const array<stopping_status>* status, bool stopped)` [#1288](#1288) ### Added Features + A distributed Schwarz preconditioner. [#1248](#1248) + A GCR solver [#1239](#1239) + Flexible Gmres solver [#1244](#1244) + Enable Gmres solver for distributed matrices and vectors [#1201](#1201) + An example that uses Kokkos to assemble the system matrix [#1216](#1216) + A symbolic LU factorization allowing the `gko::experimental::factorization::Lu` and `gko::experimental::solver::Direct` classes to be used for matrices with non-symmetric sparsity pattern [#1210](#1210) + A numerical Cholesky factorization [#1215](#1215) + Symbolic factorizations in host-side operations are now wrapped in a host-side `Operation` to make their execution visible to loggers. This means that profiling loggers and benchmarks are no longer missing a separate entry for their runtime [#1232](#1232) + Symbolic factorization benchmark [#1302](#1302) + The `ProfilerHook` logger allows annotating the Ginkgo execution (apply, operations, ...) for profiling frameworks like NVTX, ROCTX and TAU. [#1055](#1055) + `ProfilerHook::created_(nested_)summary` allows the generation of a lightweight runtime profile over all Ginkgo functions written to a user-defined stream [#1270](#1270) for both host and device timing functionality [#1313](#1313) + It is now possible to enable host buffers for MPI communications at runtime even if the compile option `GINKGO_FORCE_GPU_AWARE_MPI` is set. [#1228](#1228) + A stencil matrices generator (5-pt, 7-pt, 9-pt, and 27-pt) for benchmarks [#1204](#1204) + Distributed benchmarks (multi-vector blas, SpMV, solver) [#1204](#1204) + Benchmarks for CSR sorting and lookup [#1219](#1219) + A timer for MPI benchmarks that reports the longest time [#1217](#1217) + A `timer_method=min|max|average|median` flag for benchmark timing summary [#1294](#1294) + Support for non-default streams in CUDA and HIP executors [#1236](#1236) + METIS integration for nested dissection reordering [#1296](#1296) + SuiteSparse AMD integration for fillin-reducing reordering [#1328](#1328) + Csr mixed-precision SpMV support [#1319](#1319) + A `with_loggers` function for all `Factory` parameters [#1337](#1337) ### Improvements + Improve naming of kernel operations for loggers [#1277](#1277) + Annotate solver iterations in `ProfilerHook` [#1290](#1290) + Allow using the profiler hooks and inline input strings in benchmarks [#1342](#1342) + Allow passing smart pointers in place of raw pointers to most matrix functions. This means that things like `vec->compute_norm2(x.get())` or `vec->compute_norm2(lend(x))` can be simplified to `vec->compute_norm2(x)` [#1279](#1279) [#1261](#1261) + Catch overflows in prefix sum operations, which makes Ginkgo's operations much less likely to crash. This also improves the performance of the prefix sum kernel [#1303](#1303) + Make the installed GinkgoConfig.cmake file relocatable and follow more best practices [#1325](#1325) ### Fixes + Fix OpenMPI version check [#1200](#1200) + Fix the mpi cxx type binding by c binding [#1306](#1306) + Fix runtime failures for one-sided MPI wrapper functions observed on some OpenMPI versions [#1249](#1249) + Disable thread pinning with GPU executors due to poor performance [#1230](#1230) + Fix hwloc version detection [#1266](#1266) + Fix PAPI detection in non-implicit include directories [#1268](#1268) + Fix PAPI support for newer PAPI versions: [#1321](#1321) + Fix pkg-config file generation for library paths outside prefix [#1271](#1271) + Fix various build failures with ROCm 5.4, CUDA 12, and OneAPI 6 [#1214](#1214), [#1235](#1235), [#1251](#1251) + Fix incorrect read for skew-symmetric MatrixMarket files with explicit diagonal entries [#1272](#1272) + Fix handling of missing diagonal entries in symbolic factorizations [#1263](#1263) + Fix segmentation fault in benchmark matrix construction [#1299](#1299) + Fix the stencil matrix creation for benchmarking [#1305](#1305) + Fix the additional residual check in IR [#1307](#1307) + Fix the cuSPARSE CSR SpMM issue on single strided vector when cuda >= 11.6 [#1322](#1322) [#1331](#1331) + Fix Isai generation for large sparsity powers [#1327](#1327) + Fix Ginkgo compilation and test with NVHPC >= 22.7 [#1331](#1331) + Fix Ginkgo compilation of 32 bit binaries with MSVC [#1349](#1349)

upsj added the 1:ST:WIP This PR is a work in progress. Not ready for review. label Apr 26, 2023

upsj self-assigned this Apr 26, 2023

upsj force-pushed the suitesparse_amd branch from f56ce44 to effe4fa Compare April 26, 2023 20:54

upsj changed the title ~~Wrap SuiteSparse AMD and COLAMD~~ Wrap SuiteSparse AMD Apr 26, 2023

upsj added 1:ST:ready-for-review This PR is ready for review and removed 1:ST:WIP This PR is a work in progress. Not ready for review. labels Apr 26, 2023

upsj force-pushed the suitesparse_amd branch from effe4fa to 662a444 Compare April 26, 2023 21:18

upsj requested a review from a team April 26, 2023 21:21

upsj force-pushed the suitesparse_amd branch 2 times, most recently from a419fb7 to baf7e30 Compare April 26, 2023 21:26

ginkgo-project deleted a comment from ginkgo-bot Apr 27, 2023

upsj force-pushed the suitesparse_amd branch from baf7e30 to dbe99c0 Compare May 2, 2023 14:54

fritzgoebel reviewed May 17, 2023

View reviewed changes

core/reorder/amd.cpp Show resolved Hide resolved

core/reorder/amd.cpp Outdated Show resolved Hide resolved

core/test/reorder/amd.cpp Show resolved Hide resolved

core/test/reorder/amd.cpp Show resolved Hide resolved

yhmtsai reviewed May 22, 2023

View reviewed changes

yhmtsai approved these changes May 30, 2023

View reviewed changes

upsj added 3 commits May 30, 2023 14:30

add scaffolding

7bdd464

add plain SuiteSparse

fb427f9

fix include paths

fd15bdc

upsj and others added 8 commits May 30, 2023 14:30

remove debug code

1563ab7

wrap SuiteSparse AMD

83cfd10

add CUDA/HIP Csr fallback sorting for int64

0acd9ba

faster Csr fallback sorting

519f2da

support (skipping) symmetrization and sorting

75726e9

add AMD licensing information

f55ea90

review updates

c9c0378

- remove unused licenses - format files - clarify workspace size computation Co-authored-by: Fritz Göbel <fritz.goebel@kit.edu> Co-authored-by: Yuhsiang M. Tsai <yhmtsai@gmail.com>

use temporary float matrix in AMD

5f5590e

upsj force-pushed the suitesparse_amd branch from dbe99c0 to 5f5590e Compare May 30, 2023 12:52

upsj added the 1:ST:run-full-test label May 30, 2023

fritzgoebel approved these changes May 30, 2023

View reviewed changes

upsj added 1:ST:ready-to-merge This PR is ready to merge. and removed 1:ST:run-full-test 1:ST:ready-for-review This PR is ready for review labels May 30, 2023

upsj merged commit e3467eb into develop May 31, 2023

upsj deleted the suitesparse_amd branch May 31, 2023 05:02

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Wrap SuiteSparse AMD #1328

Wrap SuiteSparse AMD #1328

upsj commented Apr 26, 2023 •

edited

Loading

fritzgoebel commented May 2, 2023

yhmtsai left a comment

yhmtsai May 22, 2023

upsj May 23, 2023

yhmtsai May 22, 2023

upsj May 23, 2023

yhmtsai May 30, 2023

upsj May 30, 2023

yhmtsai left a comment

upsj commented May 23, 2023

yhmtsai commented May 23, 2023 •

edited

Loading

upsj commented May 29, 2023

yhmtsai left a comment

fritzgoebel left a comment

sonarqubecloud bot commented May 31, 2023

Wrap SuiteSparse AMD #1328

Wrap SuiteSparse AMD #1328

Conversation

upsj commented Apr 26, 2023 • edited Loading

fritzgoebel commented May 2, 2023

yhmtsai left a comment

Choose a reason for hiding this comment

yhmtsai May 22, 2023

Choose a reason for hiding this comment

upsj May 23, 2023

Choose a reason for hiding this comment

yhmtsai May 22, 2023

Choose a reason for hiding this comment

upsj May 23, 2023

Choose a reason for hiding this comment

yhmtsai May 30, 2023

Choose a reason for hiding this comment

upsj May 30, 2023

Choose a reason for hiding this comment

yhmtsai left a comment

Choose a reason for hiding this comment

upsj commented May 23, 2023

yhmtsai commented May 23, 2023 • edited Loading

upsj commented May 29, 2023

yhmtsai left a comment

Choose a reason for hiding this comment

fritzgoebel left a comment

Choose a reason for hiding this comment

sonarqubecloud bot commented May 31, 2023

upsj commented Apr 26, 2023 •

edited

Loading

yhmtsai commented May 23, 2023 •

edited

Loading