More on Gpu kernel fusing

[WeiqunZhang]

Summary

• Add Gpu::KernelInfo argument to ParallelFor to allow the user to indicate
  whether the kernel is an candidate for fusing.

• For MFIter, if the local size is less or equal to 3, the fuse region is
  turned on and small kernels marked fusable will be fused.

• Add launch macros for fusing.

• Add fusing to a number of functions used by linear solvers. Note that
  there are a lot more amrex functions need to be updated for fusing.

• Optimize reduction for bottom solve.

• Consolidate memcpy in communication functions.

• Option to use device memory in communication kernels for packing and
  unpacking buffers. But it's currently turned off because the performance
  was not improved in testing. In fact, it was worse than using pinned
  memory. But this might change in the future. So the option is kept.

Checklist

The proposed changes:

• fix a bug or incorrect behavior in AMReX
• add new capabilities to AMReX
• changes answers in the test suite to more than roundoff level
• are likely to significantly affect the results of downstream AMReX users
• are described in the proposed changes to the AMReX documentation, if appropriate

[WeiqunZhang]

This is still a draft. I submitted this to run CI and I like the diff provided by github.

I have done scaling test using amrex/Tutorials/LinearSolvers/ABecLaplacian_C on summit with 256^3 cells per node recently. The results are shown below.

| Branch      | # of node |   Time |
|-------------+-----------+--------|
| development |         1 | 0.2557 |
| fusing      |         1 | 0.2204 |
|-------------+-----------+--------|
| development |         8 | 0.3232 |
| fusing      |         8 | 0.2820 |
|-------------+-----------+--------|
| development |        64 | 0.3725 |
| fusing      |        64 | 0.3284 |
|-------------+-----------+--------|
| development |       512 | 0.4703 |
| fusing      |       512 | 0.4063 |
|-------------+-----------+--------|
| development |      4096 | 0.6459 |
| fusing      |      4096 | 0.5769 |

[WeiqunZhang]

This PR is ready for review now. I have done regression tests on AMReX and various application codes on both GPU and CPU. It has passed all tests except for a few CPU tests using cell-centered linear solvers. There are roundoff errors in these tests due to a change in MultiFab::Dot. See https://ccse.lbl.gov/pub/RegressionTesting/IAMR/2020-09-07-001/index.html for example. In a different branch, fusing_regtest, I reverted the CPU path in that function to the current way in development. The "failed" tests passes with the fusing_regtest. The difference of the PR branch fusing and fusing_regtest can be seen here, WeiqunZhang/amrex@fusing...WeiqunZhang:fusing_regtest.

[kngott]

Was the increase in nthreads tested? Or just changing to be consistent with other launches?

Should this be based on MAX_NUM_THREADS or another more universal variable, so it can be easily adjusted for each architecture type without being forgotten?

[WeiqunZhang]

Yes, it was tested. But I didn't really see much difference.