Enable OpenMP in particle push and coordinate transformation routines.

[atmyers]

Close #195

To Do

• turn tiling on
• set a reasonable default tile size when OMP is selected
• rebase and also OpenMP-accelerate new kernels from Space Charge Solver #162

[ax3l]

Thanks! Do we need to change some defaults, like the dynamic scheduling default we do in WarpX?

[atmyers]

Yes, at minimum we should set turn tiling on and set a reasonable default tile size when OMP is selected - I will update.

[ax3l]

@atmyers thanks! We also merged #162 now. Can you please rebase and OpenMP-enable those routines as well? :)

[atmyers]

Here are the current performance results on the expanding beam test:

dynamic_1.out:TinyProfiler total time across processes [min...avg...max]: 1.35 ... 1.35 ... 1.35
dynamic_2.out:TinyProfiler total time across processes [min...avg...max]: 0.8144 ... 0.8144 ... 0.8144
dynamic_4.out:TinyProfiler total time across processes [min...avg...max]: 0.5541 ... 0.5541 ... 0.5541
static_1.out:TinyProfiler total time across processes [min...avg...max]: 1.353 ... 1.353 ... 1.353
static_2.out:TinyProfiler total time across processes [min...avg...max]: 0.8157 ... 0.8157 ... 0.8157
static_4.out:TinyProfiler total time across processes [min...avg...max]: 0.5614 ... 0.5614 ... 0.5614

[ax3l]

Thank you! Can you perform a little test on a single CPU package comparing MPI w/ 1 OMP thread vs. all OMP threads?

[ax3l]

Can you please also update MFIter loops to be OpenMP accelerated?

[atmyers]

Here is a comparison between pure MPI and pure OMP on the expanding beam test, with diags disabled:

MPI:
1 ranks, 1 threads: 1.357
2 ranks, 1 threads: 0.9569
4 ranks, 1 threads: 0.6122

OMP:
1 ranks, 1 threads: 1.353
1 ranks, 2 threads: 0.8035
1 ranks, 4 threads: 0.6009

[ax3l]

@WeiqunZhang commented:
we can also do tiling on CPU for MFIter loops (not yet done here).

Let's investigate if this helps us here and potentially make it a user option.

Example:
https://github.com/ECP-WarpX/WarpX/blob/3fe406c9701f61e07b23f7123cf0a7bad492c6dc/Source/Diagnostics/ComputeDiagFunctors/BackTransformFunctor.cpp#L110