Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update Particle Container to Pure SoA #3850

Merged
merged 16 commits into from
Jan 28, 2024
Merged

Update Particle Container to Pure SoA #3850

merged 16 commits into from
Jan 28, 2024

Conversation

ax3l
Copy link
Member

@ax3l ax3l commented Apr 19, 2023 •
edited
Loading

Transition particle containers to pure SoA layouts.

  • AMReX: update SoA PC Increment, IncrementWithTotal, Checkpoint, WritePlotFile, Restart; SoA Particle atomicSetID
  • Update in situ vis
  • double check openPMD output.
  • double check plotfile output.
  • fix newly unearthed clang-tidy issues
  • fix QED tests
  • fix EB/scraping tests
  • fix Python_restart_runtime_components (in AMReX restarts)
  • update & pass Python scripts that read structs (positions, ids)
  • run important tests manually on GPU before merge

Fun Mini-Benchmarks on CPU, DP & SP

Same before/after PR on my laptop

./warpx.3d ../../Examples/Tests/performance_tests/automated_test_1_uniform_rest_32ppc amr.max_grid_size=64 amr.n_cell=64 64 64 max_step=5 &
taskset -cp 6 $!

We expect to need to do more work for vectorizing in the coming months.

Fun Mini-Benchmarks on GPU, DP

./warpx.3d ../../Examples/Tests/performance_tests/automated_test_1_uniform_rest_32ppc amr.max_grid_size=256 amr.n_cell=256 256 256 max_step=5

~1-2%ish faster total RT on Perlmutter A100.

Current deposition: same
GatherAndPush: 1-2%
Redistribute_partition: 12%
AddPlasma: same
PushP: same
SortParticlesForDeposition: 230%

@ax3l ax3l added help wanted Extra attention is needed Performance optimization cleaning Clean code, improve readability component: core Core WarpX functionality labels Apr 19, 2023
@ax3l ax3l requested a review from atmyers April 19, 2023 07:21
This was referenced Apr 19, 2023
Closed
Merged
Merged
ax3l
ax3l commented Apr 19, 2023
View reviewed changes
Source/Particles/NamedComponentParticleContainer.H
*/
struct PIdx
{
enum {
w = 0, ///< weight
#if !defined (WARPX_DIM_1D)
x,
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could even call this r for RZ sims :)

ax3l
ax3l commented Apr 19, 2023
View reviewed changes
Source/Particles/NamedComponentParticleContainer.H
y,
#endif
z,
w, ///< weight
ux, uy, uz,
#ifdef WARPX_DIM_RZ
theta, ///< RZ needs all three position components
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could move this forward in the enum to r,z,theta.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And even better, we should make theta a runtime parameter (see reason in AMReX-Codes/pyamrex#243)

@ax3l ax3l mentioned this pull request Apr 27, 2023
Merged
5 tasks
@ax3l ax3l force-pushed the topic-particle-soa branch 5 times, most recently from 427aaa5 to 11adef6 Compare April 27, 2023 03:22
@ax3l ax3l force-pushed the topic-particle-soa branch 6 times, most recently from 53725c5 to 5055fef Compare May 9, 2023 06:33
@ax3l ax3l mentioned this pull request May 23, 2023
Merged
13 tasks
@ax3l ax3l force-pushed the topic-particle-soa branch from 5055fef to a8aeb38 Compare May 30, 2023 15:50
@ax3l ax3l force-pushed the topic-particle-soa branch from a8aeb38 to a6fddc5 Compare June 1, 2023 04:41
@ax3l ax3l force-pushed the topic-particle-soa branch from a6fddc5 to 5a82912 Compare June 6, 2023 00:30
@ax3l ax3l force-pushed the topic-particle-soa branch 2 times, most recently from 103a487 to 08b3ff2 Compare January 24, 2024 22:22
@ax3l ax3l changed the title [WIP] Update Particle Container to Pure SoA Update Particle Container to Pure SoA Jan 24, 2024
@ax3l @atmyers
Update Particle Container to Pure SoA
a97c8bf 
Transition particle containers to pure SoA layouts.

Co-authored-by: Andrew Myers <atmyers@lbl.gov>
@ax3l ax3l force-pushed the topic-particle-soa branch from 08b3ff2 to 2ba8ec0 Compare January 25, 2024 00:52
@ax3l ax3l force-pushed the topic-particle-soa branch from 2ba8ec0 to eb6dbe9 Compare January 25, 2024 03:46
@ax3l
Copy link
Member Author

ax3l commented Jan 27, 2024

Congrats to all who helped @Thierry992, @atmyers, @AlexanderSinn et al. 🎉 👏

@ax3l
Copy link
Member Author

ax3l commented Jan 27, 2024

@atmyers @roelof-groenewald feel free to give this a final review and then we will merge :)

@ax3l ax3l merged commit 94ae119 into ECP-WarpX:development Jan 28, 2024
41 checks passed
@ax3l ax3l deleted the topic-particle-soa branch January 28, 2024 18:37
@roelof-groenewald
Copy link
Member

Congrats! This is awesome! Besides the (upcoming) performance improvements, it will make writing particle routines much easier going forward.

@ax3l
Copy link
Member Author

ax3l commented Jan 28, 2024 •
edited
Loading

Yes, and as nice Python GPU goodies we can now:

  • read & write directly (zero-copy) from/to particle positions and ids via cupy
  • can implement cudf for particle tiles with zero-copy read/write support
  • ...

ax3l added a commit to ax3l/WarpX that referenced this pull request Jan 30, 2024
@ax3l
Revert "Update Particle Container to Pure SoA (ECP-WarpX#3850)"
5363e0e 
This reverts commit 94ae119.
@ax3l ax3l mentioned this pull request Jan 30, 2024
Merged
ax3l added a commit that referenced this pull request Jan 30, 2024
@ax3l
Revert "Update Particle Container to Pure SoA (#3850)" (#4652)
56e4a2a 
This reverts commit 94ae119.
ax3l added a commit to ax3l/WarpX that referenced this pull request Jan 30, 2024
@ax3l
Particle Container to Pure SoA Again
aca62fc 
Transition to new, purely SoA particle containers.

This was originally merged in ECP-WarpX#3850 and reverted in ECP-WarpX#4652, since
we discovered issues loosing particles & laser particles on GPU.
@ax3l ax3l mentioned this pull request Jan 30, 2024
Merged
4 tasks
ax3l added a commit to ax3l/WarpX that referenced this pull request Jan 31, 2024
@ax3l
Particle Container to Pure SoA Again
295e70b 
Transition to new, purely SoA particle containers.

This was originally merged in ECP-WarpX#3850 and reverted in ECP-WarpX#4652, since
we discovered issues loosing particles & laser particles on GPU.
ax3l added a commit to ax3l/WarpX that referenced this pull request Feb 2, 2024
@ax3l
Particle Container to Pure SoA Again
659e1c0 
Transition to new, purely SoA particle containers.

This was originally merged in ECP-WarpX#3850 and reverted in ECP-WarpX#4652, since
we discovered issues loosing particles & laser particles on GPU.
ax3l added a commit to ax3l/WarpX that referenced this pull request Feb 2, 2024
@ax3l
Particle Container to Pure SoA Again
7e597cf 
Transition to new, purely SoA particle containers.

This was originally merged in ECP-WarpX#3850 and reverted in ECP-WarpX#4652, since
we discovered issues loosing particles & laser particles on GPU.
ax3l added a commit to ax3l/WarpX that referenced this pull request Feb 2, 2024
@ax3l
Particle Container to Pure SoA Again
82aec47 
Transition to new, purely SoA particle containers.

This was originally merged in ECP-WarpX#3850 and reverted in ECP-WarpX#4652, since
we discovered issues loosing particles & laser particles on GPU.
ax3l added a commit to ax3l/WarpX that referenced this pull request Feb 2, 2024
@ax3l
Particle Container to Pure SoA Again
1227c91 
Transition to new, purely SoA particle containers.

This was originally merged in ECP-WarpX#3850 and reverted in ECP-WarpX#4652, since
we discovered issues loosing particles & laser particles on GPU.
@ax3l
Copy link
Member Author

ax3l commented Feb 2, 2024

Merge coming for the 24.03 release series in #4653

RemiLehe pushed a commit that referenced this pull request Feb 2, 2024
@ax3l
Particle Container to Pure SoA Again (#4653)
6e332e9 
* AMReX & pyAMReX: Latest `development`

More pure SoA and id handling goodness.

* Particle Container to Pure SoA Again

Transition to new, purely SoA particle containers.

This was originally merged in #3850 and reverted in #4652, since
we discovered issues loosing particles & laser particles on GPU.

* Modernize `idcpu` Treatment

- faster: less emitted operations, no jumps
- cheaper: less used registers
- safer: no read-before-write warnings
- cooler: no explanation needed
atmyers pushed a commit to AMReX-Codes/amrex that referenced this pull request Aug 6, 2024
@ax3l
Bins: Index Type int (#3684)
6f039ae 
## Summary

For performance reasons, `int`/`long` are better index types since they
do not have over/underflow checks and thus vectorize better.

Also, I see narrowing warnings casting from `int` to AMReX' `unsigned
int` in WarpX.

## Additional background

Seen with clang-tidy in ECP-WarpX/WarpX#3850

## Checklist

The proposed changes:
- [ ] fix a bug or incorrect behavior in AMReX
- [ ] add new capabilities to AMReX
- [ ] changes answers in the test suite to more than roundoff level
- [ ] are likely to significantly affect the results of downstream AMReX
users
- [ ] include documentation in the code and/or rst files, if appropriate
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@PhilMiller PhilMiller PhilMiller left review comments

@WeiqunZhang WeiqunZhang WeiqunZhang left review comments

@RevathiJambunathan RevathiJambunathan RevathiJambunathan left review comments

@NeilZaim NeilZaim NeilZaim left review comments

@atmyers atmyers atmyers approved these changes

@roelof-groenewald roelof-groenewald Awaiting requested review from roelof-groenewald

Assignees

@atmyers atmyers

Labels
cleaning Clean code, improve readability component: core Core WarpX functionality Performance optimization
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
7 participants
@ax3l @roelof-groenewald @PhilMiller @atmyers @WeiqunZhang @RevathiJambunathan @NeilZaim