Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

known issue: cuda hooks / pami #78

Closed
grahamlopez opened this issue Sep 23, 2019 · 0 comments · Fixed by #173
Closed

known issue: cuda hooks / pami #78

grahamlopez opened this issue Sep 23, 2019 · 0 comments · Fixed by #173
Labels
summit known issue

Comments

@grahamlopez
Copy link
Contributor

When running simple gpu codes with jsrun that do not also have MPI support, sometimes one might run into a warning such as:

CUDA Hook Library: Failed to find symbol mem_find_dreg_entries, ./a.out: undefined symbol: __PAMI_Invalidate_region

This can be solved in a few ways:

  • use jsrun -E LD_PRELOAD=/opt/ibm/spectrum_mpi/lib/pami_451/libpami.so ...
  • use jsrun --smpiargs="off" ...
  • use jsrun --smpiargs="-disable_gpu_hooks ...

Some more discussion can be found at kokkos/kokkos#1985; reports there say it has been reported to IBM.
This was referenced Dec 6, 2019
Open
Merged
benson31 added a commit to benson31/lbann that referenced this issue Jul 12, 2021
@benson31
fix an issue where cuda wouldn't shut down properly on lassen
71f8580 
see olcf/olcf-user-docs#78 for more info
@benson31 benson31 mentioned this issue Jul 12, 2021
Merged
benson31 added a commit to LBANN/lbann that referenced this issue Jul 12, 2021
@benson31
fix an issue where cuda wouldn't shut down properly on lassen (#1916)

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Verified
Learn about vigilant mode
fab5856 
see olcf/olcf-user-docs#78 for more info
szaman19 pushed a commit to szaman19/lbann that referenced this issue Jul 23, 2021
@benson31 @szaman19
fix an issue where cuda wouldn't shut down properly on lassen (LBANN#…
da2ec0f 
…1916)

see olcf/olcf-user-docs#78 for more info
tnat410 pushed a commit to tnat410/lbann that referenced this issue Aug 3, 2021
@benson31
fix an issue where cuda wouldn't shut down properly on lassen (LBANN#…
a52ccc0 
…1916)

see olcf/olcf-user-docs#78 for more info
@nicolasbock nicolasbock mentioned this issue Sep 2, 2021
Closed
@singhbalwinder singhbalwinder mentioned this issue Aug 11, 2022
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
summit known issue
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Add description of CUDA hook issue and workarounds maximumcats/olcf-user-docs
1 participant
@grahamlopez