Fix for the v-packet logging while parallelism is enabled. #1647
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Enabling V-Packet logging while the number of threads is set to a value greater than 1 causes the Python kernel to crash. This PR fixes this issue.
The issue stems from trying to append to default Python lists inside a parallel loop in numba (using prange). To avoid multiple threads allocating new memory on the same objects at the same time, I instead cache all vpacket collections ahead of the main loop in a numba typed list. After the main loop, the relevant packet information is extracted back out to create the logging arrays. The vpacket collection caching will always happen even if logging is disabled, but the extraction of the data is not. A potential caveat of this is that there may be more memory usage when a large number of vpackets is requested than normal. This change adds a little less than a second of run time as the instantiation of the vpacket collections and their data retrieval is no longer performed in parallel.
This PR is meant to resolve issue #1614
Motivation and context
The kernel would crash when vpacket logging was enabled with more than 1 thread.
How has this been tested?
Type of change
Checklist