Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement "memory" resource #240

Open
sjdv1982 opened this issue Dec 16, 2023 · 0 comments
Open

Implement "memory" resource #240

sjdv1982 opened this issue Dec 16, 2023 · 0 comments
Labels
apifix Issues that involve a fix/usability of the high-level API enhancement medium priority performance speed/memory optimization

Comments

@sjdv1982
Copy link
Owner

Right now, the only resource that is really internal to Seamless is "ncores" (see "multi-core-transformations" test). "memory" should be added as well:

  • Use bytes2human to parse the user specification
  • Add it to Seamless itself with a lock mechanism. Something like: wait until enough memory is available, then acquire the global memory allocation lock, then decrease (allocate, claim) the available memory; then release the memory allocation lock. Have a concept of max memory, fail if that is insufficient. Also, for nesting, implement knowledge of the memory claimed by the parent process (unlike cores, don't release it when calling a child)
  • Add it to the mini and mini-dask assistant
  • Optional: add it to Dask as a Dask resource. Rather dangerous (may sliently hang) until my Dask patch has been created and accepted.
@sjdv1982 sjdv1982 added enhancement apifix Issues that involve a fix/usability of the high-level API medium priority performance speed/memory optimization labels Dec 16, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
apifix Issues that involve a fix/usability of the high-level API enhancement medium priority performance speed/memory optimization
Projects
None yet
Development

No branches or pull requests

1 participant