proposal: os: expose accessible physical system memory size

[pbnjay]

Currently runtime.NumCPU() int exposes the number of cpu cores, which is easily useful for limiting concurrent cpu-bound goroutines. It would be similarly useful if there were a runtime.TotalMemory() uint64 call to help limit memory-intensive goroutines from swapping. I think semantics similar to NumCPU are fine - just detect total memory during startup and do not attempt to track available memory.

An example use case where this would be awesome: I have multiple terabytes of data and want to do all-against-all comparisons. By dividing the data into pairwise batches that fit into memory I can process it more efficiently without needing to pre-process or configure based on what system it's running on. Also my development system uses a different OS and has significantly less memory than my production systems (e.g. using a linux-specific package would make testing and development more difficult).

FWIW, this would also provide a platform-independent way to implement #21795 in userspace.

[davecheney]

I don't think this needs to be in the standard library. I suggest proptying it in your own library first. It may never need to be integrated into the stdlib.

…

On Sat, 9 Sep 2017, 12:59 Jeremy Jay ***@***.***> wrote: Currently runtime.NumCPU() int exposes the number of cpu cores, which is easily useful for limiting concurrent cpu-bound goroutines. It would be similarly useful if there were a runtime.TotalMemory() uint64 call to help limit memory-intensive goroutines from swapping. I think semantics similar to NumCPU are fine - just detect total memory during startup and do not attempt to track available memory. An example use case where this would be awesome: I have multiple terabytes of data and want to do all-against-all comparisons. By dividing the data into pairwise batches that fit into memory I can process it more efficiently without needing to pre-process or configure based on what system it's running on. Also my development system uses a different OS and has significantly less memory than my production systems (e.g. using a linux-specific package would make testing and development more difficult). FWIW, this would also provide a platform-independent way to implement #21795 <#21795> in userspace. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#21816>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAcA4ZxXGu-mpekBA2jWMTmEiroNR6pks5sgf8lgaJpZM4PR3LA> .

[pbnjay]

An argument could be made that NumCPU doesn't "need" to be in the stdlib as well, but its utility is clear. CPUs directly relate to constraints on concurrency. Memory likewise relates to constraints on allocation/garbage collection. These are both pretty integral to Go yet memory limits don't get first-class status like cpu cores.

Personally I'd really like to avoid any java-style "-Xmx" mess without resorting to shell wrappers.

I'm happy to prototype - I submitted this proposal to initiate discussion.

[as]

In your use case, runtime.TotalMemory would provide the "detected total memory." However, if we are not providing the available memory, the metric is not useful in practice. To me it suggests that NumCPUs and TotalMemory are not the same. The user does not control what and how many CPUs are used, but might have control over indirect allocation. If all NumCPUs cant be used, the user never sees that. If TotalMemory cant be used, the runtime has a fatal error after a make().

[ianlancetaylor]

I don't understand how to do this in a portable and useful way.

For example, on GNU/Linux, the "total memory available" depends on what other processes on the system are doing. There is no single value for "total memory available" over time.

[pbnjay]

Not available (I agree that is too difficult), just what is physically installed. E.g. what is the hard limit before overcommitting/swapping.

I've started a prototype here: https://github.com/pbnjay/memory

[pbnjay]

My prototype is reasonably complete now, tested on multiple GOOS/GOARCH. I concede package runtime is probably the wrong place for it, but would like to suggest os if this has any chance of seeing the light of day. Again, I don't need/want currently available memory, only what my code has a possibility of addressing without swapping.

@as I think you're forgetting about virtual memory on modern operating systems. I can easily make() a 32GB allocation on my 16GB laptop without a fatal error. When I try to use it, it will swap to disk. If I'm using a lot of that memory all the time, the system will start thrashing, which is exactly the situation I'd like to avoid (using this function's information).

[rsc]

/cc @RLH @aclements

[aclements]

Like others, I'm concerned that this ignores multi-tenancy. There's nothing inherently wrong with the API, of course; it just doesn't seem that useful because of this. On most of my systems, using anything like the total available memory for one process would bring the system to its knees. :) One situation where this may be more useful is in a container-ized environment where you're more isolated from other tenants, but then it seems like you should be querying this information from the container manager (maybe your prototype effectively does this? I'm not sure how container memory limits affect Sysinfo. And some container managers allow much more complicated memory policies).

Certainly if we were to add this to std, it shouldn't go in runtime. The comparison to runtime.NumCPU is rather misleading. They're fundamentally different resources: time is fungible, space is not. That's why it's okay for GOMAXPROCS to default to the total number of CPUs, but not okay for the memory manager to just claim all available RAM. Plus, I'm not positive, but I suspect runtime.NumCPU mainly exists from back when GOMAXPROCS defaulted to 1 and it was often useful to explicitly set it to the number of CPUs. I suspect if we were to redo the runtime API we would leave that out.

This seems like a good package and if it's useful in certain environments, that's great. But it's not clear to me that it passes the bar for std.