OOM installing rustc-dev with 1GB of memory

[bjorn3]

Problem

I tried installing a rust toolchain in a VM I assigned 1GB of ram. (100MB used by the minimal debian install) When installing rustc-dev it got an OOM however.

Steps

1. Create a VM
2. Assign 1GB of ram
3. Install a minimal debian system in the VM. When running it only uses 100MB of ram.
4. Attempt to install a rust toolchain with the rustc-dev component in the VM.
5. OOM

Possible Solution(s)

I thought rustup 1.24 enabled installation on systems with little ram. Is this a regression? Would it be possible to stream the component rather than attempt to load it all at once in memory?

Notes

For now I will be increasing the assigned amount of memory, but IMO it should be possible to install on systems with this little ram, hence this issue.

Rustup version

rustup 1.25.1 (bb60b1e89 2022-07-12)

Installed toolchains

Default host: x86_64-unknown-linux-gnu
rustup home:  /home/bjorn/.rustup

no active toolchain

I attempted to install nightly-2022-12-13-x86_64-unknown-linux-gnu.

[bjorn3]

Assigned 2GB and now rustup maxed at 1.55GB.

[kinnison]

Sounds like the general overheads on that platform are higher than we expect. If you reduce RUSTUP_UNPACK_RAM does that help? You can set it to something tiny (e.g. 10 bytes) to get rustup to tell you the minimum it thinks it needs for your system. You can also set it to something huge (10s of gigabytes) to see if Rustup is mis-detecting the RAM available in your system.

Currently we estimate 200MB for rustup, IO logic, etc. and then we use detected max ram (https://docs.rs/effective-limits/latest/effective_limits/fn.memory_limit.html) to limit our buffer use to leave at least that estimate free.

[rbtcollins]

@bjorn3 ok, so 1GB does not work now. Have you tried @kinnison 's suggestions - in particular setting RUSTUP_UNPACK_RAM very low?

I don't recall any changes to the memory buffer management stuff recently, so not anticipating regressions. Note that the cached component dist files are not memory regulated - but we do only only one in memory cache at a time, and its pull-parsed, so I wouldn't expect a huge footprint.

Perhaps running under memgrind or massif or some such would help too.

[saethlin]

I am currently trying to install rustup on a t4g.nano instance running Ubuntu 22.04 and the unpacking is running out of memory.

info: profile set to 'minimal'
info: setting default host triple to aarch64-unknown-linux-gnu
info: syncing channel updates for 'stable-aarch64-unknown-linux-gnu'
info: latest update on 2023-10-05, rust version 1.73.0 (cc66ad468 2023-10-03)
info: downloading component 'cargo'
info: downloading component 'rust-std'
info: downloading component 'rustc'
 76.5 MiB /  76.5 MiB (100 %)  72.4 MiB/s in  1s ETA:  0s
info: installing component 'cargo'
warning: Ignoring RUSTUP_UNPACK_RAM (10) less than minimum of 33554432.
info: installing component 'rust-std'
warning: Ignoring RUSTUP_UNPACK_RAM (10) less than minimum of 33554432.
Killed

free -h looks like this:

               total        used        free      shared  buff/cache   available
Mem:           419Mi       126Mi       252Mi       0.0Ki        40Mi       274Mi
Swap:             0B          0B          0B

[guest271314]

@saethlin Right. Per the linked closed issue installing the Rust tool chain requires at least 1 GB.

[saethlin]

The linked closed issue is about installing in a tmpfs mount, i.e. a system where disk and memory together come out of a 1 GB pool. I am trying to install on a system with 128 GB of disk and 500 MB of memory (less than that in practice because OS, but you know).

I've forked rustup and started poking at this situation. I'll try to write a patch myself, but if push comes to shove I'll just set up a lot of swap.

[saethlin]

Ah-ha! If I set RUSTUP_IO_THREADS=1 I can install just fine on my instance, peak memory usage of the rustup installer only hits 109 MB.

Which is tricky, because a t4g.nano has 2 vCPUs.

[saethlin]

Another strange variation on this: rust-lang/rust#120406

[rbtcollins]

Its not clear to me that there is a bug here with this comment confirming that RUSTUP_UNPACK_RAM + RUSTUP_IO_THREADS can be used to install with only 110MB of RAM usage.

[guest271314]

FWIW I figured out how to install the Rust tool chain on a temporary file system, Linux live USB, starting out with 1 GB of totoal space available here https://rust-lang.github.io/rustup/installation/other.html

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- --profile minimal --default-toolchain nightly

[saethlin]

It would be really nice if the required configuration here were discoverable, or if rustup's attempt to detect available memory discovered the right behavior on its own. At least when I tried it, the logic correctly detected that memory was tight and reduced unpacking memory then OOMed anyway because it opts back in to too much memory via multithreading.

[djc]

Maybe we could install panic handler at the top level and print some suggestions when we catch a panic resulting from OOM? Not sure how easy to distinguish those would be programmatically. Seems straightforward and somewhat helpful at least...

[saethlin]

I don't think that will do anything. Allocation failures abort, as far as I can tell re-landing the patch that makes OOM call the panic handler is stuck in limbo with a bunch of concerns about the idea: rust-lang/rust#112331