Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

When spawning many processes, the runtime crashes when trying to call mprotect() to set up stack guards #540

Closed
yorickpeterse opened this issue May 22, 2023 · 4 comments
Labels
accepting contributions Issues that are suitable to be worked on by anybody, not just maintainers documentation Documentation related changes
Milestone

Comments

@yorickpeterse
Copy link
Collaborator

Please describe the bug

Extracted from #539:

When spawning many processes (e.g. 100 000), the runtime crashes when trying to mprotect() a memory region used as a guard page:

thread 'proc 3' panicked at 'Failed to set up the stack's guard pages: Os { code: 12, kind: OutOfMemory, message: "Cannot allocate memory" }', rt/src/stack.rs:139:14
stack backtrace:
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace

Please list the exact steps necessary to reproduce the bug

Create test.inko with the following contents:

import std::env::(arguments)
import std::process::(sleep)
import std::time::Duration

class async Sleeper {
  fn async run(output: Channel[Nil]) {
    sleep(Duration.from_secs(2))
    output.send(nil)
  }
}

class async Main {
  fn async main {
    let chan = Channel.new(size: 512)
    let num =
      arguments.opt(0).then fn (v) { Int.from_base10(v) }.unwrap_or(1_000)

    num.times fn (_) { Sleeper {}.run(chan) }
    num.times fn (_) { chan.receive }
  }
}

Then run it using inko run test.inko 100000.

Operating system

Fedora Silverblue 38

Inko version

main

Rust version

1.68.2

@yorickpeterse yorickpeterse added bug Defects, unintended behaviour, etc runtime Changes related to the Rust-based runtime library labels May 22, 2023
@yorickpeterse yorickpeterse added this to the 0.12.0 milestone May 22, 2023
@yorickpeterse yorickpeterse added the accepting contributions Issues that are suitable to be worked on by anybody, not just maintainers label May 22, 2023
@yorickpeterse
Copy link
Collaborator Author

I'm guessing this is a virtual memory limit: 100000 processes would require 100 GiB of virtual memory for the stacks. That is assuming they're allocated in parallel, and we're likely reusing them enough, meaning the limit we're hitting is probably much lower.

@yorickpeterse
Copy link
Collaborator Author

The problem seems to specifically me with mprotect: if I remove the calls to protect the guard page, the program doesn't crash and uses about 430 MiB of physical memory. That's not great, but at least it doesn't crash.

@yorickpeterse
Copy link
Collaborator Author

The problem happens for both the head and the tail of the stack, so the address used doesn't seem to matter.

@yorickpeterse
Copy link
Collaborator Author

This isn't a bug, but rather the limit /proc/sys/vm/max_map_count hitting us. This value defaults to 65530, which isn't enough in this case. If I set it to 655300 the error goes away. I'll add a documentation page about this.

@yorickpeterse yorickpeterse added documentation Documentation related changes and removed bug Defects, unintended behaviour, etc runtime Changes related to the Rust-based runtime library labels May 22, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
accepting contributions Issues that are suitable to be worked on by anybody, not just maintainers documentation Documentation related changes
Projects
None yet
Development

No branches or pull requests

1 participant