How to limit resources?

[leinad87]

With c.YarnSpawner.mem_limit and c.YarnSpawner.cpu_limit you can configure YARN limits for the notebook. The problem is that instead of generating errors when more space is allocated, YARN automatically kills the container so all the work is lost.

How can we limit the memory and cpu of the notebook to get errors when more memory is requested instead to avoid container kill?

[jcrist]

Sorry, AFAIK there's not a clean way to handle this, your application would have to monitor its resource usage itself. Java can support this because it's VM supports setting memory usage limits, whereas Python's doesn't.