[Ray Autoscaler] Ray autoscaler does not scale up effectively and fast

[raghumdani]

What happened + What you expected to happen

We run Ray jobs in production. Right after upgrading Ray version from 2.3.0 to 2.20.0, we saw a significant increase in job latency. Upon investigation, we found that autoscaler wasn't spinning up new nodes even when majority of the tasks were in queue waiting to be scheduled which resulted in latency increase overall. We only schedule by memory and these jobs weren't using full memory they requested for. However, we expect autoscaler to still spin up new nodes to serve the demand. This issues does not occur with SPREAD scheduling strategy however (not sure why!).

Versions / Dependencies

Ray: 2.20.0
OS: Ubuntu 20.04
Python: 3.10

Reproduction script

You might need to install deltacat by pip3 install deltacat. Each worker has 31 CPU, 220GB memory and 10000 max_tasks.

from deltacat.utils.ray_utils.concurrency import (
    invoke_parallel,
)
import ray
import time

ray.init(address='auto')

@ray.remote
def i_will_return_after_t_secs(t):
    import numpy as np

    all_arrs = []

    for _ in range(10):
        all_arrs.append(np.random.rand(100000000)) # 0.8GB

    # 8 GB overall

    import time
    time.sleep(t)

def options_provider(*args, **kwargs):
    return {"num_cpus": 0.01, "memory": 23387571131, "resources": {'max_tasks': 0.001} }

items_to_run = [120 for _ in range(14000)]

start = time.monotonic()
print(f"Now starting: {start}")

pending = invoke_parallel(items_to_run, i_will_return_after_t_secs, max_parallelism=4096, options_provider=options_provider)

end_invoke = time.monotonic()

print(f"Now ended invoke: {end_invoke - start}")

ray.get(pending)

end = time.monotonic()
print(f"Complete: {end - start}")

# 2.3 used 460 nodes while 2.20 used 364 nodes at max.

Issue Severity

None

[anyscalesam]

There's a bunch of these we should look into it further... kicking this back to re-triage

[Moonquakes]

I also found a similar problem, and the speed of Ray Scheduler seemed to be affected.
In my scenario, 1000 nodes were used, each with two CPUs. There were a total of 10,000 tasks, each of which required two CPUs. I found that by default, sometimes the resources of these 1000 nodes were not fully utilized, and only about 800 nodes were used. When I turned on SPREAD scheduling, there was no such problem. I would like to ask if there is any difference in the underlying implementation of these two scheduling methods? cc @anyscalesam @raghumdani

[jjyao]

@Moonquakes Seems you have a different issue about scheduler not autoscaler. Do you mind creating a new GH ticket discussing your issue with a repro?

[DmitriGekhtman]

I was able to reproduce this with the following setup:

60-CPU nodes
min_workers=0
Large upscaling speed
4000 1.5-CPU tasks, each sleeps for 10 minutes

40 tasks should fit per node, so you should get 100 nodes scaled up immediately.

Instead, upscaling happens slowly in chunks.

The pending tasks do show up in ray status, so it is surprising that the resource demand scheduler does not translate them into the appropriate node shapes.
This seems like an issue in the resource demand scheduler Python code; I can probably looking into solving this one
(, as opposed to #36926, which is a C++ Ray core problem, for which I have no idea)

[DmitriGekhtman]

One thing I find interesting is that, in the set-up I described in my last comment, I get an initial scale up of exactly 25 nodes each time, corresponding to exactly 1000 tasks being processed.
I wonder if there's some internal cap at play here.

[DmitriGekhtman]

Ok, I can definitely confirm this pattern:
4000 1.5-CPU tasks submitted -> Initial scale-up to 25 60-CPU nodes, corresponding to 1000 tasks.
2000 3.0-CPU tasks submitted -> Initial scale-up to 50 60-CPU nodes, corresponding to 1000 tasks
1000 6.0-CPU tasks submitted -> Initial scale-up to 100 60-CPU nodes, corresponding to 1000 tasks.

[DmitriGekhtman]

Ok, I think what I'm observing is AUTOSCALER_MAX_RESOURCE_DEMAND_VECTOR_SIZE = 1000

But that limit has been there for many years, so it's weird if upgrading to a recent version would suddenly slow things down.

It might be nice to make that limit configurable via env variable.

[DmitriGekhtman]

#50176 provides a workaround