Large area processing: Dynamically/adaptively start sub-jobs?

[soxofaan]

In the current "partitioned jobs" implementation all sub-jobs are started at once when the main job is started by the user.

To better take back-end load/availability into account, it might be necessary to start jobs in a more on the fly, adaptive approach

[soxofaan]

Also to cover here: retrying of jobs (e.g. on same back-end, or on another back-end?)

[jdries]

Could a more generic version of this issue be to simply support the case where a user submits a lot of jobs (e.g. 10k), which then need to be queued adequately?

[soxofaan]

Could a more generic version of this issue be to simply support the case where a user submits a lot of jobs (e.g. 10k), which then need to be queued adequately?

You mean that the aggregator would then act as an intermediate scheduler to not overload the upstream back-ends? That would indeed be a more generic solution for the original idea of this issue

[jdries]

Exactly, and it would be a useful intermediate step towards supporting large scale processing if I can do splitting myself, and let tracking be handled by the aggregator.

[soxofaan]

Another thing to consider in this context: in cross-backend processing use cases, some jobs can only be created when other dependency jobs are created or successfully finished, because a valid job id/metadata URL is necesarry in a load_result/load_stac node