Running pycistarget command aborts aburptly

[Umaarasu]

Hi,
When I run the 'run_pycistarget' command I get error that worker died and along with that the kernel also dies. I have run this command before on a almost similar data set and I did not have any issues. I am not sure why I am getting this. Has someone faced a similar issue?

Error

2023-03-19 13:42:49,008 pycisTarget_wrapper INFO Running cisTarget without promoters for topics_otsu

2023-03-19 13:43:27,989 INFO worker.py:1529 -- Started a local Ray instance. View the dashboard at http://127.0.0.1:8265

(ctx_internal_ray pid=169702) 2023-03-19 13:43:39,001 cisTarget INFO Running cisTarget for Topic1 which has 6675 regions
(ctx_internal_ray pid=169700) 2023-03-19 13:43:40,512 cisTarget INFO Running cisTarget for Topic2 which has 7648 regions
(ctx_internal_ray pid=169699) 2023-03-19 13:43:42,038 cisTarget INFO Running cisTarget for Topic3 which has 4916 regions
(ctx_internal_ray pid=169696) 2023-03-19 13:43:43,710 cisTarget INFO Running cisTarget for Topic4 which has 6803 regions
(ctx_internal_ray pid=169701) 2023-03-19 13:43:44,983 cisTarget INFO Running cisTarget for Topic5 which has 8711 regions

2023-03-19 13:43:46,681 WARNING worker.py:1851 -- A worker died or was killed while executing a task by an unexpected system error. To troubleshoot the problem, check the logs for the dead worker. RayTask ID: d2ef8305dc63ec44126e0f18568f47f296a0e8e501000000 Worker ID: 9e416243b3223d6f92adf1f3824ce48fa9f7191853e28972aa42e3ad Node ID: efc8396370beb37e784e9a7c7c31a566bda203b188819b523b002dce Worker IP address: 10.130.0.67 Worker port: 33813 Worker PID: 169700 Worker exit type: SYSTEM_ERROR Worker exit detail: Worker unexpectedly exits with a connection error code 2. End of file. There are some potential root causes. (1) The process is killed by SIGKILL by OOM killer due to high memory usage. (2) ray stop --force is called. (3) The worker is crashed unexpectedly due to SIGSEGV or other unexpected errors.
2023-03-19 13:43:46,696 WARNING worker.py:1851 -- A worker died or was killed while executing a task by an unexpected system error. To troubleshoot the problem, check the logs for the dead worker. RayTask ID: 0c72a741ce6594cd3d37f9e50c77f8f28fed20e101000000 Worker ID: 52c829fb0b696f73fceccabde8ba7b6a2bdef22b4dfb4a6ec731d09b Node ID: efc8396370beb37e784e9a7c7c31a566bda203b188819b523b002dce Worker IP address: 10.130.0.67 Worker port: 44513 Worker PID: 169702 Worker exit type: SYSTEM_ERROR Worker exit detail: The leased worker has unrecoverable failure. Worker is requested to be destroyed when it is returned.

(ctx_internal_ray pid=169703) 2023-03-19 13:43:46,754 cisTarget INFO Running cisTarget for Topic6 which has 6852 regions

2023-03-19 13:43:47,722 WARNING worker.py:1851 -- A worker died or was killed while executing a task by an unexpected system error. To troubleshoot the problem, check the logs for the dead worker. RayTask ID: 69e2493a035a95a76fa045d09397443019938ca301000000 Worker ID: b91dcfd590b0a6b00c4c34c4490f3c30b927ccc236e00f007372fee3 Node ID: efc8396370beb37e784e9a7c7c31a566bda203b188819b523b002dce Worker IP address: 10.130.0.67 Worker port: 45621 Worker PID: 169699 Worker exit type: SYSTEM_ERROR Worker exit detail: Worker unexpectedly exits with a connection error code 2. End of file. There are some potential root causes. (1) The process is killed by SIGKILL by OOM killer due to high memory usage. (2) ray stop --force is called. (3) The worker is crashed unexpectedly due to SIGSEGV or other unexpected errors.
2023-03-19 13:43:48,110 WARNING worker.py:1851 -- A worker died or was killed while executing a task by an unexpected system error. To troubleshoot the problem, check the logs for the dead worker. RayTask ID: 2f5613bf34da221260e83ef1340b9df6fb94b7be01000000 Worker ID: 9fd594b6fc7e058173cee4a828d2cb44f2b8456e9d7be7eec5c52f8c Node ID: efc8396370beb37e784e9a7c7c31a566bda203b188819b523b002dce Worker IP address: 10.130.0.67 Worker port: 43195 Worker PID: 169701 Worker exit type: SYSTEM_ERROR Worker exit detail: Worker unexpectedly exits with a connection error code 2. End of file. There are some potential root causes. (1) The process is killed by SIGKILL by OOM killer due to high memory usage. (2) ray stop --force is called. (3) The worker is crashed unexpectedly due to SIGSEGV or other unexpected errors.
2023-03-19 13:43:48,325 WARNING worker.py:1851 -- A worker died or was killed while executing a task by an unexpected system error. To troubleshoot the problem, check the logs for the dead worker. RayTask ID: 60b26f8da1ec47350024c13ba96c6f493331e55a01000000 Worker ID: af691ba9c91701f31dd3150195538b0469723ceda51074d66038f6b9 Node ID: efc8396370beb37e784e9a7c7c31a566bda203b188819b523b002dce Worker IP address: 10.130.0.67 Worker port: 39443 Worker PID: 169696 Worker exit type: SYSTEM_ERROR Worker exit detail: Worker unexpectedly exits with a connection error code 2. End of file. There are some potential root causes. (1) The process is killed by SIGKILL by OOM killer due to high memory usage. (2) ray stop --force is called. (3) The worker is crashed unexpectedly due to SIGSEGV or other unexpected errors.
2023-03-19 13:43:48,599 WARNING worker.py:1851 -- A worker died or was killed while executing a task by an unexpected system error. To troubleshoot the problem, check the logs for the dead worker. RayTask ID: 77e43cfd0762e2541c19a0ad18319b5f1ce8407301000000 Worker ID: c1676d2bf1897e2646f20cf29c50102e67a4ee27310d647e438e0ff9 Node ID: efc8396370beb37e784e9a7c7c31a566bda203b188819b523b002dce Worker IP address: 10.130.0.67 Worker port: 35435 Worker PID: 169703 Worker exit type: SYSTEM_ERROR Worker exit detail: Worker unexpectedly exits with a connection error code 2. End of file. There are some potential root causes. (1) The process is killed by SIGKILL by OOM killer due to high memory usage. (2) ray stop --force is called. (3) The worker is crashed unexpectedly due to SIGSEGV or other unexpected errors.

Code:

from scenicplus.wrappers.run_pycistarget import run_pycistarget
run_pycistarget(
region_sets = region_sets,
species = 'homo_sapiens',
save_path = os.path.join(outDir +'motifs'),
ctx_db_path = rankings_db,
dem_db_path = scores_db,
path_to_motif_annotations = motif_annotation,
run_without_promoters = True,
n_cpu = 8,
_temp_dir = os.path.join(tmp_dir),
annotation_version = 'v10nr_clust',
)

[SeppeDeWinter]

Hi @Umaarasu

Could it be that you're running out of memory?

Maybe lowering the number of cpu's (n_cpu) can help.

Best,

Seppe