CRIU dump failed since it failed to dump external device file

[zobinHuang]

Hi NVIDIA folks, thanks for this wonderful transparent C/R tool, really helpful!

I encountered an issue when criu dump a pytorch process within a container, after using this tool to suspend the CUDA state.

the commands I used are:

$ cuda-checkpoint --toggle --pid $pid
$ criu dump --tree $pid --images-dir $ckpt_dir --shell-job --display-stats

The error message from the criu dump command looks like:

Warn  (criu/kerndat.c:1593): CRIU was built without libnftables support
Error (criu/files-ext.c:94): Can't dump file 17 of that type [20666] (chr 195:255)
Error (criu/cr-dump.c:1674): Dump files (pid: 406) failed with -1
Error (criu/cr-dump.c:2098): Dumping FAILED.

The software environment I met this issue is:

• Docker image: nvidia/cuda:11.3.1-cudnn8-devel-ubuntu20.04
• CUDA: 11.3
• nvidia driver: 550.54.1
• criu: 3.19

Looks like it's due to the criu failed to dump an external file, and this file seems to be a nvidia device file

$ grep 195 /proc/devices  
195 nvidia
195 nvidia-modeset
195 nvidiactl

Does anyone has any comment on this issue? Thanks for any suggestion :-)

[rst0git]

I encountered an issue when criu dump a pytorch process within a container

@zobinHuang There is another GitHub issue about checkpointing PyTorch with more information about this problem: #4.

[zobinHuang]

I encountered an issue when criu dump a pytorch process within a container

@zobinHuang There is another GitHub issue about checkpointing PyTorch with more information about this problem: #4.

Thanks, would close this issue, looking forward to the fix!