Handle NVMLError_Unknown in NVML diagnostics

[pentschev]

Resolves another error reported in #4965, where Distributed is running on nvidia-docker without GPUs installed.

[jrbourbeau]

Thanks for fixing this @pentschev! Does pynvml have a base exception we can catch here? I'm wondering if it makes sense for us to say "try to do pynvml.nvmlInit(), but if pynvml throws any error, then bail"

cc @quasiben

[pentschev]

Thanks for fixing this @pentschev! Does pynvml have a base exception we can catch here? I'm wondering if it makes sense for us to say "try to do pynvml.nvmlInit(), but if pynvml throws any error, then bail"

I generally dislike this idea. It can be very convenient now, but if there's an actual bug in NVML (or CUDA, or something wrong with a GPU, etc.) that's not related to some unexpected ways used (e.g., using nvidia-docker without a GPU), then it becomes very difficult to find where that exception was ignored.

[jrbourbeau]

Fair point. I was mostly concerned about the number of times we've updated this portion of the codebase recently. Though most users are probably fine today since, as you mentioned, the update here is to account for a usage edge case