Adding CUDA SDK Toolkit ? #6

adrastee · 2020-06-18T08:26:15Z

Hi,
Using dizcza/docker-hashcat (latest tag), and starting hashcat -b, it outputs :

* Device #1: CUDA SDK Toolkit installation NOT detected.
             CUDA SDK Toolkit installation required for proper device support and utilization
             Falling back to OpenCL Runtime

While it's just a warning, it seems to have bad effects on performance/stability.

Could you kindly add CUDA SDK Toolkit installation to the docker file?

Thanks!

The text was updated successfully, but these errors were encountered:

dizcza · 2020-06-18T11:45:41Z

Yeap. I was thinking of adding CUDA support. But I don't have a GPU anymore to test it.

Would you be willing to test the docker build? This will probably require changing the dockerfile once I push a draft - the way I think it should work - and building it manually if the installation fails.

adrastee · 2020-06-18T13:55:55Z

Ok for testing! Let me know which docker I should load.
BTW CUDA is quicker than OpenCL according to the last version of hashcat.

dizcza · 2020-06-18T19:17:01Z

Actually, this might be easier than I thought. I just replaced manual OpenCL driver installation instructions with
FROM nvidia/cuda:10.2-runtime-ubuntu18.04
as you can find in the "cuda" branch. Check it out and give me the feedback whether it works or not:

dizcza/docker-hashcat:cuda

adrastee · 2020-06-19T18:08:31Z

Sadly it does not work.
Using dizcza/docker-hashcat:cuda on a 1080 Ti:

$ hashcat -I
hashcat (v6.0.0-3-g53fb4050) starting...

clGetPlatformIDs(): CL_PLATFORM_NOT_FOUND_KHR

ATTENTION! No OpenCL-compatible or CUDA-compatible platform found.
You are probably missing the OpenCL or CUDA runtime installation.
* NVIDIA GPUs require this runtime and/or driver (both):
  "NVIDIA Driver" (440.64 or later)
  "CUDA Toolkit" (9.0 or later)

$ nvidia-smi
Fri Jun 19 18:05:31 2020
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 440.64.00    Driver Version: 440.64.00    CUDA Version: 10.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 108...  On   | 00000000:08:00.0 Off |                  N/A |
| 23%   56C    P8    10W / 125W |      1MiB / 11178MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

dizcza · 2020-06-19T18:27:12Z

Damn. Nvidia team cannot even ship a proper docker image for their products - the same situation happened with their opencl docker image, as you mentioned in your one-year-old last issue.

Please do the steps described in this post. After you remove all cuda-related drivers, the official website proposes the next installation steps:

wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/cuda-ubuntu1804.pin
sudo mv cuda-ubuntu1804.pin /etc/apt/preferences.d/cuda-repository-pin-600
wget http://developer.download.nvidia.com/compute/cuda/11.0.1/local_installers/cuda-repo-ubuntu1804-11-0-local_11.0.1-450.36.06-1_amd64.deb
sudo dpkg -i cuda-repo-ubuntu1804-11-0-local_11.0.1-450.36.06-1_amd64.deb
sudo apt-key add /var/cuda-repo-ubuntu1804-11-0-local/7fa2af80.pub
sudo apt-get update
sudo apt-get -y install cuda

which you can easily find by following https://developer.nvidia.com/cuda-downloads.

Once you're done, please report.

adrastee · 2020-06-21T10:00:04Z

I think your should load nvidia/cuda:10.2-devel-ubuntu18.04 instead of nvidia/cuda:10.2-runtime-ubuntu18.04
runtime docker does NOT include CUDA-toolkit, while devel does => no need to uninstall/install new things.
devel: Builds on the runtime and includes headers, development tools for building CUDA images

Then simply download hashcat:

$ hashcat -I
hashcat (v6.0.0-3-g53fb4050) starting...

clGetPlatformIDs(): CL_PLATFORM_NOT_FOUND_KHR

CUDA Info:
==========

CUDA.Version.: 10.2

Backend Device ID #1
  Name...........: GeForce GTX 1080 Ti
  Processor(s)...: 28
  Clock..........: 1582
  Memory.Total...: 11178 MB
  Memory.Free....: 11039 MB

$ hashcat -b -m 2500 -w 4
hashcat (v6.0.0-3-g53fb4050) starting in benchmark mode...

clGetPlatformIDs(): CL_PLATFORM_NOT_FOUND_KHR

CUDA API (CUDA 10.2)
====================
* Device #1: GeForce GTX 1080 Ti, 11039/11178 MB, 28MCU

Benchmark relevant options:
===========================
* --workload-profile=4
Hashmode: 2500 - WPA-EAPOL-PBKDF2 (Iterations: 4095)
Speed.#1.........:   603.5 kH/s (357.10ms) @ Accel:64 Loops:512 Thr:1024 Vec:1

Issue:
We notice the error CL_PLATFORM_NOT_FOUND_KHR. Maybe because OpenCL is not properly installed? clinfo returns 0 device, so it confirms OpenCL platforms is not installed.
I don't really know if OpenCL is mandatory when CUDA is here.

adrastee · 2020-06-21T10:11:03Z

Side note , using dizcza/docker-hashcat:latest (=> using OpenCL only, no CUDA), performance on WPA (mode 2500) are lower:

$ hashcat -b -w4 -m2500
hashcat (v6.0.0) starting in benchmark mode...

* Device #1: CUDA SDK Toolkit installation NOT detected.
             CUDA SDK Toolkit installation required for proper device support and utilization
             Falling back to OpenCL Runtime

OpenCL API (OpenCL 1.2 CUDA 10.0.206) - Platform #1 [NVIDIA Corporation]
========================================================================
* Device #1: GeForce GTX 1080 Ti, 11008/11178 MB (2794 MB allocatable), 28MCU

Benchmark relevant options:
===========================
* --workload-profile=4

Hashmode: 2500 - WPA-EAPOL-PBKDF2 (Iterations: 4095)

Speed.#1.........:   578.0 kH/s (395.67ms) @ Accel:64 Loops:512 Thr:1024 Vec:1

Roughly +5% with CUDA 10.2..

dizcza · 2020-06-21T13:57:31Z

Cool, thank you! Then I'll switch to devel, as you suggest.

I don't really know if OpenCL is mandatory when CUDA is here.

According to the description opencl is not mandatory if cuda is installed. Moreover, I get the impression that having cuda only is recommended if your device supports it because hashcat priorities cuda (if it's available) over opencl and thus installing opencl-related software will only contaminate your system unless you use them elsewhere.

runtime docker does NOT include CUDA-toolkit, while devel does

Hm, this is strange. Did I misinterpret their nvidia docker description...

runtime: Builds on the base and includes the CUDA math libraries, and NCCL. A runtime image that also includes cuDNN is available.
devel: Builds on the runtime and includes headers, development tools for building CUDA images. These images are particularly useful for multi-stage builds.

I guess the warning "clGetPlatformIDs(): CL_PLATFORM_NOT_FOUND_KHR" tells not only that ocl-icd-libopencl1 (requires by clinfo) is missing system-wide but also that hashcat does not see anything in /etc/OpenCL/vendors/ folder. I think this warning can be ignored.

Here is what I did:

nvidia/cuda:10.2-devel-ubuntu18.04 for the cuda tag.
latest = cuda + opencl, meaning that apart from nvidia/cuda:10.2-devel-ubuntu18.04 I kept opencl-related installation.

I'd be nice if you try :latest tag (update it beforehand) and give me the feedback if it works for you. It should also get rid of the warning you see with cuda-only backend.

ghost · 2020-06-21T21:54:05Z

Something strange has changed here, I've been unable to load the :latest tag with Vast.ai sessions, :pocl loads correctly and I can drop in to an attached shell, but neither :latest nor :cuda seem to start and run correctly.

dizcza · 2020-06-22T06:59:36Z

Thank you for letting me know.
Do you have Nvidia GPU, @msoose ?
If yes, it'd be helpful if you test the alternative installation steps as I described here.
For now, I'll revert the changes to :latest and keep :cuda untouched until the issue is resolved.

adrastee · 2020-06-22T20:15:21Z

Using cuda tag (from nvidia/cuda:10.2-devel-ubuntu18.04) : works awesome!
And weird thing : it has nvidia version 450 + CUDA 11.0 - not consistent with the name..

$ nvidia-smi
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 450.36.06    Driver Version: 450.36.06    CUDA Version: 11.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+=================

$ hashcat -I
hashcat (v6.0.0-4-g5628317d) starting...

clGetPlatformIDs(): CL_PLATFORM_NOT_FOUND_KHR

CUDA Info:
==========

CUDA.Version.: 11.0

Backend Device ID #1
  Name...........: GeForce GTX 1080 Ti
  Processor(s)...: 28
  Clock..........: 1632
  Memory.Total...: 11178 MB
  Memory.Free....: 11039 MB

$ hashcat -b -m 2500 -w 4
...
Hashmode: 2500 - WPA-EAPOL-PBKDF2 (Iterations: 4095)
Speed.#1.........:   651.1 kH/s (351.21ms) @ Accel:64 Loops:512 Thr:1024 Vec:1

As expected, CUDA 11.0 speed for this mode is +13% compared to opencl :)

@msoose : don't need OpenCL anymore, and the cuda tag works ok on Vastai as well.

ghost · 2020-06-22T20:33:59Z

@adrastee Great, thanks for the update, just confirmed :cuda tag does work correctly on Vast instances. Excellent.

dizcza · 2020-06-22T21:19:03Z

Could then anyone test :opencl-cuda tag? I put OpenCL + CUDA back again in a separate branch:

docker pull dizcza/docker-hashcat:opencl-cuda

What's the output of hashcat -I?

ghost · 2020-06-22T23:55:13Z

@dizcza :opencl-cuda also works.

hashcat -I with some snipping out of unwanted information:


CUDA Info:
==========

CUDA.Version.: 10.2

OpenCL Info:
============

OpenCL Platform ID #1
  Vendor..: NVIDIA Corporation
  Name....: NVIDIA CUDA
  Version.: OpenCL 1.2 CUDA 10.2.131
    Version........: OpenCL 1.2 CUDA
    OpenCL.Version.: OpenCL C 1.2
    Driver.Version.: 440.59

dizcza · 2020-06-24T16:20:15Z

All right, thank you. I reverted the changes so that latest = cuda + opencl again. I hope that the previous fail was a mere coincidence.

Talking about speed, you might squeeze an extra few percent out of your gpu capabilities if you run the computation not in a docker container. That very much depends on what you're doing on top of this image. If you're into dockering and use it primarily for wpa hashes, take a look at a repo of mine: https://github.com/dizcza/hashcat-wpa-server.

Feel free to post here cuda-related issues, if you encounter any.

adrastee · 2020-06-25T08:09:49Z

Thanks a lot!

adrastee · 2021-01-25T09:25:50Z

Hi again,
Using dizcza/docker-hashcat:cuda on latest 3090 GPUs, I got the error hashcat nvrtcCompileProgram(): NVRTC_ERROR_INVALID_OPTION

It works great with docker image nvidia/cuda:11.1-devel-ubuntu18.04
Maybe you can FROM this one?
Best.

dizcza · 2021-01-25T11:27:51Z

Thanks for the heads-up!
I've pushed CUDA 11.1 devel into the cuda branch and left 10.2 in the master.
I'll keep cuda branch as experimental and master with CUDA latest_major_version-1.
Feel welcome to post here next time the 11.1 doesn't work to bump up the version again, as I'm not using it anyway and cannot capture outdated software errors.

DenkertM · 2022-04-26T01:50:23Z

I am getting an error:

nvrtcCompileProgram(): NVRTC_ERROR_INVALID_OPTION

nvrtc: error: invalid value for --gpu-architecture (-arch)

* Device #1: Kernel /usr/local/share/hashcat/OpenCL/shared.cl build failed.

* Device #1: Kernel /usr/local/share/hashcat/OpenCL/shared.cl build failed
```.

This happened on many hosts that I have tried the container on.
I will try dizcza/docker-hashcat:nvidia-full next

dizcza · 2022-04-26T13:28:35Z

@DenkertM you probably have an outdated GPU card. With CUDA 11 release, Kepler architecture support has been dropped. Try running the latest branch, not cuda. The latest tag comes with CUDA 10.2 which supports sm_30 and sm_32 compute capabilities.
I've just tested on Nvidia GTX 760 which is a quite old card - sm_30 arch. The cuda tag fails to run a benchmark with the error you described. The latest tag runs flawlessly.

DenkertM · 2022-04-26T15:04:31Z

I am wondering if this was a driver issue? It was running on a remote RTX 3060 when the issue happened, and when I used :cuda on a 2x RTX 3080 remote host it was able to start the benchmark successfully. Tonight I can do a direct A/B, I didn’t have time last night. Is there a configuration you think I should look at?

dizcza · 2022-04-26T16:48:54Z

Ah, so it's the opposite: your card is one of the newest.
Let's settle it down.
It should be a CUDA incompatibility issue (not Nvidia driver).
nvidia-full tag is obsolete, outdated, and is about to be removed. It was built at the time when HashCat had no CUDA support whatsoever.
Which tag triggers the error you described - latest or cuda? Try running both. If cuda runs and latest fails, CUDA 11 must be used. If latest runs and cuda fails, CUDA 10 must be used.
If none of the tags works, you can change the base image, the CUDA version, at the first line of the Dockerfile. The valid cuda images can be found on nvidia hub docker.

lawndoc · 2022-11-09T03:55:50Z

@dizcza Would you consider building an arm64 version of the image? I'm trying to run this on an Nvidia Jetson Nano

dizcza · 2022-11-09T06:50:36Z

@lawndoc well yes, if you test it - I don't have any arm64 hardware to test.
Let me have a look.

lawndoc · 2022-11-11T01:53:06Z

I should be able to test it, but I might need a bit to figure out how to upgrade the cuda version on the jetson. I tried building locally and when I run I get an error that says it's not detecting 11.8 (though I did install it). I am using an older version of Jetpack that might not work nicely with newer cuda versions, so I'm going to start with reflashing Jetson OS and starting with the newest Jetpack.

lawndoc · 2022-11-11T02:18:02Z

Actually nevermind, I ended up getting hashcat working locally. Sorry, and thanks for the consideration!

adrastee changed the title ~~Asding CUDA SDK Toolkit ?~~ Adding CUDA SDK Toolkit ? Jun 18, 2020

dizcza closed this as completed Jun 24, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding CUDA SDK Toolkit ? #6

Adding CUDA SDK Toolkit ? #6

adrastee commented Jun 18, 2020

dizcza commented Jun 18, 2020

adrastee commented Jun 18, 2020

dizcza commented Jun 18, 2020

adrastee commented Jun 19, 2020 •

edited

Loading

dizcza commented Jun 19, 2020 •

edited

Loading

adrastee commented Jun 21, 2020 •

edited

Loading

adrastee commented Jun 21, 2020 •

edited

Loading

dizcza commented Jun 21, 2020

ghost commented Jun 21, 2020

dizcza commented Jun 22, 2020

adrastee commented Jun 22, 2020 •

edited

Loading

ghost commented Jun 22, 2020

dizcza commented Jun 22, 2020

ghost commented Jun 22, 2020

dizcza commented Jun 24, 2020

adrastee commented Jun 25, 2020

adrastee commented Jan 25, 2021

dizcza commented Jan 25, 2021

DenkertM commented Apr 26, 2022 •

edited

Loading

dizcza commented Apr 26, 2022

DenkertM commented Apr 26, 2022

dizcza commented Apr 26, 2022

lawndoc commented Nov 9, 2022

dizcza commented Nov 9, 2022

lawndoc commented Nov 11, 2022

lawndoc commented Nov 11, 2022

Adding CUDA SDK Toolkit ? #6

Adding CUDA SDK Toolkit ? #6

Comments

adrastee commented Jun 18, 2020

dizcza commented Jun 18, 2020

adrastee commented Jun 18, 2020

dizcza commented Jun 18, 2020

adrastee commented Jun 19, 2020 • edited Loading

dizcza commented Jun 19, 2020 • edited Loading

adrastee commented Jun 21, 2020 • edited Loading

adrastee commented Jun 21, 2020 • edited Loading

dizcza commented Jun 21, 2020

ghost commented Jun 21, 2020

dizcza commented Jun 22, 2020

adrastee commented Jun 22, 2020 • edited Loading

ghost commented Jun 22, 2020

dizcza commented Jun 22, 2020

ghost commented Jun 22, 2020

dizcza commented Jun 24, 2020

adrastee commented Jun 25, 2020

adrastee commented Jan 25, 2021

dizcza commented Jan 25, 2021

DenkertM commented Apr 26, 2022 • edited Loading

dizcza commented Apr 26, 2022

DenkertM commented Apr 26, 2022

dizcza commented Apr 26, 2022

lawndoc commented Nov 9, 2022

dizcza commented Nov 9, 2022

lawndoc commented Nov 11, 2022

lawndoc commented Nov 11, 2022

adrastee commented Jun 19, 2020 •

edited

Loading

dizcza commented Jun 19, 2020 •

edited

Loading

adrastee commented Jun 21, 2020 •

edited

Loading

adrastee commented Jun 21, 2020 •

edited

Loading

adrastee commented Jun 22, 2020 •

edited

Loading

DenkertM commented Apr 26, 2022 •

edited

Loading