unrar 7 arm64: slow extraction of encrypted archives

[woiza]

Is there already an issue for your problem?

• I have checked older issues, open and closed

NZBGet Version

v23-testing

Platform

Linux/Docker

Environment

Rock 5B
arm64
docker testing image
unrar 7.0.0

Current Behavior

Less than 70 MB/s (I guess it does not utilize arm64 hardware crypto acceleration).

Expected Behavior

Over 250 MB/s

Steps To Reproduce

time unrar x -p

Logs

No response

Extra information

#78

[woiza]

I just built it on my rock 5b sbc without any build parameters:

make
c++  -march=native -O2 -std=c++11 -Wno-logical-op-parentheses -Wno-switch -Wno-dangling-else -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -DRAR_SMP -DUNRAR -c rar.cpp
c++  -march=native -O2 -std=c++11 -Wno-logical-op-parentheses -Wno-switch -Wno-dangling-else -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -DRAR_SMP -DUNRAR -c strlist.cpp

make --version
GNU Make 4.3
Built for aarch64-unknown-linux-gnu
Copyright (C) 1988-2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

I get over 230MB/s with this build. Is there anything I could try differently with your dockerfile and the compiler/build options there?

[phnzb]

@woiza
For compatibility we compile unrar with march=armv8-a for arm64 image. You right, for use crypto hardware acceleration, you must compile unrar with march=native (default) on device which supports it. You can test Dockerfile and docker-compose from this branch (https://github.com/nzbgetcom/nzbget/tree/feature/native-docker-unrar/docker), building custom image on yours device (I have added UNRAR_NATIVE build-arg).

[woiza]

@phnzb thanks for your rapid reply and the branch. I was able to build it locally and extract files with hardware acceleration.
I had to change the following in the compose file:

-        UNRAR_NATIVE: true
+        UNRAR_NATIVE: "true"

What's the idea here? Should users build nzbget locally or will you provide native builds? If so, how (another tag)?

[luckedea]

@woiza

Idea:

• Currently users have to build hw-optimized tools locally, we would support this flow, @phnzb maintains documentation and answers any questions about the matter.
• v25 would feature serious improvements on the issue where we would build extra functionality into nzbget that would detect hardware possibilities of the device it runs on and performs all possible optimizations for the user (un/archive process - which would probably mean fetching optimized native builds of various tools), with bottleneck detection (cpu, network, disk).

Having said all that - any feedback is very welcome, but we don't have plans to provide native builds separately.

[woiza]

@luckedea sounds good. I think mentioning it in the documentation and providing an example of how users can build optimized tools locally is enough.
I will build an image with an optimized version of unrar using your image as base image and provide it on ghcr and docker hub for anyone interested. Give me a couple of days, I have to do some reading on how to automate this task.

[woiza]

@phnzb @luckedea what happens if you use an image with an optimized version of unrar on an arm64 device without support for hardware crypto acceleration (e.g. raspberry pi)? Will unrar fail or will it fall back to software crypto? Unfortunately I do not own arm64 devices without hw crypto acceleration and cannot test this.

[phnzb]

@woiza will be something like that when trying to launch unrar :-(

UNRAR 7.00 freeware Copyright (c) 1993-2024 Alexander Roshal  
Illegal instruction (core dumped)

[woiza]

@phnzb thanks, what a pity. Any idea how 7zip is able to distinguish between software and hardware crypto? Or does apk get the correct version based on some checks? (just personal interest)

[phnzb]

@woiza Something like by compiling certain sections of code with optimizations, and using them dynamically in runtime if hardware crypto is available. Unrar, unfortunately, determine hardware acceleration at build time only.

[woiza]

@phnzb @luckedea might be interesting for future nzbget versions:
unrar 7 added support for arm64 crc32 instructions, namely __builtin_arm_crc32d and __builtin_arm_crc32b.
Peformance gain is noticable when adding or extracting files with -m0 switch.
With this switch extraction time of a 3.8 GB large encrypted archive reduced from 18 seconds (without -m0) to 16 seconds on my Rock 5B SBC.

[woiza]

@phnzb @luckedea
-march=native is too much, -march=armv8-a+crypto+crc is enough. I pushed an image to docker hub with an optimized unrar version using your branch and -march=armv8-a+crypto+crc:

woiza/nzbgetcom-develop-unrar7-armv8-a-crypto-crc