Add LZIP compression

[TimmThaler]

It would be great, if you could add LZIP to support .lz compression.

http://www.nongnu.org/lzip/lzip.html

[aonez]

Thought the provided binary was enough to implement support, but it lacks some features needed, so a code implementation or some code changes will be needed:

• Output file destination in compression and extraction
• Method selection
• Split
• Join multivolumes

Optional:

• Exclude files (resource forks)
• Progress percentage

Here a test version. Just for test purposes: Keka-1.1.0-beta.3r2151-lzip.zip

[TimmThaler]

This looks great!
I did a bit of reading. Perhaps that helps for coding.

• The Command Line Version of lzip provides an option for the Output file destination:

  • -o, --output=<file> : if reading standard input, write to

• The Progress Percentage is shown in Terminal, when you use the option -v twice.

  • -v -v will give you the percents and the MB of the already read data.

• The Compression has 10 stages via the options -0 (no compression) to -9 (best compression).

• Keeping a file after compression will be set with the option -k. Otherwise the input file will be deleted after (de)compressing.

• Excluding files is not possible (as far as I read the man page).

Unfortunately, the command line version of lzip does not accept folders as an input. I tested a compression with your test version from above and I got an error message, that the archive could not be created.
Because lzip is just a compression tool for single files, folders should be packed into a tar archive which then can be compressed without problems.

I hope, I could help :)

• Help File of the command line lzip: lzip.txt

[aonez]

Thanks a lot for your research! In fact yesterday I got in to it and developed a patch so the binary accepts output folders and output filenames 😁
It's in early stages, so just as reference: output-write.patch.zip

Did not noticed the -v flag (x2) shows percentage. This will do. 👍

Because lzip is just a compression tool for single files, folders should be packed into a tar archive which then can be compressed without problems.

Noticed that. I'm currently using bsdtar for gzip and bzip, with lzip I think I'll need to create a two phase (tar first, lzip then).

Right now I'm battling with the multivolume decompression, I don't understand why it decompresses each part separately.

[aonez]

By the way, you use Lzip for backup purposes?

[TimmThaler]

Really great!
I think that should work.

Yes, I use Lzip for backups of my nextcloud server to backup all the data and settings. First I put everything into one tar archive and then I compress with Lzip which saves 20% to 30% space. Actually, when I compress only the mysql backup file the result is up to 85% smaller than the input :-)

[aonez]

GNU tar has built in support for Lzip, gonna try this next, so there'll be no need to make a two step compression and extraction of tarball.

[aonez]

Trying plzip but getting performance issues, seems to saturate all the CPU threads...

[TimmThaler]

Yes, I already thought something like this would happen. This compression ist quite a "CPU owner" - demands very much processor power.
When I use lzip in the macOS Terminal, it takes up to 95% of the CPU usage ... But in the end: The files are much smaller. 😄

[TimmThaler]

When I install plzip via homebrew I got the problem, that I am not shown the percentual progress, just 0% and 100%, but nothing in between...

[aonez]

No performance issues?

[aonez]

...0% and 100%...

In fact now I remember I noticed this too, and this shows the CPU saturation.

Here you have more test builds:

• Using lzip: Keka-1.1.0-beta.3r2187-lzip.zip
• Using plzip: Keka-1.1.0-beta.3r2187-plzip.zip

The plzip binary does not have the split ability integrated, and I'm yet unable to join an lzip multivolume, so I may just remove this feature altogether.

[TimmThaler]

• With the LZIP version I got about 50% to 60% CPU usage and with the PLZIP version over 90%. That there will be some performance issues for other applications like Firefox is (for me) ok 😆 😄
• I don't think, LZIP has multivolume support. The help file doesn't contain any information about that.

[aonez]

I'm gonna use lzip for now (without split), I can't even get plzip paused on time.

[aonez]

Will add an option to use plzip instead:

defaults write com.aone.Keka UseMultithreadLzip <true|false>

[TimmThaler]

Great. This way it will not freeze the system or take too much CPU performance and users still can decide if they want this 😄 👍

[aonez]

Anyway I'll try to work on this, since plzip is much more faster for big files. Thanks to this thread I'm now using pigz and lbzip2 on the next Keka, for multithreaded gzip and bzip2 tarballs 😁