[BUG] Keka doesn’t extract tarballs with XZ compression correctly.

[VaslD]

Configuration

• Keka version: 1.2.4 (4287)
• macOS version: 11.0.1

Describe the bug

Our company’s managed account had no access to Disk Utility and DMG functionalities so I wrote a bash (probably zsh syntax) script to compress each application in /Applications using tar command. So later I could selectively untar individual apps using Keka.

For most apps this worked flawlessly. But for Affinity Publisher, Keka and macOS Archive Utility exhibited entirely different behaviors.

(I’ve been doing it this way for years. Interestingly, I think my last issue with Keka was also when compressing or uncompressing Affinity apps.)

All archives were created with tar -c -J (which means create an archive with XZ compression). Their extensions were set to .txz.

I did 4 runs for science:

• Uncompress by double-clicking when Keka was default handler for TXZ files, on the computer used to create archive.
• Uncompress by dragging to open Keka window, on another computer.
• tar -x -J (which means uncompress an XZ archive) in Terminal.
• Explicitly open archive with Archive Utility.

On first 2 runs, Keka seemed to process an archive twice: First extracting a *.tar or a file with the same filename but no filename extension; then processing the file in the first step to extract a folder named PaxHeader with the original app, which macOS deemed cannot be opened because of permission issues.

On last 2 runs, both tar command and AU extracted the original app with no extra steps/files. The app can be opened and passed GateKeeper.

I did not do a hash check for individual files in the application bundle.

To Reproduce

Not sure this is 100% reproducible though.

1. Create a tarball with XZ compression for any application.

   tar -cJf ~/Desktop/Some\ App.txz -C /Applications Some.app/

2. Uncompress new archive using Keka.

3. Uncompress the same archive using Archive Utility.

4. Uncompress with tar:

   tar -xvJf ~/Desktop/Some\ App.txz

5. Compare results.

Expected behavior

Keka should recreate the original app from archive with correct permissions, and should swallow any "header" (without actually extracting them).

Screenshots

This is what happened to Affinity Publisher and Firefox. All tarballs were created equal.

Additional context

Keka log for extracting Firefox (correctly) and Affinity Publisher (incorrectly).

Keka_2020-12-02-200230755.log

[VaslD]

By the way, if I have a file named Firefox Beta (84.0).txz, Keka extracts it to Firefox Beta (84 folder.

This isn’t that much of a problem (but you can see it in the screenshot) so I'll just leave it FYI.

[aonez]

@VaslD what tar version are you using? Just tried to reproduce with the macOS bundled version and I was unable to reproduce this in Big Sur or Catalina. I'm seeing in the log (thanks so much for sharing it) that bsdtar extraction fails so p7zip is used producing an invalid app bundle. My guts say is not your tar version, probably the same bundled one as in my tests, but you Affinity bundle that has some metadata different that the one I've just downloaded from their web.

Since the bundle is so big, maybe you can just create a txz with a file of the bundle and share it with me, for example:

tar -cJf ~/Desktop/Test.txz -C /Applications Affinity Publisher.app/Contents/Frameworks/libxmp.dylib

What I've did:

• Copy Firefox and Affinity Publisher to Applications folder
• TXZ withtar -cJf <TXZ_FILE> -C <PATH> <APP_BUNDLE>
• Extract with Keka -> No problemo

aONe-Mini:~ aone$ which tar
/usr/bin/tar
aONe-Mini:~ aone$ tar --version
bsdtar 3.3.2 - libarchive 3.3.2 zlib/1.2.11 liblzma/5.0.5 bz2lib/1.0.6

Firefox Beta (84.0).txz, Keka extracts it to Firefox Beta (84 folder

Reproduced this issue, only when using "Always extract in a subfolder" or with the context menu item "Extract to Firefox (84.0)". Fixing this one right away!

I think my last issue with Keka was also when compressing or uncompressing Affinity apps

Indeed! 😂

[VaslD]

I have the exact bsdtar 3.3.2 - libarchive 3.3.2 zlib/1.2.11 liblzma/5.0.5 bz2lib/1.0.6 (from macOS 11.0.1) on both machines.

But I can no longer reproduce the issue with just a .dylib, or modified Affinity Publisher.app less than 100 MB. I haven’t tried re-tarring the 3 GB package. I’ll get back to you when I have more info.

[aonez]

@VaslD also for me it's no problem downloading a 3GB (or more) package, so in case is no problem for you just send me a link to info@keka.io.

[VaslD]

Since I can't reproduce this anymore I'll close it until next time.

[aonez]

Hopefully we can reproduce this in the future to fix it :)

Thanks a lot @VaslD!

[aonez]

@VaslD I've finally reproduced it (thanks to a mail from Jérôme from https://foxtrot-search.com) and fixed it in this build. The fix will come eventually in v1.2.7 so hopefully you'll never find this issue again.

It was also reported in #741, it's almost the same issue, and the feedback with the build was positive :)

[VaslD]

Awesome! May I know what went wrong (in some detail)? It's been bugging me constantly, especially that it doesn't happen every time.

[aonez]

Sure! It was about the locale setting being empty while running BSDTAR. With some special characters the encoding failed. Just setting it to UTF-8 fixed the issue.

[VaslD]

Sorry I have to reopen this. The latest GitHub version of Keka (1.2.12) on macOS 11.2.3 (20D91) still has the same issue.

XZ (.tar.xz) archives created by Keka cannot be unzipped (or rather, unxzzed 😃) correctly by Keka. Just test it compressing Keka.app with XZ Slow profile and all additional options turned off. Keka was granted Full Mac and External Volumes permissions.

Logs attached:

• Keka_2021-04-12-173917910.log
• Keka_2021-04-12-174203153.log

In the first session, Keka failed to extract its own application bundle from XZ archive. But in the second session, Keka succeeded in compressing and decompressing XZ of another open-source utility App Cleaner.

I also tried excluding resource forks, leaving the app running or killing it before compressing, using different output folders. None worked.

[aonez]

No problem for me on 10.15.7. Can you share with me that Keka.tar.xz file?

[VaslD]

This is an XZ of Keka.app compressed when Keka itself was running.

(It's not a zip but an .xz or .tar.xz. One of stupid GitHub quirks.)

Keka.zip

[aonez]

Probably won't fix it, but at least I should try. This build also applies the UTF-8 variable during compression. So let me know if compressing with that version fixes the issue:

Keka-1.2.13r4522.7z

I'm mostly sure it won't. I'll have to deeply debug this one again, then.

[VaslD]

Appreciated. I’ll run it on the machine that exhibit the issue and get back to you hopefully tomorrow. If you need any additional debugging info or to run additional tests, let me know. Best, Yi

…

On Apr 13, 2021, at 22:50, aONe ***@***.***> wrote:  Probably won't fix it, but at least I should try. This build also applies the UTF-8 variable during compression. So let me know if compressing with that version fixes the issue: Keka-1.2.13r4522.7z I'm mostly sure it won't. I'll have to deeply debug this one again, then. — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub, or unsubscribe.

[VaslD]

Also, would it be possible to then fallback to system/BSD `tar` if extraction result was unexpected? E.g. checksum failure, recursively extracting archive from archive, extraction produced none-sense or internal metadata like PAXHeader, to name a few. Since system `tar` is pretty much always available and stable, it would be a better/prioritized choice then falling back to 7-zip.

…

On Apr 13, 2021, at 22:54, Yi Ding ***@***.***> wrote:  Appreciated. I’ll run it on the machine that exhibit the issue and get back to you hopefully tomorrow. If you need any additional debugging info or to run additional tests, let me know. Best, Yi >> On Apr 13, 2021, at 22:50, aONe ***@***.***> wrote: >> >  > Probably won't fix it, but at least I should try. This build also applies the UTF-8 variable during compression. So let me know if compressing with that version fixes the issue: > > Keka-1.2.13r4522.7z > > I'm mostly sure it won't. I'll have to deeply debug this one again, then. > > — > You are receiving this because you modified the open/close state. > Reply to this email directly, view it on GitHub, or unsubscribe.

[aonez]

Same error with system one (running through Keka). The problem here is sandboxing...

[aonez]

Turns out this was indeed a sandbox issue, that package contains quarantine data and sandboxed bsdtar is unable to set that quarantine back using fsetxattr.

Since all sandboxed applications (like Keka) have a quarantine applied for any newly created files (macOS system applies it), that quarantine would be replaced anyway so I'm skipping that warning.

This build should extract the package properly: Keka-v1.2.13-r4540.7z

If you have 2 minutes try to apply a color flag to Keka.app, compress it (with Exclude Mac resource forks DISABLED) and try to extract it with this build to see if the color flag is applied (should be).

Thanks again for all the feedback @VaslD!

[VaslD]

Apologies for not checking the other build today as promised. Mac should flag my Mail.app to remind me to check my flagged emails.

So, the new build worked flawlessly. Tested using another 100 MB+ app bundle to be sure, both color attribute and permissions needed to execute the app were set as expected.

Thank you for keeping Keka live and awesome. Closing this as resolved.

[aonez]

Thank you @VaslD, users like you are 🔝