Library issues with Linux AppImage observed at runtime

[freakboy3742]

A large number of reports have been received (#458, #470, #481, #504, #583, #618, #625, #631, #692) relating to AppImage support on Linux. This is a consolidation ticket to cover the general problem of Linux AppImage support.

To Reproduce

Reports are most common when using third party binaries (e.g., Pillow), or GUI libraries that have third party binaries (e.g, PySide2/6). However, problems have also been observed with Toga apps when deploying on recent versions of Linux (e.g., using Ubuntu 22.04)

The problem manifests in one of two ways.

A failure to find dependencies during build time:

-- Deploying dependencies only for ELF files -- 
Deploying files in directory /app/Hello World/Hello World.AppDir/usr/app_packages/PySide2 
WARNING: Not an ELF file: /app/Hello World/Hello World.AppDir/usr/app_packages/PySide2/_config.py 
WARNING: Not an ELF file: /app/Hello World/Hello World.AppDir/usr/app_packages/PySide2/QtScxml.pyi 
WARNING: Not an ELF file: /app/Hello World/Hello World.AppDir/usr/app_packages/PySide2/QtCore.pyi 
Deploying dependencies for ELF file in AppDir: /app/Hello World/Hello World.AppDir/usr/app_packages/PySide2/QtXmlPatterns.abi3.so 
Deploying dependencies for ELF file /app/Hello World/Hello World.AppDir/usr/app_packages/PySide2/QtXmlPatterns.abi3.so 
ERROR: Could not find dependency: libshiboken2.abi3.so.5.13 

Missing dependencies at runtime:

>>> /home/criaz/repos/briefcase-613/hello/linux/hello-0.0.1-x86_64.AppImage
Traceback (most recent call last):
  File "/tmp/.mount_hello-9Pm4Um/usr/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/tmp/.mount_hello-9Pm4Um/usr/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/tmp/.mount_hello-9Pm4Um/usr/app/hello/__main__.py", line 4, in <module>
    main().main_loop()
  File "/tmp/.mount_hello-9Pm4Um/usr/app/hello/app.py", line 27, in main
    return hello()
  File "/tmp/.mount_hello-9Pm4Um/usr/app_packages/toga/app.py", line 272, in __init__
    self.factory = get_platform_factory(factory)
  File "/tmp/.mount_hello-9Pm4Um/usr/app_packages/toga/platform.py", line 41, in get_platform_factory
    from toga_gtk import factory
  File "/tmp/.mount_hello-9Pm4Um/usr/app_packages/toga_gtk/factory.py", line 1, in <module>
    from .app import App, DocumentApp, MainWindow
  File "/tmp/.mount_hello-9Pm4Um/usr/app_packages/toga_gtk/app.py", line 8, in <module>
    import gbulb
  File "/tmp/.mount_hello-9Pm4Um/usr/app_packages/gbulb/__init__.py", line 1, in <module>
    from .glib_events import *
  File "/tmp/.mount_hello-9Pm4Um/usr/app_packages/gbulb/glib_events.py", line 12, in <module>
    from gi.repository import GLib, Gio
  File "/tmp/.mount_hello-9Pm4Um/usr/app_packages/gi/__init__.py", line 40, in <module>
    from . import _gi
ImportError: /usr/lib/libgio-2.0.so.0: undefined symbol: g_module_open_full

The specific libraries involved vary between reports, but common themes include:

• references to GDBM
• References to glib or gio - these seem particularly common on c2022 Linux distress (e.g., Ubuntu 22.04)
• Apps that use Pyside2 or Pyside6

Expected behavior

It should be possible to build and run Linux AppImages on any version of Linux.

Environment:

• Operating System: Linux; most common on non-Ubuntu, or Linux distros released c2022
• Python version: Any

Additional context

There are likely a number of underlying problems that all have similar manifestations. It seems highly likely that we're not driving LinuxDeploy correctly. Given the number of options and plugins that exist for linuxdeploy, and the complexity of binary compatibility on linux, that's not entirely surprising. We need a better understanding of how LinuxDeploy should be working, and then test on a wider range of configurations and systems.

#717 is related to this; it's possible that some of the reported problems may be avoided by adopting a manylinux base.

[tusharhero]

is this the problem I am having?
I am not able to run briefcase dev

[freakboy3742]

This is unrelated to briefcase dev. AppImages are only built on briefcase run.

[nickzoic]

Yeah, I'm getting a possibly related issue when try to do a minimalist build:

• on Ubuntu 20.04 with Python 3.8.10
• install system deps per Tutorial 0:
  sudo apt-get install git python3-dev python3-venv python3-gi python3-gi-cairo libgirepository1.0-dev libcairo2-dev libpango1.0-dev libwebkit2gtk-4.0-37 gir1.2-webkit2-4.0
• create "direnv" virtual env, install briefcase 0.3.6 and toga 0.3.0.dev33 from pypi
• run briefcase new with all default answers
• cd helloworld and run briefcase dev (it does exactly what you'd expect
• Skip "tutorial 2" entirely and stick with the default ... I don't think I'm missing any important steps in here? Going on to "tutorial 3":
• briefcase create seems to work fine with no complaints
• briefcase build runs ... there's a lot of output and some warnings, but none of them look very ominous
• briefcase run fails ... there's a warning Failed to load shared library 'libgtk-3.so.0' referenced by the typelib: /lib/x86_64-linux-gnu/libgtk-3.so.0: undefined symbol: g_mount_operation_set_is_tcrypt_hidden_volume and then a Python stacktrace ending with TypeError: must be an interface and then a message Unable to start app helloworld.
• I notice that libgtk-3.so.0 is not one of the libraries copied into linux/appimage/Hello World/Hello World.AppDir/usr/lib/ although it does exist in /usr/lib/x86_64-linux-gnu and other libs are copied from that dir
• same behaviour with briefcase at 9b3220d

I haven't dug into how briefcase even works yet :-)

[freakboy3742]

@nickzoic That sounds consistent with at least one of the class of errors we've been seeing on Linux; although I don't think it's been reported on Ubuntu 20 before. libgtk errors are usually tied to "recent" linux distros (Ubuntu 22.04, et al).

[nickzoic]

Yep, I've been picking away at it & trying to work out exactly where things are going wrong ... maybe somewhere in linuxdeploy it's not finding the right libraries, or the right versions?

Was there a known set of versions where it did always work on linux?

[freakboy3742]

So - it still works for me on my Ubuntu 20.04 box - but I set up that machine (a VM running in Parallels on macOS) some time ago (almost a year ago). It was originally a 20.04.2 install, but I've got all updates applied. Here's a zip file with the contents of apt list; and here is a link to download an AppImage that works on my machine.

[danyeaw]

There is a pretty recent linuxdeploy plugin for GTK which is doing a lot of the hard work of bundling the dependencies, maybe we could make use of it? https://github.com/linuxdeploy/linuxdeploy-plugin-gtk

[freakboy3742]

Certainly possible. If this is the fix, we may need to add a configuration item for loading linuxdeploy plugins (Toga needs the GTK plugin, but Qt-based apps won't, for example - presumably there are plugins for Qt-based apps).

If there's a way to reproduce the behavior without a plugin, that would be even better; however, I have no idea if this is even possible - we'd need a much better understanding of what plugins are doing in practice.

[nickzoic]

Yep, I got about as far as staring into the linuxdeploy source code in the hope that it'd look back into me :-)
If there's already a GTK plugin that's handy,

More generally, though, how is the deploy process meant to know which .so libraries it'll need?
Does toga (or whichever gui framework you choose) tell briefcase what plugins / libs it's using somehow?
How does this work for third-party libraries?

[freakboy3742]

So - the honest answer is that I'm not 100% confident I know for sure.

As I understand it, linuxdeploy looks at the binary that is being deployed, extracts the list of dynamic libraries that are linked by that binary, then does the same on those libraries, and so on until it has a complete list of libraries needed for execution; it then copies and rewrites them so that they are all present in the AppImage. The only library it doesn't do this for is libc, which is assumed to be present for bootstrapping purposes.

This can fall over if the libraries are loaded dynamically - e.g., in the case of Python loading third party binary libraries. To accounts for this, you explicitly pass in other paths or libraries that you explicitly want to be processed or included. We do a lot of this to ensure that dynamic libraries are loaded.

I haven't looked into the plugin system in depth, but as I understand it, the plugins exist to automate some of these inclusions, and to add into other libraries that might need to be manually excluded. I know there is a GTK plugin - but it hasn't been needed to date to get the AppImage working.

I will also add that I'm not 100% wedded to AppImage. It exists, and I was able to get it to work; but my impression is that flatpak is a lot more "industry standard" when it comes to cross-platform Linux binary distribution. If the "fix" here is "add a Flatpak backend and make it the default", I'd be completely on board with that.

[nickzoic]

Update: I've added in --plugin=gtk to the linuxdeploy invocation, plonked that plugin into ~/.briefcase/tools/ and added a couple of packages to the example's pyproject.toml's system_requires and ... IT WORKS! I can now briefcase build and briefcase run the example ...

see https://github.com/nickzoic/beeware-briefcase/tree/718-library-issues-gtk but I still need to:

• have briefcase download the plugin and save it
• modify the briefcase-template to add the system_requires packages

[freakboy3742]

I'm interested that you needed to modify packages in pyproject.toml and add the GTK plugin - unless you're explicitly testing something "external", my understanding was that the system packages list should be complete. What pyproject.toml modifications did you need to make?

[nickzoic]

So running with --plugin=gtk in briefcase but a standard pyproject.toml, the briefcase build process stops with

[gtk/stdout] Installing GTK 3.0 modules
[gtk/stderr] Package gtk+-3.0 was not found in the pkg-config search path.
[gtk/stderr] Perhaps you should add the directory containing `gtk+-3.0.pc'
[gtk/stderr] to the PKG_CONFIG_PATH environment variable
[gtk/stderr] No package 'gtk+-3.0' found
[gtk/stderr] /home/brutus/.briefcase/tools/linuxdeploy-plugin-gtk.sh: there is no 'exec_prefix' variable for 'gtk+-3.0' library.
[gtk/stderr] Please check the 'gtk+-3.0.pc' file is present in $PKG_CONFIG_PATH (you may need to install the appropriate -dev/-devel package).
ERROR: Failed to run plugin: gtk (exit code: 1) 

Error while building app helloworld.

Adding libgtk-3-dev to the pyproject.toml then running briefcase create and briefcase build then fails with the same error but for librsvg-2.0.pc.

Adding librsvg2-dev then create and build and bingo it works.

pyproject.toml.txt <-- the pyproject.toml file but renamed so I can upload it

I'm pretty much just cargo-culting at this point though so I might well be missing the point :-)

[freakboy3742]

Ok - so... I'm interested what the reverse behavior is. If you add those two dependencies in to pyproject.toml, and don't have the plugin available... what happens?

[nickzoic]

Just tried it out, so unmodified briefcase 0.3.7 and toga 0.2.15 from pypi, run briefcase new then edit pyproject.toml and run briefcase create; briefcase build; briefcase run ... build works, but briefcase run fails with the same TypeError I mentioned in my first comment in this issue.

(just tried it just in case: same with toga at cf31b7c)

[freakboy3742]

It doesn't surprise me that Toga 0.2.15 has some packaging problems; it used a different mechanism to find the GTK libraries. But if the same problem is persisting with a recent Toga trunk, then it looks like the plugin may be called for.

It would be good to know what the plugin is actually doing, though...

[nickzoic]

Weirdly, the plugin turns out to be a bash script.
https://github.com/linuxdeploy/linuxdeploy/wiki/Plugin-system
So it's both possible and painful to work out what the hell it is doing :-)
Mostly just gathering up the extra libraries I think.

[nickzoic]

@freakboy3742 > If the "fix" here is "add a Flatpak backend and make it the default", I'd be completely on board with that.

Just for fun, I've made a start on a flatpak backend at #754 ... it's a very sketchy proof of concept but it does let you run the toga "hello, world" as a flatpak. I'll open a WiP PR for it and add some notes on what needs to be done still.

[freakboy3742]

I've just had a minor breakthrough on one of these problems - the one that manifests as:

libXXXX-<hash>.so.5.0.0: ELF load command address/offset not properly aligned

Pillow is a common example where this appears - in that instance, it's usually libtiff that raises the error.

See #759 for an explanation and proposed fix.

[freakboy3742]

As noted on #717 and beeware/Python-Linux-support#2, it appears that manylinux isn't going to be a fix for these problems. A combination of #756 and #759 looks like it will address the bulk of the issues we're seeing; with #596 being the source of a more comprehensive fix.

[freakboy3742]

#759 looks like it won't work as a generic solution; however, there's a documentation only solution in #761.

I've also reported the issue upstream: linuxdeploy/linuxdeploy#204

[freakboy3742]

After a lot of investigation, it appears that #756 addresses the bulk of the GTK-related issues running on "recent" Linux releases; #761 addresses the issues related to binary modules. On that basis, I think this ticket as a "locus of Linux packaging issues" has reached the end of its useful life. If any new issues arise with Linux packaging, we can investigate further based on those specific reports.