Replicate the macOS build for Linux? #1
Comments
|
Hello, The Dockerfile already disabled the daemon. This wasn't previously supported on Linux, but I fixed the As for KF5, this is due to the fact it's not the same application. The Ring macOS version is a native Cocoa frontend on top of LibRing and LibRingClient while Ring-KDE is a native frontend on top of LibRing and a forked LibRingClient (I am the original author, but don't own the copyright, so my version is technically a fork). As for linuxdeployqt, I first tried it, but discarded that image as naive and unfixeable. The Ring daemon doesn't work perfectly with a mix of newer compiled packages and the one provided with 14.04. Upstream does not support it and their own package bundle all dependencies (with supported versions). Also, after after discussing with other KF5 devs, I decided to explore using static libraries instead of copy/pasting all As for the progress. I am 99% done. This repository Dockerfiles creates an appimage. I am currently testing on various liveCd and chrooted distribution to see how it handles. I should be done by tomorrow. |
Looks like a Qt app to me, based on the libraries found inside the bundle. But hey, as you already have 99% of the solution, that may be well irrelevant now ;-) May I ask why linuxdeployqt was not able to fulfill your needs here? |
|
Long answer:
May I ask why linuxdeployqt was not able to fulfill your needs here?
I wish to reiterate that the reason I chose Qt static mode is not because
of problems with linuxdeployqt. Both solutions can work and eventually
achieve the same goal. Neither is much more complex than the other either.
In a view of the situation, that I share with some other KF5 developers, I
think static packages are a better long term alternative. I know this view
isn't universal and only a fraction of appimages chose it. So let me detail
why I think this is a superior solution for me (and potentially other KF5
apps). Each argument have a different weight in the decision making
process. Some one them (performance) are more "meh", but some other really
reflect why this is (in my view) superior.
First of all, this isn't out of the blue. While static linking predates
modern shared libraries, it wasn't used much until recently. Since a couple
years, the Docker ecosystem started shifting from an image defaulting to
Ubuntu to an Alpine Linux one. One of the big difference is that Alpine is
much more friendly to static linking. For a container deployment
perspective, it is a much better fit as I will describe below. The AppImage
and the other self contained (or sandboxed) alternative targets such as
Snap/FlatPak/Android share most of the same attributes as container (from a
self contained portable computer program point of view). Going in that
direction seems to follow a (perceived) industry wide shift toward static
linking.
The first advantage is to reduce the attack surface of the program. Ring is
a process written in C and C++ that happens to be network facing. Using a
static binary without a shell reduces the risk of some attacks such as ROP
and remote shells. One of the advantage of having a client/server topology
(which is the default on Linux) is that the ring-daemon (network logic) can
be sandboxed without impacting the client (GUI and asset management). I
have little choice here. I can't use the daemon because it wont work if
Ring is already installed outside of the appimage (resulting in a crash if
the system daemon is running or potential data loss if the appimage daemon
is used with the Gnome client). So if I am to have a single process, better
have a single binary and get rid of all the unused symbols and code so they
can't be used in a ROP exploit. If appimage ever support sandboxing, it
would also minimize the number of files, making it much harder to upload a
remote exploit shell (as they could not link the shell to existing
libraries, it would have to be static itself. It makes it harder to upload
and hide). On the other hand, some libraries such as OpenSSL/GnuTLS should
*not* be linked statically since it makes it virtually impossible for the
users to apply a security fix until someone give her/him a new appimage. I
am a little hypocrite here because LibRing does just that due to
dependencies on newer modules not part of 14.04 version of GnuTLS...
However this is something I will eventually work on.
From a size point of view, I think it will eventually result in a smaller
image. Until all libraries are static, it does the opposite as all shared
object copy/paste the static libs. However once "complete", the aggressive
dead code elimination compiler pass[4][5] can be used and reduce the size
by a large fraction (maybe ~70%, I don't have the metrics yet). This remain
to be proven as it will take a bunch of iterations and kde-core-devel
discussions before I can upstream the required fixes in KF5. However this
strategy is commonplace for containers, so it should work fine. Some other
assets such as icons can also be added to "QRC" bundle. This is a Qt
mechanism to embed files into libraries/executables. Most windows
applications, at least back in the 9x used to work that way and have assets
dll or embedded them into .exe. This is useful for appimages are its a
cheap way to reduce memory heap allocation and avoids bundling the whole
Breeze icon set in case the system doesn't provide the icon.
From a portability PoV, this is also interesting. The main issue SGClark
(months ago) and me (last week) had is that (Lib)Ring (the logic code of
ring-daemon) requires a lot of dependencies. The macOS file list is a bit
short because it links to static libraries (but for some reasons, not all
of them...). The LibRing code has 171 preprocessor `#if` for optional
dependencies and old (deprecated) external library APIs. Compiling a mix of
old and new libraries such as what the default appimage "way of doing
things" do trigger untested and unsupported (by upstream) code paths. This
then caused bugs that could not be fixed without only using self compiled
libs (with well tested version combination, such as what the macOS image
uses). So at that point, I was already avoiding most of the system
libraries, so going static is easier than it would otherwise have been.
Note that LibRing autotools based build system somehow choses some shared
libraries even when --enable-static is used. That's a bug I will fix soon.
For now, just as the macOS image, it has a mix of static and shared libs.
Also, I tried the appimage on both a raw debootstrap/netinstall + xcb
debian Stretch and on a Gentoo system and it complained about missing
libraries. Assuming Arch will have the same problem (I didn't tested it,
yet). It seems that there is very little libraries you can really assume
are always present (at least in the expected places). Going the static way
get rid of most of these issues and increases portability. It is also more
backward compatible and future proof since virtually nothing changes. It
ensures the appimage will keep working virtually forever.
The next point I mentioned above is performance. While this isn't the
highest priority given it's already good enough, it's an interesting side
effect of shipping my own libraries (static or shared). Some more
"professional" software such as databases or web browsers use a technique
called Feedback Driven Optimization (FDO)[2] or Profile Guide Optimization
(PGO)[1]. This allows to use the runtime information from automated
integration tests to reorganize and optimize the compiled executable.
Results varies, but it can be up to 30% improvement in startup time and
~10% in general execution. I am not attacking this right now, but I find
interesting that using static libraries make this possible/easier. Both
Firefox and Google-Chrome precompiled Linux downloadable version use this.
It's why they feel faster than when you compile them yourself (ok, this
could also be confirmation bias). Other optimizations such as inlining and
vectorialization are also more efficient in static mode. However, as I
said, this is a nice side effect, but isn't a good argument to chose static
over system/shared libraries.
Finally, but, in my humble opinion, the most crucial argument is longer
term maintainability cost. While copy/pasting .so works fine and can be
semi automated like with the linuxdeployqt script, it's still in my view
fixing the problem from the wrong direction. It is in my opinion, which I
am aware some disagree with, that the copy/pasting isn't really the proper
place the "know" what is required to run the application. In fact, it's
pretty much a curated process using scripts but requiring a human. In the
whole tech stack, only 2 places really know the required files. The first
one is the distribution package (.deb), but this is again curated and
require human assistance when something changes. The other one is CMake.
CMake has all the information it needs to build the appimage. CPack is a
package generator based on this valuable data. I know you are aware of it
since it's listed in AppImage/AppImageKit#160.
CMake has literally all the information. No need to recreate any of it from
inspecting the final system. If KD5 "extra-cmake-module" package could be
extended to support the AppImage toolchain, it would not only know which
Qt5/KF5 dependencies are required, but the exact optional feature for each
of them. It could, in theory, build the smallest possible version of Qt5
and the smallest version of the framework pack. It also has the information
about the `.desktop` and icon files. So, from my point of view,
implementing static Qt5/KF5 support directly as a cmake/cpack module makes
much more sense than copy/pasting `.so`. **All changes to the package will
be known as soon as they are committed**. The time required to keep track
of the ring-kde, or any of its dependencies, is reduced to virtually zero
as long as they properly support CMake. It also scale much better. If
tomorrow you want to package another KDE app, chances are replacing the
name in the root CMakeLists.txt is the only change required[3]. Note that
my work (in this repository) is currently a bit Frankenstein-ish. I wanted
to use CMake build in way to fetch and embed dependencies, but only a
couple frameworks has the required macros. For now it just install them in
private paths and append their config.cmake files manually. It doesn't
require much work to fix it for all KF5 libs (less than a day for all 70
frameworks, regression tests included), but I want to get the first
patchset merged before I attempt it. However it doesn't affect the end
result. CMake has a correct auto generated database of all required files.
Edit: Just to note that I am not an AppImage expert and your experience with it is greater than mine with this tech. I took the decision above based on previous experience and research that I think is relevant. However I can understand that you successfully used your script many time and may not see the point of this whole message.
[1] https://developer.mozilla.org/en-US/docs/Mozilla/Developer_guide/Build_Instructions/Building_with_Profile-Guided_Optimization
[2] https://github.com/google/autofdo
[3] By root CMakeLists, I mean adding a CMake file in front of the app and
include it using `add_subdirectory(myapp)`. This allows to setup all
CMake/CPack variables without having a mile long command line or modifying
the actual application CMakeLists.
[4] https://gcc.gnu.org/onlinedocs/gccint/Tree-SSA-passes.html#Tree-SSA-passes
[5] http://elinux.org/images/2/2d/ELC2010-gc-sections_Denys_Vlasenko.pdf
|
|
Thanks for the detailed explanation, I learned a lot from this. Looks like you know a lot more about what should go into an AppImage than me (and how to best produce that payload) and I'd be thrilled to see proper CMake AppImage integration with all the good things you mentioned. Would you consider upstreaming the essence of it, eventually? |
Well, this project, as you know, is sponsored by BlueSystems and the objective is to create a Ring-KDE appimage. Writing an ultra generic and perfect CPack backend is out of scope. That being said:
That leaves the final piece, the [1] Many of those exists for totally silly reasons, like one function being capable of sending an optional email and pulling 5 libraries (kcrash). Qt5::Test is also pulled by all frameworks for nothing. Mandatory test is toxic since it brakes the ability to cross compile the appimage for ARM and Plasma-mobile. |
In the Qt-based macOS Ring.app there seem to be no traces of KF5 nor a daemon...
Can we replicate the macOS build as closely as possible for Linux? It seems to have reasonable dependencies.
If we can build it in a similar way for Linux, linuxdeployqt should have no problem bundling it as an AppImage.
The text was updated successfully, but these errors were encountered: