Debian packaging

[directhex]

This change set turns the oss-xamarin.android bundles into valid Debian source packages (albeit they need to be renamed to a Debian-valid name before dpkg-buildpackage works)

It also fixes the accidental re-inclusion of the bundle-*zip archives, so the overall file size of the bundle goes down by 200 meg.

It does not, at this time, produce the resultant .debs

[dnfclas]

Hi @directhex, I'm your friendly neighborhood .NET Foundation Pull Request Bot (You can call me DNFBOT). Thanks for your contribution!
You've already signed the contribution license agreement. Thanks!

The agreement was validated by .NET Foundation and real humans are currently evaluating your PR.

TTYL, DNFBOT;

[jonpryor]

Regarding future maintenance, would it instead be possible to make this a symlink to the top-level LICENSE file, or -- failing that -- use some Makefile voodoo so that LICENSE is copied to build-tools/debian-metadata/copyright?

I'm not looking forward to needing to maintain two separate copies of license information.

[directhex]

Good point. It started as a copy because I started off working from a bundle zip download, and didn't think to decouple it again when re-integrating with the build system.

[jonpryor]

Is this wise, placing un-prefixed binaries into /usr/bin? The files we generate are:

• cross-arm
• cross-arm64
• cross-x86
• cross-x86_64
• generator
• llc
• opt
• xabuild

@atsushieno already wants to rename generator on Linux so something less generic, and everything except xabuild just seems like it would conflict with something. For example, the llvm-3.5 Debian package installs /usr/bin/opt-3.5 and /usr/lib/llvm-3.5/bin/opt.

If nothing else, installing a /usr/bin/opt will confuse things.

I think we originally "envisioned" the Linux install as being in a subtree, e.g.:

• /opt/lib/mandroid/bin/llc
• /opt/lib/mandroid/bin/cross-arm

Having /usr/bin/xabuild seems like a good idea -- though that would likely need to differ from what we currently emit -- but otherwise, I'm not at all certain about this direction, and "polluting" $PATH with internal tools.

[directhex]

Do you want me to try and make those changes, or are they unrelated from this PR?

[jonpryor]

Do you want me to try and make those changes, or are they unrelated from this PR?

A bit of both?

The questions are twofold:

1. Which tools should be placed into /usr/bin?
2. What build system/organizational changes are required to support (1)?

This PR is a good "sounding board" to discuss those questions.

I believe that the only tool that should be in /usr/bin is xabuild.

I believe that the other tools need to be "available", but (1) don't need to be in $PATH, and (2) shouldn't be in $PATH.

That said, I don't fully understand the FlatPak and Debian directory structures, so I'm not sure where "non-public programs" should go, and what build system changes may be required to support that.

[directhex]

I don't disagree with xabuild being the only thing to go into $PATH (i.e. into /usr/bin)

The directory structure questions:

FlatPak

/app is the only writable prefix for apps, and you put whatever you like in here. When you run an app, PATH is set to /app/bin:/usr/bin (/usr is provided by the runtime on which your app is built, in our case the minimal FreeDesktop runtime). In most cases, this just means stuff being built with --prefix=/app (e.g. /app/bin/mono-sgen). We could make xabuild the only thing in /app/bin, and XA would be installed anywhere (as long as it's a subdirectory in /app)

Debian

The norm would be /usr/lib/packagename for platform-specific internal files, /usr/share/packagename for platform-independant internal files (which are really what we're talking about). If we're not shipping host-Linux (because it's only useful in the proprietary XS extension), then I don't think any of the files I noticed in the oss-xamarin.android bundle were platform specific - lots of cross-built Monos and lots of assemblies, but no binaries that would need to be rebuilt on amd64, i386, etc.

That said, I don't see a path to getting XA into distribution repositories (due to the hard dependency on an Internet connection during the build), and we can kinda do anything we like on mono-project.com packages which aren't beholden to packaging structure policy (i.e. /opt/foo is fine for non-distro packages)

[jonpryor]

If we're not shipping host-Linux... then I don't think any of the files I noticed in the oss-xamarin.android bundle were platform specific

We're not shipping host-Linux yet, but we will (eventually; C10 timeframe?), so this needs consideration. cross-arm/llc/etc., on the other hand, are platform-dependent files, and thus shouldn't go into $prefix/share.

Or I'm not sure what you mean by "binaries that would need to be rebuilt on amd64, i386, etc."

I don't see a path to getting XA into distribution repositories (due to the hard dependency on an Internet connection during the build)

This should be solvable. Just not yet.

The norm would be /usr/lib/packagename

What would be "packagename"? oss-xamarin.android?

I think what we should consider doing is "re-thinking" which files are "bin" files, such that all but xabuild are placed into bin/$(Configuration)/lib/mandroid, e.g. bin/Debug/lib/mandroid/cross-arm, and updating our build system accordingly. This would "skip" $prefix/share as a construct, but that also means that our install structure is closer-in-spirit to what Windows does as well...

[directhex]

cross-arm/llc/etc., on the other hand, are platform-dependent files, and thus shouldn't go into $prefix/share.

Right now there's no cross-* stuff going into the oss-xamarin.android bundle at all. The only stuff in bin/ is xabuild and generator. The only ELF of any description is in /bin/{Release,Debug}/lib/xbuild/Xamarin/Android/lib/*

What would be "packagename"? oss-xamarin.android?

Anything. Doesn't matter. You want oss-xamarin.android, you can have it.

I think what we should consider doing is "re-thinking" which files are "bin" files, such that all but xabuild are placed into bin/$(Configuration)/lib/mandroid, e.g. bin/Debug/lib/mandroid/cross-arm, and updating our build system accordingly. This would "skip" $prefix/share as a construct, but that also means that our install structure is closer-in-spirit to what Windows does as well...

I am okay with this

[jonpryor]

I don't think that this file is required at this time.

[directhex]

Okay, that's easily removed.

[jonpryor]

The downside to this change is that the Azure uploader (currently) only looks for .zip files.

This could be changed, but as-is, this would "break" the Azure upload step, in that files would no longer be uploaded.

Aside: what kind of space savings does this provide?

Additionally, a nice feature of using .zip is that it's easier for those poor Windows users to extract the file without using 3rd party utilities.

[directhex]

I already changed the Jenkins job config to look for any extension, so it won't break the job.

The real space saving ought to be fairly minor (around 15 meg iirc), but because the --exclude logic isn't working properly with zip, it ends up being closer to 250 meg by not including both the bundle-* zips and their contents.

The main reason for this change is that zip is not considered a valid archive type by dpkg-buildpackage, so users will need to recompress to .tar.foo before being able to build packages

[jonpryor]

Then perhaps we need to special-case things so macOS builds .zip and Linux builds .tar.bz2. The macOS artifacts are documented as the way for Windows users to test specific builds, and afaik Windows can't extract .tar.bz2 artifacts. Windows can extract .zip.

[directhex]

You're right, Windows has no built-in handler for .tar.bz2, it would require a third-party compression app like 7-zip.

[jonpryor]

This step is also executed on macOS, and this command would break macOS builds (because macOS doesn't have the dch command).

This should probably follow the spirit of the make prepare target: make it a multi-target, and have OS-specific "overloads" of the target:

package-oss $(ZIP_OUTPUT)::
    if [ -d bin/Debug/bin ]   ; then cp tools/scripts/xabuild bin/Debug/bin   ; fi
    if [ -d bin/Release/bin ] ; then cp tools/scripts/xabuild bin/Release/bin ; fi
    if [ ! -d $(ZIP_OUTPUT_BASENAME) ] ; then mkdir $(ZIP_OUTPUT_BASENAME) ; fi
    if [ ! -L $(ZIP_OUTPUT_BASENAME)/bin ] ; then ln -s ../bin $(ZIP_OUTPUT_BASENAME) ; fi
    zip -r "$(ZIP_OUTPUT)" $(ZIP_OUTPUT_BASENAME) $(_BUNDLE_ZIPS:%=--exclude %)

ifeq ($(OS),Linux)
$(ZIP_OUTPUT)::
    cp -a build-tools/debian-metadata $(ZIP_OUTPUT_BASENAME)/debian
    cd $(ZIP_OUTPUT_BASENAME) && DEBEMAIL="Xamarin Public Jenkins (auto-signing) <releng@xamarin.com>" dch --create -v $(PRODUCT_VERSION).$(-num-commits-since-version-change) --package oss-xamarin.android --force-distribution --distribution alpha "New release - please see git log for $(GIT_COMMIT)"
endif # Linux

This would prevent the macOS build from failing, while allowing Linux to also build Debian packages. (It would thus promptly fail when someone tried to build on Centos...)

[directhex]

I didn't realise this step was being used on Macs, you're right about shoving some stuff in a conditional.

[monojenkins]

Hello! I'm the build bot for the Mono project.

I need approval from a Mono team member to build this pull request. A team member should reply with "approve" to approve a build of this pull request, "whitelist" to whitelist this and all future pull requests from this contributor, or "build" to explicitly request a build, even if one has already been done.

Contributors can ignore this message.

[monojenkins]

Hello! I'm the build bot for the Mono project.

I need approval from a Mono team member to build this pull request. A team member should reply with "approve" to approve a build of this pull request, "whitelist" to whitelist this and all future pull requests from this contributor, or "build" to explicitly request a build, even if one has already been done.

Contributors can ignore this message.

[jonpryor]

PR #704 / commit 97f08f7 implements the above suggestion: instead of having:

• $prefix/bin: xabuild, the cross compilers (cross-arm, etc.), LLVM support (opt, etc.), shell scripts for utilities (generator, etc.)
• $prefix/lib/mandroid: Some of the utilities which are used by the shell scripts in $prefix/bin, e.g. generator.exe.
• $prefix/lib/xbuild: MSBuild integration
• $prefix/lib/xbuild-frameworks: Xamarin.Android profile assemblies

we now have a more Windows-style, self-contained structure of:

• $prefix/bin: contains only xabuild; generally ignorable.
• $prefix/lib/xamarin.android: Contains everything else:
  • $prefix/lib/xamarin.android/xbuild/Xamarin/Android: MSBuild integration + former contents of $prefix/lib/mandroid.
  • $prefix/lib/xamarin.android/xbuild/Xamarin/Android/$(HostOS): Operating-system specific binaries, such as the cross compilers (cross-arm).
  • $prefix/lib/xamarin.android/xbuild-frameworks: Xamarin.Android profile assemblies.

This means that a Debian package should (hopefully!) be straightforward: just stick everything into /usr/lib/xamarin.android alongside /usr/bin/xabuild.

@directhex: Would you be able to rebase and cleanup this PR?

[directhex]

Oh, wow, forgot this old PR was even still open.

Right, well, it's done I think. "good enough for government work"

There's a new make target, package-deb, which builds a .deb.

[jonpryor]

Shouldn't this be Source: xamarin.android?

[directhex]

No, the Source: line relates to the name of the source package, which is based on the compressed archive name, which is still oss-xamarin.android_versionnumberstuff

[jonpryor]

We may want to instead consider shipping bin/Release, not bin/Debug.

The question is: do we want to ship binaries full of debugging symbols? For comparison, Mono.Android.dll is ~2.5MB smaller in the Release build.

[directhex]

This... is an issue with the full builds on Jenkins, which I didn't consider from my test builds on my laptop.

I guess it's a policy decision - do we want debug symbols or not? Could we provide both, meaningfully?

[jonpryor]

It's not quite a matter of "debug symbols"; at least for managed code: we always produce .pdb files, even in the Release configuration, and thus we can redistribute them in the package. (At present, they will be redistributed.)

The primary difference is instead on whether optimizations are applied: using csc /optimize+ for managed code, and whether we use -O0 -g (debug) or -O2 (release) for native code.

I thus think we should distribute the bin/Release binaries.

Unfortunately this somewhat complicates the "local Linux build" scenario; you'd need to build with:

make prepare all package-deb CONFIGURATION=Release

which might not be desirable. Is there any way to instead generate the build-tools/debian-metadata/xamarin.android.install file, and use the value of $(CONFIGURATION)/$(lastword $(CONFIGURATIONS))?

[directhex]

Well, yes, since we're building the whole debian/ folder as part of a makefile before trying to use it, we can do anything we like.

[directhex]

OK, this is now built using $(CONFIGURATION). which works in local testing, but i expect was the wrong choice for a real build.

[jonpryor]

This is viable now, but potentially problematic if we ever get another Xamarin system on Linux.

Though at present I doubt iOS or Mac will exist on Linux, so this probably isn't something to worry about...

[directhex]

I can change it now, if you prefer? Or we wait until it becomes a problem

[jonpryor]

We can wait until it's a problem.

[jonpryor]

One problem/concern here is that $$_exclude_list isn't used, meaning all of the extra "junk" that xabuild creates to appease MSBuild will also be present in the .tar.bz2, which doesn't seem correct.

[directhex]

Turns out there's an --exclude-from flag to tar I'd missed, which takes an exclude file from the command like (like -x@ for zip), so that's unified now.

[jonpryor]

This change will break the zip -r command, as without the continuation $$_exclude_list won't be defined, preventing zip -r ... -x@$$_exclude_list from working.

The zip will thus need to be updated to instead use -x@.__exclude_list.txt.

[directhex]

Done.

[harry-cpp]

Once this gets merged I can write a general purpose .run Linux installer PR if you are interested.

[directhex]

I've turned on package building/stashing in Jenkins, this will take effect for the first commit AFTER d45a6e6

[harry-cpp]

Wuhu \o/