Generate APT repo for releases and under dev builds

[HebaruSan]

Motivation

Currently we provide .deb packages, but somewhat ironically (given what CKAN is for), the user must download a new .deb file and install it manually when a new version is released.

Changes

Now when we build the .deb file, we also create a _build/deb/apt-repo folder containing the .deb file, a Release file, and a Packages.gz file, modeled on Mono's:

• https://download.mono-project.com/repo/ubuntu/dists/stable-bionic/main/binary-amd64/

For releases, these are uploaded to: https://ksp-ckan.s3-us-west-2.amazonaws.com/deb/dists/stable/main/binary-all
For other builds, these are uploaded to: https://ksp-ckan.s3-us-west-2.amazonaws.com/deb/dists/nightly/main/binary-all

This should allow users to add one of the following lines to their sources.list and get ckan releases automatically:

deb https://ksp-ckan.s3-us-west-2.amazonaws.com/deb stable main

deb https://ksp-ckan.s3-us-west-2.amazonaws.com/deb nightly main

NOTE: This is all based on documentation of how to build an APT repo. It still needs to be tested to confirm that it will actually work with APT clients.

Open questions

• Is the level of traffic associated with a deb repo acceptable for S3?
  • Previous CKAN release (v1.28.0) has 111977 downloads total, and 3285 downloads for its .deb (https://api.github.com/repos/KSP-CKAN/CKAN/releases)
  • The Packages.gz file is 483 bytes
  • Supposing that every one of those 3285 users would sign up for the APT repo, that's 1.5 MB of traffic every however-often-they-check-for-updates (in aggregate)
  • The deb file is 748 KB, so about 2.5 GB for all of those users to download it
• Do we need the --delete flag for deploy.yml, or would it erase the release builds as well?
• Should we modify our versioning for nightly builds, and if so, how?

[techman83]

I think lets get this merged in and start doing some testing. Traffic wise it should be pretty negligible in the scheme of things.

I do however think we should consider signing the releases on the next iteration

https://wiki.debian.org/SecureApt

We can host the public key somewhere and put the private key into CI. Would be similar how the likes of spotify distribute their releases.

https://www.spotify.com/au/download/linux/

[HebaruSan]

🤞

[HebaruSan]

https://ksp-ckan.s3-us-west-2.amazonaws.com/

<Contents>
<Key>deb/dists/nightly/main/binary-all/Packages.gz</Key>
<LastModified>2020-11-17T20:10:39.000Z</LastModified>
<ETag>"499bbef01d297e20bcd50f86ccf90afb"</ETag>
<Size>482</Size>
<StorageClass>STANDARD</StorageClass>
</Contents>
<Contents>
<Key>deb/dists/nightly/main/binary-all/Release</Key>
<LastModified>2020-11-17T20:10:39.000Z</LastModified>
<ETag>"e6f45e253b75aa1129d4abd75158d0cb"</ETag>
<Size>129</Size>
<StorageClass>STANDARD</StorageClass>
</Contents>
<Contents>
<Key>deb/dists/nightly/main/binary-all/ckan_1.29.1_all.deb</Key>
<LastModified>2020-11-17T20:10:39.000Z</LastModified>
<ETag>"70eda5aa485cc3e5b1329ad0375de0a5"</ETag>
<Size>765360</Size>
<StorageClass>STANDARD</StorageClass>
</Contents>

Well, the files are there. I'm in Windows right now so I can't test immediately, but if somebody wants to try it out, this one should be active now:

deb https://ksp-ckan.s3-us-west-2.amazonaws.com/deb nightly main

[HebaruSan]

Attempting to re-run the Eddington release to generate the stable builds...

• https://github.com/KSP-CKAN/CKAN/runs/1414621626

EDIT: Oh, that won't work because the release.yml file will be the old version from the release commit. Oops, oh well.
... and it doesn't even let us replace the assets anyway. That's fine.

[DasSkelett]

It complains about a missing Release-file. Not sure where (else) it expects it though.

Fehl:22 https://ksp-ckan.s3-us-west-2.amazonaws.com/deb nightly Release
404  Not Found [IP: 52.218.233.33 443]
Paketlisten werden gelesen... Fertig
E: Das Depot »https://ksp-ckan.s3-us-west-2.amazonaws.com/deb nightly Release« enthält keine Release-Datei

[HebaruSan]

Could a --verbose or --debug flag give us more info, like the URL that is tried?

[DasSkelett]

Couldn't find a debug flag, but downgrading the repo to HTTP and some Wireshark action revealed the files it tries to access:

s3-us-west-2-r-w.amazonaws.com	http	HTTP		250		GET /deb/dists/nightly/InRelease HTTP/1.1 
s3-us-west-2-r-w.amazonaws.com	http	HTTP		248		GET /deb/dists/nightly/Release HTTP/1.1 

Which matches the info from https://wiki.debian.org/DebianRepository/Format:

To download packages from a repository apt would download a InRelease or Release file from the $ARCHIVE_ROOT/dists/$DISTRIBUTION directory.

[HebaruSan]

Yup, Mono has InRelease, Release, and Release.gpg at what would be equivalent to our /deb/dists/nightly folder:

• https://download.mono-project.com/repo/ubuntu/dists/stable-bionic/

[HebaruSan]

@DasSkelett Can we make env an array, to push multiple folders to multiple destinations? I think I've seen syntax like that in yml files but I have no idea how it works or how to check if it's allowed.

CKAN/.github/workflows/deploy.yml

Lines 76 to 88 in 984c39e

	- name: Push deb to S3
	# Send deb file to https://ksp-ckan.s3-us-west-2.amazonaws.com/
	uses: jakejarvis/s3-sync-action@master
	with:
	args: --follow-symlinks
	env:
	AWS_S3_BUCKET: ksp-ckan
	AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
	AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
	AWS_REGION: us-east-1
	SOURCE_DIR: _build/deb/apt-repo
	DEST_DIR: deb/dists/nightly/main/binary-all
	if: ${{ env.AWS_ACCESS_KEY_ID && env.AWS_SECRET_ACCESS_KEY }}

[DasSkelett]

I don't think this is possible here, at least I'm not finding anything in the docs.
The only thing that come to my mind is strategy.matrix, but that one reruns the complete job.

[HebaruSan]

Better now?

[DasSkelett]

Working on a summary of all the warnings and errors 😬

[DasSkelett]

New round:

E: Das Depot »http://ksp-ckan.s3-us-west-2.amazonaws.com/deb nightly Release« ist nicht signiert.
N: Eine Aktualisierung von solch einem Depot kann nicht auf eine sichere Art durchgeführt werden, daher ist es standardmäßig deaktiviert.
N: Weitere Details zur Erzeugung von Paketdepots sowie zu deren Benutzerkonfiguration finden Sie in der Handbuchseite apt-secure(8).
---
E: The repository »http://ksp-ckan.s3-us-west-2.amazonaws.com/deb nightly Release« is not signed.
N: Data from such a repository can't be authenticated and is therefore potentially dangerous to use.
N: See apt-secure(8) manpage for repository creation and user configuration details.

Can be bypassed by adding [trusted=yes] to the source line.

Then there are a bunch of conflicting architecture warnings, can be reduced by adding [arch=all] (so you have [trusted=yes arch=all]), but the following two architecture warnings remain:

N: Das Laden der konfigurierten Datei »main/dep11/Components-amd64.yml« wird übersprungen, da das Depot »http://ksp-ckan.s3-us-west-2.amazonaws.com/deb nightly InRelease« die Architektur »amd64« nicht unterstützt.
N: Das Laden der konfigurierten Datei »main/cnf/Commands-amd64« wird übersprungen, da das Depot »http://ksp-ckan.s3-us-west-2.amazonaws.com/deb nightly InRelease« die Architektur »amd64« nicht unterstützt.
---
N: Skipping acquire of configured file 'main/dep11/Components-amd64.yml' as repository 'http://ksp-ckan.s3-us-west-2.amazonaws.com/deb nightly InRelease' doesn't support architecture 'amd64'
N: Skipping acquire of configured file 'main/cnf/Commands-amd64' as repository 'http://ksp-ckan.s3-us-west-2.amazonaws.com/deb nightly InRelease' doesn't support architecture 'amd64'

And those warnings appear, too:

W: Kein Hash-Eintrag in Release-Datei /var/lib/apt/lists/ksp-ckan.s3-us-west-2.amazonaws.com_deb_dists_nightly_Release
W: Ungültiger »Date«-Eintrag in Release-Datei /var/lib/apt/lists/ksp-ckan.s3-us-west-2.amazonaws.com_deb_dists_nightly_Release
W: Konflikt bei Distribution: http://ksp-ckan.s3-us-west-2.amazonaws.com/deb nightly Release (nightly erwartet, aber  bekommen)
---
W: No Hash-entry in Release-file /var/lib/apt/lists/ksp-ckan.s3-us-west-2.amazonaws.com_deb_dists_nightly_Release
W: Invalid »Date«-entry in Release-Datei /var/lib/apt/lists/ksp-ckan.s3-us-west-2.amazonaws.com_deb_dists_nightly_Release
W: Conflicting distribution: http://ksp-ckan.s3-us-west-2.amazonaws.com/deb nightly Release (expected nightly, but got  ) <---- note the empty string

sudo apt install ckan also fails:

Fehl:3 http://ksp-ckan.s3-us-west-2.amazonaws.com/deb nightly/main all ckan all 1.29.1                                  
404  Not Found [IP: 52.218.240.57 80]                           
E: Fehlschlag beim Holen von http://ksp-ckan.s3-us-west-2.amazonaws.com/deb/./ckan_1.29.1_all.deb 404  Not Found [IP: 52.218.240.57 80]
---
Err:3 http://ksp-ckan.s3-us-west-2.amazonaws.com/deb nightly/main all ckan all 1.29.1                                  
404  Not Found [IP: 52.218.240.57 80]                           
E: Error while fetching http://ksp-ckan.s3-us-west-2.amazonaws.com/deb/./ckan_1.29.1_all.deb 404  Not Found [IP: 52.218.240.57 80]

[HebaruSan]

OK, clearly my guess at how to handle platform-neutral packages was wrong. Are you able to find any better methods?

[HebaruSan]

And apt-scanpackages's file path output is interpreted relative to the repo root, not the folder in which the Packages.gz file is placed. Fascinating.

[HebaruSan]

https://wiki.debian.org/DebianRepository/Format#Architectures

The presents of the architecture all in this field indicates that the architecture-specific indexes do not include information about Architecture:all packages and have instead their own index file with the architecture all. Clients must download the all index files in this case, but must not download them if the Architectures field does not include all.

I think they meant "presence" instead of "presents", but otherwise I am just not understanding these words. Any idea what the heck this is supposed to mean?

[DasSkelett]

Same, most of this guide is either written in a confusing way or seems to be missing important information to make sense.
Setting up APT repositories is an art by itself.
I'm going to dig further into it tomorrow, if you haven't solved it already until then ^^