Automatic update notifications and security advisories

[lazka]

A bit of a meta issue.

Stories:
a) a new upstream release is out which we could package and we want to be notified (with some upper version limit, in case we have separate packages for different major versions)
b) we want to link packages to cygwin/arch to share patches, see potential problems (we have https://packages.msys2.org/outofdate right now)
c) we want to know all open security advisories for the version currently in the repo and a way to ignore them on a per issue/package basis.

For (a) and (c) we need to know the real package names and real versions of each package. So for example the real pypi name, or a git repo + commit hash if we build from git, or a git tag. In various cases we have to adjust the version format for pacman, or rename the package to avoid naming conflicts or have consistent package names in our repo. If it's a git hash it would also be nice to know what the last/base version is so we can see if there is something newer.

Random list of things that might be helpful here:

• https://github.com/package-url/purl-spec - A URL format for linking to a package source
• https://repology.org/projects/?inrepo=msys2_msys2&vulnerable=1 - parses our repo right now and also lists CVEs
• https://github.com/advisories - githubs collection of security advisories for certain package repos
• https://github.blog/2022-06-17-creating-comprehensive-dependency-graph-build-time-detection/ - github API to push rust/go/python dependency information to github, so it can notify us of security issues
• https://github.com/flathub/flatpak-external-data-checker - a json format and tool to check for new upstream versions (used to automatically update flatpaks)
• https://release-monitoring.org/ - fedora release monitoring repi/api

[dscho]

See also https://github.com/git-for-windows/git/blob/main/.github/workflows/monitor-components.yml, which notifies Git for Windows about new releases in components considered crucial by opening new tickets.

Also, I vaguely recall @Biswa96 working on some Python script to monitor e.g. GitHub CLI updates. (I have a variation of Git for Windows' monitor-components workflow in a private repository of mine that monitors GitHub CLI updates.)

[Biswa96]

Disclaimer: I have no idea about python. 🙇 I did not remember working on that kind of project.

[lazka]

There is now an extra metadata file in each repo which can be used to store things related to a package: https://github.com/msys2/MSYS2-packages/blob/master/PKGMETA.yml

We only use it for https://packages.msys2.org atm

[goyalyashpal]

I do think that how f-droid manages their repo can be took some ideas from.

They have a bot, which scans for updates on the project source repos via some trigger. If update found, those are added to new build pipeline.

They do automated checks for vulnerabilities too IIRC.

Do u want me to tag an fdroid person for their input & participation?

[goyalyashpal]

Ref: #3536 above mentioned by @ Biswa96

I also wanted the information about "out-of-date" packages in the repo to be included in repo index (the repo file which pacman fetches), which is then shown by pacman while running some commands like -S, -Ss or -Qu.

Does that come under this issue?

[lazka]

I haven't found an API/service that allows easy access to CVEs and has some kind of mapping to projects which we can use. The closest is repology, but that doesn't have an API, just a web UI for the security part.

If someone knows some service, here is an example query:

Assuming we have "apr v1.7.0", is it affected by a security advisory? (spoiler: it is)

[goyalyashpal]

in case no service exist, then can web scraping the repology to extract the info help?

[lazka]

scraping is (a) slow (b) not nice the the project being scraped and (c) prone to break quickly

[dscho]

I was recently made aware of this blog post describing how so-called "Bills of Materials" ("BOMs") are generated at scale at Microsoft. These BOMs are stored in .spdx.json files conforming to the SPDX standard and can be used by a GitHub Action called spdx-to-dependency-graph (https://github.com/jhutchings1/spdx-to-dependency-graph-action) to "quickly receive Dependabot alerts for package manifests which GitHub doesn't directly support".

It should be (relatively) straight-forward to generate such .spdx.json files from the information present in the PKGBUILD files (or from the .PKGINFO files contained in the .tar.pkg.zst files). Not the least involved project, I know. But an option.

[lazka]

Thanks. From what I understand this still only supports the ecosystems listed here https://github.com/advisories, so packages from pip, ruby, rust, and go would be relevant, but that's only a fraction of our packages.

For the record, my current tactic is manually look at:

• https://repology.org/projects/?inrepo=msys2_mingw&vulnerable=1
• https://repology.org/projects/?inrepo=msys2_msys2&vulnerable=1

[dscho]

@lazka you're correct. I thought that packages using "other" package management systems could be used here, too, but it seems that there is no standardized way to label the CVEs accordingly ☹️

[lazka]

https://dependencytrack.org/ might be worth a look

(and https://github.com/CycloneDX/cdxgen to get some BOMs for testing)

[lazka]

As a first step, we now automatically create a SBOM file for the git repo content: https://github.com/msys2/MINGW-packages/releases/tag/srcinfo-cache

It only contains some pypi packages + curl atm (for pypi we have the pypi names, and curl is the only package tagged with a CPE atm)

The only public sbom scanner I found was https://snyk.io/code-checker/sbom-security/ (it has a rate limiting...)

Next step: add more CPEs, add a metadata field for the real upstream version (we sometimes change them to work with pacman)

[lazka]

https://github.com/anchore/grype also seems to work outside of the language ecosystems \o/

[lazka]

This is now implemented, see https://www.msys2.org/news/#2024-04-02-automated-vulnerability-reporting-system and https://www.msys2.org/dev/vulnerabilities/