Improve MSYS2 installation

[LeonidLapshin]

Description

Improve MSYS2 installation on Windows 2016/2019

Related issue: #3597

Check list

• Related issue / work item is attached
• Tests are written (if applicable)
• Documentation is updated (if applicable)
• Changes are tested and related VM images are successfully generated

[maxim-lobanov]

@LeonidLapshin , I don't think that we should install any additional MSYS packages to Hosted images.

According to #1572

As a result, the most obvious enhancement would be NOT to install any additional packages in this environment. Just extract the latest stable release, and optionally update the environment once. Then, setup-msys2 would take care of caching (ad-hoc for each user).

cc: @AlenaSviridenko

[LeonidLapshin]

kk, removed

[miketimofeev]

@maxim-lobanov @MSP-Greg there is no sense in this PR at all if can't add the packages

[maxim-lobanov]

My first message was about not adding any new MSYS packages at all. I remember that we had a long discussion with @eine in past in #1525 and #1572 . If I remember correctly, the main point of that discussion was to not install any MSYS packages and provide pure MSYS without any pre-installed packages at all.
Obviously, we couldn't do it because it would break some customers but I guess we don't want to install more packages.

[MSP-Greg]

The 'removed additional ucrt packages' commit either will require changes or should be reverted.

The 'toolchain' group includes several packages, and includes some generic 'build' packages. Below are the packages it installs. Note that some of these packages are also dependencies of other installed packages, (e.g. 'clang' has a dependency on 'gcc')

binutils
crt-git
gcc
gcc-ada
gcc-fortran
gcc-libgfortran
gcc-libs
gcc-objc
gdb
headers-git
libmangle-git
libwinpthread-git
make
pkgconf
tools-git
winpthreads-git
winstorecompat-git

[eine]

As explained in #1572, the current codebase of msys2/setup-msys2 was written by the most active maintainer of MSYS2. It allows using either the pre-installed MSYS2 in the virtual environments provided by GitHub or installing a clean one using the latest tarball (and using the checksum). setup-msys2 uses the caching features provided by GitHub for reducing the setup/update (regardless of using the pre-installed or the clean installation). Furthermore:

• Installing additional packages without updating the system is neither supported nor recommended. It can work, but there is no guarantee.
• Updating the system will reinstall all the pre-installed packages, even if the user does not need them.
• A simple update and installation of GCC can be used for reference:

      matrix:
        task: [ true, false ]
    defaults:
      run:
        shell: msys2 {0}
    steps:

    - uses: msys2/setup-msys2@v2
      with:
        release: ${{ matrix.task }}
        update: true
        install: base-devel gcc git

    - uses: actions/checkout@v2

    - run: gcc --version

The figures:

pre-installed?	#1572	#1572 (gcc)	today	today (gcc)
no	1min29s	1min45s	1min47s	2min26s
yes	8min9s	10min47s	5min53s	6min6s

The difference was reduced, but still users of the pre-built installation are consuming x2.5-x3 on setting up MSYS2. Therefore, given that #1572 didn't have any activity since Sep 2020, it might be sensible to fork setup-msys2 and adapt it for the necessities of the Ruby community.

Anyway, note that there are currently six environments provided by MSYS2:

• MSYS
• MINGW32
• MINGW64
• UCRT64
• CLANG32
• CLANG64

See the README in https://github.com/msys2/setup-msys2:

setup-msys2 is a JavaScript GitHub Action (GHA) to setup an MSYS2 environment (i.e. MSYS, MINGW32, MINGW64, UCRT64, CLANG32, and/or CLANG64 shells) using the GHA toolkit for automatic caching.

I would strongly discourage double or triple betting on the current strategy, as implied by @MSP-Greg. Instead, I believe there is no point in this PR, as @miketimofeev and @maxim-lobanov suggested.

[eine]

Some enhancements such as this one, which don't imply changes in the functionality, might be worth being cherry-picked.

[LeonidLapshin]

@eine thanks a lot for detailed explanation! I'll keep only minor script improvements and remove ucrt installation.

[MSP-Greg]

The issue of what to install has been discussed previously.

A lot of CI is run with code that is *unix/macOS style code, in that it compiles with *nix tools.

1. *nix build tools are installed on Ubuntu & macOS images, people seem to expect them to be installed on Windows. I suspect most do not update/upgrade the build tools on Ubuntu & macOS, they use what's on the images.

2. Most of the people coding with those platforms have never used pacman, they use 'apt' or 'brew'. Obviously, custom actions can shield them from that.

I have never suggested installing many 'free' packages, I've just suggested installing build tools, so there's parity with the Ubuntu & macOS images.

The ucrt build tools fall in that category. Regardless, installing mingw64 & mingw32, but not doing the same for ucrt seems foolish...

[MSP-Greg]

@eine

Not sure what your times are supposed to show. How about the following, which is from a very recent build of the Ruby programming language. It updates gcc and all the packages needed to build Ruby. Totals 17.22 seconds.

sed -i 's/^CheckSpace/#CheckSpace/g' C:/msys64/etc/pacman.conf
pacman.exe -Sy pacman-mirrors
  took   7.82 s
Upgrading gcc for Ruby 2.6.7
  took   6.36 s
pacman.exe -S  mingw-w64-x86_64-gdbm mingw-w64-x86_64-gmp mingw-w64-x86_64-libffi mingw-w64-x86_64-libyaml mingw-w64-x86_64-openssl mingw-w64-x86_64-ragel mingw-w64-x86_64-readline automake1.16 bison
  took   3.04 s

[eine]

@MSP-Greg, MSYS2's release model is a rolling one, unlike Ubuntu or most *nix distributions. The comparison is, thus, out of context. Should you want to use pinned versions, it'd be better to use chocolatey, conda or nuget. In fact many packages from MSYS2 are available through chocolatey or conda, for that specific reason.

people seem to expect them to be installed on Windows

Please, specify which people. AFAIAA, there is a single active user repeatly requesting more tools to be installed for a very specific use case (the Ruby project).

I have never suggested installing many 'free' packages,

So far, you requested approximately 50 packages to be installed. Moreover, you replied explicitly to this issue for requesting more.

Regardless, installing mingw64 & mingw32, but not doing the same for ucrt seems foolish...

You seem to be ignoring CLANG32 and CLANG64 and my perception is that it's motivated because they are not used in the Ruby project (yet).

Not sure what your times are supposed to show.

The execution times on GitHub Actions (not mine) show that using an up to date GCC takes at least 2.5 times longer if the pre-built installation is used, compared to using a clean setup.
Therefore, it explains why many users don't care about the pre-installed setup and it might be considered just a traffic burden for GitHub.

How about the following, which is from a very recent build of the Ruby programming language.

Please, provide reproducible examples.

[MSP-Greg]

You seem to be ignoring CLANG32 and CLANG64

I certainly did not say they shouldn't be installed. If people request them, they should be considered.

The execution times on GitHub Actions (not mine) show that using an up to date GCC...

My 17 second time is updating gcc.

Please, provide reproducible examples.

https://github.com/ruby/ruby/actions/workflows/mingw.yml

[MSP-Greg]

One point that is important is that many people do not update build tools, regardless of the OS that they're using.

Another issue is what is the purpose of MSYS2 as it relates to CI. I see it as providing build tools for compiling code that will run using standard Windows shells. It can also provide a minimal bash shell so that bash CI scripts can be run with Windows.

I don't use MSYS2 much locally, except for the above. Might as well use WSL2...

[eine]

@MSP-Greg, I run some additional tests for your specific set of packages:

• https://github.com/eine/virtual-environments/actions/runs/972005656
• https://github.com/eine/virtual-environments/actions/runs/972021038
• https://github.com/eine/virtual-environments/actions/runs/972028810
• https://github.com/eine/virtual-environments/actions/runs/972054041

Summarising:

• Using the pre-built installation and not updating the system: 20-50s.
• Using the pre-built installation and updating the system: >270s.
• Using a clean installation and either updating the system or not: 60-85s.

Therefore, I'm proposing a solution that requires 1-3min to any user, regardless of the tools they need to install. That is also the stable outcome that consumers of setup-msys2 get by ignoring the erratic handling of the host.
You are proposing a solution for saving 40s in a 35-40 min workflow (that is 2% of the execution time) at the cost of increasing the execution time of users who want to update the pre-built installation to 4-8 minutes.

---

One point that is important is that many people do not update build tools, regardless of the OS that they're using.

Please provide some reference for supporting these "important" claims.

Another issue is what is the purpose of MSYS2 as it relates to CI. I see it as providing build tools for compiling code that will run using standard Windows shells. It can also provide a minimal bash shell so that bash CI scripts can be run with Windows.

I don't use MSYS2 much locally, except for the above. Might as well use WSL2...

I think these philosophical discussions are out of the scope of this PR.

[MSP-Greg]

You are proposing a solution for saving 40s in a 35-40 min workflow (that is 2% of the execution time)

So now it's percent, not absolute? I could probably find several repos where the CI's build/test time is at least an order of magnitude shorter, but I'm not going to waste my time.

One point that is important is that many people do not update build tools, regardless of the OS that they're using
Please provide some reference for supporting these "important" claims.

Prove it wrong.

I think these philosophical discussions are out of the scope of this PR.

The main reason for using MSYS2 in CI (as opposed to local use) is certainly important.

You seem to be implying that setup-msys2 use is widespread. setup-msys2 has 85 stars and 10 forks. setup-ruby has 363 stars and 78 forks. There doesn't seem to be any data as to an action's actual use, but stars and forks indicates something. setup-ruby doesn't allow for any changes to the MSYS2 installation, but enables its use via Path additions.

If the current MSYS2 installation is unworkable, where are all the complaints?

If one doesn't like the current install, just rename the folder and start over...

[eine]

So now it's percent, not absolute?

I wrote 12 different absolute values in the previous message: 20s, 50s, 270s, 60s, 85s, 1min, 3min, 40s, 35min, 40min, 4min, 8min. Then, a single relative one, which is a relation between two of those absolute values. See https://en.wikipedia.org/wiki/Relative_change_and_difference#Comparisons for reference.

I could probably find several repos where the CI's build/test time is at least an order of magnitude shorter, but I'm not going to waste my time.

I believe you could not. Please, be coherent and don't waste our time with vacuous statements.

One point that is important is that many people do not update build tools, regardless of the OS that they're using

Please provide some reference for supporting these "important" claims.

Prove it wrong.

Unfortunately, that is not how argumentation works. You are the one interested in that argument being valid. Not me! See https://en.wikipedia.org/wiki/Argumentation_theory and https://en.wikipedia.org/wiki/Burden_of_proof_(philosophy) for reference:

When two parties are in a discussion and one makes a claim that the other disputes, the one who makes the claim typically has a burden of proof to justify or substantiate that claim especially when it challenges a perceived status quo.[1] This is also stated in Hitchens's razor, which declares that "what may be asserted without evidence, may be dismissed without evidence."

In this case, GitHub provides a search feature, which you can use for proving yourself correct or wrong: https://github.com/search?q=setup-msys2&type=code. I don't care the outcome being one or the other, I care about poor argumentation principles and development based on uneducated opinions.

I think these philosophical discussions are out of the scope of this PR.

The main reason for using MSYS2 in CI (as opposed to local use) is certainly important.

I beg to disagree. The reasons are important. The "main" is not, specially provided that it is considered the "main" based on the opinion of a sole person who is not willing to spend time in proving himself corrrect.

You seem to be implying that setup-msys2 use is widespread. setup-msys2 has 85 stars and 10 forks. setup-ruby has 363 stars and 78 forks. There doesn't seem to be any data as to an action's actual use, but stars and forks indicates something. setup-ruby doesn't allow for any changes to the MSYS2 installation, but enables its use via Path additions.

Please, try to stay on topic. Comparing the number of stars and/or the usage of setup-msys2 and setup-ruby is worthless, unless you expect setup-ruby to be a replacement and a general solution for any user to consume MSYS2. Is that what you are implying? If so, please provide an analysis of how many of the users of setup-ruby are Windows users and do care about consuming the MSYS2 build explicitly, or any other Ruby build which is usable.

Moreover, I am not implying that usage of setup-msys2 is widespread. That is a fact: https://github.com/search?q=setup-msys2&type=code. I don't need to compare numbers with any other project in order to acknowledge there is a sufficiently relevant base of users consuming MSYS2 in GitHub Actions using the officially recommended solution by MSYS2 maintainers.

If the current MSYS2 installation is unworkable, where are all the complaints?

Please, don't dress me on straw. I did not say that the pre-installed setup is unworkable. Precisely, it is supported in setup-msys2 because it is usable; otherwise it would have been removed long ago.

There are no complainers because, as I explained, most users are ignoring the pre-installed setup. It is mostly irrelevant, and generating unnecessary traffic to GitHub. Since only GitHub and a few consumers of private Actions are affected economically, it might get long until fixed.

If one doesn't like the current install, just rename the folder and start over...

That is exactly what setup-msys2 does. It allows users to consume the pre-built installation or to ignore it and use a clean one. Out of 42 random results from https://github.com/search?q=setup-msys2&type=code:

• 31 are using a clean and updated setup.
• 9 are using a clean and non updated setup.
• 2 are using the pre-installed and non updated setup.

Hence, 95.24% of the users are ignoring the pre-installed setup, and 26.19% are not updating the system. So, with regard to your previous "important" claim, 73.81% of the setup-msys2 users do explicitly care about using an updated setup. Maybe, that is because of it being the only recommended and supported use case.

[MSP-Greg]

I added installing ucrt gcc tools to a repo, the extra time varied from 35 to 71 sec, download was about 56 mB, size on disk was 440 mB.

The jobs using mingw64 take about 1 to 2 minutes (with no MSYS2 downloads).

The 56 mB could be cached, but with some workflows there maybe multiple jobs using ucrt. I've never tried using a cache that was shared between jobs...

[larskanis]

• Installing additional packages without updating the system is neither supported nor recommended. It can work, but there is no guarantee.

This is wrong. It's perfectly valid to install additional packages without updating the system in MSYS2. This is supported and it installs the package versions of the point in time of the last system update. Not updating is therefore the default in setup-msys2.

Updating the package database and updating the MSYS2 system are two distinct operations in pacman. The thing what's not supported is to update the package database and install a package from the newer package database without updating the system as well. That can work, but isn't guaranteed.

[eine]

It's perfectly valid to install additional packages without updating the system in MSYS2. This is supported and it installs the package versions of the point in time of the last system update.

@larskanis, there is no guarantee that the versions of the packages referenced by the database at that point in time are still available in the servers/mirrors. Unlike other distributions, there is no specific "contract" about how long are "old" packages to be kept in the servers; it's a rolling release. Therefore, installing additional packages without updating the system is a not recommended use case. As said, it can work (both the pre-built setup in this repo and the update option in setup-msys2 do acknowledge that), but there is no guarantee. Those do actually work because of the frequent (2-4 weeks) updates of the installations (which is not explicitly guaranteed by neither MSYS2 or GitHub) and because maintainers of MSYS2 try not to remove old packages unless there are problems with size.

Updating the package database and updating the MSYS2 system are two distinct operations in pacman. The thing what's not supported is to update the package database and install a package from the newer package database without updating the system as well. That can work, but isn't guaranteed.

Updating the database and not the system is another possible, not guaranteed and not recommended use case. This is more problematic, and that's why many users understand why this should not be used, but do not get why the previous use case is almost the same.

Anyway, if GitHub wanted to mirror MSYS2 repositories and to guarantee that all packages available at the time the image is generated are properly cached in the mirror/registry, then it would be possible to guarantee that installing packages without updating the system or the database would work. See https://www.msys2.org/dev/mirrors/#adding-a-new-mirror.

[larskanis]

@larskanis, there is no guarantee that the versions of the packages referenced by the database at that point in time are still available in the servers/mirrors.

@eine The MSYS2 team keeps the packages for a period of at least one year, some mirrors maybe longer. Combined with the regular update of MSYS2 in the Github Action environment, not updating the MSYS2 system for installing packages is a safe bed in Github Actions and should be preferred. In my experience it it is even safer than updating the MSYS2 system, since the update process has been broken from time to time in the past.

[eine]

@eine The MSYS2 team keeps the packages for a period of at least one year, some mirrors maybe longer.

@larskanis, that is a one time announcement, not an statement that guarantees a criteria to be maintained. The announcement did not specify whether they are saving all the packages since 2019-10 (one year before the reference), or 2020-07 (one year before today). Very precisely, there is a more recent announcement (just scroll up in the reference you provided): 2021-02-27 - New server for repo.msys2.org and packages.msys2.org. Moreover, yesterday they announced: 2021-07-04 - Some Mirror/Server/Repository Changes. Note:

Due to space constraints we no longer host the source packages on Sourceforge. They are still available on our main server and on all mirrors.

Hence, overall, the reference you provided is mostly invalid, because it refers to the constraints in a server/service which is not used anymore.

Combined with the regular update of MSYS2 in the Github Action environment, not updating the MSYS2 system for installing packages is a safe bed in Github Actions and should be preferred. In my experience it it is even safer than updating the MSYS2 system, since the update process has been broken from time to time in the past.

I wish that was correct. Would you mind opening an issue for discussion in the MSYS2 organisation?

Given the evolution of the MSYS2 project during the last months (it seems to be sponsored: https://www.msys2.org/#sponsors), it might be feasible to make an official statement about retention in the main repo and/or the mirrors. There are several mirror tiers (see https://www.msys2.org/dev/mirrors/). Hence, it would be sensible to require Tier 1 mirrors some retaining policy which Tier 2 mirrors might not comply with.

Moreover, as I suggested in my previous comment, I would strongly recommend GitHub to mirror MSYS2 packages in their own registry (or to provide that feature to the MSYS2 organisation), so that it can be used as the main (and maybe only) mirror in the built-in MSYS2 installation in this repo. That would guarantee that the MSYS2 installation and the mirror are frozen in the same state; as well as help reduce the traffic from MSYS2 servers by keeping it internal in GitHub. Then, we might apply the 30-50 sec optimisation from the Ruby project to setup-msys2.

[MSP-Greg]

I've been looking at creating archive files (*.7z) for the 'parts' of MSYS2 normally used by Ruby. They're stored in a GitHub release. First package was a 'ucrt gcc' package, and it installs in a reasonable amount of time.

Currently, many Ruby CI runs have several Ruby versions tested. Some update gcc packages, some may not. With the current setup, updating gcc is done by every job, and does so by hitting an MSYS2 server.

By using a 'ucrt gcc' archive file, everything is pulled from a GitHub server, and MSYS2 servers are only hit when the cron job creating the archive file is run.

I'm now working on creating a base MSYS2 archive file, and seeing the performance of that.

Hence, Ruby specific archives would be MSYS2, MinGW64, and UCRT64. This would allow a fresh set of build tools to be installed, hopefully quickly.

Given that the future of MSYS2 on the GitHub images is in question...

[eine]

With the current setup, updating gcc is done by every job, and does so by hitting an MSYS2 server.

@MSP-Greg, one of the advantages of setup-msys2 is that caching is used by default. Therefore, the packages specified in the install field are downloaded from MSYS2 once only (when the version in the cache needs to be updated), then from GitHub. That works regardless of the release option. Hence, users of the built-in installation or users of the clean setup can both benefit from the caching:

Therefore, usage of this Action is recommended to all MSYS2 users of GitHub Actions, since caching and the custom entrypoint are provided regardless of option release.

Nevertheless, it would be interesting to support users providing the "cache" as the URL to a tarball, in case they want to use that instead of the GitHub cache. Actually, some users have setup the GitHub Releases of some repo in order to use them as a mirror. Therefore, in cases such as Ruby, you might want both: have a custom cache in a release, and have the packages you will consume in your own release/mirror.

I'm now working on creating a base MSYS2 archive file, and seeing the performance of that.

Note that doing so was already prototyped in setup-msys2, and it was enabled by default for some time. However, it produced problems. That's why now packages are cached, but not the whole installation. In fact, the feature is still in the codebase, but it is disabled by default. See msys2/setup-msys2@434d91b, msys2/setup-msys2#61 and msys2/setup-msys2#50. Therefore, instead of reimplementing it yourself, you might want to just fork setup-msys2 and set that variable to true.

Moreover, in msys2/msys2-autobuild you will find how to use the Releases as a cache. In case, you really want to implement a solution on your own, I recommend to see how they do it.

Given that the future of MSYS2 on the GitHub images is in question...

How is that? Until very recently, using it as the default shell on Windows was being discussed.

[MSP-Greg]

EDIT: sorry hit the wrong key...

How is that?

Can't find it, but I thought there was a comment about removing some parts of the current MSYS2 install in Windows-2022? Not sure which parts.

I think I mentioned somewhere above that I haven't seen an Actions cache used in multiple jobs, and with typical Ruby workflows, there maybe several jobs using MSYS2 build tools.

Regardless, rather than have multiple repos caching a 'gcc' install for mingw64 and/or ucrt64, it would be better to have a separate repo that builds an archive once or more daily.

[eine]

I think I mentioned somewhere above that I haven't seen an Actions cache used in multiple jobs. Regardless,

@MSP-Greg, precisely:

Nevertheless, it would be interesting to support users providing the "cache" as the URL to a tarball, in case they want to use that instead of the GitHub cache. Actually, some users have setup the GitHub Releases of some repo in order to use them as a mirror. Therefore, in cases such as Ruby, you might want both: have a custom cache in a release, and have the packages you will consume in your own release/mirror.

Moreover, in msys2/msys2-autobuild you will find how to use the Releases as a cache.