tox.ini, build/bin/write-dockerfile.sh: Add gentoo linux, add more gentoo packages

[mkoeppe]

As a followup to #29053 and #29273, we

• extend build/bin/write-dockerfile.sh to generate a Dockerfile using https://hub.docker.com/r/sheerluck/sage-on-gentoo-stage4
• add build/pkgs/gentoo*.txt
• extend bin/sage-print-system-package-command to handle gentoo
• add gentoo to the platforms tested in GitHub CI.

To test: tox -e docker-gentoo-standard

This is a part of:

• Meta-ticket: Add Dockerfiles and CI scripts for integration testing of source and binary distributions and of downstream packages #29060: Meta-ticket: Add Dockerfiles and CI scripts for integration testing of source and binary distributions and of downstream packages

---

Resources:

• https://forums.gentoo.org/viewtopic-t-969278-start-0.html
• cschwan/sage-on-gentoo: (Unofficial) Gentoo Overlay for Sage- and Sage-related ebuilds: https://github.com/cschwan/sage-on-gentoo
• Project:Overlays/Old User Guide - Gentoo Wiki: https://wiki.gentoo.org/wiki/Project:Overlays/Old_User_Guide
• Layman - Gentoo Wiki: https://wiki.gentoo.org/wiki/Layman

Depends on #29273

CC: @orlitzky @kiwifb @dimpase @sheerluck

Component: porting

Author: Andrey Belgorodski

Branch: 9959ee9

Reviewer: Matthias Koeppe

Issue created by migration from https://trac.sagemath.org/ticket/29105

[mkoeppe]

comment:1

Creating a dockerfile for setting up sage prerequisites on a Gentoo "prefix" was previously discussed in #28905.

An attempt (incomplete) is at https://github.com/sagemath/sage-prod/files/10659346/Dockerfile-sage_gentoo.gz and would need help from Gentoo experts.

[kiwifb]

comment:2

compiler can't create executables in a configure phase of gcc typically means the compiler can't find libgmp and al. Do you have a log for me to review? Prefix on ubuntu is definitely supported. The only thing I can think of is that the bootstrap usually object being run as root.

[mkoeppe]

comment:3

Thanks, yes, turns out this was not specific to the gentoo prefix installer. I had to add some more packages. Let me see how far I get with this.

[mkoeppe]

Attachment: gentoo-log-2.txt

[mkoeppe]

comment:4

Attachment: Dockerfile-sage_gentoo.gz

OK... this is where I got now. The ./bootstrap-prefix.sh script was claiming to bootstrap some packages - such as xz and patch - and succeeded with that; but then it was missing "xz" to unpack the Python sources. I think something is wrong there, but I added xz&patch in ubuntu. This is attached Dockerfile-sage_gentoo and log from 'docker build' with that.

As you can see in the log, now it's complaining about a module missing in the Python that it just built:

  File "/gentoo/var/tmp/python-3.6.10/Python-3.6.10/Lib/xml/parsers/expat.py", line 4, in <module>
    from pyexpat import *
ModuleNotFoundError: No module named 'pyexpat'
make: *** [install] Error 1
Makefile:1102: recipe for target 'install' failed
??? Python failed to install *sigh* continuing anyway
* Python bootstrapped
http://distfiles.gentoo.org/snapshots

and then dies with:

* Compiling prefix-portage
./configure --host=x86_64-pc-linux-gnu --prefix=/gentoo/tmp/usr --mandir=/gentoo/tmp/usr/share/man --infodir=/gentoo/tmp/usr/share/info --datadir=/gentoo/tmp/usr/share --sysconfdir=/gentoo/tmp/etc --localstatedir=/gentoo/tmp/var/lib --build=x86_64-pc-linux-gnu --with-offset-prefix=/gentoo/tmp --with-portage-user=root --with-portage-group=root --with-extra-path=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
checking build system type... x86_64-pc-linux-gnu
checking host system type... x86_64-pc-linux-gnu
checking target system type... x86_64-pc-linux-gnu
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... /bin/mkdir -p
checking for gawk... no
checking for mawk... mawk
....
checking whether /usr/bin/id is good enough... yes
checking for python... no
configure: error: no python found in your path

[kiwifb]

comment:5

xz is rearing up its head again. Yes it should be an early one. Because many more sources are in .xz I am not shocked that it should be a pre-requisite along with patch. The pyexpat stuff is weird. I'll do some experimenting later. I haven't really touched prefix since ppc64 was retired (because I didn't want or could maintain it long term).

[mkoeppe]

Dependencies: #29273

[mkoeppe]

comment:9

This tickets needs help from someone who actually uses Gentoo. I found the instructions for installing stuff inadequate.

[dimpase]

comment:10

I use Gentoo, but neither prefix, not docker.

[kiwifb]

comment:11

I need to make time for this. I cannot give you a time frame.

[mkoeppe]

comment:12

Replying to @dimpase:

I use Gentoo, but [...] not docker.

After installing a docker client, it only takes

docker run -it gentoo/stage3-amd64

This gives you root prompt. I would need instructions on what to do next to install relevant gentoo packages here.

[kiwifb]

comment:13

If I understand well you have a minimal gentoo image running now. What do you want to install? Do you have a list somewhere?

[mkoeppe]

comment:14

Replying to @kiwifb:

If I understand well you have a minimal gentoo image running now. What do you want to install? Do you have a list somewhere?

The equivalent of the lists in build/pkgs/debian-bootstrap.txt and build/pkgs/debian.txt (minimal requirements for bootstrapping and building) -- which would be added in #29273 as well since gentoo packages are being added there.

[dimpase]

comment:15

Replying to @mkoeppe:

Replying to @dimpase:

I use Gentoo, but [...] not docker.

After installing a docker client, it only takes

docker run -it gentoo/stage3-amd64

This gives you root prompt. I would need instructions on what to do next to install relevant gentoo packages here.

I can try either this or Gentoo in lxc container, if such a thing exists somewhere.

[mkoeppe]

comment:16

Docker please because that's what my tox.ini infrastructure uses.
If we want to do the same with lxc instead of docker, why not, but perhaps one step at a time?

[mkoeppe]

comment:17

I have created ticket #29283 for the goal of lxc containers for testing, in case someone wants to work on that.

[mkoeppe]

comment:21

Starting points for people who want to help with this:

• https://doc.sagemath.org/html/en/developer/portability_testing.html
• https://researchseminars.org/talk/SageDays109/7/

[sheerluck]

comment:22

When I run tox -e docker-gentoo I get half-baked sage/.tox/docker-gentoo/Dockerfile, I need a way to shove the hand-crafted Dockerfile instead (for debugging purposes). What is the best way to do it?

[mkoeppe]

comment:23

Does your handcrafted Dockerfile work with docker build already?

[sheerluck]

comment:82

I don't want my name to be published

[fchapoton]

comment:83

In github, your are "Andrew" from Belgorod, Russia. What's your exact problem with our rule that contributors should give their full real name ?

[mkoeppe]

Changed author from François Bissey to Andrey Belgorodski

[vbraun]

Changed branch from u/gh-sheerluck/gentoo to 9959ee9

[sheerluck]

comment:86

What’s New in sheerluck/sage-on-gentoo-stage4:20.7

1) base image:
   FROM gentoo/portage:20200701 as portage
   FROM gentoo/stage3-amd64:20200701

2) python compiled with sqlite support
   (fixes ModuleNotFoundError: No module named 'sqlite3')

3) updates:
   pandoc             2.9.2.1 -> 2.10 
   sci-libs/amd       2.3.1   -> 2.4.6
   sci-libs/cholmod   2.1.2   -> 3.0.13
   sci-libs/colamd    2.8.0   -> 2.9.6
   sci-libs/openblas  0.3.9   -> 0.3.9-r1
   dev-lang/perl      5.30.1  -> 5.30.3
   dev-lang/R         4.0.0   -> 4.0.2
   sci-libs/umfpack   5.6.2   -> 5.7.9 

with latest image only 1 doctest failed: combinat/cluster_algebra_quiver/quiver.py

[sheerluck]

Changed commit from 9959ee9 to none

[mkoeppe]

comment:87

Great! I am assuming no change to tox.ini is required?

[sheerluck]

comment:88

no changes

[sheerluck]

comment:89

Last error from https://github.com/mkoeppe/sage/actions, namely this one:

It might be possible to solve this slot collision
by applying all of the following changes:
   - sci-libs/m4ri-20200125 (Change USE: +png)
   - sci-libs/brial-1.2.8 (Change USE: +png)

The following USE changes are necessary to proceed:
 (see "package.use" in the portage(5) man page for more details)
# required by sci-libs/brial-1.2.8::sage-on-gentoo
# required by sci-libs/brial (argument)
>=sci-libs/m4ri-20200125 -png 

is solved with png for sci-libs/brial in latest image

(checked on u/mkoeppe/fix_tox_docker_builds_broken_by__29884)

[sheerluck]

comment:90

-----------------
All tests passed!
-----------------
____ summary ____
docker-gentoo-standard: commands succeeded
congratulations :)

[sheerluck]

comment:91

I've created latest-gcc10 in addition to latest. For fun.

 version in            latest         latest-gcc10
====================================================
 sys-devel/gcc         9.3.0          10.1.0-r2
 sci-libs/arpack       3.1.5          3.5.0
 sci-mathematics/gp2c  0.0.11.2-r1    0.0.11.3
 app-misc/rlwrap       0.42           0.43

But tox -e docker-gentoo-gcc10-maximal -- build doc-html doc-pdf ptest says

checking gcc version... 10.1.0
checking g++ version... 10.1.0
configure: Installing GCC because g++ -std=gnu++11 is g++ version 10.1.0, which is too recent for this version of Sage 
configure: no suitable system package found for SPKG gcc

Sage is not ready for gcc10 yet...

but if anyone wants to play with latest-gcc10 it's available on hub.docker.com

[dimpase]

comment:92

Thanks. A related latest annoyance is that on macOS Homebrew provides gcc package, which is gcc-10, and this breaks things (even though only gfortran is used, not gcc).

[sheerluck]

comment:93

What’s New in sheerluck/sage-on-gentoo-stage4:20.8

1) base image:
   FROM gentoo/portage:20200801 as portage
   FROM gentoo/stage3-amd64:20200801

2) updates:
   arb                2.16.0  -> 2.18.0
   arpack             3.1.5   -> 3.5.0
   cython             0.29.19 -> 0.29.21 
   flint              2.5.2   -> 2.6.0
   fplll              5.3.2   -> 5.3.3
   gp2c               0.11.2  -> 0.11.3
   ipython            7.5.0   -> 7.16.1
   matplotlib         3.1.2   -> 3.2.2
   numpy              1.17.4  -> 1.19.1
   pandas             1.0.1   -> 1.0.5
   pandoc             2.10    -> 2.10.1
   pillow             7.0.0   -> 7.1.2
   python             3.7.7   -> 3.7.8
   scipy              1.4.1   -> 1.5.2 
   sqlite             3.31.1  -> 3.32.3

no changes to tox.ini is required

[sheerluck]

comment:94

Good news, everyone!
We used one latest image, now we have two images:

• latest with python-3.8.5
• latest-py37 with python-3.7.8

checking ... whether /usr/bin/python3 is good... no, Python 3.8.5 is too recent

no changes to tox.ini is required, unless we want to be prepared for #29023 when most of installed packages would be used, in that case we can have both "gentoo" and "gentoo-py37" in tox.ini

[mkoeppe]

comment:95

Great! You may want to test with #27754, which aims for Python 3.8 compatibility

[kiwifb]

comment:96

For the record sage-on-gentoo enables building with python 3.8 since 9.2.beta7. Building of the documentation with it may currently be failing though (but I managed a build between beta6 and beta7 somehow).

[sheerluck]

comment:97

Sage is ready for gcc-10, so...

What’s New in sheerluck/sage-on-gentoo-stage4:20.9

1) base image:
   FROM gentoo/portage:20200901 as portage
   FROM gentoo/stage3-amd64:20200901

2) gcc-10 + python-3.8.5

3) updates:
   arb                2.18.0  -> 2.18.1
   flint              2.6.0   -> 2.6.3
   glibc              2.30-r8 -> 2.31-r6
   libhomfly          1.02.5  -> 1.02.6
   matplotlib         3.2.2   -> 3.3.1
   mpfr               4.0.2   -> 4.1.0
   sphinx             3.1.2   -> 3.2.1

no changes to tox.ini are required

[sheerluck]

comment:98

What’s New in sheerluck/sage-on-gentoo-stage4:20.10

1) base image:
   FROM gentoo/portage:20201001 as portage
   FROM gentoo/stage3:20201001

2) gcc-10 + python-3.8.6

3) updates:
   arpack-3.7.0       new
   tox-3.19.0         new
   brial              1.2.8   -> 1.2.9
   gmpy-2.1.0         b1-r1   -> b5
   guile              2.2.4   -> 2.2.6
   ipython            7.16.1  -> 7.17.0
   sqlite             3.32.3  -> 3.33.0
   swig               3.0.12  -> 4.0.0

sheerluck/sage-on-gentoo-stage4:20.10-py37 is the same, only

2) gcc-9 + python-3.7.9

[kiwifb]

comment:99

Please mask brial-1.2.9 there are issues with it and it will be replaced by 1.2.10 in short order.

[sheerluck]

comment:100

masked, downgraded, squashed, tested, pushed

[sheerluck]

comment:101

checking for python3 >= 3.6.0, < 3.10.0 with modules ...  
checking ... whether /usr/bin/python3 is good... yes

Although overlay cschwan/sage-on-gentoo is not ready for python-3.9 yet,
develop branch of sagemath/sage accepts python-3.9.

So as if two images weren't enough, I used some ugly hacks to create third image -- latest-py39

If someone finds it usefull to play with python-3.9, tox.ini may be changed this way:

--- a/tox.ini
+++ b/tox.ini
@@ -208,6 +208,7 @@ setenv =
     gentoo:      SYSTEM=gentoo
     gentoo:      BASE_IMAGE=sheerluck/sage-on-gentoo-stage4
     gentoo-python3.7: BASE_TAG=latest-py37
+    gentoo-python3.9: BASE_TAG=latest-py39
     #    
     # https://hub.docker.com/_/archlinux/
     #    

[mkoeppe]

comment:102

I have created #30785 for this

[sheerluck]

comment:103

I was under impression that something is not right with my images until I saw #31395
That bug has ruined monthful of latest https://github.com/mkoeppe/sage/actions/ :(

[orlitzky]

comment:104

Replying to @sheerluck:

I was under impression that something is not right with my images until I saw #31395
That bug has ruined monthful of latest https://github.com/mkoeppe/sage/actions/ :(

I'm going to post a branch that removes the memlimit and then turn off my computer for a month while people yell at me.

[dimpase]

comment:105

Replying to @orlitzky:

Replying to @sheerluck:

I was under impression that something is not right with my images until I saw #31395
That bug has ruined monthful of latest https://github.com/mkoeppe/sage/actions/ :(

I'm going to post a branch that removes the memlimit and then turn off my computer for a month while people yell at me.

see, #31395 got in :-)

[sheerluck]

comment:106

latest image contains clang-12.0.1 for tox -e docker-gentoo-maximal

[sheerluck]

comment:107

At https://github.com/orlitzky/sage/runs/3851208039?check_suite_focus=true#step:7:11493 I see Pytest is not installed, skip checking tests that rely on it.

Well, that is not true, latest image got /usr/bin/py.test and /usr/bin/pytest

[dimpase]

comment:108

you need pytest in sage's venv.

e.g.

make pytest

will do the trick