Upgrade Sage's MPIR spkg to version 2.1.3

[williamstein]

Hi,

OK, after all this, the build finally completed.  The only changes I made were:

* Updated mpir to rc3
* patched ecm as explained here:
    http://lists.gforge.inria.fr/pipermail/ecm-discuss/2009-August/004070.html
   (Though this had to be slightly modified -- just search for the same command, which moved.)


I then ran the long Sage test suite, and some tests fail.  

  http://sage.math.washington.edu/home/wstein/build/mpir2/sage-4.3.5/testlong.log

----------------------------------------------------------------------
The following tests failed:


        sage -t  -long "devel/sage/doc/en/tutorial/tour_numtheory.rst"
        sage -t  -long "devel/sage/doc/fr/tutorial/tour_numtheory.rst"
        sage -t  -long "devel/sage/sage/modular/cusps.py"
        sage -t  -long "devel/sage/sage/modular/modsym/boundary.py"
        sage -t  -long "devel/sage/sage/modular/modsym/ambient.py"
        sage -t  -long "devel/sage/sage/libs/pari/gen.pyx"
        sage -t  -long "devel/sage/sage/rings/arith.py"
        sage -t  -long "devel/sage/sage/rings/integer.pyx"
        sage -t  -long "devel/sage/sage/rings/polynomial/multi_polynomial_ideal.py"
        sage -t  -long "devel/sage/sage/tests/book_stein_ent.py"
        sage -t  -long "devel/sage/sage/schemes/elliptic_curves/heegner.py"
Total time for all tests: 7170.6 seconds

--

I looked and it appears that maybe all of these are the result of the XGCD behavior in MPIR changing again.  Has it changed to be like GMP now?  That would be convenient.

william

---

New spkg: http://sage.math.washington.edu/home/leif/Sage/spkgs/mpir-2.1.3.p4.spkg

md5sum: 197c652a6bdbf20c2e47be9176a8616e mpir-2.1.3.p4.spkg

Apply attachment: trac_8664-fix_extmod_deps_for_MPIR_generically-sagelib-rebased_to_4.7.2.alpha2.patch to the Sage library (after installation of the spkgs and a rebuild of dependent packages).

---

See comment(s) below on how to "manually" install and test the new packages. (ECM has to be updated, too, package is at #5847.)

---

#9522 can be closed as invalid / duplicate once this ticket has been merged.

Depends on #5847
Depends on #9858
Depends on #11896

CC: @williamstein @nexttime @sagetrac-drkirkby @jdemeyer @wbhart @jpflori

Component: packages: standard

Keywords: sd32, GMP ECM execstack Fedora 14 extension module library dependencies Darwin 11 MacOS X 10.7 Lion

Author: Mike Hansen, Leif Leonhardy

Reviewer: Leif Leonhardy, Dmitrii Pasechnik

Merged: sage-4.8.alpha0

Issue created by migration from https://trac.sagemath.org/ticket/8664

[sagetrac-GeorgSWeber]

comment:1

The XGCD issue needs some work w.r.t. doctests (but not only, see below), maybe David Harvey has something done already in this direction when preparing the optional "true" GMP spkg he maintains. (I don't find the thread right now, but I seem to remember he said something in this direction).

Snippet from a private communication by Bill Hart (MPIR upstream):

MPIR 2.0.1 will be required for Sage because it brings the gcdext
normalisation in line with GMP's (which is what Pari needs). I think
people have skipped MPIR 1.3.0 in favour of waiting for MPIR 2.0.1.
We're in the release phase of MPIR 2.0.1 ...

Also note that as soon as this ticket here is closed, we should mark #8455 as invalid.

[mwhansen]

comment:2

This needs to be handled with #5847 and should take care of #9522 at the same time.

[mwhansen]

comment:4

I've put an spkg at http://sage.math.washington.edu/home/mhansen/mpir-2.1.1.spkg

Interestingly, I did not get any of the failures listed above.

[mwhansen]

Author: Mike Hansen

[nexttime]

comment:7

Hmmm, spkg-install could be cleaned up further, e.g.

• Dave's comment has somehow been truncated(?) a while ago ("... if" what?)
• s/`uname`/$UNAME/
• Especially for parallel builds, it's more convenient to begin every error message with "Error" (or at least the message should contain that word).
• Simplify the "fat binary" section for Linux; it doesn't make sense to call uname -m three times, nor does it to compare its result again if it already matched one of the tested. A case-esac would be appropriate, also merging the identical i386 and i686 then-branches. Btw, uname -m can also be i486 or i586 on 32-bit Linuces.
• Is the ! [ $? -eq 0 ] some required Cygwin "idiom"?

Also, spkg-check doesn't print any messages, and uses make instead of $MAKE.

The dependencies ("None") in SPKG.txt are wrong; MPIR at least depends on iconv, which Sage ships, though Sage's version is only used/built on some systems. I'm not sure if readline should be added to spkg/standard/deps. (MPIR's configure looks for much more, but apparently - as currently shipped - doesn't need/use all of that.) The Changelog could perhaps mention further changes, too.

If nobody else wants to, I could make the changes, i.e. a reviewer patch. Just let me know...

Going to test the new spkg with Sage 4.5.3.alpha2 and 4.6.prealpha3 (on Linuces only).

[nexttime]

comment:8

Dave, as it seems you're addicted to testing, you could also test this new spkg (e.g. on Solaris, OpenSolaris)... ;-)

[nexttime]

comment:9

I guess unless we bump the patch levels of all packages that depend on GMP/MPIR (and ECM), this package breaks upgrading Sage (at least with the current upgrade process), due to interface change (in conjunction with shared library versioning).

The same applies to testing this package (and/with ECM 6.3 from #5847); one has to either build from scratch or delete spkg/installed/<package-name> for all packages that depend on those two.

Perhaps one could simply (conditionally) do the latter in spkg/install, which also gets updated during upgrade.

[nexttime]

comment:10

Defining SAGE_SPKG to sage-spkg -f in spkg/standard/deps should in principle also work, but doesn't... apparently because not really all shared libraries that use libgmpxx.so get rebuilt, so some still refer to the old, deleted libgmpxx.so.3.

---

Rebuilding Sage 4.5.3.alpha2 with MPIR 2.1.1 and ECM 6.3 from scratch worked on Ubuntu 10.04 x86_64 (parallel build with 32 jobs); ptestlong passed all tests.

[nexttime]

comment:11

Replying to @nexttime:

Defining SAGE_SPKG to sage-spkg -f in spkg/standard/deps should in principle also work, but doesn't... apparently because not really all shared libraries that use libgmpxx.so get rebuilt, so some still refer to the old, deleted libgmpxx.so.3.

I think this could be fixed by making extension modules also depend on (some of) the libraries they use (in module_list.py), with the disadvantage that these modules would get rebuilt whenever their libraries "change", even if the interfaces stay the same (and even if just the modification time changed due to a rebuild/reinstallation).

---

Rebuilding Sage 4.5.3.alpha2 with MPIR 2.1.1 and ECM 6.3 from scratch worked on Ubuntu 10.04 x86_64 (parallel build with 32 jobs); ptestlong passed all tests.

Same for Fedora 13 x86 (parallel build with 6 jobs), also ptestlong.

---

Unfortunately, there seems to be some incompatibility between the new PARI (2.4.3.svn-12577.p4) or - more precisely - the required changes to Sage's interface to PARI in order to upgrade PARI (Sage 4.6.prealpha3, see #9343 and the NewPARI Wiki page) and MPIR 2.1.1:

On Ubuntu 10.04 x86_64 (Core2, gcc 4.4.3) exactly two of all doctests (ptestlong) segfault (in sage/schemes/elliptic_curves/ell_point.py).

On Fedora 13 x86 (Pentium 4 Prescott, gcc 4.4.4) the same two doctests and another one in ell_rational_field.py segfault, and in addition one doctest in the latter file raises an unexpected exception (TypeError), which seems to be the result of some other failure.

Maybe all are due to memory (or stack) corruption. See #9343 comment:327 ff. for details; we haven't yet opened a separate ticket.

(For testing with Sage 4.6.prealpha3, I copied the MPIR 2.1.1 and ECM 6.3 spkgs to spkg/standard, and built Sage from scratch - without problems. I also reinstalled the PARI, MPIR and ECM packages later with env SAGE_CHECK=yes ./sage -f ...; all test suites passed.)

[nexttime]

comment:12

Replying to @nexttime:

Unfortunately, there seems to be some incompatibility between the new PARI (2.4.3.svn-12577.p4 / Sage 4.6.prealpha3, see #9343 and the NewPARI Wiki page) and MPIR 2.1.1:

On Ubuntu 10.04 x86_64 (Core2, gcc 4.4.3) exactly two of all doctests (ptestlong) segfault (in sage/schemes/elliptic_curves/ell_point.py).

On Fedora 13 x86 (Pentium 4 Prescott, gcc 4.4.4) the same two doctests and another one in ell_rational_field.py segfault, and in addition one doctest in the latter file raises an unexpected exception (TypeError), which seems to be the result of some other failure.

This is due to the new PARI using undocumented features of GMP (that MPIR doesn't support). and is now addressed at #9837.

[nexttime]

comment:13

Replying to @nexttime:

This is due to the new PARI using undocumented features of GMP (that MPIR doesn't support). and is now addressed at #9837.

This is not necessarily the (only) cause. Jeroen has found a definite bug in MPIR 2.1.1 he'll report upstream; see #9837.

Leaving the ticket "needs review", though I don't think we should upgrade MPIR until that bug is fixed, in a later release.

[mwhansen]

comment:14

Replying to @nexttime:

Leaving the ticket "needs review", though I don't think we should upgrade MPIR until that bug is fixed, in a later release.

Sounds good to me.

[jdemeyer]

comment:15

The problem of ticket #9837 has been fixed in MPIR svn, it was purely a bug in MPIR 2.1.1.

[jdemeyer]

comment:16

Please upgrade to MPIR 2.1.2, where #9837 is fixed.

[nexttime]

comment:18

Apparently they've released MPIR 2.1.2 with just this bug fixed today.

Haven't tested that yet.

[nexttime]

comment:19

Replying to @nexttime:

Apparently they've released MPIR 2.1.2 with just this bug fixed today.

Haven't tested that yet.

At least passes all (long) tests in sage/schemes/elliptic_curves/ with Sage 4.6.prealpha3 on Fedora 13 x86 (Pentium 4 Prescott, gcc 4.4.4), where previously the segfaults occurred.

The diff shows they just fixed that single bug, bumped the MPIR version number (patch level) and incremented the revision numbers of both shared libraries. (In addition, they changed the date of the documentation without actually changing the docs themselves. The file src/.gdbinit in the MPIR 2.1.1 spkg must have been added by Mike or someone else.)

[nexttime]

comment:127

Had to fix the changelog and some file permissions.

Corrected spkg at new location.

[strogdon]

comment:128

I'm not sure if this is the best place to report this since the ticket has been merged, but in building vanilla sage-4.7.2.alpha3 on x86, mpir-2.1.3.p4 fails with

 ../strip_fPIC.sh ../yasm/yasm -I .. -f elf64 submul_1.as -o submul_1.o >/dev/null 2>&1
/bin/bash ../libtool --mode=c
 ../mpn/m4-ccas --m4=m4 gcc -std=gnu99 -c -DHAVE_CONFIG_H -m32 -O2 -fomit-frame-pointer -mtune=core2 -march=core2 -D__GMP_WITHIN_GMP -I.. -DOPERATION_lshift -I. -I.. lshift.asm  -fPIC -DPIC -o .libs/lshift.o
m4  -DHAVE_CONFIG_H -D__GMP_WITHIN_GMP -DOPERATION_lshift -DPIC lshift.asm >tmp-lshift.s
 gcc -std=gnu99 -c -DHAVE_CONFIG_H -m32 -O2 -fomit-frame-pointer -mtune=core2 -march=core2 -D__GMP_WITHIN_GMP -I.. -DOPERATION_lshift -I. -I.. tmp-lshift.s -fPIC -DPIC -o .libs/lshift.o
tmp-lshift.s: Assembler messages:
tmp-lshift.s:53: Error: bad register name `%rbx'
tmp-lshift.s:54: Error: bad register name `%rdx'
tmp-lshift.s:55: Error: bad register name `%rsi)'
tmp-lshift.s:56: Error: bad register name `%rdi)'
tmp-lshift.s:57: Error: bad register name `%rsi,%rbx,8)'
tmp-lshift.s:58: Error: bad register name `%rax'
tmp-lshift.s:59: Error: bad register name `%rdx'
tmp-lshift.s:60: Error: bad register name `%rbx'
tmp-lshift.s:64: Error: bad register name `%rsi,%rbx,8)'
tmp-lshift.s:65: Error: bad register name `%rsi,%rbx,8)'
tmp-lshift.s:66: Error: bad register name `%rsi,%rbx,8)'
tmp-lshift.s:67: Error: bad register name `%r8'
tmp-lshift.s:68: Error: bad register name `%rdx'
tmp-lshift.s:69: Error: bad register name `%r11'
tmp-lshift.s:70: Error: bad register name `%rsi,%rbx,8)'

...

tmp-lshift.s:118: Error: bad register name `%rsi,%rbx,8)'
tmp-lshift.s:119: Error: bad register name `%r8'
tmp-lshift.s:120: Error: bad register name `%rdx'
tmp-lshift.s:121: Error: bad register name `%r8'
tmp-lshift.s:122: Error: bad register name `%r8'
tmp-lshift.s:123: Error: bad register name `%rbx'
make[4]: *** [lshift.lo] Error 1
make[4]: Leaving directory `/storage/sage/sage-4.7.2.alpha3/spkg/build/mpir-2.1.3.p4/src/mpn'
make[3]: *** [all-recursive] Error 1
make[3]: Leaving directory `/storage/sage/sage-4.7.2.alpha3/spkg/build/mpir-2.1.3.p4/src'
make[2]: *** [all] Error 2
make[2]: Leaving directory `/storage/sage/sage-4.7.2.alpha3/spkg/build/mpir-2.1.3.p4/src'
Error building MPIR.

I've never seen this before. The machine is running debian, but the sage build was done from within a gentoo prefix. So the build is basically one using gentoo. There doesn't seem to be a problem on amd64. I still have the build log.

[nexttime]

comment:129

Replying to @strogdon:

I'm not sure if this is the best place to report this since the ticket has been merged, but in building vanilla sage-4.7.2.alpha3 on x86, mpir-2.1.3.p4 fails with

tmp-lshift.s: Assembler messages:
tmp-lshift.s:53: Error: bad register name `%rbx'
...
make[4]: *** [lshift.lo] Error 1
...

I've never seen this before. The machine is running debian, but the sage build was done from within a gentoo prefix. So the build is basically one using gentoo. There doesn't seem to be a problem on amd64. I still have the build log.

See sage-release and this mpir-devel thread.

I'm sorry I didn't catch this earlier, but it's a waste of resources to run 32-bit operating systems (or software in general) on 64-bit processors... ;-)

I'll provide follow-up spkgs (also at #11616) that will fix this without the need to set ABI, but perhaps not that very soon.

By the way, MPIR 2.5.0 should get released in the next days as well, so I'll presumably also make an spkg based on that.

[nexttime]

comment:130

Replying to @nexttime:

Replying to @strogdon:

I'm not sure if this is the best place to report this since the ticket has been merged, but in building vanilla sage-4.7.2.alpha3 on x86, mpir-2.1.3.p4 fails with

tmp-lshift.s: Assembler messages:
tmp-lshift.s:53: Error: bad register name `%rbx'
...
make[4]: *** [lshift.lo] Error 1
...

I'll provide follow-up spkgs (also at #11616) that will fix this without the need to set ABI, but perhaps not that very soon.

An MPIR 2.1.3.p5 spkg fixing this is now available at #11896, currently needing review.

[jdemeyer]

Changed merged from sage-4.7.2.alpha3 to none

[jdemeyer]

Changed dependencies from #5847 to #5847, #9858, #11896

[jdemeyer]

comment:131

Unmerging this because of #9858 and #11896.

[nexttime]

comment:134

Another cyclic dependency... ;-) (#11698 depends on this ticket)

---

A new FLINT spkg (still 1.5.0, a p10) fixing the FLINT test suite build error (occurring in combination with any recent GMP / MPIR version like the one here) by just applying an upstream patch from FLINT 1.5.2 is ready for review at #9858.

(FLINT 1.5.2 and 1.6 spkgs will now most probably be on follow-up tickets to #9858.)

[nexttime]

comment:135

Replying to @nexttime:

Another cyclic dependency... ;-) (#11698 depends on this ticket)

Ooops, #11896 of course. XD

(But I'll try to review #11698 anyway.)

[jdemeyer]

Merged: sage-4.7.3.alpha0

[jdemeyer]

Milestone sage-4.7.3 deleted

[jdemeyer]

Changed merged from sage-4.7.3.alpha0 to sage-4.8.alpha0

[kcrisman]

comment:139

Replying to @nexttime:

Any reason to keep

    SAGE_CONF_OPTS="--enable-shared --disable-static"

(i.e., why not build the static library, too)?

As it turns out, this change seems to break MPIR on Cygwin. See this sage-windows thread.