doctests: Detect "safe" external features even if "--optional=external" is not used #32174

mkoeppe · 2021-07-10T18:34:16Z

For some "safe" features such as imagemagick, rubiks, 4ti2 (#30887), pandoc, it really shouldn't require the use of external to check for these.

Likewise for the PythonModule features for Sage modules introduced in #32614. By delaying the test for these features, using the sage.doctest.external machinery, we fix the broken detection of some of these features. (This is why the ticket is marked "critical".)

This is a step toward #30746 (sage.doctest.control: Replace use of sage.misc.package.list_packages)

With this ticket, more doctests will be run by default. This may lead to new failures (discovered by patchbot for instance). But that does not mean it is introduced in this ticket, just that they were existing before this ticket, but simply not executed.

For example, let's compare the behavior of the following command:

sage -tp --show-skipped src/sage/tests/cmdline.py

returns:

BEFORE (9.5.beta7):

Git branch: develop
Using --optional=4ti2,build,ccache,cryptominisat,debian,dochtml,dot2tex,e_antic,fricas,glucose,latte_int,lidia,normaliz,notedown,pandoc_attributes,pip,pycosat,pynormaliz,rst2ipynb,sage,sage.geometry.polyhedron,sage.rings.real_double,sage_numerical_backends_coin,sage_spkg
Doctesting 1 file using 8 threads.
sage -t --random-seed=265206128165572261397002909756599539048 src/sage/tests/cmdline.py
    3 gdb tests not run
    12 internet tests not run
    2 pandoc tests not run
    4 r tests not run
    0 tests not run because we ran out of time
    [235 tests, 30.65 s]
----------------------------------------------------------------------
All tests passed!
----------------------------------------------------------------------

AFTER (9.5.beta7 + #32174):

$ sage -tp --show-skipped src/sage/tests/cmdline.py 
Git branch: 32174
Using --optional=4ti2,build,ccache,cryptominisat,debian,dochtml,dot2tex,e_antic,fricas,glucose,latte_int,lidia,normaliz,notedown,pandoc_attributes,pip,pycosat,pynormaliz,rst2ipynb,sage,sage_numerical_backends_coin,sage_spkg
Features to be detected: 4ti2,benzene,bliss,buckygen,conway_polynomials,csdp,database_cremona_ellcurve,database_cremona_mini_ellcurve,database_jones_numfield,database_knotinfo,dvipng,ffmpeg,graphviz,imagemagick,jupymake,kenzo,latte_int,lrslib,mcqd,meataxe,pandoc,pdf2svg,plantri,pynormaliz,rubiks,sage.combinat,sage.geometry.polyhedron,sage.graphs,sage.plot,sage.rings.number_field,sage.rings.real_double,sage.symbolic,sage_numerical_backends_coin,tdlib
Doctesting 1 file using 8 threads.
sage -t --random-seed=100100306103974678574881156032164018710 src/sage/tests/cmdline.py
**********************************************************************
File "src/sage/tests/cmdline.py", line 732, in sage.tests.cmdline.test_executable
Failed example:
    print(open(output, 'r').read() == t)          # optional - pandoc
Expected:
    True
Got:
    False
**********************************************************************
1 item had failures:
   1 of 238 in sage.tests.cmdline.test_executable
    3 gdb tests not run
    12 internet tests not run
    4 r tests not run
    0 tests not run because we ran out of time
    [237 tests, 1 failure, 32.76 s]
----------------------------------------------------------------------
sage -t --random-seed=100100306103974678574881156032164018710 src/sage/tests/cmdline.py  # 1 doctest failed
----------------------------------------------------------------------
Features detected for doctesting: pandoc

We observe that the number of tests run went from 235 to 237, because the feature pandoc was detected. Note that the doctest failure is already tracked at #32697.

Depends on #32866
Depends on #27744

CC: @dimpase @slel @antonio-rojas @kiwifb @kwankyu @seblabbe @videlec

Component: doctest framework

Author: Matthias Koeppe, Sébastien Labbé

Branch: 41c4cd4

Reviewer: Sébastien Labbé

Issue created by migration from https://trac.sagemath.org/ticket/32174

The text was updated successfully, but these errors were encountered:

mkoeppe · 2021-10-07T18:34:37Z

Dependencies: #32649

mkoeppe · 2021-10-07T23:40:20Z

Changed dependencies from #32649 to #32649, #32614

mkoeppe · 2021-10-22T20:06:53Z

comment:5

Let's use this ticket to address the problem discovered in #32732 comment:10: Detection of Sage modules fails because the check runs before sage.all has been imported, and is therefore affected by import failures from cyclic imports

mkoeppe · 2021-10-22T20:59:21Z

Changed dependencies from #32649, #32614 to #32732

mkoeppe · 2021-11-08T05:38:45Z

Changed dependencies from #32732 to #32732, #32650

mkoeppe · 2021-11-11T02:06:13Z

Branch: u/mkoeppe/doctests__detect__safe__external_features_even_if____optional_external__is_not_used

mkoeppe · 2021-11-11T02:07:40Z

comment:9

Here's an approach to do feature enumeration

Last 10 new commits:

`5d9414d`	`32650: .. note -> .. NOTE`
`345c760`	`32650: using features in has_latex method of doctest/external.py`
`6c6b332`	`32650: independent and improved method latex().is_functional()`
`63555d2`	`32650: adding spkg pdf2svg in build/pkgs`
`c8330fe`	`32650: linking pdf2svg feature to spkg pdf2svg`
`2ea4d9e`	`32650: fixing two doctests now that we latex().require() it when we use it`
`98b7db6`	`Merge #32650`
`0968f95`	`sage.features.sagemath.all_features: New`
`cdc0cb6`	`sage.features.all: New`
`c8e6c0e`	`sage.features.bliss.all_features: New`

mkoeppe · 2021-11-11T02:07:40Z

Commit: c8e6c0e

sagetrac-git · 2021-11-11T03:10:55Z

Changed commit from c8e6c0e to 69d2cfb

sagetrac-git · 2021-11-11T03:10:55Z

Branch pushed to git repo; I updated commit sha1. New commits:

`69d2cfb`	`src/sage/features/kenzo.py: Move import from sage.libs.ecl into method`

mkoeppe · 2021-11-11T04:49:00Z

Author: Matthias Koeppe, ...

slel · 2021-11-13T13:03:31Z

comment:12

Indent .. NOTE:: blocks 4 spaces, not 3
(can be a follow-up ticket).

sagetrac-git · 2021-11-13T18:18:28Z

Changed commit from 69d2cfb to 893ec23

sagetrac-git · 2021-11-13T18:18:28Z

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

`fa45c36`	`sage.features.sagemath.all_features: New`
`17e30e2`	`sage.features.all: New`
`8bfebc9`	`sage.features.bliss.all_features: New`
`893ec23`	`src/sage/features/kenzo.py: Move import from sage.libs.ecl into method`

mkoeppe · 2021-11-13T19:50:21Z

Changed dependencies from #32732, #32650 to #32866

mkoeppe · 2021-11-13T19:50:21Z

comment:14

Rebased on 9.5.beta6

sagetrac-git · 2021-11-14T22:04:47Z

Changed commit from 893ec23 to fa25155

sagetrac-git · 2021-11-14T22:04:47Z

Branch pushed to git repo; I updated commit sha1. New commits:

`eaec400`	`src/sage/features/interfaces.py: New`
`827ca03`	`src/sage/features/interfaces.py: Fix and doctest exception handling`
`52ac51e`	`src/sage/features/interfaces.py (all_features): New`
`7754fae`	`src/sage/features/interfaces.py: Fix and doctest exception handling (fixup)`
`8081f29`	`src/sage/doctest/external.py: Use sage.features.interfaces`
`09c4238`	`Merge #32866`
`c41808b`	`src/sage/features/latex.py (all_features): New`
`44c19e3`	`src/sage/features/mip_backends.py (all_features): New`
`fa25155`	`src/sage/doctest/external.py (external_software): Go through external_features`

sagetrac-git · 2021-11-14T22:25:07Z

Branch pushed to git repo; I updated commit sha1. New commits:

`945132b`	`src/sage/features/interfaces.py: Fix up`
`f63acfd`	`Merge #32866`
`7fa5b3b`	`src/sage/doctest/external.py (external_features): Add CPLEX, Gurobi`

sagetrac-git · 2021-11-14T22:25:07Z

Changed commit from fa25155 to 7fa5b3b

mkoeppe · 2021-11-14T22:27:53Z

comment:17

The has_... functions are now no longer needed.

Deprecate them? Or just remove them?

mkoeppe · 2021-12-05T17:21:59Z

comment:81

that's #32697 "sage --ipynb2rst is broken due to changes in nbconvert

sagetrac-git · 2021-12-05T18:06:17Z

Changed commit from 3bfe1a1 to fdfe1a7

sagetrac-git · 2021-12-05T18:06:17Z

Branch pushed to git repo; I updated commit sha1. New commits:

`fdfe1a7`	`src/sage/tests/cmdline.py: Mark a doctest # known bug #32697`

mkoeppe · 2021-12-05T20:32:01Z

comment:85

Thanks!

vbraun · 2021-12-16T23:48:26Z

comment:86

**********************************************************************
File "src/sage/misc/latex.py", line 1087, in sage.misc.latex.Latex.?
Failed example:
    latex.eval(r"\ThisIsAnInvalidCommand", {}) # optional -- ImageMagick
Exception raised:
    Traceback (most recent call last):
      File "/var/lib/buildbot/slave/sage_git/build/local/var/lib/sage/venv-python3.9.9/lib/python3.9/site-packages/sage/doctest/forker.py", line 694, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/var/lib/buildbot/slave/sage_git/build/local/var/lib/sage/venv-python3.9.9/lib/python3.9/site-packages/sage/doctest/forker.py", line 1088, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.misc.latex.Latex.?[1]>", line 1, in <module>
        latex.eval(r"\ThisIsAnInvalidCommand", {}) # optional -- ImageMagick
      File "/var/lib/buildbot/slave/sage_git/build/local/var/lib/sage/venv-python3.9.9/lib/python3.9/site-packages/sage/misc/latex.py", line 1128, in eval
        e = _run_latex_(os.path.join(base, filename + ".tex"), debug=debug,
      File "/var/lib/buildbot/slave/sage_git/build/local/var/lib/sage/venv-python3.9.9/lib/python3.9/site-packages/sage/misc/latex.py", line 725, in _run_latex_
        pdflatex().require()
      File "/var/lib/buildbot/slave/sage_git/build/local/var/lib/sage/venv-python3.9.9/lib/python3.9/site-packages/sage/features/__init__.py", line 206, in require
        raise FeatureNotPresentError(self, presence.reason, presence.resolution)
    sage.features.FeatureNotPresentError: pdflatex is not available.
    Executable 'pdflatex' not found on PATH.
    Further installation instructions might be available at https://www.latex-project.org/.
**********************************************************************
1 item had failures:
   1 of   3 in sage.misc.latex.Latex.?
    [250 tests, 1 failure, 0.35 s]
    Timed out
**********************************************************************

sagetrac-git · 2021-12-17T00:13:21Z

Branch pushed to git repo; I updated commit sha1. New commits:

`8f5c962`	`Merge tag '9.5.beta8' into t/32174/public/32174`
`41c4cd4`	`src/sage/misc/latex.py: Add another # optional - latex`

sagetrac-git · 2021-12-17T00:13:21Z

Changed commit from fdfe1a7 to 41c4cd4

vbraun · 2021-12-19T11:46:49Z

Changed branch from public/32174 to 41c4cd4

vbraun · 2021-12-19T11:49:22Z

Changed commit from 41c4cd4 to none

vbraun · 2021-12-19T11:49:22Z

comment:90

Followup at #33045 for ffmpeg

slel · 2022-01-01T19:26:43Z

comment:91

Followup at #33101 for lrslib.

slel · 2022-01-01T19:28:36Z

comment:92

See also #33092 for ffmpeg and imagemagick.

mkoeppe added this to the sage-9.4 milestone Jul 10, 2021

mkoeppe added c: doctest framework labels Jul 10, 2021

mkoeppe modified the milestones: sage-9.4, sage-9.5 Jul 19, 2021

This comment has been minimized.

Sign in to view

mkoeppe added p: critical / 2 and removed p: major / 3 labels Oct 22, 2021

vbraun added s: needs work and removed s: positive review labels Dec 5, 2021

mkoeppe added s: needs review and removed s: needs work labels Dec 5, 2021

seblabbe added s: positive review and removed s: needs review labels Dec 5, 2021

vbraun added s: needs work and removed s: positive review labels Dec 16, 2021

mkoeppe added s: positive review and removed s: needs work labels Dec 17, 2021

vbraun removed the s: positive review label Dec 19, 2021

vbraun closed this as completed in d5c49b9 Dec 19, 2021

jplab mentioned this issue Nov 18, 2021

Make pynormaliz a Feature #27744

Closed

tobiasdiez mentioned this issue Dec 19, 2021

Refactor to break cyclic import in complex numbers #30741

Closed

dcoudert mentioned this issue Mar 18, 2022

Update igraph to 0.9.5 and python_igraph to 0.9.8 #32510

Closed

This was referenced Jan 29, 2023

sage --ipynb2rst is broken due to changes in nbconvert #32697

Open

Add feature for pdftocairo #33005

Closed

mkoeppe added a commit to mkoeppe/sage that referenced this issue Feb 12, 2023

Merge sagemath#32174

d24e66d

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

doctests: Detect "safe" external features even if "--optional=external" is not used #32174

doctests: Detect "safe" external features even if "--optional=external" is not used #32174

mkoeppe commented Jul 10, 2021

This comment has been minimized.

mkoeppe commented Oct 7, 2021

mkoeppe commented Oct 7, 2021

mkoeppe commented Oct 22, 2021

mkoeppe commented Oct 22, 2021

mkoeppe commented Nov 8, 2021

mkoeppe commented Nov 11, 2021

mkoeppe commented Nov 11, 2021

mkoeppe commented Nov 11, 2021

sagetrac-git mannequin commented Nov 11, 2021

sagetrac-git mannequin commented Nov 11, 2021

mkoeppe commented Nov 11, 2021

slel commented Nov 13, 2021

sagetrac-git mannequin commented Nov 13, 2021

sagetrac-git mannequin commented Nov 13, 2021

mkoeppe commented Nov 13, 2021

mkoeppe commented Nov 13, 2021

sagetrac-git mannequin commented Nov 14, 2021

sagetrac-git mannequin commented Nov 14, 2021

sagetrac-git mannequin commented Nov 14, 2021

sagetrac-git mannequin commented Nov 14, 2021

mkoeppe commented Nov 14, 2021

mkoeppe commented Dec 5, 2021

sagetrac-git mannequin commented Dec 5, 2021

sagetrac-git mannequin commented Dec 5, 2021

mkoeppe commented Dec 5, 2021

vbraun commented Dec 16, 2021

sagetrac-git mannequin commented Dec 17, 2021

sagetrac-git mannequin commented Dec 17, 2021

vbraun commented Dec 19, 2021

vbraun commented Dec 19, 2021

vbraun commented Dec 19, 2021

slel commented Jan 1, 2022

slel commented Jan 1, 2022

doctests: Detect "safe" external features even if "--optional=external" is not used #32174

doctests: Detect "safe" external features even if "--optional=external" is not used #32174

Comments

mkoeppe commented Jul 10, 2021

This comment has been minimized.

mkoeppe commented Oct 7, 2021

mkoeppe commented Oct 7, 2021

mkoeppe commented Oct 22, 2021

mkoeppe commented Oct 22, 2021

mkoeppe commented Nov 8, 2021

mkoeppe commented Nov 11, 2021

mkoeppe commented Nov 11, 2021

mkoeppe commented Nov 11, 2021

sagetrac-git mannequin commented Nov 11, 2021

sagetrac-git mannequin commented Nov 11, 2021

mkoeppe commented Nov 11, 2021

slel commented Nov 13, 2021

sagetrac-git mannequin commented Nov 13, 2021

sagetrac-git mannequin commented Nov 13, 2021

mkoeppe commented Nov 13, 2021

mkoeppe commented Nov 13, 2021

sagetrac-git mannequin commented Nov 14, 2021

sagetrac-git mannequin commented Nov 14, 2021

sagetrac-git mannequin commented Nov 14, 2021

sagetrac-git mannequin commented Nov 14, 2021

mkoeppe commented Nov 14, 2021

mkoeppe commented Dec 5, 2021

sagetrac-git mannequin commented Dec 5, 2021

sagetrac-git mannequin commented Dec 5, 2021

mkoeppe commented Dec 5, 2021

vbraun commented Dec 16, 2021

sagetrac-git mannequin commented Dec 17, 2021

sagetrac-git mannequin commented Dec 17, 2021

vbraun commented Dec 19, 2021

vbraun commented Dec 19, 2021

vbraun commented Dec 19, 2021

slel commented Jan 1, 2022

slel commented Jan 1, 2022