sagemathgh-36264: sage --fixdoctests --update-known-test-failures; …

…silence modularized distributions in CI

    
<!-- ^^^^^
Please provide a concise, informative and self-explanatory title.
Don't put issue numbers in there, do this in the PR body below.
For example, instead of "Fixes sagemath#1234" use "Introduce new method to
calculate 1+1"
-->
<!-- Describe your changes here in detail -->

The distributions `sagemath-repl` and `sagemath-categories` run parts of
the testsuite in a virtual environment created by `tox` when
SAGE_CHECK=yes. This also runs as part of the Build & Test CI.

Here we make several improvements:
- Support SAGE_CHECK=warn
- Invoke the doctester with `--baseline-stats-path={toxinidir}/known-
test-failures.json`
- Make the tests silent unless new test failures, not recorded in the
`known-test-failures.json`, are encountered

To help maintain the `known-test-failures.json`, we also add new
features to the command `sage --fixdoctests`.
- `sage --fixdoctests --update-known-test-failures` reads the stats
files generated by the doctester in the virtual environments and writes
updated `known-test-failures.json` files to the source tree
- doctester stats now also include a field `ntests` - number of doctests
of a module that were run

`sage --fixdoctests` also receives new switches `--distribution all`,
`--fixed-point`, `--verbose`, `--no-diff` and some other improvements.

<!-- Why is this change required? What problem does it solve? -->
<!-- If this PR resolves an open issue, please link to it here. For
example "Fixes sagemath#12345". -->
- Part of: sagemath#29705
- Cherry-picked from: sagemath#35095
<!-- If your change requires a documentation PR, please link it
appropriately. -->

### 📝 Checklist

<!-- Put an `x` in all the boxes that apply. -->
<!-- If your change requires a documentation PR, please link it
appropriately -->
<!-- If you're unsure about any of these, don't hesitate to ask. We're
here to help! -->
<!-- Feel free to remove irrelevant items. -->

- [x] The title is concise, informative, and self-explanatory.
- [x] The description explains in detail what this PR is about.
- [x] I have linked a relevant issue or discussion.
- [ ] I have created tests covering the changes.
- [ ] I have updated the documentation accordingly.

### ⌛ Dependencies

<!-- List all open PRs that this PR logically depends on
- sagemath#12345: short description why this is a dependency
- sagemath#34567: ...
-->
- Depends on sagemath#36263 (merged here)
- Depends on sagemath#36238 (merged here)

<!-- If you're unsure about any of these, don't hesitate to ask. We're
here to help! -->
    
URL: sagemath#36264
Reported by: Matthias Köppe
Reviewer(s): Kwankyu Lee, Matthias Köppe

build/pkgs/sagemath_objects/spkg-install

@@ -21,9 +21,28 @@ export PIP_FIND_LINKS="file://$SAGE_SPKG_WHEELS"
 DIST_DIR="$(mktemp -d)"
 python3 -m build --outdir "$DIST_DIR"/dist . || sdh_die "Failure building sdist and wheel"
 
-wheel=$(cd "$DIST_DIR" && sdh_store_wheel . && echo $wheel)
+wheel=$(cd "$DIST_DIR" && sdh_store_wheel . >&2 && echo $wheel)
 ls -l "$wheel"
 
 if [ "$SAGE_CHECK" != no ]; then
-    tox -r -v -e sagepython-sagewheels-nopypi-norequirements --installpkg $wheel
+    export TOX_PARALLEL_NO_SPINNER=1
+    echo Running "tox -r -p auto -v --installpkg $wheel"
+    tox -r -p auto -v --installpkg $wheel
+    status=$?
+    case $status:$SAGE_CHECK:$([ -r known-test-failures.json ]; echo $?) in
+        0:*:0)    echo "Passed the test suite (modulo baseline known-test-failures*.json)";;
+        0:*:*)    echo "Passed the test suite";;
+        *:warn:0) echo "Warning: New failures (not in baseline known-test-failures*.json (ignored)"; status=0;;
+        *:warn:*) echo "Warning: Failures testing the package (ignored)"; status=0;;
+        *:yes:0)  echo "New failures, not in baseline known-test-failures*.json";;
+        *:yes:*)  echo "Failures testing the package";;
+    esac
+    # Show summaries of failures (suppress lines ending with '[failed in baseline]')
+    for f in $(pwd)/.tox/sagepython-sagewheels-nopypi-norequirements*/log/*-command*.log; do
+        if [ -r "$f" ]; then
+            echo "$f"
+            grep '^sage -t.*#[^]]*$' "$f"
+        fi
+    done
+    exit $status
 fi