Backporting docutils version pinning for 1.x, 2.x, and 3.x before 3.5.4

[agjohnson]

Hi Sphinx team!

I figured it would be best to open a separate issue to discuss solving the backwards incompatible changes from docutils 0.18 on older releases, separate from individual reports.

Sphinx maintainers used to maintain previous multiple releases with backports. Would the maintainers consider backporting version pinning for docutils for Sphinx 1.x and 2.x?

Based on numbers from PePy[1], Sphinx 1.x and 2.x release series still represent a fair amount of the install base. Comparing the latest in each major series, 1.x and 2.x are about 25% of installs still. This seems about accurate given the number of users requesting support around this issue on Read the Docs.

While encouraging users to upgrade to Sphinx =>3.0 and Python 3 is absolutely a great idea, unfortunately the docutils changes are forcing this upgrade very abruptly for users. Upgrading to Sphinx =>3.0 is a fairly minor upgrade, but upgrading to Python 3 to run Sphinx 3.x is a lot to force on to users without much warning.

For users that can't immediately upgrade, we're currently suggesting they manually pin docutils =<0.18. However, in order to get to that point, the user needs to hit the error from docutils 0.18, find one of the issues describing the backwards incompatibility, and then manually pin docutils. This is confusing users a lot, and it doesn't help that the error message from docutils is not helpful for users either.

If this was solved at the package level, users wouldn't have to be confused about the docutils change at all, though I understand it's annoying to maintain backport releases.

Would the maintainers consider backporting to a Sphinx 1.x and 2.x release, replicating the docutils pinning from Sphinx 3.x?

1: https://pepy.tech/project/sphinx?versions=4.2.0&versions=3.5.4&versions=2.4.4&versions=1.8.5

Refs: readthedocs/readthedocs.org#8616
Refs: #9783
Refs: #9777

[gmilde]

As these duplicates keep coming (e.g. #9811), maybe considering the long term support request for 1.x and 2.x may ease not only RTD users frustration but also the issue maintenance burden for Sphinx developers.

The alternative to fix the Python errors in Docutils 0.18.1 could lead to undedected problems in the generated documentation pages because of the remaining changes to the HTML output:

Docutils 0.17 and Docutils 0.18 contain incompatible changes that not only lead to the above errors but
also to problems with older CSS themes, including bullet lists without bullets. Cf. #9088
This means that pinning docutils < 0.18 may solve only a subset of the problems, docutils < 0.17 or even lower versions may be required (depending on your theme).

Both, Sphinx and 3rd party Documentation on pinning dependencies should make clear that pins to Sphinx versions below 3.5.4 must be amend with pins to Docutils.
Even with this feature request granted and pins in new long term support versions Sphinx 2.5 and Sphinx 1.9, this still holds for any == pins to any Sphinx version other than 1.9, 2.5, 3.5.4, and 4.x.

[agjohnson]

Great points! Agreed this seems like it should be solved at the packaging level. This doesn't just affect RTD users with old projects, it instead affects anyone using an old pinned version of Sphinx.

@tk0miya if there is anything more that RTD can do here, our team is happy to help more. At very least, the Sphinx team doesn't need to be triaging issues from RTD users with old dependencies. But I think we'd be happy to maintain backports as well.

[tk0miya]

@agjohnson Is it possible to pin docutils version for the projects that don't configure by new style on RTD side? I guess many users have still used Sphinx-1.8.5 because they don't migrate to the new style configuration. It means RTD can also pin the version of docutils for these projects.

[agjohnson]

We are talking about pinning docutils 0.16 for users that would get Sphinx 1.8.5. However, this doesn't solve the problem for a considerable number of Sphinx users, and only addresses usage on RTD, not local installations.

The reason some users still might get Sphinx 1.8.5 is:

• We install Sphinx by default for users during builds. We used to install a pinned version and the last pinned version we installed was 1.8.5
• When we stopped pinning Sphinx 1.8.5 for users, we didn't change this behavior for projects already created. Old projects will install Sphinx 1.8.5 by default, unless the project pins their own. We can safely pin docutils 0.16 for these users.

However, this is only users using 1.8.5 on RTD. There are still a large number of Sphinx users installing Sphinx < 3.5.4, !=1.8.5. This is 25-30% of Sphinx installations based on my math and data from:
https://pepy.tech/project/sphinx?versions=1.*&versions=2.*&versions=3.*&versions=4.*&versions=1.8.5&versions=3.5.4

My math is Sphinx versions prior to 3.5.4, excluding 1.8.5 (which we can affect automatically). Sphinx 1.8.5 is 7-8% of Sphinx installations, and 32-38% of Sphinx installations in total can hit this issue with docutils.

[tk0miya]

Error reports I saw are almost uses 1.8.5 and RTD. So I thought it's enough to fix it on RTD side. But no reason to pin it on Sphinx side. Okay, let's do it.

BTW, what version of docutils is good for pinning for each Sphinx version? I'm okay for either 0.16 and 0.17.

[astrojuanlu]

@gmilde also restored the backwards compatibility of nodes.Node.traverse() https://sourceforge.net/p/docutils/bugs/431/#a45c

Are these two decisions overlapping, or complementary? If the former, we need better coordination :)

[gmilde]

We have to take the statistics with a grain of salt, as downloads != installations:

• users may have installed Sphinx 1.8.5 years ago and never updated
• CI and other bots may download repeatedly for a clean installation.

For downloads since August 13, I computed 62% Sphinx versions with pin on Docutils
(3.5.4 and 4.x) which leaves 38% downloads without pin on Docutils. For downloads in November 2021, this slightly improves to 65% vs. 35%.

Reasons for users keeping Sphinx 1.x are

• It was the RTD default (and still is for "old projects").
• It is the last version supporting Python 2.
• Fixed dependency versions for Reproducible Builds.
  The last two reasons also count for keeping Sphinx 2.x or 3.x (with x<5.4).

The right thing to do would be to let Docutils 0.18 conflict with Sphinx <= 4.4. Unfortunately, I did not find a conflict mechanism similar to the one in Debian's apt for Python.

The upcoming bugfix release Docutils 0.18.1 will restore the backwards compatibility of nodes.Node.traverse() (cf. https://sourceforge.net/p/docutils/code/8885/).
This should fix the compatibility problems with Sphinx 1.x, as the HTML output only changed in the provisional "html5" writer.
Users with Python 2.7 will not be affected by future Docutils changes as we drop Python 2 support. For others, updating Sphinx is simpler.

This leaves Sphinx 2.x and Sphinx 3.x (with x < 5.4) (ca. 20% of downloads in November)
A considerable amount of these users will have fixed dependencies like sphinx==3.2.0 (for reproducible builds) and would not gain from a backported pin on Docutils.

Downloads from Sphinx 2.x are less than 6% of total downloads in November.
Updates from earlier 3.x versions to 3.5.4 should pose a minor problem.
Whether a ltr-bugfix release Sphinx 2.4.5 is sensible depends on the effort this would make compared to the slight gain.

This leaves documentation:
Please make clear, here and everywhere, that pinning a Sphinx version must be coupled with a pin on Docutils, e.g. for reproducible builds

dependencies:
  - sphinx==3.2.0
  - docutils==0.15

or allowing bugfix releases:

dependencies:
  - sphinx<3.3
  - docutils<0.16

Only with Sphinx versions >=3.5.4, this rule can be relaxed.

[agjohnson]

Thanks @gmilde that's great news! We'll get our docs/blog updated.

I haven't looked yet at the changes coming in docutils 1.0, but I assume we'll all probably have similar hurdles with unpinned dependencies from legacy Sphinx versions then too. So both fixes at docutils and Sphinx seem helpful to keeping releases a bit easier for everyone.

But no reason to pin it on Sphinx side. Okay, let's do it.

@tk0miya ping us if we can help at all here! We'll definitely do some testing on any of the releases though

[gmilde]

Anthony wrote:

I haven't looked yet at the changes coming in docutils 1.0, but I assume we'll all probably have similar hurdles with unpinned dependencies from legacy Sphinx versions then too.

After Docutils 18.1 rsp. r8885, compiling with legacy Sphinx versions should not fail until the planned removals in Docutils 1.2 (check for the DeprecationWarnings, e.g. in tests or running python -W module sphinx-build).
This makes backporting pin-patches less urgent.

Remember: any backport release needs to be enabled by users that pin a specific Sphix version. Adding a pin on Docutils is not more work than changing a pin on Sphinx.

Takeshi KOMIYA wrote:

BTW, what version of docutils is good for pinning for each Sphinx version? I'm okay for either 0.16 and 0.17.

• Sphinx 1.x had its last release in February 2019. Matches with Docutils 0.16 (2019-07-20).¹

  • Users still on Python 2 have an automatic cap to Docutils 0.18.x as this is the last version to support Python 2.7, 3.5 and 3.6.
  • Users on Python > 3.6 or users of the experimental HTML5 writer (opt-in in Sphinx 1.x) can be expected to add a pin on Docutils < 0.16 or update Sphinx.
  • Reconsidering, I see no need for a Sphinx 1.x.y pin-backport release.

• Sphinx 2.x had its last release in March 2020. Matches with Docutils 0.16 (2019-07-20).

  • Docutils 0.17 introduced changes to the experimental HTML5 writer that break older RTD themes.
  • A Sphinx 2.4.5 bugfix release pinning docutils < 0.17 may help some users still bound to the 2.x series. The more conservative docutils < 0.16 would also prevent re-occurence of bug Sphinx, unlike Docutils, incorrectly renders consecutive backslashes #7610 but prevent bugfixes in Docutils 0.16.

• Users of the Sphinx 3.x series will certainly gain from updating to 3.5.4.

  • If a backport for 3.4 is planned, 3.4.4 could pin docutils < 0.17.

¹An earlier incompatibility with Docutils 0.16 was fixed in Sphinx 1.3.2 (Nov 29, 2015).

[tk0miya]

Reconsidering, I see no need for a Sphinx 1.x.y pin-backport release.

Really? The most reports I've seen are troubles in 1.8.5. So what we really need is v1.8.6 having a pin to the old docutils. IMO, almost (or very few) people do not mind either 0.16 or 0.17 because they did not get troubles since the release of docutils-0.17. I believe those who got troubles have already migrated to the new version.

So my idea is:

• releasing Sphinx 1.8.6, 2.4.5 and 3.5.6
• They pins to docutils < 0.18

[tk0miya]

ping us if we can help at all here! We'll definitely do some testing on any of the releases though

Are there any tasks before releasing? If not, I'd like to ship the fixes soon.

[grubert]

Soon ... I learned no release on weekends.
Fast release

• current developer-version 0.18.1b on thursday
• and 0.18.1 on next tuesday

any objection ?

[agjohnson]

Are there any tasks before releasing? If not, I'd like to ship the fixes soon.

I don't think there is anything from the RTD side. Once there are some versions to test we can keep an eye on project builds and provide feedback if there are any issues.

So my idea is releasing Sphinx 1.8.6, 2.4.5 and 3.5.6

This will have to happen at some point before docutils 1.2 release anyways, but docutils 0.18.1 will make this less of a priority. If you have time for the releases, we can plan on monitoring projects using these releases.

current developer-version 0.18.1b on thursday and 0.18.1 on next tuesday

Sounds reasonable. Tues is our release day, so we can add this to our list and monitor project builds with the new release.

[tk0miya]

Now I just released v1.8.6 and v2.4.5 that pin docutils to <0.18. BTW, Sphinx-3.5.5 has already pinned. So I did not release v3.5.6.

[astrojuanlu]

Thanks a lot @tk0miya !

[grubert]

as this one is closed , should docutils 0.18.1 still be rushed out or ... give it a week more beta ?

[gmilde]

Thank you, Takeshi, for the backport releases. Am 18.11.21 schrieb engelbert gruber:

as this one is closed , should docutils 0.18.1 still be rushed out or ... give it a week more beta ?

Please go ahead as planned. The Sphinx backport releases will not help RTD users with "old" projects (pre 2020) pinned to Sphinx==1.8.5 by default or anyone using a pinned Sphinx version from before May 2021 (e.g. for reproducible builds) but not pinning Docutils.

[astrojuanlu]

It's important to note that users of old pip versions might not benefit from this pinning...

https://readthedocs.org/api/v2/build/15321608.txt

ERROR: pip's legacy dependency resolver does not consider dependency conflicts when selecting packages. This behaviour is the source of the following dependency conflicts.
sphinx 1.8.6 requires docutils<0.18,>=0.11, but you'll have docutils 0.18 which is incompatible.

(source: https://twitter.com/s_bergmann/status/1461640620573958145)

🤦🏽‍♂️

[tk0miya]

IMO, docutils-0.18.1 resolves some of the troubles. But some of them will not be resolved. That's the reason why the latest Sphinx does not still support docutils-0.18.