Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use Read the Docs command to build Sphinx documents in parallel #17

Closed

Conversation

madmiraal
Copy link
Contributor

Read the Docs, which hosts our online documentation: https://docs.rebeltoolbox.com has limits on the resources available to a build the documentation. Builds will fail if they take longer than 30 minutes. Our builds are currently taking around 30 minutes and have taken longer and failed e.g.: https://readthedocs.org/projects/rebel-documentation/builds/23365172/.

The majority of the time is taken up by the Sphinx build:

Date: 2024-02-06T00:39:55.838316Z
...
[rtd-command-info] start-time: 2024-02-06T00:40:38.959622Z, end-time: 2024-02-06T01:10:03.989662Z, duration: 1765, exit-code: 0
python -m sphinx -T -b html -d _build/doctrees -D language=en . $READTHEDOCS_OUTPUT/html

Everything before the Sphinx build is taking less than a minute, but the build is taking close to half an hour, with the failed build exceeding the 30 minute limit:

Date: 2024-02-06T18:28:43.194512Z
...
[rtd-command-info] start-time: 2024-02-06T18:29:29.603841Z, end-time: 2024-02-06T19:00:33.754201Z, duration: 1864, exit-code: 137
python -m sphinx -T -b html -d _build/doctrees -D language=en . $READTHEDOCS_OUTPUT/html

This PR attempts to use a custom build command to run the Sphinx build using multiple parallel jobs. I have tested this command using a GitHub build test, which ran for only 11 minutes! However, this can only be tested on Read the Docs once the PR is created.

@madmiraal madmiraal added the bug label Feb 11, 2024
@madmiraal madmiraal self-assigned this Feb 11, 2024
@madmiraal madmiraal force-pushed the read-the-docs-parallel-builds branch 2 times, most recently from e7e2d8c to 2e3bbb8 Compare February 11, 2024 20:39
@DanielaOrtner DanielaOrtner self-requested a review February 11, 2024 20:40
@madmiraal madmiraal force-pushed the read-the-docs-parallel-builds branch from 2e3bbb8 to c8605cd Compare February 12, 2024 17:59
@madmiraal madmiraal force-pushed the read-the-docs-parallel-builds branch from c8605cd to 266feb4 Compare February 14, 2024 09:09
@madmiraal
Copy link
Contributor Author

Closing, because running parallel builds does not significantly reduce the build time. Most of the time is spent writing the files; so this is an I/O bottle-neck not a processing bottle-neck.

Read the Docs has kindly extended our allowed build times. So, builds should no longer fail for taking slightly longer than 30 minutes going forward.

@madmiraal madmiraal closed this Feb 29, 2024
@madmiraal madmiraal added PR Type: Technical Fixes a technical issue with the website hosting, updating or building. and removed bug labels May 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
PR Type: Technical Fixes a technical issue with the website hosting, updating or building.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant