Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

✨ NEW: Add support for parallel mode #47

Merged
merged 6 commits into from
Dec 25, 2022
Merged

Conversation

jdillard
Copy link
Owner

@jdillard jdillard commented Dec 25, 2022

Summary of Changes

  • Add support for parallel mode

Notes

Huge shout-out to @xmo-odoo for #29 (closed so couldn't merge).

Future improvements to detecting when parallel mode is turned on could be made, as mentioned in the PR:

This is achieved by storing a queue in the env so the "write" workers can shove page names in there (sadly multiprocessing.Queue did not work as it apparently doesn't like the way the workers are created), and create_sitemap can then pop all that from the queue.

Anyway there's a bit of an issue here, probably not a huge one because sitemap is only invoked once per document (not hundreds of times per), but the multiprocessing queue is always created and possibly somewhat heavy (it's at least a separate process, plus whatever the proxies use to communicate with the subprocess), but while Builder has a parallel_ok flag that flag is only set during build(), which comes later than record_builder_type.

The only option I can see (aside from Sphinx being updated to resolve this earlier) is to copy the implementation details of parallel_ok in the extension and hope they don't change too much in the future, using that to decide between the seq and the parallel sitemap_links implementations, and that's gross.

Even hooking into one of the events between the reading and writing phase would not work: parallel_ok is actually set after env-check-consistency, and there doesn't seem to be any sequential event afterwards.

Potentially better support from Sphinx outlined here: sphinx-doc/sphinx#9480

@jdillard jdillard changed the title Add support for parallel mode ✨ NEW: Add support for parallel mode Dec 25, 2022
@jdillard jdillard merged commit 62ff3ce into master Dec 25, 2022
@jdillard jdillard deleted the feature/parallel-support branch December 25, 2022 04:43
jdillard pushed a commit that referenced this pull request Mar 2, 2023
…m being pickled with environment (#62)

#47 added support for parallel building via a multiprocessing.Manager().Queue() proxy object attached to the Environment, breaking incremental builds, as Sphinx later pickles the Environment object, including the Queue proxy object. The object pickles successfully, but when the Environment is later unpickled to determine if an incremental build can be used, the proxy object tries to reconnect to the server process that existed at the time of pickling, but no longer exists. This failure is near-silently swallowed with an unintuitive error message and sphinx proceeds with a full rebuild.

Sphinx specifically excludes the instance of app that's attached to the Environment from pickling via it's __getstate__() method: https://github.com/sphinx-doc/sphinx/blob/ba080286b06cb9e0cadec59a6cf1f96aa11aef5a/sphinx/environment/__init__.py#L262

This moves the Queue proxy object to the instance of app provided on the Environment in order to avoid pickling the Queue object while it is connected to it's server process.

As an alternative, It should be possible to create a custom Queue class with it's own __getstate__() method to not attempt to reconnect upon unpickling, but I believe the Most Right Answer is for upstream Sphinx to provide a way to mark custom Environment attributes as unpicklable.

As an alternative, It should be possible to create a custom Queue class with it's own __getstate__() method to not attempt to reconnect upon unpickling, but I believe the Most Right Answer is for upstream Sphinx to provide a way to mark custom Environment attributes as unpicklable
gaurav274 pushed a commit to georgia-tech-db/evadb that referenced this pull request May 22, 2023
Bumps [sphinx-sitemap](https://github.com/jdillard/sphinx-sitemap) from
2.2.0 to 2.5.0.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/jdillard/sphinx-sitemap/releases">sphinx-sitemap's
releases</a>.</em></p>
<blockquote>
<h2>Release 2.5.0 (2023-01-28)</h2>
<p>Install from <a
href="https://pypi.org/project/sphinx-sitemap/2.5.0/">PyPi</a></p>
<h2>What's Changed</h2>
<ul>
<li>:books: DOCS: Calculate version for sitemap based on current tag by
<a href="https://github.com/jdillard"><code>@​jdillard</code></a> in <a
href="https://redirect.github.com/jdillard/sphinx-sitemap/pull/53">jdillard/sphinx-sitemap#53</a></li>
<li>:wrench: MAINT: Add changelog to <code>project.urls</code> by <a
href="https://github.com/jdillard"><code>@​jdillard</code></a> in <a
href="https://redirect.github.com/jdillard/sphinx-sitemap/pull/54">jdillard/sphinx-sitemap#54</a></li>
<li>:test_tube: TESTS: Add Sphinx 6 environment to tox by <a
href="https://github.com/jdillard"><code>@​jdillard</code></a> in <a
href="https://redirect.github.com/jdillard/sphinx-sitemap/pull/55">jdillard/sphinx-sitemap#55</a></li>
<li>:sparkles: NEW: Add support for Sphinx config
<code>html_file_suffix</code> by <a
href="https://github.com/jdetaeye"><code>@​jdetaeye</code></a> in <a
href="https://redirect.github.com/jdillard/sphinx-sitemap/pull/57">jdillard/sphinx-sitemap#57</a></li>
<li>:books: DOCS: Add site search optimization by <a
href="https://github.com/jdillard"><code>@​jdillard</code></a> in <a
href="https://redirect.github.com/jdillard/sphinx-sitemap/pull/58">jdillard/sphinx-sitemap#58</a></li>
</ul>
<h2>New Contributors</h2>
<ul>
<li><a href="https://github.com/jdetaeye"><code>@​jdetaeye</code></a>
made their first contribution in <a
href="https://redirect.github.com/jdillard/sphinx-sitemap/pull/57">jdillard/sphinx-sitemap#57</a></li>
</ul>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/jdillard/sphinx-sitemap/compare/v2.4.0...v2.5.0">https://github.com/jdillard/sphinx-sitemap/compare/v2.4.0...v2.5.0</a></p>
<h2>Release 2.4.0 (2022-12-26)</h2>
<p>Install from <a
href="https://pypi.org/project/sphinx-sitemap/2.4.0/">PyPi</a></p>
<h2>What's Changed</h2>
<ul>
<li>:test_tube: TESTS: Add python 3.11 tests by <a
href="https://github.com/jdillard"><code>@​jdillard</code></a> in <a
href="https://redirect.github.com/jdillard/sphinx-sitemap/pull/43">jdillard/sphinx-sitemap#43</a></li>
<li>:books: DOCS: Add ReadTheDocs docs by <a
href="https://github.com/jdillard"><code>@​jdillard</code></a> in <a
href="https://redirect.github.com/jdillard/sphinx-sitemap/pull/45">jdillard/sphinx-sitemap#45</a></li>
<li>:wrench: MAINT: General code clean up by <a
href="https://github.com/jdillard"><code>@​jdillard</code></a> in <a
href="https://redirect.github.com/jdillard/sphinx-sitemap/pull/46">jdillard/sphinx-sitemap#46</a></li>
<li>:sparkles: NEW: Add support for parallel mode by <a
href="https://github.com/jdillard"><code>@​jdillard</code></a> in <a
href="https://redirect.github.com/jdillard/sphinx-sitemap/pull/47">jdillard/sphinx-sitemap#47</a></li>
<li>:test_tube: TESTS: Add tests for dirhtml builder by <a
href="https://github.com/jdillard"><code>@​jdillard</code></a> in <a
href="https://redirect.github.com/jdillard/sphinx-sitemap/pull/48">jdillard/sphinx-sitemap#48</a></li>
<li>:books: DOCS: Add vale support for docs by <a
href="https://github.com/jdillard"><code>@​jdillard</code></a> in <a
href="https://redirect.github.com/jdillard/sphinx-sitemap/pull/49">jdillard/sphinx-sitemap#49</a></li>
<li>:bug: FIX: Fix wheel includes so they don't include docs and tests
by <a href="https://github.com/jdillard"><code>@​jdillard</code></a> in
<a
href="https://redirect.github.com/jdillard/sphinx-sitemap/pull/51">jdillard/sphinx-sitemap#51</a></li>
<li>:books: DOCS: Add write-good and improve writing by <a
href="https://github.com/jdillard"><code>@​jdillard</code></a> in <a
href="https://redirect.github.com/jdillard/sphinx-sitemap/pull/52">jdillard/sphinx-sitemap#52</a></li>
</ul>
<p>Shout-out to <a
href="https://github.com/xmo-odoo"><code>@​xmo-odoo</code></a> in <a
href="https://redirect.github.com/jdillard/sphinx-sitemap/issues/47">#47</a></p>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/jdillard/sphinx-sitemap/compare/v2.3.0...v2.4.0">https://github.com/jdillard/sphinx-sitemap/compare/v2.3.0...v2.4.0</a></p>
<h2>Release 2.3.0 (2022-12-21)</h2>
<p>Install from <a
href="https://pypi.org/project/sphinx-sitemap/2.3.0/">PyPi</a></p>
<h2>What's Changed</h2>
<ul>
<li>:wrench: MAINT: Clean up how package versions are handled by <a
href="https://github.com/jdillard"><code>@​jdillard</code></a> in <a
href="https://redirect.github.com/jdillard/sphinx-sitemap/pull/34">jdillard/sphinx-sitemap#34</a></li>
<li>:wrench: MAINT: Install pre-commit with isort, black, and flake8 by
<a href="https://github.com/jdillard"><code>@​jdillard</code></a> in <a
href="https://redirect.github.com/jdillard/sphinx-sitemap/pull/35">jdillard/sphinx-sitemap#35</a></li>
<li>:books: DOCS: Improve the wording of the README by <a
href="https://github.com/jdillard"><code>@​jdillard</code></a> in <a
href="https://redirect.github.com/jdillard/sphinx-sitemap/pull/36">jdillard/sphinx-sitemap#36</a></li>
<li>:bug: FIX: Follow correct format for multilingual sitemaps by <a
href="https://github.com/jdillard"><code>@​jdillard</code></a> in <a
href="https://redirect.github.com/jdillard/sphinx-sitemap/pull/38">jdillard/sphinx-sitemap#38</a></li>
<li>:wrench: MAINT: Update the build process and provide wheels by <a
href="https://github.com/jdillard"><code>@​jdillard</code></a> in <a
href="https://redirect.github.com/jdillard/sphinx-sitemap/pull/39">jdillard/sphinx-sitemap#39</a></li>
<li>:wrench: MAINT: Add testing infrastructure by <a
href="https://github.com/jdillard"><code>@​jdillard</code></a> in <a
href="https://redirect.github.com/jdillard/sphinx-sitemap/pull/41">jdillard/sphinx-sitemap#41</a></li>
<li>:wrench: MAINT: Add python 3.10 tests and remove flake8 test from
tox by <a href="https://github.com/jdillard"><code>@​jdillard</code></a>
in <a
href="https://redirect.github.com/jdillard/sphinx-sitemap/pull/42">jdillard/sphinx-sitemap#42</a></li>
<li>:bug: FIX: Use logging for all logging messages by <a
href="https://github.com/jdillard"><code>@​jdillard</code></a> in <a
href="https://redirect.github.com/jdillard/sphinx-sitemap/pull/40">jdillard/sphinx-sitemap#40</a></li>
</ul>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/jdillard/sphinx-sitemap/compare/v2.2.1...v2.3.0">https://github.com/jdillard/sphinx-sitemap/compare/v2.2.1...v2.3.0</a></p>
<h2>Release 2.2.1 (2022-11-11)</h2>
<p>Install from <a
href="https://pypi.org/project/sphinx-sitemap/2.2.1/">PyPi</a></p>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/jdillard/sphinx-sitemap/blob/master/CHANGELOG.rst">sphinx-sitemap's
changelog</a>.</em></p>
<blockquote>
<h2>2.5.0</h2>
<p><em>Release date: 2023-01-28</em></p>
<ul>
<li>|:books:| DOCS: Calculate version for sitemap based on current tag
<code>[#53](jdillard/sphinx-sitemap#53)
&lt;https://github.com/jdillard/sphinx-sitemap/pull/53&gt;</code>_</li>
<li>|:test_tube:| TESTS: Add Sphinx 6 env to tox
<code>[#55](jdillard/sphinx-sitemap#55)
&lt;https://github.com/jdillard/sphinx-sitemap/pull/55&gt;</code>_</li>
<li>|:sparkles:| NEW: Add support for Sphinx config
&quot;html_file_suffix&quot;
<code>[#57](jdillard/sphinx-sitemap#57)
&lt;https://github.com/jdillard/sphinx-sitemap/pull/57&gt;</code>_</li>
<li>|:books:| DOCS: Add site search optimization
<code>[#58](jdillard/sphinx-sitemap#58)
&lt;https://github.com/jdillard/sphinx-sitemap/pull/58&gt;</code>_</li>
</ul>
<h2>2.4.0</h2>
<p><em>Release date: 2022-12-26</em></p>
<ul>
<li>|:books:| DOCS: Add ReadTheDocs docs
<code>[#45](jdillard/sphinx-sitemap#45)
&lt;https://github.com/jdillard/sphinx-sitemap/pull/45&gt;</code>_</li>
<li>|:wrench:| MAINT: General code clean up
<code>[#46](jdillard/sphinx-sitemap#46)
&lt;https://github.com/jdillard/sphinx-sitemap/pull/46&gt;</code>_</li>
<li>|:sparkles:| NEW: Add support for parallel mode
<code>[#47](jdillard/sphinx-sitemap#47)
&lt;https://github.com/jdillard/sphinx-sitemap/pull/47&gt;</code>_</li>
<li>|:test_tube:| TESTS: Add tests for <code>dirhtml</code> builder
<code>[#48](jdillard/sphinx-sitemap#48)
&lt;https://github.com/jdillard/sphinx-sitemap/pull/48&gt;</code>_</li>
<li>|:test_tube:| TESTS: Add vale support for docs
<code>[#49](jdillard/sphinx-sitemap#49)
&lt;https://github.com/jdillard/sphinx-sitemap/pull/49&gt;</code>_</li>
<li>|:bug:| FIX: Fix wheel includes so they don't include docs and tests
<code>[#51](jdillard/sphinx-sitemap#51)
&lt;https://github.com/jdillard/sphinx-sitemap/pull/51&gt;</code>_</li>
<li>|:books:| DOCS: Add write-good and improve writing
<code>[#52](jdillard/sphinx-sitemap#52)
&lt;https://github.com/jdillard/sphinx-sitemap/pull/52&gt;</code>_</li>
</ul>
<h2>2.3.0</h2>
<p><em>Release date: 2022-12-21</em></p>
<ul>
<li>|:wrench:| MAINT: Clean up how package versions are handled</li>
<li>|:test_tube:| TESTS: Install pre-commit with <code>isort</code>,
<code>black</code>, and <code>flake8</code>
<code>[#35](jdillard/sphinx-sitemap#35)
&lt;https://github.com/jdillard/sphinx-sitemap/pull/35&gt;</code>_</li>
<li>|:books:| DOCS: Improve the wording of the README to help with
issues upgrading to Sphinx 5
<code>[#36](jdillard/sphinx-sitemap#36)
&lt;https://github.com/jdillard/sphinx-sitemap/pull/36&gt;</code>_</li>
<li>|:bug:| FIX: Follow correct format for multilingual sitemaps
<code>[#38](jdillard/sphinx-sitemap#38)
&lt;https://github.com/jdillard/sphinx-sitemap/pull/38&gt;</code>_</li>
<li>|:wrench:| MAINT: Update the build process
<code>[#39](jdillard/sphinx-sitemap#39)
&lt;https://github.com/jdillard/sphinx-sitemap/pull/39&gt;</code>_</li>
<li>|:test_tube:| TESTS: Add testing infrastructure
<code>[#41](jdillard/sphinx-sitemap#41)
&lt;https://github.com/jdillard/sphinx-sitemap/pull/41&gt;</code>_</li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/jdillard/sphinx-sitemap/commit/7ec6ccc443df9b5565e68b3f21ff2f597ca14e19"><code>7ec6ccc</code></a>
Bump version to 2.5.0</li>
<li><a
href="https://github.com/jdillard/sphinx-sitemap/commit/f6c998c41ffeb15f956e587f7ec4451bd83a84b4"><code>f6c998c</code></a>
DOCS: Add site search optimization (<a
href="https://redirect.github.com/jdillard/sphinx-sitemap/issues/58">#58</a>)</li>
<li><a
href="https://github.com/jdillard/sphinx-sitemap/commit/8eff853b7623c8d911e9e9604b8c0eb56804963b"><code>8eff853</code></a>
Update changelog</li>
<li><a
href="https://github.com/jdillard/sphinx-sitemap/commit/d00702d0043130327fb051c1487979eddf9b6367"><code>d00702d</code></a>
Add test for html_file_suffix</li>
<li><a
href="https://github.com/jdillard/sphinx-sitemap/commit/06450ac9e6ea219c46201bc90edeb94d5e47789f"><code>06450ac</code></a>
NEW: Add support for Sphinx config &quot;html_file_suffix&quot;. (<a
href="https://redirect.github.com/jdillard/sphinx-sitemap/issues/57">#57</a>)</li>
<li><a
href="https://github.com/jdillard/sphinx-sitemap/commit/edf8c48339b6fe56ed2738298303e4ca4b997970"><code>edf8c48</code></a>
Be more explicit with python version classifiers</li>
<li><a
href="https://github.com/jdillard/sphinx-sitemap/commit/c8f29ee03e6d0d0b952e4a2c9a3975fef26837b0"><code>c8f29ee</code></a>
TESTS: Add Sphinx 6 env to tox (<a
href="https://redirect.github.com/jdillard/sphinx-sitemap/issues/55">#55</a>)</li>
<li><a
href="https://github.com/jdillard/sphinx-sitemap/commit/382be2a6fef956e42e6ba7705f12552d72c72eb7"><code>382be2a</code></a>
MAINT: Add changelog to project.urls (<a
href="https://redirect.github.com/jdillard/sphinx-sitemap/issues/54">#54</a>)</li>
<li><a
href="https://github.com/jdillard/sphinx-sitemap/commit/8ca21a916cda43ddf1607b641a2550627b383314"><code>8ca21a9</code></a>
DOCS: Improve wording and filenames</li>
<li><a
href="https://github.com/jdillard/sphinx-sitemap/commit/08229f74d1c72d48068a7edd4d55212cfd12172a"><code>08229f7</code></a>
DOCS: Add getting started and move CHANGELOG</li>
<li>Additional commits viewable in <a
href="https://github.com/jdillard/sphinx-sitemap/compare/v2.2.0...v2.5.0">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=sphinx-sitemap&package-manager=pip&previous-version=2.2.0&new-version=2.5.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)


</details>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant