Sphinx Multiversioning (#346)

# Description

Added Sphinx Multiversioning utilising JamesIves/github-deploy-action's
feature of saving commit history on gh-pages branch.

You can see a working example here:
https://zerglev.github.io/chatsky/master/index.html. Use the version
switcher button in the top-right to switch between versions. (It's
switched off right now, but it should work in this repo)
Major changes:
- Can build and deploy documentation for any pre-configured branch or
tag. We can build docs for versions up to v0.8.0; versions before that
are too different. (due to renaming 'DFF' -> 'Chatsky')
- Has a version switcher button. (in top right corner) It only shows
`master` (as 'latest'), `dev` and all tags like `v0.9.0` from a
specified 'start_version'.
- Documentation for different versions is saved in separate folders, and
you can conveniently switch the folder to view the same file for a
different version (if it exists)

Minor details:
- In case there is no version selected for docs, say, someone used an
old link, like `https://deeppavlov.github.io/chatsky/index.html`, the
website will redirect them to `master` by default.
- In case someone makes a mistake and types `/latest/` within their docs
link, it will redirect them to `/master/`.
- All relevant links are version-dependent.

# Checklist
- [x] Made a working example with master and dev built separately
- [x] Version switcher works correctly
- [x] Using tags to create docs for different versions
- [x] Pull Requests build docs correctly
- [x] Deleting old builds before deploying new builds
- [x] Made all relevant links version-dependent
- [x] Redirects work now
- [x] I have performed a self-review of the changes
- [x] Receive a review and do the required changes

# To Consider

- We could redirect people to `chatsky` in case someone types
`dialog_flow_framework`, it's very easy to do.
- Testing Github Pages deployment from main Chatsky repo.
- Testing documentation building from Chatsky repo.
- SEO optimization (old links will show the wrong website, even though
they will get redirected now) (just need to add a few lines into
index.html if we want to do that)
- Finding a better place/design for the version switcher button. (It's
maybe okay now, actually)
- Update CONTRIBUTING.md (if devel workflow is changed)
- Update `.ignore` files, scripts (such as `lint`), distribution
manifest (if files are added/deleted)

Notes:
- "git stash" and "git checkout" are used during the Github Workflow.
I'm not sure if that poses a problem, but everything is returned to it's
place in the end, and I don't really know how to do things otherwise. It
might just be required. If other workflows just wait for this one, then
there can't be a problem, I think. Or if they don't run at the same
time.
- We're always using old conf.py files, meaning the docs will be exactly
the same way, they were before - all brand names, pictures, website
designs might be outdated.

---------

Co-authored-by: Roman Zlobin <RLKRo@proton.me>

.github/workflows/build_and_publish_docs.yml

@@ -6,6 +6,8 @@ on:
     - dev
     - master
     - test/**
+    tags:
+    - v[0-9]+.[0-9]+.[0-9]+
   pull_request:
     branches:
     - dev
@@ -21,6 +23,8 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
 
       - name: set up python 3.9
         uses: actions/setup-python@v5
@@ -36,37 +40,26 @@ jobs:
           python -m pip install --upgrade pip poetry==1.8.5
           python -m poetry install --with tutorials,docs --all-extras --no-ansi --no-interaction
 
+      - name: save docs version into a variable
+        run: |
+          echo "DOC_VERSION=${{ github.head_ref || github.ref_name }}" >> $GITHUB_ENV
+
       - name: build documentation
         env:
           TG_BOT_TOKEN: ${{ secrets.TG_BOT_TOKEN }}
           TG_BOT_USERNAME: ${{ secrets.TG_BOT_USERNAME }}
+          DOC_VERSION: ${{ env.DOC_VERSION }}
         run: |
           python -m poetry run poe docs
 
-      - name: remove jekyll theming
-        run: touch docs/build/.nojekyll
-
-      - name: save branch name without slashes
-        if: ${{ github.ref != 'refs/heads/master' }}
-        env:
-          BRANCH_NAME: ${{ github.head_ref || github.ref_name }}
-        run: |
-          BRANCH_NAME=${{ env.BRANCH_NAME }}
-          BRANCH_NAME=${BRANCH_NAME////_}
-          echo BRANCH_NAME=${BRANCH_NAME} >> $GITHUB_ENV
-
-      - name: save artifact
-        if: ${{ github.ref != 'refs/heads/master' }}
-        uses: actions/upload-artifact@v4
-        with:
-          name: ${{ format('github-pages-for-branch-{0}', env.BRANCH_NAME) }}
-          path: docs/build/
-          retention-days: 3
-
       - name: deploy website
-        if: ${{ github.ref == 'refs/heads/master' }}
-        uses: JamesIves/github-pages-deploy-action@v4.6.4
+        uses: JamesIves/github-pages-deploy-action@v4
         with:
           branch: gh-pages
           folder: docs/build/
-          single-commit: True
+          target-folder: ${{ env.DOC_VERSION }}
+          clean: True
+
+      - name: print link to the documentation
+        run: |
+          echo "https://deeppavlov.github.io/chatsky/${{ github.head_ref || github.ref_name }}"

.github/workflows/generate_version_switcher.yml

@@ -0,0 +1,44 @@
+name: generate_version_switcher
+
+on:
+  workflow_run:
+    workflows: ["build_and_publish_docs"]
+    types:
+      - completed
+  workflow_dispatch:
+
+concurrency:
+  group: ${{ github.workflow }}
+  cancel-in-progress: true
+
+jobs:
+  publish:
+    name: generate and update version switcher's json file
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
+
+      - name: install GitPython
+        run: python -m pip install GitPython
+
+      - name: generate version switcher
+        env:
+          VERSION_SWITCHER_STARTING_TAG: ${{ vars.VERSION_SWITCHER_STARTING_TAG }}
+          VERSION_SWITCHER_TAG_BLACKLIST: ${{ vars.VERSION_SWITCHER_TAG_BLACKLIST }}
+          VERSION_SWITCHER_TAG_WHITELIST: ${{ vars.VERSION_SWITCHER_TAG_WHITELIST }}
+        run: |
+          python ./scripts/switcher_gen.py
+
+      - name: copy version switcher for updating it
+        run: |
+          mkdir docs/source/switcher/
+          cp docs/source/_static/switcher.json docs/source/switcher/switcher.json
+
+      - name: update version switcher
+        uses: JamesIves/github-pages-deploy-action@v4
+        with:
+          branch: gh-pages
+          folder: docs/source/switcher/
+          clean: False

docs/source/_templates/example-links.html

@@ -1,8 +1,9 @@
 {% if meta is defined and meta is not none and 'tutorial_name' in meta %}
+    {% set doc_version = meta['doc_version'] %}
     {% set repo_path = "/".join(meta['tutorial_name'].split("/")[-1].split(".")) %}
-    {% set github_link = 'https://github.com/deeppavlov/chatsky/blob/master/' ~ repo_path ~ '.py' %}
+    {% set github_link = 'https://github.com/deeppavlov/chatsky/blob/' ~ doc_version ~ repo_path ~ '.py' %}
     {% set notebook_link = '../' ~ meta['tutorial_name'] ~ '.ipynb' %}
-    {% set colab_link = 'https://colab.research.google.com/github/deeppavlov/chatsky/blob/gh-pages/' ~ meta['tutorial_name'] ~ '.ipynb' %}
+    {% set colab_link = 'https://colab.research.google.com/github/deeppavlov/chatsky/blob/gh-pages/' ~ doc_version ~ meta['tutorial_name'] ~ '.ipynb' %}
 
     <a href="{{ colab_link }}" data-behavior="call-to-action-event" data-response="Run in Google Colab" target="_blank">
         <div id="google-colab-link">

docs/source/_templates/source-links.html

@@ -1,5 +1,6 @@
 {% if meta is defined and meta is not none and 'source_name' in meta %}
-    {% set github_link = 'https://github.com/deeppavlov/chatsky/blob/master/' ~ meta['source_name'] ~ '.py' %}
+    {% set doc_version = meta['doc_version'] %}
+    {% set github_link = 'https://github.com/deeppavlov/chatsky/blob/' ~ doc_version ~ meta['source_name'] ~ '.py' %}
 
     <a href="{{ github_link }}" data-behavior="call-to-action-event" data-response="View on Github" target="_blank">
         <div id="github-view-link">

docs/source/conf.py

@@ -1,15 +1,13 @@
 import os
 import sys
 import re
+import git
 import importlib.metadata
 
 # -- Path setup --------------------------------------------------------------
 
 sys.path.append(os.path.abspath("."))
 from utils.notebook import py_percent_to_notebook  # noqa: E402
-from utils.generate_tutorials import generate_tutorial_links_for_notebook_creation  # noqa: E402
-from utils.link_misc_files import link_misc_files  # noqa: E402
-from utils.regenerate_apiref import regenerate_apiref  # noqa: E402
 
 # -- Project information -----------------------------------------------------
 
@@ -20,6 +18,9 @@
 author = "DeepPavlov"
 release = _distribution_metadata["Version"]
 
+current_commit = git.Repo('../../').head.commit
+today = current_commit.committed_datetime
+today = today.strftime("%b %d, %Y")
 
 # -- General configuration ---------------------------------------------------
 
@@ -51,7 +52,7 @@
 source_suffix = ".rst"
 master_doc = "index"
 
-version = re.match(r"^\d\.\d.\d", release).group()
+version = re.match(r"^\d\.\d", release).group()
 language = "en"
 
 pygments_style = "default"
@@ -89,12 +90,20 @@
 
 autosummary_generate_overwrite = False
 
+doc_version = os.getenv("DOC_VERSION", default="master")
+if doc_version != "":
+    doc_version = doc_version + '/'
 # Finding tutorials directories
 nbsphinx_custom_formats = {".py": py_percent_to_notebook}
-nbsphinx_prolog = """
-:tutorial_name: {{ env.docname }}
+nbsphinx_prolog = f"""
+:tutorial_name: {{{{ env.docname }}}}
+:doc_version: {doc_version}
 """
 
+extlinks = {
+    'github_source_link': (f"https://github.com/deeppavlov/chatsky/blob/{doc_version}%s", None),
+}
+
 html_logo = "_static/images/Chatsky-full-dark.svg"
 
 nbsphinx_thumbnails = {
@@ -112,6 +121,10 @@
     "css/custom.css",
 ]
 
+# Version switcher data
+version_data = os.getenv("DOC_VERSION", default="master")
+switcher_url = "https://deeppavlov.github.io/chatsky/switcher.json"
+
 # Theme options
 html_theme_options = {
     "header_links_before_dropdown": 5,
@@ -136,6 +149,12 @@
         },
     ],
     "secondary_sidebar_items": ["page-toc", "source-links", "example-links"],
+    "switcher": {
+        "json_url": switcher_url,
+        "version_match": version_data,
+    },
+    "navbar_persistent": ["search-button.html", "theme-switcher.html"],
+    "navbar_end": ["version-switcher.html", "navbar-icon-links.html"],
 }
 
 
@@ -155,52 +174,5 @@
 
 
 def setup(_):
-    link_misc_files(
-        [
-            "utils/db_benchmark/benchmark_schema.json",
-            "utils/db_benchmark/benchmark_streamlit.py",
-        ]
-    )
-    generate_tutorial_links_for_notebook_creation(
-        [
-            ("tutorials.context_storages", "Context Storages"),
-            (
-                "tutorials.messengers",
-                "Interfaces",
-                [
-                    ("telegram", "Telegram"),
-                    ("web_api_interface", "Web API"),
-                ],
-            ),
-            ("tutorials.service", "Service"),
-            (
-                "tutorials.script",
-                "Script",
-                [
-                    ("core", "Core"),
-                    ("responses", "Responses"),
-                ],
-            ),
-            ("tutorials.llm", "LLM Integration"),
-            ("tutorials.slots", "Slots"),
-            ("tutorials.stats", "Stats"),
-        ]
-    )
-    regenerate_apiref(
-        [
-            ("chatsky.core.service", "Core.Service"),
-            ("chatsky.core", "Core"),
-            ("chatsky.conditions", "Conditions"),
-            ("chatsky.destinations", "Destinations"),
-            ("chatsky.responses", "Responses"),
-            ("chatsky.processing", "Processing"),
-            ("chatsky.context_storages", "Context Storages"),
-            ("chatsky.messengers", "Messenger Interfaces"),
-            ("chatsky.llm", "LLM Integration"),
-            ("chatsky.slots", "Slots"),
-            ("chatsky.stats", "Stats"),
-            ("chatsky.utils.testing", "Testing Utils"),
-            ("chatsky.utils.db_benchmark", "DB Benchmark"),
-            ("chatsky.utils.devel", "Development Utils"),
-        ]
-    )
+    from setup import setup
+    setup()

docs/source/development.rst

@@ -4,7 +4,7 @@ Development
 Contribution
 ~~~~~~~~~~~~~~~
 
-`Contribution rules <https://github.com/deeppavlov/chatsky/blob/master/CONTRIBUTING.md>`_  provide
+:github_source_link:`Contribution rules <CONTRIBUTING.md>` provide
 guidelines and rules for contributing to the Chatsky project. It includes information on
 how to contribute code to the project, manage your workflow, use tests, and so on.
 

docs/source/setup.py

@@ -0,0 +1,55 @@
+from utils.generate_tutorials import generate_tutorial_links_for_notebook_creation  # noqa: E402
+from utils.link_misc_files import link_misc_files  # noqa: E402
+from utils.regenerate_apiref import regenerate_apiref  # noqa: E402
+
+
+def setup():
+    link_misc_files(
+        [
+            "utils/db_benchmark/benchmark_schema.json",
+            "utils/db_benchmark/benchmark_streamlit.py",
+        ]
+    )
+    generate_tutorial_links_for_notebook_creation(
+        [
+            ("tutorials.context_storages", "Context Storages"),
+            (
+                "tutorials.messengers",
+                "Interfaces",
+                [
+                    ("telegram", "Telegram"),
+                    ("web_api_interface", "Web API"),
+                ],
+            ),
+            ("tutorials.service", "Service"),
+            (
+                "tutorials.script",
+                "Script",
+                [
+                    ("core", "Core"),
+                    ("responses", "Responses"),
+                ],
+            ),
+            ("tutorials.llm", "LLM Integration"),
+            ("tutorials.slots", "Slots"),
+            ("tutorials.stats", "Stats"),
+        ]
+    )
+    regenerate_apiref(
+        [
+            ("chatsky.core.service", "Core.Service"),
+            ("chatsky.core", "Core"),
+            ("chatsky.conditions", "Conditions"),
+            ("chatsky.destinations", "Destinations"),
+            ("chatsky.responses", "Responses"),
+            ("chatsky.processing", "Processing"),
+            ("chatsky.context_storages", "Context Storages"),
+            ("chatsky.messengers", "Messenger Interfaces"),
+            ("chatsky.llm", "LLM Integration"),
+            ("chatsky.slots", "Slots"),
+            ("chatsky.stats", "Stats"),
+            ("chatsky.utils.testing", "Testing Utils"),
+            ("chatsky.utils.db_benchmark", "DB Benchmark"),
+            ("chatsky.utils.devel", "Development Utils"),
+        ]
+    )

docs/source/user_guides/context_guide.rst

@@ -190,8 +190,7 @@ and the connection parameters, for example, *mongodb://admin:pass@localhost:2701
 
 The GitHub-based distribution of Chatsky includes Docker images for each of the supported database types.
 Therefore, the easiest way to deploy your service together with a database is to clone the GitHub
-distribution and to take advantage of the packaged
-`docker compose file <https://github.com/deeppavlov/chatsky/blob/master/compose.yml>`_.
+distribution and to take advantage of the packaged :github_source_link:`docker compose file <compose.yml>`.
 
 .. code-block:: shell
   :linenos:
@@ -202,5 +201,5 @@ distribution and to take advantage of the packaged
   docker compose up mongo
 
 The images can be configured using the docker compose file or the
-`environment file <https://github.com/deeppavlov/chatsky/blob/master/.env_file>`_,
+:github_source_link:`environment file <.env_file>`,
 also available in the distribution. Consult these files for more options.