Using MkDocs from Github Actions with Material - Plugins not enabled?

[dcaddick]

Hi, first time posting here in Github Issues.

I have managed to use MkDocs & Material via Github Actions to publish a website to Github pages.
And I will freely admit that I'm very dangerous and chiefly code with a chainsaw and sledge hammer...

What I'm finding difficult to understand is how the Plugins work?
In my Github workflow I have understood enough to install plugins via pip here:
MakeDocsAction.yaml
But also curious as to whether or not I should be just able to "add" these to the list via the requirements.txt file?
requirements.txt

Anyway - getting them installed is one thing, easy enough, the hard part that I'm struggling with is getting them enabled?
See here for the mkdocs.yml in root:
mydocs.yml

The only ones I can get running are Search, Simple & Macros - am I doing something really dumb?
Snippet:

# Plugins
plugins:
  - search
  - simple
  #- techdocs-core
  #- glightbox
  #- social
  #- awesome-list
  #- table-reader
  #- img2fig
  #- mermaid2
  - macros
  # - minify

I've commented out the Plugins that keep erroring - here is an example of the error "Action"
Action run - with error

I've simply not worried about the Plugins errors and got on with creating the content and figured I'd get around to this at some stage.

Using pip I can consistently get the plugin to "install" but getting it enabled has consistently shown an error similar to this with just the name of the plug-in changing.

I figured it was just me doing something dumb?

[athackst]

Yeah, you just need to put them in your requirements.txt file

This action reads the requirements.txt file and installs all packages listed in there before running (see entrypoint.sh called in the action Dockerfile)

(1) shows all of the plugins that were installed into the action from requirements.txt

(2) It seems like you have mkdocs-git-revision-date-plugin but mkdocs-git-revision-date-localized-plugin is commented out in your requirements.txt file. Is there a reason for that?

The action doesn't currently support another file/name for requirements, but that would be a pretty easy change.

There shouldn't be any need to "enable" a plugin - once it's installed it should be findable by mkdocs. The plugins need to be installed within the action container, which is why your pip install lines aren't working the way you expect. That installs things into the github workflow instance, but that's isolated from the action.

[dcaddick]

Thanks Allison,

So remove my "BS" from the actions, and just revert to using the Requirements.txt as the only place to call PLugins to be installed?

I have been back and forth trying to troubleshoot why I can get Plugins "installed" - but not "enabled" - this is why some parts are commented out.

From what you're saying I don't need to enable the plugins via my mkdocs.yml file?
https://github.com/dcaddick/gsd_public/blob/main/mkdocs.yml

[athackst]

So remove my "BS" from the actions, and just revert to using the Requirements.txt as the only place to call PLugins to be installed?

Yes, the plugin should read from requirements.txt and install plugins accordingly.

A simple workflow should look something like this

jobs:
  publish:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v3
      - name: Build and push docs
        uses: athackst/mkdocs-simple-plugin@v2.2.0

I have been back and forth trying to troubleshoot why I can get Plugins "installed" - but not "enabled" - this is why some parts are commented out.

From what you're saying I don't need to enable the plugins via my mkdocs.yml file?
https://github.com/dcaddick/gsd_public/blob/main/mkdocs.yml

Ah, yes. You do need to add it to mkdocs.yml - I don't think I understood what you were talking about earlier. Adding it to mkdocs.yml will tell mkdocs to call the hooks in the plugin (otherwise it wouldn't know what was installed).

[athackst]

Maybe a tip for testing:

The image published to docker is the same one as the action. You can run it locally for testing

docker run --rm -it --network=host -v ${PWD}:/docs --user $(id -u):$(id -g) althack/mkdocs-simple-plugin

It should build and serve the site locally through the container

[dcaddick]

Ah, yes. You do need to add it to mkdocs.yml - I don't think I understood what you were talking about earlier. Adding it to mkdocs.yml will tell mkdocs to call the hooks in the plugin (otherwise it wouldn't know what was installed).

So this is where I have been seeing the error come thru - not the actual install of the Plugins - but the "call to the hooks" and the subsequent detailed config has been the typical error point.

I've now cleaned out the PIP calls and kept it clean in the workflow - testing that now.
Will then move on to checking/updating reqs.txt - then after that back to seeing if "the hooks" work or still error out

[athackst]

It looks like the issue is that something is requiring numpy which apparently doesn't like python 3.11

Try changing to v2.1.3 instead of v2.2.0

[dcaddick]

So after the numpy error - rolled back to 2.1.3 and now for some reason it's taking an age to try and complete, instead of under 2 mins it's not 13mins plus and I simply cancelled...

It just keeps going:
Installing build dependencies: still running...
Installing build dependencies: still running...
Installing build dependencies: still running...
Installing build dependencies: still running...
Installing build dependencies: still running...

https://github.com/dcaddick/gsd_public/actions/runs/3936269907/jobs/6732667943

So now still scratching my head on how to troubleshoot this correctly - even if I did pull it down local via Docker image (great call though - thanks for the tip)

[athackst]

WTF needs pandas? That's a rather heavy-weight requirement for document building.

The image I use for the action is based on apk to keep it light/quick, maybe pandas needs to be built from source because it's not available as a pre-built?

You might be better served with your own workflow instead of using the action?

jobs:
  docs:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: 3.x
          cache: pip
      - name: Setup
        run: |
          python -m pip install --upgrade pip
         pip install -r requirements.txt
      - name: Deploy docs
         run: |
           git fetch origin gh-pages --depth=1 || echo -e "skipping fetch"
           mkdocs gh-deploy

[dcaddick]

I'm glad you said that - cause this is doing my head in - looks like it's now going round in circles... ;-)
https://github.com/dcaddick/gsd_public/actions/runs/3936424193
currently just tipped over the 56 min mark

[dcaddick]

So I can only assume at this stage I'm disappearing down some almighty rabbit hole?

I'm quite happy to reset back to basics and simply see if I can get the Plugins working using any other method - I certainly didn't want to become a monumental "time suck" that this has started to become ;-(

Invalid workflow file: .github/workflows/MakeDocsAction.yaml#L11
You have an error in your yaml syntax on line 11

So this appears to be cache: pip?

jobs:
  docs:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v2
      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: 3.x
          cache: pip
      - name: Setup
        run: |
          python -m pip install --upgrade pip
         pip install -r requirements.txt
      - name: Deploy docs
         run: |
           git fetch origin gh-pages --depth=1 || echo -e "skipping fetch"
           mkdocs gh-deploy

[athackst]

It's more likely the spacing errors on the lines below it.

jobs:
  docs:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v2
      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: 3.x
          cache: pip
      - name: Setup
        run: |
          python -m pip install --upgrade pip
          pip install -r requirements.txt
      - name: Deploy docs
        run: |
          git fetch origin gh-pages --depth=1 || echo -e "skipping fetch"
          mkdocs_simple_gen --deploy

I'm also pretty sure those git-xx plugins aren't compatible with this one. This plugin builds docs in a tmp directory which would strip all of the git info.

You may be better served just doing what mkdocs expects and placing everything in a /docs folder.

And yes, it's likely a rabbit hole. I'd try to build it locally if you can.

[dcaddick]

Thanks, stripped it all back & removed all PIP commands + removed all non-essential Plug-ins from reqs.txt
Now back to building in under 2 mins - I have "isolated the errant Plugin" to being either:
table-reader or mermaid2 ...?
One of those appears to "pull in" Pandas? & blows the time out or loops...

BTW - this is without enabling the plugins - just installing them.
To date that's where the error always seemed to get me - but I'll get out of this rabbit hole first

[timvink]

WTF needs pandas?

Hi, author of mkdocs-table-reader-plugin here, yes, that one uses pandas, which in turn uses numpy which in turn is not ready for python 3.11 yet.

That's a rather heavy-weight requirement for document building.

For document building yes, but I'm assuming when you're working with CSVs / table data and python that you are using pandas anyway. Also re-implementing all those table readers with all the supported options from scratch is not worth the effort for a lighter build..

This plugin builds docs in a tmp directory which would strip all of the git info.

Would it be possible to copy the .git folder to the tmp/ folder also? mkdocs-simple-plugin seems to be getting some traction (awesome!) and users are opening issues on my git plugins (example here). Happy to open a separate issue for this one if you want.

[athackst]

Would it be possible to copy the .git folder to the tmp/ folder also? mkdocs-simple-plugin seems to be getting some traction (awesome!) and users are opening issues on my git plugins (example timvink/mkdocs-git-authors-plugin#69). Happy to open a separate issue for this one if you want.

It's already possible to copy any folder directly into the output.

include_extensions uses a glob file pattern so .git/** should copy everything in the .git directory. I'm in the process of renaming that input so that functionality is more apparent.

I'm not positive that just doing that will work appropriately though.

I've already updated the docker to use ubuntu instead of alpine which should solve the compile issue #468

[dcaddick]

.git/**

I've just given that a try - but might have the syntax incorrect?

Deploying docs to gh-pages
ERROR - Config value 'plugins': Invalid Plugins configuration
Aborted with 1 Configuration Errors!
https://github.com/dcaddick/gsd_public/actions/runs/4119980229/jobs/7114245927

What I used was:

  #- simple
  - simple:
    include_extensions: [".git/**"]

I've also tried this based on the docs here:
https://www.althack.dev/mkdocs-simple-plugin/v2.2.0/examples/ok-with-macros/#mkdocsyml

  - simple:
    include_extensions:
        - ".git/**"

Same error :(

But by playing around with the spacing (instead of using Tab) I managed to get it to this error, so figuring I'm getting closer?
https://github.com/dcaddick/gsd_public/actions/runs/4120342620/jobs/7114977430

ERROR - Config value 'plugins': Plugin 'simple' option 'include_extensions': Expected type: <class 'list'> but received: <class 'str'>

  #- simple
  - simple:
      include_extensions: .git/**

[athackst]

hm, I'll check it out

[athackst]

#481 and #482 should help you out.

That looks like the right setting in config

[dcaddick]

I have this set in mkdocs.yml as:

  - simple:
      include_extensions:
      - ".git/**"

This now flows thru without error at Build time - however I'm not sure it's fixed the original problem?
When I look thru the logging it looks like there is still a problem with the git-authors-plugin?
And it looks like it's still being built in the tmp folder?

Have I managed to do something wrong?

INFO     -  Cleaning site directory
INFO     -  Building documentation to directory: /github/workspace/site
WARNING  -  [git-authors-plugin] /tmp/mkdocs_simple_4g5lfiyk/README.md has not been committed yet. Lines are not counted
WARNING  -  [git-authors-plugin] /tmp/mkdocs_simple_4g5lfiyk/ASD.md has not been committed yet. Lines are not counted
WARNING  -  [git-authors-plugin] /tmp/mkdocs_simple_4g5lfiyk/AssumedKnowledge.md has not been committed yet. Lines are not counted
WARNING  -  [git-authors-plugin] /tmp/mkdocs_simple_4g5lfiyk/Author.md has not been committed yet. Lines are not counted

[athackst]

Yeah, I'm pretty sure the copy into tmp is not going to work with the git plugin.

I have an experimental branch that uses on_files instead of changing the docs directory https://github.com/athackst/mkdocs-simple-plugin/tree/feat/on-files

You can try it in an action by pointing to the commit

jobs:
  publish:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v3
      - name: Build and push docs
        uses: athackst/mkdocs-simple-plugin@11e81727cdfac8b224da1bb486d7072da6c5cd5d

[dcaddick]

Thanks, tested and run the action here

https://github.com/dcaddick/gsd_public/actions/runs/4213295511/jobs/7312919312

It does appear to use the git, before having an error, looks like I need to step into trying to run this locally as a docker image later today

[athackst]

You should remove the include_extensions: [".git/**"] line from your mkdocs.yml file. With this branch you shouldn't need it.

[dcaddick]

Doh! Apologies

Rerun without error, but there does appear to be some indications that the git didn't come thru?

https://github.com/dcaddick/gsd_public/actions/runs/4213383482

[dcaddick]

Checking further in the published pages I can now see that at the bottom of each page it is not correctly reporting the "last update" date of each individual page.

It's still a little early on Sunday morning here in Perth, but will dive in a bit deeper today to have a look, thanks for your persistence

[athackst]

Looks like there's a deprecation warning on the git-authors-plugin. That is probably triggered by python 3.11. You could see if the author has an update for that one.

I'm not sure why the other one says it can't find git. Git should be installed in the dockerfile and accessible. None of the files are being moved so git should be reporting the correct status. My best guess is that it might be looking for the git folder a folder above where the docs folder is located? You might want to try adding 'docs_dir: docs' to your mkdocs.yml file.

[dcaddick]

This experimental branch looks good to me

> uses: athackst/mkdocs-simple-plugin@11e81727cdfac8b224da1bb486d7072da6c5cd5d

At the bottom of the pages I am now successfully seeing the created & updated date appear

[dcaddick]

Hi Allison,

Just checking as still pointing at the experimental branch - looks like you are testing Debian?
I guess I'll leave it a week or so before switching back to the updated default branch?

Regards,
Dave C