Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create specific binder env for each scikit-decide release #131

Merged
merged 3 commits into from
Dec 7, 2021

Conversation

nhuet
Copy link
Contributor

@nhuet nhuet commented Nov 30, 2021

Summary:
During release.yml workflow, we push a new tag defining the corresponding binder environment, then build and cache it on mybinder.org. Finally the binder links in the release doc point to this specific environment.

Hypotheses:

  • a branch named "binder" exists on the repository with an environment.yml file used to define a binder env
  • the branch is up-to-date with necessary dependencies to make work the notebooks on master with master version of scikit-decide
  • environment.yml file contains the dependency in scikit-decide, and it follows the format scikit-decide[all]==...

Process:

  • we checkout binder branch
  • we replace the spec for scikit-decide by the scikit-decide version corresponding to the release triggering the workflow
  • we commit and push a tag named "binder_vx.x.x" where x.x.x stands for the inferred version number
  • we force a build of the environment on mybinder.org to accelerate later call to it
  • we use this binder env for links in the doc for the release

Note: we used a quite verbose way to define the BINDER_TAG variable. This is to make it clear that we create it exactly the same way in both jobs "create-binder-env" and "build-docs" , as there is no simple way to share variables between jobs (without using artifacts).

A fake release v0.9.3 has been created on nhuet/scikit-decide to test that the binder environment is well defined.
When the workflow will be finished, we should have a tag "binder_v0.9.3" (see https://github.com/nhuet/scikit-decide/tree/binder_v0.9.3 ) defining the binder environment with scikit-decide 0.9.3 (from pypi), and the environment should have been prebuilt on mybinder.org. Test this link: Binder

@nhuet nhuet mentioned this pull request Nov 30, 2021
.github/workflows/release.yml Outdated Show resolved Hide resolved
.github/workflows/release.yml Outdated Show resolved Hide resolved
.github/workflows/release.yml Outdated Show resolved Hide resolved
.github/workflows/release.yml Outdated Show resolved Hide resolved
nhuet added 3 commits December 6, 2021 10:54
Summary:
During release.yml workflow, we push a new tag  defining the
corresponding binder environment, then build and cache it
on mybinder.org. Finally the binder links in the release doc point to
this specific environment.

Hypotheses:
- a branch named "binder" exists on origin with environment.yml file
  used to define a binder env
- the branch is up-to-date with necessary dependencies to make work the
  notebooks on master with master version of scikit-decide
- environment.yml file contains the dependency in scikit-decide, and it
  follows the format `scikit-decide[all]==...`

Process:
- we checkout binder branch
- we replace the spec for scikit-decide by the scikit-decide version
  corresponding to the release triggering the workflow
- we commit and push a tag named "binder_vx.x.x" where x.x.x stands for
  the inferred version number
- we force a build of the environment on mybinder.org to accelerate
  later call to it
- we use this binder env for links in the doc for the release

Note:  used a quite verbose way to define the BINDER_TAG variable.
This is to make it clear that we create it exactly the same way in both jobs
"create-binder-env" and "build-docs" , as there is no simple way to
share variables between jobs (without using artifacts).
- Merge jobs: having binder creation in same job as docs build
  simplify passing variables between steps
- Use Actions <actions@github.com> as identity for committing
- Get a straight history in binder branch by committing the updated
  scikit-decide spec then reverting
- Use the sha1 to reference the new binder env to avoid creating
  unnecessary tag (we mention the release in the commit message to know
  which environment the commit is used for)
- Fix the binder env repository to current one (instead of hard-coded
  airbus one) since this is the binder branch of the current one that is
  modified. Thus the release workflow can be tested on forks.
- Remove caching oin mybinder.org. Will be done later.
@nhuet nhuet force-pushed the nh/ci_create_binder_env_on_release branch from adc6f18 to 7d3c8e3 Compare December 6, 2021 09:54
@nhuet
Copy link
Contributor Author

nhuet commented Dec 6, 2021

I took into account @dbarbier comments. I force rebased in order to take account the correction that make doc building works (fe50cb9).

Copy link
Contributor

@dbarbier dbarbier left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. Did you test it on your repo after these changes?

@nhuet
Copy link
Contributor Author

nhuet commented Dec 7, 2021

Yes, i retag a v0.9.3 on my fork. The binder env has been generated (but is now on a binder_test branch as i rebased my binder branch for PR #139) and my online doc is pointing on the proper sha1. See notebooks page here: https://nhuet.github.io/scikit-decide/version/0.9.3/notebooks/

@nhuet
Copy link
Contributor Author

nhuet commented Dec 7, 2021

Actually i just tried to follow the links in doc (thus pointing to the binder env corresponding to the appropriate sha1), and

  • it works
  • but it rebuild each time => does that mean that pointing to a sha1 prevent binder to understand that it has already been built?
  • the second build is still very short because docker has a lot of things in cache => this is because i did it in a row, but i am not sure tomorrow it will be the same. While pointing to a branch or a tag (instead of a sha1) seemed to make binder recognize it as already built and only pull the appropriate docker image from the hub.

@nhuet
Copy link
Contributor Author

nhuet commented Dec 7, 2021

  • but it rebuild each time => does that mean that pointing to a sha1 prevent binder to understand that it has already been built?

Nevermind. I tried a 3rd times, and it found the build and only pulled the image. Maybe i was unlucky. It may be due to the fact that mybinder.org is actually deploying binderhub on several cloud infrastructure (cf https://mybinder.readthedocs.io/en/latest/about/federation.html#the-binderhub-federation)

@galleon galleon merged commit 4c4f004 into airbus:master Dec 7, 2021
@nhuet nhuet deleted the nh/ci_create_binder_env_on_release branch December 12, 2023 08:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants