Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

nf-core download: Ability to add custom tags to revisions in downloaded pipelines #2938

Merged
merged 16 commits into from
May 7, 2024
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
6ab6c8d
Update Docstring of DownloadWorkflow class.
MatthiasZepper Apr 23, 2024
d2f8525
Include the '--additional_tags' argument in the CLI for 'nf-core down…
MatthiasZepper Apr 23, 2024
550da86
Started with additional_tags handling in the level of the DownloadWor…
MatthiasZepper Apr 23, 2024
620ca82
Implement a private __add_additional_tags() method for the WorkflowRe…
MatthiasZepper Apr 24, 2024
da0eab1
Experiment with capturing log messages in tests.
MatthiasZepper Apr 25, 2024
113c9d0
Finally managed to assert over log messages, thanks @adamrtalbot .
MatthiasZepper Apr 26, 2024
35b4fd6
Update the README.md to reflect the '-a' / '--additional-tags' argume…
MatthiasZepper Apr 26, 2024
3901f2d
Update CHANGELOG: Minor version jump, because CLI arguments changed.
MatthiasZepper Apr 26, 2024
00cb57e
Ensure a user is configured in Git config. Required for Github Action…
MatthiasZepper Apr 26, 2024
e089676
Catch exceptions if section is missing in the Git config.
MatthiasZepper Apr 29, 2024
6688079
Apply suggestions from code review
MatthiasZepper May 3, 2024
4d2e681
Update --additional-tag to --tag.
MatthiasZepper May 3, 2024
0b8e4ad
Change the e-mail from the nf-core bot e-mail to a sinkhole address.
MatthiasZepper May 6, 2024
3277b7c
Rearranging the sentences in README. Thanks @mashehu
MatthiasZepper May 6, 2024
1e632c2
Add deprecation warning for -t / --tower flag.
MatthiasZepper May 7, 2024
c95880f
Omit creating heads for additional tags due to UI bug in Platform.
MatthiasZepper May 7, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Update --additional-tag to --tag.
  • Loading branch information
MatthiasZepper committed May 6, 2024
commit 4d2e68148f75f876b429d03206cc0af98fa822c8
2 changes: 1 addition & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@

- Replace `--tower` with `--platform`. The former will remain for backwards compatability for now but will be removed in a future release. ([#2853](https://github.com/nf-core/tools/pull/2853))
- Better error message when GITHUB_TOKEN exists but is wrong/outdated
- New `-a` / `--additional-tag` argument to add custom tags during a pipeline download ([#2938](https://github.com/nf-core/tools/pull/2938))
- New `--tag` argument to add custom tags during a pipeline download ([#2938](https://github.com/nf-core/tools/pull/2938))

### Components

Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -437,7 +437,7 @@ Subsequently, the `*.git` folder can be moved to it's final destination and link
> [!TIP]
> Also without access to Seqera Platform, pipelines downloaded with the `--platform` flag can be run if the _absolute_ path is specified: `nextflow run -r 2.5 file:/path/to/pipelinedownload.git`. Downloads in this format allow you to include multiple revisions of a pipeline in a single file, but require that the revision (e.g. `-r 2.5`) is always explicitly specified.

Facilities and those who are setting up pipelines for others to use may find the `-a` / `--additional-tag` argument helpful. It must be followed by a string in a `key=value` format and can be provided multiple times. The `key` must refer to a valid branch, tag or commit SHA. The right-hand side must comply with the naming conventions for Git tags and may not yet exist in the repository. The `-a` / `--additional-tag` argument allows customizing the downloaded pipeline with additional tags that can be used to select particular revisions in the Seqera Platform interface. For example, an accredited facility may opt to tag particular revisions according to their structured release management process: `-a "3.12.0=testing" -a "3.9.0=validated"` so their staff can easily ensure that the correct version of the pipeline is run in production.
Facilities and those who are setting up pipelines for others to use may find the `--tag` argument helpful. It must be followed by a string in a `key=value` format and can be provided multiple times. The `key` must refer to a valid branch, tag or commit SHA. The right-hand side must comply with the naming conventions for Git tags and may not yet exist in the repository. The `--tag` argument allows customizing the downloaded pipeline with additional tags that can be used to select particular revisions in the Seqera Platform interface. For example, an accredited facility may opt to tag particular revisions according to their structured release management process: `--tag "3.12.0=testing" --tag "3.9.0=validated"` so their staff can easily ensure that the correct version of the pipeline is run in production.

## Pipeline software licences

Expand Down
11 changes: 5 additions & 6 deletions nf_core/__main__.py
Original file line number Diff line number Diff line change
Expand Up @@ -368,14 +368,14 @@ def create_params_file(pipeline, revision, output, force, show_hidden):
@click.option("-f", "--force", is_flag=True, default=False, help="Overwrite existing files")
# TODO: Remove this in a future release. Deprecated in March 2024.
@click.option(
"-t",
"--tower",
is_flag=True,
default=False,
hidden=True,
help="Download for Seqera Platform. DEPRECATED: Please use `--platform` instead.",
)
@click.option(
"-t",
"--platform",
is_flag=True,
default=False,
Expand All @@ -389,10 +389,9 @@ def create_params_file(pipeline, revision, output, force, show_hidden):
help="Include configuration profiles in download. Not available with `--platform`",
)
@click.option(
"-a",
"--additional-tags",
"--tag",
multiple=True,
help="Add custom alias tags to `--platform` downloads. For example, '-a \"3.10=validated\"' adds the custom 'validated' tag to the 3.10 release.",
help="Add custom alias tags to `--platform` downloads. For example, `--tag \"3.10=validated\"` adds the custom 'validated' tag to the 3.10 release.",
)
# -c changed to -s for consistency with other --container arguments, where it is always the first letter of the last word.
# Also -c might be used instead of -d for config in a later release, but reusing params for different options in two subsequent releases might be too error-prone.
Expand Down Expand Up @@ -436,7 +435,7 @@ def download(
tower,
platform,
download_configuration,
additional_tags,
tag,
container_system,
container_library,
container_cache_utilisation,
Expand All @@ -459,7 +458,7 @@ def download(
force,
tower or platform, # True if either specified
download_configuration,
additional_tags,
tag,
container_system,
container_library,
container_cache_utilisation,
Expand Down
19 changes: 11 additions & 8 deletions nf_core/download.py
Original file line number Diff line number Diff line change
Expand Up @@ -94,7 +94,7 @@ class DownloadWorkflow:
force (bool): Flag to force download even if files already exist (overwrite existing files). Defaults to False.
platform (bool): Flag to customize the download for Seqera Platform (convert to git bare repo). Defaults to False.
download_configuration (str): Download the configuration files from nf-core/configs. Defaults to None.
additional_tags (List[str]): Specify additional tags to add to the downloaded pipeline. Defaults to None.
tag (List[str]): Specify additional tags to add to the downloaded pipeline. Defaults to None.
container_system (str): The container system to use (e.g., "singularity"). Defaults to None.
container_library (List[str]): The container libraries (registries) to use. Defaults to None.
container_cache_utilisation (str): If a local or remote cache of already existing container images should be considered. Defaults to None.
Expand Down Expand Up @@ -1736,19 +1736,22 @@ def __add_additional_tags(self) -> None:
if self.repo.is_valid_object(anchor) and not self.repo.is_valid_object(tag):
try:
self.repo.create_tag(
tag, ref=anchor, message=f"Synonynmous tag to {anchor}; added by 'nf-core download'."
tag, ref=anchor, message=f"Synonynmous tag to {anchor}; added by `nf-core download`."
)
self.repo.create_head(tag, anchor) # should heads be created as well?
except (GitCommandError, InvalidGitRepositoryError) as e:
log.error(f"[red]Additional tag(s) could not be applied:[/]\n{e}\n")
else:
log.error(
f"[red]Adding the additional tag '{tag}' to '{anchor}' failed.[/]\n Mind that '{anchor}' must be a valid git reference that resolves to a commit, while '{tag}' must not exist hitherto."
)
if not self.repo.is_valid_object(anchor):
log.error(
f"[red]Adding tag '{tag}' to '{anchor}' failed.[/]\n Mind that '{anchor}' must be a valid git reference that resolves to a commit."
)
if self.repo.is_valid_object(tag):
log.error(
f"[red]Adding tag '{tag}' to '{anchor}' failed.[/]\n Mind that '{tag}' must not exist hitherto."
)
else:
log.error(
f"[red]Could not apply invalid '-a' / '--additional-tag' specification[/]: '{additional_tag}'"
)
log.error(f"[red]Could not apply invalid `--tag` specification[/]: '{additional_tag}'")

def bare_clone(self, destination):
if self.repo:
Expand Down
4 changes: 2 additions & 2 deletions tests/test_cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -168,7 +168,7 @@ def test_cli_download(self, mock_dl):
"force": None,
"platform": None,
"download-configuration": None,
"additional-tags": "3.12=testing",
"tag": "3.12=testing",
"container-system": "singularity",
"container-library": "quay.io",
"container-cache-utilisation": "copy",
Expand All @@ -189,7 +189,7 @@ def test_cli_download(self, mock_dl):
"force" in params,
"platform" in params,
"download-configuration" in params,
(params["additional-tags"],),
(params["tag"],),
params["container-system"],
(params["container-library"],),
params["container-cache-utilisation"],
Expand Down
37 changes: 20 additions & 17 deletions tests/test_download.py
Original file line number Diff line number Diff line change
Expand Up @@ -645,7 +645,23 @@ def test_download_workflow_for_platform(self, tmp_dir, _):
) # indirect definition via $container variable.

#
# Test adding custom tags to Seqera Platform download
# Brief test adding a single custom tag to Seqera Platform download
#
@mock.patch("nf_core.download.DownloadWorkflow.get_singularity_images")
@with_temporary_folder
def test_download_workflow_for_platform_with_one_custom_tag(self, _, tmp_dir):
download_obj = DownloadWorkflow(
pipeline="nf-core/rnaseq",
revision=("3.9"),
compress_type="none",
platform=True,
container_system=None,
additional_tags=("3.9=cool_revision",),
)
assert isinstance(download_obj.additional_tags, list) and len(download_obj.additional_tags) == 1

#
# Test adding custom tags to Seqera Platform download (full test)
#
@mock.patch("nf_core.download.DownloadWorkflow.get_singularity_images")
@with_temporary_folder
Expand Down Expand Up @@ -707,21 +723,8 @@ def test_download_workflow_for_platform_with_custom_tags(self, _, tmp_dir):
assert all(
log in self.logged_messages
for log in {
"[red]Could not apply invalid '-a' / '--additional-tag' specification[/]: '3.9=invalid tag'",
"[red]Adding the additional tag 'not_included' to '3.14.0' failed.[/]\n Mind that '3.14.0' must be a valid git reference that resolves to a commit, while 'not_included' must not exist hitherto.",
"[red]Could not apply invalid '-a' / '--additional-tag' specification[/]: 'What is this?'",
"[red]Could not apply invalid `--tag` specification[/]: '3.9=invalid tag'",
"[red]Adding tag 'not_included' to '3.14.0' failed.[/]\n Mind that '3.14.0' must be a valid git reference that resolves to a commit.",
"[red]Could not apply invalid `--tag` specification[/]: 'What is this?'",
}
)

@mock.patch("nf_core.download.DownloadWorkflow.get_singularity_images")
@with_temporary_folder
def test_download_workflow_for_platform_with_one_custom_tag(self, _, tmp_dir):
download_obj = DownloadWorkflow(
pipeline="nf-core/rnaseq",
revision=("3.9"),
compress_type="none",
platform=True,
container_system=None,
additional_tags=("3.9=cool_revision",),
)
assert isinstance(download_obj.additional_tags, list) and len(download_obj.additional_tags) == 1