Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add context command #1655

Merged
merged 36 commits into from
Nov 15, 2023
Merged
Show file tree
Hide file tree
Changes from 33 commits
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
818448b
WIP: Begin sketching context command
JimMadge Oct 26, 2023
5817e65
WIP: Adjust context settings class
JimMadge Oct 26, 2023
5e6726d
WIP: Add/Update methods
JimMadge Oct 27, 2023
3c2ad43
WIP: Add schema and from_file classmethod
JimMadge Oct 27, 2023
5603417
WIP: Correct constructor
JimMadge Oct 27, 2023
608a3d3
WIP: Add tests
JimMadge Oct 27, 2023
f3ba5a5
Add tests workflow
JimMadge Oct 27, 2023
a1feb76
WIP: Add available method
JimMadge Oct 27, 2023
11fdc19
WIP: Add tests for from_file and write methods
JimMadge Oct 30, 2023
5a79eaf
Add add method
JimMadge Oct 30, 2023
f6071f5
WIP: Add test for invalid context selection
JimMadge Oct 30, 2023
e120990
WIP: Add invalid add test
JimMadge Oct 30, 2023
38fd39d
Add remove method
JimMadge Oct 30, 2023
03456b4
Add context show command
JimMadge Oct 31, 2023
3981a83
Add context switch command
JimMadge Oct 31, 2023
5f62656
Add context add command
JimMadge Oct 31, 2023
a64a1e2
Add remove command
JimMadge Oct 31, 2023
8da10c5
Add context available command
JimMadge Oct 31, 2023
e6d6017
Add context update command
JimMadge Oct 31, 2023
afd2001
Run lint:fmt
JimMadge Oct 31, 2023
eedd004
Correct typing
JimMadge Nov 1, 2023
35dc59f
WIP: use available method
JimMadge Nov 2, 2023
bb144b0
WIP: Correct workflow step name
JimMadge Nov 2, 2023
e6c8499
WIP: Tidy entrypoints
JimMadge Nov 2, 2023
e331299
WIP: Rename 'backend' to context
JimMadge Nov 2, 2023
d12c403
Add tests for create and teardown commands
JimMadge Nov 2, 2023
087ec8d
Sort import block
JimMadge Nov 2, 2023
67b1ff4
Update renamed configuration section references
JimMadge Nov 2, 2023
1d85e25
Tidy ContextSettings tests with fixtures
JimMadge Nov 2, 2023
7805f45
Disable some rich features during tests
JimMadge Nov 3, 2023
ac99832
Allow bootstrapping context settings file
JimMadge Nov 3, 2023
878573c
Add missing help messages
JimMadge Nov 3, 2023
4bd4914
Update README
JimMadge Nov 3, 2023
b72874e
Remove old instructions
JimMadge Nov 7, 2023
6b31ea4
Update data_safe_haven/commands/context.py
JimMadge Nov 7, 2023
84dd4e2
Update argument names and command descriptions
JimMadge Nov 7, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 11 additions & 11 deletions .github/workflows/test_code.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -4,24 +4,24 @@ name: Test code
# Run workflow on pushes to matching branches
on: # yamllint disable-line rule:truthy
push:
branches: [develop]
branches: [develop, python-migration]
pull_request:
branches: [develop]
branches: [develop, python-migration]

jobs:
test_powershell:
test_python:
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v3
- name: Install requirements
shell: pwsh
run: |
Set-PSRepository PSGallery -InstallationPolicy Trusted
deployment/CheckRequirements.ps1 -InstallMissing -IncludeDev
- name: Test PowerShell
shell: pwsh
run: ./tests/Run_Pester_Tests.ps1
- name: Set up Python
uses: actions/setup-python@v4
with:
python-version: 3.11
- name: Install hatch
run: pip install hatch
- name: Test Python
run: hatch run test:test

test_markdown_links:
runs-on: ubuntu-latest
Expand Down
30 changes: 16 additions & 14 deletions data_safe_haven/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,16 +9,17 @@ Install the following requirements before starting

- Run the following to initialise the deployment [approx 5 minutes]:

```bash
> dsh init
```console
> dsh context add ...
> dsh context create
```

You will be prompted for various project settings.
JimMadge marked this conversation as resolved.
Show resolved Hide resolved
If you prefer to enter these at the command line, run `dsh init -h` to see the necessary command line flags.

- Next deploy the Safe Haven Management (SHM) infrastructure [approx 30 minutes]:

```bash
```console
> dsh deploy shm
```

Expand All @@ -29,13 +30,13 @@ Run `dsh deploy shm -h` to see the necessary command line flags and provide them
Note that the phone number must be in full international format.
Note that the country code is the two letter `ISO 3166-1 Alpha-2` code.

```bash
```console
> dsh admin add-users <my CSV users file>
```

- Next deploy the infrastructure for one or more Secure Research Environments (SREs) [approx 30 minutes]:

```bash
```console
> dsh deploy sre <SRE name>
```

Expand All @@ -44,7 +45,7 @@ Run `dsh deploy sre -h` to see the necessary command line flags and provide them

- Next add one or more existing users to your SRE

```bash
```console
> dsh admin register-users -s <SRE name> <username1> <username2>
```

Expand All @@ -54,44 +55,45 @@ where you must specify the usernames for each user you want to add to this SRE

- Run the following to list the currently available users

```bash
```console
> dsh admin list-users
```

## Removing a deployed Data Safe Haven

- Run the following if you want to teardown a deployed SRE:

```bash
```console
> dsh teardown sre <SRE name>
```

- Run the following if you want to teardown the deployed SHM:

```bash
```console
> dsh teardown shm
```

- Run the following if you want to teardown the deployed Data Safe Haven backend:
- Run the following if you want to teardown the deployed Data Safe Haven context:

```bash
> dsh teardown backend
```console
> dsh context teardown
```

## Code structure

- administration
- this is where we keep utility commands for adminstrators of a deployed DSH
- eg. "add a user"; "remove a user from an SRE"
- backend
- context
- in order to use the Pulumi Azure backend we need a KeyVault, Identity and Storage Account
- this code deploys those resources to bootstrap the rest of the Pulumi-based code
- the storage account is also used to store configuration, so that it can be shared by admins
- commands
- the main `dsh` command line entrypoint lives in `cli.py`
- the subsidiary `typer` command line entrypoints (eg. `dsh deploy shm`) live here
- config
- serialises and deserialises a config file from Azure
- `backend_settings` manages basic settings related to the Azure backend: arguably this could/should live in `backend`
- `context_settings` manages basic settings related to the context: arguably this could/should live in `context`
- exceptions
- definitions of a Python exception hierarchy
- external
Expand Down
5 changes: 0 additions & 5 deletions data_safe_haven/backend/__init__.py

This file was deleted.

10 changes: 4 additions & 6 deletions data_safe_haven/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,8 @@
from data_safe_haven import __version__
from data_safe_haven.commands import (
admin_command_group,
context_command_group,
deploy_command_group,
initialise_command,
teardown_command_group,
)
from data_safe_haven.exceptions import DataSafeHavenError
Expand Down Expand Up @@ -69,6 +69,9 @@ def main() -> None:
name="admin",
help="Perform administrative tasks for a Data Safe Haven deployment.",
)
application.add_typer(
context_command_group, name="context", help="Manage Data Safe Haven contexts."
)
application.add_typer(
deploy_command_group,
name="deploy",
Expand All @@ -80,11 +83,6 @@ def main() -> None:
help="Tear down a Data Safe Haven component.",
)

# Register direct subcommands
application.command(name="init", help="Initialise a Data Safe Haven deployment.")(
initialise_command
)

# Start the application
try:
application()
Expand Down
4 changes: 2 additions & 2 deletions data_safe_haven/commands/__init__.py
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
from .admin import admin_command_group
from .context import context_command_group
from .deploy import deploy_command_group
from .init import initialise_command
from .teardown import teardown_command_group

__all__ = [
"admin_command_group",
"context_command_group",
"deploy_command_group",
"initialise_command",
"teardown_command_group",
]
174 changes: 174 additions & 0 deletions data_safe_haven/commands/context.py
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like this in general but it raises quite a few questions. Some of this is me perhaps overthinking it worrying about ways people might mess things up, and wondering how much we should stop them from doing that.
The docs/purpose of the functions could use a bit of clarification and checking for consistency.

We start by adding a context. When adding a context, you give a key - the command is dsh context add [OPTIONS] KEY - which is described as the "name of the context to add". But there's also a --name argument, which is the "human friendly" name of the deployment. For some functions you then supply the key, like dsh context remove. But for dsh context switch you supply the name, which is also described in the help as the name of the context to switch to. So sometimes KEY is the name of the context, and sometimes NAME is the name of the context. For dsh context switch you actually need to supply the key rather than the name.

  1. I assume you're separating key and name deliberately - is it helpful to separate the context name and the deployment name, or is it better to just give a single name? e.g. You could set up two otherwise identical contexts with different keys, but why?
  2. I guess these will be addressed when we do the complete website docs, but the help info for dsh context create and dsh context update could provide a bit more info about what they do.
  3. Is it possible to get Typer to complain if no flags are passed? e.g. dsh context update does nothing if no flags are set.
  4. What is the workflow after updating the context? Suppose I update the location from uksouth to ukwest - what should happen when I run another command? If I already have something deployed, should it then deploy new things in the new location? At the moment, running, say, dsh context create after updating the location actually just seems to check the existing location -
image
  1. Could you add short form flags for each argument to the various context commands, as existed for init?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed about name/key and update confusion.
I've updated the help descriptions and argument names in 84dd4e2
Do you think that is enough? I think we need to balance keeping the CLI help messages brief, and can put a more detailed explanation in the docs.

You could set up two otherwise identical contexts with different keys, but why?

That is true, it isn't the intention but we don't prevent a user from making conflicting/overlapping contexts.
We could do that, but I think that would want to be a new issue as it is a new feature.

A context has a parameter called "name" and we refer to contexts by their keys, which must be unique. You could use a parameter of each context, but it would be awkward (and more expensive) to select them [c if c.name == name for c in contexts][0] vs contexts[key]. And you would add the additional restriction that names also need to be unique, even if they are deployed to different subscriptions/locations.

Is it possible to get Typer to complain if no flags are passed? e.g. dsh context update does nothing if no flags are set.

Probably, or at least you could raise an exception if all are None. I'm not sure if I see a strong advantage, I think saying "update nothing" and nothing changing makes sense.

What is the workflow after updating the context? Suppose I update the location from uksouth to ukwest - what should happen when I run another command? If I already have something deployed, should it then deploy new things in the new location? At the moment, running, say, dsh context create after updating the location actually just seems to check the existing location

I think that must also have been true before as I haven't touched that code.
It would be nice if context create were idempotent, but as we are using the Azure Python SDK if would be a fair bit of effort.
I've updated the description to say that update updates the settings specifically.
If we want to change/explain this I think it can be a new issue.

Could you add short form flags for each argument to the various context commands, as existed for init?

Could do. I intentionally didn't though.
I feel that the choices have fairly large consequences, so using clear names rather than letters (or even worse, positional arguments!) helps make sure we understand what each argument is.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed about name/key and update confusion. I've updated the help descriptions and argument names in 84dd4e2 Do you think that is enough? I think we need to balance keeping the CLI help messages brief, and can put a more detailed explanation in the docs.

Helps!

You could set up two otherwise identical contexts with different keys, but why?

That is true, it isn't the intention but we don't prevent a user from making conflicting/overlapping contexts. We could do that, but I think that would want to be a new issue as it is a new feature.

A context has a parameter called "name" and we refer to contexts by their keys, which must be unique. You could use a parameter of each context, but it would be awkward (and more expensive) to select them [c if c.name == name for c in contexts][0] vs contexts[key]. And you would add the additional restriction that names also need to be unique, even if they are deployed to different subscriptions/locations.

More awkward/expensive may be true, but cost doesn't seem like an issue here really - it's just searching through a very small list, once in a while. In the context of a deployment, it's the blink of an eye.

Is it possible to get Typer to complain if no flags are passed? e.g. dsh context update does nothing if no flags are set.

Probably, or at least you could raise an exception if all are None. I'm not sure if I see a strong advantage, I think saying "update nothing" and nothing changing makes sense.

Perhaps it's me - I feel like it should be explicit that nothing is changing. Why write a new config file if nothing has changed? Not something I feel strongly about I guess.

What is the workflow after updating the context? Suppose I update the location from uksouth to ukwest - what should happen when I run another command? If I already have something deployed, should it then deploy new things in the new location? At the moment, running, say, dsh context create after updating the location actually just seems to check the existing location

I think that must also have been true before as I haven't touched that code. It would be nice if context create were idempotent, but as we are using the Azure Python SDK if would be a fair bit of effort. I've updated the description to say that update updates the settings specifically. If we want to change/explain this I think it can be a new issue.

I still just think it's quite confusing. If I run dsh context add <foo>, then run dsh context create, then run dsh context update <foo>, then run dsh context create again, nothing changes:

image

It reads project settings from the context file, but does nothing with them. It just uses the config that already exists without changing it, then re-uploads it at the end, unchanged. So it seems dsh context update is only useful before dsh context create. Which isn't necessarily a problem, just, again, feel like it should be more explicit.

Could you add short form flags for each argument to the various context commands, as existed for init?

Could do. I intentionally didn't though. I feel that the choices have fairly large consequences, so using clear names rather than letters (or even worse, positional arguments!) helps make sure we understand what each argument is.

On reflection, I think I'm ok with this given we're mostly expecting these arguments to get recorded in config files now anyway.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, can you give a steer on what (if anything) you think should be changed at how?

Some of these things are (I feel) outside the scope of this issue, being in code that wasn't touched here. We can open issues if we feel like those are worth doing though.

I'm fairly strongly objected to removing keys for contexts. Removing a unique identifier feels like asking for trouble.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So I think what I'm trying to get my head round is what is changing the local config file meant to achieve? So if the purpose here is just to get a command in that changes the local config file, the code as is undoubtedly does that.

Whether what I'm wondering about it involves changes to this code or other code depends a little on what changing the config file is supposed to achieve. Currently the only use for changing the local config file seems to be changing it when nothing is deployed or before tearing down and redeploying existing resources.

I feel like either this code should say something like "You already have resources deployed, changes made here will not have an effect until the resources are deleted and redeployed", or the other code should say something like "The local config differs from existing resources, maybe you should redeploy". Or perhaps these are things that should be in accompanying documentation.

I think I'm just trying to work out when it is meant to be useful to change the local config file in this way, and thus whether it's useful for the new code here to explicitly inform people that what they are doing needs additional steps to be actually useful.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So if the purpose here is just to get a command in that changes the local config file, the code as is undoubtedly does that.

Yes, just that and nothing more 😄.

The way I'm seeing this is like git, git remotes in particular.
The context configuration file is a specification of what contexts exist and enough information to connect to them.
Similar to how git remotes are a record of repositories.
I think it is also important to understand that these settings are only for the context infrastructure, not for the TRE itself.

I don't think we should want editing the record of contexts to make changes to any infrastructure, or create warnings?
Also, I'm not certain it is possible.
How do you know if resources were already deployed or if local differs from remote?
All the program knows is the data in the contexts file.

I could see the argument to just remove the update command as it might not be much use.
Maybe it is only really useful in some rare situations like "I made one mistake when running dsh context create" or "I decided to change a parameter before deploying the context infrastructure".

Original file line number Diff line number Diff line change
@@ -0,0 +1,174 @@
"""Command group and entrypoints for managing a DSH context"""
from typing import Annotated, Optional

import typer
from rich import print

from data_safe_haven.config import Config, ContextSettings
from data_safe_haven.config.context_settings import default_config_file_path
from data_safe_haven.context import Context
from data_safe_haven.functions import validate_aad_guid

context_command_group = typer.Typer()


@context_command_group.command()
def show() -> None:
"""Show information about the selected context."""
settings = ContextSettings.from_file()

current_context_key = settings.selected
current_context = settings.context

print(f"Current context: [green]{current_context_key}")
print(f"\tName: {current_context.name}")
print(f"\tAdmin Group ID: {current_context.admin_group_id}")
print(f"\tSubscription name: {current_context.subscription_name}")
print(f"\tLocation: {current_context.location}")


@context_command_group.command()
def available() -> None:
"""Show the available contexts."""
settings = ContextSettings.from_file()

current_context_key = settings.selected
available = settings.available

available.remove(current_context_key)
available = [f"[green]{current_context_key}*[/]", *available]

print("\n".join(available))


@context_command_group.command()
def switch(
name: Annotated[str, typer.Argument(help="Name of the context to switch to.")]
) -> None:
"""Switch the context."""
settings = ContextSettings.from_file()
settings.selected = name
settings.write()


@context_command_group.command()
def add(
key: Annotated[str, typer.Argument(help="Name of the context to add.")],
admin_group: Annotated[
str,
typer.Option(
help="The ID of an Azure group containing all administrators.",
callback=validate_aad_guid,
),
],
location: Annotated[
str,
typer.Option(
help="The Azure location to deploy resources into.",
),
],
name: Annotated[
str,
typer.Option(
help="The human friendly name to give this Data Safe Haven deployment.",
),
],
subscription: Annotated[
str,
typer.Option(
help="The name of an Azure subscription to deploy resources into.",
),
],
) -> None:
"""Add a new context."""
if default_config_file_path().exists():
settings = ContextSettings.from_file()
settings.add(
key=key,
admin_group_id=admin_group,
location=location,
name=name,
subscription_name=subscription,
)
else:
# Bootstrap context settings file
settings = ContextSettings(
{
"selected": key,
"contexts": {
key: {
"admin_group_id": admin_group,
"location": location,
"name": name,
"subscription_name": subscription,
}
},
}
)
settings.write()


@context_command_group.command()
def update(
admin_group: Annotated[
Optional[str], # noqa: UP007
typer.Option(
help="The ID of an Azure group containing all administrators.",
callback=validate_aad_guid,
),
] = None,
location: Annotated[
Optional[str], # noqa: UP007
typer.Option(
help="The Azure location to deploy resources into.",
),
] = None,
name: Annotated[
Optional[str], # noqa: UP007
typer.Option(
help="The human friendly name to give this Data Safe Haven deployment.",
),
] = None,
subscription: Annotated[
Optional[str], # noqa: UP007
typer.Option(
help="The name of an Azure subscription to deploy resources into.",
),
] = None,
) -> None:
"""Update the selected context."""
settings = ContextSettings.from_file()
settings.update(
admin_group_id=admin_group,
location=location,
name=name,
subscription_name=subscription,
)
settings.write()


@context_command_group.command()
def remove(
key: Annotated[str, typer.Argument(help="Name of the context to add.")],
JimMadge marked this conversation as resolved.
Show resolved Hide resolved
) -> None:
"""Remove the selected context."""
settings = ContextSettings.from_file()
settings.remove(key)
settings.write()


@context_command_group.command()
def create() -> None:
"""Create Data Safe Haven context infrastructure."""
config = Config()
context = Context(config)
context.create()
context.config.upload()


@context_command_group.command()
def teardown() -> None:
"""Tear down Data Safe Haven context infrastructure."""
config = Config()
context = Context(config)
context.teardown()
Loading
Loading