Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use Pydantic for validation and serialisation #1661

Merged
merged 67 commits into from
Jan 19, 2024
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
67 commits
Select commit Hold shift + click to select a range
f3acf72
WIP: Use Pydantic for context settings
JimMadge Nov 7, 2023
12688ea
WIP: Fix ContextSettings tests
JimMadge Nov 7, 2023
562a6b4
Fix context command tests
JimMadge Nov 7, 2023
026f14a
Ensure selected context is defined
JimMadge Nov 7, 2023
aed1a86
Remove schema typings
JimMadge Nov 7, 2023
1abe112
Add test for invalid YAML
JimMadge Nov 13, 2023
7f314d5
Add test for YAML not being a dict
JimMadge Nov 13, 2023
d22bbd1
Add file not found test
JimMadge Nov 13, 2023
502fe5b
Fix linting errors
JimMadge Nov 13, 2023
9093b97
Add Pydantic to linting environment
JimMadge Nov 14, 2023
d374b1c
WIP: Introduce annotated types for validation
JimMadge Nov 15, 2023
7907e0c
WIP: Add Typer validators factory
JimMadge Nov 15, 2023
36f0e1c
Validate Context and ContextSettings on assignment
JimMadge Nov 15, 2023
1136b49
WIP: rewrite Config (and related) as BaseModel
JimMadge Nov 20, 2023
515668e
Update Context tests
JimMadge Nov 21, 2023
a741542
Remove ConfigSectionContext
JimMadge Nov 21, 2023
acb01b8
Improve design of config sections
JimMadge Nov 21, 2023
e7ca180
Add missing tests
JimMadge Nov 22, 2023
e8e2e5b
Add basic tests for Config
JimMadge Nov 22, 2023
8f98a9e
Add to_yaml method
JimMadge Nov 23, 2023
bde5fd4
Add upload and from remote tests
JimMadge Nov 23, 2023
e908111
Run lint:fmt
JimMadge Nov 23, 2023
e44acd6
Tidy Config
JimMadge Nov 23, 2023
6344a92
Correct sre method, add tests
JimMadge Nov 23, 2023
bf9435e
Dynamically derive config items from context
JimMadge Nov 24, 2023
c0002f6
Dynamically construct ConfigSectionTags
JimMadge Nov 24, 2023
bd358dc
Replace tags to_dict with model_dump
JimMadge Nov 24, 2023
e6eb86f
Add missing type annotation
JimMadge Nov 27, 2023
09e8427
Correct config attributes
JimMadge Nov 27, 2023
adbb464
Add missing properties to context
JimMadge Nov 27, 2023
c040e47
Update minimum Python to 3.11
JimMadge Nov 28, 2023
ba05dd7
Correct typer validator factory
JimMadge Nov 29, 2023
e1a0264
Support optional args in typer validator factory
JimMadge Nov 29, 2023
7a0cab8
Move Pulumi encryption key to Config property
JimMadge Nov 28, 2023
4727c32
Add template method
JimMadge Nov 28, 2023
36d05f3
Add config template command
JimMadge Nov 28, 2023
681379e
Add config upload command
JimMadge Nov 29, 2023
82a8d17
Add config show command
JimMadge Nov 29, 2023
b25f977
Run lint:fmt
JimMadge Nov 29, 2023
3235983
Remove dependency on config for context
JimMadge Nov 29, 2023
db7855a
Use new Config in deploy commands
JimMadge Nov 30, 2023
5fe21f8
Update Pulumi version draft README
JimMadge Nov 30, 2023
e6fb404
Use new config in teardown commands
JimMadge Dec 12, 2023
56ec257
Rename context infrastructure class
JimMadge Dec 12, 2023
9fd210c
Correct references
JimMadge Dec 12, 2023
829d7ac
Use new Config in admin commands
JimMadge Dec 12, 2023
16e73ff
Add help text for config show command
JimMadge Dec 14, 2023
c9ceb39
Fix import
JimMadge Dec 15, 2023
9b91daf
Allow no selected context
JimMadge Dec 15, 2023
9cea20e
Add test for constructor with no selected context
JimMadge Dec 15, 2023
c2f3de5
Add assert context tests
JimMadge Dec 15, 2023
4d291a1
Add test for selecting no context
JimMadge Dec 15, 2023
9321eb8
Add update test for no selected context
JimMadge Dec 15, 2023
8576f50
Add test for removing selected context
JimMadge Dec 15, 2023
3486091
Allow showing no selected context
JimMadge Dec 15, 2023
512f386
Add test for available with no selected context
JimMadge Dec 15, 2023
430926b
Add fqdn validation
JimMadge Jan 15, 2024
7d6c50d
Fix invalid fqdn in tests
JimMadge Jan 15, 2024
3990fdc
Fix test failing due to env variable
JimMadge Jan 15, 2024
26e64e0
Merge pull request #1691 from alan-turing-institute/selected
JimMadge Jan 15, 2024
625eb47
Merge pull request #1674 from alan-turing-institute/config
JimMadge Jan 15, 2024
1493dcb
Run lint:fmt
JimMadge Jan 15, 2024
50d83a9
Print Ruff version in CI
JimMadge Jan 15, 2024
672d7df
Fix linting errors
JimMadge Jan 15, 2024
3b5d552
Add fqdn typings
JimMadge Jan 16, 2024
2b9b268
Add typing hint
JimMadge Jan 16, 2024
ed4ec99
Update data_safe_haven/README.md
JimMadge Jan 19, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Dynamically derive config items from context
This simplifies the construction of a Config object and makes the
serialised configuration less verbose and redundant.
  • Loading branch information
JimMadge committed Nov 24, 2023
commit bf9435eea3ef534b94b3b6c79a5b63b5b5e63dd1
64 changes: 24 additions & 40 deletions data_safe_haven/config/config.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
from __future__ import annotations

from pathlib import Path
from typing import ClassVar
from typing import Any, ClassVar

import yaml
from pydantic import (
Expand Down Expand Up @@ -42,20 +42,14 @@


class ConfigSectionAzure(BaseModel, validate_assignment=True):
admin_group_id: Guid
location: AzureLocation
admin_group_id: Guid = Field(exclude=True)
location: AzureLocation = Field(exclude=True)
subscription_id: Guid
tenant_id: Guid

@classmethod
def from_context(
cls, context: Context, subscription_id: Guid, tenant_id: Guid
) -> ConfigSectionAzure:
return ConfigSectionAzure(
admin_group_id=context.admin_group_id,
location=context.location,
subscription_id=subscription_id,
tenant_id=tenant_id,
def __init__(self, context: Context, **kwargs: dict[Any, Any]):
super().__init__(
admin_group_id=context.admin_group_id, location=context.location, **kwargs
)


Expand All @@ -71,27 +65,11 @@ class ConfigSectionSHM(BaseModel, validate_assignment=True):
admin_email_address: EmailAdress
admin_ip_addresses: list[IpAddress]
fqdn: str
name: str
name: str = Field(exclude=True)
timezone: TimeZone

@classmethod
def from_context(
cls,
context: Context,
aad_tenant_id: Guid,
admin_email_address: EmailAdress,
admin_ip_addresses: list[IpAddress],
fqdn: str,
timezone: TimeZone,
) -> ConfigSectionSHM:
return ConfigSectionSHM(
aad_tenant_id=aad_tenant_id,
admin_email_address=admin_email_address,
admin_ip_addresses=admin_ip_addresses,
fqdn=fqdn,
name=context.shm_name,
timezone=timezone,
)
def __init__(self, context: Context, **kwargs: dict[Any, Any]):
super().__init__(name=context.shm_name, **kwargs)

def update(
self,
Expand Down Expand Up @@ -250,25 +228,24 @@ def update(


class ConfigSectionTags(BaseModel, validate_assignment=True):
deployment: str
deployment: str = Field(exclude=True)
deployed_by: ClassVar[str] = "Python"
project: ClassVar[str] = "Data Safe Haven"
version: ClassVar[str] = __version__

@classmethod
def from_context(cls, context: Context) -> ConfigSectionTags:
return ConfigSectionTags(deployment=context.name)
def __init__(self, context: Context, **kwargs: dict[Any, Any]):
super().__init__(deployment=context.name, **kwargs)


class Config(BaseModel, validate_assignment=True):
azure: ConfigSectionAzure | None = None
context: Context
context: Context = Field(exclude=True)
pulumi: ConfigSectionPulumi | None = None
shm: ConfigSectionSHM | None = None
tags: ConfigSectionTags | None = None
sres: dict[str, ConfigSectionSRE] = Field(
default_factory=dict[str, ConfigSectionSRE]
)
tags: ConfigSectionTags | None = Field(exclude=True, default=None)

@property
def work_directory(self) -> str:
Expand All @@ -286,7 +263,7 @@ def sre(self, name: str) -> ConfigSectionSRE:
"""Return the config entry for this SRE creating it if it does not exist"""
if name not in self.sres.keys():
highest_index = max(0 + sre.index for sre in self.sres.values())
self.sres[name] = ConfigSectionSRE(index=highest_index+1)
self.sres[name] = ConfigSectionSRE(index=highest_index + 1)
return self.sres[name]

def remove_sre(self, name: str) -> None:
Expand All @@ -312,7 +289,7 @@ def write_stack(self, name: str, path: Path) -> None:
f_stack.write(pulumi_cfg)

@classmethod
def from_yaml(cls, config_yaml: str) -> Config:
def from_yaml(cls, context: Context, config_yaml: str) -> Config:
try:
config_dict = yaml.safe_load(config_yaml)
except YAMLError as exc:
Expand All @@ -323,6 +300,13 @@ def from_yaml(cls, config_yaml: str) -> Config:
msg = "Unable to parse configuration as a dict."
raise DataSafeHavenConfigError(msg)

# Add context for constructors that require it
# context_dict = context.model_dump()
config_dict["context"] = context
config_dict["tags"] = {}
for section in ["azure", "shm", "tags"]:
config_dict[section]["context"] = context

try:
return Config.model_validate(config_dict)
except ValidationError as exc:
Expand All @@ -338,7 +322,7 @@ def from_remote(cls, context: Context) -> Config:
context.storage_account_name,
context.storage_container_name,
)
return Config.from_yaml(config_yaml)
return Config.from_yaml(context, config_yaml)

def to_yaml(self) -> str:
return yaml.dump(self.model_dump(), indent=2)
Expand Down
8 changes: 7 additions & 1 deletion data_safe_haven/config/context_settings.py
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,9 @@ def storage_account_name(self) -> str:
# maximum of 24 characters allowed
return f"shm{self.shm_name[:14]}context"

def to_yaml(self) -> str:
return yaml.dump(self.model_dump(), indent=2)


class ContextSettings(BaseModel, validate_assignment=True):
"""Load global and local settings from dotfiles with structure like the following
Expand Down Expand Up @@ -189,6 +192,9 @@ def from_file(cls, config_file_path: Path | None = None) -> ContextSettings:
msg = f"Could not find file {config_file_path}.\n{exc}"
raise DataSafeHavenConfigError(msg) from exc

def to_yaml(self) -> str:
return yaml.dump(self.model_dump(by_alias=True), indent=2)

def write(self, config_file_path: Path | None = None) -> None:
"""Write settings to YAML file"""
if config_file_path is None:
Expand All @@ -197,5 +203,5 @@ def write(self, config_file_path: Path | None = None) -> None:
config_file_path.parent.mkdir(parents=True, exist_ok=True)

with open(config_file_path, "w", encoding="utf-8") as f_yaml:
yaml.dump(self.model_dump(by_alias=True), f_yaml, indent=2)
f_yaml.write(self.to_yaml())
self.logger.info(f"Saved context settings to '[green]{config_file_path}[/]'.")
57 changes: 11 additions & 46 deletions tests_/config/test_config.py
Original file line number Diff line number Diff line change
Expand Up @@ -18,24 +18,16 @@

@fixture
def azure_config(context):
return ConfigSectionAzure.from_context(
return ConfigSectionAzure(
context=context,
subscription_id="d5c5c439-1115-4cb6-ab50-b8e547b6c8dd",
tenant_id="d5c5c439-1115-4cb6-ab50-b8e547b6c8dd",
)


class TestConfigSectionAzure:
def test_constructor(self):
ConfigSectionAzure(
admin_group_id="d5c5c439-1115-4cb6-ab50-b8e547b6c8dd",
location="uksouth",
subscription_id="d5c5c439-1115-4cb6-ab50-b8e547b6c8dd",
tenant_id="d5c5c439-1115-4cb6-ab50-b8e547b6c8dd",
)

def test_from_context(self, context):
azure_config = ConfigSectionAzure.from_context(
def test_constructor(self, context):
azure_config = ConfigSectionAzure(
context=context,
subscription_id="d5c5c439-1115-4cb6-ab50-b8e547b6c8dd",
tenant_id="d5c5c439-1115-4cb6-ab50-b8e547b6c8dd",
Expand All @@ -58,7 +50,7 @@ def test_constructor_defaults(self):

@fixture
def shm_config(context):
return ConfigSectionSHM.from_context(
return ConfigSectionSHM(
context=context,
aad_tenant_id="d5c5c439-1115-4cb6-ab50-b8e547b6c8dd",
admin_email_address="admin@example.com",
Expand All @@ -69,18 +61,8 @@ def shm_config(context):


class TestConfigSectionSHM:
def test_constructor(self):
ConfigSectionSHM(
aad_tenant_id="d5c5c439-1115-4cb6-ab50-b8e547b6c8dd",
admin_email_address="admin@example.com",
admin_ip_addresses=["0.0.0.0"], # noqa: S104
fqdn="shm.acme.com",
name="ACME SHM",
timezone="UTC",
)

def test_from_context(self, context):
shm_config = ConfigSectionSHM.from_context(
def test_constructor(self, context):
shm_config = ConfigSectionSHM(
context=context,
aad_tenant_id="d5c5c439-1115-4cb6-ab50-b8e547b6c8dd",
admin_email_address="admin@example.com",
Expand Down Expand Up @@ -180,19 +162,12 @@ def test_update(self):

@fixture
def tags_config(context):
return ConfigSectionTags.from_context(context)
return ConfigSectionTags(context)


class TestConfigSectionTags:
def test_constructor(self):
tags_config = ConfigSectionTags(deployment="Test Deployment")
assert tags_config.deployment == "Test Deployment"
assert tags_config.deployed_by == "Python"
assert tags_config.project == "Data Safe Haven"
assert tags_config.version == __version__

def test_from_context(self, context):
tags_config = ConfigSectionTags.from_context(context)
def test_constructor(self, context):
tags_config = ConfigSectionTags(context)
assert tags_config.deployment == "Acme Deployment"
assert tags_config.deployed_by == "Python"
assert tags_config.project == "Data Safe Haven"
Expand Down Expand Up @@ -230,15 +205,8 @@ def config_sres(context, azure_config, pulumi_config, shm_config, tags_config):
@fixture
def config_yaml():
return """azure:
admin_group_id: d5c5c439-1115-4cb6-ab50-b8e547b6c8dd
location: uksouth
subscription_id: d5c5c439-1115-4cb6-ab50-b8e547b6c8dd
tenant_id: d5c5c439-1115-4cb6-ab50-b8e547b6c8dd
context:
admin_group_id: d5c5c439-1115-4cb6-ab50-b8e547b6c8dd
location: uksouth
name: Acme Deployment
subscription_name: Data Safe Haven (Acme)
pulumi:
encryption_key_name: pulumi-encryption-key
encryption_key_version: lorem
Expand All @@ -250,7 +218,6 @@ def config_yaml():
admin_ip_addresses:
- 0.0.0.0/32
fqdn: shm.acme.com
name: acmedeployment
timezone: UTC
sres:
sre1:
Expand All @@ -273,8 +240,6 @@ def config_yaml():
research_user_ip_addresses: []
software_packages: none
workspace_skus: []
tags:
deployment: Acme Deployment
"""


Expand Down Expand Up @@ -336,8 +301,8 @@ def test_remove_sre(self, config_sres):
assert "sre2" in config_sres.sres.keys()
assert "sre1" not in config_sres.sres.keys()

def test_from_yaml(self, config_sres, config_yaml):
config = Config.from_yaml(config_yaml)
def test_from_yaml(self, config_sres, context, config_yaml):
config = Config.from_yaml(context, config_yaml)
assert config == config_sres
assert isinstance(
config.sres["sre1"].software_packages, SoftwarePackageCategory
Expand Down