diff --git a/CHANGELOG.md b/CHANGELOG.md index 79afc9a1..85858981 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -20,6 +20,7 @@ This project adheres to [Semantic Versioning](http://semver.org/) and [Keep a Ch ### New ### Changes +- adds support for npm mirrors to be set ### Fixes diff --git a/docs/source/manifests.md b/docs/source/manifests.md index 0a783530..562f95f7 100644 --- a/docs/source/manifests.md +++ b/docs/source/manifests.md @@ -31,8 +31,9 @@ targetAccountMappings: default: true codebuildImage: XXXXXXXXXXXX.dkr.ecr.us-east-1.amazonaws.com/aws-codeseeder/code-build-base:5.5.0 npmMirror: https://registry.npmjs.org/ + npmMirrorSecret: /something/aws-addf-mirror-credentials pypiMirror: https://pypi.python.org/simple - pypiMirrorSecret: /something/aws-addf-mirror-secret + pypiMirrorSecret: /something/aws-addf-mirror-mirror-credentials parametersGlobal: dockerCredentialsSecret: nameofsecret permissionsBoundaryName: policyname @@ -41,8 +42,9 @@ targetAccountMappings: default: true codebuildImage: XXXXXXXXXXXX.dkr.ecr.us-east-1.amazonaws.com/aws-codeseeder/code-build-base:4.4.0 npmMirror: https://registry.npmjs.org/ + npmMirrorSecret: /something/aws-addf-mirror-credentials pypiMirror: https://pypi.python.org/simple - pypiMirrorSecret: /something/aws-addf-mirror-secret + pypiMirrorSecret: /something/aws-addf-mirror-credentials parametersRegional: dockerCredentialsSecret: nameofsecret permissionsBoundaryName: policyname @@ -103,6 +105,7 @@ targetAccountMappings: - **default** - this designates this mapping as the default account for all modules unless otherwise specified. This is primarily for supporting migrating from `seedfarmer v1` to the current version. - **codebuildImage** - a custom build image to use (see [Build Image Override](buildimageoverride)) - **npmMirror** - the NPM registry mirror to use (see [Mirror Override](mirroroverride)) + - **npmMirrorSecret** - the AWS SecretManager to use when setting the mirror (see [Mirror Override](mirroroverride)) - **pypiMirror** - the Pypi mirror to use (see [Mirror Override](mirroroverride)) - **pypiMirrorSecret** - the AWS SecretManager to use when setting the mirror (see [Mirror Override](mirroroverride)) - **parametersGlobal** - these are parameters that apply to all region mappings unless otherwise overridden at the region level @@ -258,8 +261,9 @@ targetAccount: secondary targetRegion: us-west-2 codebuildImage: XXXXXXXXXXXX.dkr.ecr.us-east-1.amazonaws.com/aws-codeseeder/code-build-base:3.3.0 npmMirror: https://registry.npmjs.org/ +npmMirrorSecret: /something/aws-addf-mirror-credentials pypiMirror: https://pypi.python.org/simple -pypiMirrorSecret: /something/aws-addf-mirror-secret +pypiMirrorSecret: /something/aws-addf-mirror-credentials parameters: - name: encryption-type value: SSE @@ -285,6 +289,7 @@ dataFiles: - **targetRegion** - the name of the region to deploy to - this overrides any mappings - **codebuildImage** - a custom build image to use (see [Build Image Override](buildimageoverride)) - **npmMirror** - the NPM registry mirror to use (see [Mirror Override](mirroroverride)) +- **npmMirrorSecret** - the NPM registry mirror to use (see [Mirror Override](mirroroverride)) - **pypiMirror** - the Pypi mirror to use (see [Mirror Override](mirroroverride)) - **pypiMirrorSecret** - the AWS SecretManager to use when setting the mirror (see [Mirror Override](mirroroverride)) - **parameters** - the parameters section .... see [Parameters](parameters) @@ -481,6 +486,46 @@ This would result in the creation of the url `https://derekpypi:thepasswordpypi@ pip config set global.index-url https://derekpypi:thepasswordpypi@the-mirror-dns/simple/pypi ``` +#### NPM Mirror +NPM mirror authentication is also supported via a registry url and ssl token. This can be added to the above mirror credentials secret. For example: +```json +{ + "npm" : { + "ssl_token": "mybase64encodedssltoken" + }, + "pypi": { + "username": "derekpypi", + "password": "thepasswordpypi" + }, + "artifactory": { + "username": "myuser@amazon.com", + "password": "agobbleygookofahexcodehere" + }, + "pypi2": { + "username": "hey", + "password": "yooooo" + }, +} +``` + +The secret for npm and the url of the npm registry would then need to be referenced in the manifest. + +```yaml +... +npmMirror: https://the-mirror-dns/npm/ +npmMirrorSecret: /aws-addf-mirror-credentials::npm +... + +``` +This would result in the creation of an `_auth` entry in npm config (`.npmrc`) with the following convention: +``` +//the-mirror-dns/npm/:_auth="mybase64encodedssltoken" and the global config in the runtime will be set via: + +```bash +npm config set //the-mirror-dns/npm/:_auth="mybase64encodedssltoken" +``` + + ### Archive Secret If using an archive store that is not public or needs an authentication scheme, the `archiveSecret` provides a means to set a username / password, so that the archived modules can be downloaded. diff --git a/docs/source/upgrades.md b/docs/source/upgrades.md index 145e1694..5c6a9dd2 100644 --- a/docs/source/upgrades.md +++ b/docs/source/upgrades.md @@ -62,6 +62,7 @@ This is a **BREAKING CHANGE !!!** `seed-farmer` 5.0.0 introduces support for downloading modules from HTTPS archives. This includes support for both secure HTTPS URLs which require authentication, as well as support for S3 HTTPS downloads. + In order to able to use secure HTTPS URLs or S3 HTTPS, you must upgrade the toolchain role permissions. To upgrade: @@ -73,3 +74,23 @@ To upgrade: ```bash seedfarmer bootstrap toolchain <--as-target> --trusted-principal ``` + +`seed-farmer` 5.0.0 also introduces the use of `npmMirrorSecret` to support configuring a npm mirror with credentials (see [Manifests - Mirrors](./manifests.md#mirroroverride)). + +**The following upgrade is optional** + +Seedkits must be upgraded if **both** of the following is true. +- Both `npmMirrorSecret` & `pypiMirrorSecret` are set. +- Mirror secrets are using explicit paths. I.e. (`my-mirror-credentials::npm` && `my-mirror-credentials::pypi`). + +**Note: You can still use this feature by specifying your secret name `my-mirror-credentials` without a path `::pypi` for both secrets.** + +To upgrade: +1. Update your version of `aws-codeseeder` via + ```bash + pip install --upgrade codeseeder==1.1.0 + ``` +2. Run seedfarmer with the `--update-seedkit` flag set + ```bash + seedfarmer apply my/manifest/path --update-seedkit + ``` diff --git a/seedfarmer/commands/_bootstrap_commands.py b/seedfarmer/commands/_bootstrap_commands.py index c50e500e..5b03ab5b 100644 --- a/seedfarmer/commands/_bootstrap_commands.py +++ b/seedfarmer/commands/_bootstrap_commands.py @@ -98,7 +98,7 @@ def bootstrap_toolchain_account( raise seedfarmer.errors.InvalidConfigurationError("The Qualifier must be alphanumeric and 6 characters or less") for arn in principal_arns: - if not re.match(r"arn:aws:(sts|iam)::(\d{12}|\*):.*$", arn): + if not re.match(r"arn:aws.*:(sts|iam)::(\d{12}|\*):.*$", arn): raise seedfarmer.errors.InvalidConfigurationError(f"Trusted principal: {arn} is not a valid principal arn") role_stack_name = get_toolchain_role_name(project_name=project_name, qualifier=cast(str, qualifier)) diff --git a/seedfarmer/commands/_module_commands.py b/seedfarmer/commands/_module_commands.py index da6e735f..3fe9aaa3 100644 --- a/seedfarmer/commands/_module_commands.py +++ b/seedfarmer/commands/_module_commands.py @@ -56,6 +56,7 @@ def _env_vars( session: Optional[Session] = None, use_project_prefix: Optional[bool] = True, pypi_mirror_secret: Optional[str] = None, + npm_mirror_secret: Optional[str] = None, ) -> Dict[str, str]: env_vars = ( { @@ -79,7 +80,9 @@ def _env_vars( if permissions_boundary_arn: env_vars[_param("PERMISSIONS_BOUNDARY_ARN", use_project_prefix)] = permissions_boundary_arn if pypi_mirror_secret is not None: - env_vars["AWS_CODESEEDER_MIRROR_SECRET"] = pypi_mirror_secret + env_vars["AWS_CODESEEDER_PYPI_MIRROR_SECRET"] = pypi_mirror_secret + if npm_mirror_secret is not None: + env_vars["AWS_CODESEEDER_NPM_MIRROR_SECRET"] = npm_mirror_secret # Add the partition to env for ease of fetching env_vars["AWS_PARTITION"] = deployment_partition env_vars["AWS_CODESEEDER_VERSION"] = aws_codeseeder.__version__ @@ -127,6 +130,9 @@ def deploy_module(mdo: ModuleDeployObject) -> ModuleDeploymentResponse: pypi_mirror_secret=( module_manifest.pypi_mirror_secret if module_manifest.pypi_mirror_secret else mdo.pypi_mirror_secret ), + npm_mirror_secret=( + module_manifest.npm_mirror_secret if module_manifest.npm_mirror_secret else mdo.npm_mirror_secret + ), ) env_vars[_param("MODULE_MD5", use_project_prefix)] = ( module_manifest.bundle_md5 if module_manifest.bundle_md5 is not None else "" @@ -248,6 +254,9 @@ def destroy_module(mdo: ModuleDeployObject) -> ModuleDeploymentResponse: pypi_mirror_secret=( module_manifest.pypi_mirror_secret if module_manifest.pypi_mirror_secret else mdo.pypi_mirror_secret ), + npm_mirror_secret=( + module_manifest.npm_mirror_secret if module_manifest.npm_mirror_secret else mdo.npm_mirror_secret + ), ) remove_ssm = [ diff --git a/seedfarmer/models/manifests/_deployment_manifest.py b/seedfarmer/models/manifests/_deployment_manifest.py index ac78cbbc..ca9ff9da 100644 --- a/seedfarmer/models/manifests/_deployment_manifest.py +++ b/seedfarmer/models/manifests/_deployment_manifest.py @@ -60,6 +60,7 @@ class RegionMapping(CamelModel): npm_mirror: Optional[str] = None pypi_mirror: Optional[str] = None pypi_mirror_secret: Optional[str] = None + npm_mirror_secret: Optional[str] = None seedkit_metadata: Optional[Dict[str, Any]] = None seedfarmer_artifact_bucket: Optional[str] = None @@ -77,6 +78,7 @@ class TargetAccountMapping(CamelModel): region_mappings: List[RegionMapping] = [] codebuild_image: Optional[str] = None npm_mirror: Optional[str] = None + npm_mirror_secret: Optional[str] = None pypi_mirror: Optional[str] = None pypi_mirror_secret: Optional[str] = None _default_region: Optional[RegionMapping] = PrivateAttr(default=None) @@ -372,9 +374,10 @@ def get_region_pypi_mirror( else: return None - def get_region_pypi_mirror_secret( + def get_region_mirror_secret( self, *, + mirror_type: Optional[str] = "pypi", account_alias: Optional[str] = None, account_id: Optional[str] = None, region: Optional[str] = None, @@ -382,6 +385,9 @@ def get_region_pypi_mirror_secret( if account_alias is not None and account_id is not None: raise seedfarmer.errors.InvalidManifestError("Only one of 'account_alias' and 'account_id' is allowed") + if mirror_type not in ["pypi", "npm"]: + raise seedfarmer.errors.InvalidManifestError("Mirror type must be of type 'npm' or 'pypi'") + use_default_account = account_alias is None and account_id is None use_default_region = region is None for target_account in self.target_account_mappings: @@ -390,15 +396,23 @@ def get_region_pypi_mirror_secret( or account_id == target_account.actual_account_id or (use_default_account and target_account.default) ): - # Search the region_mappings for the region, if the pypi_mirror_secret is in region + # Search the region_mappings for the region, if the [pypi|npm]_mirror_secret is in region for region_mapping in target_account.region_mappings: if region == region_mapping.region or (use_default_region and region_mapping.default): - pypi_mirror_secret = ( - region_mapping.pypi_mirror_secret - if region_mapping.pypi_mirror_secret is not None - else target_account.pypi_mirror_secret - ) - return pypi_mirror_secret + if mirror_type == "pypi": + pypi_mirror_secret = ( + region_mapping.pypi_mirror_secret + if region_mapping.pypi_mirror_secret is not None + else target_account.pypi_mirror_secret + ) + return pypi_mirror_secret + elif mirror_type == "npm": + npm_mirror_secret = ( + region_mapping.npm_mirror_secret + if region_mapping.npm_mirror_secret is not None + else target_account.npm_mirror_secret + ) + return npm_mirror_secret else: return None diff --git a/seedfarmer/models/manifests/_module_manifest.py b/seedfarmer/models/manifests/_module_manifest.py index 452b12b1..008ad9ee 100644 --- a/seedfarmer/models/manifests/_module_manifest.py +++ b/seedfarmer/models/manifests/_module_manifest.py @@ -97,6 +97,7 @@ class ModuleManifest(CamelModel): data_files: Optional[List[DataFile]] = None commit_hash: SkipJsonSchema[Optional[str]] = None npm_mirror: Optional[str] = None + npm_mirror_secret: Optional[str] = None pypi_mirror: Optional[str] = None pypi_mirror_secret: Optional[str] = None _target_account_id: Optional[str] = PrivateAttr(default=None) diff --git a/seedfarmer/models/transfer/_module_deploy_object.py b/seedfarmer/models/transfer/_module_deploy_object.py index 2b085d0b..a5684c88 100644 --- a/seedfarmer/models/transfer/_module_deploy_object.py +++ b/seedfarmer/models/transfer/_module_deploy_object.py @@ -16,6 +16,7 @@ class ModuleDeployObject(CamelModel): module_role_name: Optional[str] = None codebuild_image: Optional[str] = None npm_mirror: Optional[str] = None + npm_mirror_secret: Optional[str] = None pypi_mirror: Optional[str] = None pypi_mirror_secret: Optional[str] = None seedfarmer_bucket: Optional[str] = None @@ -52,12 +53,16 @@ def __init__(self, **kwargs: Any) -> None: account_alias=_module.target_account, region=_module.target_region ) + npm_mirror_secret = self.deployment_manifest.get_region_mirror_secret( + account_alias=_module.target_account, region=_module.target_region, mirror_type="npm" + ) + pypi_mirror = self.deployment_manifest.get_region_pypi_mirror( account_alias=_module.target_account, region=_module.target_region ) - pypi_mirror_secret = self.deployment_manifest.get_region_pypi_mirror_secret( - account_alias=_module.target_account, region=_module.target_region + pypi_mirror_secret = self.deployment_manifest.get_region_mirror_secret( + account_alias=_module.target_account, region=_module.target_region, mirror_type="pypi" ) sf_bucket = self.deployment_manifest.get_region_seedfarmer_bucket( @@ -68,6 +73,7 @@ def __init__(self, **kwargs: Any) -> None: self.codebuild_image = codebuild_image if codebuild_image is not None else None self.docker_credentials_secret = dcs if dcs else None self.npm_mirror = npm_mirror if npm_mirror is not None else None + self.npm_mirror_secret = npm_mirror_secret if npm_mirror_secret is not None else None self.pypi_mirror = pypi_mirror if pypi_mirror is not None else None self.pypi_mirror_secret = pypi_mirror_secret if pypi_mirror_secret is not None else None self.seedfarmer_bucket = sf_bucket if sf_bucket is not None else None diff --git a/test/unit-test/test_commands_module.py b/test/unit-test/test_commands_module.py index b391a07c..c21d68bd 100644 --- a/test/unit-test/test_commands_module.py +++ b/test/unit-test/test_commands_module.py @@ -182,6 +182,10 @@ def test_deploy_modules(session_manager, mocker): mdo.permissions_boundary_arn = ("arn:aws:iam::123456789012:policy/boundary",) mdo.module_role_name = ("mlops-optionals-efs",) mdo.docker_credentials_secret = ("aws-addf-docker-credentials",) + mdo.pypi_mirror = "https://mypypimirror.com/here" + mdo.pypi_mirror_secret = "user-mirror-credentials" + mdo.npm_mirror = "https://mynpmmirror.com/here" + mdo.npm_mirror_secret = "user-mirror-credentials" mdo.module_metadata = (json.dumps(dummy_list_params),) mc.deploy_module(mdo) diff --git a/test/unit-test/test_models.py b/test/unit-test/test_models.py index 178c0aea..7542cc74 100644 --- a/test/unit-test/test_models.py +++ b/test/unit-test/test_models.py @@ -62,6 +62,34 @@ def test_deserialize_deployment_manifest(): assert manifest.name == "test" +@pytest.mark.models +@pytest.mark.models_deployment_manifest +def test_deployment_manifest_with_custom_mirrors(): + deployment_yaml["targetAccountMappings"][0]["npmMirror"] = "https://mynpmmirror.com/here" + deployment_yaml["targetAccountMappings"][0]["npmMirrorSecret"] = "user-mirror-credentials" + deployment_yaml["targetAccountMappings"][0]["pypiMirror"] = "https://mypypimirror.com/here" + deployment_yaml["targetAccountMappings"][0]["pypiMirrorSecret"] = "user-mirror-credentials" + manifest = DeploymentManifest(**deployment_yaml) + assert manifest.target_account_mappings[0].npm_mirror == "https://mynpmmirror.com/here" + assert manifest.target_account_mappings[0].npm_mirror_secret == "user-mirror-credentials" + assert manifest.target_account_mappings[0].pypi_mirror == "https://mypypimirror.com/here" + assert manifest.target_account_mappings[0].pypi_mirror_secret == "user-mirror-credentials" + + +@pytest.mark.models +@pytest.mark.models_deployment_manifest +def test_get_region_mirror_secret(): + secret_name = "user-mirror-credentials" + deployment_yaml["targetAccountMappings"][0]["npmMirror"] = "https://mynpmmirror.com/here" + deployment_yaml["targetAccountMappings"][0]["npmMirrorSecret"] = secret_name + deployment_yaml["targetAccountMappings"][0]["pypiMirror"] = "https://mypypimirror.com/here" + deployment_yaml["targetAccountMappings"][0]["pypiMirrorSecret"] = secret_name + manifest = DeploymentManifest(**deployment_yaml) + assert manifest.get_region_mirror_secret() == secret_name + assert manifest.get_region_npm_mirror() == "https://mynpmmirror.com/here" + assert manifest.get_region_pypi_mirror() == "https://mypypimirror.com/here" + + @pytest.mark.models @pytest.mark.models_deployment_manifest def test_deployment_manifest_get_parameter_with_defaults():