feat: adding npm_mirror_secret to manifest and module deploy objects

CHANGELOG.md

@@ -20,6 +20,7 @@ This project adheres to [Semantic Versioning](http://semver.org/) and [Keep a Ch
 ### New
 
 ### Changes
+- adds support for npm mirrors to be set 
 
 ### Fixes
 

docs/source/manifests.md

@@ -31,8 +31,9 @@ targetAccountMappings:
     default: true
     codebuildImage:  XXXXXXXXXXXX.dkr.ecr.us-east-1.amazonaws.com/aws-codeseeder/code-build-base:5.5.0
     npmMirror: https://registry.npmjs.org/
+    npmMirrorSecret: /something/aws-addf-mirror-credentials
     pypiMirror: https://pypi.python.org/simple
-    pypiMirrorSecret: /something/aws-addf-mirror-secret
+    pypiMirrorSecret: /something/aws-addf-mirror-mirror-credentials
     parametersGlobal:
       dockerCredentialsSecret: nameofsecret
       permissionsBoundaryName: policyname
@@ -41,8 +42,9 @@ targetAccountMappings:
         default: true
         codebuildImage:  XXXXXXXXXXXX.dkr.ecr.us-east-1.amazonaws.com/aws-codeseeder/code-build-base:4.4.0
         npmMirror: https://registry.npmjs.org/
+        npmMirrorSecret: /something/aws-addf-mirror-credentials
         pypiMirror: https://pypi.python.org/simple
-        pypiMirrorSecret: /something/aws-addf-mirror-secret
+        pypiMirrorSecret: /something/aws-addf-mirror-credentials
         parametersRegional:
           dockerCredentialsSecret: nameofsecret
           permissionsBoundaryName: policyname
@@ -103,6 +105,7 @@ targetAccountMappings:
   - **default** - this designates this mapping as the default account for all modules unless otherwise specified.  This is primarily for supporting migrating from `seedfarmer v1` to the current version.
   - **codebuildImage** - a custom build image to use (see [Build Image Override](buildimageoverride))
   - **npmMirror** - the NPM registry mirror to use (see [Mirror Override](mirroroverride))
+  - **npmMirrorSecret** - the AWS SecretManager to use when setting the mirror (see [Mirror Override](mirroroverride))
   - **pypiMirror** - the Pypi mirror to use (see [Mirror Override](mirroroverride))
   - **pypiMirrorSecret** - the AWS SecretManager to use when setting the mirror (see [Mirror Override](mirroroverride))
   - **parametersGlobal** - these are parameters that apply to all region mappings unless otherwise overridden at the region level
@@ -258,8 +261,9 @@ targetAccount: secondary
 targetRegion: us-west-2
 codebuildImage:  XXXXXXXXXXXX.dkr.ecr.us-east-1.amazonaws.com/aws-codeseeder/code-build-base:3.3.0
 npmMirror: https://registry.npmjs.org/
+npmMirrorSecret: /something/aws-addf-mirror-credentials
 pypiMirror: https://pypi.python.org/simple
-pypiMirrorSecret: /something/aws-addf-mirror-secret
+pypiMirrorSecret: /something/aws-addf-mirror-credentials
 parameters:
   - name: encryption-type
     value: SSE
@@ -285,6 +289,7 @@ dataFiles:
 - **targetRegion** - the name of the region to deploy to - this overrides any mappings
 - **codebuildImage** - a custom build image to use (see [Build Image Override](buildimageoverride))
 - **npmMirror** - the NPM registry mirror to use (see [Mirror Override](mirroroverride))
+- **npmMirrorSecret** - the NPM registry mirror to use (see [Mirror Override](mirroroverride))
 - **pypiMirror** - the Pypi mirror to use (see [Mirror Override](mirroroverride))
 - **pypiMirrorSecret** - the AWS SecretManager to use when setting the mirror (see [Mirror Override](mirroroverride))
 - **parameters** - the parameters section .... see [Parameters](parameters)
@@ -481,6 +486,46 @@ This would result in the creation of the url `https://derekpypi:thepasswordpypi@
 pip config set global.index-url https://derekpypi:thepasswordpypi@the-mirror-dns/simple/pypi
 ```
 
+#### NPM Mirror
+NPM mirror authentication is also supported via a registry url and ssl token. This can be added to the above mirror credentials secret. For example:
+```json
+{
+  "npm" : {
+    "ssl_token": "mybase64encodedssltoken"
+  },
+  "pypi": { 
+    "username": "derekpypi", 
+    "password": "thepasswordpypi" 
+  },
+  "artifactory": {
+    "username": "myuser@amazon.com",
+    "password": "agobbleygookofahexcodehere"
+  },
+  "pypi2": { 
+    "username": "hey", 
+    "password": "yooooo" 
+  },
+}
+```
+
+The secret for npm and the url of the npm registry would then need to be referenced in the manifest.
+
+```yaml
+...
+npmMirror: https://the-mirror-dns/npm/
+npmMirrorSecret: /aws-addf-mirror-credentials::npm
+...
+
+```
+This would result in the creation of an `_auth` entry in npm config (`.npmrc`) with the following convention:
+```
+//the-mirror-dns/npm/:_auth="mybase64encodedssltoken" and the global config in the runtime will be set via:
+
+```bash
+npm config set //the-mirror-dns/npm/:_auth="mybase64encodedssltoken"
+```
+
+
 ### Archive Secret
 
 If using an archive store that is not public or needs an authentication scheme, the `archiveSecret` provides a means to set a username / password, so that the archived modules can be downloaded.

docs/source/upgrades.md

@@ -62,6 +62,7 @@ This is a **BREAKING CHANGE !!!**
 
 `seed-farmer` 5.0.0 introduces support for downloading modules from HTTPS archives.
 This includes support for both secure HTTPS URLs which require authentication, as well as support for S3 HTTPS downloads.
+
 In order to able to use secure HTTPS URLs or S3 HTTPS, you must upgrade the toolchain role permissions.
 
 To upgrade:
@@ -73,3 +74,23 @@ To upgrade:
     ```bash
     seedfarmer bootstrap toolchain <--as-target> --trusted-principal <trusted-principal-arn>
     ```
+
+`seed-farmer` 5.0.0 also introduces the use of `npmMirrorSecret` to support configuring a npm mirror with credentials (see [Manifests - Mirrors](./manifests.md#mirroroverride)).
+
+**The following upgrade is optional**
+
+Seedkits must be upgraded if **both** of the following is true.
+- Both `npmMirrorSecret` & `pypiMirrorSecret` are set.
+- Mirror secrets are using explicit paths. I.e. (`my-mirror-credentials::npm` && `my-mirror-credentials::pypi`). 
+
+**Note: You can still use this feature by specifying your secret name `my-mirror-credentials` without a path `::pypi` for both secrets.**
+
+To upgrade:
+1. Update your version of `aws-codeseeder` via
+    ```bash
+    pip install --upgrade codeseeder==1.1.0
+     ```
+2. Run seedfarmer with the `--update-seedkit` flag set
+    ```bash
+    seedfarmer apply my/manifest/path --update-seedkit
+    ```

seedfarmer/commands/_bootstrap_commands.py

@@ -98,7 +98,7 @@ def bootstrap_toolchain_account(
         raise seedfarmer.errors.InvalidConfigurationError("The Qualifier must be alphanumeric and 6 characters or less")
 
     for arn in principal_arns:
-        if not re.match(r"arn:aws:(sts|iam)::(\d{12}|\*):.*$", arn):
+        if not re.match(r"arn:aws.*:(sts|iam)::(\d{12}|\*):.*$", arn):
             raise seedfarmer.errors.InvalidConfigurationError(f"Trusted principal: {arn} is not a valid principal arn")
 
     role_stack_name = get_toolchain_role_name(project_name=project_name, qualifier=cast(str, qualifier))

seedfarmer/commands/_module_commands.py

@@ -56,6 +56,7 @@ def _env_vars(
     session: Optional[Session] = None,
     use_project_prefix: Optional[bool] = True,
     pypi_mirror_secret: Optional[str] = None,
+    npm_mirror_secret: Optional[str] = None,
 ) -> Dict[str, str]:
     env_vars = (
         {
@@ -79,7 +80,9 @@ def _env_vars(
     if permissions_boundary_arn:
         env_vars[_param("PERMISSIONS_BOUNDARY_ARN", use_project_prefix)] = permissions_boundary_arn
     if pypi_mirror_secret is not None:
-        env_vars["AWS_CODESEEDER_MIRROR_SECRET"] = pypi_mirror_secret
+        env_vars["AWS_CODESEEDER_PYPI_MIRROR_SECRET"] = pypi_mirror_secret
+    if npm_mirror_secret is not None:
+        env_vars["AWS_CODESEEDER_NPM_MIRROR_SECRET"] = npm_mirror_secret
     # Add the partition to env for ease of fetching
     env_vars["AWS_PARTITION"] = deployment_partition
     env_vars["AWS_CODESEEDER_VERSION"] = aws_codeseeder.__version__
@@ -127,6 +130,9 @@ def deploy_module(mdo: ModuleDeployObject) -> ModuleDeploymentResponse:
         pypi_mirror_secret=(
             module_manifest.pypi_mirror_secret if module_manifest.pypi_mirror_secret else mdo.pypi_mirror_secret
         ),
+        npm_mirror_secret=(
+            module_manifest.npm_mirror_secret if module_manifest.npm_mirror_secret else mdo.npm_mirror_secret
+        ),
     )
     env_vars[_param("MODULE_MD5", use_project_prefix)] = (
         module_manifest.bundle_md5 if module_manifest.bundle_md5 is not None else ""
@@ -248,6 +254,9 @@ def destroy_module(mdo: ModuleDeployObject) -> ModuleDeploymentResponse:
         pypi_mirror_secret=(
             module_manifest.pypi_mirror_secret if module_manifest.pypi_mirror_secret else mdo.pypi_mirror_secret
         ),
+        npm_mirror_secret=(
+            module_manifest.npm_mirror_secret if module_manifest.npm_mirror_secret else mdo.npm_mirror_secret
+        ),
     )
 
     remove_ssm = [

seedfarmer/models/manifests/_deployment_manifest.py

@@ -60,6 +60,7 @@ class RegionMapping(CamelModel):
     npm_mirror: Optional[str] = None
     pypi_mirror: Optional[str] = None
     pypi_mirror_secret: Optional[str] = None
+    npm_mirror_secret: Optional[str] = None
     seedkit_metadata: Optional[Dict[str, Any]] = None
     seedfarmer_artifact_bucket: Optional[str] = None
 
@@ -77,6 +78,7 @@ class TargetAccountMapping(CamelModel):
     region_mappings: List[RegionMapping] = []
     codebuild_image: Optional[str] = None
     npm_mirror: Optional[str] = None
+    npm_mirror_secret: Optional[str] = None
     pypi_mirror: Optional[str] = None
     pypi_mirror_secret: Optional[str] = None
     _default_region: Optional[RegionMapping] = PrivateAttr(default=None)
@@ -372,16 +374,20 @@ def get_region_pypi_mirror(
         else:
             return None
 
-    def get_region_pypi_mirror_secret(
+    def get_region_mirror_secret(
         self,
         *,
+        mirror_type: Optional[str] = "pypi",
         account_alias: Optional[str] = None,
         account_id: Optional[str] = None,
         region: Optional[str] = None,
     ) -> Optional[str]:
         if account_alias is not None and account_id is not None:
             raise seedfarmer.errors.InvalidManifestError("Only one of 'account_alias' and 'account_id' is allowed")
 
+        if mirror_type not in ["pypi", "npm"]:
+            raise seedfarmer.errors.InvalidManifestError("Mirror type must be of type 'npm' or 'pypi'")
+
         use_default_account = account_alias is None and account_id is None
         use_default_region = region is None
         for target_account in self.target_account_mappings:
@@ -390,15 +396,23 @@ def get_region_pypi_mirror_secret(
                 or account_id == target_account.actual_account_id
                 or (use_default_account and target_account.default)
             ):
-                # Search the region_mappings for the region, if the pypi_mirror_secret is in region
+                # Search the region_mappings for the region, if the [pypi|npm]_mirror_secret is in region
                 for region_mapping in target_account.region_mappings:
                     if region == region_mapping.region or (use_default_region and region_mapping.default):
-                        pypi_mirror_secret = (
-                            region_mapping.pypi_mirror_secret
-                            if region_mapping.pypi_mirror_secret is not None
-                            else target_account.pypi_mirror_secret
-                        )
-                        return pypi_mirror_secret
+                        if mirror_type == "pypi":
+                            pypi_mirror_secret = (
+                                region_mapping.pypi_mirror_secret
+                                if region_mapping.pypi_mirror_secret is not None
+                                else target_account.pypi_mirror_secret
+                            )
+                            return pypi_mirror_secret
+                        elif mirror_type == "npm":
+                            npm_mirror_secret = (
+                                region_mapping.npm_mirror_secret
+                                if region_mapping.npm_mirror_secret is not None
+                                else target_account.npm_mirror_secret
+                            )
+                            return npm_mirror_secret
         else:
             return None
 

seedfarmer/models/manifests/_module_manifest.py

@@ -97,6 +97,7 @@ class ModuleManifest(CamelModel):
     data_files: Optional[List[DataFile]] = None
     commit_hash: SkipJsonSchema[Optional[str]] = None
     npm_mirror: Optional[str] = None
+    npm_mirror_secret: Optional[str] = None
     pypi_mirror: Optional[str] = None
     pypi_mirror_secret: Optional[str] = None
     _target_account_id: Optional[str] = PrivateAttr(default=None)

seedfarmer/models/transfer/_module_deploy_object.py

@@ -16,6 +16,7 @@ class ModuleDeployObject(CamelModel):
     module_role_name: Optional[str] = None
     codebuild_image: Optional[str] = None
     npm_mirror: Optional[str] = None
+    npm_mirror_secret: Optional[str] = None
     pypi_mirror: Optional[str] = None
     pypi_mirror_secret: Optional[str] = None
     seedfarmer_bucket: Optional[str] = None
@@ -52,12 +53,16 @@ def __init__(self, **kwargs: Any) -> None:
             account_alias=_module.target_account, region=_module.target_region
         )
 
+        npm_mirror_secret = self.deployment_manifest.get_region_mirror_secret(
+            account_alias=_module.target_account, region=_module.target_region, mirror_type="npm"
+        )
+
         pypi_mirror = self.deployment_manifest.get_region_pypi_mirror(
             account_alias=_module.target_account, region=_module.target_region
         )
 
-        pypi_mirror_secret = self.deployment_manifest.get_region_pypi_mirror_secret(
-            account_alias=_module.target_account, region=_module.target_region
+        pypi_mirror_secret = self.deployment_manifest.get_region_mirror_secret(
+            account_alias=_module.target_account, region=_module.target_region, mirror_type="pypi"
         )
 
         sf_bucket = self.deployment_manifest.get_region_seedfarmer_bucket(
@@ -68,6 +73,7 @@ def __init__(self, **kwargs: Any) -> None:
         self.codebuild_image = codebuild_image if codebuild_image is not None else None
         self.docker_credentials_secret = dcs if dcs else None
         self.npm_mirror = npm_mirror if npm_mirror is not None else None
+        self.npm_mirror_secret = npm_mirror_secret if npm_mirror_secret is not None else None
         self.pypi_mirror = pypi_mirror if pypi_mirror is not None else None
         self.pypi_mirror_secret = pypi_mirror_secret if pypi_mirror_secret is not None else None
         self.seedfarmer_bucket = sf_bucket if sf_bucket is not None else None

test/unit-test/test_commands_module.py

@@ -182,6 +182,10 @@ def test_deploy_modules(session_manager, mocker):
     mdo.permissions_boundary_arn = ("arn:aws:iam::123456789012:policy/boundary",)
     mdo.module_role_name = ("mlops-optionals-efs",)
     mdo.docker_credentials_secret = ("aws-addf-docker-credentials",)
+    mdo.pypi_mirror = "https://mypypimirror.com/here"
+    mdo.pypi_mirror_secret = "user-mirror-credentials"
+    mdo.npm_mirror = "https://mynpmmirror.com/here"
+    mdo.npm_mirror_secret = "user-mirror-credentials"
     mdo.module_metadata = (json.dumps(dummy_list_params),)
     mc.deploy_module(mdo)
 

test/unit-test/test_models.py

@@ -62,6 +62,34 @@ def test_deserialize_deployment_manifest():
     assert manifest.name == "test"
 
 
+@pytest.mark.models
+@pytest.mark.models_deployment_manifest
+def test_deployment_manifest_with_custom_mirrors():
+    deployment_yaml["targetAccountMappings"][0]["npmMirror"] = "https://mynpmmirror.com/here"
+    deployment_yaml["targetAccountMappings"][0]["npmMirrorSecret"] = "user-mirror-credentials"
+    deployment_yaml["targetAccountMappings"][0]["pypiMirror"] = "https://mypypimirror.com/here"
+    deployment_yaml["targetAccountMappings"][0]["pypiMirrorSecret"] = "user-mirror-credentials"
+    manifest = DeploymentManifest(**deployment_yaml)
+    assert manifest.target_account_mappings[0].npm_mirror == "https://mynpmmirror.com/here"
+    assert manifest.target_account_mappings[0].npm_mirror_secret == "user-mirror-credentials"
+    assert manifest.target_account_mappings[0].pypi_mirror == "https://mypypimirror.com/here"
+    assert manifest.target_account_mappings[0].pypi_mirror_secret == "user-mirror-credentials"
+
+
+@pytest.mark.models
+@pytest.mark.models_deployment_manifest
+def test_get_region_mirror_secret():
+    secret_name = "user-mirror-credentials"
+    deployment_yaml["targetAccountMappings"][0]["npmMirror"] = "https://mynpmmirror.com/here"
+    deployment_yaml["targetAccountMappings"][0]["npmMirrorSecret"] = secret_name
+    deployment_yaml["targetAccountMappings"][0]["pypiMirror"] = "https://mypypimirror.com/here"
+    deployment_yaml["targetAccountMappings"][0]["pypiMirrorSecret"] = secret_name
+    manifest = DeploymentManifest(**deployment_yaml)
+    assert manifest.get_region_mirror_secret() == secret_name
+    assert manifest.get_region_npm_mirror() == "https://mynpmmirror.com/here"
+    assert manifest.get_region_pypi_mirror() == "https://mypypimirror.com/here"
+
+
 @pytest.mark.models
 @pytest.mark.models_deployment_manifest
 def test_deployment_manifest_get_parameter_with_defaults():