diff --git a/README.md b/README.md index f893f76..f1e03b8 100644 --- a/README.md +++ b/README.md @@ -915,7 +915,8 @@ aws cloudformation create-stack \ ParameterKey=EnvName,ParameterValue=$ENV_NAME \ ParameterKey=EnvType,ParameterValue=dev \ ParameterKey=AvailabilityZones,ParameterValue=${AWS_DEFAULT_REGION}a\\,${AWS_DEFAULT_REGION}b \ - ParameterKey=NumberOfAZs,ParameterValue=2 + ParameterKey=NumberOfAZs,ParameterValue=2 \ + ParameterKey=SeedCodeS3BucketName,ParameterValue=$S3_BUCKET_NAME ``` If you would like to use **multi-account model deployment**, you must provide the valid values for OU ids and the name for the `SetupStackSetExecutionRole`: @@ -938,11 +939,14 @@ aws cloudformation create-stack \ ParameterKey=AvailabilityZones,ParameterValue=${AWS_DEFAULT_REGION}a\\,${AWS_DEFAULT_REGION}b \ ParameterKey=NumberOfAZs,ParameterValue=2 \ ParameterKey=StartKernelGatewayApps,ParameterValue=YES \ + ParameterKey=SeedCodeS3BucketName,ParameterValue=$S3_BUCKET_NAME \ ParameterKey=OrganizationalUnitStagingId,ParameterValue=$STAGING_OU_ID \ ParameterKey=OrganizationalUnitProdId,ParameterValue=$PROD_OU_ID \ ParameterKey=SetupStackSetExecutionRoleName,ParameterValue=$SETUP_STACKSET_ROLE_NAME ``` +If you do not have an AWS Organization setup, you can omit the `OrganizationalUnitStagingId` and `OrganizationalUnitProdId` parameters from the previous call. + ### Cleanup First, delete the two root stacks from AWS CloudFormation console or command line: ```bash @@ -974,6 +978,7 @@ aws cloudformation describe-stacks \ Copy and paste the `AssumeDSAdministratorRole` link to a web browser and switch role to DS Administrator. Go to AWS Service Catalog in the AWS console and select **Products** on the left pane: + ![service-catalog-end-user-products](img/service-catalog-end-user-products.png) You will see the list of available products for your user role: @@ -984,7 +989,7 @@ Click on the product name and and then on the **Launch product** on the product ![service-catalog-launch-product](img/service-catalog-launch-product.png) -Fill the product parameters with values specific for your environment. Provide the valid values for OU ids and the name for the `SetupStackSetExecutionRole` if you would like to enable multi-account model deployment. +Fill the product parameters with values specific for your environment. Provide the valid values for OU ids and the name for the `SetupStackSetExecutionRole` if you would like to enable multi-account model deployment, otherwise keep these parameters empty. Wait until AWS Service Catalog finishes the provisioning of the Data Science environment stack and the product status becomes **Available**. The data science environmetn provisioning takes about 20 minutes to complete. @@ -1145,6 +1150,26 @@ aws cloudformation deploy \ --capabilities CAPABILITY_NAMED_IAM ``` +Deploy the setup stack set execution role in each of the staging and target accounts. This step is only needed if: +1. You are going to use multi-account model deployment option +2. You want that the deployment of the data science environment provisions the network infrastructure and IAM roles in the target accounts. + +```sh +ENV_NAME=ds-team +ADMIN_ACCOUNT_ID=#Data science account with SageMaker Studio +SETUP_STACKSET_ROLE_NAME=$ENV_NAME-setup-stackset-role + +aws cloudformation deploy \ + --template-file build/$AWS_DEFAULT_REGION/env-iam-setup-stackset-role.yaml \ + --stack-name $ENV_NAME-setup-stackset-execution-role \ + --capabilities CAPABILITY_NAMED_IAM \ + --parameter-overrides \ + EnvName=$ENV_NAME \ + EnvType=$ENV_TYPE \ + StackSetExecutionRoleName=$SETUP_STACKSET_ROLE_NAME \ + AdministratorAccountId=$ADMIN_ACCOUNT_ID +``` + Deploy IAM shared roles: ```bash STACK_SET_NAME=ds-team @@ -1172,6 +1197,8 @@ aws cloudformation deploy \ EnvType=dev ``` +If you want to provision the target account infrastructure during the data science environment deployment, you must provide the value for the `SetupStackSetExecutionRoleName` parameter. + Deploy SageMaker model deployment roles in development, staging, and production AWS accounts: ```bash aws cloudformation deploy \ @@ -1181,11 +1208,15 @@ aws cloudformation deploy \ --parameter-overrides \ EnvName=$ENV_NAME \ EnvType=dev \ - PipelineExecutionRoleArn=arn:aws:iam::ACCOUNT_ID:role/service-role/AmazonSageMakerServiceCatalogProductsUseRole \ + PipelineExecutionRoleArn=arn:aws:iam::ACCOUNT_ID:role/service-role/ + AmazonSageMakerServiceCatalogProductsUseRole \ + AdministratorAccountId= \ ModelS3KMSKeyArn= \ ModelBucketName= ``` +If you do not use multi-account deployment, you do not need to deploy these roles into the staging and production accounts. Deploy to the development account only. + Show IAM role ARNs: ```bash aws cloudformation describe-stacks \ @@ -1271,7 +1302,8 @@ aws cloudformation create-stack \ ParameterKey=CreateVPCFlowLogsToCloudWatch,ParameterValue=NO \ ParameterKey=CreateVPCFlowLogsRole,ParameterValue=NO \ ParameterKey=AvailabilityZones,ParameterValue=${AWS_DEFAULT_REGION}a\\,${AWS_DEFAULT_REGION}b\\,${AWS_DEFAULT_REGION}c \ - ParameterKey=NumberOfAZs,ParameterValue=3 + ParameterKey=NumberOfAZs,ParameterValue=3 \ + ParameterKey=SeedCodeS3BucketName,ParameterValue=$S3_BUCKET_NAME ``` ## Clean-up diff --git a/cfn_templates/core-iam-sc-sm-projects-roles.yaml b/cfn_templates/core-iam-sc-sm-projects-roles.yaml index b601673..583c7c8 100644 --- a/cfn_templates/core-iam-sc-sm-projects-roles.yaml +++ b/cfn_templates/core-iam-sc-sm-projects-roles.yaml @@ -382,12 +382,12 @@ Resources: - cloudformation:TagResource Resource: "*" Effect: Allow - - - Action: - - organizations:DescribeOrganizationalUnit - - organizations:ListAccountsForParent - Resource: "arn:aws:organizations::*:ou/o-*/ou-*" - Effect: Allow + #- + # Action: + # - organizations:DescribeOrganizationalUnit + # - organizations:ListAccountsForParent + # Effect: Allow + # Resource: "arn:aws:organizations::*:ou/o-*/ou-*" - Action: - cloudwatch:PutMetricData diff --git a/cfn_templates/data-science-environment-quickstart.yaml b/cfn_templates/data-science-environment-quickstart.yaml index f983abf..0514037 100644 --- a/cfn_templates/data-science-environment-quickstart.yaml +++ b/cfn_templates/data-science-environment-quickstart.yaml @@ -29,6 +29,10 @@ Metadata: default: Deployment Options Parameters: - CreateSharedServices + - Label: + default: S3 Bucket Name with MLOps Seed Code + Parameters: + - SeedCodeS3BucketName - Label: default: Network Configuration Parameters: @@ -47,6 +51,8 @@ Metadata: default: Environment type CreateSharedServices: default: Create Shared Services (PyPI mirror) + SeedCodeS3BucketName: + default: Existing S3 bucket name where MLOps seed code will be stored VPCCIDR: default: VPC CIDR block PrivateSubnet1ACIDR: @@ -94,6 +100,10 @@ Parameters: - 'NO' Description: Set to YES if you do want to provision the shared services VPC network and PyPi mirror repository + SeedCodeS3BucketName: + Description: S3 bucket name to store MLOps seed code (the S3 bucket must exist) + Type: String + VPCCIDR: AllowedPattern: ^(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])(\/(1[6-9]|2[0-8]))$ ConstraintDescription: CIDR block parameter must be in the form x.x.x.x/16-28 @@ -194,6 +204,7 @@ Resources: PrivateSubnet2ACIDR: !Ref PrivateSubnet2ACIDR PublicSubnet1CIDR: !Ref PublicSubnet1CIDR PublicSubnet2CIDR: !Ref PublicSubnet2CIDR + SeedCodeS3BucketName: !Ref SeedCodeS3BucketName TemplateURL: env-main.yaml Tags: - Key: EnvironmentName diff --git a/cfn_templates/env-main.yaml b/cfn_templates/env-main.yaml index fcab770..1ab8a8a 100644 --- a/cfn_templates/env-main.yaml +++ b/cfn_templates/env-main.yaml @@ -40,10 +40,16 @@ Metadata: - DataBucketName - ModelBucketName - Label: - default: Organizational Unit Ids for multi-account model deployment + default: S3 Bucket Name with MLOps Seed Code + Parameters: + - SeedCodeS3BucketName + - Label: + default: Multi-account model deployment setup Parameters: - OrganizationalUnitStagingId - OrganizationalUnitProdId + - StagingAccountList + - ProductionAccountList - Label: default: Infrastructure setup in the target accounts in staging and production OUs (multi-account only) Parameters: @@ -117,6 +123,8 @@ Metadata: default: Existing S3 bucket name to store ML data (only if created outside of this stack) ModelBucketName: default: Existing S3 bucket name to store ML models (only if created outside of this stack) + SeedCodeS3BucketName: + default: Existing S3 bucket name where MLOps seed code will be stored CreateEnvironmentIAMRoles: default: Create environment IAM roles CreateEnvironmentS3Buckets: @@ -177,6 +185,10 @@ Metadata: default: OU Id for the organizational unit with data science staging account. Leave empty for single-account deployment OrganizationalUnitProdId: default: OU Id for the organizational unit with data science production account. Leave empty for single-account deployment + StagingAccountList: + default: Comma-delimited list of the staging accounts for multi-account model deployment. Leave empty for single-account deployment + ProductionAccountList: + default: Comma-delimited list of the production accounts for multi-account model deployment. Leave empty for single-account deployment SetupStackSetExecutionRoleName: default: Stack set execution role name for initial infrastructure setup in the target accounts in the staging and production OUs CreateTargetAccountNetworkInfra: @@ -297,6 +309,10 @@ Parameters: Description: S3 bucket name to store models (only if created outside of this stack) Type: String Default: '' + + SeedCodeS3BucketName: + Description: S3 bucket name to store MLOps seed code (the S3 bucket must exist) + Type: String CreateEnvironmentIAMRoles: Description: If YES, all IAM Roles for Data Science environment will be automatically provisioned. If NO, you have to provide the IAM roles @@ -505,6 +521,16 @@ Parameters: Description: OU Id of the organizational unit that holds the data science production account Default: '' + StagingAccountList: + Type: String + Description: Comma-delimited list of staging account ids for multi-account model deployment + Default: '' + + ProductionAccountList: + Type: String + Description: Comma-delimited list of production account ids for multi-account model deployment + Default: '' + SetupStackSetExecutionRoleName: Type: String Description: Stack set execution role name for initial infrastructure setup in the target accounts (staging and production OUs). @@ -704,9 +730,15 @@ Conditions: IAMRoles&VPCFlowLogsRoleCondition: !And [ !Condition IAMRolesCondition, !Condition VPCFlowLogsRoleCondition] S3BucketsConditions: !Equals [ !Ref CreateEnvironmentS3Buckets, 'YES' ] SharedServicesPyPiMirrorCondition: !Equals [ !Ref UseSharedServicesPyPiMirror, 'YES' ] - MultiAccountDeploymentCondition: !And - - !Not [ !Equals [ !Ref OrganizationalUnitStagingId, ''] ] - - !Not [ !Equals [ !Ref OrganizationalUnitProdId, ''] ] + OrganizationalUnitCondition: !And + - !Not [ !Equals [ !Ref OrganizationalUnitStagingId, ''] ] + - !Not [ !Equals [ !Ref OrganizationalUnitProdId, ''] ] + AccountListCondition: !And + - !Not [ !Equals [ !Ref StagingAccountList, ''] ] + - !Not [ !Equals [ !Ref ProductionAccountList, ''] ] + MultiAccountDeploymentCondition: !Or + - !Condition OrganizationalUnitCondition + - !Condition AccountListCondition IAMRolesStackSetDeploymentCondition: !And - !Condition MultiAccountDeploymentCondition - !Condition IAMRolesCondition @@ -715,6 +747,7 @@ Conditions: - !Condition MultiAccountDeploymentCondition - !Condition TargetAccountNetworkInfraCondition + Rules: IAMRoles: RuleCondition: !Equals [ !Ref CreateEnvironmentIAMRoles, 'NO' ] @@ -737,17 +770,29 @@ Rules: - Assert: !Equals [ !Ref CreateVPCFlowLogsRole, 'NO' ] AssertDescription: Create VPC Flow Logs execution role cannot be set to YES if Create environemtn IAM Roles is set NO - OrganizationalUnitStagingId: - RuleCondition: !Equals [ !Ref OrganizationalUnitStagingId, '' ] + OrganizationalUnitIds: + RuleCondition: !Or + - !Not [ !Equals [ !Ref OrganizationalUnitStagingId, '' ] ] + - !Not [ !Equals [ !Ref OrganizationalUnitProdId, '' ] ] Assertions: - - Assert: !Equals [ !Ref OrganizationalUnitProdId, '' ] - AssertDescription: OU Ids for both staging and production OUs must be provided if one (staging or production) id is provided - - OrganizationalUnitProdId: - RuleCondition: !Equals [ !Ref OrganizationalUnitProdId, '' ] + - Assert: !And + - !Not [ !Equals [ !Ref OrganizationalUnitProdId, '' ] ] + - !Not [ !Equals [ !Ref OrganizationalUnitStagingId, '' ] ] + - !Equals [ !Ref StagingAccountList, ''] + - !Equals [ !Ref ProductionAccountList, ''] + AssertDescription: Both staging and production OU ids must be provided if one OU id is provided and both StagingAccountList and ProductionAccountList must be empty + + AccountList: + RuleCondition: !Or + - !Not [ !Equals [ !Ref StagingAccountList, '' ] ] + - !Not [ !Equals [ !Ref ProductionAccountList, '' ] ] Assertions: - - Assert: !Equals [ !Ref OrganizationalUnitStagingId, '' ] - AssertDescription: OU Ids for both staging and production OUs must be provided if one (staging or production) id is provided + - Assert: !And + - !Equals [ !Ref OrganizationalUnitProdId, '' ] + - !Equals [ !Ref OrganizationalUnitStagingId, '' ] + - !Not [ !Equals [ !Ref StagingAccountList, ''] ] + - !Not [ !Equals [ !Ref ProductionAccountList, ''] ] + AssertDescription: Both StagingAccountList and ProductionAccountList must be provided if one list is provided and both OrganizationalUnitProdId and OrganizationalUnitProdId must be empty SetupStackSetExecutionRole: RuleCondition: !And @@ -849,7 +894,11 @@ Resources: Regions: - !Ref AWS::Region DeploymentTargets: - Accounts: !GetAtt GetOUAccountsStaging.Accounts + Accounts: !If + - OrganizationalUnitCondition + - !GetAtt GetOUAccountsStaging.Accounts + - !Split [',', !Ref StagingAccountList] + StackSetName: !Sub '${EnvName}-${EnvTypeStagingName}-${AWS::Region}-target-account-roles' TemplateURL: 'https://s3.amazonaws.com/< S3_CFN_STAGING_BUCKET_PATH >/env-iam-target-account-roles.yaml' Tags: @@ -900,7 +949,11 @@ Resources: Regions: - !Ref AWS::Region DeploymentTargets: - Accounts: !GetAtt GetOUAccountsProd.Accounts + Accounts: !If + - OrganizationalUnitCondition + - !GetAtt GetOUAccountsProd.Accounts + - !Split [',', !Ref ProductionAccountList] + StackSetName: !Sub '${EnvName}-${EnvTypeProdName}-${AWS::Region}-target-account-roles' TemplateURL: 'https://s3.amazonaws.com/< S3_CFN_STAGING_BUCKET_PATH >/env-iam-target-account-roles.yaml' Tags: @@ -1134,7 +1187,11 @@ Resources: Regions: - !Ref AWS::Region DeploymentTargets: - Accounts: !GetAtt GetOUAccountsStaging.Accounts + Accounts: !If + - OrganizationalUnitCondition + - !GetAtt GetOUAccountsStaging.Accounts + - !Split [',', !Ref StagingAccountList] + StackSetName: !Sub '${EnvName}-${EnvTypeStagingName}-${AWS::Region}-target-account-vpc' TemplateURL: 'https://s3.amazonaws.com/< S3_CFN_STAGING_BUCKET_PATH >/env-vpc.yaml' Tags: @@ -1252,7 +1309,11 @@ Resources: Regions: - !Ref AWS::Region DeploymentTargets: - Accounts: !GetAtt GetOUAccountsProd.Accounts + Accounts: !If + - OrganizationalUnitCondition + - !GetAtt GetOUAccountsProd.Accounts + - !Split [',', !Ref ProductionAccountList] + StackSetName: !Sub '${EnvName}-${EnvTypeProdName}-${AWS::Region}-target-account-vpc' TemplateURL: 'https://s3.amazonaws.com/< S3_CFN_STAGING_BUCKET_PATH >/env-vpc.yaml' Tags: @@ -1389,7 +1450,10 @@ Resources: DependsOn: EnvironmentSageMakerStudio Properties: ServiceToken: !GetAtt SetupCrossAccountPermissionsLambda.Arn - Accounts: !GetAtt GetOUAccountsStaging.Accounts + Accounts: !If + - OrganizationalUnitCondition + - !GetAtt GetOUAccountsStaging.Accounts + - !Split [',', !Ref StagingAccountList] PrincipalRoleName: !If - IAMRolesCondition - !GetAtt EnvironmentTargetAccountRoles.Outputs.SageMakerModelExecutionRoleName @@ -1413,7 +1477,10 @@ Resources: DependsOn: SetupCrossAccountPermissionsStaging Properties: ServiceToken: !GetAtt SetupCrossAccountPermissionsLambda.Arn - Accounts: !GetAtt GetOUAccountsProd.Accounts + Accounts: !If + - OrganizationalUnitCondition + - !GetAtt GetOUAccountsProd.Accounts + - !Split [',', !Ref ProductionAccountList] PrincipalRoleName: !If - IAMRolesCondition - !GetAtt EnvironmentTargetAccountRoles.Outputs.SageMakerModelExecutionRoleName @@ -1704,9 +1771,33 @@ Resources: - Key: EnvironmentType Value: !Ref EnvType - OrganizationalUnitStagingIdSSM: + StagingAccountListSSM: Type: 'AWS::SSM::Parameter' Condition: MultiAccountDeploymentCondition + Properties: + Name: !Sub '${EnvName}-${EnvType}-staging-account-list' + Type: String + Value: !If + - AccountListCondition + - !Ref StagingAccountList + - !Join [',', !GetAtt GetOUAccountsStaging.Accounts] + Description: !Sub 'List of staging accounts for model deployment for ${EnvName}-${EnvType} data science environment' + + ProductionAccountListSSM: + Type: 'AWS::SSM::Parameter' + Condition: MultiAccountDeploymentCondition + Properties: + Name: !Sub '${EnvName}-${EnvType}-production-account-list' + Type: String + Value: !If + - AccountListCondition + - !Ref ProductionAccountList + - !Join [',', !GetAtt GetOUAccountsProd.Accounts] + Description: !Sub 'List of production accounts for model deployment for ${EnvName}-${EnvType} data science environment' + + OrganizationalUnitStagingIdSSM: + Type: 'AWS::SSM::Parameter' + Condition: OrganizationalUnitCondition Properties: Name: !Sub '${EnvName}-${EnvType}-ou-staging-id' Type: String @@ -1715,7 +1806,7 @@ Resources: OrganizationalUnitProdIdSSM: Type: 'AWS::SSM::Parameter' - Condition: MultiAccountDeploymentCondition + Condition: OrganizationalUnitCondition Properties: Name: !Sub '${EnvName}-${EnvType}-ou-prod-id' Type: String @@ -1796,4 +1887,12 @@ Resources: Name: !Sub "${EnvName}-${EnvType}-env-type-prod-name" Type: String Value: !Ref EnvTypeProdName - Description: Environment name for production environment \ No newline at end of file + Description: Environment name for production environment + + SeedCodeS3BucketNameSSM: + Type: 'AWS::SSM::Parameter' + Properties: + Name: !Sub "${EnvName}-${EnvType}-seed-code-s3bucket-name" + Type: String + Value: !Ref SeedCodeS3BucketName + Description: S3 bucket name where MLOps seed code is stored \ No newline at end of file diff --git a/cfn_templates/project-model-build-train.yaml b/cfn_templates/project-model-build-train.yaml index 2a0a636..7285041 100644 --- a/cfn_templates/project-model-build-train.yaml +++ b/cfn_templates/project-model-build-train.yaml @@ -15,10 +15,6 @@ Parameters: Type: String Description: Service generated Id of the project. - SeedCodeS3BucketName: - Type: String - Description: S3 bucket name where the seed code for MLOps projects for CodeCommit repository is stored - Conditions: MLOpsArtifactBucketCondition: !Equals [ 'true', 'true' ] @@ -30,15 +26,22 @@ Resources: ServiceToken: !ImportValue 'ds-get-environment-configuration-lambda-arn' SageMakerProjectName: !Ref SageMakerProjectName SSMParams: - - VariableName: 'DataBucketName' + - + VariableName: 'DataBucketName' ParameterName: 'data-bucket-name' - - VariableName: 'ModelBucketName' + - + VariableName: 'ModelBucketName' ParameterName: 'model-bucket-name' - - VariableName: 'S3KmsKeyId' + - + VariableName: 'S3KmsKeyId' ParameterName: 'kms-s3-key-arn' - - VariableName: 'PipelineExecutionRole' + - + VariableName: 'PipelineExecutionRole' ParameterName: 'sm-pipeline-execution-role-arn' - + - + VariableName: 'SeedCodeS3BucketName' + ParameterName: 'seed-code-s3bucket-name' + MlOpsArtifactsBucket: Type: AWS::S3::Bucket Condition: MLOpsArtifactBucketCondition @@ -102,7 +105,7 @@ Resources: RepositoryDescription: !Sub SageMaker Model building infrastructure as code for the project ${SageMakerProjectName} Code: S3: - Bucket: !Ref SeedCodeS3BucketName + Bucket: !GetAtt GetEnvironmentConfiguration.SeedCodeS3BucketName Key: sagemaker-mlops/seed-code/mlops-model-build-train-v1.0.zip BranchName: main Tags: diff --git a/cfn_templates/project-model-deploy.yaml b/cfn_templates/project-model-deploy.yaml index 0453de6..524b04c 100644 --- a/cfn_templates/project-model-deploy.yaml +++ b/cfn_templates/project-model-deploy.yaml @@ -15,10 +15,6 @@ Parameters: Type: String Description: Service generated Id of the project. - SeedCodeS3BucketName: - Type: String - Description: S3 bucket name where the seed code for MLOps projects for CodeCommit repository is stored - ModelPackageGroupName: Type: String Description: Model package group name to monitor for model package state changes. Leave 'Auto' to auto generate. @@ -38,24 +34,6 @@ Conditions: MultiAccountDeploymentCondition: !Equals [ !Ref MultiAccountDeployment, 'YES' ] Resources: - # Retrieve accounts from the staging OU - GetOUAccountsStaging: - Type: Custom::GetOUAccounts - Condition: MultiAccountDeploymentCondition - Properties: - ServiceToken: !ImportValue 'ds-ou-accounts-lambda-arn' - OUIds: - - !GetAtt GetEnvironmentConfiguration.OUStagingId - - # Retrieve accounts from the production OU - GetOUAccountsProd: - Type: Custom::GetOUAccounts - Condition: MultiAccountDeploymentCondition - Properties: - ServiceToken: !ImportValue 'ds-ou-accounts-lambda-arn' - OUIds: - - !GetAtt GetEnvironmentConfiguration.OUProdId - # Retrieve the environment variables GetEnvironmentConfiguration: Type: Custom::GetEnvironmentConfiguration @@ -84,6 +62,12 @@ Resources: - VariableName: 'OUProdId' ParameterName: 'ou-prod-id' + - + VariableName: 'ProdAccountList' + ParameterName: 'production-account-list' + - + VariableName: 'StagingAccountList' + ParameterName: 'staging-account-list' - VariableName: 'ModelExecutionRole' ParameterName: 'sm-model-execution-role-name' @@ -102,6 +86,9 @@ Resources: - VariableName: 'EnvTypeProdName' ParameterName: 'env-type-prod-name' + - + VariableName: 'SeedCodeS3BucketName' + ParameterName: 'seed-code-s3bucket-name' MlOpsArtifactsBucket: Type: AWS::S3::Bucket @@ -192,7 +179,7 @@ Resources: RepositoryDescription: !Sub SageMaker Endpoint deployment infrastructure as code for the project ${SageMakerProjectName} Code: S3: - Bucket: !Ref SeedCodeS3BucketName + Bucket: !GetAtt GetEnvironmentConfiguration.SeedCodeS3BucketName Key: sagemaker-mlops/seed-code/mlops-model-deploy-v1.0.zip BranchName: main Tags: @@ -245,10 +232,16 @@ Resources: Value: !Ref AWS::Region - Name: MULTI_ACCOUNT_DEPLOYMENT Value: !Ref MultiAccountDeployment - - Name: ORGANIZATIONAL_UNIT_STAGING_ID - Value: !GetAtt GetEnvironmentConfiguration.OUStagingId - - Name: ORGANIZATIONAL_UNIT_PROD_ID - Value: !GetAtt GetEnvironmentConfiguration.OUProdId + - Name: STAGING_ACCOUNT_LIST + Value: !If + - MultiAccountDeploymentCondition + - !GetAtt GetEnvironmentConfiguration.StagingAccountList + - !Ref 'AWS::AccountId' + - Name: PROD_ACCOUNT_LIST + Value: !If + - MultiAccountDeploymentCondition + - !GetAtt GetEnvironmentConfiguration.ProdAccountList + - !Ref 'AWS::AccountId' - Name: STAGING_CONFIG_NAME Value: 'staging-config' - Name: PROD_CONFIG_NAME @@ -304,8 +297,6 @@ Resources: Value: 'staging-config' - Name: TEST_RESULTS Value: 'test-results' - - Name: MULTI_ACCOUNT_DEPLOYMENT - Value: !Ref MultiAccountDeployment Source: Type: CODEPIPELINE @@ -406,7 +397,7 @@ Resources: # For a self-managed model, targets can only be AWS accounts DeploymentTargets: !If - MultiAccountDeploymentCondition - - !Join [',', !GetAtt GetOUAccountsStaging.Accounts] + - !GetAtt GetEnvironmentConfiguration.StagingAccountList - !Ref 'AWS::AccountId' Regions: !Ref 'AWS::Region' RunOrder: 1 @@ -437,13 +428,16 @@ Resources: Provider: Manual Configuration: CustomData: !Sub - - "Model ${ModelPackageGroupName} for the project ${SageMakerProjectName} is deployed to the staging accounts for OU ${OUStagingId}" + - "Model ${ModelPackageGroupName} for the project ${SageMakerProjectName} is deployed to the staging accounts ${AccountList}" - ModelPackageGroupName: !If - GenerateModelPackageNameCondition - !Sub '${SageMakerProjectName}-${SageMakerProjectId}' - !Ref ModelPackageGroupName SageMakerProjectName: !Ref SageMakerProjectName - OUStagingId: !GetAtt GetEnvironmentConfiguration.OUStagingId + AccountList: !If + - MultiAccountDeploymentCondition + - !GetAtt GetEnvironmentConfiguration.StagingAccountList + - !Ref 'AWS::AccountId' ExternalEntityLink: !Sub 'https://${AWS::Region}.console.aws.amazon.com/codesuite/codepipeline/pipelines/sagemaker-${SageMakerProjectName}-${SageMakerProjectId}-modeldeploy/view?region=${AWS::Region}' RunOrder: 3 @@ -470,7 +464,7 @@ Resources: # For a self-managed model, targets can only be AWS accounts DeploymentTargets: !If - MultiAccountDeploymentCondition - - !Join [',', !GetAtt GetOUAccountsProd.Accounts] + - !GetAtt GetEnvironmentConfiguration.ProdAccountList - !Ref 'AWS::AccountId' Regions: !Ref 'AWS::Region' RunOrder: 1 \ No newline at end of file diff --git a/mlops-seed-code/model-deploy/build.py b/mlops-seed-code/model-deploy/build.py index 548b06b..3a535c2 100644 --- a/mlops-seed-code/model-deploy/build.py +++ b/mlops-seed-code/model-deploy/build.py @@ -61,7 +61,7 @@ def prepare_config(args, model_package_arn, config_name, params): # Optional: Add validation of config parameters if needed # Add deployment-time parameters - config.append({ "ParameterKey": "OrgUnitId", "ParameterValue": params["OUId"] }) + config.append({ "ParameterKey": "Accounts", "ParameterValue": params["Accounts"] }) config.append({ "ParameterKey": "ExecutionRoleName", "ParameterValue": params["ExecutionRoleName"] }) config.append({ "ParameterKey": "SageMakerProjectName", "ParameterValue": args.sagemaker_project_name, }) config.append({ "ParameterKey": "SageMakerProjectId", "ParameterValue": args.sagemaker_project_id }) @@ -86,8 +86,8 @@ def prepare_config(args, model_package_arn, config_name, params): parser.add_argument("--prod-config-name", type=str, default="prod-config") parser.add_argument("--sagemaker-execution-role-staging-name", type=str, required=True) parser.add_argument("--sagemaker-execution-role-prod-name", type=str, required=True) - parser.add_argument("--organizational-unit-staging-id", type=str, required=True) - parser.add_argument("--organizational-unit-prod-id", type=str, required=True) + parser.add_argument("--staging-accounts", type=str, default='') + parser.add_argument("--prod-accounts", type=str, default='') parser.add_argument("--env-name", type=str, required=True) parser.add_argument("--ebs-kms-key-arn", type=str, required=True) parser.add_argument("--env-type-staging-name", type=str, required=True) @@ -106,12 +106,12 @@ def prepare_config(args, model_package_arn, config_name, params): for k, v in { args.staging_config_name:{ "ExecutionRoleName":args.sagemaker_execution_role_staging_name, - "OUId":args.organizational_unit_staging_id, + "Accounts":args.staging_accounts, "EnvType":args.env_type_staging_name }, args.prod_config_name:{ "ExecutionRoleName":args.sagemaker_execution_role_prod_name, - "OUId":args.organizational_unit_prod_id, + "Accounts":args.prod_accounts, "EnvType":args.env_type_prod_name } }.items(): diff --git a/mlops-seed-code/model-deploy/buildspec.yml b/mlops-seed-code/model-deploy/buildspec.yml index ee75364..79a6e6b 100644 --- a/mlops-seed-code/model-deploy/buildspec.yml +++ b/mlops-seed-code/model-deploy/buildspec.yml @@ -15,8 +15,8 @@ phases: python setup.py \ --sagemaker-project-id "$SAGEMAKER_PROJECT_ID" --sagemaker-project-name "$SAGEMAKER_PROJECT_NAME" \ --model-package-group-name "$SOURCE_MODEL_PACKAGE_GROUP_NAME" \ - --organizational-unit-staging-id "$ORGANIZATIONAL_UNIT_STAGING_ID" \ - --organizational-unit-prod-id "$ORGANIZATIONAL_UNIT_PROD_ID" \ + --staging-accounts "$STAGING_ACCOUNT_LIST" \ + --prod-accounts "$PROD_ACCOUNT_LIST" \ --env-name "$ENV_NAME" --env-type "$ENV_TYPE" \ --multi-account-deployment "$MULTI_ACCOUNT_DEPLOYMENT" @@ -28,8 +28,8 @@ phases: --staging-config-name "$STAGING_CONFIG_NAME" --prod-config-name "$PROD_CONFIG_NAME" \ --sagemaker-execution-role-staging-name "$SAGEMAKER_EXECUTION_ROLE_STAGING_NAME" \ --sagemaker-execution-role-prod-name "$SAGEMAKER_EXECUTION_ROLE_PROD_NAME" \ - --organizational-unit-staging-id "$ORGANIZATIONAL_UNIT_STAGING_ID" \ - --organizational-unit-prod-id "$ORGANIZATIONAL_UNIT_PROD_ID" \ + --staging-accounts "$STAGING_ACCOUNT_LIST" \ + --prod-accounts "$PROD_ACCOUNT_LIST" \ --env-name "$ENV_NAME" \ --ebs-kms-key-arn "$SAGEMAKER_EBS_KMS_KEY_ARN" \ --env-type-staging-name "$ENV_TYPE_STAGING_NAME" \ diff --git a/mlops-seed-code/model-deploy/setup.py b/mlops-seed-code/model-deploy/setup.py index 0ca5c44..4627001 100644 --- a/mlops-seed-code/model-deploy/setup.py +++ b/mlops-seed-code/model-deploy/setup.py @@ -18,8 +18,8 @@ parser.add_argument("--sagemaker-project-id", type=str, required=True) parser.add_argument("--sagemaker-project-name", type=str, required=True) parser.add_argument("--model-package-group-name", type=str, required=True) - parser.add_argument("--organizational-unit-staging-id", type=str, default='') - parser.add_argument("--organizational-unit-prod-id", type=str, default='') + parser.add_argument("--staging-accounts", type=str, default='') + parser.add_argument("--prod-accounts", type=str, default='') parser.add_argument("--env-name", type=str, required=True) parser.add_argument("--env-type", type=str, required=True) parser.add_argument("--multi-account-deployment", type=str, required=True) @@ -62,22 +62,19 @@ raise e if args.multi_account_deployment == "YES": - staging_ou_id = args.organizational_unit_staging_id - prod_ou_id = args.organizational_unit_prod_id - if not staging_ou_id or not prod_ou_id: + if not len(args.staging_accounts) or not len(args.prod_accounts.split): error_message = ( - f"Staging OU {staging_ou_id} or production OU {prod_ou_id} are not provided for multi-account-deployment" + f"Staging accounts {args.staging_accounts} or production accounts {args.prod_accounts.split} are not provided for multi-account-deployment" ) logger.error(error_message) raise Exception(error_message) - logger.info(f"Staging OU: {staging_ou_id} and Production OU: {prod_ou_id} are provided. Setting up the permissions...") + logger.info(f"Staging accounts: {args.staging_accounts} and production accounts: {args.prod_accounts} are provided. Setting up the permissions...") - # Get the account ids based on staging and prod ids + # Construct the principals for the account ids principals = [f"arn:aws:iam::{acc}:root" for acc in - [i['Id'] for i in org_client.list_accounts_for_parent(ParentId=staging_ou_id)['Accounts']] + - [i['Id'] for i in org_client.list_accounts_for_parent(ParentId=prod_ou_id)['Accounts']]] + args.staging_accounts.split(",") + args.prod_accounts.split(",")] # create policy for cross-account access to the ModelPackageGroup sm_client.put_model_package_group_policy( diff --git a/mlops-seed-code/model-deploy/test/buildspec.yml b/mlops-seed-code/model-deploy/test/buildspec.yml index dff4059..1018440 100644 --- a/mlops-seed-code/model-deploy/test/buildspec.yml +++ b/mlops-seed-code/model-deploy/test/buildspec.yml @@ -10,9 +10,8 @@ phases: - | python test/test.py \ --build-config $CODEBUILD_SRC_DIR_BuildArtifact/${BUILD_CONFIG}.json \ - --test-results-output ${TEST_RESULTS}.json \ - --multi-account-deployment "$MULTI_ACCOUNT_DEPLOYMENT" - + --test-results-output ${TEST_RESULTS}.json + # Show the test results file - cat ${TEST_RESULTS}.json diff --git a/mlops-seed-code/model-deploy/test/test.py b/mlops-seed-code/model-deploy/test/test.py index 6edaaed..1e26935 100644 --- a/mlops-seed-code/model-deploy/test/test.py +++ b/mlops-seed-code/model-deploy/test/test.py @@ -55,7 +55,6 @@ def test_endpoint(endpoint_name, sm_client): parser.add_argument("--log-level", type=str, default=os.environ.get("LOGLEVEL", "INFO").upper()) parser.add_argument("--build-config", type=str, required=True) parser.add_argument("--test-results-output", type=str, required=True) - parser.add_argument("--multi-account-deployment", type=str, required=True) args, _ = parser.parse_known_args() # Configure logging to output the line number and message @@ -68,23 +67,13 @@ def test_endpoint(endpoint_name, sm_client): boto_sts=boto3.client('sts') - # get the caller account for single-account deployment - account_ids = [boto_sts.get_caller_identity()["Account"]] - - if args.multi_account_deployment == "YES": - # Multi-account deployment to all accounts in the OU if multi-account-deployment set to YES - if config["OrgUnitId"]: - account_ids = [i['Id'] for i in org_client.list_accounts_for_parent(ParentId=config["OrgUnitId"])['Accounts']] - else: - error_message = ( - f"OU is not provided for multi-account-deployment" - ) - logger.error(error_message) - raise Exception(error_message) - - logger.info(f"Multi-account deployment enabled. Test endpoint for the accounts {account_ids} in {config['OrgUnitId']}") - - # Test the endpoint in each account of the target organizational unit + # Get the target account list + if config["Accounts"]: + account_ids = config["Accounts"].split(",") + else: # get the caller account for single-account deployment + account_ids = [boto_sts.get_caller_identity()["Account"]] + + # Test the endpoint in each account of the target account list logger.info(f"Test endpoint for the accounts: {account_ids}") for account_id in account_ids: # Request to assume the specified role in the target account diff --git a/package-cfn.sh b/package-cfn.sh index 6a0038a..50c7e77 100755 --- a/package-cfn.sh +++ b/package-cfn.sh @@ -116,6 +116,9 @@ do aws s3 cp ${CFN_OUTPUT_DIR}/${fname} s3://${CFN_BUCKET_NAME}/${PROJECT_NAME}/${fname} fi + echo "To validate template ${fname}:" + echo "aws cloudformation validate-template --template-url https://s3.${DEPLOYMENT_REGION}.amazonaws.com/${CFN_BUCKET_NAME}/${PROJECT_NAME}/${fname}" + echo "To deploy stack execute:" echo "aws cloudformation create-stack --template-url https://s3.${DEPLOYMENT_REGION}.amazonaws.com/${CFN_BUCKET_NAME}/${PROJECT_NAME}/${fname} --region ${DEPLOYMENT_REGION} --stack-name --disable-rollback --capabilities CAPABILITY_IAM CAPABILITY_NAMED_IAM --parameters ParameterKey=,ParameterValue=" diff --git a/test/cfn-test-e2e.sh b/test/cfn-test-e2e.sh index 490a114..476ab96 100644 --- a/test/cfn-test-e2e.sh +++ b/test/cfn-test-e2e.sh @@ -79,6 +79,8 @@ STACK_NAME="sm-mlops-env" ENV_NAME="sm-mlops" STAGING_OU_ID="ou-fi18-56v340tb" PROD_OU_ID="ou-fi18-9fex2edg" +STAGING_ACCOUNTS="322676557848" +PROD_ACCOUNTS="404668521988" SETUP_STACKSET_ROLE_NAME=$ENV_NAME-setup-stackset-execution-role aws cloudformation create-stack \ @@ -92,10 +94,12 @@ aws cloudformation create-stack \ ParameterKey=EnvType,ParameterValue=dev \ ParameterKey=AvailabilityZones,ParameterValue=${AWS_DEFAULT_REGION}a\\,${AWS_DEFAULT_REGION}b \ ParameterKey=NumberOfAZs,ParameterValue=2 \ - ParameterKey=StartKernelGatewayApps,ParameterValue=YES \ - # This parameter block is only needed for multi-account model deployment + ParameterKey=StartKernelGatewayApps,ParameterValue=NO \ + ParameterKey=SeedCodeS3BucketName,ParameterValue=$S3_BUCKET_NAME \ ParameterKey=OrganizationalUnitStagingId,ParameterValue=$STAGING_OU_ID \ ParameterKey=OrganizationalUnitProdId,ParameterValue=$PROD_OU_ID \ + ParameterKey=StagingAccountList,ParameterValue=$STAGING_ACCOUNTS \ + ParameterKey=ProductionAccountList,ParameterValue=$PROD_ACCOUNTS \ ParameterKey=SetupStackSetExecutionRoleName,ParameterValue=$SETUP_STACKSET_ROLE_NAME @@ -121,9 +125,9 @@ ENV_STACK_NAME="sm-mlops-env" CORE_STACK_NAME="sm-mlops-core" ENV_NAME="sm-mlops-dev" -MLOPS_PROJECT_NAME_LIST=("test3-deploy" "test4-train" "test4-deploy") -MLOPS_PROJECT_ID_LIST=("p-mc3zkgsbl8v3" "p-uode4b0qf2un" "p-y1iloulknggg") -SM_DOMAIN_ID="d-hlnftwb2ywkw" +MLOPS_PROJECT_NAME_LIST=("test18-train" "test20-deploy") +MLOPS_PROJECT_ID_LIST=("p-jetabw9lhqhl" "p-1cmngbt4ogao") +SM_DOMAIN_ID="d-h9kgl9eqt4ia" STACKSET_NAME_LIST=("sagemaker-test4-deploy-p-y1iloulknggg-deploy-staging" "sagemaker-test4-deploy-p-y1iloulknggg-deploy-prod") ACCOUNT_IDS="949335012047" @@ -192,7 +196,7 @@ aws sagemaker delete-app \ --app-type KernelGateway \ --app-name -# The following commands are only for manual deployment (not with CI/CD pipelines) +# The following commands are for clean-up after for manual deployment only (and not after automated tests with CI/CD pipelines) echo "Delete data science stack" aws cloudformation delete-stack --stack-name $ENV_STACK_NAME diff --git a/test/cfn-test.sh b/test/cfn-test.sh index e10b07c..59493ef 100644 --- a/test/cfn-test.sh +++ b/test/cfn-test.sh @@ -238,7 +238,8 @@ aws cloudformation create-stack \ ParameterKey=CreateVPCFlowLogsToCloudWatch,ParameterValue=NO \ ParameterKey=CreateVPCFlowLogsRole,ParameterValue=NO \ ParameterKey=AvailabilityZones,ParameterValue=${AWS_DEFAULT_REGION}a\\,${AWS_DEFAULT_REGION}b\\,${AWS_DEFAULT_REGION}c \ - ParameterKey=NumberOfAZs,ParameterValue=3 + ParameterKey=NumberOfAZs,ParameterValue=3 \ + ParameterKey=SeedCodeS3BucketName,ParameterValue=$S3_BUCKET_NAME # Clean up aws cloudformation delete-stack --stack-name ds-team-env