From c2d591d9bad829b90bdf3f3f2ff1abaaeea4547b Mon Sep 17 00:00:00 2001 From: Brandon Liu Date: Thu, 12 Sep 2024 17:28:00 +0800 Subject: [PATCH] update AWS docs for CloudFormation. --- deploy/aws.md | 174 ++++++++++++-------------------------------------- 1 file changed, 42 insertions(+), 132 deletions(-) diff --git a/deploy/aws.md b/deploy/aws.md index 43b17f5..f955bf6 100644 --- a/deploy/aws.md +++ b/deploy/aws.md @@ -5,150 +5,60 @@ outline: deep # AWS Integration +The AWS Deployment option is based on [CloudFormation](https://aws.amazon.com/cloudformation/), which automates the creation and deletion of all resources. You don't need to install anything; simply upload a YAML to the AWS Console. + +At the end of this walkthrough, you'll have a CDN-cached ZXY API, compatible with all major map renderers, serving tiles from a private S3 bucket. + ## Installation ### 1. Upload to S3 +The CloudFormation template is designed to work with an existing S3 bucket. + +If you need to create a new one: + * Open the [S3 Console](https://s3.console.aws.amazon.com/s3/home) and choose **Create Bucket**. -* The name must be globally unique. Choose any region, just remember the **region name.** +* Choose a globally unique bucket name, and any region, just remember the **region name.** * We'll use `protomaps-example` as a placeholder bucket name. * Leave the default *ACLs disabled* and *Block all public access* setting. - * Leave other options as the default and proceed with Create Bucket. -* **Upload** a PMTiles archive: we'll use `my_file.pmtiles` as an example: to your bucket via the Web Console or a tool like `pmtiles` or `rclone`. - -### 2. Lambda function - -* Open the **Lambda** dashboard in the **same region as your bucket.** -* Choose **Create Function**. - * Name your function `protomaps`. - * For Runtime, leave the default choice `Node.js 20.x`. - * For Architecture, choose `arm64`. - * Under Change Default Execution Role, leave the default `Create a new role with basic Lambda Permissions`. - * This will auto-generate a role name. - * Under Advanced Settings, Choose **Enable Function URL.** - * Under **Auth Type**, choose `NONE`. - * Proceed with Create Function. -* On the Configuration tab, choose **General Configuration** > **Edit**. - * set `Memory` to **512 MB**. This is required, and more cost effective than the default of 128. -* On the Configuration tab, choose **Environment Variables** > **Edit**. - * set `BUCKET` to your unique **bucket name** from Step 1. - * set `PUBLIC_HOSTNAME` to the **public custom domain name you'll assign to your CloudFront distribution.** *TileJSON responses won't work without setting this.* Example: `tiles.example.com` -* In the **Code** tab, replace the code contents with the bundled [index.mjs](https://pmtiles.io/lambda_function.zip) from [PMTiles/serverless/aws](https://github.com/protomaps/PMTiles/tree/main/serverless/aws). -* Choose **Deploy** to deploy the function. - -### 3. Lambda role permissions - -* In the **Configuration** > **Permissions** tab, follow the link under **Execution Role > Role Name** to navigate to the function's IAM role. -* On the right side, choose **Add Permissions > Create Inline Policy**. -* Choose the **JSON** tab and paste in the following: - -(Replace `protomaps-example` with your bucket created in Step 1) - -```json -{ - "Version": "2012-10-17", - "Statement": [ - { - "Sid": "VisualEditor0", - "Effect": "Allow", - "Action": "s3:GetObject", - "Resource": "arn:aws:s3:::protomaps-example/*" - } - ] -} -``` - -* Give this policy any name eg `protomaps-lambda` and **Create Policy**. -* Return to your Lambda function's **Configuration** and **Function URL** page. - -The Function URL that looks like `https://AAAA.lambda-url.region-name.on.aws/` is now active. - -Make a test request for the `/my_file/0/0/0.mvt` (or `.png`, `.jpg` - whatever matches your data) path in your browser. This should succeed with your tile data! - -::: warning -This Public Lambda URL should not be used directly by browsers because it lacks caching and CORS headers, which we'll configure next. -::: - -### 4. CloudFront - -* Navigate to the CloudFront Dashboard and choose **Create Distribution**. - -* Enter your Lambda Function URL under **Origin Domain**. - -* for Cache Policy, choose **CachingOptimized** which sets a default TTL of 86400 seconds. + * Leave other options as the default and proceed with **Create Bucket.** +* **Upload** a PMTiles archive. For example, save [this file](https://pmtiles.io/protomaps(vector)ODbL_firenze.pmtiles) as `example.pmtiles`. Upload to your bucket via the Web Console or [a tool like rclone.](/pmtiles/cloud-storage#uploading) -* for Viewer Protocol Policy, choose **Redirect HTTP to HTTPS**. +### 2. CloudFormation Template -* Under Response Headers Policy, choose **Create Policy**. +1. In the CloudFormation console for your AWS region, choose **Create Stack > With new resources (standard)**. - * enter a name `protomaps-cors` +2. Choose **Specify Template > Upload a template file** and upload the file [cloudformation-stack.yaml](http://pmtiles.io/cloudformation-stack.yaml). - * Enable the slider **Configure CORS**. +3. Provide a stack name of your choice. - * Choose **Customize** under Access-Control-Allow-Origin and enter full allowed origins e.g. `https://example.com` +4. for **Parameters**, specify: - * Leave other settings as the default and proceed with **Create**. + * The allowed CORS origins. By default, all sites (`*`) are authorized to make requests. Specify a comma-separated allowlist of sites e.g. `example.com,example.io`. - * Return to the CloudFront Configuration and choose `protomaps-cors` in the **Response headers policy** dropdown (you might need to refresh). + * The name of the bucket from step 1. -* Under Settings, check **HTTP/3** in addition to HTTP/2. + * The CloudFront cache TTL, which is how long tiles will be cached at the edge. Default 1 day. -* Enter a **Description** like `protomaps`. + * (Optional) the public hostname for TileJSON. If you plan to add a custom domain like `tiles.example.com`, enter that here. Otherwise leave this blank for the default `*.cloudfront.net` hostname. -* Proceed with Create Distribution. -This may take a few minutes, where the `Last modified` value will display `... Deploying`. When that's complete, you will have a working CloudFront distribution at a URL like `AAAA.cloudfront.net` that can be accessed directly from browsers. +5. Proceed with **Next > Submit**, acknowledging that it might create IAM resources. -Accessing your Distribution from a web map should verify that tiles are cached on second request. Tile headers should include: +This may take a few minutes to create the CDN distribution. When that's complete, you will have a URL like `AAAA.cloudfront.net` that can be accessed directly from browsers: ``` -x-cache: Hit from cloudfront +https://SUBDOMAIN.cloudfront.net/TILESET.json # TileJSON for MapLibre +https://SUBDOMAIN.cloudfront.net/TILESET/{z}/{x}/{y}.{ext} ``` -You may next want to assign a custom domain name to your distribution through Route 53 and Certificate Manager. - -## Configuration - -Configure these Lambda environment variables: - -* `BUCKET`: the S3 bucket name. -* `PMTILES_PATH`: optional, define how a tileset name is translated into an S3 key. Default `{name}.pmtiles` - * Example path setting for objects in a directory: `my_folder/{name}/file.pmtiles` -* `CORS`: optional, set the value of the `Access-Control-Allow-Origin` response header. Examples: `https://example.com`, `*`. Only supports one origin, so useful for development or staging environments only. For production use you should use CloudFront CORS configuration. -* ~~`CACHE_MAX_AGE`: max age in the CloudFront cache, in seconds. default 86400, or 1 day.~~ -* `CACHE_CONTROL`: HTTP header value to control caching, default `public, max-age=86400` (1 day). - -## Accessing your Tiles - -The default Cloudfront URL for your tiles: - -``` -https://SUBDOMAIN.cloudfront.net/ARCHIVE_NAME/{z}/{x}/{y}.{ext} -``` - -where `{ext}` is the file extension - `mvt`, `jpg`, or `png` - matching your tileset, and `SUBDOMAIN` is found at the **Distribution Domain Name** in the [CloudFront Console](https://us-east-1.console.aws.amazon.com/cloudfront/v3/). - -If you're using [MapLibre](/pmtiles/maplibre), it's more convenient to fetch [TileJSON](https://github.com/mapbox/tilejson-spec/tree/master/3.0.0), which will detect the tileset `minzoom` and `maxzoom`: - -``` -https://SUBDOMAIN.cloudfront.net/ARCHIVE_NAME.json -``` - -## TileJSON - -You can access a [TileJSON](https://github.com/mapbox/tilejson-spec) document for each tileset: +Accessing your Distribution from a web map should verify that tiles are cached on second request. Tile headers should include: ``` -https://PUBLIC_HOSTNAME/ARCHIVE_NAME.json +x-cache: Hit from cloudfront ``` -These endpoints will return a 404 unless the `PUBLIC_HOSTNAME` environment variable is set. - - -## Cache Invalidation - -* For AWS Cloudfront, issue prefix-based invalidations at the [Cloudfront Console](https://us-east-1.console.aws.amazon.com/cloudfront/v3/home). The first 1000 invalidations per month are free (a prefix = 1 invalidation). -* A cache purge will result in new billable events in Lambda and reads from the origin S3 object. +You may next want to assign a custom domain name to your distribution through Route 53 and Certificate Manager and update the public hostname in step 4 above. ## Monitoring @@ -158,33 +68,33 @@ These endpoints will return a 404 unless the `PUBLIC_HOSTNAME` environment varia ### Recommended Metrics * **Cache Hit Rate**: All Metrics > Cloudfront > Per-Distribution Metrics > Requests -* **Lambda Invocations**: All Metrics > Lambda > By Function Name (protomaps) > Invocations +* **Lambda Invocations**: All Metrics > Lambda > By Function Name > Invocations -* **Lambda Execution Time**: All Metrics > Lambda > By Funcion Name (protomaps) > Duration +* **Lambda Execution Time**: All Metrics > Lambda > By Funcion Name > Duration ::: info Typical execution times for a properly configured AWS install are 125 ms p50 (mean), 800 ms p99. ::: - ## Cleanup +Deleting the CloudFormation stack will delete the CloudFront distribution, Lambda function and all associated resources, but leave the bucket untouched. -* **Disable** and then **Delete** the **CloudFront Distribution**. -* Delete the **Lambda function and associated policy** -* Delete the **S3 Bucket.** -* Delete the **IAM role.** -* (Custom domain name) **Delete** the certificate in **Certificate Manager.** -* (Custom domain name) **Delete** the A and AAAA entries in **Route 53**. +## Cache Invalidation +* For AWS Cloudfront, issue prefix-based invalidations at the [Cloudfront Console](https://us-east-1.console.aws.amazon.com/cloudfront/v3/home). The first 1000 invalidations per month are free (a prefix = 1 invalidation). +* A cache purge will result in new billable events in Lambda and reads from the origin S3 object. -## Other Deployment Options +## Lambda Configuration -### Lambda@Edge +The raw Lambda function embedded in the CloudFormation template is also available at [lambda_function.zip](https://pmtiles.io/lambda_function.zip). Note that this must be uploaded as a `.zip` file containing the single file `index.js`. -Lambda@Edge's multi-region features have little benefit when fetching data from S3 in a single region, and Lambda@Edge [doesn't support](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/edge-functions-restrictions.html) environment variables or responses over 1 MB. For globally distributed caching, use CloudFront in combination with Lambda Function URLs. +Lambda environment variables: -### API Gateway +* `BUCKET`: the S3 bucket name. +* `PMTILES_PATH`: optional, define how a tileset name is translated into an S3 key. Default `{name}.pmtiles` + * Example path setting for objects in a directory: `my_folder/{name}/file.pmtiles` +* `CORS`: optional, set the value of the `Access-Control-Allow-Origin` response header. Examples: `https://example.com`, `*`. Only supports one origin, so useful for development or staging environments only. For production use you should use CloudFront CORS configuration. +* ~~`CACHE_MAX_AGE`: max age in the CloudFront cache, in seconds. default 86400, or 1 day.~~ +* `CACHE_CONTROL`: HTTP header value to control caching, default `public, max-age=86400` (1 day). -* your Lambda Proxy Integration route will need to specify a greedy capturing parameter called `proxy` e.g. `/{proxy+}` (the default). -* API Gateway responses will always be GZIP-encoded, to work around binary content detection problems.