Skip to content

Commit

Permalink
change AWS guide to use CloudFormation. (#59)
Browse files Browse the repository at this point in the history
  • Loading branch information
@bdon
bdon authored Sep 12, 2024
1 parent 007d69d commit 119d242
Showing 1 changed file with 42 additions and 132 deletions.
174 changes: 42 additions & 132 deletions deploy/aws.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,150 +5,60 @@ outline: deep

# AWS Integration

The AWS Deployment option is based on [CloudFormation](https://aws.amazon.com/cloudformation/), which automates the creation and deletion of all resources. You don't need to install anything; simply upload a YAML to the AWS Console.

At the end of this walkthrough, you'll have a CDN-cached ZXY API, compatible with all major map renderers, serving tiles from a private S3 bucket.

## Installation

### 1. Upload to S3

The CloudFormation template is designed to work with an existing S3 bucket.

If you need to create a new one:

* Open the [S3 Console](https://s3.console.aws.amazon.com/s3/home) and choose **Create Bucket**.
* The name must be globally unique. Choose any region, just remember the **region name.**
* Choose a globally unique bucket name, and any region, just remember the **region name.**
* We'll use `protomaps-example` as a placeholder bucket name.
* Leave the default *ACLs disabled* and *Block all public access* setting.
* Leave other options as the default and proceed with Create Bucket.
* **Upload** a PMTiles archive: we'll use `my_file.pmtiles` as an example: to your bucket via the Web Console or a tool like `pmtiles` or `rclone`.

### 2. Lambda function

* Open the **Lambda** dashboard in the **same region as your bucket.**
* Choose **Create Function**.
* Name your function `protomaps`.
* For Runtime, leave the default choice `Node.js 20.x`.
* For Architecture, choose `arm64`.
* Under Change Default Execution Role, leave the default `Create a new role with basic Lambda Permissions`.
* This will auto-generate a role name.
* Under Advanced Settings, Choose **Enable Function URL.**
* Under **Auth Type**, choose `NONE`.
* Proceed with Create Function.
* On the Configuration tab, choose **General Configuration** > **Edit**.
* set `Memory` to **512 MB**. This is required, and more cost effective than the default of 128.
* On the Configuration tab, choose **Environment Variables** > **Edit**.
* set `BUCKET` to your unique **bucket name** from Step 1.
* set `PUBLIC_HOSTNAME` to the **public custom domain name you'll assign to your CloudFront distribution.** *TileJSON responses won't work without setting this.* Example: `tiles.example.com`
* In the **Code** tab, replace the code contents with the bundled [index.mjs](https://pmtiles.io/lambda_function.zip) from [PMTiles/serverless/aws](https://github.com/protomaps/PMTiles/tree/main/serverless/aws).
* Choose **Deploy** to deploy the function.

### 3. Lambda role permissions

* In the **Configuration** > **Permissions** tab, follow the link under **Execution Role > Role Name** to navigate to the function's IAM role.
* On the right side, choose **Add Permissions > Create Inline Policy**.
* Choose the **JSON** tab and paste in the following:

(Replace `protomaps-example` with your bucket created in Step 1)

```json
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": "s3:GetObject",
"Resource": "arn:aws:s3:::protomaps-example/*"
}
]
}
```

* Give this policy any name eg `protomaps-lambda` and **Create Policy**.
* Return to your Lambda function's **Configuration** and **Function URL** page.

The Function URL that looks like `https://AAAA.lambda-url.region-name.on.aws/` is now active.

Make a test request for the `/my_file/0/0/0.mvt` (or `.png`, `.jpg` - whatever matches your data) path in your browser. This should succeed with your tile data!

::: warning
This Public Lambda URL should not be used directly by browsers because it lacks caching and CORS headers, which we'll configure next.
:::

### 4. CloudFront

* Navigate to the CloudFront Dashboard and choose **Create Distribution**.

* Enter your Lambda Function URL under **Origin Domain**.

* for Cache Policy, choose **CachingOptimized** which sets a default TTL of 86400 seconds.
* Leave other options as the default and proceed with **Create Bucket.**
* **Upload** a PMTiles archive. For example, save [this file](https://pmtiles.io/protomaps(vector)ODbL_firenze.pmtiles) as `example.pmtiles`. Upload to your bucket via the Web Console or [a tool like rclone.](/pmtiles/cloud-storage#uploading)

* for Viewer Protocol Policy, choose **Redirect HTTP to HTTPS**.
### 2. CloudFormation Template

* Under Response Headers Policy, choose **Create Policy**.
1. In the CloudFormation console for your AWS region, choose **Create Stack > With new resources (standard)**.

* enter a name `protomaps-cors`
2. Choose **Specify Template > Upload a template file** and upload the file [cloudformation-stack.yaml](http://pmtiles.io/cloudformation-stack.yaml).

* Enable the slider **Configure CORS**.
3. Provide a stack name of your choice.

* Choose **Customize** under Access-Control-Allow-Origin and enter full allowed origins e.g. `https://example.com`
4. for **Parameters**, specify:

* Leave other settings as the default and proceed with **Create**.
* The allowed CORS origins. By default, all sites (`*`) are authorized to make requests. Specify a comma-separated allowlist of sites e.g. `example.com,example.io`.

* Return to the CloudFront Configuration and choose `protomaps-cors` in the **Response headers policy** dropdown (you might need to refresh).
* The name of the bucket from step 1.

* Under Settings, check **HTTP/3** in addition to HTTP/2.
* The CloudFront cache TTL, which is how long tiles will be cached at the edge. Default 1 day.

* Enter a **Description** like `protomaps`.
* (Optional) the public hostname for TileJSON. If you plan to add a custom domain like `tiles.example.com`, enter that here. Otherwise leave this blank for the default `*.cloudfront.net` hostname.

* Proceed with Create Distribution.

This may take a few minutes, where the `Last modified` value will display `... Deploying`. When that's complete, you will have a working CloudFront distribution at a URL like `AAAA.cloudfront.net` that can be accessed directly from browsers.
5. Proceed with **Next > Submit**, acknowledging that it might create IAM resources.

Accessing your Distribution from a web map should verify that tiles are cached on second request. Tile headers should include:
This may take a few minutes to create the CDN distribution. When that's complete, you will have a URL like `AAAA.cloudfront.net` that can be accessed directly from browsers:

```
x-cache: Hit from cloudfront
https://SUBDOMAIN.cloudfront.net/TILESET.json # TileJSON for MapLibre
https://SUBDOMAIN.cloudfront.net/TILESET/{z}/{x}/{y}.{ext}
```

You may next want to assign a custom domain name to your distribution through Route 53 and Certificate Manager.

## Configuration

Configure these Lambda environment variables:

* `BUCKET`: the S3 bucket name.
* `PMTILES_PATH`: optional, define how a tileset name is translated into an S3 key. Default `{name}.pmtiles`
* Example path setting for objects in a directory: `my_folder/{name}/file.pmtiles`
* `CORS`: optional, set the value of the `Access-Control-Allow-Origin` response header. Examples: `https://example.com`, `*`. Only supports one origin, so useful for development or staging environments only. For production use you should use CloudFront CORS configuration.
* ~~`CACHE_MAX_AGE`: max age in the CloudFront cache, in seconds. default 86400, or 1 day.~~
* `CACHE_CONTROL`: HTTP header value to control caching, default `public, max-age=86400` (1 day).

## Accessing your Tiles

The default Cloudfront URL for your tiles:

```
https://SUBDOMAIN.cloudfront.net/ARCHIVE_NAME/{z}/{x}/{y}.{ext}
```

where `{ext}` is the file extension - `mvt`, `jpg`, or `png` - matching your tileset, and `SUBDOMAIN` is found at the **Distribution Domain Name** in the [CloudFront Console](https://us-east-1.console.aws.amazon.com/cloudfront/v3/).

If you're using [MapLibre](/pmtiles/maplibre), it's more convenient to fetch [TileJSON](https://github.com/mapbox/tilejson-spec/tree/master/3.0.0), which will detect the tileset `minzoom` and `maxzoom`:

```
https://SUBDOMAIN.cloudfront.net/ARCHIVE_NAME.json
```

## TileJSON

You can access a [TileJSON](https://github.com/mapbox/tilejson-spec) document for each tileset:
Accessing your Distribution from a web map should verify that tiles are cached on second request. Tile headers should include:

```
https://PUBLIC_HOSTNAME/ARCHIVE_NAME.json
x-cache: Hit from cloudfront
```

These endpoints will return a 404 unless the `PUBLIC_HOSTNAME` environment variable is set.


## Cache Invalidation

* For AWS Cloudfront, issue prefix-based invalidations at the [Cloudfront Console](https://us-east-1.console.aws.amazon.com/cloudfront/v3/home). The first 1000 invalidations per month are free (a prefix = 1 invalidation).
* A cache purge will result in new billable events in Lambda and reads from the origin S3 object.
You may next want to assign a custom domain name to your distribution through Route 53 and Certificate Manager and update the public hostname in step 4 above.

## Monitoring

Expand All @@ -158,33 +68,33 @@ These endpoints will return a 404 unless the `PUBLIC_HOSTNAME` environment varia
### Recommended Metrics

* **Cache Hit Rate**: All Metrics > Cloudfront > Per-Distribution Metrics > Requests
* **Lambda Invocations**: All Metrics > Lambda > By Function Name (protomaps) > Invocations
* **Lambda Invocations**: All Metrics > Lambda > By Function Name > Invocations

* **Lambda Execution Time**: All Metrics > Lambda > By Funcion Name (protomaps) > Duration
* **Lambda Execution Time**: All Metrics > Lambda > By Funcion Name > Duration

::: info
Typical execution times for a properly configured AWS install are 125 ms p50 (mean), 800 ms p99.
:::


## Cleanup

Deleting the CloudFormation stack will delete the CloudFront distribution, Lambda function and all associated resources, but leave the bucket untouched.

* **Disable** and then **Delete** the **CloudFront Distribution**.
* Delete the **Lambda function and associated policy**
* Delete the **S3 Bucket.**
* Delete the **IAM role.**
* (Custom domain name) **Delete** the certificate in **Certificate Manager.**
* (Custom domain name) **Delete** the A and AAAA entries in **Route 53**.
## Cache Invalidation

* For AWS Cloudfront, issue prefix-based invalidations at the [Cloudfront Console](https://us-east-1.console.aws.amazon.com/cloudfront/v3/home). The first 1000 invalidations per month are free (a prefix = 1 invalidation).
* A cache purge will result in new billable events in Lambda and reads from the origin S3 object.

## Other Deployment Options
## Lambda Configuration

### Lambda@Edge
The raw Lambda function embedded in the CloudFormation template is also available at [lambda_function.zip](https://pmtiles.io/lambda_function.zip). Note that this must be uploaded as a `.zip` file containing the single file `index.js`.

Lambda@Edge's multi-region features have little benefit when fetching data from S3 in a single region, and Lambda@Edge [doesn't support](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/edge-functions-restrictions.html) environment variables or responses over 1 MB. For globally distributed caching, use CloudFront in combination with Lambda Function URLs.
Lambda environment variables:

### API Gateway
* `BUCKET`: the S3 bucket name.
* `PMTILES_PATH`: optional, define how a tileset name is translated into an S3 key. Default `{name}.pmtiles`
* Example path setting for objects in a directory: `my_folder/{name}/file.pmtiles`
* `CORS`: optional, set the value of the `Access-Control-Allow-Origin` response header. Examples: `https://example.com`, `*`. Only supports one origin, so useful for development or staging environments only. For production use you should use CloudFront CORS configuration.
* ~~`CACHE_MAX_AGE`: max age in the CloudFront cache, in seconds. default 86400, or 1 day.~~
* `CACHE_CONTROL`: HTTP header value to control caching, default `public, max-age=86400` (1 day).

* your Lambda Proxy Integration route will need to specify a greedy capturing parameter called `proxy` e.g. `/{proxy+}` (the default).
* API Gateway responses will always be GZIP-encoded, to work around binary content detection problems.

0 comments on commit 119d242

Please sign in to comment.