From 0139ac798e5d874145efc6a71954540ddeb61fa5 Mon Sep 17 00:00:00 2001 From: Gus Luxton Date: Thu, 5 Mar 2020 13:29:04 -0400 Subject: [PATCH 01/13] First draft of Terraform docs --- docs/4.2.yaml | 1 + docs/4.2/aws_terraform_guide.md | 391 ++++++++++++++++++++++++++++++++ 2 files changed, 392 insertions(+) create mode 100644 docs/4.2/aws_terraform_guide.md diff --git a/docs/4.2.yaml b/docs/4.2.yaml index 6ab6fe804b9ae..0dc3ffe4b165a 100644 --- a/docs/4.2.yaml +++ b/docs/4.2.yaml @@ -48,6 +48,7 @@ nav: - Using Teleport with PAM: features/ssh_pam.md - Infrastructure Guides: - AWS: aws_oss_guide.md + - AWS HA with Terraform: aws_terraform_guide.md - GCP: gcp_guide.md - IBM: ibm_cloud_guide.md - Kubernetes Guide: kubernetes_ssh.md diff --git a/docs/4.2/aws_terraform_guide.md b/docs/4.2/aws_terraform_guide.md new file mode 100644 index 0000000000000..990dd7a2b4bdc --- /dev/null +++ b/docs/4.2/aws_terraform_guide.md @@ -0,0 +1,391 @@ +# Running Teleport Enterprise in a high-availiability configuration on AWS + +This guide is designed to accompany our reference Terraform code (https://github.com/gravitational/teleport/tree/master/examples/aws/terraform) and describe how to use and administrate the resulting Teleport deployment. + +## Prerequisites + +Our code requires Terraform 0.12+. You can [download Terraform here](https://www.terraform.io/downloads.html). We will assume that you have `terraform` installed and available on your path. + +```bash +$ terraform version +Terraform v0.12.20 +``` + +You will also require the `aws` command line tool. This is available in Ubuntu/Debian/Fedora/CentOS +and MacOS Homebrew as the `awscli` package. + +When possible, installing via a package is always preferable. If you can't find a package available for +your distribution, you can install `python3` and `pip`, then use these to install `awscli` using `pip3 install awscli` + +We will assume that you have configured your AWS cli access with credentials available at `~/.aws/credentials`: + +```bash +$ cat ~/.aws/credentials +[default] +aws_access_key_id = AKIA.... +aws_secret_access_key = 8ZRxy.... +``` + +You should also have a default region set under `~/.aws/config`: + +```bash +$ cat ~/.aws/config +[default] +region = us-east-1 +``` + +As a result, you should be able to run a command like `aws ec2 describe-images --owners 126027368216 --filters 'Name=name,Values=gravitational-teleport-ami-ent*` +to list available EC2 images matching a pattern. If you get an "access denied" or similar message, you will need +to grant additional permissions to the AWS IAM user that your `aws_access_key_id` and `aws_secret_access_key` refers to. + +# TODO(gus): detailed explanation of IAM permissions needed + +## AWS Services required to run Teleport in HA + +- [EC2 / Autoscale](#ec2-autoscale) +- [DynamoDB](#dynamodb) +- [S3](#s3://) +- [Route53](#route53) +- [NLB](#nlb-network-load-balancer) +- [IAM](#iam) +- [ACM](#acm) +- [SSM](#aws-systems-manager-parameter-store) + + +## Get the Terraform code + +Firstly, you'll need to clone the Teleport repo to get the Terraform code available on your system. + +```bash +$ git clone https://github.com/gravitational/teleport +Cloning into 'teleport'... +remote: Enumerating objects: 106, done. +remote: Counting objects: 100% (106/106), done. +remote: Compressing objects: 100% (95/95), done. +remote: Total 61144 (delta 33), reused 35 (delta 11), pack-reused 61038 +Receiving objects: 100% (61144/61144), 85.17 MiB | 4.66 MiB/s, done. +Resolving deltas: 100% (39141/39141), done. +``` + +Once this is done, you can change into the directory where the Terraform code is checked out and run `terraform init`: + +```bash +$ cd teleport/examples/aws/terraform +$ terraform init + +Initializing the backend... + +Initializing provider plugins... +- Checking for available provider plugins... +- Downloading plugin for provider "template" (hashicorp/template) 2.1.2... +- Downloading plugin for provider "aws" (hashicorp/aws) 2.51.0... +- Downloading plugin for provider "random" (hashicorp/random) 2.2.1... + +Terraform has been successfully initialized! + +You may now begin working with Terraform. Try running "terraform plan" to see +any changes that are required for your infrastructure. All Terraform commands +should now work. + +If you ever set or change modules or backend configuration for Terraform, +rerun this command to reinitialize your working directory. If you forget, other +commands will detect it and remind you to do so if necessary. +``` + +This will download the appropriate Terraform plugins needed to spin up Teleport using our +reference code. + +## Set up variables + +Terraform modules use variables to pass in input. You can do this on the command line to `terraform plan`, +by editing the `vars.tf` file or the way we do it - by setting environment variables. + +Any environment variable starting with `TF_VAR_` is automatically stripped down, so `TF_VAR_test_variable` becomes +`test_variable` to Terraform. + +We maintain an up-to-date list of the variables and what they do in the README.md file under [the +`examples/aws/terraform` section of the Teleport repo](https://github.com/gravitational/teleport/blob/master/examples/aws/terraform/README.md) +but we'll run through an example list here. + +Things you will need to decide on: + +### region + +!!! note "How to set" + `export TF_VAR_region="us-west-2"` + +The AWS region to run in, pick from the supported list in the README + + +### cluster_name + +!!! note "How to set" + `export TF_VAR_cluster_name="teleport.example.com"` + +The cluster name is the Teleport cluster name to use. This should be unique, and not contain spaces or other special characters. +This will appear in the web UI for your cluster and cannot be changed after creation without rebuilding your +cluster from scratch, so choose carefully. A good example might be `teleport.domain.com` where `domain.com` +is your company domain. + + +### ami_name + +!!! note "How to set" + `export TF_VAR_ami_name="gravitational-teleport-ami-ent-4.2.3"` + +Gravitational automatically builds and publishes OSS, Enterprise and Enterprise FIPS 140-2 AMIs when we +release a new version of Teleport. The AMIs follow the format: `gravitational-teleport-ami--` +where `` is either `oss` or `ent` (Enterprise) and `version` is the version of Teleport e.g. `4.2.3`. +FIPS 140-2 compatible AMIs (which deploy Teleport in FIPS 140-2 mode by default) have the `-fips` suffix. + +The AWS account ID which publishes these AMIs is `126027368216`. You can list the available AMIs with +the example `awscli` commands below. The output is in JSON format by default. + +!!! tip "List OSS AMIs" + `aws ec2 describe-images --owners 126027368216 --filters 'Name=name,Values=gravitational-teleport-ami-oss*'` + +!!! tip "List Enterprise AMIs" + `aws ec2 describe-images --owners 126027368216 --filters 'Name=name,Values=gravitational-teleport-ami-ent*'` + +!!! tip "List Enterprise FIPS 140-2 AMIs" + `aws ec2 describe-images --owners 126027368216 --filters 'Name=name,Values=gravitational-teleport-ami-ent*-fips'` + + +### key_name + +!!! note "How to set" + `export TF_VAR_key_name="exampleuser"` + +The AWS keypair name to use when deploying EC2 instances. This must exist in the same region as you +specify in the `region` variable, and you will need a copy of this keypair to connect to the deployed EC2 instances. + + +### license_path + +!!! note "How to set" + `export TF_VAR_license_path="/home/user/teleport-license.pem"` + +The full local path to your Teleport license file, which customers can download from [the Gravitational dashboard](https://dashboard.gravitational.com/) +This license will be copied into AWS SSM and automatically added to Teleport auth nodes in order to enable +Teleport Enterprise/Pro functionality. + + +### route53_zone + +!!! note "How to set" + `export TF_VAR_route53_zone="example.com"` + +Our Terraform setup requires you to have your domain provisioned in AWS Route 53 - it will automatically add +DNS records for `route53_domain` as set up below. You can list these with `aws route53 list-hosted-zones` - +you should should use the appropriate "Name" field here. + + +### route53_domain + +!!! note "How to set" + `export TF_VAR_route53_domain="teleport.example.com"` + +A subdomain to set up as a CNAME to the Teleport load balancer for web access. We generally advise that this should be +the same as the `cluster_name` picked above. + + +### s3_bucket_name + +!!! note "How to set" + `export TF_VAR_s3_bucket_name="teleport.example.com"` + +The Terraform example also provisions an S3 bucket to hold certificates provisioned by LetsEncrypt and distribute these +to EC2 instances. This can be any S3-compatible name, and will be generated in the same region as set above. + + +### email + +!!! note "How to set" + `export TF_VAR_email="support@example.com"` + +LetsEncrypt requires an email address for every certificate registered which can be used to send notifications and +useful information. We recommend a generic ops/support email address which the team deploying Teleport has access to. + + +### grafana_pass + +!!! note "How to set" + `export TF_VAR_grafana_pass="CHANGE_THIS_VALUE"` + +We deploy Grafana along with every Terraform deployment and automatically make stats on cluster usage available. +This variable sets up the password for the Grafana `admin` user. The Grafana web UI is served on the same subdomain +as specified above in `route53_domain` on port 8443. + +With the variables set in this example, it would be available on https://cluster.example.com:8443 + +If you do not change this from the default (`CHANGE_THIS_VALUE`), then it will be set to a random value and written to AWS +SSM as `grafana_pass` - you will need to query this value from SSM. + + +### use_acm + +!!! note "How to set" + `export TF_VAR_use_acm="false"` + +If set to the string `"false"`, Terraform will use [LetsEncrypt](https://letsencrypt.org/) to provision the public-facing +web UI certificate for the Teleport cluster (`route53_subdomain` - so https://teleport.example.com in this example). This uses an [AWS network load balancer](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/introduction.html) to load-balance connections to the Teleport cluster's web UI, and its SSL termination is handled by Teleport itself. + +If set to the string `"true"`, Terraform will use [AWS ACM](https://aws.amazon.com/certificate-manager/) to +provision the public-facing web UI certificate for the cluster. This uses an [AWS application load balancer](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/introduction.html) to load-balance connections to the Teleport cluster's web UI, and its SSL termination is handled by the load balancer. + +If you wish to use a pre-existing ACM certificate rather than having Terraform generate one for you, you can import it +with this command: `terraform import aws_acm_certificate.cert ` + +## Reference deployment defaults + +### Instances + +Our reference deployment will provision the following instances for your cluster by default: + +- 2 x m4.large Teleport auth instances in an ASG, behind an internal network load balancer, configured using DynamoDB +for shared storage ([the desired size of the ASG is configured here](https://github.com/gravitational/teleport/blob/master/examples/aws/terraform/auth_asg.tf#L11)) +- 2 x m4.large Teleport proxy instances in an ASG, behind a public-facing load balancer (NLB for LetsEncrypt, ALB for ACM) ([the desired size of the ASG is configured here](https://github.com/gravitational/teleport/blob/master/examples/aws/terraform/proxy_asg.tf#L12)) +- 1 x m4.large Teleport node instance in an ASG ([the desired size of the ASG is configured here](https://github.com/gravitational/teleport/blob/master/examples/aws/terraform/node_asg.tf#L10)) +- 1 x m4.large monitoring server in an ASG which hosts the Grafana instance and receives monitoring data from each part of the cluster ([the desired size of the ASG is configured here](https://github.com/gravitational/teleport/blob/master/examples/aws/terraform/monitor_asg.tf#L12)) +- 1 x t2.medium bastion server which is the only permitted source for inbound SSH traffic to the instances + +[The instance types used for each ASG can be configured here](https://github.com/gravitational/teleport/blob/master/examples/aws/terraform/vars.tf#L23-L44) + + +### Cluster state database storage + +The reference Terraform deployment sets Teleport up to store its cluster state database in DynamoDB. The name of the +table for cluster state will be the same as the cluster name configured in the [`cluster_name`](#cluster_name) variable above. + +In our example, the DynamoDB table would be called `teleport.example.com`. + +More information about how Teleport works with DynamoDB can be found in our [DynamoDB Admin Guide](https://gravitational.com/teleport/docs/admin-guide/#using-dynamodb) + + +### Audit event storage + +The reference Terraform deployment sets Teleport up to store cluster audit logs in DynamoDB. The name of the table for +audit event storage will be the same as the cluster name configured in the [`cluster_name`](#cluster_name) variable above +with `-events` appended to the end. + +In our example, the DynamoDB table would be called `teleport.example.com-events`. + +More information about how Teleport works with DynamoDB can be found in our [DynamoDB Admin Guide](https://gravitational.com/teleport/docs/admin-guide/#using-dynamodb) + + +### Recorded session storage + +The reference Terraform deployment sets Teleport up to store recorded session logs in the same S3 bucket configured in +the [`s3_bucket_name`](#s3_bucket_name) variable, under the `records` directory. In our example this would be `s3://teleport.example.com/records` + +!!! tip "Tip" + + S3 provides [Amazon S3 Object Lock](https://docs.aws.amazon.com/AmazonS3/latest/dev/object-lock.html), + which is useful for customers deploying Teleport in regulated environments. + +### Cluster domain + +The reference Terraform deployment sets the Teleport cluster up to be available on a domain defined in Route53, referenced +by the `route53_domain`[#route53_domain] variable. In our example this would be `teleport.example.com` + +Teleport's web interface will be available on port 443 - https://teleport.example.com + +Teleport's proxy SSH interface will be available on port 3023 (this is the default port used when connecting with the +`tsh` client) + + +## Technical details + +### Ports exposed by the Teleport deployment + +443 - HTTPS access to the Teleport web UI +3023 - Used for SSH access to the Teleport proxy server +3024 - Used for inbound tunnel connections from Teleport [trusted clusters](trustedclusters.md) +3025 - Used for nodes to connect to the Teleport cluster. This is exposed via an internal load balancer inside the VPC +where Teleport is deployed + + +## Accessing the cluster after Terraform setup + +Once the Terraform setup is finished, your Teleport cluster's web UI should be available on https:// + +### Adding an admin user to the Teleport cluster + +To add users to the Teleport cluster, you will need to connect to a Teleport auth server via SSH and run the `tctl` command. + +1. Use the AWS cli to get the IP of the bastion server: + +```bash +$ export BASTION_IP=$(aws ec2 describe-instances --filters "Name=tag:TeleportCluster,Values=${TF_VAR_cluster_name},Name=tag:TeleportRole,Values=bastion" --query "Reservations[*].Instances[*].PublicIpAddress" --output text) +$ echo ${BASTION_IP} +1.2.3.4 +``` + +2. Use the AWS cli to get the IP of an auth server: + +```bash +$ export AUTH_IP=$(aws ec2 describe-instances --filters "Name=tag:TeleportCluster,Values=${TF_VAR_cluster_name},Name=tag:TeleportRole,Values=auth" --query "Reservations[0].Instances[*].PrivateIpAddress" --output text) +$ echo ${AUTH_IP} +172.31.0.196 +``` + +3. Use both these values to SSH into the auth server (make sure that the AWS keypair that you specified in the +`key_name`[#key_name] variable is available in the current directory): + +```bash +$ ssh -i ${TF_VAR_key_name}.pem -J ec2-user@${BASTION_IP} ec2-user@${AUTH_IP} +The authenticity of host '1.2.3.4 (1.2.3.4)' can't be established. +ECDSA key fingerprint is SHA256:vFPnCFliRsRQ1Dk+muIv2B1Owm96hXiihlOUsj5H3bg. +Are you sure you want to continue connecting (yes/no/[fingerprint])? yes +Warning: Permanently added '1.2.3.4' (ECDSA) to the list of known hosts. +The authenticity of host '172.31.0.196 ()' can't be established. +ECDSA key fingerprint is SHA256:vFPnCFliRsRQ1Dk+muIv2B1Owm96hXiihlOUsj5H3bg. +Are you sure you want to continue connecting (yes/no/[fingerprint])? yes +Warning: Permanently added '172.31.0.196' (ECDSA) to the list of known hosts. +Last login: Tue Mar 3 18:57:12 2020 from 1.2.3.5 + + __| __|_ ) + _| ( / Amazon Linux 2 AMI + ___|\___|___| + +https://aws.amazon.com/amazon-linux-2/ +1 package(s) needed for security, out of 6 available +Run "sudo yum update" to apply all updates. +[ec2-user@ip-172-31-0-196 ~]$ +``` + +4. Use the `tctl` command to create an admin user for Teleport: + +```bash +[ec2-user@ip-172-31-0-196 ~]$ sudo tctl users add teleport-admin --roles=admin +Signup token has been created and is valid for 1 hours. Share this URL with the user: +https://teleport.example.com:443/web/newuser/6489ae886babf4232826076279bcb2fb + +NOTE: Make sure teleport.example.com:443 points at a Teleport proxy which users can access. +When the user 'teleport-admin' activates their account, they will be assigned roles [admin] +``` + +5. Click the link to launch the Teleport web UI and finish setting up your user. You will need to scan the QR +code with an TOTP-compatible app like Google Authenticator or Authy. You will also set a password for the +`teleportadmin` user on this page. + +Once this user is successfully configured, you should be logged into the Teleport web UI. + + +### Teleport service names + + +TODO(gus) + + +### Adding EC2 instances to your Teleport cluster +Customers run many workloads within EC2 and depending on how you work there are many +ways to integrate Teleport onto your servers. We recommend looking at our [Admin manual](https://gravitational.com/teleport/docs/admin-guide/#installing). + +In short, to add new nodes / EC2 servers that you can "SSH into" you'll need to +- [Install the Teleport Binary on the Server](admin-guide.md#installing) +- [Run Teleport - we recommend using systemd](admin-guide.md#systemd-unit-file) +- [Set the correct settings in /etc/teleport.yaml](admin-guide.md#configuration-file) +- [Add EC2 nodes to the Teleport cluster](admin-guide.md#adding-nodes-to-the-cluster) + +The hostname to use with `auth_servers` in your Teleport config file can be found with this command: \ No newline at end of file From b57003b956f6777aafa9caf80c826ac0858988d5 Mon Sep 17 00:00:00 2001 From: Gus Luxton Date: Thu, 5 Mar 2020 18:12:27 -0400 Subject: [PATCH 02/13] More detail --- docs/4.2/aws_terraform_guide.md | 508 ++++++++++++++++++++++++++++---- 1 file changed, 448 insertions(+), 60 deletions(-) diff --git a/docs/4.2/aws_terraform_guide.md b/docs/4.2/aws_terraform_guide.md index 990dd7a2b4bdc..50988f8b8ca57 100644 --- a/docs/4.2/aws_terraform_guide.md +++ b/docs/4.2/aws_terraform_guide.md @@ -1,10 +1,14 @@ # Running Teleport Enterprise in a high-availiability configuration on AWS -This guide is designed to accompany our reference Terraform code (https://github.com/gravitational/teleport/tree/master/examples/aws/terraform) and describe how to use and administrate the resulting Teleport deployment. +This guide is designed to accompany our reference Terraform code (https://github.com/gravitational/teleport/tree/master/examples/aws/terraform) +and describe how to use and administrate the resulting Teleport deployment. + +[TOC] ## Prerequisites -Our code requires Terraform 0.12+. You can [download Terraform here](https://www.terraform.io/downloads.html). We will assume that you have `terraform` installed and available on your path. +Our code requires Terraform 0.12+. You can [download Terraform here](https://www.terraform.io/downloads.html). We will assume that you have +`terraform` installed and available on your path. ```bash $ terraform version @@ -34,11 +38,10 @@ $ cat ~/.aws/config region = us-east-1 ``` -As a result, you should be able to run a command like `aws ec2 describe-images --owners 126027368216 --filters 'Name=name,Values=gravitational-teleport-ami-ent*` -to list available EC2 images matching a pattern. If you get an "access denied" or similar message, you will need -to grant additional permissions to the AWS IAM user that your `aws_access_key_id` and `aws_secret_access_key` refers to. +As a result, you should be able to run a command like `aws ec2 describe-instances` to list running EC2 instances. +If you get an "access denied", "403 Forbidden" or similar message, you will need to grant additional permissions to the +AWS IAM user that your `aws_access_key_id` and `aws_secret_access_key` refers to. -# TODO(gus): detailed explanation of IAM permissions needed ## AWS Services required to run Teleport in HA @@ -97,11 +100,17 @@ reference code. ## Set up variables -Terraform modules use variables to pass in input. You can do this on the command line to `terraform plan`, -by editing the `vars.tf` file or the way we do it - by setting environment variables. +Terraform modules use variables to pass in input. You can do this in a few ways: + +- on the command line to `terraform apply` +- by editing the `vars.tf` file +- by setting environment variables + +For this guide, we are going to make extensive use of environment variables. This is because it makes it easier for us +to reference values from our configuration when running Teleport commands. -Any environment variable starting with `TF_VAR_` is automatically stripped down, so `TF_VAR_test_variable` becomes -`test_variable` to Terraform. +Any set environment variable starting with `TF_VAR_` is automatically processed and stripped down by Terraform, so +`TF_VAR_test_variable` becomes `test_variable`. We maintain an up-to-date list of the variables and what they do in the README.md file under [the `examples/aws/terraform` section of the Teleport repo](https://github.com/gravitational/teleport/blob/master/examples/aws/terraform/README.md) @@ -114,18 +123,17 @@ Things you will need to decide on: !!! note "How to set" `export TF_VAR_region="us-west-2"` -The AWS region to run in, pick from the supported list in the README +The AWS region to run in. You should pick from the supported list as detailed in the [README](https://github.com/gravitational/teleport/blob/master/examples/aws/terraform/README.md). ### cluster_name !!! note "How to set" - `export TF_VAR_cluster_name="teleport.example.com"` + `export TF_VAR_cluster_name="example-cluster"` -The cluster name is the Teleport cluster name to use. This should be unique, and not contain spaces or other special characters. -This will appear in the web UI for your cluster and cannot be changed after creation without rebuilding your -cluster from scratch, so choose carefully. A good example might be `teleport.domain.com` where `domain.com` -is your company domain. +The cluster name is the internal Teleport cluster name to use. This should be unique, and not contain spaces, dots (.) +or other special characters. This will appear in the web UI for your cluster and cannot be changed after creation +without rebuilding your cluster from scratch, so choose carefully. ### ami_name @@ -134,8 +142,9 @@ is your company domain. `export TF_VAR_ami_name="gravitational-teleport-ami-ent-4.2.3"` Gravitational automatically builds and publishes OSS, Enterprise and Enterprise FIPS 140-2 AMIs when we -release a new version of Teleport. The AMIs follow the format: `gravitational-teleport-ami--` +release a new version of Teleport. The AMI names follow the format: `gravitational-teleport-ami--` where `` is either `oss` or `ent` (Enterprise) and `version` is the version of Teleport e.g. `4.2.3`. + FIPS 140-2 compatible AMIs (which deploy Teleport in FIPS 140-2 mode by default) have the `-fips` suffix. The AWS account ID which publishes these AMIs is `126027368216`. You can list the available AMIs with @@ -165,9 +174,11 @@ specify in the `region` variable, and you will need a copy of this keypair to co !!! note "How to set" `export TF_VAR_license_path="/home/user/teleport-license.pem"` -The full local path to your Teleport license file, which customers can download from [the Gravitational dashboard](https://dashboard.gravitational.com/) -This license will be copied into AWS SSM and automatically added to Teleport auth nodes in order to enable -Teleport Enterprise/Pro functionality. +The full local path to your Teleport license file, which customers can download from +[the Gravitational dashboard](https://dashboard.gravitational.com/) This license will be copied into AWS SSM and +automatically added to Teleport auth nodes in order to enable Teleport Enterprise/Pro functionality. + +OSS users can provide any valid file here - it isn't used by the auth server in a Teleport OSS install. ### route53_zone @@ -176,8 +187,18 @@ Teleport Enterprise/Pro functionality. `export TF_VAR_route53_zone="example.com"` Our Terraform setup requires you to have your domain provisioned in AWS Route 53 - it will automatically add -DNS records for `route53_domain` as set up below. You can list these with `aws route53 list-hosted-zones` - -you should should use the appropriate "Name" field here. +DNS records for [`route53_domain`](#route53_domain) as set up below. You can list these with this command: + +```bash +$ aws route53 list-hosted-zones --query "HostedZones[*].Name" --output text +[ + "example.com.", + "testing.net.", + "subdomain.wow.org." +] +``` + +You should use the appropriate domain without the trailing dot. ### route53_domain @@ -185,14 +206,16 @@ you should should use the appropriate "Name" field here. !!! note "How to set" `export TF_VAR_route53_domain="teleport.example.com"` -A subdomain to set up as a CNAME to the Teleport load balancer for web access. We generally advise that this should be -the same as the `cluster_name` picked above. +A subdomain to set up as a CNAME to the Teleport load balancer for web access. This will be the public-facing domain +that people use to connect to your Teleport cluster, so choose wisely. + +This must be a subdomain of the domain you chose for [`route53_zone`](#route53_zone) above. ### s3_bucket_name !!! note "How to set" - `export TF_VAR_s3_bucket_name="teleport.example.com"` + `export TF_VAR_s3_bucket_name="example-cluster"` The Terraform example also provisions an S3 bucket to hold certificates provisioned by LetsEncrypt and distribute these to EC2 instances. This can be any S3-compatible name, and will be generated in the same region as set above. @@ -214,12 +237,12 @@ useful information. We recommend a generic ops/support email address which the t We deploy Grafana along with every Terraform deployment and automatically make stats on cluster usage available. This variable sets up the password for the Grafana `admin` user. The Grafana web UI is served on the same subdomain -as specified above in `route53_domain` on port 8443. +as specified above in [`route53_domain`](#route53_domain) on port 8443. -With the variables set in this example, it would be available on https://cluster.example.com:8443 +With the variables set in this example, it would be available on https://teleport.example.com:8443 -If you do not change this from the default (`CHANGE_THIS_VALUE`), then it will be set to a random value and written to AWS -SSM as `grafana_pass` - you will need to query this value from SSM. +If you do not change this from the default (`CHANGE_THIS_VALUE`), then it will be set to a random value for security. +As such, we recommend setting this to a known value. ### use_acm @@ -228,7 +251,9 @@ SSM as `grafana_pass` - you will need to query this value from SSM. `export TF_VAR_use_acm="false"` If set to the string `"false"`, Terraform will use [LetsEncrypt](https://letsencrypt.org/) to provision the public-facing -web UI certificate for the Teleport cluster (`route53_subdomain` - so https://teleport.example.com in this example). This uses an [AWS network load balancer](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/introduction.html) to load-balance connections to the Teleport cluster's web UI, and its SSL termination is handled by Teleport itself. +web UI certificate for the Teleport cluster ([`route53_subdomain`](#route53_subdomain) - so https://teleport.example.com in this example). +This uses an [AWS network load balancer](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/introduction.html) +to load-balance connections to the Teleport cluster's web UI, and its SSL termination is handled by Teleport itself. If set to the string `"true"`, Terraform will use [AWS ACM](https://aws.amazon.com/certificate-manager/) to provision the public-facing web UI certificate for the cluster. This uses an [AWS application load balancer](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/introduction.html) to load-balance connections to the Teleport cluster's web UI, and its SSL termination is handled by the load balancer. @@ -242,22 +267,25 @@ with this command: `terraform import aws_acm_certificate.cert ` Our reference deployment will provision the following instances for your cluster by default: -- 2 x m4.large Teleport auth instances in an ASG, behind an internal network load balancer, configured using DynamoDB -for shared storage ([the desired size of the ASG is configured here](https://github.com/gravitational/teleport/blob/master/examples/aws/terraform/auth_asg.tf#L11)) -- 2 x m4.large Teleport proxy instances in an ASG, behind a public-facing load balancer (NLB for LetsEncrypt, ALB for ACM) ([the desired size of the ASG is configured here](https://github.com/gravitational/teleport/blob/master/examples/aws/terraform/proxy_asg.tf#L12)) -- 1 x m4.large Teleport node instance in an ASG ([the desired size of the ASG is configured here](https://github.com/gravitational/teleport/blob/master/examples/aws/terraform/node_asg.tf#L10)) -- 1 x m4.large monitoring server in an ASG which hosts the Grafana instance and receives monitoring data from each part of the cluster ([the desired size of the ASG is configured here](https://github.com/gravitational/teleport/blob/master/examples/aws/terraform/monitor_asg.tf#L12)) -- 1 x t2.medium bastion server which is the only permitted source for inbound SSH traffic to the instances +- 2 x m4.large Teleport auth instances in an ASG, behind an internal network load balancer, configured using DynamoDB for +shared storage. [The desired size of the ASG is configured here](https://github.com/gravitational/teleport/blob/master/examples/aws/terraform/auth_asg.tf#L11) +- 2 x m4.large Teleport proxy instances in an ASG, behind a public-facing load balancer - NLB for LetsEncrypt, ALB for ACM. [The desired size of the ASG is configured here](https://github.com/gravitational/teleport/blob/master/examples/aws/terraform/proxy_asg.tf#L12) +- 1 x m4.large Teleport node instance in an ASG. [The desired size of the ASG is configured here](https://github.com/gravitational/teleport/blob/master/examples/aws/terraform/node_asg.tf#L10) +- 1 x m4.large monitoring server in an ASG which hosts the Grafana instance and receives monitoring data from each part of the cluster. [The desired size of the ASG is configured here](https://github.com/gravitational/teleport/blob/master/examples/aws/terraform/monitor_asg.tf#L12) +- 1 x t2.medium bastion server which is the only permitted source for inbound SSH traffic to the instances. [The instance types used for each ASG can be configured here](https://github.com/gravitational/teleport/blob/master/examples/aws/terraform/vars.tf#L23-L44) +If you don't wish to set up a node or the monitoring services, you can set the `desired_size` and `min_size` for the ASG +to `0` and Terraform won't provision it. + ### Cluster state database storage The reference Terraform deployment sets Teleport up to store its cluster state database in DynamoDB. The name of the table for cluster state will be the same as the cluster name configured in the [`cluster_name`](#cluster_name) variable above. -In our example, the DynamoDB table would be called `teleport.example.com`. +In our example, the DynamoDB table would be called `example-cluster`. More information about how Teleport works with DynamoDB can be found in our [DynamoDB Admin Guide](https://gravitational.com/teleport/docs/admin-guide/#using-dynamodb) @@ -268,7 +296,7 @@ The reference Terraform deployment sets Teleport up to store cluster audit logs audit event storage will be the same as the cluster name configured in the [`cluster_name`](#cluster_name) variable above with `-events` appended to the end. -In our example, the DynamoDB table would be called `teleport.example.com-events`. +In our example, the DynamoDB table would be called `example-cluster-events`. More information about how Teleport works with DynamoDB can be found in our [DynamoDB Admin Guide](https://gravitational.com/teleport/docs/admin-guide/#using-dynamodb) @@ -276,7 +304,7 @@ More information about how Teleport works with DynamoDB can be found in our [Dyn ### Recorded session storage The reference Terraform deployment sets Teleport up to store recorded session logs in the same S3 bucket configured in -the [`s3_bucket_name`](#s3_bucket_name) variable, under the `records` directory. In our example this would be `s3://teleport.example.com/records` +the [`s3_bucket_name`](#s3_bucket_name) variable, under the `records` directory. In our example this would be `s3://example-cluster/records` !!! tip "Tip" @@ -293,27 +321,77 @@ Teleport's web interface will be available on port 443 - https://teleport.exampl Teleport's proxy SSH interface will be available on port 3023 (this is the default port used when connecting with the `tsh` client) +## Deploying with Terraform + +Once you have set values for and exported all the variables detailed above, you should run `terraform plan` to validate the +configuration. + +```bash +$ terraform plan +Refreshing Terraform state in-memory prior to plan... +The refreshed state will be used to calculate this plan, but will not be +persisted to local or remote state storage. + +data.template_file.monitor_user_data: Refreshing state... +data.aws_kms_alias.ssm: Refreshing state... +data.aws_caller_identity.current: Refreshing state... +data.aws_ami.base: Refreshing state... +data.aws_availability_zones.available: Refreshing state... +data.aws_route53_zone.proxy: Refreshing state... +data.aws_region.current: Refreshing state... -## Technical details +------------------------------------------------------------------------ -### Ports exposed by the Teleport deployment +An execution plan has been generated and is shown below. +Resource actions are indicated with the following symbols: + + create + <= read (data resources) -443 - HTTPS access to the Teleport web UI -3023 - Used for SSH access to the Teleport proxy server -3024 - Used for inbound tunnel connections from Teleport [trusted clusters](trustedclusters.md) -3025 - Used for nodes to connect to the Teleport cluster. This is exposed via an internal load balancer inside the VPC -where Teleport is deployed +Terraform will perform the following actions: + + +Plan: 121 to add, 0 to change, 0 to destroy. + +------------------------------------------------------------------------ + +Note: You didn't specify an "-out" parameter to save this plan, so Terraform +can't guarantee that exactly these actions will be performed if +"terraform apply" is subsequently run. +``` + +This looks good (no errors produced by Terraform) so we can run `terraform apply`: + +```bash +$ terraform apply + + +Plan: 121 to add, 0 to change, 0 to destroy. + +Do you want to perform these actions? + Terraform will perform the actions described above. + Only 'yes' will be accepted to approve. + + Enter a value: +``` + +Entering `yes` here will start the Terraform deployment. It takes around 8-10 minutes to deploy in full. + +#### Destroy/shutdown + +If you need to tear down a running deployment for any reason, you can run `terraform destroy`. ## Accessing the cluster after Terraform setup -Once the Terraform setup is finished, your Teleport cluster's web UI should be available on https:// +Once the Terraform setup is finished, your Teleport cluster's web UI should be available on +https://[route53_domain](#route53_domain) - this is https://teleport.example.com in our example. + ### Adding an admin user to the Teleport cluster To add users to the Teleport cluster, you will need to connect to a Teleport auth server via SSH and run the `tctl` command. -1. Use the AWS cli to get the IP of the bastion server: +1 - Use the AWS cli to get the IP of the bastion server: ```bash $ export BASTION_IP=$(aws ec2 describe-instances --filters "Name=tag:TeleportCluster,Values=${TF_VAR_cluster_name},Name=tag:TeleportRole,Values=bastion" --query "Reservations[*].Instances[*].PublicIpAddress" --output text) @@ -321,7 +399,7 @@ $ echo ${BASTION_IP} 1.2.3.4 ``` -2. Use the AWS cli to get the IP of an auth server: +2 - Use the AWS cli to get the IP of an auth server: ```bash $ export AUTH_IP=$(aws ec2 describe-instances --filters "Name=tag:TeleportCluster,Values=${TF_VAR_cluster_name},Name=tag:TeleportRole,Values=auth" --query "Reservations[0].Instances[*].PrivateIpAddress" --output text) @@ -329,8 +407,8 @@ $ echo ${AUTH_IP} 172.31.0.196 ``` -3. Use both these values to SSH into the auth server (make sure that the AWS keypair that you specified in the -`key_name`[#key_name] variable is available in the current directory): +3 - Use both these values to SSH into the auth server (make sure that the AWS keypair that you specified in the +[`key_name`](#key_name) variable is available in the current directory): ```bash $ ssh -i ${TF_VAR_key_name}.pem -J ec2-user@${BASTION_IP} ec2-user@${AUTH_IP} @@ -354,7 +432,7 @@ Run "sudo yum update" to apply all updates. [ec2-user@ip-172-31-0-196 ~]$ ``` -4. Use the `tctl` command to create an admin user for Teleport: +4 - Use the `tctl` command to create an admin user for Teleport: ```bash [ec2-user@ip-172-31-0-196 ~]$ sudo tctl users add teleport-admin --roles=admin @@ -365,27 +443,337 @@ NOTE: Make sure teleport.example.com:443 points at a Teleport proxy which users When the user 'teleport-admin' activates their account, they will be assigned roles [admin] ``` -5. Click the link to launch the Teleport web UI and finish setting up your user. You will need to scan the QR +5 - Click the link to launch the Teleport web UI and finish setting up your user. You will need to scan the QR code with an TOTP-compatible app like Google Authenticator or Authy. You will also set a password for the `teleportadmin` user on this page. Once this user is successfully configured, you should be logged into the Teleport web UI. -### Teleport service names +## Restarting/checking Teleport services + +### LetsEncrypt + +!!! note + You are using LetsEncrypt if your `use_acm` variable is set to `"false"`. + +#### Auth service + +```bash +$ systemctl status teleport-auth.service +● teleport-auth.service - Teleport Auth Service + Loaded: loaded (/etc/systemd/system/teleport-auth.service; enabled; vendor preset: disabled) + Active: active (running) since Thu 2020-03-05 16:45:18 UTC; 4h 14min ago + Main PID: 3766 (teleport) + CGroup: /system.slice/teleport-auth.service + └─3766 /usr/bin/teleport start --config=/etc/teleport.yaml --diag-addr=127.0.0.1:3434 --pid-file=/run/teleport/teleport.pid + +Mar 05 17:54:58 ip-172-31-0-196.ec2.internal /usr/bin/teleport[3766]: INFO [CA] Generating TLS certificate {0x3767920 0xc0012802f0 CN=teleport-admin,O=admin,POSTALCODE={\"kubernetes_groups\":null\,\"logins\":null},STREET=,L=root 2020-03-06 05:54:58.846233666 +0000 UTC []}. common_name:teleport-admin dns_name... +Mar 05 18:04:39 ip-172-31-0-196.ec2.internal /usr/bin/teleport[3766]: INFO [CA] Generating TLS certificate {0x3767920 0xc00155d200 CN=teleport-admin,O=admin,POSTALCODE={\"kubernetes_groups\":null\,\"logins\":null},STREET=,L=root 2020-03-06 06:04:39.844777551 +0000 UTC []}. common_name:teleport-admin dns_name... +``` + +You can get detailed logs for the Teleport auth servers using the `journalctl` command: + +```bash +$ journalctl -u teleport-auth.service +``` + +Remember that there is more than one auth server in an HA deployment. You should use this command to get the IP addresses +of each auth server that you'll need to connect to: + +```bash +$ aws ec2 describe-instances --filters "Name=tag:TeleportCluster,Values=${TF_VAR_cluster_name},Name=tag:TeleportRole,Values=auth" --query "Reservations[*].Instances[*].PrivateIpAddress" --output text +172.31.0.196 +172.31.1.78 +``` + +#### Proxy service + +```bash +$ systemctl status teleport-proxy.service +● teleport-proxy.service - Teleport Proxy Service + Loaded: loaded (/etc/systemd/system/teleport-proxy.service; enabled; vendor preset: disabled) + Active: active (running) since Thu 2020-03-05 17:14:37 UTC; 3h 47min ago + Process: 4502 ExecStartPre=/usr/bin/teleport-ssm-get-token (code=exited, status=0/SUCCESS) + Main PID: 4514 (teleport) + CGroup: /system.slice/teleport-proxy.service + └─4514 /usr/bin/teleport start --config=/etc/teleport.yaml --diag-addr=127.0.0.1:3434 --pid-file=/run/teleport/teleport.pid + +Mar 05 20:58:25 ip-172-31-2-109.ec2.internal /usr/bin/teleport[4514]: ERRO read tcp 172.31.2.109:3024->172.31.2.143:1577: read: connection reset by peer +Mar 05 20:58:50 ip-172-31-2-109.ec2.internal /usr/bin/teleport[4514]: ERRO read tcp 172.31.2.109:3023->172.31.2.143:38011: read: connection reset by peer +``` + +You can get detailed logs for the Teleport proxy service using the `journalctl` command: + +```bash +$ journalctl -u teleport-proxy.service +``` + +Remember that there is more than one proxy instance in an HA deployment. You should use this command to get the IP addresses +of each auth instance that you'll need to connect to: + +```bash +$ aws ec2 describe-instances --filters "Name=tag:TeleportCluster,Values=${TF_VAR_cluster_name},Name=tag:TeleportRole,Values=proxy" --query "Reservations[*].Instances[*].PrivateIpAddress" --output text +172.31.2.109 +172.31.3.215 +``` + +#### Node service + +```bash +$ systemctl status teleport-node.service +● teleport-node.service - Teleport SSH Node Service + Loaded: loaded (/etc/systemd/system/teleport-node.service; enabled; vendor preset: disabled) + Active: active (running) since Thu 2020-03-05 17:18:25 UTC; 3h 44min ago + Process: 4444 ExecStartPre=/usr/bin/teleport-ssm-get-token (code=exited, status=0/SUCCESS) + Main PID: 4456 (teleport) + CGroup: /system.slice/teleport-node.service + └─4456 /usr/bin/teleport start --config=/etc/teleport.yaml --diag-addr=127.0.0.1:3434 --pid-file=/run/teleport/teleport.pid + +Mar 05 17:18:25 ip-172-31-11-69.ec2.internal /usr/bin/teleport[4456]: INFO [AUDIT:1] Creating directory /var/lib/teleport/log/upload/sessions. ser...o:1630 +Mar 05 17:18:25 ip-172-31-11-69.ec2.internal /usr/bin/teleport[4456]: INFO [AUDIT:1] Setting directory /var/lib/teleport/log/upload/sessions owner...o:1639 +``` + +You can get detailed logs for the Teleport node service using the `journalctl` command: + +```bash +$ journalctl -u teleport-node.service +``` + +### ACM + +!!! note + You are using ACM if your `use_acm` variable is set to `"true"`. + +When using ACM, the service name for the proxy is different (`teleport-proxy-acm.service` vs `teleport-proxy.service`). + +#### Auth service + +```bash +$ systemctl status teleport-auth.service +● teleport-auth.service - Teleport Auth Service + Loaded: loaded (/etc/systemd/system/teleport-auth.service; enabled; vendor preset: disabled) + Active: active (running) since Thu 2020-03-05 16:45:18 UTC; 4h 14min ago + Main PID: 3766 (teleport) + CGroup: /system.slice/teleport-auth.service + └─3766 /usr/bin/teleport start --config=/etc/teleport.yaml --diag-addr=127.0.0.1:3434 --pid-file=/run/teleport/teleport.pid + +Mar 05 17:54:58 ip-172-31-0-196.ec2.internal /usr/bin/teleport[3766]: INFO [CA] Generating TLS certificate {0x3767920 0xc0012802f0 CN=teleport-admin,O=admin,POSTALCODE={\"kubernetes_groups\":null\,\"logins\":null},STREET=,L=root 2020-03-06 05:54:58.846233666 +0000 UTC []}. common_name:teleport-admin dns_name... +Mar 05 18:04:39 ip-172-31-0-196.ec2.internal /usr/bin/teleport[3766]: INFO [CA] Generating TLS certificate {0x3767920 0xc00155d200 CN=teleport-admin,O=admin,POSTALCODE={\"kubernetes_groups\":null\,\"logins\":null},STREET=,L=root 2020-03-06 06:04:39.844777551 +0000 UTC []}. common_name:teleport-admin dns_name... +``` + +You can get detailed logs for the Teleport auth server using the `journalctl` command: + +```bash +$ journalctl -u teleport-auth.service +``` + +Remember that there is more than one auth instance in an HA deployment. You should use this command to get the IP addresses +of each auth instance that you'll need to connect to: + +```bash +$ aws ec2 describe-instances --filters "Name=tag:TeleportCluster,Values=${TF_VAR_cluster_name},Name=tag:TeleportRole,Values=auth" --query "Reservations[*].Instances[*].PrivateIpAddress" --output text +172.31.0.196 +172.31.1.78 +``` + +#### Proxy service (ACM) + +```bash +$ systemctl status teleport-proxy-acm.service +● teleport-proxy-acm.service - Teleport Proxy Service (ACM) + Loaded: loaded (/etc/systemd/system/teleport-proxy-acm.service; enabled; vendor preset: disabled) + Active: active (running) since Thu 2020-03-05 17:14:37 UTC; 3h 47min ago + Process: 4502 ExecStartPre=/usr/bin/teleport-ssm-get-token (code=exited, status=0/SUCCESS) + Main PID: 4514 (teleport) + CGroup: /system.slice/teleport-proxy-acm.service + └─4514 /usr/bin/teleport start --config=/etc/teleport.yaml --diag-addr=127.0.0.1:3434 --pid-file=/run/teleport/teleport.pid + +Mar 05 20:58:25 ip-172-31-2-109.ec2.internal /usr/bin/teleport[4514]: ERRO read tcp 172.31.2.109:3024->172.31.2.143:1577: read: connection reset by peer +Mar 05 20:58:50 ip-172-31-2-109.ec2.internal /usr/bin/teleport[4514]: ERRO read tcp 172.31.2.109:3023->172.31.2.143:38011: read: connection reset by peer +``` + +You can get detailed logs for the Teleport proxy service using the `journalctl` command: + +```bash +$ journalctl -u teleport-proxy-acm.service +``` + +Remember that there is more than one proxy instance in an HA deployment. You should use this command to get the IP addresses +of each auth instance that you'll need to connect to: + +```bash +$ aws ec2 describe-instances --filters "Name=tag:TeleportCluster,Values=${TF_VAR_cluster_name},Name=tag:TeleportRole,Values=proxy" --query "Reservations[*].Instances[*].PrivateIpAddress" --output text +172.31.2.109 +172.31.3.215 +``` +#### Node service -TODO(gus) +```bash +$ systemctl status teleport-node.service +● teleport-node.service - Teleport SSH Node Service + Loaded: loaded (/etc/systemd/system/teleport-node.service; enabled; vendor preset: disabled) + Active: active (running) since Thu 2020-03-05 17:18:25 UTC; 3h 44min ago + Process: 4444 ExecStartPre=/usr/bin/teleport-ssm-get-token (code=exited, status=0/SUCCESS) + Main PID: 4456 (teleport) + CGroup: /system.slice/teleport-node.service + └─4456 /usr/bin/teleport start --config=/etc/teleport.yaml --diag-addr=127.0.0.1:3434 --pid-file=/run/teleport/teleport.pid + +Mar 05 17:18:25 ip-172-31-11-69.ec2.internal /usr/bin/teleport[4456]: INFO [AUDIT:1] Creating directory /var/lib/teleport/log/upload/sessions. ser...o:1630 +Mar 05 17:18:25 ip-172-31-11-69.ec2.internal /usr/bin/teleport[4456]: INFO [AUDIT:1] Setting directory /var/lib/teleport/log/upload/sessions owner...o:1639 +``` +You can get detailed logs for the Teleport node service using the `journalctl` command: -### Adding EC2 instances to your Teleport cluster +```bash +$ journalctl -u teleport-node.service +``` + +## Adding EC2 instances to your Teleport cluster Customers run many workloads within EC2 and depending on how you work there are many ways to integrate Teleport onto your servers. We recommend looking at our [Admin manual](https://gravitational.com/teleport/docs/admin-guide/#installing). -In short, to add new nodes / EC2 servers that you can "SSH into" you'll need to -- [Install the Teleport Binary on the Server](admin-guide.md#installing) +To add new nodes/EC2 servers that you can "SSH into" you'll need to: + +- [Install the Teleport binary on the Server](admin-guide.md#installing) + - [Run Teleport - we recommend using systemd](admin-guide.md#systemd-unit-file) + - [Set the correct settings in /etc/teleport.yaml](admin-guide.md#configuration-file) -- [Add EC2 nodes to the Teleport cluster](admin-guide.md#adding-nodes-to-the-cluster) -The hostname to use with `auth_servers` in your Teleport config file can be found with this command: \ No newline at end of file +- [Add nodes to the Teleport cluster](admin-guide.md#adding-nodes-to-the-cluster) + + +### Getting the CA pin hash + +You can use this command to get the CA pin hash for your Teleport cluster: + +```bash +$ aws ssm get-parameter --name "/teleport/${TF_VAR_cluster_name}/ca-pin-hash" --query "Parameter.Value" --output text +sha256:d021ef54aaf8633c4e15c5cc59479fb2f19b1bbc5432bb95213ee047000689dd +``` + +You should use this so that nodes can validate the auth server's identity when joining your cluster. + + +### Getting the node join token + +You can use this command to get a join token for your Teleport cluster: + +```bash +$ aws ssm get-parameter --name "/teleport/${TF_VAR_cluster_name}/tokens/node" --query "Parameter.Value" --output text +AQICAHgLq8feq4riNouuw8Wxs5EEPlS2qKIVE5Z/qEo1i6mqfwGX3dW56SdoS6PinTWbZL1RAAAAgzCBgAYJKoZIhvcNAQcGoHMwcQIBADBsBgkqhkiG9w0BBwEwHgYJYIZIAWUDBAEuMBEEDNdu5TxaT8gyJx63eAIBEIA/JEpX2Vte90UmufIzZzvBQcQaKgWr95aN9xZYMEjWbAiNitxkvZgb98FgFn8d9GNwKQgDGfUYDbzsX8EqTtx9 +``` + +You should use this so that nodes can validate the auth server's identity when joining your cluster. + +You can also generate a node join token using `tctl tokens add --type=node` [as detailed here in our admin guide](admin-guide.md#adding-nodes-to-the-cluster). + + +### Joining nodes via the Teleport auth server + +To join Teleport nodes in the same VPC via the auth server, you can find the hostname for the auth load balancer with +this command: + +```bash +$ aws elbv2 describe-load-balancers --names "${TF_VAR_cluster_name}-auth" --query "LoadBalancers[*].DNSName" --output text +example-cluster-auth-c5b0fc2764ee015b.elb.us-east-1.amazonaws.com +``` + +With this method, the nodes should be configured like so: + +```yaml +auth_servers: + - example-cluster-auth-c5b0fc2764ee015b.elb.us-east-1.amazonaws.com:3025 +``` + + +### Joining nodes via Teleport IoT/node tunnelling + +To join Teleport nodes from outside the same VPC, you will either need to investigate VPC peering/gateways (out of scope +for this document) or join your nodes using [Teleport's node tunnelling](admin-guide.md#adding-a-node-located-behind-nat) functionality. + +With this method, you can join the nodes using the public facing proxy address - `teleport.example.com:443` for our +example. + +```yaml +auth_servers: + - teleport.example.com:443 +``` + + +### Trusted clusters + +To add a trusted cluster, you'll need the hostname of the proxy load balancer. You can get it using this command: + +```bash +$ aws elbv2 describe-load-balancers --names "${TF_VAR_cluster_name}-proxy" --query "LoadBalancers[*].DNSName" --output text +example-cluster-proxy-7c97b76593d6bf21.elb.us-east-1.amazonaws.com +``` + +In this example, the `tunnel_addr` and `web_proxy_addr` in the trusted cluster configuration should be set up like this: + +```yaml +spec: + tunnel_addr: example-cluster-proxy-7c97b76593d6bf21.elb.us-east-1.amazonaws.com:3024 + web_proxy_addr: teleport.example.com:443 +``` + +You can generate a token for adding the trusted cluster via the `tctl` command line tool after +connecting to an auth server. Follow the instructions in our [trusted cluster guide](https://gravitational.com/teleport/docs/trustedclusters/#dynamic-join-tokens). + + +## Some quick commands + +Here's a bash script that you can use to quickly connect to instances: + +```bash +#!/bin/bash +if [[ "$1" != "" ]]; then + INSTANCE_TYPE=$1 +else + INSTANCE_TYPE="auth" +fi +if [[ "$2" != "" ]]; then + INSTANCE_ID=$2 +else + INSTANCE_ID="0" +fi + +export BASTION_IP=$(aws ec2 describe-instances --filters "Name=tag:TeleportCluster,Values=${TF_VAR_cluster_name},Name=tag:TeleportRole,Values=bastion" --query "Reservations[*].Instances[*].PublicIpAddress" --output text) +echo "Bastion IP: ${BASTION_IP}" + +if [[ "${INSTANCE_TYPE}" == "auth" ]]; then + export SERVER_IP=$(aws ec2 describe-instances --filters "Name=tag:TeleportCluster,Values=${TF_VAR_cluster_name},Name=tag:TeleportRole,Values=auth" --query "Reservations[${INSTANCE_ID}].Instances[*].PrivateIpAddress" --output text) + echo "Auth ${INSTANCE_ID} IP: ${SERVER_IP}" +elif [[ "${INSTANCE_TYPE}" == "proxy" ]]; then + export SERVER_IP=$(aws ec2 describe-instances --filters "Name=tag:TeleportCluster,Values=${TF_VAR_cluster_name},Name=tag:TeleportRole,Values=proxy" --query "Reservations[${INSTANCE_ID}].Instances[*].PrivateIpAddress" --output text) + echo "Proxy ${INSTANCE_ID} IP: ${SERVER_IP}" +elif [[ "${INSTANCE_TYPE}" == "node" ]]; then + export SERVER_IP=$(aws ec2 describe-instances --filters "Name=tag:TeleportCluster,Values=${TF_VAR_cluster_name},Name=tag:TeleportRole,Values=node" --query "Reservations[*].Instances[*].PrivateIpAddress" --output text) + echo "Node IP: ${SERVER_IP}" +fi + +echo "Keypair name: ${TF_VAR_key_name}" +ssh -i ${TF_VAR_key_name}.pem -J ec2-user@${BASTION_IP} ec2-user@${SERVER_IP} +``` + +Save this as `connect.sh`, run `chmod +x connect.sh` to make it executable, then use it like so: + +```bash +# connect to the first auth server +$ ./connect.sh auth 0 + +# connect to the second auth server +$ ./connect.sh auth 1 + +# connect to the first proxy server +$ ./connect.sh proxy 0 + +# connect to the second proxy server +$ ./connect.sh proxy 1 + +# connect to the node +$ ./connect.sh node +``` \ No newline at end of file From c949733c3904709497b95d078b9263543634916f Mon Sep 17 00:00:00 2001 From: Gus Luxton Date: Thu, 5 Mar 2020 18:45:27 -0400 Subject: [PATCH 03/13] Fixes --- docs/4.2/aws_terraform_guide.md | 16 ++-------------- docs/4.2/milv.config.yaml | 12 ++++++++++++ 2 files changed, 14 insertions(+), 14 deletions(-) diff --git a/docs/4.2/aws_terraform_guide.md b/docs/4.2/aws_terraform_guide.md index 50988f8b8ca57..885cd2356adf8 100644 --- a/docs/4.2/aws_terraform_guide.md +++ b/docs/4.2/aws_terraform_guide.md @@ -43,18 +43,6 @@ If you get an "access denied", "403 Forbidden" or similar message, you will need AWS IAM user that your `aws_access_key_id` and `aws_secret_access_key` refers to. -## AWS Services required to run Teleport in HA - -- [EC2 / Autoscale](#ec2-autoscale) -- [DynamoDB](#dynamodb) -- [S3](#s3://) -- [Route53](#route53) -- [NLB](#nlb-network-load-balancer) -- [IAM](#iam) -- [ACM](#acm) -- [SSM](#aws-systems-manager-parameter-store) - - ## Get the Terraform code Firstly, you'll need to clone the Teleport repo to get the Terraform code available on your system. @@ -251,7 +239,7 @@ As such, we recommend setting this to a known value. `export TF_VAR_use_acm="false"` If set to the string `"false"`, Terraform will use [LetsEncrypt](https://letsencrypt.org/) to provision the public-facing -web UI certificate for the Teleport cluster ([`route53_subdomain`](#route53_subdomain) - so https://teleport.example.com in this example). +web UI certificate for the Teleport cluster ([`route53_domain`](#route53_domain) - so https://teleport.example.com in this example). This uses an [AWS network load balancer](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/introduction.html) to load-balance connections to the Teleport cluster's web UI, and its SSL termination is handled by Teleport itself. @@ -314,7 +302,7 @@ the [`s3_bucket_name`](#s3_bucket_name) variable, under the `records` directory. ### Cluster domain The reference Terraform deployment sets the Teleport cluster up to be available on a domain defined in Route53, referenced -by the `route53_domain`[#route53_domain] variable. In our example this would be `teleport.example.com` +by the [`route53_domain`](#route53_domain) variable. In our example this would be `teleport.example.com` Teleport's web interface will be available on port 443 - https://teleport.example.com diff --git a/docs/4.2/milv.config.yaml b/docs/4.2/milv.config.yaml index e2b93327da9a2..1c7f46f4613ea 100644 --- a/docs/4.2/milv.config.yaml +++ b/docs/4.2/milv.config.yaml @@ -26,6 +26,9 @@ white-list-external: - "http://127.0.0.1:3000/debug/pprof" - "https://app.onelogin.com/saml/metadata/123456" - "https://en.wikipedia.org/wiki/Shebang_(Unix" + - "https://teleport.example.com:8443" + - "https://teleport.example.com" + - "https://[route53_domain](#route53_domain" black-list: - "./architecture/teleport_architecture.md" - "./quickstart__archive.md" @@ -37,7 +40,16 @@ files: - "#ec2-autoscale" - "#acm" - "#single-oss-teleport-amis-manual-gui-setup" + - path: "./aws_terraform_guide.md" + config: + white-list-internal: + - "#cluster_name" + - "#key_name" + - "#route53_domain" + - "#route53_zone" + - "#s3_bucket_name" - path: "./enterprise/index.md" config: white-list-internal: - "#fedrampfips" + From 2a112c328a7c504276b241ea9a9bb687a47b69d9 Mon Sep 17 00:00:00 2001 From: Gus Luxton Date: Thu, 5 Mar 2020 18:46:24 -0400 Subject: [PATCH 04/13] Update milv --- docs/4.2/milv.config.yaml | 1 - 1 file changed, 1 deletion(-) diff --git a/docs/4.2/milv.config.yaml b/docs/4.2/milv.config.yaml index 1c7f46f4613ea..796e205b6a4f1 100644 --- a/docs/4.2/milv.config.yaml +++ b/docs/4.2/milv.config.yaml @@ -52,4 +52,3 @@ files: config: white-list-internal: - "#fedrampfips" - From c071ba2c8c8a8bdc0277f7280958d40f66935326 Mon Sep 17 00:00:00 2001 From: Gus Luxton Date: Thu, 5 Mar 2020 18:47:49 -0400 Subject: [PATCH 05/13] Update line number --- docs/4.2/aws_terraform_guide.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/4.2/aws_terraform_guide.md b/docs/4.2/aws_terraform_guide.md index 885cd2356adf8..d36224507c00d 100644 --- a/docs/4.2/aws_terraform_guide.md +++ b/docs/4.2/aws_terraform_guide.md @@ -262,7 +262,7 @@ shared storage. [The desired size of the ASG is configured here](https://github. - 1 x m4.large monitoring server in an ASG which hosts the Grafana instance and receives monitoring data from each part of the cluster. [The desired size of the ASG is configured here](https://github.com/gravitational/teleport/blob/master/examples/aws/terraform/monitor_asg.tf#L12) - 1 x t2.medium bastion server which is the only permitted source for inbound SSH traffic to the instances. -[The instance types used for each ASG can be configured here](https://github.com/gravitational/teleport/blob/master/examples/aws/terraform/vars.tf#L23-L44) +[The instance types used for each ASG can be configured here](https://github.com/gravitational/teleport/blob/master/examples/aws/terraform/vars.tf#L23-L45) If you don't wish to set up a node or the monitoring services, you can set the `desired_size` and `min_size` for the ASG to `0` and Terraform won't provision it. From baa0f2814c2fe3df2e0e32b8561bee9aaf1209f3 Mon Sep 17 00:00:00 2001 From: Gus Luxton Date: Thu, 5 Mar 2020 18:49:50 -0400 Subject: [PATCH 06/13] Update title --- docs/4.2/aws_terraform_guide.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/4.2/aws_terraform_guide.md b/docs/4.2/aws_terraform_guide.md index d36224507c00d..de395de70a3e6 100644 --- a/docs/4.2/aws_terraform_guide.md +++ b/docs/4.2/aws_terraform_guide.md @@ -1,4 +1,4 @@ -# Running Teleport Enterprise in a high-availiability configuration on AWS +# Running Teleport Enterprise in HA mode on AWS This guide is designed to accompany our reference Terraform code (https://github.com/gravitational/teleport/tree/master/examples/aws/terraform) and describe how to use and administrate the resulting Teleport deployment. From 23b968403bad3b20026b14617f2bb7d6bdaaed31 Mon Sep 17 00:00:00 2001 From: Gus Luxton Date: Thu, 5 Mar 2020 18:55:04 -0400 Subject: [PATCH 07/13] Add a note about IAM --- docs/4.2/aws_terraform_guide.md | 18 +++++++++++++++++- 1 file changed, 17 insertions(+), 1 deletion(-) diff --git a/docs/4.2/aws_terraform_guide.md b/docs/4.2/aws_terraform_guide.md index de395de70a3e6..d7ee963785f2d 100644 --- a/docs/4.2/aws_terraform_guide.md +++ b/docs/4.2/aws_terraform_guide.md @@ -1,6 +1,6 @@ # Running Teleport Enterprise in HA mode on AWS -This guide is designed to accompany our reference Terraform code (https://github.com/gravitational/teleport/tree/master/examples/aws/terraform) +This guide is designed to accompany our [reference Terraform code](https://github.com/gravitational/teleport/tree/master/examples/aws/terraform) and describe how to use and administrate the resulting Teleport deployment. [TOC] @@ -11,6 +11,8 @@ Our code requires Terraform 0.12+. You can [download Terraform here](https://www `terraform` installed and available on your path. ```bash +$ which terraform +/usr/local/bin/terraform $ terraform version Terraform v0.12.20 ``` @@ -42,6 +44,20 @@ As a result, you should be able to run a command like `aws ec2 describe-instance If you get an "access denied", "403 Forbidden" or similar message, you will need to grant additional permissions to the AWS IAM user that your `aws_access_key_id` and `aws_secret_access_key` refers to. +As a general rule, we assume that any user running Terraform has administrator-level permissions for the following +AWS services: + +- [EC2](https://aws.amazon.com/ec2/) +- [S3](https://aws.amazon.com/s3/) +- [Route 53](https://aws.amazon.com/route53/) +- [DynamoDB](https://aws.amazon.com/dynamodb/) +- [Elastic Load Balancing](https://aws.amazon.com/elasticloadbalancing/) +- [IAM](https://aws.amazon.com/iam/) +- [SSM Parameter Store](https://docs.aws.amazon.com/systems-manager/latest/userguide/systems-manager-parameter-store.html) + +The Terraform deployment itself will create new IAM roles to be used by Teleport instances which only have limited +permission scopes for AWS services, however the initial setup must be done by a user with a high level of permissions. + ## Get the Terraform code From 4430a5e5ec7f52a8aaec9e36a380124c74458f50 Mon Sep 17 00:00:00 2001 From: Gus Luxton Date: Thu, 5 Mar 2020 19:02:21 -0400 Subject: [PATCH 08/13] More tweaks --- docs/4.2/aws_terraform_guide.md | 50 ++++++++++++++++++++------------- 1 file changed, 30 insertions(+), 20 deletions(-) diff --git a/docs/4.2/aws_terraform_guide.md b/docs/4.2/aws_terraform_guide.md index d7ee963785f2d..4445bde29cf57 100644 --- a/docs/4.2/aws_terraform_guide.md +++ b/docs/4.2/aws_terraform_guide.md @@ -179,8 +179,10 @@ specify in the `region` variable, and you will need a copy of this keypair to co `export TF_VAR_license_path="/home/user/teleport-license.pem"` The full local path to your Teleport license file, which customers can download from -[the Gravitational dashboard](https://dashboard.gravitational.com/) This license will be copied into AWS SSM and -automatically added to Teleport auth nodes in order to enable Teleport Enterprise/Pro functionality. +[the Gravitational dashboard](https://dashboard.gravitational.com/) + +This license will be copied into AWS SSM and automatically added to Teleport auth nodes in order to enable +Teleport Enterprise/Pro functionality. OSS users can provide any valid file here - it isn't used by the auth server in a Teleport OSS install. @@ -224,6 +226,8 @@ This must be a subdomain of the domain you chose for [`route53_zone`](#route53_z The Terraform example also provisions an S3 bucket to hold certificates provisioned by LetsEncrypt and distribute these to EC2 instances. This can be any S3-compatible name, and will be generated in the same region as set above. +This bucket is still provisioned when using ACM, as it is also used to store Teleport session logs. + ### email @@ -239,14 +243,15 @@ useful information. We recommend a generic ops/support email address which the t !!! note "How to set" `export TF_VAR_grafana_pass="CHANGE_THIS_VALUE"` -We deploy Grafana along with every Terraform deployment and automatically make stats on cluster usage available. -This variable sets up the password for the Grafana `admin` user. The Grafana web UI is served on the same subdomain -as specified above in [`route53_domain`](#route53_domain) on port 8443. +We deploy Grafana along with every Terraform deployment and automatically make stats on cluster usage available in +a custom dashboard. This variable sets up the password for the Grafana `admin` user. The Grafana web UI is served +on the same subdomain as specified above in [`route53_domain`](#route53_domain) on port 8443. With the variables set in this example, it would be available on https://teleport.example.com:8443 -If you do not change this from the default (`CHANGE_THIS_VALUE`), then it will be set to a random value for security. -As such, we recommend setting this to a known value. +If you do not change this from the default (`CHANGE_THIS_VALUE`), then it will be set to a random value for security +and you will need to log into the monitoring instance to discover this manually. As such, we recommend setting this +to a known value at the outset. ### use_acm @@ -271,12 +276,13 @@ with this command: `terraform import aws_acm_certificate.cert ` Our reference deployment will provision the following instances for your cluster by default: -- 2 x m4.large Teleport auth instances in an ASG, behind an internal network load balancer, configured using DynamoDB for +- 2 x m4.large Teleport **auth** instances in an ASG, behind an internal network load balancer, configured using DynamoDB for shared storage. [The desired size of the ASG is configured here](https://github.com/gravitational/teleport/blob/master/examples/aws/terraform/auth_asg.tf#L11) -- 2 x m4.large Teleport proxy instances in an ASG, behind a public-facing load balancer - NLB for LetsEncrypt, ALB for ACM. [The desired size of the ASG is configured here](https://github.com/gravitational/teleport/blob/master/examples/aws/terraform/proxy_asg.tf#L12) -- 1 x m4.large Teleport node instance in an ASG. [The desired size of the ASG is configured here](https://github.com/gravitational/teleport/blob/master/examples/aws/terraform/node_asg.tf#L10) -- 1 x m4.large monitoring server in an ASG which hosts the Grafana instance and receives monitoring data from each part of the cluster. [The desired size of the ASG is configured here](https://github.com/gravitational/teleport/blob/master/examples/aws/terraform/monitor_asg.tf#L12) -- 1 x t2.medium bastion server which is the only permitted source for inbound SSH traffic to the instances. +- 2 x m4.large Teleport **proxy** instances in an ASG, behind a public-facing load balancer - NLB for LetsEncrypt, ALB for ACM. [The desired size of the ASG is configured here](https://github.com/gravitational/teleport/blob/master/examples/aws/terraform/proxy_asg.tf#L12) +- 1 x m4.large Teleport **node** instance in an ASG. [The desired size of the ASG is configured here](https://github.com/gravitational/teleport/blob/master/examples/aws/terraform/node_asg.tf#L10) +- 1 x m4.large monitoring server in an ASG which hosts the Grafana instance and receives monitoring data from each service in the cluster. [The desired size of the ASG is configured here](https://github.com/gravitational/teleport/blob/master/examples/aws/terraform/monitor_asg.tf#L12) +- 1 x t2.medium bastion server which is the only permitted source for inbound SSH traffic to the instances. This is done +to avoid exposing each instance to the internet directly. [The instance types used for each ASG can be configured here](https://github.com/gravitational/teleport/blob/master/examples/aws/terraform/vars.tf#L23-L45) @@ -313,7 +319,9 @@ the [`s3_bucket_name`](#s3_bucket_name) variable, under the `records` directory. !!! tip "Tip" S3 provides [Amazon S3 Object Lock](https://docs.aws.amazon.com/AmazonS3/latest/dev/object-lock.html), - which is useful for customers deploying Teleport in regulated environments. + which is useful for customers deploying Teleport in regulated environments. Configuration of object lock is out of + the scope of this guide. + ### Cluster domain @@ -322,8 +330,9 @@ by the [`route53_domain`](#route53_domain) variable. In our example this would b Teleport's web interface will be available on port 443 - https://teleport.example.com -Teleport's proxy SSH interface will be available on port 3023 (this is the default port used when connecting with the -`tsh` client) +Teleport's proxy SSH interface will be available on port 3023 - this is the default port used when connecting with the +`tsh` client and will not require any additional configuration. + ## Deploying with Terraform @@ -380,7 +389,8 @@ Do you want to perform these actions? Entering `yes` here will start the Terraform deployment. It takes around 8-10 minutes to deploy in full. -#### Destroy/shutdown + +#### Destroy/shutdown a Terraform deployment If you need to tear down a running deployment for any reason, you can run `terraform destroy`. @@ -411,8 +421,8 @@ $ echo ${AUTH_IP} 172.31.0.196 ``` -3 - Use both these values to SSH into the auth server (make sure that the AWS keypair that you specified in the -[`key_name`](#key_name) variable is available in the current directory): +3 - Use both these values to SSH into the auth server. Make sure that the AWS keypair that you specified in the +[`key_name`](#key_name) variable is available in the current directory, or update the `-i` parameter to point to it: ```bash $ ssh -i ${TF_VAR_key_name}.pem -J ec2-user@${BASTION_IP} ec2-user@${AUTH_IP} @@ -433,7 +443,7 @@ Last login: Tue Mar 3 18:57:12 2020 from 1.2.3.5 https://aws.amazon.com/amazon-linux-2/ 1 package(s) needed for security, out of 6 available Run "sudo yum update" to apply all updates. -[ec2-user@ip-172-31-0-196 ~]$ +[ec2-user@ip-172-31-0-196 ~]$ ``` 4 - Use the `tctl` command to create an admin user for Teleport: @@ -780,4 +790,4 @@ $ ./connect.sh proxy 1 # connect to the node $ ./connect.sh node -``` \ No newline at end of file +``` From 4e5ef9e2443089e5a0619017bac1f7fd73ce713f Mon Sep 17 00:00:00 2001 From: Ben Arent Date: Thu, 5 Mar 2020 15:59:25 -0800 Subject: [PATCH 09/13] Redcude toc_depth, and added a few tweaks to aws terraform guide --- docs/4.2.yaml | 1 + docs/4.2/aws_terraform_guide.md | 11 ++++------- 2 files changed, 5 insertions(+), 7 deletions(-) diff --git a/docs/4.2.yaml b/docs/4.2.yaml index 0dc3ffe4b165a..d653a07385aa7 100644 --- a/docs/4.2.yaml +++ b/docs/4.2.yaml @@ -21,6 +21,7 @@ markdown_extensions: - footnotes - toc: marker: '[TOC]' + toc_depth: 2 extra_css: [] plugins: - search: diff --git a/docs/4.2/aws_terraform_guide.md b/docs/4.2/aws_terraform_guide.md index 4445bde29cf57..b18a6871e9b6f 100644 --- a/docs/4.2/aws_terraform_guide.md +++ b/docs/4.2/aws_terraform_guide.md @@ -1,6 +1,6 @@ # Running Teleport Enterprise in HA mode on AWS -This guide is designed to accompany our [reference Terraform code](https://github.com/gravitational/teleport/tree/master/examples/aws/terraform) +This guide is designed to accompany our [reference Terraform code](https://github.com/gravitational/teleport/tree/master/examples/aws/terraform#terraform-based-provisioning-example-amazon-single-ami) and describe how to use and administrate the resulting Teleport deployment. [TOC] @@ -17,8 +17,8 @@ $ terraform version Terraform v0.12.20 ``` -You will also require the `aws` command line tool. This is available in Ubuntu/Debian/Fedora/CentOS -and MacOS Homebrew as the `awscli` package. +You will also require the `aws` command line tool. This is available in Ubuntu/Debian/Fedora/CentOS from [https://aws.amazon.com/cli/](https://aws.amazon.com/cli/) +and MacOS Homebrew as the `awscli` package using `brew When possible, installing via a package is always preferable. If you can't find a package available for your distribution, you can install `python3` and `pip`, then use these to install `awscli` using `pip3 install awscli` @@ -459,7 +459,7 @@ When the user 'teleport-admin' activates their account, they will be assigned ro 5 - Click the link to launch the Teleport web UI and finish setting up your user. You will need to scan the QR code with an TOTP-compatible app like Google Authenticator or Authy. You will also set a password for the -`teleportadmin` user on this page. +`teleport-admin` user on this page. Once this user is successfully configured, you should be logged into the Teleport web UI. @@ -651,11 +651,8 @@ ways to integrate Teleport onto your servers. We recommend looking at our [Admin To add new nodes/EC2 servers that you can "SSH into" you'll need to: - [Install the Teleport binary on the Server](admin-guide.md#installing) - - [Run Teleport - we recommend using systemd](admin-guide.md#systemd-unit-file) - - [Set the correct settings in /etc/teleport.yaml](admin-guide.md#configuration-file) - - [Add nodes to the Teleport cluster](admin-guide.md#adding-nodes-to-the-cluster) From de99dec3ab4c4bbff924ff7e363afda966a6e215 Mon Sep 17 00:00:00 2001 From: Ben Arent Date: Thu, 5 Mar 2020 16:29:14 -0800 Subject: [PATCH 10/13] Reduce amount of tip for TF_VAR --- docs/4.2/aws_terraform_guide.md | 40 +++++++++++++-------------------- 1 file changed, 15 insertions(+), 25 deletions(-) diff --git a/docs/4.2/aws_terraform_guide.md b/docs/4.2/aws_terraform_guide.md index b18a6871e9b6f..e4a187e44aefc 100644 --- a/docs/4.2/aws_terraform_guide.md +++ b/docs/4.2/aws_terraform_guide.md @@ -124,16 +124,14 @@ Things you will need to decide on: ### region -!!! note "How to set" - `export TF_VAR_region="us-west-2"` +Setting `export TF_VAR_region="us-west-2"` The AWS region to run in. You should pick from the supported list as detailed in the [README](https://github.com/gravitational/teleport/blob/master/examples/aws/terraform/README.md). ### cluster_name -!!! note "How to set" - `export TF_VAR_cluster_name="example-cluster"` +Setting `export TF_VAR_cluster_name="example-cluster"` The cluster name is the internal Teleport cluster name to use. This should be unique, and not contain spaces, dots (.) or other special characters. This will appear in the web UI for your cluster and cannot be changed after creation @@ -142,8 +140,7 @@ without rebuilding your cluster from scratch, so choose carefully. ### ami_name -!!! note "How to set" - `export TF_VAR_ami_name="gravitational-teleport-ami-ent-4.2.3"` +Setting `export TF_VAR_ami_name="gravitational-teleport-ami-ent-4.2.3"` Gravitational automatically builds and publishes OSS, Enterprise and Enterprise FIPS 140-2 AMIs when we release a new version of Teleport. The AMI names follow the format: `gravitational-teleport-ami--` @@ -154,20 +151,20 @@ FIPS 140-2 compatible AMIs (which deploy Teleport in FIPS 140-2 mode by default) The AWS account ID which publishes these AMIs is `126027368216`. You can list the available AMIs with the example `awscli` commands below. The output is in JSON format by default. -!!! tip "List OSS AMIs" +!!! tip "List Gravitational AMIs" + OSS AMIs
`aws ec2 describe-images --owners 126027368216 --filters 'Name=name,Values=gravitational-teleport-ami-oss*'` -!!! tip "List Enterprise AMIs" + Enterprise AMIs
`aws ec2 describe-images --owners 126027368216 --filters 'Name=name,Values=gravitational-teleport-ami-ent*'` -!!! tip "List Enterprise FIPS 140-2 AMIs" + List Enterprise FIPS 140-2 AMIs
`aws ec2 describe-images --owners 126027368216 --filters 'Name=name,Values=gravitational-teleport-ami-ent*-fips'` ### key_name -!!! note "How to set" - `export TF_VAR_key_name="exampleuser"` +Setting `export TF_VAR_key_name="exampleuser"` The AWS keypair name to use when deploying EC2 instances. This must exist in the same region as you specify in the `region` variable, and you will need a copy of this keypair to connect to the deployed EC2 instances. @@ -175,8 +172,7 @@ specify in the `region` variable, and you will need a copy of this keypair to co ### license_path -!!! note "How to set" - `export TF_VAR_license_path="/home/user/teleport-license.pem"` +Setting `export TF_VAR_license_path="/home/user/teleport-license.pem"` The full local path to your Teleport license file, which customers can download from [the Gravitational dashboard](https://dashboard.gravitational.com/) @@ -189,8 +185,7 @@ OSS users can provide any valid file here - it isn't used by the auth server in ### route53_zone -!!! note "How to set" - `export TF_VAR_route53_zone="example.com"` +Setting `export TF_VAR_route53_zone="example.com"` Our Terraform setup requires you to have your domain provisioned in AWS Route 53 - it will automatically add DNS records for [`route53_domain`](#route53_domain) as set up below. You can list these with this command: @@ -209,8 +204,7 @@ You should use the appropriate domain without the trailing dot. ### route53_domain -!!! note "How to set" - `export TF_VAR_route53_domain="teleport.example.com"` +Setting `export TF_VAR_route53_domain="teleport.example.com"` A subdomain to set up as a CNAME to the Teleport load balancer for web access. This will be the public-facing domain that people use to connect to your Teleport cluster, so choose wisely. @@ -220,8 +214,7 @@ This must be a subdomain of the domain you chose for [`route53_zone`](#route53_z ### s3_bucket_name -!!! note "How to set" - `export TF_VAR_s3_bucket_name="example-cluster"` +Setting `export TF_VAR_s3_bucket_name="example-cluster"` The Terraform example also provisions an S3 bucket to hold certificates provisioned by LetsEncrypt and distribute these to EC2 instances. This can be any S3-compatible name, and will be generated in the same region as set above. @@ -231,8 +224,7 @@ This bucket is still provisioned when using ACM, as it is also used to store Tel ### email -!!! note "How to set" - `export TF_VAR_email="support@example.com"` +Setting `export TF_VAR_email="support@example.com"` LetsEncrypt requires an email address for every certificate registered which can be used to send notifications and useful information. We recommend a generic ops/support email address which the team deploying Teleport has access to. @@ -240,8 +232,7 @@ useful information. We recommend a generic ops/support email address which the t ### grafana_pass -!!! note "How to set" - `export TF_VAR_grafana_pass="CHANGE_THIS_VALUE"` +Setting `export TF_VAR_grafana_pass="CHANGE_THIS_VALUE"` We deploy Grafana along with every Terraform deployment and automatically make stats on cluster usage available in a custom dashboard. This variable sets up the password for the Grafana `admin` user. The Grafana web UI is served @@ -256,8 +247,7 @@ to a known value at the outset. ### use_acm -!!! note "How to set" - `export TF_VAR_use_acm="false"` +Setting `export TF_VAR_use_acm="false"` If set to the string `"false"`, Terraform will use [LetsEncrypt](https://letsencrypt.org/) to provision the public-facing web UI certificate for the Teleport cluster ([`route53_domain`](#route53_domain) - so https://teleport.example.com in this example). From d2b6a10753989cd4efdd27618818df0c99d4ed58 Mon Sep 17 00:00:00 2001 From: Gus Luxton Date: Fri, 6 Mar 2020 12:39:05 -0400 Subject: [PATCH 11/13] Add tsh --- docs/4.2/aws_terraform_guide.md | 112 +++++++++++++++++++++++--------- 1 file changed, 83 insertions(+), 29 deletions(-) diff --git a/docs/4.2/aws_terraform_guide.md b/docs/4.2/aws_terraform_guide.md index e4a187e44aefc..05d69afc7b848 100644 --- a/docs/4.2/aws_terraform_guide.md +++ b/docs/4.2/aws_terraform_guide.md @@ -17,11 +17,17 @@ $ terraform version Terraform v0.12.20 ``` -You will also require the `aws` command line tool. This is available in Ubuntu/Debian/Fedora/CentOS from [https://aws.amazon.com/cli/](https://aws.amazon.com/cli/) -and MacOS Homebrew as the `awscli` package using `brew +You will also require the `aws` command line tool. This is available in Ubuntu/Debian/Fedora/CentOS and MacOS Homebrew +as the `awscli` package. + +Fedora/CentOS: `yum -y install awscli` + +Ubuntu/Debian: `apt-get -y install awscli` + +MacOS (with [Homebrew](https://brew.sh/)): `brew install awscli` When possible, installing via a package is always preferable. If you can't find a package available for -your distribution, you can install `python3` and `pip`, then use these to install `awscli` using `pip3 install awscli` +your distribution, you can also download the tool from [https://aws.amazon.com/cli/](https://aws.amazon.com/cli/) We will assume that you have configured your AWS cli access with credentials available at `~/.aws/credentials`: @@ -55,13 +61,14 @@ AWS services: - [IAM](https://aws.amazon.com/iam/) - [SSM Parameter Store](https://docs.aws.amazon.com/systems-manager/latest/userguide/systems-manager-parameter-store.html) -The Terraform deployment itself will create new IAM roles to be used by Teleport instances which only have limited -permission scopes for AWS services, however the initial setup must be done by a user with a high level of permissions. +The Terraform deployment itself will create new IAM roles to be used by Teleport instances which have appropriately +limited permission scopes for AWS services. However, the initial cluster setup must be done by a user with a high +level of AWS permissions. ## Get the Terraform code -Firstly, you'll need to clone the Teleport repo to get the Terraform code available on your system. +Firstly, you'll need to clone the Teleport repo to get the Terraform code available on your system: ```bash $ git clone https://github.com/gravitational/teleport @@ -111,7 +118,7 @@ Terraform modules use variables to pass in input. You can do this in a few ways: - by setting environment variables For this guide, we are going to make extensive use of environment variables. This is because it makes it easier for us -to reference values from our configuration when running Teleport commands. +to reference values from our configuration when running Teleport commands after the cluster has been created. Any set environment variable starting with `TF_VAR_` is automatically processed and stripped down by Terraform, so `TF_VAR_test_variable` becomes `test_variable`. @@ -126,16 +133,18 @@ Things you will need to decide on: Setting `export TF_VAR_region="us-west-2"` -The AWS region to run in. You should pick from the supported list as detailed in the [README](https://github.com/gravitational/teleport/blob/master/examples/aws/terraform/README.md). +The AWS region to run in. You should pick from the supported list as detailed in the [README](https://github.com/gravitational/teleport/blob/master/examples/aws/terraform/README.md). These are regions which support [DynamoDB encryption +at rest](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/EncryptionAtRest.html). ### cluster_name Setting `export TF_VAR_cluster_name="example-cluster"` -The cluster name is the internal Teleport cluster name to use. This should be unique, and not contain spaces, dots (.) -or other special characters. This will appear in the web UI for your cluster and cannot be changed after creation -without rebuilding your cluster from scratch, so choose carefully. +This is the internal Teleport cluster name to use. This should be unique, and not contain spaces, dots (.) or other +special characters. Some AWS services will not allow you to use dots in a name, so this should not be set to a domain +name. This will appear in the web UI for your cluster and cannot be changed after creation without rebuilding your +cluster from scratch, so choose carefully. ### ami_name @@ -167,7 +176,8 @@ the example `awscli` commands below. The output is in JSON format by default. Setting `export TF_VAR_key_name="exampleuser"` The AWS keypair name to use when deploying EC2 instances. This must exist in the same region as you -specify in the `region` variable, and you will need a copy of this keypair to connect to the deployed EC2 instances. +specify in the `region` variable, and you will need a copy of this keypair available to connect to the deployed +EC2 instances. Do not use a keypair that you do not have access to. ### license_path @@ -175,12 +185,12 @@ specify in the `region` variable, and you will need a copy of this keypair to co Setting `export TF_VAR_license_path="/home/user/teleport-license.pem"` The full local path to your Teleport license file, which customers can download from -[the Gravitational dashboard](https://dashboard.gravitational.com/) +[the Gravitational dashboard](https://dashboard.gravitational.com/). -This license will be copied into AWS SSM and automatically added to Teleport auth nodes in order to enable +This license will be uploaded to AWS SSM and automatically downloaded to Teleport auth nodes in order to enable Teleport Enterprise/Pro functionality. -OSS users can provide any valid file here - it isn't used by the auth server in a Teleport OSS install. +(OSS users can provide any valid local file path here - it isn't used by the auth server in a Teleport OSS install) ### route53_zone @@ -257,8 +267,12 @@ to load-balance connections to the Teleport cluster's web UI, and its SSL termin If set to the string `"true"`, Terraform will use [AWS ACM](https://aws.amazon.com/certificate-manager/) to provision the public-facing web UI certificate for the cluster. This uses an [AWS application load balancer](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/introduction.html) to load-balance connections to the Teleport cluster's web UI, and its SSL termination is handled by the load balancer. -If you wish to use a pre-existing ACM certificate rather than having Terraform generate one for you, you can import it -with this command: `terraform import aws_acm_certificate.cert ` +If you wish to use a pre-existing ACM certificate rather than having Terraform generate one for you, you can make +Terraform use it by running this command before `terraform apply`: + +```bash +terraform import aws_acm_certificate.cert +``` ## Reference deployment defaults @@ -276,8 +290,8 @@ to avoid exposing each instance to the internet directly. [The instance types used for each ASG can be configured here](https://github.com/gravitational/teleport/blob/master/examples/aws/terraform/vars.tf#L23-L45) -If you don't wish to set up a node or the monitoring services, you can set the `desired_size` and `min_size` for the ASG -to `0` and Terraform won't provision it. +If you don't wish to set up a node or the monitoring services, you can set the `desired_size` and `min_size` for an ASG +to `0` and Terraform will not provision it. ### Cluster state database storage @@ -287,7 +301,7 @@ table for cluster state will be the same as the cluster name configured in the [ In our example, the DynamoDB table would be called `example-cluster`. -More information about how Teleport works with DynamoDB can be found in our [DynamoDB Admin Guide](https://gravitational.com/teleport/docs/admin-guide/#using-dynamodb) +More information about how Teleport works with DynamoDB can be found in our [DynamoDB Admin Guide](https://gravitational.com/teleport/docs/admin-guide/#using-dynamodb). ### Audit event storage @@ -298,13 +312,15 @@ with `-events` appended to the end. In our example, the DynamoDB table would be called `example-cluster-events`. -More information about how Teleport works with DynamoDB can be found in our [DynamoDB Admin Guide](https://gravitational.com/teleport/docs/admin-guide/#using-dynamodb) +More information about how Teleport works with DynamoDB can be found in our [DynamoDB Admin Guide](https://gravitational.com/teleport/docs/admin-guide/#using-dynamodb). ### Recorded session storage The reference Terraform deployment sets Teleport up to store recorded session logs in the same S3 bucket configured in -the [`s3_bucket_name`](#s3_bucket_name) variable, under the `records` directory. In our example this would be `s3://example-cluster/records` +the [`s3_bucket_name`](#s3_bucket_name) variable, under the `records` directory. + +In our example this would be `s3://example-cluster/records` !!! tip "Tip" @@ -454,6 +470,40 @@ code with an TOTP-compatible app like Google Authenticator or Authy. You will al Once this user is successfully configured, you should be logged into the Teleport web UI. +### Logging into the cluster with tsh + +You can use the Teleport command line tool (`tsh`) to log into your Teleport cluster after provisioning a user. + +You can [download the Teleport package containing the `tsh` client from here](https://gravitational.com/teleport/download) +- the client is the same for both OSS and Enterprise versions of Teleport. + +```bash +$ tsh login --proxy=${TF_VAR_route53_domain}:443 --user=teleport-admin +Enter password for Teleport user teleport-admin: +Enter your OTP token: +567989 +> Profile URL: https://teleport.example.com:443 + Logged in as: teleport-admin + Cluster: example-cluster + Roles: admin* + Logins: root + Valid until: 2020-03-06 22:07:11 -0400 AST [valid for 12h0m0s] + Extensions: permit-agent-forwarding, permit-port-forwarding, permit-pty + + +* RBAC is only available in Teleport Enterprise + https://gravitational.com/teleport/docs/enterprise + +$ tsh ls +Node Name Address Labels +---------------------------- ----------------- ------ +ip-172-31-11-69-ec2-internal 172.31.11.69:3022 + +$ tsh ssh root@ip-172-31-11-69-ec2-internal +[root@ip-172-31-11-69 ~]# +``` + + ## Restarting/checking Teleport services ### LetsEncrypt @@ -491,6 +541,8 @@ $ aws ec2 describe-instances --filters "Name=tag:TeleportCluster,Values=${TF_VAR 172.31.1.78 ``` +You can run `tctl` commands on **any** of the auth instances connected to your cluster, however. + #### Proxy service ```bash @@ -573,7 +625,7 @@ $ journalctl -u teleport-auth.service ``` Remember that there is more than one auth instance in an HA deployment. You should use this command to get the IP addresses -of each auth instance that you'll need to connect to: +of each auth instance that you'd need to connect to for checking logs: ```bash $ aws ec2 describe-instances --filters "Name=tag:TeleportCluster,Values=${TF_VAR_cluster_name},Name=tag:TeleportRole,Values=auth" --query "Reservations[*].Instances[*].PrivateIpAddress" --output text @@ -581,6 +633,8 @@ $ aws ec2 describe-instances --filters "Name=tag:TeleportCluster,Values=${TF_VAR 172.31.1.78 ``` +You can run `tctl` commands on **any** of the auth instances connected to your cluster, however. + #### Proxy service (ACM) ```bash @@ -603,8 +657,8 @@ You can get detailed logs for the Teleport proxy service using the `journalctl` $ journalctl -u teleport-proxy-acm.service ``` -Remember that there is more than one proxy instance in an HA deployment. You should use this command to get the IP addresses -of each auth instance that you'll need to connect to: +Remember that there is more than one proxy instance in an HA deployment. You can use this command to get the IP addresses +of each proxy instance that you'd need to connect to for checking logs: ```bash $ aws ec2 describe-instances --filters "Name=tag:TeleportCluster,Values=${TF_VAR_cluster_name},Name=tag:TeleportRole,Values=proxy" --query "Reservations[*].Instances[*].PrivateIpAddress" --output text @@ -635,7 +689,7 @@ $ journalctl -u teleport-node.service ``` ## Adding EC2 instances to your Teleport cluster -Customers run many workloads within EC2 and depending on how you work there are many +Customers run many workloads within EC2 and depending on how you work, there are many ways to integrate Teleport onto your servers. We recommend looking at our [Admin manual](https://gravitational.com/teleport/docs/admin-guide/#installing). To add new nodes/EC2 servers that you can "SSH into" you'll need to: @@ -721,11 +775,11 @@ spec: web_proxy_addr: teleport.example.com:443 ``` -You can generate a token for adding the trusted cluster via the `tctl` command line tool after -connecting to an auth server. Follow the instructions in our [trusted cluster guide](https://gravitational.com/teleport/docs/trustedclusters/#dynamic-join-tokens). +You can generate a token for adding the trusted cluster using `tctl tokens add --type=trusted_cluster` after connecting +to an auth server. Follow the instructions in our [trusted cluster guide](https://gravitational.com/teleport/docs/trustedclusters/#dynamic-join-tokens). -## Some quick commands +## Script to quickly connect to instances Here's a bash script that you can use to quickly connect to instances: From 30bf1c8e387f6ff261100baa8d6c43789f7d6c8b Mon Sep 17 00:00:00 2001 From: Gus Luxton Date: Fri, 6 Mar 2020 12:48:42 -0400 Subject: [PATCH 12/13] Add extra LB details --- docs/4.2/aws_terraform_guide.md | 26 +++++++++++++++++++++++--- 1 file changed, 23 insertions(+), 3 deletions(-) diff --git a/docs/4.2/aws_terraform_guide.md b/docs/4.2/aws_terraform_guide.md index 05d69afc7b848..110e16d926948 100644 --- a/docs/4.2/aws_terraform_guide.md +++ b/docs/4.2/aws_terraform_guide.md @@ -334,10 +334,30 @@ In our example this would be `s3://example-cluster/records` The reference Terraform deployment sets the Teleport cluster up to be available on a domain defined in Route53, referenced by the [`route53_domain`](#route53_domain) variable. In our example this would be `teleport.example.com` -Teleport's web interface will be available on port 443 - https://teleport.example.com +Teleport's web interface will be available on port 443 - https://teleport.example.com - this is via a configured CNAME +to the AWS load balancer. -Teleport's proxy SSH interface will be available on port 3023 - this is the default port used when connecting with the -`tsh` client and will not require any additional configuration. +Teleport's proxy SSH interface will be available via a network load balancer with an AWS-controlled hostname on port 3023. +This is the default port used when connecting with the `tsh` client and will not require any additional configuration. + +Teleport's tunnel interface will be available via the same network load balancer with an AWS-controlled hostname on port +3024. This allows trusted clusters and nodes connected via node tunnelling to access the cluster. + +You can get the hostname of the proxy/tunnel network load balancer if needed with this command: + +```bash +$ aws elbv2 describe-load-balancers --names "${TF_VAR_cluster_name}-proxy" --query "LoadBalancers[*].DNSName" --output text +example-cluster-proxy-7c97b76593d6bf21.elb.us-east-1.amazonaws.com +``` + +Teleport's auth server interface will be available via an internal load balancer with an AWS-controlled hostname on port 3025. + +You can get the hostname of the internal auth load balancer if needed with this command: + +```bash +$ aws elbv2 describe-load-balancers --names "${TF_VAR_cluster_name}-auth" --query "LoadBalancers[*].DNSName" --output text +example-cluster-auth-c5b0fc2764ee015b.elb.us-east-1.amazonaws.com +``` ## Deploying with Terraform From 1dc3022e126973d43351a71a6ca2845947a9c2f3 Mon Sep 17 00:00:00 2001 From: Gus Luxton Date: Fri, 6 Mar 2020 12:49:16 -0400 Subject: [PATCH 13/13] Note about timing --- docs/4.2/aws_terraform_guide.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/4.2/aws_terraform_guide.md b/docs/4.2/aws_terraform_guide.md index 110e16d926948..56278520feca2 100644 --- a/docs/4.2/aws_terraform_guide.md +++ b/docs/4.2/aws_terraform_guide.md @@ -343,7 +343,7 @@ This is the default port used when connecting with the `tsh` client and will not Teleport's tunnel interface will be available via the same network load balancer with an AWS-controlled hostname on port 3024. This allows trusted clusters and nodes connected via node tunnelling to access the cluster. -You can get the hostname of the proxy/tunnel network load balancer if needed with this command: +After deploying, you can get the hostname of the proxy/tunnel network load balancer if needed with this command: ```bash $ aws elbv2 describe-load-balancers --names "${TF_VAR_cluster_name}-proxy" --query "LoadBalancers[*].DNSName" --output text @@ -352,7 +352,7 @@ example-cluster-proxy-7c97b76593d6bf21.elb.us-east-1.amazonaws.com Teleport's auth server interface will be available via an internal load balancer with an AWS-controlled hostname on port 3025. -You can get the hostname of the internal auth load balancer if needed with this command: +After deploying, you can get the hostname of the internal auth load balancer if needed with this command: ```bash $ aws elbv2 describe-load-balancers --names "${TF_VAR_cluster_name}-auth" --query "LoadBalancers[*].DNSName" --output text