Cluster Configuration

[kontsevoy]

Problem

We have two different concerns when it comes to configuration: machine configuration and cluster configuration and we have been struggling with how to solve this problem. Thus far we have been mixing and matching two different approaches to solve the problem.

1. The teleport.yaml configuration file. This approach is easy to understand and manage, but it is a pain to configure complex clusters with multiple Auth servers (HA).
2. The concept of "dynamic resources" which we introduced in Teleport 2.x to solve the HA problem by moving some configuration to the database, but they can conflict with what's in teleport.yaml so we added this clumsy "dynamic_config" switch. Basically we treat the same thing as a resource and also as a configuration setting.

Another problem with "dynamic resources" is that resources and configuration are two separate things (configuration is a singleton, resources are not).

As a result, dynamically configuring HA clusters is not easy.

Solution

The basic idea is simple. Everything in Teleport must be one of:

• Configuration setting (a singleton) which exists in teleport.yaml.
• Resource which you can perform CRUD operations on with tctl.

But not both at the same time.

Implementation

The above outlined problem does not affect proxies or nodes as they only have machine specific configuration. This issues only affects the Auth service section.

Fields to be removed

u2f: Configuration for "universal 2nd factor". This field was deprecated in Teleport 2.0.

Fields which will be ignored

The following fields will be ignored: dynamic_config, trusted_clusters, oidc_connectors. We will print a warning to the logs that these fields are not being ignored.

High Availability (HA) Behavior

To have consistent behavior across multiple Auth Servers and make configuration simple, we will make the following changes:

1. Configuration is a resource on the backend called cluster-configuration and will contain all cluster configuration (cluster_name, tokens, authorities, reverse_tunnels, and authentication). Upon start we will read configuration from a file and update this resource on the backend so all the settings change atomically and all Auth Servers return a consistent answer. The cluster-configuration resource can only be updated from file configuration.

2. The last Auth Server that starts will override the cluster configuration resource.

Migration

The following fields do not need to be migrated: cluster_name, tokens, authorities, and reverse_tunnels.

The authentication section will need to be migrated. We will need to convert ClusterAuthPreference and UniversalSecondFactorSettings to sit within authentication then push that to the backend.

[russjones]

The core of my proposal is to make a singleton resource on the backend that contains all cluster configuration. This resource can only be updated from this section of file configuration, no other way. We need to do this for the following reason:

1. We need a way to share information across multiple Auth Server processes and have them all have them all return a consistent response.
2. Our existing customers have shown a preference to configure authentication using file configuration.
3. If we leave connectors as resources that can be manipulated outside of file configuration we're at the same place as before (configuration is not a singleton).

Teleport 2.3 Clean Install

I propose the Auth section for Teleport 2.3 look like below.

type Auth struct {
  // Service common configuration of a Teleport service.
  Service

  // ClusterName is the name of the Teleport cluster.
  ClusterName string `yaml:"cluster_name,omitempty"`

  // Authentication contains the authentication configuration for this cluster.
  Authentication Authentication `yaml:"authentication,omitempty"`
}

type Authentication struct {
  // Type can be any of the following: local, oidc, saml. If the value is "oidc"
  // or "saml" a OIDCConnector or SAMLConnector connector will be used
  Type string `yaml:"type"`
  
  // SecondFactor can be any of the following: off, otp, u2f. If the value is
  // "off" authentication will be password only. If the value is "otp" Google
  // Authenticator and TOTP will be used. If the value is "u2f" the 
  // UniversalSecondFactor settings below will be used.
  SecondFactor string `yaml:"second_factor,omitempty"`

  // U2F contains configuration for the Universal Second Factor.
  U2F *UniversalSecondFactor `yaml:"u2f,omitempty"`
}

type UniversalSecondFactor struct {
  // AppID is the application ID for U2F.
  AppID string `yaml:"app_id"`

  // Facets is the facets for U2F.
  Facets []string `yaml:"facets"`
}

Teleport 2.x Migration

We need to migrate users on Teleport 2.x to the new format, I propose we migrate the existing fields in the following manner:

type Auth struct {
    Before: DomainName string
    After:  ClusterName gets migrated to a resource.

    Before: TrustedClusters []TrustedCluster
    After:  We log that configuring Trusted Clusters using file configuration
            is no longer supported but existing trust relationships and tunnels
            will be maintained. In the future Trusted Cluster configuration
            should be done using resources.

    Before: Authorities []Authority
    After:  Since this was only used internally I propose we remove it for the
            following reasons: 1) and we now have an publicly supported API to add
            a CertAuthority, and 2) this can potentially cause inconsistent
            behavior because it will append a CertAuthority to your backend but
            does not remove a CertAuthority if you remove it here.
         
    Before: ReverseTunnels []ReverseTunnel
    After:  Since this was only used internally I propose we remove it for the
            following reasons: 1) and we now have an publicly supported API to add
            a ReverseTunnel, and 2) this can potentially cause inconsistent
            behavior because it will append a ReverseTunnel to your backend but
            does not remove a ReverseTunnel if you remove it here.

    Before: StaticTokens []StaticToken
    After:  We log that configuring static tokens using file configuration is no
            longer supported but existing tokens in this file will be migrated
            to the backend. We tell people to use dynamic tokens with an infinite TTL.
            I propose we do that because at the moment these
            are stored in memory for each Auth Server so you can get inconsistent
            behavior from Auth Servers which we need to resolve. If we put these
            into the backend from here we will have inconsistent behavior because
            they can't be removed using file configuration.

    Before: Authentication *AuthenticationConfig
    After:  We maintain this structure but remove OIDC support from it. If OIDC
            is found we print a warning.

    Before: OIDCConnectors []OIDCConnector
    After:  This was depreciated in Teleport 2.0 and I propose we remove it
            completely in Teleport 2.3. Configuration of a OIDC connector should be
            done in the Authentication section.

    Before: U2F U2F
    After:  This was depreciated in Teleport 2.0 and I propose we remove it
            completely in Teleport 2.3.

    Before: DynamicConfig *bool
    After:  We log that this field is no longer being used and will be removed in
            Teleport 2.4.
}