Login Performance with DynamoDB

[russjones]

Problem

Logins when using DynamoDB are fairly slow and take about 4 seconds when the proxy and auth servers are running locally.

Proposed Solution

During login and creation of a session, Teleport frequently checks what mode it's running in to change it's behavior from regular Teleport to recording proxy mode. All of these checks hit DynamoDB and there are about 5-10 of these requests.

To improve performance, Teleport should request the cluster configuration once, then use that (cached) copy for the remainder of the session.

[russjones]

I'm going to reopen this issue for a little bit longer.

I ran some testing with the latest version of Teleport, this was the configuration:

• Teleport 2.5.4.
• Teleport was recording sessions at the proxy.
• Teleport was completely idle.
• DynamoDB was the backend in us-east-1.
• Auth, Proxy, and Node were running on separate servers connected to each other the network of a cloud provider.
• Connections were made to sshd from ssh.

$ time for i in {1..10}; do ssh node.example.com date; done
Fri May  4 22:00:09 UTC 2018
Fri May  4 22:00:12 UTC 2018
Fri May  4 22:00:15 UTC 2018
Fri May  4 22:00:17 UTC 2018
Fri May  4 22:00:20 UTC 2018
Fri May  4 22:00:22 UTC 2018
Fri May  4 22:00:25 UTC 2018
Fri May  4 22:00:27 UTC 2018
Fri May  4 22:00:30 UTC 2018
Fri May  4 22:00:32 UTC 2018

real	0m25.627s
user	0m0.120s
sys	0m0.008s

Averaging out 10 runs for ssh node.example.com date took 2.56 seconds.

[russjones]

Exact same test on a Teleport 2.4.7 cluster.

$ time for i in {1..10}; do ssh node.example.com date; done
Fri May  4 22:08:00 UTC 2018
Fri May  4 22:08:03 UTC 2018
Fri May  4 22:08:07 UTC 2018
Fri May  4 22:08:10 UTC 2018
Fri May  4 22:08:14 UTC 2018
Fri May  4 22:08:17 UTC 2018
Fri May  4 22:08:20 UTC 2018
Fri May  4 22:08:23 UTC 2018
Fri May  4 22:08:27 UTC 2018
Fri May  4 22:08:30 UTC 2018

real	0m33.307s
user	0m0.168s
sys	0m0.036s

Averaging out 10 runs for ssh node.example.com date took 3.33 seconds.

[russjones]

Exact same test on a Teleport 2.6.0-beta.1 cluster.

$ time for i in {1..10}; do ssh node.example.com date; done
Fri May  4 23:12:50 UTC 2018
Fri May  4 23:12:52 UTC 2018
Fri May  4 23:12:54 UTC 2018
Fri May  4 23:12:57 UTC 2018
Fri May  4 23:12:59 UTC 2018
Fri May  4 23:13:01 UTC 2018
Fri May  4 23:13:04 UTC 2018
Fri May  4 23:13:06 UTC 2018
Fri May  4 23:13:08 UTC 2018
Fri May  4 23:13:10 UTC 2018

real	0m23.224s
user	0m0.140s
sys	0m0.032s

Averaging out 10 runs for ssh node.example.com date took 2.32 seconds.