Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Null route can cause high latency when connecting to hosts #8866

Closed
Tracked by #8745
russjones opened this issue Nov 5, 2021 · 1 comment · Fixed by #9254
Closed
Tracked by #8745

Null route can cause high latency when connecting to hosts #8866

russjones opened this issue Nov 5, 2021 · 1 comment · Fixed by #9254
Assignees
@rosstimothy
Labels
bug c-ju Internal Customer Reference robustness Resistance to crashes and reliability

Comments

@russjones
Copy link
Contributor

If you add a null route using ip route add ... to auth on a node then attempt to tsh ssh into that node, it will take a very long time to run (about 6 minutes). The following appears in the logs indicating that session establishment may be hung on attempting to create an active session.

Failed to create new session: Post "https://teleport.cluster.local/v2/namespaces/default/sessions":
   dial tcp w.x.y.z:3025: connect: connection timed out
@russjones russjones added bug c-ju Internal Customer Reference robustness Resistance to crashes and reliability labels Nov 5, 2021
@russjones russjones mentioned this issue Nov 5, 2021
Open
14 tasks
@rosstimothy
Copy link
Contributor

Neither ip route add -host $auth_ip reject or ip route add -host $auth_ip blackhole cause latency issues with tsh ssh. I was only able to reproduce this with ip route add $auth_ip dev lo.

rosstimothy added a commit that referenced this issue Dec 6, 2021
@rosstimothy
Prevent infinite dialing of Auth
b7f40c8 
NewHTTPClient was not calling CheckAndSetDefault on the provided client.Config which
allowed the DialTimeout to be 0. This allowed dialing auth to hang forever if
auth is unreachable.
Fixes #8866
@rosstimothy rosstimothy mentioned this issue Dec 6, 2021
Merged
rosstimothy added a commit that referenced this issue Dec 8, 2021
@rosstimothy
Prevent infinite dialing of Auth
ac3b751 
NewHTTPClient was not calling CheckAndSetDefault on the provided client.Config which
allowed the DialTimeout to be 0. This allowed dialing auth to hang forever if
auth is unreachable.
Fixes #8866
rosstimothy added a commit that referenced this issue Dec 8, 2021
@rosstimothy
Prevent infinite dialing to Auth (#9254)
3fcbe17 
NewHTTPClient was not calling CheckAndSetDefault on the provided client.Config which
allowed the DialTimeout to be 0. This allowed dialing auth to hang forever if
auth is unreachable.
Fixes #8866
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@rosstimothy rosstimothy

Labels
bug c-ju Internal Customer Reference robustness Resistance to crashes and reliability
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Prevent infinite dialing to Auth gravitational/teleport
3 participants
@russjones @fspmarshall @rosstimothy