Under high load, haproxy reloads using incomplete config files

[matthewpucc]

Randomly, we will see an haproxy ingress start returning the default backend for all requests. We see this the most when the controllers are under heavy load. We are using the default reloader.

I think this is the same issue @westse is investigating with PR #123.

[matthewpucc]

There are additional details here.

[matthewpucc]

I didn't see a central issue tracking these efforts. Since more than one person is experiencing this issue, I wanted to change that.

[jcmoraisjr]

Hi, so you are using the reload-strategy=native (or didn't change this as it is the default), right? Did you change the rate-limit?

What do you mean with controllers under heavy load - is the ingress controller managing a big amount of updates and reloading HAProxy or the HAProxy process with a big amount of requests?

About incomplete config files - do you mean stale config as of #124 or do you mean that, for an unkown reason, a valid config file was built missing backends that wasn't removed from ingress resources?

[matthewpucc]

We use reload-strategy=native (unset and as the default). We have not changed the rate-limit.

By under load, I am talking about the amount of requests. An average day will see between 500 to 1000 rps per pod.

Our haproxy config file doesn't change as often as @westse describes in that ticket. The symptoms we see are the pods will randomly start returning our default backend for every request to the pod.

We have not pushed the change to keep rotated config files out to production yet.

[jcmoraisjr]

Hi, I don't have a clear idea yet on how to reproduce this. A few more questions and suggestions:

• How much time HAProxy send requests to the default backend? eg for a few seconds? until the next ingress resource change? until the whole controller restart?
• What tag are you using?
• If using v0.4 try v0.5-beta.3 which fixes ssl certificate "<namespace/secret>" does not exist in local store #78 that may cause these same symptoms
• Rotated config may be a good idea - you can diff them in order to see what changed and this will help us to find where is the root cause of the config failure
• What the log says just before and during the config failure?

[matthewpucc]

Sorry for late reply.

• It's usually for a few seconds and it self corrects
• I think we are on the 0.2 tag
• Will upgrade to 0.5-beta.3 asap to see if that corrects the issue
• Will try to cherry pick this into beta
• TBD. I'll have the team pull logs next time this happens.

[jipperinbham]

we're seeing similar issues with traffic being routed to the default backend for seemingly no reason. Our ingress controller isn't logging anything during the time period of the default backend being selected so I don't suspect a config reload.

[matthewpucc]

ok. Been running 0.5-beta3 for about a week now. It has had no impact on the situation.

[matthewpucc]

When I pulled logs, we are seeing reloads about every 10-15 seconds.

[jcmoraisjr]

Hey there, tons of synchronization improvements were made since the v0.5 days. Maybe new versions fixed the issues you're experiencing, otherwise please fill a new issue describing version and as much info as possible. Closing this one.