-
Notifications
You must be signed in to change notification settings - Fork 352
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
net/redisclient: fix nil pointer dereference #2939
Conversation
Fix use of logger before initialization. Followup on #2261 Signed-off-by: Alexander Yastrebov <alexander.yastrebov@zalando.de>
if ro.Log == nil { | ||
ro.Log = &logging.DefaultLog{} | ||
} | ||
|
||
if ro.AddrUpdater != nil { | ||
address, err := ro.AddrUpdater() | ||
for i := 0; i < retryCount; i++ { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we should remove this probe from the client - it simply delays creation of the client and client will recover after redis is up. Also client provides RingAvailable method that could be used to check if ring is up.
We also check AddrUpdater once at startup
Lines 1717 to 1730 in 37c474a
_, err = redisOptions.AddrUpdater() | |
if err != nil { | |
log.Errorf("Failed to update redis addresses from kubernetes: %v", err) | |
return err | |
} | |
} else if redisOptions != nil && o.SwarmRedisEndpointsRemoteURL != "" { | |
log.Infof("Use remote address %s to fetch updates redis shards", o.SwarmRedisEndpointsRemoteURL) | |
redisOptions.AddrUpdater = getRemoteURLRedisAddrUpdater(o.SwarmRedisEndpointsRemoteURL) | |
_, err = redisOptions.AddrUpdater() | |
if err != nil { | |
log.Errorf("Failed to update redis addresses from URL: %v", err) | |
return err | |
} |
which might be a bigger problem - imagine permanent redis failure prevents new pods startup leading to cascading failure. I think its better to fail open/close with ratelimits then fail startup when redis is not avaialble.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I get your point but 10 sec extra startup time is not that much and it will take longer with container restart. Maybe we don't need to fail startup but it's still good to have the retries, wdyt?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
which might be a bigger problem - imagine permanent redis failure prevents new pods startup leading to cascading failure. I think its better to fail open/close with ratelimits then fail startup when redis is not avaialble.
I disagree here and you should page if this is a permanent issue. A new bad deployment will block and you get a page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I get your point but 10 sec extra startup time is not that much and it will take longer with container restart.
Check the code, its just a delay, it only logs an error.
👍 |
1 similar comment
👍 |
Fix use of logger before initialization.
Followup on #2261