Watcher could implement either keep alive or retry mechanism

[petermicuch]

Similar to issue mentioned here for python client, it would be really nice if Watcher does not drop connection after 30 minutes. Right now I am handling OnClose and OnError events and recreate Watcher after short period of time since connection was closed. Even better would be if watch supports Observable and one could use Rx to handle changes.

[brendandburns]

We need to build a higher level "Informer" on top of the Watcher to handle reconnecting after a watch stops.

In general, you can't rely on a watch lasting forever. You have to List/Watch in a loop, but we should build a utility to make that easy for people to do correctly.

[petermicuch]

@brendandburns does it mean this will be considered? Loop is exactly what I am doing now, so in case this would be already build in, it would be great. Use-cases for watch are usually long running tasks in my opinion. Observable would be even better as you can trigger delayed processing, buffering or throttling of events.

[brendandburns]

We definitely should build up the Informer code. There are some starting points here:

#394

But it got hung up in the design phase.

[tomaszgawlik]

@petermicuch Would you be able to share your look example please?
I'm trying to write an application, which is watching kubernetes events and sends them to Grafana.
But having troubles with restarting the stream after it's closed.

Edit, that's what I'm testing now:

            var response = await _kubernetesClient.ListEventForAllNamespacesWithHttpMessagesAsync(watch: true);
            while (true)
            {
                _logger.LogInformation($"Starting new watch stream");
                var streamTokenSource = new CancellationTokenSource();
                
                var watch = response.Watch<V1Event, V1EventList>((type, @event) =>
                    {
                        _logger.LogInformation($"Event received action here..");
                    }, (error) => { _logger.LogError(error, $"Watch error"); },
                    () =>
                    {
                        _logger.LogInformation("Event watch onClosed triggered");
                        streamTokenSource.Cancel();
                    });

                await Task.Delay(-1, streamTokenSource.Token);
            }

[petermicuch]

You have to call _kubernetesClient.ListEventForAllNamespacesWithHttpMessagesAsync(watch: true); inside while loop and not reuse the stream, then it will work. Also why do you await to get a response? That is responsibility of Watcher you are getting from watch call.

            while (true)
            {
                _logger.LogInformation($"Starting new watch stream");
                var streamTokenSource = new CancellationTokenSource();
                
                var response = _kubernetesClient.ListEventForAllNamespacesWithHttpMessagesAsync(watch: true);
                var watch = response.Watch<V1Event, V1EventList>((type, @event) =>
                    {
                        _logger.LogInformation($"Event received action here..");
                    }, (error) => { _logger.LogError(error, $"Watch error"); },
                    () =>
                    {
                        _logger.LogInformation("Event watch onClosed triggered");
                        streamTokenSource.Cancel();
                    });

                await Task.Delay(-1, streamTokenSource.Token);
                watch.Dispose();
            }

I have not compiled or tried this modified code, but should work. Also consider retrying on error with some delay. Would be nice if there is some kind of resilience implemented.

If you want to know more, just check code of Watcher itself.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[tg123]

close and move discuss to #533
workaround here
#533 (comment)