Skip to content
This repository has been archived by the owner on Mar 9, 2022. It is now read-only.

Support for parallel filter instances #55

Closed
oplehto opened this issue Mar 22, 2018 · 0 comments
Closed

Support for parallel filter instances #55

oplehto opened this issue Mar 22, 2018 · 0 comments

Comments

@oplehto
Copy link
Contributor

oplehto commented Mar 22, 2018

Right now the filter is a bit of a SPOF, although any disruption should be fairly minor due to auto-recovery mechanisms in the service management level (supervisord, k8s, systemd) and NATS buffering. At some point we will also start hitting the limits of how far we can scale the filter within the confines of a single physical server which is a slightly bigger concern.

To make things more scalable and redundant, we could run multiple filter instances in parallel and have the incoming data distributed across them. In practice I think it may be as simple as using QueueSubscribe (https://github.com/nats-io/go-nats#queue-groups) instead of Subscribe to subscribe to subjects in the filter.

@oplehto oplehto changed the title Support for parallel filters Support for parallel filter instances Mar 22, 2018
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant