Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sliding window throttling implementation #511

Merged
merged 1 commit into from
Feb 19, 2018
Merged

Conversation

onorua
Copy link
Contributor

@onorua onorua commented Feb 18, 2018

This is more advanced throttling mechanism based on #410

There is a window of some size, if interval is not set it defaults to 1 second, otherwise it got parsed by similar to GNU/sleep parser.
This interval is window pane, we slide one pane at interval, and calculate average over entire window. We also have a slow start, to prevent overflowing at the beginning of the window.
Basically rate of 100 with window of 5 with interval set to 1m is different than rate of 3 with window 300 and interval of 1s, because in first example it is possible to get all 100 messages in first second, while second one, will accept only 3 and wait for the next second to accept the rest.

We have this working in production for more than 3 months, and everything is working as expected.

P.S. I'm sorry that this took so long, I had problems with my spine, and was waiting for standing desk at the office :(

@onorua onorua mentioned this pull request Feb 18, 2018
@onorua onorua force-pushed the ratelimit branch 2 times, most recently from 73958a1 to f529e47 Compare February 18, 2018 14:58
@edsiper
Copy link
Member

edsiper commented Feb 19, 2018

@onorua

thanks for the PR.

I did a quick non-tech review and I have two comments:

  • there are two strtod.[ch] files which are GNU licensed. Since this project is Apache license we cannot link them together. We need to get a similar function under a different license (apache, MIT).

  • the source code have "Anchorfree" copyright. In order to make future changes we need this follow the other project files with "Treasure Data" name. The goal is to have just one copyright to make the transition smoothly to "Fluent Bit Authors" or "Cloud Native Computing Foundation". You can add a MAINTAINERS file inside the filter repo where you mention that Anchorfree has contributed and maintain current plugin.

@onorua
Copy link
Contributor Author

onorua commented Feb 19, 2018

Removed Anchorfree mentioning from source code. Not sure about License, isn't Apache 2.0 compatible with GNU v3, at least Wikipedia told me that. Do you want me to create similar function under Apache license or to remove code completely?

@edsiper
Copy link
Member

edsiper commented Feb 19, 2018

they are compatible if that code base lives in a separated shared library, but they cannot live together in one binary. Better to get rid of any GPL code in our code base,

btw, why not to use direct strtod() provided by stdlib.h ? that should be fine

Based on sliding window algorithm
@onorua
Copy link
Contributor Author

onorua commented Feb 19, 2018

Indeed, I've reworked it to use standard strtod.

@edsiper edsiper merged commit 33544ae into fluent:master Feb 19, 2018
@edsiper
Copy link
Member

edsiper commented Feb 19, 2018

thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants