Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(parsers): Memory leak for plugins using ParserFunc. #11815

Merged
merged 3 commits into from
Sep 16, 2022

Conversation

srebhan
Copy link
Member

@srebhan srebhan commented Sep 15, 2022

resolves #11809

When input plugins (or processors) use the passed ParserFunc, the new parser gets added to the list of running parsers in config.Parsers. As this instantiation of parsers can happen often and is out of the control of Telegraf's general framework, the array of running parsers will keep growing and is newer cleaned up. This means that a reference to the no-longer-used parsers is kept, garbage collection cannot free the parsers' space and we slowly eat up all memory.

The PR completely removes tracking the running parsers and thus solves the memory leak. However, once we want to go to dynamic plugin starting/stopping this problem will hit us for other categories as well as there is currently no way to remove running plugin instances from the corresponding lists.

@telegraf-tiger telegraf-tiger bot added the fix pr to fix corresponding bug label Sep 15, 2022
@srebhan srebhan mentioned this pull request Sep 15, 2022
@sspaink
Copy link
Contributor

sspaink commented Sep 15, 2022

@srebhan why was Telegraf keeping track of the running parsers?

@srebhan
Copy link
Member Author

srebhan commented Sep 15, 2022

@sspaink when moving to the more plugin-like structure we used it to initialize the parsers just as we do with the other plugins, but I don't think it is necessary.

@telegraf-tiger
Copy link
Contributor

@srebhan srebhan requested a review from reimda September 16, 2022 10:40
@reimda reimda merged commit 3b3584b into influxdata:master Sep 16, 2022
@srebhan srebhan added regression something that used to work, but is now broken plugin/parser 1. Request for new parser plugins 2. Issues/PRs that are related to parser plugins labels Sep 16, 2022
reimda pushed a commit that referenced this pull request Sep 19, 2022
dba-leshop pushed a commit to dba-leshop/telegraf that referenced this pull request Oct 30, 2022
@srebhan srebhan deleted the issue_11809 branch November 7, 2022 11:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
fix pr to fix corresponding bug plugin/parser 1. Request for new parser plugins 2. Issues/PRs that are related to parser plugins regression something that used to work, but is now broken
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Memory Leak in 1.24.0
4 participants