feat(output): add mongodb output plugin #9923

bustedware · 2021-10-13T21:48:18Z

Updated associated README.md.
Wrote appropriate unit tests.
Pull request title or commits are in conventional commit format

Added an output plugin to gather metrics and output them to mongodb v5.0+ time series collections

…g. updated README

telegraf-tiger · 2021-10-13T21:55:31Z

Thanks so much for the pull request!
🤝 ✒️ Just a reminder that the CLA has not yet been signed, and we'll need it before merging. Please sign the CLA when you get a chance, then post a comment here saying !signed-cla

bustedware · 2021-10-13T21:57:16Z

!signed-cla

srebhan

Hey @bustedware, thanks for this nice PR!
Please check if you can implement this plugin using the go.mongodb.org/mongo-driver driver instead of using the unmaintained library used now. I have some more comments in the code...

go.mod

plugins/outputs/mongodb/README.md

plugins/outputs/mongodb/mongodb.go

plugins/outputs/mongodb/mongodb_test.go

plugins/outputs/mongodb/mongodb.go

Co-authored-by: Sven Rebhan <36194019+srebhan@users.noreply.github.com>

Co-authored-by: Thomas Casteleyn <thomas.casteleyn@me.com>

… member variable. fmt statements used previouslly for logging/debugging are now using log instead

plugins/outputs/mongodb/mongodb.go

…e plugin interface. refactored some additional variable names and modified README.md to reflect those names.

plugins/outputs/mongodb/README.md

plugins/outputs/mongodb/mongodb_test.go

go.mod

go.sum

…. using testutil for test metrics in mongodb_test output test.

- MongoDBGetCollections is now member function - removed TODO from Init() - ctx now using Background() from Connect() method

- no longer storing context in struct - removed errorneous comments and TODOs

…und MongoDBCreateTimeSeriesCollection

plugins/outputs/mongodb/README.md

plugins/outputs/mongodb/mongodb.go

srebhan · 2021-10-18T14:12:57Z

plugins/outputs/mongodb/mongodb.go

+	CAFile              string          `toml:"cafile"`
+	X509clientpem       string          `toml:"x509clientpem"`
+	X509clientpempwd    string          `toml:"x509clientpempwd"`
+	Allow_tls_insecure  bool            `toml:"allow_tls_insecure"`


Maybe you should add this option to TLSConfig then for other plugins to profit as well. Do you think this can be done?

srebhan

@bustedware, we are getting closer. Some more comments in the code. Please try to export as less as possible. Looking forward to the next round!

plugins/outputs/mongodb/mongodb.go

srebhan · 2021-10-18T14:31:11Z

plugins/outputs/mongodb/mongodb_test.go

+	// create time series collection when it doesn't exist
+	myTestMetricName := "testMetricName"
+	if !s.DoesCollectionExist(myTestMetricName) {
+		err = s.MongoDBCreateTimeSeriesCollection(myTestMetricName)
+		require.NoError(t, err)
+	}


Shouldn't this be done by Write automatically?

This is a different test, should be in its own func.

Correct. Write does this automatically. I have added a lot more tests. Please let me know if the tests span enough cases.

note that MongoDBCreateTimeSeriesCollection is no longer exported and is now named createTimeSeriesCollection. I think leaving Write to perform the testing of this specific functionality should be enough.

plugins/outputs/mongodb/mongodb_test.go

…Dockerfile and mongodb.sh in mongodb output plugin

assign client only after checking err status. Co-authored-by: Sven Rebhan <36194019+srebhan@users.noreply.github.com>

Hipska

Great improvements, 2 small comments!

plugins/outputs/mongodb/mongodb.go

Hipska · 2021-10-26T07:34:55Z

plugins/outputs/mongodb/mongodb.go

+		serverSelectionTimeoutSeconds := int64(s.ServerSelectTimeout / 1000000000)
+		s.clientOptions = s.clientOptions.SetServerSelectionTimeout(time.Duration(serverSelectionTimeoutSeconds) * time.Second)


Why is this conversion? s.ServerSelectTimeout is already a Duration type.

invalid operation: mismatched types config.Duration and time.Duration - compiler MismatchedTypes. when I try:
s.clientOptions = s.clientOptions.SetServerSelectionTimeout(s.ServerSelectTimeout * time.Second)

check slack thread for more details

fixed misleading error message regarding a missing username from configuration Co-authored-by: Thomas Casteleyn <thomas.casteleyn@me.com>

srebhan

Hey @bustedware you nicely worked this trough with @Hipska. Good job! I only have some minor comments, the most severe is the handling of the timeout which already is a duration.

Can you also elaborate on what you use that PKI (certificates) for?

plugins/outputs/mongodb/mongodb.go

plugins/outputs/mongodb/mongodb_test.go

error checking added to ensure name of collection returned is of type string to prevent panic. default value added for database option default value added for granularity option updated to use switch statement for metric granularity value checks using HasPrefix to be explicit instead of Contains() for Dsn checks updated to use err within local/in scope contexts of method calls removed superfilous lines from NoAuth test for brevity and cleanliness Co-authored-by: Sven Rebhan <36194019+srebhan@users.noreply.github.com>

…y now defaulted. removed return statement which will always be nil. golang fixed space formatting from previous commit suggestions

…ranularity setting. fixed mongodb.go to actually set the default values and not set the sample config. updated mongodb_test.go to pass new short test cases where config no longer requires database or granularity field to pass init

srebhan

This leaves us with the timeout/duration setting. :-)

plugins/outputs/mongodb/mongodb.go

verified and applied suggestion for invoking SetServerSelectionTimeout Co-authored-by: Sven Rebhan <36194019+srebhan@users.noreply.github.com>

…Seconds variable which no longer exists

srebhan

Looks good to me. Thanks @bustedware for the nice PR and the constructive discussions!

plugins/outputs/mongodb/mongodb.go

…of config.Duration

telegraf-tiger · 2021-10-28T00:53:09Z

🥳 This pull request decreases the Telegraf binary size by -1.09 % for linux amd64 (new size: 130.7 MB, nightly size 132.1 MB)

📦 Looks like new artifacts were built from this PR.

Expand this list to get them here! 🐯

Artifact URLs

DEB	RPM	TAR GZ	ZIP
amd64.deb	aarch64.rpm	darwin_amd64.tar.gz	windows_amd64.zip
arm64.deb	armel.rpm	freebsd_amd64.tar.gz	windows_i386.zip
armel.deb	armv6hl.rpm	freebsd_armv7.tar.gz
armhf.deb	i386.rpm	freebsd_i386.tar.gz
i386.deb	ppc64le.rpm	linux_amd64.tar.gz
mips.deb	s390x.rpm	linux_arm64.tar.gz
mipsel.deb	x86_64.rpm	linux_armel.tar.gz
ppc64el.deb		linux_armhf.tar.gz
s390x.deb		linux_i386.tar.gz
		linux_mips.tar.gz
		linux_mipsel.tar.gz
		linux_ppc64le.tar.gz
		linux_s390x.tar.gz
		static_linux_amd64.tar.gz

Hipska

Perfect! Nice collaboration 👏

bustedware and others added 4 commits October 13, 2021 06:46

initial commit for monogdb output plugin

2bd13fb

Merge branch 'influxdata:master' into master

6b9f8de

initial commit for mongodb output unit test. added more error checkin…

1362bba

…g. updated README

updated go.mod and go.sum files

4e79116

bustedware added 2 commits October 13, 2021 20:05

fixed formatting for mongodb output plugin

2465282

added license for gopkg.in/mgo.v2

ee193be

srebhan requested changes Oct 14, 2021

View reviewed changes

srebhan self-assigned this Oct 14, 2021

srebhan added area/mongodb new plugin plugin/output 1. Request for new output plugins 2. Issues/PRs that are related to out plugins labels Oct 14, 2021

Hipska suggested changes Oct 14, 2021

View reviewed changes

plugins/outputs/mongodb/mongodb.go Outdated Show resolved Hide resolved

bustedware and others added 3 commits October 14, 2021 07:15

formatting README.md to highlight version requirements

49761b2

Co-authored-by: Sven Rebhan <36194019+srebhan@users.noreply.github.com>

update to variable names for consistency

0a7c486

Co-authored-by: Thomas Casteleyn <thomas.casteleyn@me.com>

updated to reference new variable names. removed unused serialization…

efb7fb9

… member variable. fmt statements used previouslly for logging/debugging are now using log instead

Hipska reviewed Oct 14, 2021

View reviewed changes

plugins/outputs/mongodb/mongodb.go Outdated Show resolved Hide resolved

removed unmaintained library from project. updated mongodb_test to us…

f8fa611

…e plugin interface. refactored some additional variable names and modified README.md to reflect those names.

Hipska suggested changes Oct 15, 2021

View reviewed changes

plugins/outputs/mongodb/README.md Outdated Show resolved Hide resolved

plugins/outputs/mongodb/mongodb_test.go Outdated Show resolved Hide resolved

Hipska suggested changes Oct 15, 2021

View reviewed changes

go.mod Outdated Show resolved Hide resolved

go.sum Outdated Show resolved Hide resolved

bustedware and others added 6 commits October 15, 2021 18:53

Merge branch 'influxdata:master' into master

5eb9553

updated go.mod and go.sum. removed data_format from config and readme…

0c4e4f6

…. using testutil for test metrics in mongodb_test output test.

fixed make check by using primitive.E types

708616e

- added clientOptions to struct and configured within Init()

4aa3294

- MongoDBGetCollections is now member function - removed TODO from Init() - ctx now using Background() from Connect() method

- use proper logging facility

0929edd

- no longer storing context in struct - removed errorneous comments and TODOs

modified variable names to pass lint checks. added an error check aro…

ad4011b

…und MongoDBCreateTimeSeriesCollection

srebhan reviewed Oct 18, 2021

View reviewed changes

srebhan requested changes Oct 18, 2021

View reviewed changes

Hipska self-assigned this Oct 18, 2021

bustedware and others added 4 commits October 25, 2021 17:17

resolved merge conflict with config.go. added new line characters to …

c06a21e

…Dockerfile and mongodb.sh in mongodb output plugin

executed make tidy to resolve go.mod and go.sum conflicts

48d6ad7

appears that github.com/aws/aws-sdk-go is no longer used

0bb78e9

Update plugins/outputs/mongodb/mongodb.go

160970d

assign client only after checking err status. Co-authored-by: Sven Rebhan <36194019+srebhan@users.noreply.github.com>

bustedware requested review from Hipska and srebhan October 25, 2021 23:07

Hipska suggested changes Oct 26, 2021

View reviewed changes

Update plugins/outputs/mongodb/mongodb.go

d92e298

fixed misleading error message regarding a missing username from configuration Co-authored-by: Thomas Casteleyn <thomas.casteleyn@me.com>

srebhan reviewed Oct 26, 2021

View reviewed changes

bustedware and others added 4 commits October 26, 2021 21:16

Merge branch 'influxdata:master' into master

2cae811

finish apply remaining open suggestions. both database and granularit…

4445eb6

…y now defaulted. removed return statement which will always be nil. golang fixed space formatting from previous commit suggestions

Hipska approved these changes Oct 27, 2021

View reviewed changes

srebhan reviewed Oct 27, 2021

View reviewed changes

plugins/outputs/mongodb/mongodb.go Outdated Show resolved Hide resolved

Update plugins/outputs/mongodb/mongodb.go

c655015

verified and applied suggestion for invoking SetServerSelectionTimeout Co-authored-by: Sven Rebhan <36194019+srebhan@users.noreply.github.com>

bustedware requested a review from srebhan October 27, 2021 10:36

update to use s.ServerSelectTimeout instead of serverSelectionTimeout…

02152ce

…Seconds variable which no longer exists

srebhan approved these changes Oct 27, 2021

View reviewed changes

srebhan added the ready for final review This pull request has been reviewed and/or tested by multiple users and is ready for a final review. label Oct 27, 2021

Hipska suggested changes Oct 27, 2021

View reviewed changes

plugins/outputs/mongodb/mongodb.go Outdated Show resolved Hide resolved

plugins/outputs/mongodb/mongodb.go Outdated Show resolved Hide resolved

removed assignment back to s.clientOptions. updated for better usage …

ab60455

…of config.Duration

bustedware requested a review from Hipska October 28, 2021 00:58

srebhan approved these changes Oct 28, 2021

View reviewed changes

Hipska approved these changes Oct 28, 2021

View reviewed changes

powersj merged commit 7d6672c into influxdata:master Oct 28, 2021

srebhan mentioned this pull request Nov 5, 2021

MongoDB output plugin #8556

Closed

3 tasks

VladislavSenkevich pushed a commit to gwos/telegraf that referenced this pull request Nov 23, 2021

feat: add mongodb output plugin (influxdata#9923)

5df26c8

bustedware mentioned this pull request Apr 11, 2022

feat: added support for multi pem block files for all plugins #10020

Closed

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(output): add mongodb output plugin #9923

feat(output): add mongodb output plugin #9923

bustedware commented Oct 13, 2021 •

edited

Loading

telegraf-tiger bot commented Oct 13, 2021

bustedware commented Oct 13, 2021

srebhan left a comment

srebhan Oct 18, 2021

srebhan left a comment

srebhan Oct 18, 2021

Hipska Oct 18, 2021

bustedware Oct 25, 2021

bustedware Oct 25, 2021

Hipska left a comment

Hipska Oct 26, 2021

bustedware Oct 26, 2021 •

edited

Loading

bustedware Oct 27, 2021

srebhan left a comment

srebhan left a comment

srebhan left a comment

telegraf-tiger bot commented Oct 28, 2021

Artifact URLs

Hipska left a comment

		serverSelectionTimeoutSeconds := int64(s.ServerSelectTimeout / 1000000000)
		s.clientOptions = s.clientOptions.SetServerSelectionTimeout(time.Duration(serverSelectionTimeoutSeconds) * time.Second)

feat(output): add mongodb output plugin #9923

feat(output): add mongodb output plugin #9923

Conversation

bustedware commented Oct 13, 2021 • edited Loading

telegraf-tiger bot commented Oct 13, 2021

bustedware commented Oct 13, 2021

srebhan left a comment

Choose a reason for hiding this comment

srebhan Oct 18, 2021

Choose a reason for hiding this comment

srebhan left a comment

Choose a reason for hiding this comment

srebhan Oct 18, 2021

Choose a reason for hiding this comment

Hipska Oct 18, 2021

Choose a reason for hiding this comment

bustedware Oct 25, 2021

Choose a reason for hiding this comment

bustedware Oct 25, 2021

Choose a reason for hiding this comment

Hipska left a comment

Choose a reason for hiding this comment

Hipska Oct 26, 2021

Choose a reason for hiding this comment

bustedware Oct 26, 2021 • edited Loading

Choose a reason for hiding this comment

bustedware Oct 27, 2021

Choose a reason for hiding this comment

srebhan left a comment

Choose a reason for hiding this comment

srebhan left a comment

Choose a reason for hiding this comment

srebhan left a comment

Choose a reason for hiding this comment

telegraf-tiger bot commented Oct 28, 2021

Artifact URLs

Hipska left a comment

Choose a reason for hiding this comment

bustedware commented Oct 13, 2021 •

edited

Loading

bustedware Oct 26, 2021 •

edited

Loading