feat(inputs.gcs): Google Cloud Storage Input Plugin

[gkatzioura]

This is a pull request for the issue #8412

Required for all PRs:

• Signed CLA.
• Associated README.md updated.
• Has appropriate unit tests.

How this Telegraf plugin works with Google Cloud Storage

Google Cloud Storage list files in lexicographic order. Files are stored with a date prefix, or their lexicographic order is based on the time the files were created. A GCS crawler will list the files on GCS in order to parse them and submit them to an output. After every Gather execution the last file processed will be stored on GCS as an offset. The next Gather command will continue from that offset.

How Gather works

• Entries on GCS are stored with a name that represents the time they got created ie. 1604148850991 or 2020-01-10T05:45:01
• Telegraf input plugin is configured to crawl data on a GCS bucket with a specified prefix.
• Input plugin can use an offset to list the data after that offset
• If there is no offset will start from beginning
• On every Gather() execution the Input plugin will save the last file it processed on GCS. The purpose is to use this file in order to retrieve the offset and to use it for the next iteration
• The files on GCS should contain measurements so that telegraph can parse them
• Due to a page of items fetched from GCS can be bigger than what Telegraph can process in one Gather() action user can specify a limit of files processed

[gkatzioura]

The error are related with the protobuf update and the ericchiang/k8s@v1.2.0/watch/versioned/generated.pb.go

Actually it is described here as an issue.
ericchiang/k8s#125

What could be the options?
Swapping the generated offending file?
Maybe an external plugin can be a solution (although this is a go pull request)

[gkatzioura]

I added those lines on a circleci build to highlight the dependency problem (on purpose to highlight the issue, don't want to change the build)

telegraf/.circleci/config.yml

Lines 31 to 33 in 98d2403

	- run: chmod 777 /go/pkg/mod/github.com/ericchiang/k8s@v1.2.0/watch/versioned/generated.pb.go
	- run: sed 's/github.com\/ericchiang.k8s.watch.versioned.Event/k8s.io.kubernetes.pkg.watch.versioned.Event/g' /go/pkg/mod/github.com/ericchiang/k8s@v1.2.0/watch/versioned/generated.pb.go > newfile.go
	- run: mv newfile.go /go/pkg/mod/github.com/ericchiang/k8s@v1.2.0/watch/versioned/generated.pb.go

A google sdk upgrade due to the need of the new protobuf package will have this problem, due to that line ericchiang/k8s#125

[sspaink]

@gkatzioura sorry no one has responded to you yet, are you still interested in working on this plugin? We recently moved away from ericchiang/k8s to kubernetes/client-go maybe the problem has been resolved in the newer repo? Creating an external plugin as you suggested might be a good option as well.

[gkatzioura]

Hi @sspaink More than happy! Will spend some time to get back on track.

[gkatzioura]

Hi @sspaink, seems the issues on the build are resolved. Will do some extra testing.
I think it's good to go for a review.

[sspaink]

!signed-cla

[sspaink]

Thank you for your continued worked on this plugin, you will also need to add a README.md that contains the sample configuration, a description of the plugin, how to use it, etc. The more detail the better.

[gkatzioura]

Hi @sspaink . Did the changes, some renaming and registering the plugin among the other input plugins.

[sspaink]

@gkatzioura thank you for your continued work on this. Recently the Telegraf project changed how the sample configs are maintained, instead of duplicating it in the README.md and the code it is now generated from a single source of truth. I added some suggested changes to help migrate your plugin to use the new structure, hopefully that will make it easy. You will also need to add a file called "sample.conf" that contains the sample configuration.

The input plugin guidelines were updated as well if that makes it more clear: https://github.com/influxdata/telegraf/blob/master/docs/INPUTS.md#input-plugin-guidelines

[sspaink]

@gkatzioura do you have time to resolve the conflicts and address the sample config change? No worries if you don't, I wouldn't mind creating a separate pull request to resolve the issues.

[gkatzioura]

Hi @sspaink checking!

[telegraf-tiger]

Download PR build artifacts for linux_amd64.tar.gz, darwin_amd64.tar.gz, and windows_amd64.zip.
Downloads for additional architectures and packages are available below.

⚠️ This pull request increases the Telegraf binary size by 1.01 % for linux amd64 (new size: 154.1 MB, nightly size 152.6 MB)

📦 Click here to get additional PR build artifacts

Artifact URLs

DEB	RPM	TAR GZ	ZIP
amd64.deb	aarch64.rpm	darwin_amd64.tar.gz	windows_amd64.zip
arm64.deb	armel.rpm	darwin_arm64.tar.gz	windows_i386.zip
armel.deb	armv6hl.rpm	freebsd_amd64.tar.gz
armhf.deb	i386.rpm	freebsd_armv7.tar.gz
i386.deb	ppc64le.rpm	freebsd_i386.tar.gz
mips.deb	riscv64.rpm	linux_amd64.tar.gz
mipsel.deb	s390x.rpm	linux_arm64.tar.gz
ppc64el.deb	x86_64.rpm	linux_armel.tar.gz
riscv64.deb		linux_armhf.tar.gz
s390x.deb		linux_i386.tar.gz
		linux_mips.tar.gz
		linux_mipsel.tar.gz
		linux_ppc64le.tar.gz
		linux_riscv64.tar.gz
		linux_s390x.tar.gz
		static_linux_amd64.tar.gz

[gkatzioura]

@sspaink all green