Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add OpenTSDB data format parser (mark two) #13167

Merged
merged 9 commits into from
May 5, 2023
Merged

feat: add OpenTSDB data format parser (mark two) #13167

merged 9 commits into from
May 5, 2023

Conversation

nick-kentik
Copy link
Contributor

@nick-kentik nick-kentik commented Apr 28, 2023

Required for all PRs

resolves #8559

Added a parser for OpenTSDB's Telnet style put API format.
For more detail of the format, see

http://opentsdb.net/docs/build/html/api_telnet/put.html
http://opentsdb.net/docs/build/html/user_guide/writing/index.html#data-specification

This is heavily based on the effort by @envzhu in #9690 - I merged that branch against latest master, then made the (minimal) changes necessary to get it working, and the existing tests passing.

A quick end-to-end test also looked good. Here's the telegraf.conf I used:

[[outputs.file]]
files = ["stdout"]
data_format = "prometheus"

[[inputs.socket_listener]]
service_address = "tcp://127.0.0.1:4242"
data_format = "opentsdb"

A telnet session as input:

telnet localhost 4242
Trying ::1...
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
version
put sys.if.bytes.out 1479496100 1.3E3 host=web01 interface=eth0
^]
telnet> quit
Connection closed.

And the telegraf process output:

./telegraf --config telegraf.conf 
2023-05-05T09:49:26Z I! Loading config: telegraf.conf
2023-05-05T09:49:26Z I! Starting Telegraf 1.27.0-e36d3a8c
2023-05-05T09:49:26Z I! Available plugins: 235 inputs, 9 aggregators, 28 processors, 23 parsers, 57 outputs, 3 secret-stores
2023-05-05T09:49:26Z I! Loaded inputs: socket_listener
2023-05-05T09:49:26Z I! Loaded aggregators: 
2023-05-05T09:49:26Z I! Loaded processors: 
2023-05-05T09:49:26Z I! Loaded secretstores: 
2023-05-05T09:49:26Z I! Loaded outputs: file
2023-05-05T09:49:26Z I! Tags enabled: host=starbook
2023-05-05T09:49:26Z I! [agent] Config: Interval:10s, Quiet:false, Hostname:"starbook", Flush Interval:10s
2023-05-05T09:49:26Z I! [inputs.socket_listener] Listening on tcp://127.0.0.1:4242
2023-05-05T09:49:38Z E! [parsers.opentsdb::socket_listener] Error parsing "version" as opentsdb: doesn't have required fields
# HELP sys_if_bytes_out_value Telegraf collected metric
# TYPE sys_if_bytes_out_value untyped
sys_if_bytes_out_value{host="web01",interface="eth0"} 1300

@telegraf-tiger
Copy link
Contributor

Thanks so much for the pull request!
🤝 ✒️ Just a reminder that the CLA has not yet been signed, and we'll need it before merging. Please sign the CLA when you get a chance, then post a comment here saying !signed-cla

@telegraf-tiger telegraf-tiger bot added the feat Improvement on an existing feature such as adding a new setting/mode to an existing plugin label Apr 28, 2023
@nick-kentik nick-kentik marked this pull request as draft April 28, 2023 16:11
@nick-kentik
Copy link
Contributor Author

I'll need to get advice on the CLA; I'll leave this as draft until then.

@nick-kentik nick-kentik marked this pull request as ready for review May 2, 2023 09:24
@nick-kentik
Copy link
Contributor Author

!signed-cla

@nick-kentik
Copy link
Contributor Author

OK, I think this is ready for review now.

cc @powersj @Hipska @srebhan - and thanks again to @envzhu for doing the hard work on mk1 ^^

Any problems, just let me know; I'm motivated to get this merged.

@Hipska Hipska added the plugin/parser 1. Request for new parser plugins 2. Issues/PRs that are related to parser plugins label May 2, 2023
@Hipska Hipska added ready for final review This pull request has been reviewed and/or tested by multiple users and is ready for a final review. area/opentsdb labels May 2, 2023
Copy link
Member

@srebhan srebhan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the nice addition @nick-kentik and sorry for the long response time! I have some minor comments...

@srebhan srebhan self-assigned this May 2, 2023
@nick-kentik
Copy link
Contributor Author

Thanks for the review @srebhan ! I've addressed your comments, with a few tweaks in places; I've resolved the comments where I've applied your suggestion unchanged.

There's just the discussion about what to do when some lines in the input are invalid; I've not made any changes there yet.

@nick-kentik nick-kentik requested a review from srebhan May 4, 2023 16:41
@telegraf-tiger
Copy link
Contributor

telegraf-tiger bot commented May 5, 2023

Download PR build artifacts for linux_amd64.tar.gz, darwin_amd64.tar.gz, and windows_amd64.zip.
Downloads for additional architectures and packages are available below.

🥳 This pull request decreases the Telegraf binary size by -3.60 % for linux amd64 (new size: 168.8 MB, nightly size 175.1 MB)

📦 Click here to get additional PR build artifacts

Artifact URLs

DEB RPM TAR GZ ZIP
amd64.deb aarch64.rpm darwin_amd64.tar.gz windows_amd64.zip
arm64.deb armel.rpm darwin_arm64.tar.gz windows_arm64.zip
armel.deb armv6hl.rpm freebsd_amd64.tar.gz windows_i386.zip
armhf.deb i386.rpm freebsd_armv7.tar.gz
i386.deb ppc64le.rpm freebsd_i386.tar.gz
mips.deb riscv64.rpm linux_amd64.tar.gz
mipsel.deb s390x.rpm linux_arm64.tar.gz
ppc64el.deb x86_64.rpm linux_armel.tar.gz
riscv64.deb linux_armhf.tar.gz
s390x.deb linux_i386.tar.gz
linux_mips.tar.gz
linux_mipsel.tar.gz
linux_ppc64le.tar.gz
linux_riscv64.tar.gz
linux_s390x.tar.gz

Copy link
Member

@srebhan srebhan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me. Thanks for your nice contribution @nick-kentik!

@srebhan srebhan assigned powersj and unassigned srebhan May 5, 2023
Copy link
Contributor

@powersj powersj left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you!

@powersj powersj merged commit a808b9f into influxdata:master May 5, 2023
@nick-kentik
Copy link
Contributor Author

Great, thanks for the merge!

@srebhan srebhan added this to the v1.27.0 milestone Jun 21, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/opentsdb feat Improvement on an existing feature such as adding a new setting/mode to an existing plugin plugin/parser 1. Request for new parser plugins 2. Issues/PRs that are related to parser plugins ready for final review This pull request has been reviewed and/or tested by multiple users and is ready for a final review.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Parsers :: Add OpenTSDB data format support
5 participants