Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

internet_speed configuration options #10656

Closed
beacherpille88 opened this issue Feb 15, 2022 · 7 comments
Closed

internet_speed configuration options #10656

beacherpille88 opened this issue Feb 15, 2022 · 7 comments
Assignees
Labels
area/ping feature request Requests for new plugin and for new features to existing plugins waiting for response waiting for response from contributor

Comments

@beacherpille88
Copy link

Feature Request

Opening a feature request kicks off a discussion.

Proposal:

Adding more configuration options, e.g.:

  • do everything OR only download OR only latency PR only two of the three (download, upload, latency) checks
  • error handling
  • latency - fallback

Current behavior:

If one of the three checks fails, nothing gets written to the Bucket.

Desired behavior:

If anything fails, I still want to have the information already gathered or still able to gather. Additionally I want to use a latency-fallback-mode.

Use case:

The speed test is based on Speedtest by Ookla. Regularly this happens: #9852, sometimes, this happens:

Error in plugin: upload test failed failed: Post "http://speedtest02.daxisweb.net:8080/speedtest/upload.php": write tcp local.ip:56140->178.251.31.40:8080: use of closed network connection

Scenario 1: Step 1 (Download) fails
Maybe it is an issue with my ISP or an issue with Speedtest by Ookla. Each following steps (Upload and latency) arent executed. I still want to know.

Scenario 2: Step 2 or 3 (Upload or latency) fails
Maybe it is an issue with my ISP or an issue with Speedtest by Ookla. The following step (latency) isnt executed. I still want to know and I still want to have the information already gatherd (download and upload).

Scenario 3: Step 1 fails with the issue mentioned in #9852
A latency check to some ip-address (like in inputs.ping) has to be executed. At the moment, I would need two plugins to know, if my ISP is broken or Speedtest by Ookla.

@beacherpille88 beacherpille88 added the feature request Requests for new plugin and for new features to existing plugins label Feb 15, 2022
@powersj
Copy link
Contributor

powersj commented Feb 15, 2022

Hi,

When there is an error in a plugin that occurs, it ends the collection attempt. This makes a lot or more sense when you are trying to connect to a database for example and if the initial connect fails, there is no use in going on.

In the case of the internet speed plugin, I can see the desire to try to collect download speed if the upload speed test fails and vis versa. When you said it could be an issue with your ISP, or the Speedtest server you are reaching out to in both cases wouldn't we fail the same way with some sort of error about networking?

Scenario 3: Step 1 fails with the issue mentioned in #9852
A latency check to some ip-address (like in inputs.ping) has to be executed. At the moment, I would need two plugins to know, if my ISP is broken or Speedtest by Ookla.

How many of each issue are you seeing? I am curious to learn what the error rate is for fails on latency versus fails on download/upload. I have run it over a 24-hour period, and minus some issues in #9852 which should be fixed or more limited due to the recent #10530, I have not seen really any failures. Might be worth running with a nightly build as well to get a sense for failure rate.

Also how often are you running the plugin? I have seen issues where one run tops another run.

Thanks!

@powersj powersj added the waiting for response waiting for response from contributor label Feb 15, 2022
@powersj powersj self-assigned this Feb 15, 2022
@beacherpille88
Copy link
Author

In the case of the internet speed plugin, I can see the desire to try to collect download speed if the upload speed test fails and vis versa. When you said it could be an issue with your ISP, or the Speedtest server you are reaching out to in both cases wouldn't we fail the same way with some sort of error about networking?

In case my ISP has problems, I would get a timeout because the response doesnt get back to me. If speedtest had some reason to terminate the connection, they would actively send a response with the info, e.g. terminating the connection.

which should be fixed or more limited due to the recent #10530

I was not aware of this, thank you. I think this would limit my issues massively.

Also how often are you running the plugin?

I was running the plugin every 9mins. If you tell me how to measure the failure rates, I will do it. At the moment I have no idea how to distinguish between the different errors I was seeing during telegraf --debug --test

@telegraf-tiger telegraf-tiger bot removed the waiting for response waiting for response from contributor label Feb 15, 2022
@powersj
Copy link
Contributor

powersj commented Feb 15, 2022

If you could try the nightly images I linked to that would be good, I'd be curious to see how many error messages even show up.

Can you please run Telegraf with: telegraf --debug, overnight combined with your 9-minute timer and share the log with us here.

Thanks!

@powersj powersj added the waiting for response waiting for response from contributor label Feb 15, 2022
@beacherpille88
Copy link
Author

I started it right now. Here is some preview of the first round:

Feb 16 00:42:00 sun.beach.lan telegraf[69981]: 2022-02-15T23:42:00Z D! [inputs.internet_speed] Starting Speed Test
Feb 16 00:42:00 sun.beach.lan telegraf[69981]: 2022-02-15T23:42:00Z D! [inputs.internet_speed] Running Ping...
Feb 16 00:42:00 sun.beach.lan telegraf[69981]: 2022-02-15T23:42:00Z D! [inputs.internet_speed] Running Download...
Feb 16 00:42:27 sun.beach.lan telegraf[69981]: 2022-02-15T23:42:27Z D! [inputs.internet_speed] Running Upload...
Feb 16 00:43:10 sun.beach.lan telegraf[69981]: 2022-02-15T23:43:10Z E! [inputs.internet_speed] Error in plugin: upload test failed failed: Post "http://speedtest02.daxisweb.net:8080/speedtest/upload.php": write tcp 192.168.0.7:34150->178.251.31.40:8080: use of closed network connection

I will deliver the rest tomorrow.

@telegraf-tiger telegraf-tiger bot removed the waiting for response waiting for response from contributor label Feb 15, 2022
@beacherpille88
Copy link
Author

Here it comes. About 20 hours. I filtered to just the errors (E!). If this is not sufficient, hit me up and I will post everything.

Feb 15 10:00:00 sun.beach.lan telegraf[11461]: 2022-02-15T09:00:00Z E! [inputs.internet_speed] Error in plugin: fetching server list failed: unable to retrieve server list
Feb 15 11:30:00 sun.beach.lan telegraf[14629]: 2022-02-15T10:30:00Z E! [inputs.internet_speed] Error in plugin: fetching server list failed: unable to retrieve server list
Feb 15 11:40:40 sun.beach.lan telegraf[14629]: 2022-02-15T10:40:40Z E! [inputs.internet_speed] Error in plugin: upload test failed failed: Post "http://speedtest.meerfarbig.net:8080/speedtest/upload.php": write tcp 192.168.0.7:38862->185.37.147.2:8080: use of closed network connection
Feb 15 13:00:00 sun.beach.lan telegraf[14629]: 2022-02-15T12:00:00Z E! [inputs.internet_speed] Error in plugin: fetching server list failed: unable to retrieve server list
Feb 15 14:04:00 sun.beach.lan telegraf[14629]: 2022-02-15T13:04:00Z E! [inputs.internet_speed] Error in plugin: upload test failed failed: Post "http://speedtest.previder.nl:8080/speedtest/upload.php": EOF
Feb 15 14:13:11 sun.beach.lan telegraf[14629]: 2022-02-15T13:13:11Z E! [inputs.internet_speed] Error in plugin: upload test failed failed: Post "http://speedtest02.daxisweb.net:8080/speedtest/upload.php": write tcp 192.168.0.7:55750->178.251.31.40:8080: use of closed network connection
Feb 15 14:22:11 sun.beach.lan telegraf[14629]: 2022-02-15T13:22:11Z E! [inputs.internet_speed] Error in plugin: upload test failed failed: Post "http://speedtest02.daxisweb.net:8080/speedtest/upload.php": write tcp 192.168.0.7:55900->178.251.31.40:8080: write: connection reset by peer
Feb 15 14:30:00 sun.beach.lan telegraf[14629]: 2022-02-15T13:30:00Z E! [inputs.internet_speed] Error in plugin: fetching server list failed: unable to retrieve server list
Feb 15 15:16:04 sun.beach.lan telegraf[14629]: 2022-02-15T14:16:04Z E! [inputs.internet_speed] Error in plugin: upload test failed failed: Post "http://speedtest02.daxisweb.net:8080/speedtest/upload.php": write tcp 192.168.0.7:56826->178.251.31.40:8080: use of closed network connection
Feb 15 15:25:13 sun.beach.lan telegraf[14629]: 2022-02-15T14:25:13Z E! [inputs.internet_speed] Error in plugin: upload test failed failed: Post "http://speedtest02.daxisweb.net:8080/speedtest/upload.php": EOF
Feb 15 16:00:00 sun.beach.lan telegraf[14629]: 2022-02-15T15:00:00Z E! [inputs.internet_speed] Error in plugin: fetching server list failed: unable to retrieve server list
Feb 15 16:19:13 sun.beach.lan telegraf[14629]: 2022-02-15T15:19:13Z E! [inputs.internet_speed] Error in plugin: upload test failed failed: Post "http://speedtest02.daxisweb.net:8080/speedtest/upload.php": write tcp 192.168.0.7:57580->178.251.31.40:8080: use of closed network connection
Feb 15 17:30:00 sun.beach.lan telegraf[14629]: 2022-02-15T16:30:00Z E! [inputs.internet_speed] Error in plugin: fetching server list failed: unable to retrieve server list
Feb 15 19:22:04 sun.beach.lan telegraf[30719]: 2022-02-15T18:22:04Z E! [inputs.internet_speed] Error in plugin: upload test failed failed: Post "http://speedtest.qonnected.net:8080/speedtest/upload.php": EOF
Feb 15 21:00:00 sun.beach.lan telegraf[30719]: 2022-02-15T20:00:00Z E! [inputs.internet_speed] Error in plugin: fetching server list failed: unable to retrieve server list
Feb 16 00:19:09 sun.beach.lan telegraf[40022]: 2022-02-15T23:19:09Z E! [inputs.internet_speed] Error in plugin: upload test failed failed: Post "http://speedtest02.daxisweb.net:8080/speedtest/upload.php": write tcp 192.168.0.7:33926->178.251.31.40:8080: use of closed network connection
Feb 16 00:43:10 sun.beach.lan telegraf[69981]: 2022-02-15T23:43:10Z E! [inputs.internet_speed] Error in plugin: upload test failed failed: Post "http://speedtest02.daxisweb.net:8080/speedtest/upload.php": write tcp 192.168.0.7:34150->178.251.31.40:8080: use of closed network connection
Feb 16 00:52:08 sun.beach.lan telegraf[69981]: 2022-02-15T23:52:08Z E! [inputs.internet_speed] Error in plugin: upload test failed failed: Post "http://speedtest02.daxisweb.net:8080/speedtest/upload.php": write tcp 192.168.0.7:34260->178.251.31.40:8080: use of closed network connection
Feb 16 01:00:00 sun.beach.lan telegraf[69981]: 2022-02-16T00:00:00Z E! [inputs.internet_speed] Error in plugin: fetching server list failed: unable to retrieve server list
Feb 16 01:10:11 sun.beach.lan telegraf[69981]: 2022-02-16T00:10:11Z E! [inputs.internet_speed] Error in plugin: upload test failed failed: Post "http://speedtest02.daxisweb.net:8080/speedtest/upload.php": write tcp 192.168.0.7:34484->178.251.31.40:8080: use of closed network connection
Feb 16 01:19:09 sun.beach.lan telegraf[69981]: 2022-02-16T00:19:09Z E! [inputs.internet_speed] Error in plugin: upload test failed failed: Post "http://speedtest02.daxisweb.net:8080/speedtest/upload.php": write tcp 192.168.0.7:34588->178.251.31.40:8080: use of closed network connection
Feb 16 02:04:10 sun.beach.lan telegraf[69981]: 2022-02-16T01:04:10Z E! [inputs.internet_speed] Error in plugin: upload test failed failed: Post "http://speedtest02.daxisweb.net:8080/speedtest/upload.php": write tcp 192.168.0.7:35208->178.251.31.40:8080: use of closed network connection
Feb 16 02:30:00 sun.beach.lan telegraf[69981]: 2022-02-16T01:30:00Z E! [inputs.internet_speed] Error in plugin: fetching server list failed: unable to retrieve server list
Feb 16 04:00:00 sun.beach.lan telegraf[69981]: 2022-02-16T03:00:00Z E! [inputs.internet_speed] Error in plugin: fetching server list failed: unable to retrieve server list
Feb 16 05:30:00 sun.beach.lan telegraf[69981]: 2022-02-16T04:30:00Z E! [inputs.internet_speed] Error in plugin: fetching server list failed: unable to retrieve server list
Feb 16 07:00:00 sun.beach.lan telegraf[69981]: 2022-02-16T06:00:00Z E! [inputs.internet_speed] Error in plugin: fetching server list failed: unable to retrieve server list
Feb 16 08:30:00 sun.beach.lan telegraf[69981]: 2022-02-16T07:30:00Z E! [inputs.internet_speed] Error in plugin: fetching server list failed: unable to retrieve server list
Feb 16 09:43:04 sun.beach.lan telegraf[69981]: 2022-02-16T08:43:04Z E! [inputs.internet_speed] Error in plugin: upload test failed failed: Post "http://speedtest.stadtwerke-soest.de:8080/speedtest/upload.php": net/http: HTTP/1.x transport connection broken: malformed HTTP version "\x17\x03\x03\x00\x13ҹ8@\xa2:M\xe8\xdb\xdeN\x95\xf2\x18V2\u007f~\x04HTTP/1.1"
Feb 16 10:00:00 sun.beach.lan telegraf[69981]: 2022-02-16T09:00:00Z E! [inputs.internet_speed] Error in plugin: fetching server list failed: unable to retrieve server list
Feb 16 11:30:00 sun.beach.lan telegraf[69981]: 2022-02-16T10:30:00Z E! [inputs.internet_speed] Error in plugin: fetching server list failed: unable to retrieve server list
Feb 16 11:40:02 sun.beach.lan telegraf[69981]: 2022-02-16T10:40:02Z E! [inputs.internet_speed] Error in plugin: upload test failed failed: Post "http://ook-fra-x1.puregig.net:8080/speedtest/upload.php": EOF
Feb 16 12:25:06 sun.beach.lan telegraf[69981]: 2022-02-16T11:25:06Z E! [inputs.internet_speed] Error in plugin: upload test failed failed: Post "http://ook-fra-x1.puregig.net:8080/speedtest/upload.php": write tcp 192.168.0.7:57720->216.131.114.246:8080: use of closed network connection
Feb 16 13:00:00 sun.beach.lan telegraf[69981]: 2022-02-16T12:00:00Z E! [inputs.internet_speed] Error in plugin: fetching server list failed: unable to retrieve server list
Feb 16 14:30:00 sun.beach.lan telegraf[69981]: 2022-02-16T13:30:00Z E! [inputs.internet_speed] Error in plugin: fetching server list failed: unable to retrieve server list
Feb 16 14:49:02 sun.beach.lan telegraf[69981]: 2022-02-16T13:49:02Z E! [inputs.internet_speed] Error in plugin: upload test failed failed: Post "http://speedtest.plutex.de:8080/speedtest/upload.php": EOF
Feb 16 16:00:00 sun.beach.lan telegraf[69981]: 2022-02-16T15:00:00Z E! [inputs.internet_speed] Error in plugin: fetching server list failed: unable to retrieve server list
Feb 16 17:30:00 sun.beach.lan telegraf[69981]: 2022-02-16T16:30:00Z E! [inputs.internet_speed] Error in plugin: fetching server list failed: unable to retrieve server list
Feb 16 19:00:00 sun.beach.lan telegraf[69981]: 2022-02-16T18:00:00Z E! [inputs.internet_speed] Error in plugin: fetching server list failed: unable to retrieve server list
Feb 16 20:30:00 sun.beach.lan telegraf[69981]: 2022-02-16T19:30:00Z E! [inputs.internet_speed] Error in plugin: fetching server list failed: unable to retrieve server list

@powersj
Copy link
Contributor

powersj commented Feb 17, 2022

Did you do this with the nightly builds? And if so did you enable the cache option?

fetching server list failed: unable to retrieve server list

As you originally pointed out this error message is known and is covered by #9852. The root cause is the underlying library and the fix in #10530 should help prevent those from showing up going forward assuming you enabled the option to caches the results.

All the other errors seem to be centered around the upload test after the ping and download tests have already been completed:

use of closed network connection
EOF
write: connection reset by peer
malformed HTTP version

Of the proposed solutions in your original post:

error handling

this probably makes the most sense. I can see us re-trying once but after that throwing an error in the event of some sort of network blip. I am not a fan of only reporting some of the metrics either as that could hide a real issue or cause confusion with users as to why some data shows up and others does not.

If you agree, I think the next step then is to add a retry around the 3 three tests in the event of an error.

@powersj powersj added the waiting for response waiting for response from contributor label Feb 22, 2022
@telegraf-tiger
Copy link
Contributor

telegraf-tiger bot commented Mar 8, 2022

Hello! I am closing this issue due to inactivity. I hope you were able to resolve your problem, if not please try posting this question in our Community Slack or Community Page. Thank you!

@telegraf-tiger telegraf-tiger bot closed this as completed Mar 8, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/ping feature request Requests for new plugin and for new features to existing plugins waiting for response waiting for response from contributor
Projects
None yet
Development

No branches or pull requests

2 participants