Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Context deadline exceeded error with Docker plugin #2233

Closed
itskingori opened this issue Jan 6, 2017 · 12 comments · Fixed by #6016
Closed

Context deadline exceeded error with Docker plugin #2233

itskingori opened this issue Jan 6, 2017 · 12 comments · Fixed by #6016
Assignees
@goller
Labels
area/docker bug unexpected problem or unintended behavior
Milestone
1.12.0

Comments

@itskingori
Copy link

itskingori commented Jan 6, 2017 •
edited
Loading

Bug report

Relevant telegraf.conf

[global_tags]
  infrastructure = "kubernetes"
  namespace = "$POD_NAMESPACE"

[agent]
  interval = "10s"
  round_interval = true
  metric_batch_size = 1000
  metric_buffer_limit = 10000
  collection_jitter = "5s"
  flush_interval = "10s"
  flush_jitter = "5s"
  precision = ""
  debug = false
  quiet = false
  logfile = ""
  hostname = "$AGENT_HOSTNAME"
  omit_hostname = false

[[outputs.influxdb]]
  urls = ["http://$INFLUXDB_HOST:$INFLUXDB_PORT"]
  database = "$INFLUXDB_DATABASE"
  retention_policy = ""
  timeout = "5s"

[[inputs.docker]]
  endpoint = "unix:///var/run/docker.sock"
  container_names = []
  timeout = "5s"
  perdevice = true
  total = false

[[inputs.kubernetes]]
  url = "$KUBELET_SERVICE_URL"
  bearer_token = "$SERVICE_ACCOUNT_TOKEN_PATH"

Relevant environment variables

echo "$POD_NAMESPACE | $AGENT_HOSTNAME | http://$INFLUXDB_HOST:$INFLUXDB_PORT | $INFLUXDB_DATABASE | $KUBELET_SERVICE_URL | $SERVICE_ACCOUNT_TOKEN_PATH"
production | ip-192-168-97-124.ec2.internal | http://influxdb.production.svc.cluster.local:8086 | telegraf | http://192.168.97.124:10255 | /var/run/secrets/kubernetes.io/serviceaccount/token

System info:

Running on Kubernetes on AWS using the official Telegraf Docker image.

$ telegraf version
Telegraf v1.1.2 (git: release-1.1.0 3da3a2279108d02c2158782794532de7cd7deacd)

Steps to reproduce:

Just try run Telegraf and configure fetch Kubernetes metrics with the above configuration.

Expected behavior:

No errors when accessing Kubernetes metrics.

Actual behavior:

telegraf --test --config /etc/telegraf/telegraf2.toml
* Plugin: inputs.docker, Collection 1
> docker,engine_host=ip-192-168-97-124,host=ip-192-168-97-124.ec2.internal,infrastructure=kubernetes,namespace=production n_containers=22i,n_containers_paused=0i,n_containers_running=19i,n_containers_stopped=3i,n_cpus=2i,n_goroutines=169i,n_images=19i,n_listener_events=0i,n_used_file_descriptors=80i 1483799610000000000
> docker,engine_host=ip-192-168-97-124,host=ip-192-168-97-124.ec2.internal,infrastructure=kubernetes,namespace=production,unit=bytes memory_total=8374394880i 1483799610000000000
> docker_container_mem,container_image=b.gcr.io/kops-images/protokube,container_name=stoic_mahavira,container_version=1.5.0,engine_host=ip-192-168-97-124,host=ip-192-168-97-124.ec2.internal,infrastructure=kubernetes,namespace=production active_anon=10612736i,active_file=634880i,cache=1064960i,container_id="1a1d2536d5991e129bf8bc1fdc9d53378e1702aa7b9e7b5cf726f4e6042d1d7a",fail_count=0i,hierarchical_memory_limit=9223372036854771712i,inactive_anon=0i,inactive_file=430080i,limit=8374394880i,mapped_file=0i,max_usage=483180544i,pgfault=21697686i,pgmajfault=0i,pgpgin=12040981i,pgpgout=0i,rss=10612736i,rss_huge=0i,total_active_anon=10612736i,total_active_file=634880i,total_cache=1064960i,total_inactive_anon=0i,total_inactive_file=430080i,total_mapped_file=0i,total_pgfault=21697686i,total_pgmafault=0i,total_pgpgin=12040981i,total_pgpgout=12038130i,total_rss=10612736i,total_rss_huge=0i,total_unevictable=0i,total_writeback=0i,unevictable=0i,usage=11677696i,usage_percent=0.13944525147588932,writeback=0i 1483799611000000000
> docker_container_cpu,container_image=b.gcr.io/kops-images/protokube,container_name=stoic_mahavira,container_version=1.5.0,cpu=cpu-total,engine_host=ip-192-168-97-124,host=ip-192-168-97-124.ec2.internal,infrastructure=kubernetes,namespace=production container_id="1a1d2536d5991e129bf8bc1fdc9d53378e1702aa7b9e7b5cf726f4e6042d1d7a",throttling_periods=0i,throttling_throttled_periods=0i,throttling_throttled_time=0i,usage_in_kernelmode=80640000000i,usage_in_usermode=689760000000i,usage_percent=0,usage_system=719882790000000i,usage_total=814253669784i 1483799611000000000
> docker_container_cpu,container_image=b.gcr.io/kops-images/protokube,container_name=stoic_mahavira,container_version=1.5.0,cpu=cpu0,engine_host=ip-192-168-97-124,host=ip-192-168-97-124.ec2.internal,infrastructure=kubernetes,namespace=production container_id="1a1d2536d5991e129bf8bc1fdc9d53378e1702aa7b9e7b5cf726f4e6042d1d7a",usage_total=403900716695i 1483799611000000000
> docker_container_cpu,container_image=b.gcr.io/kops-images/protokube,container_name=stoic_mahavira,container_version=1.5.0,cpu=cpu1,engine_host=ip-192-168-97-124,host=ip-192-168-97-124.ec2.internal,infrastructure=kubernetes,namespace=production container_id="1a1d2536d5991e129bf8bc1fdc9d53378e1702aa7b9e7b5cf726f4e6042d1d7a",usage_total=410352953089i 1483799611000000000
> docker_container_blkio,container_image=b.gcr.io/kops-images/protokube,container_name=stoic_mahavira,container_version=1.5.0,device=202:5376,engine_host=ip-192-168-97-124,host=ip-192-168-97-124.ec2.internal,infrastructure=kubernetes,namespace=production container_id="1a1d2536d5991e129bf8bc1fdc9d53378e1702aa7b9e7b5cf726f4e6042d1d7a",io_service_bytes_recursive_async=1032192i,io_service_bytes_recursive_read=1032192i,io_service_bytes_recursive_sync=184332288i,io_service_bytes_recursive_total=185364480i,io_service_bytes_recursive_write=184332288i,io_serviced_recursive_async=144i,io_serviced_recursive_read=144i,io_serviced_recursive_sync=45003i,io_serviced_recursive_total=45147i,io_serviced_recursive_write=45003i 1483799611000000000
> docker_container_blkio,container_image=b.gcr.io/kops-images/protokube,container_name=stoic_mahavira,container_version=1.5.0,device=202:5120,engine_host=ip-192-168-97-124,host=ip-192-168-97-124.ec2.internal,infrastructure=kubernetes,namespace=production container_id="1a1d2536d5991e129bf8bc1fdc9d53378e1702aa7b9e7b5cf726f4e6042d1d7a",io_service_bytes_recursive_async=1052672i,io_service_bytes_recursive_read=1052672i,io_service_bytes_recursive_sync=189435904i,io_service_bytes_recursive_total=190488576i,io_service_bytes_recursive_write=189435904i,io_serviced_recursive_async=149i,io_serviced_recursive_read=149i,io_serviced_recursive_sync=46249i,io_serviced_recursive_total=46398i,io_serviced_recursive_write=46249i 1483799611000000000
> docker_container_blkio,container_image=b.gcr.io/kops-images/protokube,container_name=stoic_mahavira,container_version=1.5.0,device=202:0,engine_host=ip-192-168-97-124,host=ip-192-168-97-124.ec2.internal,infrastructure=kubernetes,namespace=production container_id="1a1d2536d5991e129bf8bc1fdc9d53378e1702aa7b9e7b5cf726f4e6042d1d7a",io_service_bytes_recursive_async=4096i,io_service_bytes_recursive_read=4096i,io_service_bytes_recursive_sync=0i,io_service_bytes_recursive_total=4096i,io_service_bytes_recursive_write=0i,io_serviced_recursive_async=1i,io_serviced_recursive_read=1i,io_serviced_recursive_sync=0i,io_serviced_recursive_total=1i,io_serviced_recursive_write=0i 1483799611000000000
> docker_container_mem,container_image=gcr.io/google_containers/pause-amd64,container_name=k8s_POD.d8dbe16c_etcd-server-events-ip-192-168-97-124.ec2.internal_kube-system_2b310f818153713c2adba2fa1bbe1a20_a9930641,container_version=3.0,engine_host=ip-192-168-97-124,host=ip-192-168-97-124.ec2.internal,infrastructure=kubernetes,io.kubernetes.container.hash=d8dbe16c,io.kubernetes.container.name=POD,io.kubernetes.container.restartCount=0,io.kubernetes.pod.name=etcd-server-events-ip-192-168-97-124.ec2.internal,io.kubernetes.pod.namespace=kube-system,io.kubernetes.pod.terminationGracePeriod=30,io.kubernetes.pod.uid=2b310f818153713c2adba2fa1bbe1a20,namespace=production active_anon=45056i,active_file=0i,cache=0i,container_id="88be29b37933b77d929e2b4f7538833eff1b3d5fbe57962a9d074e7b6f1c276d",fail_count=0i,hierarchical_memory_limit=9223372036854771712i,inactive_anon=0i,inactive_file=0i,limit=8374394880i,mapped_file=0i,max_usage=524288i,pgfault=123i,pgmajfault=0i,pgpgin=111i,pgpgout=0i,rss=45056i,rss_huge=0i,total_active_anon=45056i,total_active_file=0i,total_cache=0i,total_inactive_anon=0i,total_inactive_file=0i,total_mapped_file=0i,total_pgfault=123i,total_pgmafault=0i,total_pgpgin=111i,total_pgpgout=100i,total_rss=45056i,total_rss_huge=0i,total_unevictable=0i,total_writeback=0i,unevictable=0i,usage=45056i,usage_percent=0.0005380209632531681,writeback=0i 1483799611000000000
> docker_container_cpu,container_image=gcr.io/google_containers/pause-amd64,container_name=k8s_POD.d8dbe16c_etcd-server-events-ip-192-168-97-124.ec2.internal_kube-system_2b310f818153713c2adba2fa1bbe1a20_a9930641,container_version=3.0,cpu=cpu-total,engine_host=ip-192-168-97-124,host=ip-192-168-97-124.ec2.internal,infrastructure=kubernetes,io.kubernetes.container.hash=d8dbe16c,io.kubernetes.container.name=POD,io.kubernetes.container.restartCount=0,io.kubernetes.pod.name=etcd-server-events-ip-192-168-97-124.ec2.internal,io.kubernetes.pod.namespace=kube-system,io.kubernetes.pod.terminationGracePeriod=30,io.kubernetes.pod.uid=2b310f818153713c2adba2fa1bbe1a20,namespace=production container_id="88be29b37933b77d929e2b4f7538833eff1b3d5fbe57962a9d074e7b6f1c276d",throttling_periods=0i,throttling_throttled_periods=0i,throttling_throttled_time=0i,usage_in_kernelmode=0i,usage_in_usermode=0i,usage_percent=0,usage_system=719882790000000i,usage_total=5063367i 1483799611000000000
> docker_container_cpu,container_image=gcr.io/google_containers/pause-amd64,container_name=k8s_POD.d8dbe16c_etcd-server-events-ip-192-168-97-124.ec2.internal_kube-system_2b310f818153713c2adba2fa1bbe1a20_a9930641,container_version=3.0,cpu=cpu0,engine_host=ip-192-168-97-124,host=ip-192-168-97-124.ec2.internal,infrastructure=kubernetes,io.kubernetes.container.hash=d8dbe16c,io.kubernetes.container.name=POD,io.kubernetes.container.restartCount=0,io.kubernetes.pod.name=etcd-server-events-ip-192-168-97-124.ec2.internal,io.kubernetes.pod.namespace=kube-system,io.kubernetes.pod.terminationGracePeriod=30,io.kubernetes.pod.uid=2b310f818153713c2adba2fa1bbe1a20,namespace=production container_id="88be29b37933b77d929e2b4f7538833eff1b3d5fbe57962a9d074e7b6f1c276d",usage_total=933730i 1483799611000000000
> docker_container_cpu,container_image=gcr.io/google_containers/pause-amd64,container_name=k8s_POD.d8dbe16c_etcd-server-events-ip-192-168-97-124.ec2.internal_kube-system_2b310f818153713c2adba2fa1bbe1a20_a9930641,container_version=3.0,cpu=cpu1,engine_host=ip-192-168-97-124,host=ip-192-168-97-124.ec2.internal,infrastructure=kubernetes,io.kubernetes.container.hash=d8dbe16c,io.kubernetes.container.name=POD,io.kubernetes.container.restartCount=0,io.kubernetes.pod.name=etcd-server-events-ip-192-168-97-124.ec2.internal,io.kubernetes.pod.namespace=kube-system,io.kubernetes.pod.terminationGracePeriod=30,io.kubernetes.pod.uid=2b310f818153713c2adba2fa1bbe1a20,namespace=production container_id="88be29b37933b77d929e2b4f7538833eff1b3d5fbe57962a9d074e7b6f1c276d",usage_total=4129637i 1483799611000000000
> docker_container_mem,container_image=gcr.io/google_containers/pause-amd64,container_name=k8s_POD.d8dbe16c_etcd-server-ip-192-168-97-124.ec2.internal_kube-system_397cce481a573eafc904eb7b68940825_93ef016c,container_version=3.0,engine_host=ip-192-168-97-124,host=ip-192-168-97-124.ec2.internal,infrastructure=kubernetes,io.kubernetes.container.hash=d8dbe16c,io.kubernetes.container.name=POD,io.kubernetes.container.restartCount=0,io.kubernetes.pod.name=etcd-server-ip-192-168-97-124.ec2.internal,io.kubernetes.pod.namespace=kube-system,io.kubernetes.pod.terminationGracePeriod=30,io.kubernetes.pod.uid=397cce481a573eafc904eb7b68940825,namespace=production active_anon=40960i,active_file=4096i,cache=4096i,container_id="641f22f7c7fa0b27201c7d48e5b4c2434e4c317c41bd84043f061908337d301a",fail_count=0i,hierarchical_memory_limit=9223372036854771712i,inactive_anon=0i,inactive_file=0i,limit=8374394880i,mapped_file=0i,max_usage=524288i,pgfault=99i,pgmajfault=0i,pgpgin=89i,pgpgout=0i,rss=40960i,rss_huge=0i,total_active_anon=40960i,total_active_file=4096i,total_cache=4096i,total_inactive_anon=0i,total_inactive_file=0i,total_mapped_file=0i,total_pgfault=99i,total_pgmafault=0i,total_pgpgin=89i,total_pgpgout=78i,total_rss=40960i,total_rss_huge=0i,total_unevictable=0i,total_writeback=0i,unevictable=0i,usage=45056i,usage_percent=0.0005380209632531681,writeback=0i 1483799611000000000
> docker_container_cpu,container_image=gcr.io/google_containers/pause-amd64,container_name=k8s_POD.d8dbe16c_etcd-server-ip-192-168-97-124.ec2.internal_kube-system_397cce481a573eafc904eb7b68940825_93ef016c,container_version=3.0,cpu=cpu-total,engine_host=ip-192-168-97-124,host=ip-192-168-97-124.ec2.internal,infrastructure=kubernetes,io.kubernetes.container.hash=d8dbe16c,io.kubernetes.container.name=POD,io.kubernetes.container.restartCount=0,io.kubernetes.pod.name=etcd-server-ip-192-168-97-124.ec2.internal,io.kubernetes.pod.namespace=kube-system,io.kubernetes.pod.terminationGracePeriod=30,io.kubernetes.pod.uid=397cce481a573eafc904eb7b68940825,namespace=production container_id="641f22f7c7fa0b27201c7d48e5b4c2434e4c317c41bd84043f061908337d301a",throttling_periods=0i,throttling_throttled_periods=0i,throttling_throttled_time=0i,usage_in_kernelmode=0i,usage_in_usermode=0i,usage_percent=0,usage_system=719882790000000i,usage_total=5097011i 1483799611000000000
> docker_container_cpu,container_image=gcr.io/google_containers/pause-amd64,container_name=k8s_POD.d8dbe16c_etcd-server-ip-192-168-97-124.ec2.internal_kube-system_397cce481a573eafc904eb7b68940825_93ef016c,container_version=3.0,cpu=cpu0,engine_host=ip-192-168-97-124,host=ip-192-168-97-124.ec2.internal,infrastructure=kubernetes,io.kubernetes.container.hash=d8dbe16c,io.kubernetes.container.name=POD,io.kubernetes.container.restartCount=0,io.kubernetes.pod.name=etcd-server-ip-192-168-97-124.ec2.internal,io.kubernetes.pod.namespace=kube-system,io.kubernetes.pod.terminationGracePeriod=30,io.kubernetes.pod.uid=397cce481a573eafc904eb7b68940825,namespace=production container_id="641f22f7c7fa0b27201c7d48e5b4c2434e4c317c41bd84043f061908337d301a",usage_total=576864i 1483799611000000000
> docker_container_cpu,container_image=gcr.io/google_containers/pause-amd64,container_name=k8s_POD.d8dbe16c_etcd-server-ip-192-168-97-124.ec2.internal_kube-system_397cce481a573eafc904eb7b68940825_93ef016c,container_version=3.0,cpu=cpu1,engine_host=ip-192-168-97-124,host=ip-192-168-97-124.ec2.internal,infrastructure=kubernetes,io.kubernetes.container.hash=d8dbe16c,io.kubernetes.container.name=POD,io.kubernetes.container.restartCount=0,io.kubernetes.pod.name=etcd-server-ip-192-168-97-124.ec2.internal,io.kubernetes.pod.namespace=kube-system,io.kubernetes.pod.terminationGracePeriod=30,io.kubernetes.pod.uid=397cce481a573eafc904eb7b68940825,namespace=production container_id="641f22f7c7fa0b27201c7d48e5b4c2434e4c317c41bd84043f061908337d301a",usage_total=4520147i 1483799611000000000
> docker_container_mem,container_image=gcr.io/google_containers/pause-amd64,container_name=k8s_POD.d8dbe16c_kube-scheduler-ip-192-168-97-124.ec2.internal_kube-system_b94e9fc6e899eb4a4b5b1f94be012ac2_901da924,container_version=3.0,engine_host=ip-192-168-97-124,host=ip-192-168-97-124.ec2.internal,infrastructure=kubernetes,io.kubernetes.container.hash=d8dbe16c,io.kubernetes.container.name=POD,io.kubernetes.container.restartCount=0,io.kubernetes.pod.name=kube-scheduler-ip-192-168-97-124.ec2.internal,io.kubernetes.pod.namespace=kube-system,io.kubernetes.pod.terminationGracePeriod=30,io.kubernetes.pod.uid=b94e9fc6e899eb4a4b5b1f94be012ac2,namespace=production active_anon=45056i,active_file=135168i,cache=135168i,container_id="915f626d803c873c2244c2264a2bbd3e1b82827e1ba43901700aa7971816ac20",fail_count=0i,hierarchical_memory_limit=9223372036854771712i,inactive_anon=0i,inactive_file=0i,limit=8374394880i,mapped_file=0i,max_usage=548864i,pgfault=101i,pgmajfault=0i,pgpgin=117i,pgpgout=0i,rss=45056i,rss_huge=0i,total_active_anon=45056i,total_active_file=135168i,total_cache=135168i,total_inactive_anon=0i,total_inactive_file=0i,total_mapped_file=0i,total_pgfault=101i,total_pgmafault=0i,total_pgpgin=117i,total_pgpgout=73i,total_rss=45056i,total_rss_huge=0i,total_unevictable=0i,total_writeback=0i,unevictable=0i,usage=180224i,usage_percent=0.0021520838530126726,writeback=0i 1483799611000000000
> docker_container_cpu,container_image=gcr.io/google_containers/pause-amd64,container_name=k8s_POD.d8dbe16c_kube-scheduler-ip-192-168-97-124.ec2.internal_kube-system_b94e9fc6e899eb4a4b5b1f94be012ac2_901da924,container_version=3.0,cpu=cpu-total,engine_host=ip-192-168-97-124,host=ip-192-168-97-124.ec2.internal,infrastructure=kubernetes,io.kubernetes.container.hash=d8dbe16c,io.kubernetes.container.name=POD,io.kubernetes.container.restartCount=0,io.kubernetes.pod.name=kube-scheduler-ip-192-168-97-124.ec2.internal,io.kubernetes.pod.namespace=kube-system,io.kubernetes.pod.terminationGracePeriod=30,io.kubernetes.pod.uid=b94e9fc6e899eb4a4b5b1f94be012ac2,namespace=production container_id="915f626d803c873c2244c2264a2bbd3e1b82827e1ba43901700aa7971816ac20",throttling_periods=0i,throttling_throttled_periods=0i,throttling_throttled_time=0i,usage_in_kernelmode=0i,usage_in_usermode=0i,usage_percent=0,usage_system=719882790000000i,usage_total=5762169i 1483799611000000000
> docker_container_cpu,container_image=gcr.io/google_containers/pause-amd64,container_name=k8s_POD.d8dbe16c_kube-scheduler-ip-192-168-97-124.ec2.internal_kube-system_b94e9fc6e899eb4a4b5b1f94be012ac2_901da924,container_version=3.0,cpu=cpu0,engine_host=ip-192-168-97-124,host=ip-192-168-97-124.ec2.internal,infrastructure=kubernetes,io.kubernetes.container.hash=d8dbe16c,io.kubernetes.container.name=POD,io.kubernetes.container.restartCount=0,io.kubernetes.pod.name=kube-scheduler-ip-192-168-97-124.ec2.internal,io.kubernetes.pod.namespace=kube-system,io.kubernetes.pod.terminationGracePeriod=30,io.kubernetes.pod.uid=b94e9fc6e899eb4a4b5b1f94be012ac2,namespace=production container_id="915f626d803c873c2244c2264a2bbd3e1b82827e1ba43901700aa7971816ac20",usage_total=4151832i 1483799611000000000
> docker_container_cpu,container_image=gcr.io/google_containers/pause-amd64,container_name=k8s_POD.d8dbe16c_kube-scheduler-ip-192-168-97-124.ec2.internal_kube-system_b94e9fc6e899eb4a4b5b1f94be012ac2_901da924,container_version=3.0,cpu=cpu1,engine_host=ip-192-168-97-124,host=ip-192-168-97-124.ec2.internal,infrastructure=kubernetes,io.kubernetes.container.hash=d8dbe16c,io.kubernetes.container.name=POD,io.kubernetes.container.restartCount=0,io.kubernetes.pod.name=kube-scheduler-ip-192-168-97-124.ec2.internal,io.kubernetes.pod.namespace=kube-system,io.kubernetes.pod.terminationGracePeriod=30,io.kubernetes.pod.uid=b94e9fc6e899eb4a4b5b1f94be012ac2,namespace=production container_id="915f626d803c873c2244c2264a2bbd3e1b82827e1ba43901700aa7971816ac20",usage_total=1610337i 1483799611000000000
> docker_container_blkio,container_image=gcr.io/google_containers/pause-amd64,container_name=k8s_POD.d8dbe16c_kube-scheduler-ip-192-168-97-124.ec2.internal_kube-system_b94e9fc6e899eb4a4b5b1f94be012ac2_901da924,container_version=3.0,device=202:0,engine_host=ip-192-168-97-124,host=ip-192-168-97-124.ec2.internal,infrastructure=kubernetes,io.kubernetes.container.hash=d8dbe16c,io.kubernetes.container.name=POD,io.kubernetes.container.restartCount=0,io.kubernetes.pod.name=kube-scheduler-ip-192-168-97-124.ec2.internal,io.kubernetes.pod.namespace=kube-system,io.kubernetes.pod.terminationGracePeriod=30,io.kubernetes.pod.uid=b94e9fc6e899eb4a4b5b1f94be012ac2,namespace=production container_id="915f626d803c873c2244c2264a2bbd3e1b82827e1ba43901700aa7971816ac20",io_service_bytes_recursive_async=131072i,io_service_bytes_recursive_read=131072i,io_service_bytes_recursive_sync=0i,io_service_bytes_recursive_total=131072i,io_service_bytes_recursive_write=0i,io_serviced_recursive_async=1i,io_serviced_recursive_read=1i,io_serviced_recursive_sync=0i,io_serviced_recursive_total=1i,io_serviced_recursive_write=0i 1483799611000000000
> docker_container_mem,container_image=gcr.io/google_containers/pause-amd64,container_name=k8s_POD.d8dbe16c_kube-apiserver-ip-192-168-97-124.ec2.internal_kube-system_39748d1ca3f847081acfff1e2913a43e_13ef2139,container_version=3.0,engine_host=ip-192-168-97-124,host=ip-192-168-97-124.ec2.internal,infrastructure=kubernetes,io.kubernetes.container.hash=d8dbe16c,io.kubernetes.container.name=POD,io.kubernetes.container.restartCount=0,io.kubernetes.pod.name=kube-apiserver-ip-192-168-97-124.ec2.internal,io.kubernetes.pod.namespace=kube-system,io.kubernetes.pod.terminationGracePeriod=30,io.kubernetes.pod.uid=39748d1ca3f847081acfff1e2913a43e,namespace=production active_anon=40960i,active_file=4096i,cache=16384i,container_id="196476e714ba9f5ef348c7bdcbaf7509a4e88dde73890c31445545e0c3ab1930",fail_count=0i,hierarchical_memory_limit=9223372036854771712i,inactive_anon=0i,inactive_file=12288i,limit=8374394880i,mapped_file=0i,max_usage=593920i,pgfault=103i,pgmajfault=0i,pgpgin=94i,pgpgout=0i,rss=40960i,rss_huge=0i,total_active_anon=40960i,total_active_file=4096i,total_cache=16384i,total_inactive_anon=0i,total_inactive_file=12288i,total_mapped_file=0i,total_pgfault=103i,total_pgmafault=0i,total_pgpgin=94i,total_pgpgout=80i,total_rss=40960i,total_rss_huge=0i,total_unevictable=0i,total_writeback=0i,unevictable=0i,usage=57344i,usage_percent=0.000684753953231305,writeback=0i 1483799611000000000
> docker_container_cpu,container_image=gcr.io/google_containers/pause-amd64,container_name=k8s_POD.d8dbe16c_kube-apiserver-ip-192-168-97-124.ec2.internal_kube-system_39748d1ca3f847081acfff1e2913a43e_13ef2139,container_version=3.0,cpu=cpu-total,engine_host=ip-192-168-97-124,host=ip-192-168-97-124.ec2.internal,infrastructure=kubernetes,io.kubernetes.container.hash=d8dbe16c,io.kubernetes.container.name=POD,io.kubernetes.container.restartCount=0,io.kubernetes.pod.name=kube-apiserver-ip-192-168-97-124.ec2.internal,io.kubernetes.pod.namespace=kube-system,io.kubernetes.pod.terminationGracePeriod=30,io.kubernetes.pod.uid=39748d1ca3f847081acfff1e2913a43e,namespace=production container_id="196476e714ba9f5ef348c7bdcbaf7509a4e88dde73890c31445545e0c3ab1930",throttling_periods=0i,throttling_throttled_periods=0i,throttling_throttled_time=0i,usage_in_kernelmode=0i,usage_in_usermode=0i,usage_percent=0,usage_system=719882790000000i,usage_total=7360653i 1483799611000000000
> docker_container_cpu,container_image=gcr.io/google_containers/pause-amd64,container_name=k8s_POD.d8dbe16c_kube-apiserver-ip-192-168-97-124.ec2.internal_kube-system_39748d1ca3f847081acfff1e2913a43e_13ef2139,container_version=3.0,cpu=cpu0,engine_host=ip-192-168-97-124,host=ip-192-168-97-124.ec2.internal,infrastructure=kubernetes,io.kubernetes.container.hash=d8dbe16c,io.kubernetes.container.name=POD,io.kubernetes.container.restartCount=0,io.kubernetes.pod.name=kube-apiserver-ip-192-168-97-124.ec2.internal,io.kubernetes.pod.namespace=kube-system,io.kubernetes.pod.terminationGracePeriod=30,io.kubernetes.pod.uid=39748d1ca3f847081acfff1e2913a43e,namespace=production container_id="196476e714ba9f5ef348c7bdcbaf7509a4e88dde73890c31445545e0c3ab1930",usage_total=7074682i 1483799611000000000
> docker_container_cpu,container_image=gcr.io/google_containers/pause-amd64,container_name=k8s_POD.d8dbe16c_kube-apiserver-ip-192-168-97-124.ec2.internal_kube-system_39748d1ca3f847081acfff1e2913a43e_13ef2139,container_version=3.0,cpu=cpu1,engine_host=ip-192-168-97-124,host=ip-192-168-97-124.ec2.internal,infrastructure=kubernetes,io.kubernetes.container.hash=d8dbe16c,io.kubernetes.container.name=POD,io.kubernetes.container.restartCount=0,io.kubernetes.pod.name=kube-apiserver-ip-192-168-97-124.ec2.internal,io.kubernetes.pod.namespace=kube-system,io.kubernetes.pod.terminationGracePeriod=30,io.kubernetes.pod.uid=39748d1ca3f847081acfff1e2913a43e,namespace=production container_id="196476e714ba9f5ef348c7bdcbaf7509a4e88dde73890c31445545e0c3ab1930",usage_total=285971i 1483799611000000000
> docker_container_blkio,container_image=gcr.io/google_containers/pause-amd64,container_name=k8s_POD.d8dbe16c_kube-apiserver-ip-192-168-97-124.ec2.internal_kube-system_39748d1ca3f847081acfff1e2913a43e_13ef2139,container_version=3.0,device=202:0,engine_host=ip-192-168-97-124,host=ip-192-168-97-124.ec2.internal,infrastructure=kubernetes,io.kubernetes.container.hash=d8dbe16c,io.kubernetes.container.name=POD,io.kubernetes.container.restartCount=0,io.kubernetes.pod.name=kube-apiserver-ip-192-168-97-124.ec2.internal,io.kubernetes.pod.namespace=kube-system,io.kubernetes.pod.terminationGracePeriod=30,io.kubernetes.pod.uid=39748d1ca3f847081acfff1e2913a43e,namespace=production container_id="196476e714ba9f5ef348c7bdcbaf7509a4e88dde73890c31445545e0c3ab1930",io_service_bytes_recursive_async=12288i,io_service_bytes_recursive_read=12288i,io_service_bytes_recursive_sync=0i,io_service_bytes_recursive_total=12288i,io_service_bytes_recursive_write=0i,io_serviced_recursive_async=2i,io_serviced_recursive_read=2i,io_serviced_recursive_sync=0i,io_serviced_recursive_total=2i,io_serviced_recursive_write=0i 1483799611000000000
> docker_container_mem,container_image=gcr.io/google_containers/pause-amd64,container_name=k8s_POD.d8dbe16c_telegraf-aa5q5_production_24e28a0c-d4e4-11e6-af2d-0a209ce05b0a_a1fa361c,container_version=3.0,engine_host=ip-192-168-97-124,host=ip-192-168-97-124.ec2.internal,infrastructure=kubernetes,io.kubernetes.container.hash=d8dbe16c,io.kubernetes.container.name=POD,io.kubernetes.container.restartCount=0,io.kubernetes.pod.name=telegraf-aa5q5,io.kubernetes.pod.namespace=production,io.kubernetes.pod.terminationGracePeriod=30,io.kubernetes.pod.uid=24e28a0c-d4e4-11e6-af2d-0a209ce05b0a,namespace=production active_anon=40960i,active_file=4096i,cache=4096i,container_id="df0f1de816ba3b0b3799132c1d6cf01e0341e7016629ab3bd18d724ad23c8465",fail_count=0i,hierarchical_memory_limit=9223372036854771712i,inactive_anon=0i,inactive_file=0i,limit=8374394880i,mapped_file=0i,max_usage=655360i,pgfault=133i,pgmajfault=0i,pgpgin=121i,pgpgout=0i,rss=40960i,rss_huge=0i,total_active_anon=40960i,total_active_file=4096i,total_cache=4096i,total_inactive_anon=0i,total_inactive_file=0i,total_mapped_file=0i,total_pgfault=133i,total_pgmafault=0i,total_pgpgin=121i,total_pgpgout=110i,total_rss=40960i,total_rss_huge=0i,total_unevictable=0i,total_writeback=0i,unevictable=0i,usage=45056i,usage_percent=0.0005380209632531681,writeback=0i 1483799611000000000
> docker_container_cpu,container_image=gcr.io/google_containers/pause-amd64,container_name=k8s_POD.d8dbe16c_telegraf-aa5q5_production_24e28a0c-d4e4-11e6-af2d-0a209ce05b0a_a1fa361c,container_version=3.0,cpu=cpu-total,engine_host=ip-192-168-97-124,host=ip-192-168-97-124.ec2.internal,infrastructure=kubernetes,io.kubernetes.container.hash=d8dbe16c,io.kubernetes.container.name=POD,io.kubernetes.container.restartCount=0,io.kubernetes.pod.name=telegraf-aa5q5,io.kubernetes.pod.namespace=production,io.kubernetes.pod.terminationGracePeriod=30,io.kubernetes.pod.uid=24e28a0c-d4e4-11e6-af2d-0a209ce05b0a,namespace=production container_id="df0f1de816ba3b0b3799132c1d6cf01e0341e7016629ab3bd18d724ad23c8465",throttling_periods=0i,throttling_throttled_periods=0i,throttling_throttled_time=0i,usage_in_kernelmode=0i,usage_in_usermode=0i,usage_percent=0,usage_system=719882790000000i,usage_total=6145777i 1483799611000000000
> docker_container_cpu,container_image=gcr.io/google_containers/pause-amd64,container_name=k8s_POD.d8dbe16c_telegraf-aa5q5_production_24e28a0c-d4e4-11e6-af2d-0a209ce05b0a_a1fa361c,container_version=3.0,cpu=cpu0,engine_host=ip-192-168-97-124,host=ip-192-168-97-124.ec2.internal,infrastructure=kubernetes,io.kubernetes.container.hash=d8dbe16c,io.kubernetes.container.name=POD,io.kubernetes.container.restartCount=0,io.kubernetes.pod.name=telegraf-aa5q5,io.kubernetes.pod.namespace=production,io.kubernetes.pod.terminationGracePeriod=30,io.kubernetes.pod.uid=24e28a0c-d4e4-11e6-af2d-0a209ce05b0a,namespace=production container_id="df0f1de816ba3b0b3799132c1d6cf01e0341e7016629ab3bd18d724ad23c8465",usage_total=4986515i 1483799611000000000
> docker_container_cpu,container_image=gcr.io/google_containers/pause-amd64,container_name=k8s_POD.d8dbe16c_telegraf-aa5q5_production_24e28a0c-d4e4-11e6-af2d-0a209ce05b0a_a1fa361c,container_version=3.0,cpu=cpu1,engine_host=ip-192-168-97-124,host=ip-192-168-97-124.ec2.internal,infrastructure=kubernetes,io.kubernetes.container.hash=d8dbe16c,io.kubernetes.container.name=POD,io.kubernetes.container.restartCount=0,io.kubernetes.pod.name=telegraf-aa5q5,io.kubernetes.pod.namespace=production,io.kubernetes.pod.terminationGracePeriod=30,io.kubernetes.pod.uid=24e28a0c-d4e4-11e6-af2d-0a209ce05b0a,namespace=production container_id="df0f1de816ba3b0b3799132c1d6cf01e0341e7016629ab3bd18d724ad23c8465",usage_total=1159262i 1483799611000000000
> docker_container_mem,container_image=gcr.io/google_containers/pause-amd64,container_name=k8s_POD.d8dbe16c_kube-controller-manager-ip-192-168-97-124.ec2.internal_kube-system_f0713bd8968299c1008273c9b6b0896c_2203f3f6,container_version=3.0,engine_host=ip-192-168-97-124,host=ip-192-168-97-124.ec2.internal,infrastructure=kubernetes,io.kubernetes.container.hash=d8dbe16c,io.kubernetes.container.name=POD,io.kubernetes.container.restartCount=0,io.kubernetes.pod.name=kube-controller-manager-ip-192-168-97-124.ec2.internal,io.kubernetes.pod.namespace=kube-system,io.kubernetes.pod.terminationGracePeriod=30,io.kubernetes.pod.uid=f0713bd8968299c1008273c9b6b0896c,namespace=production active_anon=40960i,active_file=4096i,cache=4096i,container_id="5e4f15495e7b221c2d7cf991deb7dd0bf3afea1ca3e55366310e7648f3668854",fail_count=0i,hierarchical_memory_limit=9223372036854771712i,inactive_anon=0i,inactive_file=0i,limit=8374394880i,mapped_file=0i,max_usage=634880i,pgfault=117i,pgmajfault=0i,pgpgin=105i,pgpgout=0i,rss=40960i,rss_huge=0i,total_active_anon=40960i,total_active_file=4096i,total_cache=4096i,total_inactive_anon=0i,total_inactive_file=0i,total_mapped_file=0i,total_pgfault=117i,total_pgmafault=0i,total_pgpgin=105i,total_pgpgout=94i,total_rss=40960i,total_rss_huge=0i,total_unevictable=0i,total_writeback=0i,unevictable=0i,usage=45056i,usage_percent=0.0005380209632531681,writeback=0i 1483799611000000000
> docker_container_cpu,container_image=gcr.io/google_containers/pause-amd64,container_name=k8s_POD.d8dbe16c_kube-controller-manager-ip-192-168-97-124.ec2.internal_kube-system_f0713bd8968299c1008273c9b6b0896c_2203f3f6,container_version=3.0,cpu=cpu-total,engine_host=ip-192-168-97-124,host=ip-192-168-97-124.ec2.internal,infrastructure=kubernetes,io.kubernetes.container.hash=d8dbe16c,io.kubernetes.container.name=POD,io.kubernetes.container.restartCount=0,io.kubernetes.pod.name=kube-controller-manager-ip-192-168-97-124.ec2.internal,io.kubernetes.pod.namespace=kube-system,io.kubernetes.pod.terminationGracePeriod=30,io.kubernetes.pod.uid=f0713bd8968299c1008273c9b6b0896c,namespace=production container_id="5e4f15495e7b221c2d7cf991deb7dd0bf3afea1ca3e55366310e7648f3668854",throttling_periods=0i,throttling_throttled_periods=0i,throttling_throttled_time=0i,usage_in_kernelmode=0i,usage_in_usermode=0i,usage_percent=0,usage_system=719882790000000i,usage_total=4596345i 1483799611000000000
> docker_container_cpu,container_image=gcr.io/google_containers/pause-amd64,container_name=k8s_POD.d8dbe16c_kube-controller-manager-ip-192-168-97-124.ec2.internal_kube-system_f0713bd8968299c1008273c9b6b0896c_2203f3f6,container_version=3.0,cpu=cpu0,engine_host=ip-192-168-97-124,host=ip-192-168-97-124.ec2.internal,infrastructure=kubernetes,io.kubernetes.container.hash=d8dbe16c,io.kubernetes.container.name=POD,io.kubernetes.container.restartCount=0,io.kubernetes.pod.name=kube-controller-manager-ip-192-168-97-124.ec2.internal,io.kubernetes.pod.namespace=kube-system,io.kubernetes.pod.terminationGracePeriod=30,io.kubernetes.pod.uid=f0713bd8968299c1008273c9b6b0896c,namespace=production container_id="5e4f15495e7b221c2d7cf991deb7dd0bf3afea1ca3e55366310e7648f3668854",usage_total=4198951i 1483799611000000000
> docker_container_cpu,container_image=gcr.io/google_containers/pause-amd64,container_name=k8s_POD.d8dbe16c_kube-controller-manager-ip-192-168-97-124.ec2.internal_kube-system_f0713bd8968299c1008273c9b6b0896c_2203f3f6,container_version=3.0,cpu=cpu1,engine_host=ip-192-168-97-124,host=ip-192-168-97-124.ec2.internal,infrastructure=kubernetes,io.kubernetes.container.hash=d8dbe16c,io.kubernetes.container.name=POD,io.kubernetes.container.restartCount=0,io.kubernetes.pod.name=kube-controller-manager-ip-192-168-97-124.ec2.internal,io.kubernetes.pod.namespace=kube-system,io.kubernetes.pod.terminationGracePeriod=30,io.kubernetes.pod.uid=f0713bd8968299c1008273c9b6b0896c,namespace=production container_id="5e4f15495e7b221c2d7cf991deb7dd0bf3afea1ca3e55366310e7648f3668854",usage_total=397394i 1483799611000000000
> docker_container_mem,container_image=gcr.io/google_containers/pause-amd64,container_name=k8s_POD.d8dbe16c_weave-net-f7cch_kube-system_1f393821-d197-11e6-af2d-0a209ce05b0a_9215c9a5,container_version=3.0,engine_host=ip-192-168-97-124,host=ip-192-168-97-124.ec2.internal,infrastructure=kubernetes,io.kubernetes.container.hash=d8dbe16c,io.kubernetes.container.name=POD,io.kubernetes.container.restartCount=0,io.kubernetes.pod.name=weave-net-f7cch,io.kubernetes.pod.namespace=kube-system,io.kubernetes.pod.terminationGracePeriod=30,io.kubernetes.pod.uid=1f393821-d197-11e6-af2d-0a209ce05b0a,namespace=production active_anon=40960i,active_file=4096i,cache=4096i,container_id="bdab1d60a3abac5595ce56c68dc68a46aeadb8775c56611c6e5cea6f0e07d16e",fail_count=0i,hierarchical_memory_limit=9223372036854771712i,inactive_anon=0i,inactive_file=0i,limit=8374394880i,mapped_file=0i,max_usage=598016i,pgfault=108i,pgmajfault=0i,pgpgin=97i,pgpgout=0i,rss=40960i,rss_huge=0i,total_active_anon=40960i,total_active_file=4096i,total_cache=4096i,total_inactive_anon=0i,total_inactive_file=0i,total_mapped_file=0i,total_pgfault=108i,total_pgmafault=0i,total_pgpgin=97i,total_pgpgout=86i,total_rss=40960i,total_rss_huge=0i,total_unevictable=0i,total_writeback=0i,unevictable=0i,usage=45056i,usage_percent=0.0005380209632531681,writeback=0i 1483799611000000000
> docker_container_cpu,container_image=gcr.io/google_containers/pause-amd64,container_name=k8s_POD.d8dbe16c_weave-net-f7cch_kube-system_1f393821-d197-11e6-af2d-0a209ce05b0a_9215c9a5,container_version=3.0,cpu=cpu-total,engine_host=ip-192-168-97-124,host=ip-192-168-97-124.ec2.internal,infrastructure=kubernetes,io.kubernetes.container.hash=d8dbe16c,io.kubernetes.container.name=POD,io.kubernetes.container.restartCount=0,io.kubernetes.pod.name=weave-net-f7cch,io.kubernetes.pod.namespace=kube-system,io.kubernetes.pod.terminationGracePeriod=30,io.kubernetes.pod.uid=1f393821-d197-11e6-af2d-0a209ce05b0a,namespace=production container_id="bdab1d60a3abac5595ce56c68dc68a46aeadb8775c56611c6e5cea6f0e07d16e",throttling_periods=0i,throttling_throttled_periods=0i,throttling_throttled_time=0i,usage_in_kernelmode=0i,usage_in_usermode=0i,usage_percent=0,usage_system=719882790000000i,usage_total=5437260i 1483799611000000000
> docker_container_cpu,container_image=gcr.io/google_containers/pause-amd64,container_name=k8s_POD.d8dbe16c_weave-net-f7cch_kube-system_1f393821-d197-11e6-af2d-0a209ce05b0a_9215c9a5,container_version=3.0,cpu=cpu0,engine_host=ip-192-168-97-124,host=ip-192-168-97-124.ec2.internal,infrastructure=kubernetes,io.kubernetes.container.hash=d8dbe16c,io.kubernetes.container.name=POD,io.kubernetes.container.restartCount=0,io.kubernetes.pod.name=weave-net-f7cch,io.kubernetes.pod.namespace=kube-system,io.kubernetes.pod.terminationGracePeriod=30,io.kubernetes.pod.uid=1f393821-d197-11e6-af2d-0a209ce05b0a,namespace=production container_id="bdab1d60a3abac5595ce56c68dc68a46aeadb8775c56611c6e5cea6f0e07d16e",usage_total=1071894i 1483799611000000000
> docker_container_cpu,container_image=gcr.io/google_containers/pause-amd64,container_name=k8s_POD.d8dbe16c_weave-net-f7cch_kube-system_1f393821-d197-11e6-af2d-0a209ce05b0a_9215c9a5,container_version=3.0,cpu=cpu1,engine_host=ip-192-168-97-124,host=ip-192-168-97-124.ec2.internal,infrastructure=kubernetes,io.kubernetes.container.hash=d8dbe16c,io.kubernetes.container.name=POD,io.kubernetes.container.restartCount=0,io.kubernetes.pod.name=weave-net-f7cch,io.kubernetes.pod.namespace=kube-system,io.kubernetes.pod.terminationGracePeriod=30,io.kubernetes.pod.uid=1f393821-d197-11e6-af2d-0a209ce05b0a,namespace=production container_id="bdab1d60a3abac5595ce56c68dc68a46aeadb8775c56611c6e5cea6f0e07d16e",usage_total=4365366i 1483799611000000000
> docker_container_mem,container_image=gcr.io/google_containers/pause-amd64,container_name=k8s_POD.d8dbe16c_kube-proxy-ip-192-168-97-124.ec2.internal_kube-system_49a4db0a459cac1a65ee4c30588c05dc_518d613e,container_version=3.0,engine_host=ip-192-168-97-124,host=ip-192-168-97-124.ec2.internal,infrastructure=kubernetes,io.kubernetes.container.hash=d8dbe16c,io.kubernetes.container.name=POD,io.kubernetes.container.restartCount=0,io.kubernetes.pod.name=kube-proxy-ip-192-168-97-124.ec2.internal,io.kubernetes.pod.namespace=kube-system,io.kubernetes.pod.terminationGracePeriod=30,io.kubernetes.pod.uid=49a4db0a459cac1a65ee4c30588c05dc,namespace=production active_anon=36864i,active_file=122880i,cache=126976i,container_id="ebc8b6c30d3bc30c31d86fcf96de1159fc531a398ed3e1039e3878a031c8d35d",fail_count=0i,hierarchical_memory_limit=9223372036854771712i,inactive_anon=0i,inactive_file=4096i,limit=8374394880i,mapped_file=0i,max_usage=671744i,pgfault=101i,pgmajfault=1i,pgpgin=113i,pgpgout=0i,rss=36864i,rss_huge=0i,total_active_anon=36864i,total_active_file=122880i,total_cache=126976i,total_inactive_anon=0i,total_inactive_file=4096i,total_mapped_file=0i,total_pgfault=101i,total_pgmafault=1i,total_pgpgin=113i,total_pgpgout=73i,total_rss=36864i,total_rss_huge=0i,total_unevictable=0i,total_writeback=0i,unevictable=0i,usage=163840i,usage_percent=0.0019564398663751573,writeback=0i 1483799611000000000
> docker_container_cpu,container_image=gcr.io/google_containers/pause-amd64,container_name=k8s_POD.d8dbe16c_kube-proxy-ip-192-168-97-124.ec2.internal_kube-system_49a4db0a459cac1a65ee4c30588c05dc_518d613e,container_version=3.0,cpu=cpu-total,engine_host=ip-192-168-97-124,host=ip-192-168-97-124.ec2.internal,infrastructure=kubernetes,io.kubernetes.container.hash=d8dbe16c,io.kubernetes.container.name=POD,io.kubernetes.container.restartCount=0,io.kubernetes.pod.name=kube-proxy-ip-192-168-97-124.ec2.internal,io.kubernetes.pod.namespace=kube-system,io.kubernetes.pod.terminationGracePeriod=30,io.kubernetes.pod.uid=49a4db0a459cac1a65ee4c30588c05dc,namespace=production container_id="ebc8b6c30d3bc30c31d86fcf96de1159fc531a398ed3e1039e3878a031c8d35d",throttling_periods=0i,throttling_throttled_periods=0i,throttling_throttled_time=0i,usage_in_kernelmode=0i,usage_in_usermode=0i,usage_percent=0,usage_system=719882790000000i,usage_total=7597776i 1483799611000000000
> docker_container_cpu,container_image=gcr.io/google_containers/pause-amd64,container_name=k8s_POD.d8dbe16c_kube-proxy-ip-192-168-97-124.ec2.internal_kube-system_49a4db0a459cac1a65ee4c30588c05dc_518d613e,container_version=3.0,cpu=cpu0,engine_host=ip-192-168-97-124,host=ip-192-168-97-124.ec2.internal,infrastructure=kubernetes,io.kubernetes.container.hash=d8dbe16c,io.kubernetes.container.name=POD,io.kubernetes.container.restartCount=0,io.kubernetes.pod.name=kube-proxy-ip-192-168-97-124.ec2.internal,io.kubernetes.pod.namespace=kube-system,io.kubernetes.pod.terminationGracePeriod=30,io.kubernetes.pod.uid=49a4db0a459cac1a65ee4c30588c05dc,namespace=production container_id="ebc8b6c30d3bc30c31d86fcf96de1159fc531a398ed3e1039e3878a031c8d35d",usage_total=6979363i 1483799611000000000
> docker_container_cpu,container_image=gcr.io/google_containers/pause-amd64,container_name=k8s_POD.d8dbe16c_kube-proxy-ip-192-168-97-124.ec2.internal_kube-system_49a4db0a459cac1a65ee4c30588c05dc_518d613e,container_version=3.0,cpu=cpu1,engine_host=ip-192-168-97-124,host=ip-192-168-97-124.ec2.internal,infrastructure=kubernetes,io.kubernetes.container.hash=d8dbe16c,io.kubernetes.container.name=POD,io.kubernetes.container.restartCount=0,io.kubernetes.pod.name=kube-proxy-ip-192-168-97-124.ec2.internal,io.kubernetes.pod.namespace=kube-system,io.kubernetes.pod.terminationGracePeriod=30,io.kubernetes.pod.uid=49a4db0a459cac1a65ee4c30588c05dc,namespace=production container_id="ebc8b6c30d3bc30c31d86fcf96de1159fc531a398ed3e1039e3878a031c8d35d",usage_total=618413i 1483799611000000000
> docker_container_blkio,container_image=gcr.io/google_containers/pause-amd64,container_name=k8s_POD.d8dbe16c_kube-proxy-ip-192-168-97-124.ec2.internal_kube-system_49a4db0a459cac1a65ee4c30588c05dc_518d613e,container_version=3.0,device=202:0,engine_host=ip-192-168-97-124,host=ip-192-168-97-124.ec2.internal,infrastructure=kubernetes,io.kubernetes.container.hash=d8dbe16c,io.kubernetes.container.name=POD,io.kubernetes.container.restartCount=0,io.kubernetes.pod.name=kube-proxy-ip-192-168-97-124.ec2.internal,io.kubernetes.pod.namespace=kube-system,io.kubernetes.pod.terminationGracePeriod=30,io.kubernetes.pod.uid=49a4db0a459cac1a65ee4c30588c05dc,namespace=production container_id="ebc8b6c30d3bc30c31d86fcf96de1159fc531a398ed3e1039e3878a031c8d35d",io_service_bytes_recursive_async=118784i,io_service_bytes_recursive_read=118784i,io_service_bytes_recursive_sync=0i,io_service_bytes_recursive_total=118784i,io_service_bytes_recursive_write=0i,io_serviced_recursive_async=1i,io_serviced_recursive_read=1i,io_serviced_recursive_sync=0i,io_serviced_recursive_total=1i,io_serviced_recursive_write=0i 1483799611000000000
2017/01/07 14:33:35 E! Error gathering container [/k8s_kubectl-proxy.e48a0908_telegraf-aa5q5_production_24e28a0c-d4e4-11e6-af2d-0a209ce05b0a_28b88670] stats: Error getting docker stats: An error occurred trying to connect: context deadline exceeded
2017/01/07 14:33:35 E! Error gathering container [/k8s_telegraf.1c6b442d_telegraf-aa5q5_production_24e28a0c-d4e4-11e6-af2d-0a209ce05b0a_65a778f6] stats: Error getting docker stats: An error occurred trying to connect: context deadline exceeded
2017/01/07 14:33:35 E! Error gathering container [/k8s_weave-npc.85b894ce_weave-net-f7cch_kube-system_1f393821-d197-11e6-af2d-0a209ce05b0a_4d5d053e] stats: Error getting docker stats: An error occurred trying to connect: context deadline exceeded
2017/01/07 14:33:35 E! Error gathering container [/k8s_weave.71dbac7f_weave-net-f7cch_kube-system_1f393821-d197-11e6-af2d-0a209ce05b0a_d84f78f5] stats: Error getting docker stats: An error occurred trying to connect: context deadline exceeded
2017/01/07 14:33:35 E! Error gathering container [/k8s_kube-apiserver.c6947346_kube-apiserver-ip-192-168-97-124.ec2.internal_kube-system_39748d1ca3f847081acfff1e2913a43e_30f31f58] stats: Error getting docker stats: An error occurred trying to connect: context deadline exceeded
2017/01/07 14:33:35 E! Error gathering container [/k8s_etcd-container.5fcb3aeb_etcd-server-events-ip-192-168-97-124.ec2.internal_kube-system_2b310f818153713c2adba2fa1bbe1a20_2fbc5caf] stats: Error getting docker stats: An error occurred trying to connect: context deadline exceeded
2017/01/07 14:33:35 E! Error gathering container [/k8s_kube-scheduler.95470df9_kube-scheduler-ip-192-168-97-124.ec2.internal_kube-system_b94e9fc6e899eb4a4b5b1f94be012ac2_3b1052b9] stats: Error getting docker stats: An error occurred trying to connect: context deadline exceeded
2017/01/07 14:33:35 E! Error gathering container [/k8s_etcd-container.5bb319d1_etcd-server-ip-192-168-97-124.ec2.internal_kube-system_397cce481a573eafc904eb7b68940825_ccf59f3d] stats: Error getting docker stats: An error occurred trying to connect: context deadline exceeded
2017/01/07 14:33:35 E! Error gathering container [/k8s_kube-proxy.96950a4a_kube-proxy-ip-192-168-97-124.ec2.internal_kube-system_49a4db0a459cac1a65ee4c30588c05dc_f6de0520] stats: Error getting docker stats: An error occurred trying to connect: context deadline exceeded
2017/01/07 14:33:35 E! Error gathering container [/k8s_kube-controller-manager.c4c5cf2c_kube-controller-manager-ip-192-168-97-124.ec2.internal_kube-system_f0713bd8968299c1008273c9b6b0896c_f7907087] stats: Error getting docker stats: An error occurred trying to connect: context deadline exceeded
* Plugin: inputs.kubernetes, Collection 1
> kubernetes_node,host=ip-192-168-97-124.ec2.internal,infrastructure=kubernetes,namespace=production,node_name=ip-192-168-97-124.ec2.internal cpu_usage_core_nanoseconds=27076684087116i,cpu_usage_nanocores=107033127i,fs_available_bytes=14148374528i,fs_capacity_bytes=19988041728i,fs_used_bytes=4878471168i,memory_available_bytes=5352833024i,memory_major_page_faults=1609i,memory_page_faults=1359778212i,memory_rss_bytes=291483648i,memory_usage_bytes=5142806528i,memory_working_set_bytes=3021561856i,network_rx_bytes=7661019521i,network_rx_errors=0i,network_tx_bytes=6962459573i,network_tx_errors=0i,runtime_image_fs_available_bytes=14148374528i,runtime_image_fs_capacity_bytes=19988041728i,runtime_image_fs_used_bytes=1471873736i 1483799615000000000
> kubernetes_pod_container,container_name=etcd-container,host=ip-192-168-97-124.ec2.internal,infrastructure=kubernetes,namespace=kube-system,node_name=ip-192-168-97-124.ec2.internal,pod_name=etcd-server-events-ip-192-168-97-124.ec2.internal cpu_usage_core_nanoseconds=5379665672i,cpu_usage_nanocores=0i,logsfs_avaialble_bytes=14148374528i,logsfs_capacity_bytes=19988041728i,logsfs_used_bytes=16384i,memory_major_page_faults=0i,memory_page_faults=8641i,memory_rss_bytes=11378688i,memory_usage_bytes=13639680i,memory_working_set_bytes=13639680i,rootfs_available_bytes=14148374528i,rootfs_capacity_bytes=19988041728i,rootfs_used_bytes=65536i 1483799615000000000
> kubernetes_pod_network,host=ip-192-168-97-124.ec2.internal,infrastructure=kubernetes,namespace=kube-system,node_name=ip-192-168-97-124.ec2.internal,pod_name=etcd-server-events-ip-192-168-97-124.ec2.internal rx_bytes=7661249990i,rx_errors=0i,tx_bytes=6962906723i,tx_errors=0i 1483799615000000000
> kubernetes_pod_container,container_name=kube-controller-manager,host=ip-192-168-97-124.ec2.internal,infrastructure=kubernetes,namespace=kube-system,node_name=ip-192-168-97-124.ec2.internal,pod_name=kube-controller-manager-ip-192-168-97-124.ec2.internal cpu_usage_core_nanoseconds=283139512i,cpu_usage_nanocores=0i,logsfs_avaialble_bytes=14148374528i,logsfs_capacity_bytes=19988041728i,logsfs_used_bytes=16384i,memory_major_page_faults=0i,memory_page_faults=2189i,memory_rss_bytes=5214208i,memory_usage_bytes=5259264i,memory_working_set_bytes=5259264i,rootfs_available_bytes=14148374528i,rootfs_capacity_bytes=19988041728i,rootfs_used_bytes=122880i 1483799615000000000
> kubernetes_pod_network,host=ip-192-168-97-124.ec2.internal,infrastructure=kubernetes,namespace=kube-system,node_name=ip-192-168-97-124.ec2.internal,pod_name=kube-controller-manager-ip-192-168-97-124.ec2.internal rx_bytes=7661243422i,rx_errors=0i,tx_bytes=6962900295i,tx_errors=0i 1483799615000000000
> kubernetes_pod_container,container_name=kube-scheduler,host=ip-192-168-97-124.ec2.internal,infrastructure=kubernetes,namespace=kube-system,node_name=ip-192-168-97-124.ec2.internal,pod_name=kube-scheduler-ip-192-168-97-124.ec2.internal cpu_usage_core_nanoseconds=1091290342i,cpu_usage_nanocores=0i,logsfs_avaialble_bytes=14148374528i,logsfs_capacity_bytes=19988041728i,logsfs_used_bytes=12288i,memory_major_page_faults=0i,memory_page_faults=2422i,memory_rss_bytes=6860800i,memory_usage_bytes=6873088i,memory_working_set_bytes=6873088i,rootfs_available_bytes=14148374528i,rootfs_capacity_bytes=19988041728i,rootfs_used_bytes=57344i 1483799615000000000
> kubernetes_pod_network,host=ip-192-168-97-124.ec2.internal,infrastructure=kubernetes,namespace=kube-system,node_name=ip-192-168-97-124.ec2.internal,pod_name=kube-scheduler-ip-192-168-97-124.ec2.internal rx_bytes=7660987454i,rx_errors=0i,tx_bytes=6962406061i,tx_errors=0i 1483799615000000000
> kubernetes_pod_container,container_name=weave-npc,host=ip-192-168-97-124.ec2.internal,infrastructure=kubernetes,namespace=kube-system,node_name=ip-192-168-97-124.ec2.internal,pod_name=weave-net-f7cch cpu_usage_core_nanoseconds=886090253i,cpu_usage_nanocores=0i,logsfs_avaialble_bytes=14148374528i,logsfs_capacity_bytes=19988041728i,logsfs_used_bytes=3256320i,memory_major_page_faults=0i,memory_page_faults=19408i,memory_rss_bytes=29483008i,memory_usage_bytes=29536256i,memory_working_set_bytes=29528064i,rootfs_available_bytes=14148374528i,rootfs_capacity_bytes=19988041728i,rootfs_used_bytes=77824i 1483799615000000000
> kubernetes_pod_container,container_name=weave,host=ip-192-168-97-124.ec2.internal,infrastructure=kubernetes,namespace=kube-system,node_name=ip-192-168-97-124.ec2.internal,pod_name=weave-net-f7cch cpu_usage_core_nanoseconds=1791071747i,cpu_usage_nanocores=0i,logsfs_avaialble_bytes=14148374528i,logsfs_capacity_bytes=19988041728i,logsfs_used_bytes=77824i,memory_major_page_faults=0i,memory_page_faults=37799i,memory_rss_bytes=33513472i,memory_usage_bytes=46321664i,memory_working_set_bytes=41607168i,rootfs_available_bytes=14148374528i,rootfs_capacity_bytes=19988041728i,rootfs_used_bytes=94208i 1483799615000000000
> kubernetes_pod_volume,host=ip-192-168-97-124.ec2.internal,infrastructure=kubernetes,namespace=kube-system,node_name=ip-192-168-97-124.ec2.internal,pod_name=weave-net-f7cch,volume_name=default-token-cz82a available_bytes=4187185152i,capacity_bytes=4187197440i,used_bytes=12288i 1483799615000000000
> kubernetes_pod_volume,host=ip-192-168-97-124.ec2.internal,infrastructure=kubernetes,namespace=kube-system,node_name=ip-192-168-97-124.ec2.internal,pod_name=weave-net-f7cch,volume_name=weavedb available_bytes=14148374528i,capacity_bytes=19988041728i,used_bytes=28672i 1483799615000000000
> kubernetes_pod_network,host=ip-192-168-97-124.ec2.internal,infrastructure=kubernetes,namespace=kube-system,node_name=ip-192-168-97-124.ec2.internal,pod_name=weave-net-f7cch rx_bytes=7661041474i,rx_errors=0i,tx_bytes=6962502070i,tx_errors=0i 1483799615000000000
> kubernetes_pod_container,container_name=kube-proxy,host=ip-192-168-97-124.ec2.internal,infrastructure=kubernetes,namespace=kube-system,node_name=ip-192-168-97-124.ec2.internal,pod_name=kube-proxy-ip-192-168-97-124.ec2.internal cpu_usage_core_nanoseconds=9811826226i,cpu_usage_nanocores=0i,logsfs_avaialble_bytes=14148374528i,logsfs_capacity_bytes=19988041728i,logsfs_used_bytes=12288i,memory_major_page_faults=0i,memory_page_faults=1262733i,memory_rss_bytes=2473984i,memory_usage_bytes=2502656i,memory_working_set_bytes=2502656i,rootfs_available_bytes=14148374528i,rootfs_capacity_bytes=19988041728i,rootfs_used_bytes=61440i 1483799615000000000
> kubernetes_pod_network,host=ip-192-168-97-124.ec2.internal,infrastructure=kubernetes,namespace=kube-system,node_name=ip-192-168-97-124.ec2.internal,pod_name=kube-proxy-ip-192-168-97-124.ec2.internal rx_bytes=7661221048i,rx_errors=0i,tx_bytes=6962832805i,tx_errors=0i 1483799615000000000
> kubernetes_pod_container,container_name=telegraf,host=ip-192-168-97-124.ec2.internal,infrastructure=kubernetes,namespace=production,node_name=ip-192-168-97-124.ec2.internal,pod_name=telegraf-aa5q5 cpu_usage_core_nanoseconds=2038676364i,cpu_usage_nanocores=1588564i,logsfs_avaialble_bytes=14148374528i,logsfs_capacity_bytes=19988041728i,logsfs_used_bytes=331776i,memory_major_page_faults=0i,memory_page_faults=37605i,memory_rss_bytes=10006528i,memory_usage_bytes=10264576i,memory_working_set_bytes=10264576i,rootfs_available_bytes=14148374528i,rootfs_capacity_bytes=19988041728i,rootfs_used_bytes=69632i 1483799615000000000
> kubernetes_pod_container,container_name=kubectl-proxy,host=ip-192-168-97-124.ec2.internal,infrastructure=kubernetes,namespace=production,node_name=ip-192-168-97-124.ec2.internal,pod_name=telegraf-aa5q5 cpu_usage_core_nanoseconds=130967093i,cpu_usage_nanocores=30749i,logsfs_avaialble_bytes=14148374528i,logsfs_capacity_bytes=19988041728i,logsfs_used_bytes=16384i,memory_major_page_faults=0i,memory_page_faults=2031i,memory_rss_bytes=4321280i,memory_usage_bytes=4349952i,memory_working_set_bytes=4349952i,rootfs_available_bytes=14148374528i,rootfs_capacity_bytes=19988041728i,rootfs_used_bytes=53248i 1483799615000000000
> kubernetes_pod_volume,host=ip-192-168-97-124.ec2.internal,infrastructure=kubernetes,namespace=production,node_name=ip-192-168-97-124.ec2.internal,pod_name=telegraf-aa5q5,volume_name=telegraf-token-k3x0u available_bytes=4187185152i,capacity_bytes=4187197440i,used_bytes=12288i 1483799615000000000
> kubernetes_pod_network,host=ip-192-168-97-124.ec2.internal,infrastructure=kubernetes,namespace=production,node_name=ip-192-168-97-124.ec2.internal,pod_name=telegraf-aa5q5 rx_bytes=1401294i,rx_errors=0i,tx_bytes=4587653i,tx_errors=0i 1483799615000000000
> kubernetes_pod_container,container_name=kube-apiserver,host=ip-192-168-97-124.ec2.internal,infrastructure=kubernetes,namespace=kube-system,node_name=ip-192-168-97-124.ec2.internal,pod_name=kube-apiserver-ip-192-168-97-124.ec2.internal cpu_usage_core_nanoseconds=4311812903i,cpu_usage_nanocores=0i,logsfs_avaialble_bytes=14148374528i,logsfs_capacity_bytes=19988041728i,logsfs_used_bytes=16384i,memory_major_page_faults=0i,memory_page_faults=16640i,memory_rss_bytes=54161408i,memory_usage_bytes=54247424i,memory_working_set_bytes=54247424i,rootfs_available_bytes=14148374528i,rootfs_capacity_bytes=19988041728i,rootfs_used_bytes=122880i 1483799615000000000
> kubernetes_pod_network,host=ip-192-168-97-124.ec2.internal,infrastructure=kubernetes,namespace=kube-system,node_name=ip-192-168-97-124.ec2.internal,pod_name=kube-apiserver-ip-192-168-97-124.ec2.internal rx_bytes=7661238183i,rx_errors=0i,tx_bytes=6962892036i,tx_errors=0i 1483799615000000000
> kubernetes_pod_container,container_name=etcd-container,host=ip-192-168-97-124.ec2.internal,infrastructure=kubernetes,namespace=kube-system,node_name=ip-192-168-97-124.ec2.internal,pod_name=etcd-server-ip-192-168-97-124.ec2.internal cpu_usage_core_nanoseconds=6286557652i,cpu_usage_nanocores=0i,logsfs_avaialble_bytes=14148374528i,logsfs_capacity_bytes=19988041728i,logsfs_used_bytes=16384i,memory_major_page_faults=0i,memory_page_faults=9043i,memory_rss_bytes=14024704i,memory_usage_bytes=14086144i,memory_working_set_bytes=14077952i,rootfs_available_bytes=14148374528i,rootfs_capacity_bytes=19988041728i,rootfs_used_bytes=65536i 1483799615000000000
> kubernetes_pod_network,host=ip-192-168-97-124.ec2.internal,infrastructure=kubernetes,namespace=kube-system,node_name=ip-192-168-97-124.ec2.internal,pod_name=etcd-server-ip-192-168-97-124.ec2.internal rx_bytes=7661057585i,rx_errors=0i,tx_bytes=6962577533i,tx_errors=0i 1483799615000000000

Additional info:

Willing to provide any more information that I might have not provided and is required to debug further.
@itskingori
Copy link
Author

(edited description)

@itskingori itskingori changed the title Context deadline exceeded error with Kubernetes plugin Context deadline exceeded error with Docker plugin Jan 7, 2017
@itskingori
Copy link
Author

I've edited the description of the issue (again), provided a better bug report and thereby changed the title to reflect the new discovery. 🤓

Initially I thought the issue was the Kubernetes plugin but now can conclude with reasonable confidence that it's the Docker plugin (as seen in the telegraf --test output).

Which now leads me to issue #1195 ... seems I'm having the same problem (yet there was a fix added by @sparrc that was/should have been merged in in v0.13. 😰

@sparrc
Copy link
Contributor

sparrc commented Jan 9, 2017

this is unrelated to #1195, did you try increasing the timeout parameter in your docker plugin configuration?

@itskingori
Copy link
Author

@sparrc Yes, I did. I set it to 10 seconds and then used with telegraf --test. In hindsight I don't think that would really capture a change in timeout 🤔🙈 .

I hope you don't mind me leaving this open for a little while longer ... so I can change the timeout and leave it running on the cluster (actual deployment as opposed to a --test test).

@sparrc
Copy link
Contributor

sparrc commented Jan 24, 2017

I'm going to close this for lack of updates, if you have any more details and still think it's an issue please feel free to open a new one

@sparrc sparrc closed this as completed Jan 24, 2017
@itskingori
Copy link
Author

Sure. Sorry about that ... forgot to report back.

I changed timeout to 10s but issue persisted. Opted to leave out Docker input and Kubernetes plugin gave me much of what I needed.

@sybrandy
Copy link

sybrandy commented Mar 6, 2017

I just saw this same issue today using version 1.2, however I don't have any additional information in terms of stack traces as it apparently tried to access an invalid memory location and segfaulted. This was a random occurrence, so I'm not sure if simply changing the timeout will make a difference.

@xuqingfeng
Copy link

Same here.

2018-01-04T04:03:30Z E! Error in plugin [inputs.docker]: E! Error gathering container [/auth-proxy] stats: Error getting docker stats: context deadline exceeded
---
$ telegraf version
Telegraf v1.3.5 (git: release-1.3 7192e68b2423997177692834f53cdf171aee1a88)

PS. docker plugin works days ago in this host and still works in other hosts.

@ghost
Copy link

ghost commented Nov 13, 2018

have the same with Telegraf 1.8.1 (git: HEAD ae9efb2f)

Nov 12 16:25:50 myhost.dev telegraf[855]: 2018-11-12T16:25:50Z E! Error in plugin [inputs.docker]: E! Error gathering container [/test_container] stats: Error getting docker stats: context deadline exceeded

plugin config:

[[inputs.docker]]
  endpoint = "unix:///var/run/docker.sock"
  gather_services = false
  container_names = []
  container_name_include = []
  container_name_exclude = []
  timeout = "5s"
  perdevice = true
  total = false
  docker_label_include = []
  docker_label_exclude = []
  tag_env = []

it works for some time and then get sutck.
restart helps for few days

@danielnelson danielnelson reopened this Nov 13, 2018
@danielnelson danielnelson added bug unexpected problem or unintended behavior area/docker and removed need more info labels Nov 13, 2018
@robermorales
Copy link

same here,

some news??

@danielnelson
Copy link
Contributor

I believe this error indicates a timeout talking to the docker api, can you try setting the timeout higher? You may be overloading the docker engine, it also might help to increase the interval or limit the containers that are matching.

@danielnelson
Copy link
Contributor

As a fix let's change the log message to better indicate a timeout.

@danielnelson danielnelson added this to the 1.10.0 milestone Nov 28, 2018
@danielnelson danielnelson modified the milestones: 1.10.0, 1.11.0 Feb 4, 2019
@danielnelson danielnelson modified the milestones: 1.11.0, 1.12.0 May 24, 2019
@goller goller mentioned this issue Jun 20, 2019
Merged
3 tasks
@goller goller self-assigned this Jun 20, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@goller goller

Labels
area/docker bug unexpected problem or unintended behavior
Projects
None yet
Milestone
1.12.0
Development

Successfully merging a pull request may close this issue.

feat(inputs/docker): add better user-facing errors for API timeouts influxdata/telegraf
7 participants
@robermorales @itskingori @danielnelson @sybrandy @goller @xuqingfeng @sparrc