Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

High CPU consumption with JMX errors #313

Closed
alq666 opened this issue Dec 23, 2012 · 13 comments
Closed

High CPU consumption with JMX errors #313

alq666 opened this issue Dec 23, 2012 · 13 comments
Assignees
@remh
Labels
checks request
Milestone
3.4.x

Comments

@alq666
Copy link
Member

alq666 commented Dec 23, 2012

from the collector log

Dec 22 02:20:32 rig dd-agent - root - INFO - Logging to syslog is set up
Dec 22 02:20:32 rig dd-agent - root - INFO - Running in foreground
Dec 22 02:20:32 rig dd-agent - checks - INFO - Dogstream parsers: []
Dec 22 02:20:32 rig dd-agent - checks - WARNING - Configuring tomcat in datadog.conf is deprecated. Please use conf.d. In a future release, support for the old style of configuration will be dropped.
Dec 22 02:20:32 rig dd-agent - checks - WARNING - Configuring postgres in datadog.conf is deprecated. Please use conf.d. In a future release, support for the old style of configuration will be dropped.
Dec 22 02:20:32 rig dd-agent - checks - WARNING - Configuring jmx in datadog.conf is deprecated. Please use conf.d. In a future release, support for the old style of configuration will be dropped.
Dec 22 02:20:32 rig dd-agent - checks - INFO - checks.d checks: ['tomcat', 'postgres', 'jmx']
Dec 22 02:20:32 rig dd-agent - collector - INFO - Starting collection run #1
Dec 22 02:20:41 rig dd-agent - checks.tomcat - CRITICAL - Error while fetching JVM metrics <class 'pexpect.EOF'>
Dec 22 02:20:41 rig dd-agent - checks.tomcat - ERROR - Check 'tomcat' instance #0 failed
Traceback (most recent call last):
  File "/usr/share/datadog/agent/checks/__init__.py", line 423, in run
    self.check(instance)
  File "/usr/share/datadog/agent/checks.d/tomcat.py", line 45, in check
    (host, port, user, password, jmx, instance_name) = self._load_config(instance)
  File "/usr/share/datadog/agent/checks/jmx_connector.py", line 314, in _load_config
    jmx = connect()
  File "/usr/share/datadog/agent/checks/jmx_connector.py", line 309, in connect
    jmx.connect("%s:%s" % (host, port), user, password)
  File "/usr/share/datadog/agent/checks/jmx_connector.py", line 99, in connect
    raise Exception('Error while fetching JVM metrics at attdress: %s:%s' % (connection, passwd))
Exception: Error while fetching JVM metrics at attdress: localhost:8090:XXX
Dec 22 02:20:41 rig dd-agent - agent - INFO - Exiting. Bye bye.
Dec 22 02:21:22 rig dd-agent - root - INFO - Logging to syslog is set up
Dec 22 02:21:22 rig dd-agent - root - INFO - Running in foreground
Dec 22 02:21:22 rig dd-agent - checks - INFO - Dogstream parsers: []
Dec 22 02:21:22 rig dd-agent - checks - WARNING - Configuring tomcat in datadog.conf is deprecated. Please use conf.d. In a future release, support for the old style of configuration will be dropped.
Dec 22 02:21:22 rig dd-agent - checks - WARNING - Configuring postgres in datadog.conf is deprecated. Please use conf.d. In a future release, support for the old style of configuration will be dropped.
Dec 22 02:21:22 rig dd-agent - checks - WARNING - Configuring jmx in datadog.conf is deprecated. Please use conf.d. In a future release, support for the old style of configuration will be dropped.
Dec 22 02:21:22 rig dd-agent - checks - INFO - checks.d checks: ['tomcat', 'postgres', 'jmx']
Dec 22 02:21:22 rig dd-agent - collector - INFO - Starting collection run #1
Dec 22 02:21:31 rig dd-agent - checks.tomcat - CRITICAL - Error while fetching JVM metrics <class 'pexpect.EOF'>
Dec 22 02:21:31 rig dd-agent - checks.tomcat - ERROR - Check 'tomcat' instance #0 failed
Traceback (most recent call last):
  File "/usr/share/datadog/agent/checks/__init__.py", line 423, in run
    self.check(instance)
  File "/usr/share/datadog/agent/checks.d/tomcat.py", line 45, in check
    (host, port, user, password, jmx, instance_name) = self._load_config(instance)
  File "/usr/share/datadog/agent/checks/jmx_connector.py", line 314, in _load_config
    jmx = connect()
  File "/usr/share/datadog/agent/checks/jmx_connector.py", line 309, in connect
    jmx.connect("%s:%s" % (host, port), user, password)
  File "/usr/share/datadog/agent/checks/jmx_connector.py", line 99, in connect
    raise Exception('Error while fetching JVM metrics at attdress: %s:%s' % (connection, passwd))
Exception: Error while fetching JVM metrics at attdress: localhost:8090:XXX
Dec 22 02:21:31 rig dd-agent - checks.postgres - ERROR - Check 'postgres' instance #0 failed
Traceback (most recent call last):
  File "/usr/share/datadog/agent/checks/__init__.py", line 423, in run
    self.check(instance)
  File "/usr/share/datadog/agent/checks.d/postgres.py", line 72, in check
    database='postgres')
  File "/usr/lib/python2.7/dist-packages/psycopg2/__init__.py", line 179, in connect
    connection_factory=connection_factory, async=async)
OperationalError: could not connect to server: Connection refused
Is the server running on host "localhost" (127.0.0.1) and accepting
TCP/IP connections on port 5432?
Dec 22 02:21:35 rig dd-agent - checks.jmx - CRITICAL - Error while fetching JVM metrics <class 'pexpect.EOF'>
Dec 22 02:21:35 rig dd-agent - checks.jmx - ERROR - Check 'jmx' instance #0 failed
Traceback (most recent call last):
  File "/usr/share/datadog/agent/checks/__init__.py", line 423, in run
    self.check(instance)
  File "/usr/share/datadog/agent/checks.d/jmx.py", line 25, in check
    (host, port, user, password, jmx, instance_name) = self._load_config(instance)
  File "/usr/share/datadog/agent/checks/jmx_connector.py", line 314, in _load_config
    jmx = connect()
  File "/usr/share/datadog/agent/checks/jmx_connector.py", line 309, in connect
    jmx.connect("%s:%s" % (host, port), user, password)
  File "/usr/share/datadog/agent/checks/jmx_connector.py", line 99, in connect
    raise Exception('Error while fetching JVM metrics at attdress: %s:%s' % (connection, passwd))
Exception: Error while fetching JVM metrics at attdress: localhost:8090:XXX
Dec 22 02:21:35 rig dd-agent - collector - INFO - Finished run #1. Collection time: 13.27s. Emit time: 0.01s
Dec 22 02:21:50 rig dd-agent - collector - INFO - Starting collection run #2
Dec 22 02:22:37 rig dd-agent - checks.jmx - ERROR - Check 'jmx' instance #0 failed
Traceback (most recent call last):
  File "/usr/share/datadog/agent/checks/__init__.py", line 423, in run
    self.check(instance)
  File "/usr/share/datadog/agent/checks.d/jmx.py", line 29, in check
    dump = jmx.dump()
  File "/usr/share/datadog/agent/checks/jmx_connector.py", line 129, in dump
    jsonvar = json.loads(content)
  File "/usr/lib/python2.7/json/__init__.py", line 326, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python2.7/json/decoder.py", line 366, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python2.7/json/decoder.py", line 384, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded
Dec 22 02:22:37 rig dd-agent - collector - INFO - Finished run #2. Collection time: 47.23s. Emit time: 0.01s
Dec 22 02:22:52 rig dd-agent - collector - INFO - Starting collection run #3
Dec 22 02:23:16 rig dd-agent - checks.tomcat - ERROR - Check 'tomcat' instance #0 failed
Traceback (most recent call last):
  File "/usr/share/datadog/agent/checks/__init__.py", line 423, in run
    self.check(instance)
  File "/usr/share/datadog/agent/checks.d/tomcat.py", line 49, in check
    dump = jmx.dump()
  File "/usr/share/datadog/agent/checks/jmx_connector.py", line 129, in dump
    jsonvar = json.loads(content)
  File "/usr/lib/python2.7/json/__init__.py", line 326, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python2.7/json/decoder.py", line 366, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python2.7/json/decoder.py", line 384, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded
Dec 22 02:23:36 rig dd-agent - checks.jmx - ERROR - Check 'jmx' instance #0 failed
Traceback (most recent call last):
  File "/usr/share/datadog/agent/checks/__init__.py", line 423, in run
    self.check(instance)
  File "/usr/share/datadog/agent/checks.d/jmx.py", line 29, in check
    dump = jmx.dump()
  File "/usr/share/datadog/agent/checks/jmx_connector.py", line 127, in dump
    self._wait_prompt()
  File "/usr/share/datadog/agent/checks/jmx_connector.py", line 61, in _wait_prompt
    self._jmx.expect_exact("$>") # got prompt, we can continue
  File "/usr/lib/python2.7/dist-packages/pexpect.py", line 1343, in expect_exact
    return self.expect_loop(searcher_string(pattern_list), timeout, searchwindowsize)
  File "/usr/lib/python2.7/dist-packages/pexpect.py", line 1409, in expect_loop
    raise TIMEOUT (str(e) + '\n' + str(self))
TIMEOUT: Timeout exceeded in read_nonblocking().
<pexpect.spawn object at 0x2263810>
version: 2.3 ($Revision: 399 $)
command: /usr/bin/java
args: ['/usr/bin/java', '-jar', '/usr/share/datadog/agent/checks/libs/jmxterm-1.0-DATADOG-uber.jar', '-l', 'localhost:8090', '-u', 'bigroad', '-p', 'XXX']
searcher: searcher_string:
    0: "$>"
buffer (last 100 chars): dump#015

before (last 100 chars): dump#015

after: <class 'pexpect.TIMEOUT'>
match: None
match_index: None
exitstatus: None
flag_eof: False
pid: 25990
child_fd: 7
closed: False
timeout: 20
delimiter: <class 'pexpect.EOF'>
logfile: None
logfile_read: None
logfile_send: None
maxread: 2000
ignorecase: False
searchwindowsize: None
delaybeforesend: 0
delayafterclose: 0.1
delayafterterminate: 0.1
Dec 22 02:23:36 rig dd-agent - collector - INFO - Finished run #3. Collection time: 43.7s. Emit time: 0.01s
Dec 22 02:23:51 rig dd-agent - collector - INFO - Starting collection run #4
Dec 22 02:24:15 rig dd-agent - checks.tomcat - ERROR - Check 'tomcat' instance #0 failed
Traceback (most recent call last):
  File "/usr/share/datadog/agent/checks/__init__.py", line 423, in run
    self.check(instance)
  File "/usr/share/datadog/agent/checks.d/tomcat.py", line 49, in check
    dump = jmx.dump()
  File "/usr/share/datadog/agent/checks/jmx_connector.py", line 127, in dump
    self._wait_prompt()
  File "/usr/share/datadog/agent/checks/jmx_connector.py", line 61, in _wait_prompt
    self._jmx.expect_exact("$>") # got prompt, we can continue
  File "/usr/lib/python2.7/dist-packages/pexpect.py", line 1343, in expect_exact
    return self.expect_loop(searcher_string(pattern_list), timeout, searchwindowsize)
  File "/usr/lib/python2.7/dist-packages/pexpect.py", line 1409, in expect_loop
    raise TIMEOUT (str(e) + '\n' + str(self))
TIMEOUT: Timeout exceeded in read_nonblocking().
<pexpect.spawn object at 0x20b56d0>
version: 2.3 ($Revision: 399 $)
command: /usr/bin/java
args: ['/usr/bin/java', '-jar', '/usr/share/datadog/agent/checks/libs/jmxterm-1.0-DATADOG-uber.jar', '-l', 'localhost:8090', '-u', 'bigroad', '-p', 'XXX']
searcher: searcher_string:
    0: "$>"
buffer (last 100 chars): dump#015

before (last 100 chars): dump#015

after: <class 'pexpect.TIMEOUT'>
match: None
match_index: None
exitstatus: None
flag_eof: False
pid: 25773
child_fd: 5
closed: False
timeout: 20
delimiter: <class 'pexpect.EOF'>
logfile: None
logfile_read: None
logfile_send: None
maxread: 2000
ignorecase: False
searchwindowsize: None
delaybeforesend: 0
delayafterclose: 0.1
delayafterterminate: 0.1
Dec 22 02:24:16 rig dd-agent - collector - INFO - Finished run #4. Collection time: 25.01s. Emit time: 0.01s
Dec 22 02:24:31 rig dd-agent - collector - INFO - Starting collection run #5
Dec 22 02:24:35 rig dd-agent - checks.tomcat - ERROR - Check 'tomcat' instance #0 failed
Traceback (most recent call last):
  File "/usr/share/datadog/agent/checks/__init__.py", line 423, in run
    self.check(instance)
  File "/usr/share/datadog/agent/checks.d/tomcat.py", line 49, in check
    dump = jmx.dump()
  File "/usr/share/datadog/agent/checks/jmx_connector.py", line 129, in dump
    jsonvar = json.loads(content)
  File "/usr/lib/python2.7/json/__init__.py", line 326, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python2.7/json/decoder.py", line 366, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python2.7/json/decoder.py", line 384, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded
Dec 22 02:24:35 rig dd-agent - checks.jmx - ERROR - Check 'jmx' instance #0 failed
Traceback (most recent call last):
  File "/usr/share/datadog/agent/checks/__init__.py", line 423, in run
    self.check(instance)
  File "/usr/share/datadog/agent/checks.d/jmx.py", line 29, in check
    dump = jmx.dump()
  File "/usr/share/datadog/agent/checks/jmx_connector.py", line 129, in dump
    jsonvar = json.loads(content)
  File "/usr/lib/python2.7/json/__init__.py", line 326, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python2.7/json/decoder.py", line 366, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python2.7/json/decoder.py", line 384, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded
Dec 22 02:24:35 rig dd-agent - collector - INFO - Finished run #5. Collection time: 4.09s. Emit time: 0.01s
Dec 22 02:24:50 rig dd-agent - collector - INFO - Starting collection run #6
Dec 22 02:25:03 rig dd-agent - collector - INFO - Finished run #6. Collection time: 13.35s. Emit time: 0.01s
Dec 22 02:25:18 rig dd-agent - collector - INFO - Starting collection run #7
Dec 22 02:25:29 rig dd-agent - collector - INFO - Finished run #7. Collection time: 10.05s. Emit time: 0.01s
Dec 22 02:25:44 rig dd-agent - collector - INFO - Starting collection run #8
Dec 22 02:25:48 rig dd-agent - checks.tomcat - ERROR - Check 'tomcat' instance #0 failed
Traceback (most recent call last):
  File "/usr/share/datadog/agent/checks/__init__.py", line 423, in run
    self.check(instance)
  File "/usr/share/datadog/agent/checks.d/tomcat.py", line 49, in check
    dump = jmx.dump()
  File "/usr/share/datadog/agent/checks/jmx_connector.py", line 129, in dump
    jsonvar = json.loads(content)
  File "/usr/lib/python2.7/json/__init__.py", line 326, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python2.7/json/decoder.py", line 366, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python2.7/json/decoder.py", line 384, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded
Dec 22 02:25:48 rig dd-agent - checks.jmx - ERROR - Check 'jmx' instance #0 failed
Traceback (most recent call last):
  File "/usr/share/datadog/agent/checks/__init__.py", line 423, in run
    self.check(instance)
  File "/usr/share/datadog/agent/checks.d/jmx.py", line 29, in check
    dump = jmx.dump()
  File "/usr/share/datadog/agent/checks/jmx_connector.py", line 129, in dump
    jsonvar = json.loads(content)
  File "/usr/lib/python2.7/json/__init__.py", line 326, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python2.7/json/decoder.py", line 366, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python2.7/json/decoder.py", line 384, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded
Dec 22 02:25:48 rig dd-agent - collector - INFO - Finished run #8. Collection time: 4.1s. Emit time: 0.01s
Dec 22 02:26:03 rig dd-agent - collector - INFO - Starting collection run #9
Dec 22 02:26:17 rig dd-agent - collector - INFO - Finished run #9. Collection time: 13.95s. Emit time: 0.01s
Dec 22 02:26:32 rig dd-agent - collector - INFO - Starting collection run #10
Dec 22 02:26:43 rig dd-agent - collector - INFO - Finished run #10. Collection time: 10.94s. Emit time: 0.01s
Dec 22 02:26:58 rig dd-agent - collector - INFO - Starting collection run #11
Dec 22 02:27:08 rig dd-agent - collector - INFO - Finished run #11. Collection time: 10.8s. Emit time: 0.01s
Dec 22 02:27:23 rig dd-agent - collector - INFO - Starting collection run #12
Dec 22 02:27:35 rig dd-agent - collector - INFO - Finished run #12. Collection time: 11.61s. Emit time: 0.01s
Dec 22 02:27:50 rig dd-agent - collector - INFO - Starting collection run #13
Dec 22 02:28:01 rig dd-agent - collector - INFO - Finished run #13. Collection time: 11.41s. Emit time: 0.01s
Dec 22 02:28:16 rig dd-agent - collector - INFO - Starting collection run #14
Dec 22 02:28:28 rig dd-agent - collector - INFO - Finished run #14. Collection time: 11.91s. Emit time: 0.01s
Dec 22 02:28:43 rig dd-agent - collector - INFO - Starting collection run #15
Dec 22 02:28:55 rig dd-agent - collector - INFO - Finished run #15. Collection time: 12.04s. Emit time: 0.01s
Dec 22 02:29:10 rig dd-agent - collector - INFO - Starting collection run #16
Dec 22 02:29:23 rig dd-agent - collector - INFO - Finished run #16. Collection time: 12.15s. Emit time: 0.01s
Dec 22 02:29:38 rig dd-agent - collector - INFO - Starting collection run #17
Dec 22 02:29:50 rig dd-agent - collector - INFO - Finished run #17. Collection time: 12.19s. Emit time: 0.01s
Dec 22 02:30:05 rig dd-agent - collector - INFO - Starting collection run #18
Dec 22 02:30:17 rig dd-agent - collector - INFO - Finished run #18. Collection time: 12.52s. Emit time: 0.01s
Dec 22 02:30:32 rig dd-agent - collector - INFO - Starting collection run #19
Dec 22 02:30:45 rig dd-agent - collector - INFO - Finished run #19. Collection time: 12.48s. Emit time: 0.01s
Dec 22 02:31:00 rig dd-agent - collector - INFO - Starting collection run #20
Dec 22 02:31:12 rig dd-agent - collector - INFO - Finished run #20. Collection time: 12.56s. Emit time: 0.01s
Dec 22 02:31:15 rig dd-agent - agent - INFO - Exiting. Bye bye.
Dec 22 02:31:53 rig dd-agent - root - INFO - Logging to syslog is set up
Dec 22 02:31:53 rig dd-agent - root - INFO - Running in foreground
Dec 22 02:31:53 rig dd-agent - checks - INFO - Dogstream parsers: []
Dec 22 02:31:54 rig dd-agent - checks - WARNING - Configuring tomcat in datadog.conf is deprecated. Please use conf.d. In a future release, support for the old style of configuration will be dropped.
Dec 22 02:31:54 rig dd-agent - checks - WARNING - Configuring postgres in datadog.conf is deprecated. Please use conf.d. In a future release, support for the old style of configuration will be dropped.
Dec 22 02:31:54 rig dd-agent - checks - WARNING - Configuring jmx in datadog.conf is deprecated. Please use conf.d. In a future release, support for the old style of configuration will be dropped.
Dec 22 02:31:54 rig dd-agent - checks - INFO - checks.d checks: ['tomcat', 'postgres', 'jmx']
Dec 22 02:31:54 rig dd-agent - collector - INFO - Starting collection run #1
Dec 22 02:32:23 rig dd-agent - checks.tomcat - ERROR - Check 'tomcat' instance #0 failed
Traceback (most recent call last):
  File "/usr/share/datadog/agent/checks/__init__.py", line 423, in run
    self.check(instance)
  File "/usr/share/datadog/agent/checks.d/tomcat.py", line 49, in check
    dump = jmx.dump()
  File "/usr/share/datadog/agent/checks/jmx_connector.py", line 127, in dump
    self._wait_prompt()
  File "/usr/share/datadog/agent/checks/jmx_connector.py", line 61, in _wait_prompt
    self._jmx.expect_exact("$>") # got prompt, we can continue
  File "/usr/lib/python2.7/dist-packages/pexpect.py", line 1343, in expect_exact
    return self.expect_loop(searcher_string(pattern_list), timeout, searchwindowsize)
  File "/usr/lib/python2.7/dist-packages/pexpect.py", line 1409, in expect_loop
    raise TIMEOUT (str(e) + '\n' + str(self))
TIMEOUT: Timeout exceeded in read_nonblocking().
<pexpect.spawn object at 0x11b9e10>
version: 2.3 ($Revision: 399 $)
command: /usr/bin/java
args: ['/usr/bin/java', '-jar', '/usr/share/datadog/agent/checks/libs/jmxterm-1.0-DATADOG-uber.jar', '-l', 'localhost:8090', '-u', 'bigroad', '-p', 'XXX']
searcher: searcher_string:
    0: "$>"
buffer (last 100 chars): dump#015

before (last 100 chars): dump#015

after: <class 'pexpect.TIMEOUT'>
match: None
match_index: None
exitstatus: None
flag_eof: False
pid: 26373
child_fd: 5
closed: False
timeout: 20
delimiter: <class 'pexpect.EOF'>
logfile: None
logfile_read: None
logfile_send: None
maxread: 2000
ignorecase: False
searchwindowsize: None
delaybeforesend: 0
delayafterclose: 0.1
delayafterterminate: 0.1
Dec 22 02:32:47 rig dd-agent - checks.jmx - ERROR - Check 'jmx' instance #0 failed
Traceback (most recent call last):
  File "/usr/share/datadog/agent/checks/__init__.py", line 423, in run
    self.check(instance)
  File "/usr/share/datadog/agent/checks.d/jmx.py", line 29, in check
    dump = jmx.dump()
  File "/usr/share/datadog/agent/checks/jmx_connector.py", line 127, in dump
    self._wait_prompt()
  File "/usr/share/datadog/agent/checks/jmx_connector.py", line 61, in _wait_prompt
    self._jmx.expect_exact("$>") # got prompt, we can continue
  File "/usr/lib/python2.7/dist-packages/pexpect.py", line 1343, in expect_exact
    return self.expect_loop(searcher_string(pattern_list), timeout, searchwindowsize)
  File "/usr/lib/python2.7/dist-packages/pexpect.py", line 1409, in expect_loop
    raise TIMEOUT (str(e) + '\n' + str(self))
TIMEOUT: Timeout exceeded in read_nonblocking().
<pexpect.spawn object at 0x1395610>
version: 2.3 ($Revision: 399 $)
command: /usr/bin/java
args: ['/usr/bin/java', '-jar', '/usr/share/datadog/agent/checks/libs/jmxterm-1.0-DATADOG-uber.jar', '-l', 'localhost:8090', '-u', 'bigroad', '-p', 'XXX']
searcher: searcher_string:
    0: "$>"
buffer (last 100 chars): dump#015

before (last 100 chars): dump#015

after: <class 'pexpect.TIMEOUT'>
match: None
match_index: None
exitstatus: None
flag_eof: False
pid: 26413
child_fd: 7
closed: False
timeout: 20
delimiter: <class 'pexpect.EOF'>
logfile: None
logfile_read: None
logfile_send: None
maxread: 2000
ignorecase: False
searchwindowsize: None
delaybeforesend: 0
delayafterclose: 0.1
delayafterterminate: 0.1
Dec 22 02:32:47 rig dd-agent - collector - INFO - Finished run #1. Collection time: 53.41s. Emit time: 0.09s
Dec 22 02:33:02 rig dd-agent - collector - INFO - Starting collection run #2
Dec 22 02:33:10 rig dd-agent - collector - INFO - Finished run #2. Collection time: 8.4s. Emit time: 0.01s
Dec 22 02:33:25 rig dd-agent - collector - INFO - Starting collection run #3
Dec 22 02:33:41 rig dd-agent - collector - INFO - Finished run #3. Collection time: 15.54s. Emit time: 0.01s
Dec 22 02:33:56 rig dd-agent - collector - INFO - Starting collection run #4
Dec 22 02:34:11 rig dd-agent - collector - INFO - Finished run #4. Collection time: 15.03s. Emit time: 0.01s
Dec 22 02:34:26 rig dd-agent - collector - INFO - Starting collection run #5
Dec 22 02:34:42 rig dd-agent - collector - INFO - Finished run #5. Collection time: 15.81s. Emit time: 0.01s
Dec 22 02:34:57 rig dd-agent - collector - INFO - Starting collection run #6
Dec 22 02:35:13 rig dd-agent - collector - INFO - Finished run #6. Collection time: 15.6s. Emit time: 0.01s
Dec 22 02:35:28 rig dd-agent - collector - INFO - Starting collection run #7
Dec 22 02:35:44 rig dd-agent - collector - INFO - Finished run #7. Collection time: 16.16s. Emit time: 0.01s
Dec 22 02:35:59 rig dd-agent - collector - INFO - Starting collection run #8
Dec 22 02:36:03 rig dd-agent - checks.tomcat - ERROR - Check 'tomcat' instance #0 failed
Traceback (most recent call last):
  File "/usr/share/datadog/agent/checks/__init__.py", line 423, in run
    self.check(instance)
  File "/usr/share/datadog/agent/checks.d/tomcat.py", line 49, in check
    dump = jmx.dump()
  File "/usr/share/datadog/agent/checks/jmx_connector.py", line 129, in dump
    jsonvar = json.loads(content)
  File "/usr/lib/python2.7/json/__init__.py", line 326, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python2.7/json/decoder.py", line 366, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python2.7/json/decoder.py", line 384, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded
Dec 22 02:36:07 rig dd-agent - checks.jmx - ERROR - Check 'jmx' instance #0 failed
Traceback (most recent call last):
  File "/usr/share/datadog/agent/checks/__init__.py", line 423, in run
    self.check(instance)
  File "/usr/share/datadog/agent/checks.d/jmx.py", line 29, in check
    dump = jmx.dump()
  File "/usr/share/datadog/agent/checks/jmx_connector.py", line 129, in dump
    jsonvar = json.loads(content)
  File "/usr/lib/python2.7/json/__init__.py", line 326, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python2.7/json/decoder.py", line 366, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python2.7/json/decoder.py", line 384, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded
Dec 22 02:36:07 rig dd-agent - collector - INFO - Finished run #8. Collection time: 8.65s. Emit time: 0.01s
Dec 22 02:36:22 rig dd-agent - collector - INFO - Starting collection run #9
Dec 22 02:36:40 rig dd-agent - collector - INFO - Finished run #9. Collection time: 17.35s. Emit time: 0.01s
Dec 22 02:36:55 rig dd-agent - collector - INFO - Starting collection run #10
Dec 22 02:37:11 rig dd-agent - collector - INFO - Finished run #10. Collection time: 16.48s. Emit time: 0.01s
Dec 22 02:37:26 rig dd-agent - collector - INFO - Starting collection run #11
Dec 22 02:37:43 rig dd-agent - collector - INFO - Finished run #11. Collection time: 16.89s. Emit time: 0.01s
Dec 22 02:37:58 rig dd-agent - collector - INFO - Starting collection run #12
Dec 22 02:38:02 rig dd-agent - checks.tomcat - ERROR - Check 'tomcat' instance #0 failed
Traceback (most recent call last):
  File "/usr/share/datadog/agent/checks/__init__.py", line 423, in run
    self.check(instance)
  File "/usr/share/datadog/agent/checks.d/tomcat.py", line 49, in check
    dump = jmx.dump()
  File "/usr/share/datadog/agent/checks/jmx_connector.py", line 129, in dump
    jsonvar = json.loads(content)
  File "/usr/lib/python2.7/json/__init__.py", line 326, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python2.7/json/decoder.py", line 366, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python2.7/json/decoder.py", line 384, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded
Dec 22 02:38:02 rig dd-agent - checks.jmx - ERROR - Check 'jmx' instance #0 failed
Traceback (most recent call last):
  File "/usr/share/datadog/agent/checks/__init__.py", line 423, in run
    self.check(instance)
  File "/usr/share/datadog/agent/checks.d/jmx.py", line 29, in check
    dump = jmx.dump()
  File "/usr/share/datadog/agent/checks/jmx_connector.py", line 129, in dump
    jsonvar = json.loads(content)
  File "/usr/lib/python2.7/json/__init__.py", line 326, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python2.7/json/decoder.py", line 366, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python2.7/json/decoder.py", line 384, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded
Dec 22 02:38:02 rig dd-agent - collector - INFO - Finished run #12. Collection time: 4.1s. Emit time: 0.01s
Dec 22 02:38:17 rig dd-agent - collector - INFO - Starting collection run #13
Dec 22 02:38:38 rig dd-agent - collector - INFO - Finished run #13. Collection time: 21.0s. Emit time: 0.01s
Dec 22 02:38:53 rig dd-agent - collector - INFO - Starting collection run #14
Dec 22 02:38:57 rig dd-agent - checks.tomcat - ERROR - Check 'tomcat' instance #0 failed
Traceback (most recent call last):
  File "/usr/share/datadog/agent/checks/__init__.py", line 423, in run
    self.check(instance)
  File "/usr/share/datadog/agent/checks.d/tomcat.py", line 49, in check
    dump = jmx.dump()
  File "/usr/share/datadog/agent/checks/jmx_connector.py", line 129, in dump
    jsonvar = json.loads(content)
  File "/usr/lib/python2.7/json/__init__.py", line 326, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python2.7/json/decoder.py", line 366, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python2.7/json/decoder.py", line 384, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded
Dec 22 02:38:57 rig dd-agent - checks.jmx - ERROR - Check 'jmx' instance #0 failed
Traceback (most recent call last):
  File "/usr/share/datadog/agent/checks/__init__.py", line 423, in run
    self.check(instance)
  File "/usr/share/datadog/agent/checks.d/jmx.py", line 29, in check
    dump = jmx.dump()
  File "/usr/share/datadog/agent/checks/jmx_connector.py", line 129, in dump
    jsonvar = json.loads(content)
  File "/usr/lib/python2.7/json/__init__.py", line 326, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python2.7/json/decoder.py", line 366, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python2.7/json/decoder.py", line 384, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded
Dec 22 02:38:57 rig dd-agent - collector - INFO - Finished run #14. Collection time: 4.1s. Emit time: 0.01s
Dec 22 02:39:12 rig dd-agent - collector - INFO - Starting collection run #15
Dec 22 02:39:33 rig dd-agent - collector - INFO - Finished run #15. Collection time: 20.74s. Emit time: 0.01s
Dec 22 02:39:48 rig dd-agent - collector - INFO - Starting collection run #16
Dec 22 02:40:05 rig dd-agent - collector - INFO - Finished run #16. Collection time: 16.58s. Emit time: 0.01s
Dec 22 02:40:20 rig dd-agent - collector - INFO - Starting collection run #17
Dec 22 02:40:36 rig dd-agent - collector - INFO - Finished run #17. Collection time: 16.47s. Emit time: 0.01s
Dec 22 02:40:51 rig dd-agent - collector - INFO - Starting collection run #18
Dec 22 02:41:08 rig dd-agent - collector - INFO - Finished run #18. Collection time: 16.9s. Emit time: 0.01s
Dec 22 02:41:23 rig dd-agent - collector - INFO - Starting collection run #19
Dec 22 02:41:40 rig dd-agent - collector - INFO - Finished run #19. Collection time: 16.9s. Emit time: 0.01s
Dec 22 02:41:55 rig dd-agent - collector - INFO - Starting collection run #20
Dec 22 02:42:12 rig dd-agent - collector - INFO - Finished run #20. Collection time: 17.07s. Emit time: 0.01s
Dec 22 02:42:27 rig dd-agent - collector - INFO - Starting collection run #21
Dec 22 02:42:45 rig dd-agent - collector - INFO - Finished run #21. Collection time: 18.05s. Emit time: 0.01s
Dec 22 02:43:00 rig dd-agent - collector - INFO - Starting collection run #22
Dec 22 02:43:19 rig dd-agent - collector - INFO - Finished run #22. Collection time: 18.27s. Emit time: 0.01s
Dec 22 02:43:34 rig dd-agent - collector - INFO - Starting collection run #23
Dec 22 02:43:52 rig dd-agent - collector - INFO - Finished run #23. Collection time: 18.61s. Emit time: 0.01s
Dec 22 02:44:07 rig dd-agent - collector - INFO - Starting collection run #24
Dec 22 02:44:27 rig dd-agent - collector - INFO - Finished run #24. Collection time: 19.51s. Emit time: 0.01s
Dec 22 02:44:42 rig dd-agent - collector - INFO - Starting collection run #25
Dec 22 02:45:01 rig dd-agent - collector - INFO - Finished run #25. Collection time: 19.24s. Emit time: 0.01s
Dec 22 02:45:16 rig dd-agent - collector - INFO - Starting collection run #26
Dec 22 02:45:36 rig dd-agent - collector - INFO - Finished run #26. Collection time: 19.87s. Emit time: 0.01s
Dec 22 02:45:51 rig dd-agent - collector - INFO - Starting collection run #27
Dec 22 02:46:11 rig dd-agent - collector - INFO - Finished run #27. Collection time: 20.1s. Emit time: 0.01s
Dec 22 02:46:26 rig dd-agent - collector - INFO - Starting collection run #28
Dec 22 02:46:47 rig dd-agent - collector - INFO - Finished run #28. Collection time: 20.71s. Emit time: 0.01s
Dec 22 02:47:02 rig dd-agent - collector - INFO - Starting collection run #29
Dec 22 02:47:23 rig dd-agent - collector - INFO - Finished run #29. Collection time: 21.1s. Emit time: 0.01s
Dec 22 02:47:38 rig dd-agent - collector - INFO - Starting collection run #30
Dec 22 02:47:59 rig dd-agent - collector - INFO - Finished run #30. Collection time: 20.8s. Emit time: 0.01s
Dec 22 02:48:14 rig dd-agent - collector - INFO - Starting collection run #31
Dec 22 02:48:36 rig dd-agent - collector - INFO - Finished run #31. Collection time: 21.94s. Emit time: 0.01s
Dec 22 02:48:51 rig dd-agent - collector - INFO - Starting collection run #32
Dec 22 02:49:12 rig dd-agent - collector - INFO - Finished run #32. Collection time: 21.69s. Emit time: 0.01s
Dec 22 02:49:27 rig dd-agent - collector - INFO - Starting collection run #33
Dec 22 02:49:49 rig dd-agent - collector - INFO - Finished run #33. Collection time: 21.81s. Emit time: 0.01s
Dec 22 02:50:04 rig dd-agent - collector - INFO - Starting collection run #34
Dec 22 02:50:27 rig dd-agent - collector - INFO - Finished run #34. Collection time: 22.54s. Emit time: 0.01s
Dec 22 02:50:42 rig dd-agent - collector - INFO - Starting collection run #35
Dec 22 02:51:04 rig dd-agent - collector - INFO - Finished run #35. Collection time: 22.58s. Emit time: 0.01s
Dec 22 02:51:19 rig dd-agent - collector - INFO - Starting collection run #36
Dec 22 02:51:42 rig dd-agent - collector - INFO - Finished run #36. Collection time: 22.75s. Emit time: 0.01s
Dec 22 02:51:57 rig dd-agent - collector - INFO - Starting collection run #37
Dec 22 02:52:20 rig dd-agent - collector - INFO - Finished run #37. Collection time: 22.65s. Emit time: 0.01s
Dec 22 02:52:35 rig dd-agent - collector - INFO - Starting collection run #38
Dec 22 02:52:57 rig dd-agent - collector - INFO - Finished run #38. Collection time: 22.41s. Emit time: 0.01s
Dec 22 02:53:12 rig dd-agent - collector - INFO - Starting collection run #39
Dec 22 02:53:36 rig dd-agent - collector - INFO - Finished run #39. Collection time: 23.4s. Emit time: 0.01s
Dec 22 02:53:51 rig dd-agent - collector - INFO - Starting collection run #40
Dec 22 02:54:14 rig dd-agent - collector - INFO - Finished run #40. Collection time: 23.5s. Emit time: 0.01s
Dec 22 02:54:29 rig dd-agent - collector - INFO - Starting collection run #41
Dec 22 02:54:33 rig dd-agent - checks.tomcat - ERROR - Check 'tomcat' instance #0 failed
Traceback (most recent call last):
  File "/usr/share/datadog/agent/checks/__init__.py", line 423, in run
    self.check(instance)
  File "/usr/share/datadog/agent/checks.d/tomcat.py", line 49, in check
    dump = jmx.dump()
  File "/usr/share/datadog/agent/checks/jmx_connector.py", line 129, in dump
    jsonvar = json.loads(content)
  File "/usr/lib/python2.7/json/__init__.py", line 326, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python2.7/json/decoder.py", line 366, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python2.7/json/decoder.py", line 384, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded
Dec 22 02:54:33 rig dd-agent - checks.jmx - ERROR - Check 'jmx' instance #0 failed
Traceback (most recent call last):
  File "/usr/share/datadog/agent/checks/__init__.py", line 423, in run
    self.check(instance)
  File "/usr/share/datadog/agent/checks.d/jmx.py", line 29, in check
    dump = jmx.dump()
  File "/usr/share/datadog/agent/checks/jmx_connector.py", line 129, in dump
    jsonvar = json.loads(content)
  File "/usr/lib/python2.7/json/__init__.py", line 326, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python2.7/json/decoder.py", line 366, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python2.7/json/decoder.py", line 384, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded
Dec 22 02:54:33 rig dd-agent - collector - INFO - Finished run #41. Collection time: 4.1s. Emit time: 0.0s
Dec 22 02:54:48 rig dd-agent - collector - INFO - Starting collection run #42
Dec 22 02:55:18 rig dd-agent - collector - INFO - Finished run #42. Collection time: 29.21s. Emit time: 0.01s
Dec 22 02:55:33 rig dd-agent - collector - INFO - Starting collection run #43
Dec 22 02:55:57 rig dd-agent - collector - INFO - Finished run #43. Collection time: 24.06s. Emit time: 0.01s
Dec 22 02:56:12 rig dd-agent - collector - INFO - Starting collection run #44
Dec 22 02:56:36 rig dd-agent - collector - INFO - Finished run #44. Collection time: 24.04s. Emit time: 0.01s
Dec 22 02:56:51 rig dd-agent - collector - INFO - Starting collection run #45
Dec 22 02:57:15 rig dd-agent - collector - INFO - Finished run #45. Collection time: 23.99s. Emit time: 0.01s
Dec 22 02:57:30 rig dd-agent - collector - INFO - Starting collection run #46
Dec 22 02:57:34 rig dd-agent - checks.tomcat - ERROR - Check 'tomcat' instance #0 failed
Traceback (most recent call last):
  File "/usr/share/datadog/agent/checks/__init__.py", line 423, in run
    self.check(instance)
  File "/usr/share/datadog/agent/checks.d/tomcat.py", line 49, in check
    dump = jmx.dump()
  File "/usr/share/datadog/agent/checks/jmx_connector.py", line 129, in dump
    jsonvar = json.loads(content)
  File "/usr/lib/python2.7/json/__init__.py", line 326, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python2.7/json/decoder.py", line 366, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python2.7/json/decoder.py", line 384, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded
Dec 22 02:57:34 rig dd-agent - checks.jmx - ERROR - Check 'jmx' instance #0 failed
Traceback (most recent call last):
  File "/usr/share/datadog/agent/checks/__init__.py", line 423, in run
    self.check(instance)
  File "/usr/share/datadog/agent/checks.d/jmx.py", line 29, in check
    dump = jmx.dump()
  File "/usr/share/datadog/agent/checks/jmx_connector.py", line 129, in dump
    jsonvar = json.loads(content)
  File "/usr/lib/python2.7/json/__init__.py", line 326, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python2.7/json/decoder.py", line 366, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python2.7/json/decoder.py", line 384, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded
Dec 22 02:57:34 rig dd-agent - collector - INFO - Finished run #46. Collection time: 4.12s. Emit time: 0.01s
Dec 22 02:57:49 rig dd-agent - collector - INFO - Starting collection run #47
Dec 22 02:57:53 rig dd-agent - checks.tomcat - ERROR - Check 'tomcat' instance #0 failed
Traceback (most recent call last):
  File "/usr/share/datadog/agent/checks/__init__.py", line 423, in run
    self.check(instance)
  File "/usr/share/datadog/agent/checks.d/tomcat.py", line 49, in check
    dump = jmx.dump()
  File "/usr/share/datadog/agent/checks/jmx_connector.py", line 129, in dump
    jsonvar = json.loads(content)
  File "/usr/lib/python2.7/json/__init__.py", line 326, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python2.7/json/decoder.py", line 366, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python2.7/json/decoder.py", line 384, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded
Dec 22 02:57:53 rig dd-agent - checks.jmx - ERROR - Check 'jmx' instance #0 failed
Traceback (most recent call last):
  File "/usr/share/datadog/agent/checks/__init__.py", line 423, in run
    self.check(instance)
  File "/usr/share/datadog/agent/checks.d/jmx.py", line 29, in check
    dump = jmx.dump()
  File "/usr/share/datadog/agent/checks/jmx_connector.py", line 129, in dump
    jsonvar = json.loads(content)
  File "/usr/lib/python2.7/json/__init__.py", line 326, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python2.7/json/decoder.py", line 366, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python2.7/json/decoder.py", line 384, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded
Dec 22 02:57:53 rig dd-agent - collector - INFO - Finished run #47. Collection time: 4.1s. Emit time: 0.01s
Dec 22 02:58:08 rig dd-agent - collector - INFO - Starting collection run #48
Dec 22 02:58:32 rig dd-agent - checks.tomcat - ERROR - Check 'tomcat' instance #0 failed
Traceback (most recent call last):
  File "/usr/share/datadog/agent/checks/__init__.py", line 423, in run
    self.check(instance)
  File "/usr/share/datadog/agent/checks.d/tomcat.py", line 49, in check
    dump = jmx.dump()
  File "/usr/share/datadog/agent/checks/jmx_connector.py", line 127, in dump
    self._wait_prompt()
  File "/usr/share/datadog/agent/checks/jmx_connector.py", line 61, in _wait_prompt
    self._jmx.expect_exact("$>") # got prompt, we can continue
  File "/usr/lib/python2.7/dist-packages/pexpect.py", line 1343, in expect_exact
    return self.expect_loop(searcher_string(pattern_list), timeout, searchwindowsize)
  File "/usr/lib/python2.7/dist-packages/pexpect.py", line 1409, in expect_loop
    raise TIMEOUT (str(e) + '\n' + str(self))
TIMEOUT: Timeout exceeded in expect_any().
<pexpect.spawn object at 0x11b9e10>
version: 2.3 ($Revision: 399 $)
command: /usr/bin/java
args: ['/usr/bin/java', '-jar', '/usr/share/datadog/agent/checks/libs/jmxterm-1.0-DATADOG-uber.jar', '-l', 'localhost:8090', '-u', 'bigroad', '-p', 'XXX']
searcher: searcher_string:
    0: "$>"
buffer (last 100 chars): 9.197.117"},"Catalina:name=HttpRequest2607,type=RequestProcessor,worker=http-80":{"requestProcessing
before (last 100 chars): 9.197.117"},"Catalina:name=HttpRequest2607,type=RequestProcessor,worker=http-80":{"requestProcessing
after: <class 'pexpect.TIMEOUT'>
match: None
match_index: None
exitstatus: None
flag_eof: False
pid: 26373
child_fd: 5
closed: False
timeout: 20
delimiter: <class 'pexpect.EOF'>
logfile: None
logfile_read: None
logfile_send: None
maxread: 2000
ignorecase: False
searchwindowsize: None
delaybeforesend: 0
delayafterclose: 0.1
delayafterterminate: 0.1
Dec 22 02:58:34 rig dd-agent - collector - INFO - Finished run #48. Collection time: 25.62s. Emit time: 0.01s
Dec 22 02:58:49 rig dd-agent - collector - INFO - Starting collection run #49
Dec 22 02:59:14 rig dd-agent - collector - INFO - Finished run #49. Collection time: 25.11s. Emit time: 0.01s
Dec 22 02:59:29 rig dd-agent - collector - INFO - Starting collection run #50
Dec 22 02:59:53 rig dd-agent - collector - INFO - Finished run #50. Collection time: 24.41s. Emit time: 0.01s
Dec 22 03:00:08 rig dd-agent - collector - INFO - Starting collection run #51
Dec 22 03:00:33 rig dd-agent - collector - INFO - Finished run #51. Collection time: 24.49s. Emit time: 0.01s
Dec 22 03:00:48 rig dd-agent - collector - INFO - Starting collection run #52
Dec 22 03:01:12 rig dd-agent - collector - INFO - Finished run #52. Collection time: 24.68s. Emit time: 0.01s
Dec 22 03:01:27 rig dd-agent - collector - INFO - Starting collection run #53
Dec 22 03:01:32 rig dd-agent - checks.tomcat - ERROR - Check 'tomcat' instance #0 failed
Traceback (most recent call last):
  File "/usr/share/datadog/agent/checks/__init__.py", line 423, in run
    self.check(instance)
  File "/usr/share/datadog/agent/checks.d/tomcat.py", line 49, in check
    dump = jmx.dump()
  File "/usr/share/datadog/agent/checks/jmx_connector.py", line 129, in dump
    jsonvar = json.loads(content)
  File "/usr/lib/python2.7/json/__init__.py", line 326, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python2.7/json/decoder.py", line 366, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python2.7/json/decoder.py", line 384, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded
Dec 22 03:01:32 rig dd-agent - checks.jmx - ERROR - Check 'jmx' instance #0 failed
Traceback (most recent call last):
  File "/usr/share/datadog/agent/checks/__init__.py", line 423, in run
    self.check(instance)
  File "/usr/share/datadog/agent/checks.d/jmx.py", line 29, in check
    dump = jmx.dump()
  File "/usr/share/datadog/agent/checks/jmx_connector.py", line 129, in dump
    jsonvar = json.loads(content)
  File "/usr/lib/python2.7/json/__init__.py", line 326, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python2.7/json/decoder.py", line 366, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python2.7/json/decoder.py", line 384, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded
Dec 22 03:01:32 rig dd-agent - collector - INFO - Finished run #53. Collection time: 4.1s. Emit time: 0.01s
@ghost ghost assigned alq666 Dec 23, 2012
@alq666
Copy link
Member Author

alq666 commented Dec 23, 2012

@remh have you seen this before? CPU seems busier than it should be, on top of not getting the data.

@remh
Copy link

remh commented Dec 23, 2012

The cpu must be rising because the jmx connection is not working. So at each collection the agent spawns a new jmx connector process, launching the jvm which may be cpu consuming.

What's happening when we manually launch jmx connector ?
java -jar /usr/share/datadog/agent/checks/libs/jmxterm-1.0-DATADOG-uber.jar -l localhost:8090 -u USERNAME -p PASSWORD
Is the connection working ?

@olidb2
Copy link
Member

olidb2 commented Dec 28, 2012

Independently from knowing why the connection doesn't work, we need to
guard against behaviors that end-up consuming tons of CPU. Should we add
some back-off when that happens? What status does the agent report?

On Sun, Dec 23, 2012 at 11:29 AM, Remi Hakim notifications@github.comwrote:

The cpu must be rising because the jmx connection is not working. So at
each collection the agent spawns a new jmx connector process, launching the
jvm which may be cpu consuming.

What's happening when we manually launch jmx connector ?
java -jar
/usr/share/datadog/agent/checks/libs/jmxterm-1.0-DATADOG-uber.jar -l
localhost:8090 -u USERNAME -p PASSWORD

Is the connection working ?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/313#issuecomment-11647657.

@clofresh
Copy link
Contributor

An easy-ish way to limit our cpu is to nice the process when we run it so that our cpu usage gets deprioritized.

@olidb2
Copy link
Member

olidb2 commented Dec 28, 2012

Yep, that's a start.
Seems like spawning tons of new outside processes with immediate failure
could have a number of nasty consequences, though. Wondering if it would
make sense to add a utility into the agent core for handling back-offs /
giving-up and reporting of these situations.

On Fri, Dec 28, 2012 at 4:26 PM, Carlo Cabanilla
notifications@github.comwrote:

An easy-ish way to limit our cpu is to nice the process when we run it so
that our cpu usage gets deprioritized.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/313#issuecomment-11742021.

@ghost ghost assigned remh Jan 7, 2013
@remh
Copy link

remh commented Jan 7, 2013

Ok so here the things I did:

  1. When the connection fails 3 times in a row, we won't try to open it up again.
  2. The jmx process will be launched with nice -n 15
  3. I added an option in the jmx terminal to do a prefiltering of the values that are fetched which seems to add more stability to the jmx connector
  4. Better failure handling
  5. The check will only dump jmx domains that are actually used

Performances are ok but not great, the jmx connector uses a mean value of 2% cpu on my machine, with high peak when performing the dump. Collector is ok (less than 0.5 %)

In the overall, it will fix the problem encountered. I used ab to apply heavy load to a tomcat instance and it seems to be working as expected, although from time to time the jmx connection breaks without an obvious reason, but the improved failure handling will relaunch the connection at the next collection.

@alq666
Copy link
Member Author

alq666 commented Jan 7, 2013

How many concurrent connections did you use with ab?

@remh
Copy link

remh commented Jan 7, 2013

200, I couldn't get more

@alq666
Copy link
Member Author

alq666 commented Jan 8, 2013

Worth spawning a big ec2 box to crank that number up. 5000 is a good
target.

On Monday, January 7, 2013, Remi Hakim wrote:

200, I couldn't get more

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/313#issuecomment-11977600.

@remh
Copy link

remh commented Jan 8, 2013

good idea. doing it

@remh
Copy link

remh commented Jan 11, 2013

#337

@remh remh closed this as completed Jan 30, 2013
@remh remh reopened this Dec 5, 2013
@remh
Copy link

remh commented Dec 5, 2013

JMXFetch is going to be way more efficient.

@remh remh mentioned this issue Dec 5, 2013
Closed
@remh
Copy link

remh commented Dec 18, 2013

JMXFetch is way more efficient.

@remh remh closed this as completed Dec 18, 2013
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@remh remh

Labels
checks request
Projects
None yet
Milestone
3.4.x
Development

No branches or pull requests
4 participants
@clofresh @alq666 @olidb2 @remh