unclear TravisError in get_response_contents when grabbing build log from Travis

[boegel]

I've been using travispy for a long time now, has worked flawlessly for months.

It started failing on me recently though, for reasons unclear to me (my best guess is a change in the Travis API for now).

Here's the traceback I'm seeing:

Traceback (most recent call last):
  File "/tmp/eb-scripts/pr_check.py", line 478, in <module>
    main()
  File "/tmp/eb-scripts/pr_check.py", line 441, in main
    res = travis(github_account, repository, github_token, owner=owner)
  File "/tmp/eb-scripts/pr_check.py", line 354, in travis
    retained_log_lines = jobs[0][1].log.body.split('\n')
  File "/tmp/travispy/travispy/entities/job.py", line 94, in log
    return self._load_one_lazy_information(Log)
  File "/tmp/travispy/travispy/entities/_entity.py", line 292, in _load_one_lazy_information
    'entity_id',
  File "/tmp/travispy/travispy/entities/_entity.py", line 252, in _load_lazy_information
    result = load_method(self._session, **{load_kwarg: property_ref})
  File "/tmp/travispy/travispy/entities/_entity.py", line 92, in find_one
    contents = get_response_contents(response)
  File "/tmp/travispy/travispy/_helpers.py", line 34, in get_response_contents
    raise TravisError(contents)
travispy.errors.TravisError

My pr_check.py script is available at https://github.com/boegel/eb-scripts/blob/master/pr_check.py .

It fails when trying to grab the log from https://travis-ci.org/easybuilders/easybuild-easyconfigs/builds/340553691 .

Any ideas on what may be broken here?

[boegel]

Some more information: this is the actual error that occurs in get_response_contents:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "travispy/entities/job.py", line 94, in log
    return self._load_one_lazy_information(Log)
  File "travispy/entities/_entity.py", line 292, in _load_one_lazy_information
    'entity_id',
  File "travispy/entities/_entity.py", line 252, in _load_lazy_information
    result = load_method(self._session, **{load_kwarg: property_ref})
  File "travispy/entities/_entity.py", line 92, in find_one
    contents = get_response_contents(response)
  File "travispy/_helpers.py", line 20, in get_response_contents
    contents = response.json()
  File "/prefix/lib/python2.7/site-packages/requests-2.18.4-py2.7.egg/requests/models.py", line 892, in json
    return complexjson.loads(self.text, **kwargs)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/__init__.py", line 338, in loads
    return _default_decoder.decode(s)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 366, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 384, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded

[boegel]

After a bit of debugging, I found a workaround, but no real fix yet...

diff --git a/travispy/_helpers.py b/travispy/_helpers.py
index 348d816..f54636b 100644
--- a/travispy/_helpers.py
+++ b/travispy/_helpers.py
@@ -16,7 +16,10 @@ def get_response_contents(response):
     '''
     status_code = response.status_code
     try:
-        contents = response.json()
+        if 'json' in response.headers['content-type']:
+            contents = response.json()
+        else:
+            contents = {'log': {'type': 'Log', 'body': response.text}}
     except:
         error = response.text.strip()
         if not error:

The problem is that the log is provided as a value of type text/plain, while get_response_contents assumes it's always dealing with a value of type application/json.

The return value of get_response_contents should be a dict value though (usually a parsed JSON value), that's assumed by the rest of the code.

A dirty workaround is to manually construct the expected dict value for logs when the response value type is not application/json, see patch above.

There must be a cleaner way to fix this though, at least only do this when the response is actually for a log, and not for something else that may be of type text/plain...

@menegazzo Any suggestions for a better way of dealing with this?

[SilleBille]

Even my team is having the same issue. Is there a fix or a PR request already for this. The workaround seems to be a fix. We can send this as a pull request. @menegazzo

Thanks for this awesome API!

[boegel]

@SilleBille I don't think my proposed workaround is a proper fix, there must be a better way to fix this, but we'll need some input from @menegazzo on this I think...

[SilleBille]

Updated: @boegel FYI, if you do pip3 install travispy this issue still exists.

[menegazzo]

Hey guys! Sorry for taking so long to get back to you. I will take a look at this right away and ASAP I bring some news. Thanks for your patience and support!

[menegazzo]

The response related to logs where changed at the Travis CI API. That is the cause of the error. Considering that the v2 of this API will be discontinued soon I don't think that te workaround suggested by @boegel here is that bad.

I will dig a bit more. Let's see.

[SilleBille]

@menegazzo if you are going to drop support for v2, my CI infrastructure will become unstable. I tried to install v3 Version but it gets installed only to the user who is installing it. I tried sudo pip3 install as well. Didn't work.

Tried this:

• logged in as user
• (sudo) pip3 install travispy
• tried importing travispy (it worked)
• login as jenkins
• tried importing travispy

The last step throws "Module not found error". I know this is not relevant to the bug but since you said you will be dropping v2 support, I am posting it here.

Thanks for the heads up!

[SilleBille]

@menegazzo @boegel Just wanted to update. I face the same issue in the pip3 package as well! Sorry for the wrong post above.

Proof:

---------------------------------------------------------------------------
JSONDecodeError                           Traceback (most recent call last)
~/.local/lib/python3.6/site-packages/travispy/_helpers.py in get_response_contents(response)
     18     try:
---> 19         contents = response.json()
     20     except:

/usr/lib/python3.6/site-packages/requests/models.py in json(self, **kwargs)
    891                     pass
--> 892         return complexjson.loads(self.text, **kwargs)
    893 

/usr/lib64/python3.6/site-packages/simplejson/__init__.py in loads(s, encoding, cls, object_hook, parse_float, parse_int, parse_constant, object_pairs_hook, use_decimal, **kw)
    515             and not use_decimal and not kw):
--> 516         return _default_decoder.decode(s)
    517     if cls is None:

/usr/lib64/python3.6/site-packages/simplejson/decoder.py in decode(self, s, _w, _PY3)
    373             s = s.decode(self.encoding)
--> 374         obj, end = self.raw_decode(s)
    375         end = _w(s, end).end()

/usr/lib64/python3.6/site-packages/simplejson/decoder.py in raw_decode(self, s, idx, _w, _PY3)
    403                 idx += 3
--> 404         return self.scan_once(s, idx=_w(s, idx).end())

JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

TravisError                               Traceback (most recent call last)
<ipython-input-11-0d54204df712> in <module>()
----> 1 fullLogObj = t.log(build.jobs[0].log_id)

~/.local/lib/python3.6/site-packages/travispy/travispy.py in log(self, log_id)
    230         :rtype: :class:`.Log`
    231         '''
--> 232         return Log.find_one(self._session, log_id)
    233 
    234     def repos(self, **kwargs):

~/.local/lib/python3.6/site-packages/travispy/entities/_entity.py in find_one(cls, session, entity_id, **kwargs)
     88         )
     89 
---> 90         contents = get_response_contents(response)
     91         if command not in contents:
     92             return

~/.local/lib/python3.6/site-packages/travispy/_helpers.py in get_response_contents(response)
     32             'error': error,
     33         }
---> 34         raise TravisError(contents)
     35 
     36     if status_code == 200:

Worker information
hostname: af49ecba-0fcb-449b-ad79-7c8eeb1a68fb@1.i-03b0435-production-2-worker-org-ec2.travisci.net
version: v3.1.0 https://github.com/travis-ci/worker/tree/5b106cb773802277034d07b0a865af58d67bb1d2
instance: f219a02:travisci/ci-garnet:packer-1503972846 (via amqp)
startup: 531.38626ms
Build system information

[boegel]

@menegazzo Any updates on this?

[menegazzo]

Hi. Unfortunately I don't have the time to keep the support on this library. If you have the interest in taking the ownership of this, let me know.