Few failing internet doctests in mma_free_integrator #25501

seblabbe · 2018-06-03T12:09:39Z

sage -tp --optional=sage,internet --logfile=logs/25501.log src/sage/symbolic/integration/integral.py src/sage/symbolic/integration/external.py

gives

**********************************************************************
File "src/sage/symbolic/integration/external.py", line 65, in sage.symbolic.integration.external.mma_free_integrator
Failed example:
    mma_free_integrator(sin(x), x) # optional - internet
Exception raised:
    Traceback (most recent call last):
      File "/home/slabbe/GitBox/sage/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 572, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/home/slabbe/GitBox/sage/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 982, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.symbolic.integration.external.mma_free_integrator[1]>", line 1, in <module>
        mma_free_integrator(sin(x), x) # optional - internet
      File "/home/slabbe/GitBox/sage/local/lib/python2.7/site-packages/sage/symbolic/integration/external.py", line 97, in mma_free_integrator
        page = page[page.index('"inputForm"'):page.index('"outputForm"')]
    ValueError: substring not found
**********************************************************************

...

----------------------------------------------------------------------
sage -t src/sage/symbolic/integration/external.py  # 3 doctests failed
sage -t src/sage/symbolic/integration/integral.py  # 1 doctest failed
----------------------------------------------------------------------

It can be reproduced with:

sage: from sage.symbolic.integration.external import mma_free_integrator
sage: mma_free_integrator(sin(x), x)

...
ValueError: substring not found

CC: @slel

Component: doctest coverage

Keywords: thursdaysbdx

Author: Sébastien Labbé, Amaury Pouly

Branch: f083393

Reviewer: Frédéric Chapoton

Issue created by migration from https://trac.sagemath.org/ticket/25501

The text was updated successfully, but these errors were encountered:

seblabbe · 2018-06-07T20:14:58Z

comment:2

I created independent tickets #25534 and #25535 to deal with the independent issues. Only the problem based on the mma_free_integrator is kept in this ticket.

fchapoton · 2018-07-18T08:22:44Z

comment:4

It seems that Ma#######ca is now hiding carefully its results, so we can no longer parse them..

slel · 2018-10-22T11:32:05Z

comment:5

From the starting point provided in the ticket description:

sage: from sage.symbolic.integration.external import mma_free_integrator
sage: mma_free_integrator(sin(x), x)
---------------------------------------------------------------------------
Traceback (most recent call last)
<ipython-input-2-9d83ac0f1d23> in <module>()
----> 1 mma_free_integrator(sin(x), x)

.../local/lib/python2.7/site-packages/sage/symbolic/integration/external.pyc in mma_free_integrator(expression, v, a, b)
     95     params = urlencode({'expr': expression._mathematica_init_(), 'random': 'false'})
     96     page = urlopen("http://integrals.wolfram.com/home.jsp", params).read()
---> 97     page = page[page.index('"inputForm"'):page.index('"outputForm"')]
     98     page = re.sub(r"\s", "", page)
     99     mexpr = re.match(r".*Integrate.*==</em><br/>(.*)</p>", page).groups()[0]

ValueError: substring not found

one can check what this function is doing:

sage: mma_free_integrator??

and follow its content step by step to trace the error:

sage: expression = sin(x)
sage: v = x
sage: import re
sage: from six.moves.urllib.request import urlopen
sage: from six.moves.urllib.parse import urlencode
sage: vars = [str(x) for x in expression.variables()]
sage: any(len(x)>1 for x in vars)
False
sage: x = SR.var('x')
sage: repr(v) != 'x'
False
sage: params = urlencode({'expr': expression._mathematica_init_(), 'random': 'false'})
sage: params
'expr=Sin%5Bx%5D&random=false'
sage: page = urlopen("http://integrals.wolfram.com/home.jsp", params).read()

Since the next line is the one that gives the error:

sage: page = page[page.index('"inputForm"'):page.index('"outputForm"')]
Traceback (most recent call last)
...
ValueError: substring not found

we can explore page in search of a possible reason; the last few lines
indicate that the integral calculator has moved to Wolfram|Alpha.

sage: print(page)
...
    <!-- we inline this because of handlebars -->
    <script type="text/javascript">
        if (getURLParam("redirected") == "true") {
          $('#input-wrapper').before(
              '<div id="redirectBanner"><div id="bannerContent"><div id="bannerText">The integral calculator has moved to Wolfram|Alpha. &nbsp;'
              + '</div><a id="redirectLink" href="http://integrals.wolfram.com/home.jsp">Visit the old one &raquo;</a></div></div>'
          );
        }

        $('form#input-form').submit(function() {
        var submittedData = $(this).find('input#calc-input').val();
        if (!submittedData) {
          $(this).find('input#calc-input').val('integrate x sin(x^2)');
        }
      });

    </script>
  </body>
</html>

Now if we use Wolfram|Alpha to collect the page describing the integral:

sage: page = urlopen("https://www.wolframalpha.com/input/?i=integrate+x+sin(x%5E2)").read()
sage: print(page)

we see that the display of the solution happens in JavaScript.

I see no way to parse the html page for the computed integral.

fchapoton · 2018-10-22T11:34:00Z

comment:6

Yes, the web site tries very hard to hide the result, so that no machine can extract it.

slel · 2018-10-23T23:33:07Z

comment:7

The API access for Wolfram|Alpha requires signing in with a Wolfram ID.

It can be used for free, limited to 2000 queries per month.

Wolfram Alpha API

seblabbe · 2018-11-22T18:16:29Z

New commits:

`0035b14`	`#26361: package glucose SAT solver`
`1159e03`	`#26361 : glucose depends on zlib`
`38d09eb`	`#26361 : fix glucose interface + doctests`
`878769d`	`#26361 : add a warning about the license of glucose-syrup`
`19ac7ec`	`Merge branch 'u/tmonteil/glucose_sat_solver' into 8.5.beta0`
`521f595`	26361 adding doctests, adding glucose as an option to `SAT`
`41fa426`	`26361: adding optional tags`
`bec05bb`	`Merge branch 'u/slabbe/glucose_sat_solver' into 8.5.beta4`
`d0c6a1a`	`25501: fixing mathematica_free optional doctests`

seblabbe · 2018-11-22T18:16:29Z

Branch: u/slabbe/25501

seblabbe · 2018-11-22T18:16:29Z

Commit: d0c6a1a

sagetrac-git · 2018-11-22T18:26:27Z

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

`58e2af4`	`25501: fixing mathematica_free optional doctests`

sagetrac-git · 2018-11-22T18:26:27Z

Changed commit from d0c6a1a to 58e2af4

seblabbe · 2018-11-22T18:27:55Z

comment:10

Oups, my first push was containing commits from another ticket.

I remade the branch on top of the most recent beta.

Sorry for the noise.

seblabbe · 2018-11-22T18:30:39Z

Author: Sébastien Labbé

pamaury · 2018-11-22T22:27:33Z

comment:12

Attachment: wolfram.py.gz

I was able to make the Wolfram API work without login. I basically traced what the Wolfram page does (it's full of javascript) using Firefox dev tools, and produced the smallest Python program I could to reproduce it. At the moment it only prints the result, which is big-ish JSON with several results. Example use:

./wolfram.py "integrate x sin(x^2)"

Some random notes: it seems cookies are important, the "Referer" header was important, there is a proxy_code which is important but it's not clear if it can be reused. Obviously Wolfram may change this interface at any time. I have not checked if the params of the requests are the same as the official API.

EDIT: I uploaded a second program that uses the synchronous interface and drops some of the (seemingly) useless parameters of the requests. Some parameters are documented in the Full Reference of the Wolfram API.

pamaury · 2018-11-26T13:27:14Z

Attachment: wolfram2.py.gz

seblabbe · 2018-12-12T22:21:28Z

comment:13

I just updated the branch with a fix based on the suggestion made by gh-pamaury.

I stil need to add some doctests...

New commits:

`9517244`	`25501: fixing the mma integrator`

seblabbe · 2018-12-12T22:21:28Z

Changed branch from u/slabbe/25501 to u/slabbe/25501-fix

seblabbe · 2018-12-12T22:21:28Z

Changed commit from 58e2af4 to 9517244

sagetrac-git · 2018-12-12T22:29:38Z

Changed commit from 9517244 to 60bcbe7

sagetrac-git · 2018-12-12T22:29:38Z

Branch pushed to git repo; I updated commit sha1. New commits:

`60bcbe7`	`25501: small fix`

seblabbe · 2018-12-13T08:23:06Z

Changed keywords from none to thursdaysbdx

seblabbe · 2018-12-20T10:14:54Z

Changed commit from e5cb272 to 157850c

seblabbe · 2018-12-20T10:14:54Z

Changed branch from u/slabbe/25501-fix to u/slabbe/25501

seblabbe · 2018-12-20T10:14:54Z

comment:21

fixed Python 3 issues, rebased on 8.5.beta6
New commits:

`6b02bc2`	`25501: fixing the mma integrator`
`157850c`	`25501: Python 3 care`

sagetrac-git · 2018-12-20T10:31:32Z

Changed commit from 157850c to 76ad1cb

sagetrac-git · 2018-12-20T10:31:32Z

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

`11791c5`	`25501: fixing the mma integrator`
`2eabdcb`	`25501: Python 3 care`
`76ad1cb`	`25501: fix back for Python 2`

seblabbe · 2018-12-20T10:32:41Z

comment:23

After fixing Python 3, I needed to adapt the doctests for Python 2 again...

Rebased on 8.5.rc0.

Needs review.

fchapoton · 2018-12-20T14:39:48Z

comment:24

oh, well. Let it be..

fchapoton · 2018-12-20T14:39:48Z

Reviewer: Frédéric Chapoton

vbraun · 2018-12-23T10:23:04Z

comment:25

There is a internet-using doctest thats not marked as such:

**********************************************************************
File "src/sage/symbolic/integration/external.py", line 240, in sage.symbolic.integration.external.parse_moutput_from_json
Failed example:
    page_data = request_wolfram_alpha('Integrate(Sin[z], y)')
Exception raised:
    Traceback (most recent call last):
      File "/var/lib/buildbot/slave/sage_git/build/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 671, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/var/lib/buildbot/slave/sage_git/build/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 1086, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.symbolic.integration.external.parse_moutput_from_json[2]>", line 1, in <module>
        page_data = request_wolfram_alpha('Integrate(Sin[z], y)')
      File "/var/lib/buildbot/slave/sage_git/build/local/lib/python2.7/site-packages/sage/symbolic/integration/external.py", line 157, in request_wolfram_alpha
        resp = opener.open(req)
      File "/var/lib/buildbot/slave/sage_git/build/local/lib/python2.7/urllib2.py", line 429, in open
        response = self._open(req, data)
      File "/var/lib/buildbot/slave/sage_git/build/local/lib/python2.7/urllib2.py", line 447, in _open
        '_open', req)
      File "/var/lib/buildbot/slave/sage_git/build/local/lib/python2.7/urllib2.py", line 407, in _call_chain
        result = func(*args)
      File "/var/lib/buildbot/slave/sage_git/build/local/lib/python2.7/urllib2.py", line 1228, in http_open
        return self.do_open(httplib.HTTPConnection, req)
      File "/var/lib/buildbot/slave/sage_git/build/local/lib/python2.7/urllib2.py", line 1198, in do_open
        raise URLError(err)
    URLError: <urlopen error [Errno 110] Connection timed out>
**********************************************************************
File "src/sage/symbolic/integration/external.py", line 241, in sage.symbolic.integration.external.parse_moutput_from_json
Failed example:
    parse_moutput_from_json(page_data)
Expected:
    Traceback (most recent call last):
    ...
    ValueError: asking wolframalpha.com was not successful
Got:
    <BLANKLINE>
    Traceback (most recent call last):
      File "/var/lib/buildbot/slave/sage_git/build/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 671, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/var/lib/buildbot/slave/sage_git/build/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 1086, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.symbolic.integration.external.parse_moutput_from_json[3]>", line 1, in <module>
        parse_moutput_from_json(page_data)
    NameError: name 'page_data' is not defined

sagetrac-git · 2018-12-31T12:33:07Z

Changed commit from 76ad1cb to f083393

sagetrac-git · 2018-12-31T12:33:07Z

Branch pushed to git repo; I updated commit sha1. New commits:

`f083393`	`25501: adding missing optional tags`

fchapoton · 2018-12-31T15:29:57Z

comment:29

ok

vbraun · 2019-01-02T08:17:30Z

Changed branch from u/slabbe/25501 to f083393

seblabbe · 2019-01-02T10:18:06Z

Changed commit from f083393 to none

seblabbe added this to the sage-8.3 milestone Jun 3, 2018

seblabbe added t: tests labels Jun 3, 2018

This comment has been minimized.

Sign in to view

seblabbe changed the title ~~Few internet doctests failed in 4 files~~ Few failing internet doctests in mma_free_integrator Jun 7, 2018

This comment has been minimized.

Sign in to view

seblabbe added the s: needs review label Nov 22, 2018

seblabbe removed the s: needs review label Dec 13, 2018

fchapoton added the s: needs work label Dec 16, 2018

seblabbe added s: needs review and removed s: needs work labels Dec 20, 2018

fchapoton added s: positive review and removed s: needs review labels Dec 20, 2018

vbraun added s: needs work and removed s: positive review labels Dec 23, 2018

seblabbe added s: needs review and removed s: needs work labels Dec 31, 2018

fchapoton added s: positive review and removed s: needs review labels Dec 31, 2018

vbraun removed the s: positive review label Jan 2, 2019

vbraun closed this as completed in 9454882 Jan 2, 2019

seblabbe modified the milestones: sage-8.3, sage-8.6 Jan 2, 2019

seblabbe mentioned this issue Oct 21, 2022

meta-ticket: failing internet/optional doctests #25536

Open

Few failing internet doctests in mma_free_integrator #25501

Few failing internet doctests in mma_free_integrator #25501

Comments

seblabbe commented Jun 3, 2018

This comment has been minimized.

seblabbe commented Jun 7, 2018

This comment has been minimized.

This comment has been minimized.

fchapoton commented Jul 18, 2018

slel commented Oct 22, 2018

fchapoton commented Oct 22, 2018

slel commented Oct 23, 2018

seblabbe commented Nov 22, 2018

seblabbe commented Nov 22, 2018

seblabbe commented Nov 22, 2018

sagetrac-git mannequin commented Nov 22, 2018

sagetrac-git mannequin commented Nov 22, 2018

seblabbe commented Nov 22, 2018

seblabbe commented Nov 22, 2018

pamaury mannequin commented Nov 22, 2018

pamaury mannequin commented Nov 26, 2018

seblabbe commented Dec 12, 2018

seblabbe commented Dec 12, 2018

seblabbe commented Dec 12, 2018

sagetrac-git mannequin commented Dec 12, 2018

sagetrac-git mannequin commented Dec 12, 2018

seblabbe commented Dec 13, 2018

seblabbe commented Dec 20, 2018

seblabbe commented Dec 20, 2018

seblabbe commented Dec 20, 2018

sagetrac-git mannequin commented Dec 20, 2018

sagetrac-git mannequin commented Dec 20, 2018

seblabbe commented Dec 20, 2018

fchapoton commented Dec 20, 2018

fchapoton commented Dec 20, 2018

vbraun commented Dec 23, 2018

sagetrac-git mannequin commented Dec 31, 2018

sagetrac-git mannequin commented Dec 31, 2018

fchapoton commented Dec 31, 2018

vbraun commented Jan 2, 2019

seblabbe commented Jan 2, 2019