Syntaxerror for working doctest

[ArneBachmannDLR]

Hi,

For this code:

def slurpFile(path:str, mode:str = READ, maxbytes:int = 0, **kwds) -> Union[str,bytes]:
  ''' Read and return entire file.

  path: file path
  mode: open parameters
  kwds: optional parameters like encoding

  >>> import os; slurpFile(os.path.abspath(__file__), mode = 'rb')[:9]
  b'# coding='
  >>> import os; slurpFile(os.path.abspath(__file__), encoding='utf-8')[:9]
  '# coding='
  '''
  assert 'w' not in mode and 'a' not in mode, mode
  with open(path, mode=mode, **kwds) as fd: return fd.read(maxbytes) if maxbytes else fd.read()

I get this:

Traceback (most recent call last):
  File "d:\apps\miniforge3\lib\runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "d:\apps\miniforge3\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "d:\apps\Miniforge3\Scripts\xdoctest.exe\__main__.py", line 7, in <module>
  File "d:\apps\miniforge3\lib\site-packages\xdoctest\__main__.py", line 160, in main
    run_summary = xdoctest.doctest_module(modname, argv=[command], style=style,
  File "d:\apps\miniforge3\lib\site-packages\xdoctest\runner.py", line 302, in doctest_module
    run_summary = _run_examples(enabled_examples, verbose, config,
  File "d:\apps\miniforge3\lib\site-packages\xdoctest\runner.py", line 465, in _run_examples
    summary = example.run(verbose=verbose, on_error=on_error)
  File "d:\apps\miniforge3\lib\site-packages\xdoctest\doctest_example.py", line 612, in run
    code = compile(
  File "<doctest:D:\forks\HACE\hacelibs\hacelibs\commonlib.py::slurpFile:0>", line 1
    import os; slurpFile(os.path.abspath(__file__), mode = 'rb')[:9]
    ^
SyntaxError: invalid syntax

Maybe a parsing error?

[Erotemic]

Looks like a similar issue to #106 but not the exact same.

This breaks:

compile("import os; slurpFile(os.path.abspath(__file__), mode = 'rb')[:9]", filename="", mode='eval')

This works

compile("import os; slurpFile(os.path.abspath(__file__), mode = 'rb')[:9]", filename="", mode='exec')
compile("import os; slurpFile(os.path.abspath(__file__), mode = 'rb')[:9]", filename="", mode='single')

[Erotemic]

I added a fix to this issue in the other parse patch:

#107

This issue was to detect this and change the "mode_hint" (which I had a nice change to refactor from an awful non-descriptive name I will not utter here) such that if there was a semicolon in the doctest code it would change from "eval" to "single". This was done in a robust way with the tokenize package.

This does add further overhead to the parser (I probably pass the code to tokenize / ast.parse more than is needed, so there is a lot of room to refactor and improve there), but it doesn't seem noticeable yet, and I care about getting correctness right first. Just noting this so I don't forget that the parser does need an efficiency refactor. There is likely a way we would only need to tokenize and run the ast once. If not, I think we can reduce the number of calls to these expensive steps.

[Erotemic]

This should be fixed in the released 0.15.9. Please let me know if there is still an issue.