Cannot access source code in tests

[drjbarker]

Is your feature request related to a problem? Please describe.
I'm aiming to walk an ast in a test but cannot access the source of the function for example using inspect.getsource(some_function). This results in a OSError: source code not available presumably because we only have access to a compiled object within the grading framework (everything works fine in the original notebook).

In general there seems to be no way to access an ast of the solution cell even though it seems otter generates this to do the evaluation (based on this: https://otter-grader.readthedocs.io/en/latest/execution.html).

Describe the solution you'd like
It would be useful to natively support inspect.getsource within the grader, maybe doing some 'injection' like with the random number seeding.

Describe alternatives you've considered
Possibly allowing access to the ast at least within a test .py file (even if not in the notebook format) as an advanced feature.

[chrispyles]

I believe #231 fixes this issue. Can you try installing that version of Otter and let me know if it works?

pip install git+https://github.com/ucbds-infra/otter-grader.git@fix-229

[drjbarker]

Doesn't seem to work. I've made a minimal example and tested (ast_test.ipynb.zip). The issue seems to be that doctest can't find the source because it never exists in a file.

ast_test results:

Trying:
    import ast;
Expecting nothing
ok
Trying:
    import inspect;
Expecting nothing
ok
Trying:
    found = False;
Expecting nothing
ok
Trying:
    for node in ast.walk(ast.parse(inspect.getsource(bar))):
        if isinstance(node, ast.Call) and hasattr(node, 'func'):
            # must check only for ast.Name because some ast.Call funcs are of type ast.Attribute
            if isinstance(node.func, ast.Name) and node.func.id == 'foo':
                found = True;
Expecting nothing
**********************************************************************
Line 9, in ast_test 0
Failed example:
    for node in ast.walk(ast.parse(inspect.getsource(bar))):
        if isinstance(node, ast.Call) and hasattr(node, 'func'):
            # must check only for ast.Name because some ast.Call funcs are of type ast.Attribute
            if isinstance(node.func, ast.Name) and node.func.id == 'foo':
                found = True;
Exception raised:
    Traceback (most recent call last):
      File "/opt/miniconda3/envs/computing1-beta/lib/python3.8/doctest.py", line 1336, in __run
        exec(compile(example.source, filename, "single",
      File "<doctest ast_test 0[3]>", line 1, in <module>
        for node in ast.walk(ast.parse(inspect.getsource(bar))):
      File "/opt/miniconda3/envs/computing1-beta/lib/python3.8/inspect.py", line 985, in getsource
        lines, lnum = getsourcelines(object)
      File "/opt/miniconda3/envs/computing1-beta/lib/python3.8/inspect.py", line 967, in getsourcelines
        lines, lnum = findsource(object)
      File "/opt/miniconda3/envs/computing1-beta/lib/python3.8/inspect.py", line 798, in findsource
        raise OSError('could not get source code')
    OSError: could not get source code
Trying:
    print(found)
Expecting:
    True
**********************************************************************
Line 14, in ast_test 0
Failed example:
    print(found)
Expected:
    True
Got:
    False

Inside of doctest they play some tricks with the linecache to allow access to the source but only when debugging.

The two options I can think of at the moment are:

• Write the source to a temporary file and read this (not ideal)
• Play similar games with linecache (but I'm not 100% sure that will work)

I'm a bit busy at the moment (writing exams with otter!) but I might be able to take a look at a later date.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[chrispyles]

I think I've implemented a fix for this while refactoring our execution internals for v4. The fix uses tempfile to create a temporary file with the source before compiling, and I tested inspect.getsource in a test with this structure and it worked for me. This will be included as a feature in Otter v4.