"Public tests" section on gradescope show "Passed" even when tests are failing

[joelostblom]

Describe the bug
The "Public tests" section on gradescope always shows up as "Passed", even when some of the tests fail:

In the stdout they always show up as NA for points, regardless how many points there are:

Expected behavior

The headings does not show that everything passed when some public tests fail.

In general I think it would be good to remove this section if show_hidden is used in the metadata, because otherwise the info about these tests will show up twice, both under "Public" and then again under the question header, which can be confusing.

Versions

otter 4.0.2 python 3.9.13

Additional context
Add any other context about the problem here.

[chrispyles]

I think that the Public Tests section has always shown as passing since it's never had a point value assigned to it, but now that Gradescope lets you customize how it shows, I agree that showing it as failing when some public tests don't pass is a good idea.

w.r.t. not showing it if show_hidden is true, I think that it's fine to leave it shown as the default but make this behavior configurable, e.g. with a show_public_summary configuration. thoughts?

[joelostblom]

An option to control whether the public section is shown would be nice! I do think it makes more sense for the default to be to hide it if the tests are already showing up under their respective question heading as I don't see the value of seeing the same test in two places. I don't feel super strongly about that though and maybe I am missing something useful with seeing it twice?

[chrispyles]

The reason I think leaving it shown as the default is better is because show_hidden only makes the results of the hidden tests visible after grades are published, so not showing the public tests summary would mean that the student gets no feedback on their submission until grades are published.

[chrispyles]

Just merged #549 which fixes this bug and adds the force_public_test_summary configuration, which defaults to true. If you want to hide the public tests summary when show_hidden is true, set force_public_test_summary to false. This will be included in v4.1.0.

[joelostblom]

Awesome, thank for adding that so quickly!

The reason I think leaving it shown as the default is better is because show_hidden only makes the results of the hidden tests visible after grades are published, so not showing the public tests summary would mean that the student gets no feedback on their submission until grades are published.

Ah ok that makes sense. I was thinking that students would always get this feedback locally in their own notebook, but I realize that it is possible to have a workflow where you design assignments that don't include those test cells in the notebook itself and they would have to view it on gradescope instead.

[joelostblom]

@chrispyles I updated my autograder image to otter 4.1.0 and used it for an assignment that I distributed using otter 4.0.2. In this context, I can see that the Public tests now correctly show up in the failed section when a test is failing, thanks for that! What does not seem to work for me, is the new force_public_test_summary: false, I still see the Public test section even after adding this to my autograder image:

Full assignment config

requirements: requirements.txt
export_cell: false
check_all_cell: false
show_question_points: true
force_public_test_summary: false
generate:
    show_hidden: true
    show_stdout: true

[chrispyles]

force_public_test_summary needs to be inside generate

[joelostblom]

Ah of course, it works now, thanks!