Otter assign - tests fail when answers are correct, "Error: object not found"

[jerrybonnell]

Describe the bug
I am following the otter sample at https://github.com/ucbds-infra/ottr-sample to create an assignment, and all of my tests are failing on Gradescope in the tested Rmd submission (a submission with all correct solutions). It is notable that all tests pass successfully when testing locally using otter run lab04.Rmd. I am attaching a ZIP file containing the master notebook file that I am using and a screenshot of the failing tests. It seems that the tests are unable to recognize any of the variables that are created in the immediately preceding solution cell.

Archive.zip

To Reproduce
Steps to reproduce the behavior:

1. Unzip the Archive.zip attached. cd into its directory.
2. Use Otter assign to prepare the autograder.zip file: otter assign lab04.Rmd dist. Note that the header of the master notebook has the following:

BEGIN ASSIGNMENT
requirements: requirements.R
generate: true

3. Upload the autograder.zip from dist/autograder to Gradescope.
4. Test the autograder using the solution Rmd file.
5. The above error is produced.

Expected behavior
All tests should pass successfully for the tested submission .Rmd file on Gradescope.

Versions
Python version: 3.9.5
Otter-Grader version: 2.2.2

[jerrybonnell]

Apologizes for the double post -- I have updated my issue with a reproducible master Rmd file (please see the attached ZIP in original post)

[chrispyles]

Hi @jerrybonnell -- sorry for the long wait. I think the issue here is that your dependencies aren't installing correctly. For some reason, packages won't install on Gradescope unless you specify a CRAN mirror. Try adding repos="http://cran.us.r-project.org" in the call to install.packages in your requirements.R file and let me know if that fixes the issue.

[jerrybonnell]

Thank you @chrispyles! That fixes it. There is one more issue we are experiencing which is that the results from the hidden tests remain hidden on Gradescope even after the grades are published -- students only get to see how they did on the public tests. We tried the following settings in the assignment header but no dice:

BEGIN ASSIGNMENT
requirements: requirements.R
generate:
   show_hidden: true
   show_stdout: true

Let me know if it would be best to post this problem as a separate issue. Thanks again!

[chrispyles]

I don't think this is possible with the latest stable release of ottr.

This is something that should be fixed with the new releases of Otter v3 and Ottr v1 (which should be in beta shortly).

[chrispyles]

Although IIRC there should be a dataframe printed out summarizing scores by test case (incl. hidden tests) which can be shown to students by setting show_stdout: true.

[chrispyles]

I've actually just released Otter 3.0.0.b0 and Ottr 1.0.0.b0. If you want to install these you can test out the new behavior.

$ pip install otter-grader==3.0.0.b0

devtools::install_github("ucbds-infra/ottr@1.0.0.b0")

[jerrybonnell]

Although IIRC there should be a dataframe printed out summarizing scores by test case (incl. hidden tests) which can be shown to students by setting show_stdout: true.

Perfect -- yes, this works.

I've actually just released Otter 3.0.0.b0 and Ottr 1.0.0.b0. If you want to install these you can test out the new behavior.

$ pip install otter-grader==3.0.0.b0

devtools::install_github("ucbds-infra/ottr@1.0.0.b0")

I tried the new version, but the autograder errored when building. It looks like it had trouble installing the "stringi" package.

[chrispyles]

Can you paste the error message @jerrybonnell?

[jerrybonnell]

Here is what I get (full log is here:

ERROR: configuration failed for package ‘stringi’
* removing ‘/root/miniconda3/envs/otter-env/lib/R/library/stringi’
* restoring previous ‘/root/miniconda3/envs/otter-env/lib/R/library/stringi’
Error in i.p(...) : 
  (converted from warning) installation of package ‘stringi’ had non-zero exit status
Calls: <Anonymous> ... with_rprofile_user -> with_envvar -> force -> force -> i.p
Execution halted

*** *********************************************************************
*** stringi cannot be built with these settings in place.
*** See the INSTALL file for the solutions to the most common problems.
*** Moreover, explore the list of open and closed issues at
*** https://github.com/gagolews/stringi/issues/
*** ********************************************************************

This is after running on the example Archive.zip in my original post and with Otter version 3.0.0.b0.

[chrispyles]

@jerrybonnell I've just released Otter v3.0.0.b1 which should fix this issue. Can you install and let me know if that works?

[jerrybonnell]

@chrispyles I installed Otter v3.0.0.b1 and can confirm the "stringi" issue is fixed. However, I tried testing the built autograder with the student file that has no solutions (in dist/student) and the autograder is marking everything correct when it shouldn't -- any ideas how to remedy this? The behavior can be observed using the same Archive.zip file.

Thanks so much for your help!

[chrispyles]

@jerrybonnell Thanks for pointing this out. I forgot to mention some changes to the Rmd format for Otter Assign that this new version brings. I'm attaching an updated Rmd file but the the basic difference is that test cells should now just raise errors, not be wrapped in calls to testthat::test_that. You can still use testthat's expect_* functions to raise those errors, as the attached does. I ran the output of the attached through Gradescope and I think it should work.

Archive.zip

[jerrybonnell]

@chrispyles Thanks for this -- yes that fixed it. There are two minor issues I am having now:

• It seems the point values in the Rmd file are being ignored. Every test seems to always be worth 1 point.
• In Otter v3.0.0.b1 how are manually graded questions handled? I believe the documentation mentioned that the autograder will produce a PDF with the manual questions, but I am not finding it.

[chrispyles]

• Yes, this is an issue I'm aware of. I think I fixed it with Update docs to reflect the new version of Ottr #300 and this should be included in v3.0.0.b2.
• PDF generation is not currently supported with R assignments, only Python. It is a feature we hope to add but I'm not sure yet if it will be included in v3.

[jerrybonnell]

Perfect -- thanks a lot! Is there any way to know when v3.0.0.b2 will be released?

[chrispyles]

Shortly (tonight 🤞). I've got another feature to add but it shouldn't be a big one, then it can be released.

[chrispyles]

Just released v3.0.0.b2. Also, you may want to follow #302 which is the issue that will track adding PDF support for R.

[chrispyles]

@jerrybonnell did v3.0.0.b2 resolve your issue?

[jerrybonnell]

@chrispyles Thanks for checking in -- sorry for my late reply. Yes, v3.0.0.b2 fixed the points issue and everything looks in order 👍

I will reach out when we come across more issues. I followed #302 for PDF support.