Show executed code for JN during autograding

[cindygr]

[I've been using otter-grader for a year now - thank you, despite some problems I've run into I really like how I can use JN, Python, an IDE, and still have autograding work in Gradescope]

I am running Python 3.11 with the latest version of otter. Last year (2 years ago?) I vaguely recall that the autograder would show the results of running the JN (or at least generated errors) not just the results of running the tests on the JN output. I just ran into a problem where the student's notebooks (and mine) would work fine on our machines (windows/mac) but not when submitted to Gradescope. I finally realized the problem was that the data directory was not capitalized correctly (which mac & windows "fix", but Ubuntu does not...) Long story short, if the results of running the JN (or at least any errors that were generated) had showed up in the results window, this would have been much easier to debug.

(BTW, I did do the debug w/ssh, but that didn't really help much, because doing run_autograder generates the exact same thing - I was deep in trying to figure out where in the otter code it actually ran the JN when I realized the file name problem.)

Describe the solution you'd like
Is it possible to have a flag (or does one exist?) that shows any errors generated by the JN when it is executed, and not just the results of the test?

Describe alternatives you've considered
Or... the documentation for running the otter grade manually is very confusing. It mentions doing otter grade -n name .ipynb, but not what name should be or how that relates to the run_autograder batch file...

So if there's a way to run the autograder from the debug ssh and tell it to spew out everything, documenting that would fix the problem. (and maybe add a line to the doc about how you actually run it in debug ssh, and that you need your github public key password when you log into debug ssh).

Additional context
Nothing I can think of. Thanks!

[chrispyles]

There is a way to use Gradescope's debug via SSH (or otter grade or otter run) to do this. If you set "debug": true in the otter_config.json file in the autograder configuration zip file, this will tell otter to stop executing when it reaches an error and raise the error. To use this with Gradescope's debug via SSH feature (so you don't have to rebuild the grading image), you can just add this key to /autograder/source/otter_config.json and then run the run_autograder script. I don't think this is documented clearly anywhere, so I'll leave this issue open to track adding these steps to the documentation.

N.B. to do a similar thing with otter grade, you can create a docker container using the grading image it created for you (the value of the -n flag is the tag name, so otter-grade -n hw01 creates an image called otter-grade:hw01) with docker run -it otter-grade:hw01 bash. Then copy a submission into the container with docker cp subm.ipynb <container id>:/autograder/submission/subm.ipynb. Then in the bash prompt opened by the first command you can recreate the steps above as if you were SSHed into the Gradescope container.

[chrispyles]

this section describes the assignment names but doesn't actually mention that this is the -nflag, which is a pretty large oversight.

[cindygr]

If you could kindly add to the documentation instructions for actually running the tests once you've logged in (you do "run_autograder" in that folder to run the autograder...) Seems stupid, but when you login there are multiple run files, that all kinda do the same thing, but it's confusing.

Is it true that if I set "show_stdout" to true, then this would have solved my problem (i.e., it would show the results of the JN calculation before the tests are run)?

[chrispyles]

If you could kindly add to the documentation instructions for actually running the tests once you've logged in (you do "run_autograder" in that folder to run the autograder...) Seems stupid, but when you login there are multiple run files, that all kinda do the same thing, but it's confusing.

will do

Is it true that if I set "show_stdout" to true, then this would have solved my problem (i.e., it would show the results of the JN calculation before the tests are run)?

No. show_stdout controls whether students are shown the autograder stdout when grades are published; instructors can always seen the autograder stdout. To debug this without using debug via SSH, you would've needed to set debug in the otter_config.json in the autograder zip file, reupload it to Gradescope and rebuild the image, and then rerun the autograder on the submission. (You would've also had to upload the old zip file after debugging to ensure that other submissions didn't run with the debug flag enabled.)

[chrispyles]

#759 adds a debugging guide to the docs