Add submission debugging guide

docs/_static/grading-environment-r.yml

@@ -18,6 +18,7 @@ dependencies:
   - r-startup
   - r-rmarkdown
   - r-stringi
+  - r-ottr==1.4.0
   - pip:
       - datascience
       - jupyter_client

docs/_static/r_setup.sh

@@ -30,7 +30,4 @@ mamba env create -f /autograder/source/environment.yml
 mamba run -n otter-env Rscript /autograder/source/requirements.r
 
 # set mamba shell
-mamba init --all
-
-# install ottr
-mamba run -n otter-env Rscript -e 'install.packages("https://cran.r-project.org/package=ottr&version=", dependencies=TRUE, repos=NULL)'
+mamba init --all

docs/debugging.rst

@@ -0,0 +1,87 @@
+Debugging Submissions
+=====================
+
+To help debug submissions that are not passing tests or which are behaving oddly, it can be helpful
+to use Otter's debug mode. The main use for debug mode is to examine errors that are being thrown
+in the submission and which Otter swallows during normal execution. This is a guide for setting up
+an environment where Otter's debug mode is enabled and for using it.
+
+
+Setting Up a Debugging Environment
+----------------------------------
+
+The first step is to set up a debugging environment. This step depends on how you're executing the
+submissions. If you're using containerized local grading (``otter grade``) or Gradescope, follow the
+steps in this section. If you're using non-containerized grading (``otter run``), follow the steps
+in :ref:`debugging_with_otter_run`.
+
+
+Otter Grade Setup
++++++++++++++++++
+
+To create a container for debugging a submission, use the Docker image that was created for your
+assignment when you ran ``otter grade``. The image's tag will use the assignment name (the value of
+the ``-n`` or ``--name`` flag). If you passed ``-n hw01``, the image name will be
+``otter-grade:hw01``. Create a container using this image and run bash:
+
+.. code-block:: console
+
+    docker run -it otter-grade:<assignment name> bash
+
+This should create the container and drop you into a bash prompt inside of it. Now, in a separate
+terminal, copy the submission file into the container you've just created:
+
+.. code-block:: console
+
+    docker cp path/to/subm.ipynb <container id>:/autograder/submission/subm.ipynb
+
+You can get the container ID either from the prompt in the container's bash shell or from the
+``docker container ls`` command.
+
+Now that you've set up the grading environment, continue with the debugging steps
+:ref:`below <debugging_container>`. Once you're finished debugging, remember to stop the container
+and remove it.
+
+
+Gradescope Setup
+++++++++++++++++
+
+Find the submission you would like to debug on Gradescope and use Gradescope's `Debug via SSH
+feature <https://gradescope-autograders.readthedocs.io/en/latest/ssh/>`_ to create a container with
+that submission and SSH into it, then continue with the debugging steps
+:ref:`below <debugging_container>`.
+
+
+.. _debugging_container:
+
+Debugging in a Container
+------------------------
+
+In the container, ``cd`` into the ``/autograder/source`` directory. There should be an
+``otter_config.json`` file in this directory containing Otter's autograder configuration. (If there
+isn't one present, make one.) Edit this file to set ``"debug"`` to ``true`` (this enables debug
+mode).
+
+Now ``cd`` into ``/autograder`` and run
+
+.. code-block:: console
+
+    ./run_autograder
+
+This shell script runs Otter and its results should be printed to the console. If an error is thrown
+while executing the notebook, it will be shown and the autograder will stop. If you're running on
+Gradescope, no changes or autograder runs made here will affect the submission or its grade.
+
+If you need to edit the submission file(s), they are located in the ``/autograder/submission``
+directory.
+
+
+.. _debugging_with_otter_run:
+
+Debugging with Otter Run
+------------------------
+
+Read the :ref:`previous section <debugging_container>` first. Because Otter Run relies on an
+autograder zip file for its configuration intsead of a Docker container, you will need to manually
+edit the ``otter_config.json`` file in your autograder zip file to set ``"debug"`` to ``true``.
+Then, re-zip the zip file's contents and use this new autograder zip file for ``otter run``.

docs/index.rst

@@ -13,6 +13,7 @@ Otter-Grader Documentation
     otter_check/index
     workflow/index
     execution
+    debugging
     plugins/index
     pdfs
     seeding

docs/workflow/executing_submissions/otter_grade.rst

@@ -22,12 +22,12 @@ re-downloaded when the image for an assignment is built for the first time.
 Assignment Names
 ++++++++++++++++
 
-Whenever you use Otter Grade, you must specify an assignment name. This assignment name is used as
-the tag for the Docker image that Otter creates (so you will have an image called
-``otter-grade:{assignment name}`` for each assignment). These assignment names are required so that
-Otter can make effective user of Docker's image layer cache. This means that if you make changes to
-tests or need to grade an assignment twice, Docker doesn't need to reinstall all of the dependencies
-Otter defines.
+Whenever you use Otter Grade, you must specify an assignment name with the ``-n`` or ``--name``
+flag. This assignment name is used as the tag for the Docker image that Otter creates (so you will
+have an image called ``otter-grade:{assignment name}`` for each assignment). These assignment names
+are required so that Otter can make effective user of Docker's image layer cache. This means that if
+you make changes to tests or need to grade an assignment twice, Docker doesn't need to reinstall all
+of the dependencies Otter defines.
 
 These images can be quite large (~4GB), so Otter provides a way to easily prune all of the Docker
 images it has created:

otter/check/logs.py

@@ -14,17 +14,6 @@
 class EventType(Enum):
     """
     Enum of event types for log entries
-
-    Attributes:
-        AUTH: an auth event
-        BEGIN_CHECK_ALL: beginning of a check-all call
-        BEGIN_EXPORT: beginning of an assignment export
-        CHECK: a check of a single question
-        END_CHECK_ALL: ending of a check-all call
-        END_EXPORT: ending of an assignment export
-        INIT: initialization of an ``otter.Notebook`` object
-        SUBMIT: submission of an assignment
-        TO_PDF: PDF export of a notebook
     """
 
     AUTH = auto()