From 0246e349130fbf89e2dd22616702bf959c31ab06 Mon Sep 17 00:00:00 2001 From: Markku Riekkinen Date: Wed, 26 May 2021 18:17:46 +0300 Subject: [PATCH] Update outdated information in README and docs --- README.md | 65 +++---- doc/README.md | 8 +- doc/etc-default-celeryd | 29 --- doc/etc-init-uwsgi.conf | 7 - doc/etc-init.d-celeryd | 387 ---------------------------------------- doc/etc-init.d-xvfb | 36 ---- 6 files changed, 37 insertions(+), 495 deletions(-) delete mode 100644 doc/etc-default-celeryd delete mode 100644 doc/etc-init-uwsgi.conf delete mode 100644 doc/etc-init.d-celeryd delete mode 100644 doc/etc-init.d-xvfb diff --git a/README.md b/README.md index 61e1b4bc..04b7337d 100644 --- a/README.md +++ b/README.md @@ -1,14 +1,16 @@ # mooc-grader -This grading service accepts anonymous submissions for grading via HTTP. The -grading can be done synchronously or asynchronously using a submission queue. +This grading service accepts anonymous submissions for grading via HTTP. +The submissions are graded either synchronously in the web application or +asynchronously in containers (usually in Kubernetes, though it is possible +to run Docker containers in the web server for a small number of submissions). The results are delivered to the calling system and the grader does not keep any record other than service logs. The grader is designed to serve exercises -for the A+ learning system. A farm of individual grading servers can be setup -to handle large amount of submissions. +for the A+ learning system. If the number of submissions is large, we recommend +setting up a Kubernetes cluster for the grader. -The grader is implemented on Django 1.9 (`grader/settings.py`). The application -is tested on both Python 2.7 and 3.4. +The grader is implemented on Django 2.2 (`grader/settings.py`). The application +is tested on Python 3.7 and newer versions, but Python 3.5+ should also work. The grader can be run stand alone without the full stack to test graders in the local system environment. The grader is designed to be extended for @@ -17,7 +19,13 @@ different courses and exercise types. Course and exercise configuration is in ## Installing for development -> 6/2014 - Ubuntu 12.04.4 +You may run the app with Docker without installing the whole software stack locally. +It is easy to get started with the aplus-manual course: +[apluslms/aplus-manual](https://github.com/apluslms/aplus-manual). +The Docker image is intended for local development and testing, not production: +[apluslms/run-mooc-grader](https://github.com/apluslms/run-mooc-grader). + +> Ubuntu 20.04 ### 1. Clone the software @@ -31,7 +39,7 @@ Install software git clone https://github.com/apluslms/mooc-grader.git mkdir mooc-grader/uploads -### 2. Python requirements (2.7 should work too) +### 2. Python requirements sudo apt-get install python3 python3-dev python3-pip python3-venv @@ -43,13 +51,24 @@ Then, create virtual environment with grader requirements. pip install wheel pip install -r mooc-grader/requirements.txt +If you enable the gitmanager app in the MOOC-Grader (see the section +"Django application settings for deployment"), you need to install its +requirements in the (virtual) environment that it uses. It may use a different +environment than the Django application. + + pip install -r mooc-grader/requirements_gitmanager.txt + # If you need to use the old Sphinx version 1.6, then install this one instead: + pip install -r mooc-grader/requirements_gitmng_sphinx16.txt + ### 3. Testing grader application +Run the Django app locally: + cd mooc-grader python manage.py runserver -In addition, the exercise configuration and grading of individual -exercises can be tested from command line. +The exercise configuration and grading of individual +exercises can be tested from the command line. python manage.py exercises python manage.py grade @@ -60,7 +79,7 @@ exercises can be tested from command line. ## Installing the full stack -> 6/2014 - Ubuntu 12.04.4 +> Ubuntu 20.04 ### 0. User account @@ -87,27 +106,6 @@ user account. Install uwsgi to run WSGI processes. The **mooc-grader directory and user must** be set in the configuration files. -#### uWSGI with Upstart (Ubuntu < 15.04) (deprecated) - - source venv/bin/activate - pip install uwsgi - sudo mkdir -p /etc/uwsgi - sudo mkdir -p /var/log/uwsgi - sudo cp doc/etc-uwsgi-grader.ini /etc/uwsgi/grader.ini - sudo cp doc/etc-init-uwsgi.conf /etc/init/uwsgi.conf - # EDIT /etc/uwsgi/grader.ini - # EDIT /etc/init/uwsgi.conf - sudo touch /var/log/uwsgi/grader.log - sudo chown -R [shell-username]:users /etc/uwsgi /var/log/uwsgi - -NOTE that the ownership of the log file is required for graceful -restarts using touch. Operate the workers using: - - sudo status uwsgi - sudo start uwsgi - # Graceful application reload - touch /etc/uwsgi/grader.ini - #### uWSGI with systemd (Ubuntu >= 15.04) source ~/venv/bin/activate @@ -163,6 +161,9 @@ Django must install the database schema for the `gitmanager` (Python virtual env python manage.py migrate +Note that if the file path of the Sqlite database is changed in `local_settings.py`, +the same path must also be hardcoded in the code `gitmanager/cron.sh`. + The `gitmanager` requires a crontab for the grader account: sudo crontab -u grader doc/gitmanager-crontab diff --git a/doc/README.md b/doc/README.md index d9f4d7b3..57177441 100644 --- a/doc/README.md +++ b/doc/README.md @@ -1,5 +1,5 @@ * For installation, see /README.md -* For exercise configuration, see /exercises/README.md +* For exercise configuration, see /courses/README.md # Grader Filesystem Walkthrough @@ -9,7 +9,7 @@ * `/templates`: Base templates for default grader pages. -* `/static`: Statical files for default grader pages. +* `/static`: Static files for default grader pages. * `/access`: Django application presenting exercises and accepting submissions. @@ -21,7 +21,7 @@ * `/util`: Utility modules for HTTP, shell, filesystem access etc. -* `/exercises`: Course directories holding exercise configuration and material. +* `/courses`: Course directories holding exercise configuration and material. * `sample_course`: Different exercise types sampled. @@ -30,5 +30,5 @@ * `/uploads`: Asynchronous graders store submission data in unique directories here. After accepting submission a `user` subdirectory holds the user data. Grading actions get this directory as a parameter and can change the - contents. When grading is finished and feedback sent the submission + contents. When grading is finished and feedback sent, the submission data is removed and submission is completely forgotten. diff --git a/doc/etc-default-celeryd b/doc/etc-default-celeryd deleted file mode 100644 index 2076e0d9..00000000 --- a/doc/etc-default-celeryd +++ /dev/null @@ -1,29 +0,0 @@ -# Names of nodes to start -CELERYD_NODES="tasks" - -# Absolute or relative path to the 'celery' command: -CELERY_BIN="/srv/grader/venv/bin/celery" - -# App instance to use -CELERY_APP="grader.celery" - -# Where to chdir at start. -CELERYD_CHDIR="/srv/grader/mooc-grader/" - -# Extra command-line arguments to the worker -# --concurrency: amount of concurrent task processes per worker -CELERYD_OPTS="--concurrency=1" - -# %N will be replaced with the first part of the nodename. -CELERYD_LOG_FILE="/var/log/celery/%N.log" -CELERYD_PID_FILE="/var/run/celery/%N.pid" - -# Workers should run as an unprivileged user. -# You need to create this user manually (or you can choose -# a user/group combination that already exists, e.g. nobody). -CELERYD_USER="grader" -CELERYD_GROUP="grader" - -# If enabled pid and log directories will be created if missing, -# and owned by the userid/group configured. -CELERY_CREATE_DIRS=1 diff --git a/doc/etc-init-uwsgi.conf b/doc/etc-init-uwsgi.conf deleted file mode 100644 index e52011a8..00000000 --- a/doc/etc-init-uwsgi.conf +++ /dev/null @@ -1,7 +0,0 @@ -description "uWSGI Python Web Server" -start on runlevel [2345] -stop on runlevel [06] - -respawn - -exec /srv/grader/venv/bin/uwsgi --emperor /etc/uwsgi diff --git a/doc/etc-init.d-celeryd b/doc/etc-init.d-celeryd deleted file mode 100644 index 0fe704fd..00000000 --- a/doc/etc-init.d-celeryd +++ /dev/null @@ -1,387 +0,0 @@ -#!/bin/sh -e -# ============================================ -# celeryd - Starts the Celery worker daemon. -# ============================================ -# -# :Usage: /etc/init.d/celeryd {start|stop|force-reload|restart|try-restart|status} -# :Configuration file: /etc/default/celeryd -# -# See http://docs.celeryproject.org/en/latest/tutorials/daemonizing.html#generic-init-scripts - - -### BEGIN INIT INFO -# Provides: celeryd -# Required-Start: $network $local_fs $remote_fs -# Required-Stop: $network $local_fs $remote_fs -# Default-Start: 2 3 4 5 -# Default-Stop: 0 1 6 -# Short-Description: celery task worker daemon -### END INIT INFO -# -# -# To implement separate init scripts, copy this script and give it a different -# name: -# I.e., if my new application, "little-worker" needs an init, I -# should just use: -# -# cp /etc/init.d/celeryd /etc/init.d/little-worker -# -# You can then configure this by manipulating /etc/default/little-worker. -# -VERSION=10.0 -echo "celery init v${VERSION}." -if [ $(id -u) -ne 0 ]; then - echo "Error: This program can only be used by the root user." - echo " Unprivileged users must use the 'celery multi' utility, " - echo " or 'celery worker --detach'." - exit 1 -fi - - -# Can be a runlevel symlink (e.g. S02celeryd) -if [ -L "$0" ]; then - SCRIPT_FILE=$(readlink "$0") -else - SCRIPT_FILE="$0" -fi -SCRIPT_NAME="$(basename "$SCRIPT_FILE")" - -DEFAULT_USER="celery" -DEFAULT_PID_FILE="/var/run/celery/%n.pid" -DEFAULT_LOG_FILE="/var/log/celery/%n.log" -DEFAULT_LOG_LEVEL="INFO" -DEFAULT_NODES="celery" -DEFAULT_CELERYD="-m celery worker --detach" - -CELERY_DEFAULTS=${CELERY_DEFAULTS:-"/etc/default/${SCRIPT_NAME}"} - -# Make sure executable configuration script is owned by root -_config_sanity() { - local path="$1" - local owner=$(ls -ld "$path" | awk '{print $3}') - local iwgrp=$(ls -ld "$path" | cut -b 6) - local iwoth=$(ls -ld "$path" | cut -b 9) - - if [ "$(id -u $owner)" != "0" ]; then - echo "Error: Config script '$path' must be owned by root!" - echo - echo "Resolution:" - echo "Review the file carefully and make sure it has not been " - echo "modified with mailicious intent. When sure the " - echo "script is safe to execute with superuser privileges " - echo "you can change ownership of the script:" - echo " $ sudo chown root '$path'" - exit 1 - fi - - if [ "$iwoth" != "-" ]; then # S_IWOTH - echo "Error: Config script '$path' cannot be writable by others!" - echo - echo "Resolution:" - echo "Review the file carefully and make sure it has not been " - echo "modified with malicious intent. When sure the " - echo "script is safe to execute with superuser privileges " - echo "you can change the scripts permissions:" - echo " $ sudo chmod 640 '$path'" - exit 1 - fi - if [ "$iwgrp" != "-" ]; then # S_IWGRP - echo "Error: Config script '$path' cannot be writable by group!" - echo - echo "Resolution:" - echo "Review the file carefully and make sure it has not been " - echo "modified with malicious intent. When sure the " - echo "script is safe to execute with superuser privileges " - echo "you can change the scripts permissions:" - echo " $ sudo chmod 640 '$path'" - exit 1 - fi -} - -if [ -f "$CELERY_DEFAULTS" ]; then - _config_sanity "$CELERY_DEFAULTS" - echo "Using config script: $CELERY_DEFAULTS" - . "$CELERY_DEFAULTS" -fi - -# Sets --app argument for CELERY_BIN -CELERY_APP_ARG="" -if [ ! -z "$CELERY_APP" ]; then - CELERY_APP_ARG="--app=$CELERY_APP" -fi - -CELERYD_USER=${CELERYD_USER:-$DEFAULT_USER} - -# Set CELERY_CREATE_DIRS to always create log/pid dirs. -CELERY_CREATE_DIRS=${CELERY_CREATE_DIRS:-0} -CELERY_CREATE_RUNDIR=$CELERY_CREATE_DIRS -CELERY_CREATE_LOGDIR=$CELERY_CREATE_DIRS -if [ -z "$CELERYD_PID_FILE" ]; then - CELERYD_PID_FILE="$DEFAULT_PID_FILE" - CELERY_CREATE_RUNDIR=1 -fi -if [ -z "$CELERYD_LOG_FILE" ]; then - CELERYD_LOG_FILE="$DEFAULT_LOG_FILE" - CELERY_CREATE_LOGDIR=1 -fi - -CELERYD_LOG_LEVEL=${CELERYD_LOG_LEVEL:-${CELERYD_LOGLEVEL:-$DEFAULT_LOG_LEVEL}} -CELERY_BIN=${CELERY_BIN:-"celery"} -CELERYD_MULTI=${CELERYD_MULTI:-"$CELERY_BIN multi"} -CELERYD_NODES=${CELERYD_NODES:-$DEFAULT_NODES} - -export CELERY_LOADER - -if [ -n "$2" ]; then - CELERYD_OPTS="$CELERYD_OPTS $2" -fi - -CELERYD_LOG_DIR=`dirname $CELERYD_LOG_FILE` -CELERYD_PID_DIR=`dirname $CELERYD_PID_FILE` - -# Extra start-stop-daemon options, like user/group. -if [ -n "$CELERYD_CHDIR" ]; then - DAEMON_OPTS="$DAEMON_OPTS --workdir=$CELERYD_CHDIR" -fi - - -check_dev_null() { - if [ ! -c /dev/null ]; then - echo "/dev/null is not a character device!" - exit 75 # EX_TEMPFAIL - fi -} - - -maybe_die() { - if [ $? -ne 0 ]; then - echo "Exiting: $* (errno $?)" - exit 77 # EX_NOPERM - fi -} - -create_default_dir() { - if [ ! -d "$1" ]; then - echo "- Creating default directory: '$1'" - mkdir -p "$1" - maybe_die "Couldn't create directory $1" - echo "- Changing permissions of '$1' to 02755" - chmod 02755 "$1" - maybe_die "Couldn't change permissions for $1" - if [ -n "$CELERYD_USER" ]; then - echo "- Changing owner of '$1' to '$CELERYD_USER'" - chown "$CELERYD_USER" "$1" - maybe_die "Couldn't change owner of $1" - fi - if [ -n "$CELERYD_GROUP" ]; then - echo "- Changing group of '$1' to '$CELERYD_GROUP'" - chgrp "$CELERYD_GROUP" "$1" - maybe_die "Couldn't change group of $1" - fi - fi -} - - -check_paths() { - if [ $CELERY_CREATE_LOGDIR -eq 1 ]; then - create_default_dir "$CELERYD_LOG_DIR" - fi - if [ $CELERY_CREATE_RUNDIR -eq 1 ]; then - create_default_dir "$CELERYD_PID_DIR" - fi -} - -create_paths() { - create_default_dir "$CELERYD_LOG_DIR" - create_default_dir "$CELERYD_PID_DIR" -} - -export PATH="${PATH:+$PATH:}/usr/sbin:/sbin" - - -_get_pids() { - found_pids=0 - my_exitcode=0 - - for pid_file in "$CELERYD_PID_DIR"/*.pid; do - local pid=`cat "$pid_file"` - local cleaned_pid=`echo "$pid" | sed -e 's/[^0-9]//g'` - if [ -z "$pid" ] || [ "$cleaned_pid" != "$pid" ]; then - echo "bad pid file ($pid_file)" - one_failed=true - my_exitcode=1 - else - found_pids=1 - echo "$pid" - fi - - if [ $found_pids -eq 0 ]; then - echo "${SCRIPT_NAME}: All nodes down" - exit $my_exitcode - fi - done -} - - -_chuid () { - su "$CELERYD_USER" -c "$CELERYD_MULTI $*" -} - - -start_workers () { - if [ ! -z "$CELERYD_ULIMIT" ]; then - ulimit $CELERYD_ULIMIT - fi - _chuid $* start $CELERYD_NODES $DAEMON_OPTS \ - --pidfile="$CELERYD_PID_FILE" \ - --logfile="$CELERYD_LOG_FILE" \ - --loglevel="$CELERYD_LOG_LEVEL" \ - $CELERY_APP_ARG \ - $CELERYD_OPTS -} - - -dryrun () { - (C_FAKEFORK=1 start_workers --verbose) -} - - -stop_workers () { - _chuid stopwait $CELERYD_NODES --pidfile="$CELERYD_PID_FILE" -} - - -restart_workers () { - _chuid restart $CELERYD_NODES $DAEMON_OPTS \ - --pidfile="$CELERYD_PID_FILE" \ - --logfile="$CELERYD_LOG_FILE" \ - --loglevel="$CELERYD_LOG_LEVEL" \ - $CELERY_APP_ARG \ - $CELERYD_OPTS -} - - -kill_workers() { - _chuid kill $CELERYD_NODES --pidfile="$CELERYD_PID_FILE" -} - - -restart_workers_graceful () { - local worker_pids= - worker_pids=`_get_pids` - [ "$one_failed" ] && exit 1 - - for worker_pid in $worker_pids; do - local failed= - kill -HUP $worker_pid 2> /dev/null || failed=true - if [ "$failed" ]; then - echo "${SCRIPT_NAME} worker (pid $worker_pid) could not be restarted" - one_failed=true - else - echo "${SCRIPT_NAME} worker (pid $worker_pid) received SIGHUP" - fi - done - - [ "$one_failed" ] && exit 1 || exit 0 -} - - -check_status () { - my_exitcode=0 - found_pids=0 - - local one_failed= - for pid_file in "$CELERYD_PID_DIR"/*.pid; do - if [ ! -r $pid_file ]; then - echo "${SCRIPT_NAME} is stopped: no pids were found" - one_failed=true - break - fi - - local node=`basename "$pid_file" .pid` - local pid=`cat "$pid_file"` - local cleaned_pid=`echo "$pid" | sed -e 's/[^0-9]//g'` - if [ -z "$pid" ] || [ "$cleaned_pid" != "$pid" ]; then - echo "bad pid file ($pid_file)" - one_failed=true - else - local failed= - kill -0 $pid 2> /dev/null || failed=true - if [ "$failed" ]; then - echo "${SCRIPT_NAME} (node $node) (pid $pid) is stopped, but pid file exists!" - one_failed=true - else - echo "${SCRIPT_NAME} (node $node) (pid $pid) is running..." - fi - fi - done - - [ "$one_failed" ] && exit 1 || exit 0 -} - - -case "$1" in - start) - check_dev_null - check_paths - start_workers - ;; - - stop) - check_dev_null - check_paths - stop_workers - ;; - - reload|force-reload) - echo "Use restart" - ;; - - status) - check_status - ;; - - restart) - check_dev_null - check_paths - restart_workers - ;; - - graceful) - check_dev_null - restart_workers_graceful - ;; - - kill) - check_dev_null - kill_workers - ;; - - dryrun) - check_dev_null - dryrun - ;; - - try-restart) - check_dev_null - check_paths - restart_workers - ;; - - create-paths) - check_dev_null - create_paths - ;; - - check-paths) - check_dev_null - check_paths - ;; - - *) - echo "Usage: /etc/init.d/${SCRIPT_NAME} {start|stop|restart|graceful|kill|dryrun|create-paths}" - exit 64 # EX_USAGE - ;; -esac - -exit 0 diff --git a/doc/etc-init.d-xvfb b/doc/etc-init.d-xvfb deleted file mode 100644 index 59007b7d..00000000 --- a/doc/etc-init.d-xvfb +++ /dev/null @@ -1,36 +0,0 @@ -#!/bin/sh -e - -XVFB=/usr/bin/Xvfb -XVFBARGS=":0 -screen 0 1024x768x24 -ac +extension GLX +render -noreset -nolisten tcp" -PIDFILE=/var/run/xvfb.pid - -### BEGIN INIT INFO -# Provides: xvfb -# Required-Start: $local_fs $remote_fs -# Required-Stop: $local_fs $remote_fs -# Default-Start: 2 3 4 5 -# Default-Stop: 0 1 6 -# Short-Description: X virtual frame buffer -### END INIT INFO - -case "$1" in - start) - echo -n "Starting virtual X frame buffer: Xvfb" - start-stop-daemon --start --quiet --pidfile $PIDFILE --make-pidfile --background --exec $XVFB -- $XVFBARGS - echo "." - ;; - stop) - echo -n "Stopping virtual X frame buffer: Xvfb" - start-stop-daemon --stop --quiet --pidfile $PIDFILE - echo "." - ;; - restart) - $0 stop - $0 start - ;; - *) - echo "Usage: /etc/init.d/xvfb {start|stop|restart}" - exit 1 -esac - -exit 0 \ No newline at end of file