Auto-grade a student assignment created as a Jupyter notebook, using the nbgrader Jupyter extension, and write the score in the Open edX gradebook
See also the Jupyter Notebook Viewer XBlock to populate course content from publicly available Jupyter notebooks.
This XBlock uses Docker and nbgrader to create a Python environment and auto-grade a Jupyter Notebook, and tracks the resulting score as a problem within an EdX graded sub-section.
It allows an instructor to upload an assignment created with nbgrader, upload a requirements.txt
file to configure the environment, set the maximum number of tries for the student, and set a deadline for the submission.
The student downloads the assignment file, answers the questions (executing all cells), and uploads the solution, which gets immediately auto-graded.
The student gets a visual score report, and the score gets added to his/her progress in the Open edX gradebook.
- Integrated into EdX Grading System
- Maximum point values are pulled from the instructor version of the notebook
- A separate Python3 virtual environment is kept for each course
- Each student's notebook is run within its own Docker container
- Several Other Configuration Options
- Only supports auto-graded cells - Does not support manually graded cells.
Efforts have been made to isolate the student notebook from the EdX Server by running them in an isolated docker container with no network access (by default). The only links back to the EdX server when the notebook is executed are the following mapped files/folders:
- /var/www/nbgrader/courses/<course-id>/source/<unit-id>/<nb-name> (set to read-only)
- /var/www/nbgrader/courses/<course-id>/submitted/<username>/<unit-id>/<nb-name> (set to read-only)
- /var/www/nbgrader/courses/<course-id>/autograded/<username>/<unit-id>/<nb-name>
- /var/www/nbgrader/courses/<course-id>/feedback/<username>/<unit-id>/<nb-name>
- /tmp/<random-filename> (temporary config file, set to read-only)
PLEASE REMEMBER: Jupyter Notebooks do allow arbitrary code execution
Until more granular network controls are implemented it would be best to leave network access off unless you know and trust your student population
- login as the root user:
sudo -i
- New Installation:
/edx/bin/pip.edxapp install git+https://github.com/ibleducation/jupyter-edx-grader-xblock.git
- Re-Installation:
/edx/bin/pip.edxapp install --upgrade --no-deps --force-reinstall git+https://github.com/ibleducation/jupyter-edx-grader-xblock.git
- Install Docker
- Create the
docker
group (this may already be done via installing docker)sudo groupadd docker
- Create jupyter user
adduser --no-create-home jupyter
- A strong password is recommended as any user within the
docker
group is considered a user with root privileges
- A strong password is recommended as any user within the
usermod -aG docker jupyter
- Allow
www-data
user to switch tojupyter
user and executedocker
commands without a password- As a root user, enter:
visudo
- Enter the following line at the bottom of the file:
www-data ALL=(jupyter) NOPASSWD:/usr/bin/docker
- This allows
www-data
user to executedocker
commands as thejupyter
user without using a password
- As a root user, enter:
In the following two files:
/edx/app/edxapp/edx-platform/lms/urls.py
/edx/app/edxapp/edx-platform/cms/urls.py
Add the following to the bottom of each file:
# Jupyter Graded XBlock Endpoints
urlpatterns += (
url(r'^api/jupyter_graded/', include('xblock_jupyter_graded.rest.urls', namespace='xblock_jupyter_graded')),
)
In the following two files:
/edx/app/edxapp/edx-platform/lms/envs/common.py
/edx/app/edxapp/edx-platform/cms/envs/common.py
Add the following at the bottom of the INSTALLED_APPS
section:
# Jupyter Notebook Graded XBlock
'xblock_jupyter_graded',
Restart edxapp
via /edx/bin/supervisorctl restart edxapp:
Restart edxapp_worker
's via /edx/bin/supervisorctl restart edxapp_worker:
Run the following as a root user to create the necessary database models:
sudo su edxapp -s /bin/bash
cd ~/edx-platform
source ../venvs/edxapp/bin/activate
./manage.py cms migrate xblock_jupyter_graded --settings=aws
Currently, this XBlock makes some long running Ajax calls which can exceed the default nginx timeout of 60s on some machines. We will temporarily increase this timeout until the architecture can be updated to make this a background task that can be polled for state.
In the followingfile:
/etc/nginx/nginx.conf
Add the following line in the http
section under # Basic Settings
:
proxy_read_timeout 300;
Restart nginx via: service nginx restart
Login to the EdX Studio and navigate to the course you would like to implement the XBlock in.
On the top Menu bar, select:
- Settings => Advanced Settings
In the Advanced Module List
, add:
- "xblock_jupyter_graded"
Click Save
at the bottom of the screen.
On the main author screen in the Studio, there are three major instructor related sections:
- Python Virtual Environment
- Instructor Notebook Upload
- Uploaded Notebook Details
- Allows you select a
requirements.txt
file to upload for this course.- You only need to add the packages that are imported from within the notebook.
- You do not need to include nbgrader, jupyter, ipykernel, or any other jupyter related packages.
ipykernel
is included by default as it is required to install the student virtualenv as an execution environment
- This python environment will be shared by all Notebooks in this course
- A list of the currently installed packages is listed under Installed Packages
requirements.txt
should be formatted like a normalrequirements.txt
file, but only supports the following formats:package_name==package_version
- Pinned version of a packagepackage_name
- Unpinned version - installs newest version of the package
- The docker container will be built each time a new environment is uploaded that doesn't already exist
- Allows you to select and upload a
source
notebook that will be used to generate the student version.
- Shows details about the currently uploaded notebook and allows the instructor to download the version currently attached.
- Display Name: The title of this XBlock that the student will see
- Student Instructions: A set of instructions that will be shown to the student.
- Allowed Submissions (Default: 0): The max number of submissions that a student may make. If set to
0
, infinite attempts are allowed. - Network Allowed (Default: False): If True, allows network access from within the container.
- CAUTION: Because Jupyter Notebooks allow arbitrary execution of code, it is best to leave this to
False
otherwise a student could make unauthorized web requests from the perspective of the server. - More work should be done here to create more granular network isolation
- CAUTION: Because Jupyter Notebooks allow arbitrary execution of code, it is best to leave this to
- Cell Timeout (Default: 15s): Max amount of time to allow a single cell to execute before raising a
KeyboardInterrupt
Exception to attempt to stop the cell - Allow Graded NB Download (Default: False): If
True
, allows student to download the autograded.html
file generated by runningnbgrader feedback
.- Note: This will show all
### HIDDEN TESTS
from the instructor notebook as they are included in the autograded feedback
- Note: This will show all
- Max File Size (B): Maximum file size student is allowed to upload (in bytes).
- Default is 10kB > the size of the instructor version of the notebook if this field is unset when the instructor version is uploaded. Otherwise it can be modified to your liking, but it must be set.
- Add a
Graded Jupyter Notebook
from theAdvanced
button in the course Studio. - Optionally upload a
requirements.txt
file containing all packages required for this course. - Create the source jupyter notebook from a local installation of jupyter and nbgrader and save/export it to your system.
- It is suggested to set the cell
ID
value for eachAutograder tests
cell to something meaningful to the student - For instance, if your notebook has two sections and two graded exercises per section:
- Section 1
- Exercise 1
- Exercise 2
- Section 2
- Exercise 1
- Exercise 2
- Section 1
- naming the test cells something like:
section1_excercise1
will help the student associate the point value with a test or section in the notebook when they see their results.
- It is suggested to set the cell
- Upload the source notebook in the
Instructor Notebook Upload
section.- The max score and notebook details will be reflected in the
Uploaded Notebook Details
section. - An error will be thrown if the text
BEGIN SOLUTION
is not found somewhere in the notebook as this likely indicates the appropriate version was not uploaded
- The max score and notebook details will be reflected in the
- The student version will now be generated and can be downloaded in the Student Section
- Update the settings for this XBlock as appropriate
- Publish the Unit
- Select the
Download Student Notebook
link to download the student version - Complete the notebook via a local installation of jupyter
- Upload the completed version of the student notebook
- The notebook must be named the same as what is specified in the
Notebook Name:
label
- The notebook must be named the same as what is specified in the
- The notebook will be graded and results reflected in the
Results
section- If the student has no more attempts left, they will no longer be able to upload a new notebook
- The most recent submission is the one recorded for grading
(c) 2017 IBL Studios and Lorena A. Barba, code is under BSD-3 clause.