Skip to content

Commit

Permalink
add documentation about dask gateway
Browse files Browse the repository at this point in the history
closes #7
  • Loading branch information
zonca committed Feb 22, 2021
1 parent 0550703 commit 4028959
Showing 1 changed file with 18 additions and 0 deletions.
18 changes: 18 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,24 @@ the user can remove them by deleting the right folder from:

~/.local/share/jupyter/kernels/

### Use Dask for distributed computing

Dask is a Python package that has an array syntax very similar to `numpy` but under the hood can distribute the computation across many different "workers".
In the SuperCDMS JupyterHub deployment on Jetstream, all users by default have access only to the Jupyter Notebook instance. However, they can leverage Dask to launch more workers inside the Kubernetes cluster and parallelize their computation, having access to way more CPU and RAM.

Access to Dask is available via ["Dask Gateway"](https://gateway.dask.org/), check their documentation to know more. The deployment itself is explained in [this tutorial](https://zonca.dev/2020/08/dask-gateway-jupyterhub.html), but this is not necessary for users.

Before testing it, please notify `@zonca` on [the relevant github issue](https://github.com/det-lab/jupyterhub-deploy-kubernetes-jetstream/issues/51).

In order to test it:

* check [this notebook](https://nbviewer.jupyter.org/gist/zonca/355a7ec6b5bd3f84b1413a8c29fbc877)
* download it to your local machine clicking on the download icon on the top
* upload it to the SuperCDMS JupyterHub and open it
* replace the `js-XXX-YYY` string in one of the first cells to `supercdms`
* add `services/dask-gateway` to the URL to access the Dask dashboard
* execute the whole notebook

## Information about the deployment

* Check project progress at <https://github.com/det-lab/jupyterhub-deploy-kubernetes-jetstream/projects/1>
Expand Down

0 comments on commit 4028959

Please sign in to comment.