Skip to content

Commit

Permalink
Update Dask example (#38)
Browse files Browse the repository at this point in the history
  • Loading branch information
pavithraes authored Nov 4, 2022
1 parent dc9ba18 commit b8c3ace
Show file tree
Hide file tree
Showing 6 changed files with 4,312 additions and 190 deletions.
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -130,3 +130,6 @@ dmypy.json

# macOS
.DS_Store

# Dask
dask-worker-space/
22 changes: 14 additions & 8 deletions 02_dask.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -49,10 +49,17 @@
">\n",
"> Dask can scale up to your full laptop capacity and out to a cloud cluster.\n",
">\n",
"> *~ Source: [Dask Tutorial](https://tutorial.dask.org/00_overview.html)*\n",
"\n",
"\n",
"<!-- TODO add screenshot of an ongoin computation and the dask dashboard -->"
"> *~ Source: [Dask Tutorial](https://tutorial.dask.org/00_overview.html)*"
]
},
{
"cell_type": "markdown",
"id": "59e71afd-4537-49b6-a5f3-5eb97295db44",
"metadata": {
"tags": []
},
"source": [
"<img src=\"assets/dask-example-dashboard.gif\" alt=\"Dask computation with task stream annd progress bar\" width=\"70%\">"
]
},
{
Expand Down Expand Up @@ -90,10 +97,10 @@
"\n",
"> **Dask Gateway** allows you to launch and use Dask clusters in a shared, centrally managed cluster environment, without requiring users to have direct access to the underlying cluster backend (e.g.. Kubernetes, Hadoop/YARN, HPC Job queues, etc.)\n",
">\n",
"> <img src=\"./assets/dask-gateway-overview.png\" alt-text=\"Diagram of Dask-Gateway architecture\" width=\"50%\"></img>\n",
"> \n",
"> *~ Source: [gateway.dask.org](https://gateway.dask.org/)*\n",
"\n",
"<img src=\"./assets/dask-gateway-overview.png\" alt-text=\"Diagram of Dask-Gateway architecture\" width=\"50%\"></img>\n",
"\n",
"For the purposes of this Nebari tutorial, when we refer to Dask running on a distributed cluster, we mean connecting to Dask Gateway.\n",
"\n",
"This means that users with access to Dask Gateway (more on user permissions in a later notebook) need to connect to the gateway to submit their workloads to the Dask cluster. For a concrete example of how this done, check out the links at the bottom of this page."
Expand All @@ -107,7 +114,6 @@
"## Why do we include Dask and Dask Gateway with Nebari?\n",
"\n",
"* In PyData, Dask has become foundational for out-of-memory computation and lots of Nebari uses also use Dask.\n",
"* Dask Gateway is an open source and general-purpose deployment solution for Dask, so it fits in well with Nebari's OSS-first values.\n",
"* Dask's features like adaptive scaling and diagnostic dashboards can help you manage your big data computation and costs.\n",
"* Making sure Dask deployments work on your cloud platform is non-trivial, so we ship it built-in to make your workflows more efficient."
]
Expand All @@ -119,7 +125,7 @@
"source": [
"## 👀 Watch this:\n",
"\n",
"Here we run through a basic example of how to use Dask Gateway in Nebari: [finance_examples/02_dask_gateway_adaptive_scaling.ipynb](./finance_examples/02_dask_gateway_adaptive_scaling.ipynb)"
"Here we run through a basic example of how to use Dask Gateway in Nebari: [finance_examples/02a_dask_gateway_adaptive_scaling.ipynb](./finance_examples/02a_dask_gateway_adaptive_scaling.ipynb)"
]
},
{
Expand Down
Loading

0 comments on commit b8c3ace

Please sign in to comment.