Analytics / data science projects with Dagster

[yuhan]

This discussion is for folks to share how you use Dagster for analytics or data science projects.

Question was originally asked in Dagster Slack. Reposting it here for posterity and discoverability.

"""
Question on project best practices (with regards to analytics/data science pipelines i.e. reading in a csv, dropping columns, renaming columns, filtering, modelling, etc.): what is the recommended project structure? I searched a bit on the dagster documentation and found 'Create a new project' but that seems more geared toward generating a general project skeleton. We've got a bunch of ipynb files that perform each step in the pipeline - does it make more sense to: wire them together using dagstermill? Convert them into python files and then wire them together with dagster? It's helpful to see cell outputs when the process runs, but it seems like you lose a bit of flexibility. Additionally when we are quickly validating data, conducting EDA, etc. it's nice that I can see plots/visualizations right in an ipynb. Before I embarked on a full blown conversion, just thought I'd check with the community of experts! Thank you in advanced!
"""

[yuhan]

What I personally found helpful is

1. Begin with plain notebooks (so load data from an api, process then, and plot), so I can see the results immediately and maximize the flexibility.
2. Then, when I’m satisfied with the first step in the notebook (say load_data), I’d move it out to a dagster op, and chain the load_data op with a notebook using dagstermill, so I can leverage Dagit and the re-execution functionalities which helps me persist the intermediate artifacts and not have to load data over and over again (esp when im calling some external APIs that I don’t want to hit the limit while I’m debugging/developing the notebook). — The alternative is to develop everything inside the notebook but I found it cumbersome to have to comment and uncomment code.
3. incrementally iterate (do step 1 and 2 again and again) until I have a “prod-read” job with a notebook which has the plot.