New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Investigate pipeline frameworks #203

Open

buggtb opened this issue Sep 29, 2020 · 1 comment

Milestone

Collaborator

buggtb commented Sep 29, 2020

Would something like Apache Beam, be a more modern way of doing the same Spark stuff but in an agnostic fashion?

This would allow us to be less dependant on spark versions, which is a jar/packaging PITA and give users the option to run on a range of different engines: https://beam.apache.org/documentation/runners/capability-matrix/

chrismattmann modified the milestones: Sparkler 1.0 Release, Sparkler Next

Member

lewismc commented Mar 6, 2021 •

edited

I recently saw liminal which builds off of airflow.
The Airflow programming model is very intuitive. I've never used Beam so I cannot comment on it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment