Skip to content
@ddeutils

Data Developer & Engineering

Provide Data Developer & Engineering utility objects and functions.

Welcome to Data Developer and Engineer 👋

For knowledge document of this organization, you can follow this like Document.

flowchart LR
   1([ddeutil]) --> 2([ddeutil-io]) --> 3([🏃 ddeutil-workflow]) --> 4([📡 ddeutil-observe])
   1 --> 4
   2 --> 5([ddeutil-vendors])
   0([fmtutil]) -.-> 2
   0 -.-> 5
Loading

📌 Components

This organization has the propose to make lightweight data orchestration framework for small to middle project scale (1-1000 workflows).

Firstly, I will implement base projects, Core (utility functions) and IO (Input/Output transport utility objects) for the first dependency packages because it has a lot of base code to make main package and I do not want to develop this code on the main package, for example, it do not good if I want fix bug on the merge key function that no relate with the workflow package

🎯 The main package of this organize orchestration framework has 2 layers and I split it with 2 projects for optional installation requirement (you can only use just one of these layers without raise error).

  • 🏃 Workflow - Lightweight workflow orchestration in Python with less dependencies.
  • 📡 Observe - Lightweight observation application with FastAPI for the workflow package.

Note

I have some 3rd-party projects, Vendors, for keeping an additional practices to use any 3rd API connect data source, like polars, duckdb, etc. It be dynamic data processing & transformation objects from external vendor packages.

🍸 External

This organize has some mini-projects that develop for specific usecase:

  • ddeapp-flask - Full-Stack Data Orchestration from Yaml template with Flask & HTMX
  • ddeapp-fastapi - Routing Application Service deploy to On-Premise server with FastAPI

Warning

The above projects have a lot of bugs and need time to fix and refactored the code.

Pinned Loading

  1. ddeutil-observe ddeutil-observe Public

    Lightweight observation application with FastAPI

    Python

  2. ddeutil-workflow ddeutil-workflow Public

    Lightweight workflow orchestration in Python with less dependencies

    Python

  3. ddeutil ddeutil Public

    Core data utility functions and objects

    Python

  4. ddeutil-io ddeutil-io Public

    Input/Output transport utility objects with less dependencies

    Python

  5. ddedocs ddedocs Public

    Data Developer & Engineer Documents and Hands-On

    Python 1 1

  6. ddeutil-vendors ddeutil-vendors Public

    Dynamic data processing & transformation objects from external vendor packages

    Python

Repositories

Showing 9 of 9 repositories
  • ddeutil-observe Public

    Lightweight observation application with FastAPI

    ddeutils/ddeutil-observe’s past year of commit activity
    Python 0 MIT 0 0 4 Updated Oct 1, 2024
  • ddedocs Public

    Data Developer & Engineer Documents and Hands-On

    ddeutils/ddedocs’s past year of commit activity
    Python 1 MIT 1 0 0 Updated Oct 1, 2024
  • ddeutil Public

    Core data utility functions and objects

    ddeutils/ddeutil’s past year of commit activity
    Python 0 MIT 0 0 0 Updated Oct 1, 2024
  • ddeutil-workflow Public

    Lightweight workflow orchestration in Python with less dependencies

    ddeutils/ddeutil-workflow’s past year of commit activity
    Python 0 MIT 0 0 0 Updated Sep 30, 2024
  • ddeutil-io Public

    Input/Output transport utility objects with less dependencies

    ddeutils/ddeutil-io’s past year of commit activity
    Python 0 MIT 0 0 0 Updated Sep 26, 2024
  • .github Public

    README page on the ddeutils organization

    ddeutils/.github’s past year of commit activity
    0 MIT 0 0 0 Updated Sep 20, 2024
  • ddeutil-vendors Public

    Dynamic data processing & transformation objects from external vendor packages

    ddeutils/ddeutil-vendors’s past year of commit activity
    Python 0 MIT 0 0 0 Updated Sep 13, 2024
  • ddeapp-flask Public

    Full-Stack Data Orchestration from Yaml template with Flask & HTMX

    ddeutils/ddeapp-flask’s past year of commit activity
    Python 0 MIT 0 0 0 Updated May 11, 2024
  • ddeapp-fastapi Public

    Routing Application Service deploy to On-Premise server with FastAPI

    ddeutils/ddeapp-fastapi’s past year of commit activity
    Python 0 MIT 0 0 0 Updated May 6, 2024

People

This organization has no public members. You must be a member to see who’s a part of this organization.

Top languages

Loading…

Most used topics

Loading…