Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature/inLineSource: adding another source type #131

Draft
wants to merge 15 commits into
base: main
Choose a base branch
from

Conversation

gnguyen87
Copy link

@gnguyen87 gnguyen87 commented Oct 9, 2024

Description & motivation

From this Monday item:

Consider adding another source type, possibly called inline, where the data is specified in the earthmover YAML:

sources:
  inlineSource:
    orientation: rows # or columns
    data:
      - key1: value1a
        key2: value2a
        key3: value3a
      - key1: value1b
        key2: value2b
        key3: value3b
      - key1: value1c
        key2: value2c
        key3: value3c
    # or, if orientation=columns
    data:
      key1: [ "value1a", "value1b", "value1c" ]
      key2: [ "value2a", "value2b", "value2c" ]
      key3: [ "value3a", "value3b", "value3c" ]

This PR supports this feature by adding a new inLineSource class with minor refactoring to other helper classes.

PR Merge Priority:

  • Low
  • Medium
  • High

Changes to existing files:

  • source.py : Add an inLineSource class
  • runs_file.py: Import inLineSource class to take care of file-hashing
  • column.py: Cast dict vals to str type in a lambdato appropriately join row values together
  • earthmover/tests: add an inLineSource to the test suite

Tests and QC done:

  • Tested locally via the test suite as well as earthmover_edfi_bundles

@@ -447,3 +451,51 @@ def _verify_packages(self, connection: str):
"connecting to a database requires additional libraries... please install using `pip install earthmover[sql]`"
)
raise

class inLineSource(Source):
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider InlineSource to fit convention of other nodes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants