Feature/Optional Fields from External Source

[AngelicaLastra]

Description

This PR augments the optional fields feature to support specifying optional fields from an external YAML source under an optional_fields key. This allows earthmover templates that have a large number of optional fields to preserve a cleaner and legible earthmover.yml (e.g. CLI Circle's earthmover.yml)

Changes

• Added a load_optional_fields_from_file function to support external YAMLs to define optional fields.
• Updated documentation to reflect the new configuration format.
• Added validation for the YAML structure.

New Configuration Format

version: 2

optional_fields:
  input:
    - optional_field1
    - optional_field2
  source2:
    - optional_field3
    - optional_field4

Example Usage

In earthmover.yml:

sources:
  input:
    file: ./data/input.csv
    optional_fields: ./optional_fields.yml

Testing

• Test loading optional fields from external YAML file
• Test with multiple sources in the same YAML file
• Test error handling for invalid YAML structure
• Test with missing source names in the YAML file

Documentation

• Updated configuration.md with the new YAML structure
• Added examples and usage instructions