Skip to content

implementation of new lightbeam create functionality#56

Merged
tomreitz merged 5 commits intomainfrom
feature/lightbeam_create
Jul 10, 2025
Merged

implementation of new lightbeam create functionality#56
tomreitz merged 5 commits intomainfrom
feature/lightbeam_create

Conversation

@tomreitz
Copy link
Collaborator

@tomreitz tomreitz commented Oct 7, 2024

This is a PR with an implementation of a new function, lightbeam create (typically used with a -s selector), which, in the current dir, derived from the Ed-Fi API's Swagger:

  • creates (if it doesn't already exist, otherwise adds/overwrites) a partial earthmover.yml with empty sources and transformations but destinations for each selected endpoint, plus comments indicating what column names and data types/values are required, and the required grain of the table (based on Swagger isIdentity fields)
  • creates (or overwrites) templates/*.jsont for each selected endpoint, with skeleton JSON that includes all the required fields (including nested ones), optional fields wrapped in conditionals, and comments with some of the property metadata from Swagger, like type, description, isIdentity, etc.
  • outputs a message that points to docs somewhere that explain how to install earthmover and finish out the earthmover.yml's sources and transformations

(The idea for this feature arose from discussions with a developer who figured out how to use earthmover but was struggling to learn how to construct the Ed-Fi payloads. lightbeam create will help scaffold developers by automatically creating some of the boilerplate code required for an earthmover Ed-Fi project.)

This implements api.get_params_for_endpoint(..., type='all') to build the sample payload for each endpoint, such as

{
  "property_bool": true,
  "property_int": 1,
  "property_float": 1.0,
  "property_string": "string",
  "property_date": "date",
  "property_string_optional": "string",
  "property_descriptor": "uri://ed-fi.org/SomeDescriptor#SomeValue",
  "property_object": {
    "property_object_1": "string",
    "property_object_2": "string"
  },
  "property_array": [
    {
      "property_array_1": "string",
      "property_array_2": "string"
    }
  ]
}

which can then be turned into a JSON template such as

{
  "property_bool": {{property_bool}},
  "property_int": {{property_int}},
  "property_float": {{property_float}},
  "property_string": "{{property_string}}",
  "property_date": "{{property_date}}",
  {% if property_string_optional %}
  "property_string_optional": "{{property_string_optional}}",
  {% endif %}
  "property_descriptor": "uri://ed-fi.org/SomeDescriptor#{{property_descriptor}}",
  "property_object": {
    "property_object_1": "{{property_object_1}}",
    "property_object_2": "{{property_object_2}}"
  },
  "property_array": [
    {% for item in property_array %}
    {
      "property_array_1": "{{item.property_array_1}}",
      "property_array_2": "{{item.property_array_2}}"
    } {% if not loop.last %},{% endif %}
    {% endfor %}
  ]
}

@tomreitz tomreitz marked this pull request as ready for review November 22, 2024 22:17
@tomreitz tomreitz changed the title partial implementation of new lightbeam create functionality implementation of new lightbeam create functionality Nov 25, 2024
@tomreitz tomreitz merged commit 48790bb into main Jul 10, 2025
@tomreitz tomreitz deleted the feature/lightbeam_create branch July 10, 2025 19:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants