Add Fast Bridge Math assessment bundle

assessments/FastBridge_Math/README.md

@@ -0,0 +1,87 @@
+## FastBridge Math (Early Math)
+
+* **Title:** FastBridge Math Early Math Assessment Results
+* **Description:** This template maps FastBridge Math Early Math Assessment Results, providing comprehensive evaluation of various mathematical skills including composing, counting objects, decomposing, equal partitioning, match quantity, number sequence, numeral identification, place value, quantity discrimination, subitizing, verbal addition, verbal subtraction, and story problems. The bundle includes pre-processing to pivot season-based columns into student assessment rows, handle growth metrics, and map positional error columns.
+* **API version:** 5.3
+* **Submitter name:** Bruk Woldearegay
+* **Submitter organization:** Crocus LLC.
+
+To run this bundle, please add your own source file(s) and column(s):
+<details>
+This template works with vendor layout file structure. The pre-execute script transforms the wide CSV format (seasons as columns) into a long format (seasons as rows) suitable for Ed-Fi ingestion. See the sample anonymized file.
+</details>
+
+Sample file: `data/Sample_earlyMath_deidentified.csv`
+
+### CLI Parameters
+
+### Required
+- **OUTPUT_DIR**: Where output files will be written
+- **STATE_FILE**: Where to store the earthmover runs.csv file
+- **INPUT_FILE**: The student assessment file to be mapped
+- **STUDENT_ID_NAME**: Which column to use as the Ed-Fi `studentUniqueId`. Default column is the 'StateID' from the vendor layout file.
+- **SCHOOL_YEAR**: The year of the assessment file (format as 'YYYY' e.g. '2024', etc).
+
+### Examples
+
+**Step 1: Running the pre-execute script to transform the file structure**
+The FastBridge Math CSV comes in a wide format with seasons as columns. The pre-execute script pivots this into a long format and handles growth metrics and error column mapping:
+
+```python
+fast_bridge_math_pre_exec(source_file, output_file)
+```
+
+Example:
+```python
+fast_bridge_math_pre_exec(
+    'data/Sample_earlyMath_deidentified.csv',
+    'data/Sample_earlyMath_deidentified_pivoted.csv'
+)
+```
+
+**Step 2: Running earthmover with the transformed file:**
+```bash
+earthmover run -c ./earthmover.yaml -p '{
+"INPUT_FILE": "data/Sample_earlyMath_deidentified_pivoted.csv",
+"OUTPUT_DIR": "output/",
+"STUDENT_ID_NAME": "State ID",
+"SCHOOL_YEAR": "2024"}'
+```
+
+### Pre-Execute Script Features
+
+The pre-execute script (`python_pre_exec/pre-execute.py`) performs the following transformations:
+
+1. **Season Pivoting**: Converts wide format (seasons as columns) to long format (seasons as rows)
+2. **Growth Metrics Processing**: Handles growth columns (e.g., "Composing from Fall to Winter") and pivots them based on ending season
+3. **Error Column Mapping**: Maps generic "Error" columns to objective-specific error fields using positional anchors (IC per minute, NRC per minute)
+4. **Column Normalization**: Converts all column names to snake_case format for consistency
+5. **Data Validation**: Filters out empty rows and ensures data quality
+
+### Error Column Handling
+
+The script automatically detects and maps generic "Error" columns that appear positionally after anchor metrics:
+- Generic columns named "Error", "Error.1", "Error.2", etc.
+- Maps to objective-specific names like "numeral_identification_one_error", "numeral_identification_kg_error"
+- Uses "IC per minute" and "NRC per minute" columns as anchors to determine which objective each error belongs to
+
+Once you have inspected the output JSONL for issues, check the settings in `lightbeam.yaml` and transmit them to your Ed-Fi API with
+```bash
+lightbeam validate+send -c ./lightbeam.yaml -p '{
+"DATA_DIR": "./output/",
+"STATE_DIR": "./tmp/.lightbeam/",
+"EDFI_API_BASE_URL": "<yourURL>",
+"EDFI_API_CLIENT_ID": "<yourID>",
+"EDFI_API_CLIENT_SECRET": "<yourSecret>",
+"SCHOOL_YEAR": "<yourAPIYear>"}'
+```
+
+
+### Output Structure
+
+After transformation, the pivoted file will contain:
+- One row per student per season with assessment data
+- Snake_case column names for consistency
+- Growth metrics attached to records based on ending season
+- Mapped error columns with objective-specific names (e.g., `numeral_identification_one_error`, `numeral_identification_kg_error`)
+

assessments/FastBridge_Math/earthmover.yaml

@@ -0,0 +1,305 @@
+version: 2
+
+config:
+  log_level: INFO
+  output_dir: ${OUTPUT_DIR}
+  memory_limit: 1GB
+  state_file: ${STATE_FILE}
+  show_graph: false
+  show_stacktrace: true
+  parameter_defaults:
+    # You can set these values at the command line or leave the defaults
+    OUTPUT_DIR: ./output/
+    STUDENT_ID_NAME: 'edFi_studentUniqueID' # default to the column added by the apply_xwalk package of student ID xwalking feature
+    POSSIBLE_STUDENT_ID_COLUMNS: Local ID,State ID
+    DESCRIPTOR_NAMESPACE: uri://ed-fi.org
+    SCHOOL_YEAR: 2024
+
+sources:
+  input:
+    file: ${INPUT_FILE} # Path to the input CSV file
+    header_rows: 1
+  assessments:
+    file: ./seeds/assessments.csv
+    header_rows: 1
+  assessmentReportingMethodDescriptors:
+    file: ./seeds/assessmentReportingMethodDescriptors.csv
+    header_rows: 1
+  assessmentCategoryDescriptors:
+    file: ./seeds/assessmentCategoryDescriptors.csv
+    header_rows: 1
+  assessmentPeriodDescriptors:
+    file: ./seeds/assessmentPeriodDescriptors.csv
+    header_rows: 1
+  objectiveAssessments:
+    file: ./seeds/objectiveAssessments.csv
+    header_rows: 1
+  performanceLevelDescriptors:
+    file: ./seeds/performanceLevelDescriptors.csv
+    header_rows: 1
+  gradeLevelDescriptors:
+    file: ./seeds/gradeLevelDescriptors.csv
+    header_rows: 1
+  objectiveAssessmentScores:
+    file: ./seeds/objectiveAssessmentScores.csv
+    header_rows: 1
+
+
+transformations:
+  input:
+    source: $sources.input
+    operations: []
+
+  # Clean and prepare input data
+  cleaned_input:
+    source: $transformations.input
+    operations:
+      # Duplicate key columns for easier template access
+      - operation: duplicate_columns
+        columns:
+          Final_Date: administrationDate
+          ${STUDENT_ID_NAME}: studentUniqueId
+      # Convert column names to snake_case for easier template access
+      - operation: snake_case_columns
+      # Add required columns
+      - operation: add_columns
+        columns:
+          assessmentIdentifier: "FastBridge_earlyMath"
+          namespace: "uri://fastbridge.org/assessment"
+          descriptor_namespace: ${DESCRIPTOR_NAMESPACE}
+      # Create unique student assessment ID
+      - operation: combine_columns
+        columns:
+          - final_date
+          - student_unique_id
+          - season
+        new_column: studentAssessmentIdentifier
+        separator: '-'
+      - operation: modify_columns
+        columns:
+          studentAssessmentIdentifier: "{%raw%}{{ md5(studentAssessmentIdentifier) }}{%endraw%}"
+      # Format the administration date
+      - operation: date_format
+        column: administration_date
+        from_format: "%m/%d/%Y"
+        to_format: "%Y-%m-%d"
+      - operation: add_columns
+        columns:
+          school_year: "{%raw%}{{administration_date[:4]|int +1 if administration_date[5:7]|int >=7 else administration_date[:4]|int }}{%endraw%}"
+      #join gradeleveldescriptor
+      - operation: join
+        sources:
+          - $sources.gradeLevelDescriptors
+        join_type: left
+        left_key: grade
+        right_key: original_grade
+        right_keep_columns:
+        - gradeLevelDescriptor
+      # filter out rows with missing student IDs
+      - operation: filter_rows
+        query: "student_unique_id	 != ''"
+        behavior: include
+
+
+  # Generate grade level JSON for assessments
+  grade_level_descriptors:
+    source: $sources.gradeLevelDescriptors
+    operations:
+      - operation: add_columns
+        columns:
+          assessmentIdentifier: "FastBridge_earlyMath"
+          grade_json: >
+            {%raw-%}
+            {
+              "gradeLevelDescriptor": "{{gradeLevelDescriptor}}"
+            },
+            {%-endraw%}
+      - operation: modify_columns
+        columns:
+          grade_json: "{%raw%}{{grade_json | replace('\n', '')-}}{%endraw%}"
+      # group and aggregate:
+      - operation: group_by
+        group_by_columns:
+          - assessmentIdentifier
+        create_columns:
+          grade_json: agg(grade_json,)
+      # there will be a trailing comma, want to get rid of that
+      - operation: modify_columns
+        columns:
+          grade_json: "{%raw%}{{value.rstrip().rstrip(',')}}{%endraw%}"
+
+  # Generate performance level JSON for assessments
+  performance_level_descriptors:
+    source: $sources.performanceLevelDescriptors
+    operations:
+      - operation: add_columns
+        columns:
+          base_namespace: "uri://fastbridge.org/assessment"
+          descriptor_namespace: ${DESCRIPTOR_NAMESPACE}
+          performance_level_json: >
+            {%raw-%}
+            {
+              "assessmentReportingMethodDescriptor": "{{base_namespace}}/AssessmentReportingMethodDescriptor#Risk Level",
+              "performanceLevelDescriptor": "{{namespace}}#{{codeValue}}",
+              "resultDatatypeTypeDescriptor": "{{descriptor_namespace}}/ResultDatatypeTypeDescriptor#Level"
+            },
+            {%-endraw%}
+      - operation: modify_columns
+        columns:
+          performance_level_json: "{%raw%}{{performance_level_json | replace('\n', '')-}}{%endraw%}"
+      # group and aggregate:
+      - operation: group_by
+        group_by_columns:
+          - assessmentIdentifier
+        create_columns:
+          performance_level_json: agg(performance_level_json,)
+      # there will be a trailing comma, want to get rid of that
+      - operation: modify_columns
+        columns:
+          performance_level_json: "{%raw%}{{value.rstrip().rstrip(',')}}{%endraw%}"
+
+  # Generate assessment period JSON for assessments
+  assessment_period_descriptors:
+    source: $sources.assessmentPeriodDescriptors
+    operations:
+      - operation: add_columns
+        columns:
+          period_json: >
+            {%raw-%}
+            {
+              "assessmentPeriodDescriptor": "{{namespace}}#{{codeValue}}"
+            },
+            {%-endraw%}
+      - operation: modify_columns
+        columns:
+          period_json: "{%raw%}{{period_json | replace('\n', '')-}}{%endraw%}"
+      # group and aggregate:
+      - operation: group_by
+        group_by_columns:
+          - assessmentIdentifier
+        create_columns:
+          period_json: agg(period_json,)
+      # there will be a trailing comma, want to get rid of that
+      - operation: modify_columns
+        columns:
+          period_json: "{%raw%}{{value.rstrip().rstrip(',')}}{%endraw%}"
+
+  # Generate scores JSON for objective assessments
+  objective_assessment_scores:
+    source: $sources.objectiveAssessmentScores
+    operations:
+      - operation: add_columns
+        columns:
+          base_namespace: "uri://fastbridge.org/assessment"
+          score_json: >
+            {%raw-%}
+            {
+              "assessmentReportingMethodDescriptor": "{{base_namespace}}/AssessmentReportingMethodDescriptor#{{scoreType}}",
+              "resultDatatypeTypeDescriptor": "{{resultDatatypeTypeDescriptor}}"
+            },
+            {%-endraw%}
+      - operation: modify_columns
+        columns:
+          score_json: "{%raw%}{{score_json | replace('\n', '')-}}{%endraw%}"
+      # group and aggregate by identificationCode:
+      - operation: group_by
+        group_by_columns:
+          - identificationCode
+        create_columns:
+          score_json: agg(score_json,)
+      # there will be a trailing comma, want to get rid of that
+      - operation: modify_columns
+        columns:
+          score_json: "{%raw%}{{value.rstrip().rstrip(',')}}{%endraw%}"
+
+  # Prepare assessments with proper structure
+  assessments:
+    source: $sources.assessments
+    operations:
+      - operation: join
+        sources:
+          - $transformations.grade_level_descriptors
+        join_type: inner
+        left_key: assessmentIdentifier
+        right_key: assessmentIdentifier
+      - operation: join
+        sources:
+          - $transformations.performance_level_descriptors
+        join_type: inner
+        left_key: assessmentIdentifier
+        right_key: assessmentIdentifier
+      - operation: join
+        sources:
+          - $transformations.assessment_period_descriptors
+        join_type: inner
+        left_key: assessmentIdentifier
+        right_key: assessmentIdentifier
+      - operation: add_columns
+        columns:
+          descriptor_namespace: ${DESCRIPTOR_NAMESPACE}
+
+  # Prepare objective assessments
+  objective_assessments:
+    source: $sources.objectiveAssessments
+    operations:
+      - operation: join
+        sources:
+          - $transformations.performance_level_descriptors
+        join_type: inner
+        left_key: assessmentIdentifier
+        right_key: assessmentIdentifier
+      - operation: join
+        sources:
+          - $transformations.objective_assessment_scores
+        join_type: inner
+        left_key: identificationCode
+        right_key: identificationCode
+      - operation: add_columns
+        columns:
+          descriptor_namespace: ${DESCRIPTOR_NAMESPACE}
+
+  #Prepare Student Assessment Educationorganization Association
+  student_assessment_edorg:
+    source: $transformations.cleaned_input
+    operations:
+      - operation: filter_rows
+        query: school != ''
+        behavior: include
+
+destinations:
+  assessments:
+    source: $transformations.assessments
+    template: ./templates/assessments.jsont
+    extension: jsonl
+  objectiveAssessments:
+    source: $transformations.objective_assessments
+    template: ./templates/objectiveAssessments.jsont
+    extension: jsonl
+  studentAssessments:
+    source: $transformations.cleaned_input
+    template: ./templates/studentAssessments.jsont
+    extension: jsonl
+  studentAssessmentEducationOrganizationAssociations:
+    source: $transformations.student_assessment_edorg
+    template: ./templates/studentAssessmentEducationOrganizationAssociations.jsont
+    extension: jsonl
+  assessmentReportingMethodDescriptors:
+    source: $sources.assessmentReportingMethodDescriptors
+    template: ./templates/descriptors.jsont
+    extension: jsonl
+    linearize: True
+  assessmentCategoryDescriptors:
+    source: $sources.assessmentCategoryDescriptors
+    template: ./templates/descriptors.jsont
+    extension: jsonl
+    linearize: True
+  assessmentPeriodDescriptors:
+    source: $sources.assessmentPeriodDescriptors
+    template: ./templates/descriptors.jsont
+    extension: jsonl
+    linearize: True
+  performanceLevelDescriptors:
+    source: $sources.performanceLevelDescriptors
+    template: ./templates/descriptors.jsont
+    extension: jsonl
+    linearize: True