Feature/iread 3

assessments/IN_IREAD/README.md

@@ -0,0 +1,77 @@
+This is an earthmover bundle created from the following Ed-Fi Data Import Tool mapping:
+* **Title**: Indiana IREAD-3 - API 3.X
+* **Description**: This template includes IREAD-3 subjects ELA, Math, Social Studies, Science, Biology, US Government.
+* **API version**: 7.1
+* **Submitter name**: Na Nguyen
+* **Submitter organization**: Education Analytics
+
+To run this bundle, please add your own source file(s) and column(s):
+<details>
+<summary><code>data/IREAD-3.csv</code></summary>
+This bundle currently works with IREAD-3 scores as exported from CRS, as of SY 2024.  It has not yet been updated for the new Checkpoints format that is being piloted for SY 2025.  
+
+</details>
+
+## CLI Parameters
+- OUTPUT_DIR: Where output files will be written
+- STATE_FILE: Where to store the earthmover runs.csv file
+- INPUT_FILE: The student assessment file to be mapped 
+- API_YEAR: The API year of the ODS for which we would send these records
+
+### Examples
+Using an ID column from the assessment file:
+```bash
+earthmover run -c ./earthmover.yaml -f -p '{
+"OUTPUT_DIR": "./output",
+"STATE_FILE": "./runs.csv",
+"INPUT_FILE": "./data/sample_anonymized_file.csv"
+}'
+```
+
+Once you have inspected the output JSONL for issues, check the settings in `lightbeam.yaml` and transmit them to your Ed-Fi API with
+```bash
+lightbeam validate+send -c ./lightbeam.yaml -p '{
+"DATA_DIR": "./output/",
+"API_YEAR": "youAPIYear",
+"BASE_URL": "yourURL",
+"EDFI_API_CLIENT_ID": "yourID",
+"EDFI_API_CLIENT_SECRET": "yourSecret" }'
+```
+
+### Notes
+- This assessment is created and administered by the state of Indiana, and is only ever expected to be used within Indiana.
+
+### Reference
+Here is a complete list of columns provided by the export
+
+- Student_First_Name
+- Student_Last_Name
+- STN
+- Student_DOB
+- Gender
+- Ethnicity
+- Special_Education_Status
+- Identified_English_Learner_Status
+- Section_504_Status
+- Free/Reduced_Price_Meals
+- Enrolled_Grade
+- Enrolled_School
+- Enrolled_School_ID
+- Enrolled_Corporation
+- Enrolled_Corporation_ID
+- Test_name
+- Overall_scale_score
+- Overall_proficiency_level
+- IREAD-3_Reported_Lexile_Measure
+- IREAD-3_Reported_Quantile_Measure
+- IREAD-3_Passing_Status
+- IREAD-3_Grade_2_Passing_Status
+- IREAD-3_Overall_scale_score
+- IREAD-3_Grade_2_Overall_scale_score
+- IREAD-3_Reported_Lexile_Measure
+- IREAD-3_Grade_2_Reported_Lexile_Measure
+- IREAD-3_Reporting_Category_1_Performance
+- IREAD-3_Reporting_Category_2_Performance
+- IREAD-3_Reporting_Category_3_Performance
+- Test Start Date
+- Test Completion Date

assessments/IN_IREAD/data/sample_anonymized_file.csv

@@ -0,0 +1,6 @@
+Student First Name,Student Last Name,STN,Student DOB,Gender,Ethnicity,Special Education Status,Identified English Learner Status,Section 504 Status,Free/Reduced Price Meals,Enrolled Grade,Enrolled School,Enrolled School ID,Enrolled Corporation,Enrolled Corporation ID,Test name,Overall scale score,Overall proficiency level,ILEARN Reported Lexile® Measure,ILEARN Reported Quantile® Measure,College and Career Readiness Indicator,ILEARN Checkpoints State Percentile Ranking,ILEARN Checkpoints Corporation Percentile Ranking,IREAD-3 Passing Status,IREAD-3 Grade 2 Passing Status,IREAD-3 Overall scale score,IREAD-3 Grade 2 Overall scale score,IREAD-3 Reported Lexile® Measure,IREAD-3 Grade 2 Reported Lexile® Measure,ILEARN U.S. Government Passing Status,ILEARN Checkpoints Reporting Category 1 Performance,ILEARN Reporting Category 1 Performance,I AM Reporting Category 1 Performance,IREAD-3 Reporting Category 1 Performance,Reporting Category 1 Percent Correct,ILEARN Checkpoints Reporting Category 2 Performance,ILEARN Reporting Category 2 Performance,I AM Reporting Category 2 Performance,IREAD-3 Reporting Category 2 Performance,Reporting Category 2 Percent Correct,ILEARN Checkpoints Reporting Category 3 Performance,ILEARN Reporting Category 3 Performance,I AM Reporting Category 3 Performance,IREAD-3 Reporting Category 3 Performance,Reporting Category 3 Percent Correct,ILEARN Reporting Category 4 Performance,I AM Reporting Category 4 Performance,ILEARN Reporting Category 5 Performance,Argumentative Organization,Argumentative Evidence/Elaboration,Argumentative Conventions,Informative Organization,Informative Evidence/Elaboration,Informative Conventions,Narrative Organization,Narrative Evidence/Elaboration,Narrative Conventions,Persuasive Organization,Persuasive Evidence/Elaboration,Persuasive Conventions,Explanatory Organization/Purpose,Explanatory Evidence/Development & Elaboration,Explanatory Conventions,Test Start Date,Test Completion Date
+StuFirst1,StuLast1,1000000,4062017,F,6,Y,N,N,Y,2,School_1 Name,1000_1000,Corp_1 Name,1000,IREAD-3 Grade 2,,,,,,,,,Pass,,527,,NoScore,,,,,At/Near,,,,,Above,,,,,Above,,,,,,,,,,,,,,,,,,,,3/4/2025 10:13,3/6/2025 10:16
+StuFirst2,StuLast2,1000001,10102016,F,4,Y,Y,Y,Y,2,School_2 Name,1001_1001,Corp_2 Name,1001,IREAD-3 Grade 2,,,,,,,,,On Track,,442,,435L,,,,,At/Near,,,,,Above,,,,,At/Near,,,,,,,,,,,,,,,,,,,,3/4/2025 10:13,3/6/2025 10:16
+StuFirst3,StuLast3,1000002,9192016,F,3,Y,N,N,N,2,School_3 Name,1002_1002,Corp_3 Name,1002,IREAD-3 Grade 2,,,,,,,,,Pass,,456,,NoScore,,,,,Above,,,,,At/Near,,,,,Above,,,,,,,,,,,,,,,,,,,,3/4/2025 10:13,3/6/2025 10:16
+StuFirst4,StuLast4,1000003,6202016,F,2,N,N,Y,Y,2,School_4 Name,1003_1003,Corp_4 Name,1003,IREAD-3 Grade 2,,,,,,,,,Pass,,451,,NoScore,,,,,Above,,,,,Above,,,,,At/Near,,,,,,,,,,,,,,,,,,,,3/4/2025 10:13,3/6/2025 10:16
+StuFirst5,StuLast5,1000004,7022016,F,2,N,N,Y,N,2,School_5 Name,1004_1004,Corp_5 Name,1004,IREAD-3 Grade 2,,,,,,,,,Pass,,514,,NoScore,,,,,At/Near,,,,,Above,,,,,Above,,,,,,,,,,,,,,,,,,,,3/4/2025 10:13,3/6/2025 10:16

assessments/IN_IREAD/earthmover.yaml

@@ -0,0 +1,147 @@
+version: 2
+
+config:
+  log_level: INFO
+  output_dir: ${OUTPUT_DIR}
+  memory_limit: 1GB
+  show_graph: False
+  show_stacktrace: true
+  parameter_defaults:
+    STUDENT_ID_NAME: 'edFi_studentUniqueID'
+    POSSIBLE_STUDENT_ID_COLUMNS: 'STN'
+    #INPUT_FILE: ./data/sample_anonymized_file.csv # Sample input file with plausible made-up data
+    INPUT_FILETYPE: csv
+    DESCRIPTOR_NAMESPACE: 'uri://doe.in.gov/IREAD/Assessment'
+
+sources:
+  # THIS FILE DOES NOT EXIST; IT MUST BE SUPPLIED BY THE USER!
+  # See the accompanying bundle_metadata.json for a list of required columns for this file
+  # Or see data/sample_anonymized_file.csv
+  input:
+    file: ${INPUT_FILE}
+    type: ${INPUT_FILETYPE}
+    header_rows: 1
+
+  assessments:
+    file: ./seeds/assessments.csv
+    header_rows: 1
+  objectiveAssessments:
+    file: ./seeds/objectiveAssessments.csv
+    header_rows: 1
+  assessmentReportingMethodDescriptors:
+    file: ./seeds/assessmentReportingMethodDescriptors.csv
+    header_rows: 1
+  gradeLevelDescriptors:
+    file: ./seeds/gradeLevelDescriptors.csv
+    header_rows: 1
+  performanceLevelDescriptors:
+    file: ./seeds/performanceLevelDescriptors.csv
+    header_rows: 1
+
+transformations:
+  input:
+    source: $sources.input
+    operations: []
+
+  studentAssessments:
+    source: $transformations.input
+    operations:
+      # An additional python pre-processing step occurs on IREAD's col names due to special characters (e.g. "®").
+      - operation: snake_case_columns
+
+        # school_year: Extracts year from the test+start_date column 
+        # assessmentIdentifier: We need to extract the assessment name from the test_name column
+        # generated_test_id: Hash to define a unique identifier for the student assessment record.
+        # namespace: Added namespace to re-use in template file.
+      - operation: add_columns
+        columns:
+          school_year: "{% raw%} {{ test_start_date.split('/')[2].split(' ')[0]}} {%endraw%}"
+          assessmentIdentifier: IREAD
+          generated_test_id: "{%raw%}{{ stn }}_{{ enrolled_school_id }}_{{ test_name }}{%endraw%}"
+          namespace: ${DESCRIPTOR_NAMESPACE}
+
+      - operation: modify_columns
+        columns:
+          test_start_date: "{% raw %} {{ test_start_date.split(' ')[0]}} {%endraw%}"
+
+      # Rename columns for brevity.
+      - operation: rename_columns
+        columns:
+          iread_3_reporting_category_1_performance: rc1_performance
+          iread_3_reporting_category_2_performance: rc2_performance
+          iread_3_reporting_category_3_performance: rc3_performance
+          iread_3_overall_scale_score: scale_score
+          iread_3_grade_2_overall_scale_score: scale_score_grade_2
+          iread_3_passing_status: passing_status
+          iread_3_grade_2_passing_status: passing_status_grade_2
+          iread_3_reported_lexile_measure: lexile_measure
+          iread_3_grade_2_reported_lexile_measure: lexile_measure_grade_2
+
+      - operation: modify_columns
+        columns:
+          scale_score: "{%raw%}{%if scale_score|int != 0 %}{{scale_score}}{%else%}{%endif%}{%endraw%}"
+          scale_score_grade_2: "{%raw%}{%if scale_score_grade_2|int != 0 %}{{scale_score_grade_2}}{%else%}{%endif%}{%endraw%}"
+
+  # Generate grade level json for assessment template based on seeds
+  gradeLevelDescriptors:
+    source: $sources.gradeLevelDescriptors
+    operations:
+      - operation: add_columns
+        columns:
+          grade_json: >
+            {%raw-%}  
+            {
+              "gradeLevelDescriptor": "{{gradeLevelDescriptor}}"
+            },
+            {%-endraw%}
+      - operation: modify_columns
+        columns:
+          grade_json: "{%raw%}{{grade_json | replace('\n', '')-}}{%endraw%}"
+      # group and aggregate:
+      - operation: group_by
+        group_by_columns:
+          - assessmentIdentifier
+        create_columns: 
+          grade_json: agg(grade_json,)
+      # there will be a trailing comma, want to get rid of that
+      - operation: modify_columns
+        columns:
+          grade_json: "{%raw%}{{value.rstrip().rstrip(',')}}{%endraw%}"
+
+
+  assessments:
+    source: $sources.assessments
+    operations:
+      - operation: join
+        sources:
+          - $transformations.gradeLevelDescriptors
+        join_type: inner
+        left_key: assessmentIdentifier
+        right_key: assessmentIdentifier
+
+destinations:
+  studentAssessments:
+    source: $transformations.studentAssessments
+    template: ./templates/studentAssessments.jsont
+    extension: jsonl
+    linearize: True
+  assessments:
+    source: $transformations.assessments
+    template: ./templates/assessments.jsont
+    extension: jsonl
+    linearize: True
+  objectiveAssessments:
+    source: $sources.objectiveAssessments
+    template: ./templates/objectiveAssessments.jsont
+    extension: jsonl
+    linearize: True
+  assessmentReportingMethodDescriptors:
+    source: $sources.assessmentReportingMethodDescriptors
+    template: ./templates/assessmentReportingMethodDescriptors.jsont
+    extension: jsonl
+    linearize: True
+  performanceLevelDescriptors:
+    source: $sources.performanceLevelDescriptors
+    template: ./templates/performanceLevelDescriptors.jsont
+    extension: jsonl
+    linearize: True

assessments/IN_IREAD/lightbeam.yaml

@@ -0,0 +1,18 @@
+state_dir: ${STATE_DIR}
+data_dir: ${DATA_DIR}
+edfi_api:
+  base_url: ${EDFI_API_BASE_URL}
+  version: 3
+  mode: district_specific
+  year: ${API_YEAR}
+  client_id: ${EDFI_API_CLIENT_ID}
+  client_secret: ${EDFI_API_CLIENT_SECRET}
+connection:
+  pool_size: 2
+  timeout: 6000
+  num_retries: 10
+  backoff_factor: 1.5
+  retry_statuses: [429, 500, 502, 503, 504]
+  verify_ssl: True
+log_level: DEBUG
+show_stacktrace: True

assessments/IN_IREAD/seeds/assessmentReportingMethodDescriptors.csv

@@ -0,0 +1,6 @@
+namespace,codeValue,description,shortDescription
+uri://doe.in.gov/IREAD/Assessment/AssessmentReportingMethodDescriptor,Passing Status,Passing Status,Passing Status
+uri://doe.in.gov/IREAD/Assessment/AssessmentReportingMethodDescriptor,Overall scale score,Overall scale score,Overall scale score
+uri://doe.in.gov/IREAD/Assessment/AssessmentReportingMethodDescriptor,Lexile Measure,Lexile Measure,Lexile Measure
+uri://doe.in.gov/IREAD/Assessment/AssessmentReportingMethodDescriptor,Reporting Category Performance Level, Reporting Category Performance Level, Reporting Category Performance Level
+

assessments/IN_IREAD/seeds/assessments.csv

@@ -0,0 +1,2 @@
+assessmentIdentifier,assessmentTitle,assessmentFamily,namespace,assessmentCategoryDescriptor,academicSubjectDescriptor,academicSubject
+IREAD,IREAD,IREAD,uri://doe.in.gov/IREAD/Assessment,uri://ed-fi.org/AssessmentCategoryDescriptor#State assessment,uri://ed-fi.org/AcademicSubjectDescriptor#English Language Arts,English/Language Arts

assessments/IN_IREAD/seeds/gradeLevelDescriptors.csv

@@ -0,0 +1,6 @@
+gradeLevel,assessmentIdentifier,gradeLevelDescriptor
+2,IREAD,uri://doe.in.gov/GradeLevelDescriptor#02
+3,IREAD,uri://doe.in.gov/GradeLevelDescriptor#03
+4,IREAD,uri://doe.in.gov/GradeLevelDescriptor#04
+5,IREAD,uri://doe.in.gov/GradeLevelDescriptor#05
+6,IREAD,uri://doe.in.gov/GradeLevelDescriptor#06

assessments/IN_IREAD/seeds/objectiveAssessments.csv

@@ -0,0 +1,4 @@
+identificationCode,assessmentIdentifier,namespace,description,parentIdentificationCode
+rc1_performance,IREAD,uri://doe.in.gov/IREAD/Assessment,IREAD Reporting Category 1 Performance,
+rc2_performance,IREAD,uri://doe.in.gov/IREAD/Assessment,IREAD Reporting Category 2 Performance,
+rc3_performance,IREAD,uri://doe.in.gov/IREAD/Assessment,IREAD Reporting Category 3 Performance,

assessments/IN_IREAD/seeds/performanceLevelDescriptors.csv

@@ -0,0 +1,11 @@
+namespace,codeValue,description,shortDescription
+uri://doe.in.gov/IREAD/Assessment/PerformanceLevelDescriptor,Pass,Pass,Pass
+uri://doe.in.gov/IREAD/Assessment/PerformanceLevelDescriptor,Did Not Pass,Did Not Pass,Did Not Pass
+uri://doe.in.gov/IREAD/Assessment/PerformanceLevelDescriptor,At Risk,At Risk,At Risk
+uri://doe.in.gov/IREAD/Assessment/PerformanceLevelDescriptor,On Track,On Track,On Track
+uri://doe.in.gov/IREAD/Assessment/PerformanceLevelDescriptor,Below,Below,Below
+uri://doe.in.gov/IREAD/Assessment/PerformanceLevelDescriptor,At/Near,At/Near,At/Near
+uri://doe.in.gov/IREAD/Assessment/PerformanceLevelDescriptor,Above,Above,Above
+uri://doe.in.gov/IREAD/Assessment/PerformanceLevelDescriptor,UND,Undetermined,Undetermined
+uri://doe.in.gov/IREAD/Assessment/PerformanceLevelDescriptor,Invalidated,Invalidated,Invalidated
+uri://doe.in.gov/IREAD/Assessment/PerformanceLevelDescriptor,Insufficient to Score,Insufficient to Score,Insufficient to Score

assessments/IN_IREAD/templates/assessmentReportingMethodDescriptors.jsont

@@ -0,0 +1,6 @@
+{
+    "codeValue": "{{codeValue}}",
+    "description": "{{description}}",
+    "namespace": "{{namespace}}",
+    "shortDescription": "{{shortDescription}}"
+}

assessments/IN_IREAD/templates/assessments.jsont

@@ -0,0 +1,44 @@
+{
+	"assessmentIdentifier": "{{assessmentIdentifier}}",
+	"assessmentTitle": "{{assessmentTitle}}",
+	"namespace": "{{namespace}}",
+	"assessmentCategoryDescriptor": "{{assessmentCategoryDescriptor}}",
+    "assessmentFamily": "{{assessmentFamily}}",
+	"assessedGradeLevels":[{{grade_json}}],
+	"academicSubjects": [
+		{
+		"academicSubjectDescriptor": "{{academicSubjectDescriptor}}"
+		}
+	],
+	"scores": [
+		{
+			"assessmentReportingMethodDescriptor": "{{namespace}}/AssessmentReportingMethodDescriptor#Overall scale score",
+			"resultDatatypeTypeDescriptor": "uri://ed-fi.org/ResultDatatypeTypeDescriptor#Integer"
+		},
+		{
+			"assessmentReportingMethodDescriptor": "{{namespace}}/AssessmentReportingMethodDescriptor#Lexile Measure",
+			"resultDatatypeTypeDescriptor": "uri://ed-fi.org/ResultDatatypeTypeDescriptor#Level"
+		}
+	],
+		{%- set passing_status_perf_desc = ["Did Not Pass", "Pass", "At Risk", "On Track", "Invalidated", "Insufficient to Score", "UND"] -%}
+		{%- set reporting_cat_perf_desc = ["Below", "At/Near", "Above",  "Invalidated", "Insufficient to Score", "UND"] -%}
+
+	"performanceLevels": [
+		{%- for desc in passing_status_perf_desc -%}
+		{
+			"assessmentReportingMethodDescriptor": "{{namespace}}/AssessmentReportingMethodDescriptor#Passing Status",
+			"performanceLevelDescriptor": "{{namespace}}/PerformanceLevelDescriptor#{{desc}}",
+			"resultDatatypeTypeDescriptor": "uri://ed-fi.org/ResultDatatypeTypeDescriptor#Level"
+		},
+		{%- endfor -%}
+
+		{%- for desc in reporting_cat_perf_desc -%}
+		{
+			"assessmentReportingMethodDescriptor": "{{namespace}}/AssessmentReportingMethodDescriptor#Reporting Category Performance Level",
+			"performanceLevelDescriptor": "{{namespace}}/PerformanceLevelDescriptor#{{desc}}",
+			"resultDatatypeTypeDescriptor": "uri://ed-fi.org/ResultDatatypeTypeDescriptor#Level"
+		} {%- if not loop.last-%},{%-else-%}{%-endif-%}
+		{%- endfor -%}
+
+	]
+}

assessments/IN_IREAD/templates/objectiveAssessments.jsont

@@ -0,0 +1,29 @@
+{
+    "identificationCode": "{{identificationCode}}",
+    "assessmentReference": {
+      "assessmentIdentifier": "{{assessmentIdentifier}}",
+      "namespace": "{{namespace}}"
+    },
+    "description": "{{description}}",
+    {% if parentIdentificationCode is not none and parentIdentificationCode | length %}
+    "parentObjectiveAssessmentReference": {
+        "assessmentIdentifier": "{{assessmentIdentifier}}",
+        "identificationCode": "{{parentIdentificationCode}}",
+        "namespace": "{{namespace}}"
+    },
+    {% endif %}
+    "performanceLevels": [
+		{
+			"assessmentReportingMethodDescriptor": "{{namespace}}/AssessmentReportingMethodDescriptor#Reporting Category Performance Level",
+			"performanceLevelDescriptor": "{{namespace}}/PerformanceLevelDescriptor#Above"
+		},
+		{
+			"assessmentReportingMethodDescriptor": "{{namespace}}/AssessmentReportingMethodDescriptor#Reporting Category Performance Level",
+			"performanceLevelDescriptor": "{{namespace}}/PerformanceLevelDescriptor#Below"
+		},
+		{
+			"assessmentReportingMethodDescriptor": "{{namespace}}/AssessmentReportingMethodDescriptor#Reporting Category Performance Level",
+			"performanceLevelDescriptor": "{{namespace}}/PerformanceLevelDescriptor#At/Near"
+		}
+    ]
+  }

assessments/IN_IREAD/templates/performanceLevelDescriptors.jsont

@@ -0,0 +1,6 @@
+{
+    "codeValue": "{{codeValue}}",
+    "description": "{{description}}",
+    "namespace": "{{namespace}}",
+    "shortDescription": "{{shortDescription}}"
+  }