Skip to content

Commit 5068cb6

Browse files
jalvord1jayckaiseralchenist
authored
Hotfix/scready assessment grade level (#178)
* Populate `whenAssessedGradeLevelDescriptor` with assessment grade level, and move `Grade` to generic score results. * Define columns list separately and pass to both keep columns and optional fields in case of schema drift. * Attempt array unification using jinja. * Reorder to reattempt aliasing. * Refactor to avoid array merging. * Another attempt at including student_unique_id in the main transformation. * Attempt to escape special characters in score results using tojson Jinja filter. * Map SC Ready academic subject to assessment identifier, not across all assessments. * Force all Ed-Fi descriptors in SC READY to their South-Carolina specific alternatives. * Fix formatting error in academic subject descriptor. * Fix/scready 2025 (#169) * Trim whitespace from assessment grade and duplicate instead of replace STUDENT_NAME_ID * Handle changes from SchoolID to RSchoolID and TSchoolID in SC READY * Handle 2025 changes from TDA to TDW for SC READY * Handle 2025 changes to SC READY scores, perf levels * Handle 2025 changes to SC READY, removal of ELA1Acc11 * Handle 2025 changes to SC READY, IEP changes * SC READY: move SCD Type II operations into common definitions * SC READY: clean up SCD Type 2 logic * SC READY: Add transformations and template for student assessment education organization associations * SC READY: Trim score results; some in sample file were whitespace only * SC READY: fix case in educationOrganizationId * SC_READY: Set default API_YEAR * SC READY: remove string -> int pipe for API_YEAR to test if setting default is sufficient * fixup! SC READY: remove string -> int pipe for API_YEAR to test if setting default is sufficient SC READY: remove extra quotes * swap debug to info for log level in earthmover.yaml --------- Co-authored-by: Julianna Alvord <[email protected]> --------- Co-authored-by: jayckaiser <[email protected]> Co-authored-by: Alex C. <[email protected]>
1 parent b624aec commit 5068cb6

File tree

6 files changed

+461
-333
lines changed

6 files changed

+461
-333
lines changed

‎assessments/SC_READY/earthmover.yaml‎

Lines changed: 95 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -11,16 +11,37 @@ config:
1111
STATE_FILE: ./runs.csv
1212
STUDENT_ID_NAME: 'edFi_studentUniqueID' # default to the column added by the apply_xwalk package of student ID xwalking feature
1313
INPUT_FILE: ''
14-
API_YEAR: ''
14+
API_YEAR: 2024
1515
INPUT_FILETYPE: csv
1616
POSSIBLE_STUDENT_ID_COLUMNS: StateID
1717

18+
definitions:
19+
- &scd2-year-fill-nas
20+
operation: modify_columns
21+
columns:
22+
start_year: "{%raw%}{{ start_year or 2024}}{%endraw%}"
23+
end_year: "{%raw%}{{ end_year or 2100 }}{%endraw%}"
24+
- &scd2-flag-current-rows
25+
operation: add_columns
26+
columns:
27+
current: "{%raw%}{% if ${API_YEAR} >= (start_year | int) and ${API_YEAR} <= (end_year | int) %}true{% endif %}{%endraw%}"
28+
- &scd2-filter-to-current-rows
29+
operation: filter_rows
30+
query: "current == 'true'"
31+
behavior: include
32+
33+
1834
# Define columns list as anchor to pass to keep-cols and optional_fields.
1935
columns_list: &columns_list
2036
# Generic fields
2137
- student_unique_id # Initialized empty and renamed in first transformation.
2238
- StateID
39+
{% if ${API_YEAR} < 2025 %}
2340
- SchoolID
41+
{% else %}
42+
- TSchoolID
43+
- RSchoolID
44+
{% endif %}
2445
- TestAdmin
2546
- UID
2647
- Grade
@@ -55,6 +76,11 @@ columns_list: &columns_list
5576
- IEP_TBI
5677
- IEP_TM
5778
- IEP_VH
79+
- IEP_ID
80+
- IEP_HI
81+
- IEP_TB
82+
- EFA_DB
83+
- EFA_MD
5884
- InstrSetting
5985
- LName
6086
- MName
@@ -363,19 +389,29 @@ transformations:
363389
- operation: drop_columns
364390
columns:
365391
- student_unique_id
366-
- operation: rename_columns
392+
- operation: duplicate_columns
367393
columns:
368394
${STUDENT_ID_NAME}: student_unique_id
369395
- operation: keep_columns
370396
columns: *columns_list
397+
- operation: modify_columns
398+
columns:
399+
ELAGrade: "{%raw%}{{ value | trim }}{%endraw%}"
400+
MathGrade: "{%raw%}{{ value | trim }}{%endraw%}"
401+
SciGrade: "{%raw%}{{ value | trim }}{%endraw%}"
402+
SocGrade: "{%raw%}{{ value | trim }}{%endraw%}"
371403
- operation: filter_rows
372404
query: student_unique_id.notnull() & (student_unique_id != "") & (student_unique_id != "0")
373405
behavior: include
374406
- operation: add_columns
375407
columns:
376408
school_year: ${API_YEAR}
377409
administration_date: "{%raw%}{{${API_YEAR} ~ '-05-15'}}{%endraw%}"
410+
{% if ${API_YEAR} < 2025 %}
378411
lea_id: "{%raw%}{{SchoolID[0:-3]}}{%endraw%}"
412+
{% else %}
413+
lea_id: "{%raw%}{{RSchoolID[0:-3]}}{%endraw%}"
414+
{% endif %}
379415
namespace: "uri://datarecognitioncorp.com/Assessment"
380416

381417
scready_student_assessment_ela:
@@ -422,12 +458,10 @@ transformations:
422458
- $transformations.scready_student_assessment_math
423459
- $transformations.scready_student_assessment_science
424460
- $transformations.scready_student_assessment_social_studies
425-
426461
# Only populated assessments have `{Subject}Grade` column populated
427462
- operation: filter_rows
428463
query: assessment_grade != ''
429464
behavior: include
430-
431465
- operation: add_columns
432466
columns:
433467
student_assessment_id: "{%raw%}{{UID}}__{{subject}}{%endraw%}"
@@ -460,13 +494,48 @@ transformations:
460494
SocLev5_name: "{%raw%}{{assessment_grade | int}}_SocLev5{%endraw%}"
461495
SocLev6_name: "{%raw%}{{assessment_grade | int}}_SocLev6{%endraw%}"
462496
SocLev7_name: "{%raw%}{{assessment_grade | int}}_SocLev7{%endraw%}"
497+
498+
## StudentAssessmentEducationOrganizationAssociations
499+
# Stack reported school ID (RSchoolID 2025 and later, SchoolID before 2025)
500+
# and tested school ID (TSchoolID 2025 and later)
501+
502+
student_assessment_edu_org_associations__administration:
503+
source: $transformations.scready_student_assessment_stacked
504+
operations:
505+
- operation: add_columns
506+
columns:
507+
educationOrganizationId: "{%raw%}{{ TSchoolID }}{%endraw%}"
508+
educationOrganizationAssociationTypeDescriptor: "uri://ed.sc.gov/EducationOrganizationAssociationTypeDescriptor#Administration"
509+
510+
student_assessment_edu_org_associations__attribution:
511+
source: $transformations.scready_student_assessment_stacked
512+
operations:
513+
- operation: add_columns
514+
columns:
515+
# TSchoolID added in 2025
516+
educationOrganizationId: "{%raw%}{{ RSchoolID or SchoolID }}{%endraw%}"
517+
educationOrganizationAssociationTypeDescriptor: "uri://ed.sc.gov/EducationOrganizationAssociationTypeDescriptor#Attribution"
518+
519+
student_assessment_edu_org_associations_stacked:
520+
source: $transformations.student_assessment_edu_org_associations__administration
521+
operations:
522+
- operation: union
523+
sources:
524+
- $transformations.student_assessment_edu_org_associations__attribution
525+
- operation: filter_rows
526+
query: "educationOrganizationId.notnull() and (educationOrganizationId != '') and (educationOrganizationId != 'None')"
527+
behavior: include
528+
463529

464530
objective_assessments:
465531
source: $sources.objectiveAssessments
466532
operations:
467533
- operation: modify_columns
468534
columns:
469535
assessmentIdentifier: "{%raw%}{{value}}_${API_YEAR}{%endraw%}"
536+
- *scd2-year-fill-nas
537+
- *scd2-flag-current-rows
538+
- *scd2-filter-to-current-rows
470539

471540

472541
assessments:
@@ -475,7 +544,21 @@ transformations:
475544
- operation: modify_columns
476545
columns:
477546
assessmentIdentifier: "{%raw%}{{value}}_${API_YEAR}{%endraw%}"
478-
547+
548+
549+
assessmentReportingMethodDescriptors:
550+
source: $sources.assessmentReportingMethodDescriptors
551+
operations:
552+
- *scd2-year-fill-nas
553+
- *scd2-flag-current-rows
554+
- *scd2-filter-to-current-rows
555+
556+
accommodationDescriptors:
557+
source: $sources.accommodationDescriptors
558+
operations:
559+
- *scd2-year-fill-nas
560+
- *scd2-flag-current-rows
561+
- *scd2-filter-to-current-rows
479562

480563

481564
destinations:
@@ -496,12 +579,12 @@ destinations:
496579
extension: jsonl
497580
linearize: True
498581
assessmentReportingMethodDescriptors:
499-
source: $sources.assessmentReportingMethodDescriptors
582+
source: $transformations.assessmentReportingMethodDescriptors
500583
template: ./templates/descriptors.jsont
501584
extension: jsonl
502585
linearize: True
503586
accommodationDescriptors:
504-
source: $sources.accommodationDescriptors
587+
source: $transformations.accommodationDescriptors
505588
template: ./templates/descriptors.jsont
506589
extension: jsonl
507590
linearize: True
@@ -510,3 +593,8 @@ destinations:
510593
template: ./templates/studentAssessments.jsont
511594
extension: jsonl
512595
linearize: True
596+
studentAssessmentEducationOrganizationAssociations:
597+
source: $transformations.student_assessment_edu_org_associations_stacked
598+
template: ./templates/studentAssessmentEducationOrganizationAssociations.jsont
599+
extension: jsonl
600+
linearize: True

‎assessments/SC_READY/seeds/accommodationDescriptors.csv‎

Lines changed: 49 additions & 49 deletions
Original file line numberDiff line numberDiff line change
@@ -1,49 +1,49 @@
1-
codeValue,description,namespace,shortDescription
2-
Acc1,Accomodations 1,uri://datarecognitioncorp.com/AccommodationDescriptor,Acc1
3-
Acc2,Accomodations 2,uri://datarecognitioncorp.com/AccommodationDescriptor,Acc2
4-
Acc3,Accomodations 3,uri://datarecognitioncorp.com/AccommodationDescriptor,Acc3
5-
Acc4,Accomodations 4,uri://datarecognitioncorp.com/AccommodationDescriptor,Acc4
6-
Acc5,Accomodations 5,uri://datarecognitioncorp.com/AccommodationDescriptor,Acc5
7-
Acc6,Accomodations 6,uri://datarecognitioncorp.com/AccommodationDescriptor,Acc6
8-
Acc7,Accomodations 7,uri://datarecognitioncorp.com/AccommodationDescriptor,Acc7
9-
Acc8,Accomodations 8,uri://datarecognitioncorp.com/AccommodationDescriptor,Acc8
10-
Acc9,Accomodations 9,uri://datarecognitioncorp.com/AccommodationDescriptor,Acc9
11-
Acc10,Accomodations 10,uri://datarecognitioncorp.com/AccommodationDescriptor,Acc10
12-
ESLAcc1,ESL Accomodations 1,uri://datarecognitioncorp.com/AccommodationDescriptor,ESLAcc1
13-
ESLAcc2,ESL Accomodations 2,uri://datarecognitioncorp.com/AccommodationDescriptor,ESLAcc2
14-
ESLAcc3,ESL Accomodations 3,uri://datarecognitioncorp.com/AccommodationDescriptor,ESLAcc3
15-
ESLAcc4,ESL Accomodations 4,uri://datarecognitioncorp.com/AccommodationDescriptor,ESLAcc4
16-
ESLAcc5,ESL Accomodations 5,uri://datarecognitioncorp.com/AccommodationDescriptor,ESLAcc5
17-
ESLAcc6,ESL Accomodations 6,uri://datarecognitioncorp.com/AccommodationDescriptor,ESLAcc6
18-
1Acc1,Session 1 Accomodations 1,uri://datarecognitioncorp.com/AccommodationDescriptor,1Acc1
19-
1Acc2,Session 1 Accomodations 2,uri://datarecognitioncorp.com/AccommodationDescriptor,1Acc2
20-
1Acc3,Session 1 Accomodations 3,uri://datarecognitioncorp.com/AccommodationDescriptor,1Acc3
21-
1Acc4,Session 1 Accomodations 4,uri://datarecognitioncorp.com/AccommodationDescriptor,1Acc4
22-
1Acc5,Session 1 Accomodations 5,uri://datarecognitioncorp.com/AccommodationDescriptor,1Acc5
23-
1Acc6,Session 1 Accomodations 6,uri://datarecognitioncorp.com/AccommodationDescriptor,1Acc6
24-
1Acc7,Session 1 Accomodations 7,uri://datarecognitioncorp.com/AccommodationDescriptor,1Acc7
25-
1Acc8,Session 1 Accomodations 8,uri://datarecognitioncorp.com/AccommodationDescriptor,1Acc8
26-
1Acc9,Session 1 Accomodations 9,uri://datarecognitioncorp.com/AccommodationDescriptor,1Acc9
27-
1Acc10,Session 1 Accomodations 10,uri://datarecognitioncorp.com/AccommodationDescriptor,1Acc10
28-
1Acc11,Session 1 Accomodations 11,uri://datarecognitioncorp.com/AccommodationDescriptor,1Acc11
29-
1ESLAcc1,Session 1 ESL Accomodations 1,uri://datarecognitioncorp.com/AccommodationDescriptor,1ESLAcc1
30-
1ESLAcc2,Session 1 ESL Accomodations 2,uri://datarecognitioncorp.com/AccommodationDescriptor,1ESLAcc2
31-
1ESLAcc3,Session 1 ESL Accomodations 3,uri://datarecognitioncorp.com/AccommodationDescriptor,1ESLAcc3
32-
1ESLAcc4,Session 1 ESL Accomodations 4,uri://datarecognitioncorp.com/AccommodationDescriptor,1ESLAcc4
33-
1ESLAcc5,Session 1 ESL Accomodations 5,uri://datarecognitioncorp.com/AccommodationDescriptor,1ESLAcc5
34-
1ESLAcc6,Session 1 ESL Accomodations 6,uri://datarecognitioncorp.com/AccommodationDescriptor,1ESLAcc6
35-
2Acc1,Session 2 Accomodations 1,uri://datarecognitioncorp.com/AccommodationDescriptor,2Acc1
36-
2Acc2,Session 2 Accomodations 2,uri://datarecognitioncorp.com/AccommodationDescriptor,2Acc2
37-
2Acc3,Session 2 Accomodations 3,uri://datarecognitioncorp.com/AccommodationDescriptor,2Acc3
38-
2Acc4,Session 2 Accomodations 4,uri://datarecognitioncorp.com/AccommodationDescriptor,2Acc4
39-
2Acc5,Session 2 Accomodations 5,uri://datarecognitioncorp.com/AccommodationDescriptor,2Acc5
40-
2Acc6,Session 2 Accomodations 6,uri://datarecognitioncorp.com/AccommodationDescriptor,2Acc6
41-
2Acc7,Session 2 Accomodations 7,uri://datarecognitioncorp.com/AccommodationDescriptor,2Acc7
42-
2Acc8,Session 2 Accomodations 8,uri://datarecognitioncorp.com/AccommodationDescriptor,2Acc8
43-
2Acc9,Session 2 Accomodations 9,uri://datarecognitioncorp.com/AccommodationDescriptor,2Acc9
44-
2Acc10,Session 2 Accomodations 10,uri://datarecognitioncorp.com/AccommodationDescriptor,2Acc10
45-
2ESLAcc1,Session 2 ESL Accomodations 1,uri://datarecognitioncorp.com/AccommodationDescriptor,2ESLAcc1
46-
2ESLAcc2,Session 2 ESL Accomodations 2,uri://datarecognitioncorp.com/AccommodationDescriptor,2ESLAcc2
47-
2ESLAcc3,Session 2 ESL Accomodations 3,uri://datarecognitioncorp.com/AccommodationDescriptor,2ESLAcc3
48-
2ESLAcc4,Session 2 ESL Accomodations 4,uri://datarecognitioncorp.com/AccommodationDescriptor,2ESLAcc4
49-
2ESLAcc5,Session 2 ESL Accomodations 5,uri://datarecognitioncorp.com/AccommodationDescriptor,2ESLAcc5
1+
codeValue,description,namespace,shortDescription,start_year,end_year
2+
Acc1,Accomodations 1,uri://datarecognitioncorp.com/AccommodationDescriptor,Acc1,,
3+
Acc2,Accomodations 2,uri://datarecognitioncorp.com/AccommodationDescriptor,Acc2,,
4+
Acc3,Accomodations 3,uri://datarecognitioncorp.com/AccommodationDescriptor,Acc3,,
5+
Acc4,Accomodations 4,uri://datarecognitioncorp.com/AccommodationDescriptor,Acc4,,
6+
Acc5,Accomodations 5,uri://datarecognitioncorp.com/AccommodationDescriptor,Acc5,,
7+
Acc6,Accomodations 6,uri://datarecognitioncorp.com/AccommodationDescriptor,Acc6,,
8+
Acc7,Accomodations 7,uri://datarecognitioncorp.com/AccommodationDescriptor,Acc7,,
9+
Acc8,Accomodations 8,uri://datarecognitioncorp.com/AccommodationDescriptor,Acc8,,
10+
Acc9,Accomodations 9,uri://datarecognitioncorp.com/AccommodationDescriptor,Acc9,,
11+
Acc10,Accomodations 10,uri://datarecognitioncorp.com/AccommodationDescriptor,Acc10,,
12+
ESLAcc1,ESL Accomodations 1,uri://datarecognitioncorp.com/AccommodationDescriptor,ESLAcc1,,
13+
ESLAcc2,ESL Accomodations 2,uri://datarecognitioncorp.com/AccommodationDescriptor,ESLAcc2,,
14+
ESLAcc3,ESL Accomodations 3,uri://datarecognitioncorp.com/AccommodationDescriptor,ESLAcc3,,
15+
ESLAcc4,ESL Accomodations 4,uri://datarecognitioncorp.com/AccommodationDescriptor,ESLAcc4,,
16+
ESLAcc5,ESL Accomodations 5,uri://datarecognitioncorp.com/AccommodationDescriptor,ESLAcc5,,
17+
ESLAcc6,ESL Accomodations 6,uri://datarecognitioncorp.com/AccommodationDescriptor,ESLAcc6,,
18+
1Acc1,Session 1 Accomodations 1,uri://datarecognitioncorp.com/AccommodationDescriptor,1Acc1,,
19+
1Acc2,Session 1 Accomodations 2,uri://datarecognitioncorp.com/AccommodationDescriptor,1Acc2,,
20+
1Acc3,Session 1 Accomodations 3,uri://datarecognitioncorp.com/AccommodationDescriptor,1Acc3,,
21+
1Acc4,Session 1 Accomodations 4,uri://datarecognitioncorp.com/AccommodationDescriptor,1Acc4,,
22+
1Acc5,Session 1 Accomodations 5,uri://datarecognitioncorp.com/AccommodationDescriptor,1Acc5,,
23+
1Acc6,Session 1 Accomodations 6,uri://datarecognitioncorp.com/AccommodationDescriptor,1Acc6,,
24+
1Acc7,Session 1 Accomodations 7,uri://datarecognitioncorp.com/AccommodationDescriptor,1Acc7,,
25+
1Acc8,Session 1 Accomodations 8,uri://datarecognitioncorp.com/AccommodationDescriptor,1Acc8,,
26+
1Acc9,Session 1 Accomodations 9,uri://datarecognitioncorp.com/AccommodationDescriptor,1Acc9,,
27+
1Acc10,Session 1 Accomodations 10,uri://datarecognitioncorp.com/AccommodationDescriptor,1Acc10,,
28+
1Acc11,Session 1 Accomodations 11,uri://datarecognitioncorp.com/AccommodationDescriptor,1Acc11,,2024
29+
1ESLAcc1,Session 1 ESL Accomodations 1,uri://datarecognitioncorp.com/AccommodationDescriptor,1ESLAcc1,,
30+
1ESLAcc2,Session 1 ESL Accomodations 2,uri://datarecognitioncorp.com/AccommodationDescriptor,1ESLAcc2,,
31+
1ESLAcc3,Session 1 ESL Accomodations 3,uri://datarecognitioncorp.com/AccommodationDescriptor,1ESLAcc3,,
32+
1ESLAcc4,Session 1 ESL Accomodations 4,uri://datarecognitioncorp.com/AccommodationDescriptor,1ESLAcc4,,
33+
1ESLAcc5,Session 1 ESL Accomodations 5,uri://datarecognitioncorp.com/AccommodationDescriptor,1ESLAcc5,,
34+
1ESLAcc6,Session 1 ESL Accomodations 6,uri://datarecognitioncorp.com/AccommodationDescriptor,1ESLAcc6,,
35+
2Acc1,Session 2 Accomodations 1,uri://datarecognitioncorp.com/AccommodationDescriptor,2Acc1,,
36+
2Acc2,Session 2 Accomodations 2,uri://datarecognitioncorp.com/AccommodationDescriptor,2Acc2,,
37+
2Acc3,Session 2 Accomodations 3,uri://datarecognitioncorp.com/AccommodationDescriptor,2Acc3,,
38+
2Acc4,Session 2 Accomodations 4,uri://datarecognitioncorp.com/AccommodationDescriptor,2Acc4,,
39+
2Acc5,Session 2 Accomodations 5,uri://datarecognitioncorp.com/AccommodationDescriptor,2Acc5,,
40+
2Acc6,Session 2 Accomodations 6,uri://datarecognitioncorp.com/AccommodationDescriptor,2Acc6,,
41+
2Acc7,Session 2 Accomodations 7,uri://datarecognitioncorp.com/AccommodationDescriptor,2Acc7,,
42+
2Acc8,Session 2 Accomodations 8,uri://datarecognitioncorp.com/AccommodationDescriptor,2Acc8,,
43+
2Acc9,Session 2 Accomodations 9,uri://datarecognitioncorp.com/AccommodationDescriptor,2Acc9,,
44+
2Acc10,Session 2 Accomodations 10,uri://datarecognitioncorp.com/AccommodationDescriptor,2Acc10,,
45+
2ESLAcc1,Session 2 ESL Accomodations 1,uri://datarecognitioncorp.com/AccommodationDescriptor,2ESLAcc1,,
46+
2ESLAcc2,Session 2 ESL Accomodations 2,uri://datarecognitioncorp.com/AccommodationDescriptor,2ESLAcc2,,
47+
2ESLAcc3,Session 2 ESL Accomodations 3,uri://datarecognitioncorp.com/AccommodationDescriptor,2ESLAcc3,,
48+
2ESLAcc4,Session 2 ESL Accomodations 4,uri://datarecognitioncorp.com/AccommodationDescriptor,2ESLAcc4,,
49+
2ESLAcc5,Session 2 ESL Accomodations 5,uri://datarecognitioncorp.com/AccommodationDescriptor,2ESLAcc5,,

0 commit comments

Comments
 (0)