Skip to content

Commit 2ad7b76

Browse files
jalvord1sleblanc23ryanaguilar
authored
Feature/cambridge (#212)
* cambridge first draft * Update asssessmentReportingMethodDescriptors * Update earthmover.yaml to match new spec and sample * Update possible scores * Update sample file * Fix sample fix state * Add comment about courseLevel * Remove extra qualifications score from possible scores * Remove score if result == '-' * Update assessments template * Remove series * Add series to studentAssessmentIdentifier * Fix courseLevel * Update sample series col * Update sample IDs * Update testAdminDate to use series * Fix sample file formatting * Update sample file series * Update series format * Remove debug statement * Update possible_student_id_columns * Update possible_student_id_columns * Remove snake case * Change student_state_id to student_id in studentAssessmentIdentifier * Fix case * Add series * Change grade to Grade * small fixes for cambridge --------- Co-authored-by: sleblanc23 <[email protected]> Co-authored-by: Ryan Aguilar <[email protected]>
1 parent 2afcc2e commit 2ad7b76

13 files changed

+451
-0
lines changed

‎assessments/Cambridge/.gitignore‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
packages/*
2+
packages

‎assessments/Cambridge/README.md‎

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
This is an earthmover bundle created from the following Ed-Fi Data Import Tool mapping:
2+
* **Title**: Cambridge International AS & A Levels Results - API 3.X
3+
* **Description**: This template maps Cambridge International assessment results.
4+
* **API version**: 5.3
5+
* **Submitter name**: Samantha LeBlanc
6+
* **Submitter organization**: Education Analytics
7+
8+
To run this bundle, please add your own source file(s) and column(s):
9+
<details>
10+
<summary><code>data/cambridge.csv</code></summary>
11+
This template will only work with the vendor specification of the Cambridge results file. See `data/sample_anonymized_file.csv` for column names.
12+
</details>
13+
14+
Or use the sample file (data/sample_anonymized_file.csv).
15+
16+
## CLI Parameters
17+
- OUTPUT_DIR: Where output files will be written
18+
- STATE_FILE: Where to store the earthmover runs.csv file
19+
- INPUT_FILE: The path to the ASVAB .csv file you want to transform
20+
- SCHOOL_YEAR: The school year associated with the results file
21+
- STUDENT_ID_NAME: Which column to use as the Ed-Fi studentUniqueId
22+
- EDFI_DS_VERSION: The Ed-Fi data standard major version for your ODS. Integer only; supported versions are 3, 4, and 5
23+
24+
### Examples
25+
Running earthmover for a Cambridge file:
26+
```bash
27+
earthmover run -c ./earthmover.yaml -p '{
28+
"OUTPUT_DIR": "./output",
29+
"STATE_FILE": "./runs.csv",
30+
"INPUT_FILE": "./data/sample_anonymized_file.csv",
31+
"SCHOOL_YEAR": "2025",
32+
"STUDENT_ID_NAME": "STUDENT_STATE_ID",
33+
"EDFI_DS_VERSION": 3 }'
34+
```
35+
36+
Once you have inspected the output JSONL for issues, check the settings in `lightbeam.yaml` and transmit them to your Ed-Fi API with
37+
```bash
38+
lightbeam validate+send -c ./lightbeam.yaml -p '{
39+
"DATA_DIR": "./output/",
40+
"EDFI_API_BASE_URL": "yourURL",
41+
"EDFI_API_CLIENT_ID": "yourID",
42+
"EDFI_API_CLIENT_SECRET": "yourSecret",
43+
"API_YEAR": yourAPIYear }'
44+
```

‎assessments/Cambridge/data/sample_anonymized_file.csv‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
STATE,DISTRICT,CENTRE,CANDIDATE_NUMBER,STUDENT_NAME,STUDENT_STATE_ID,ETHNICITY,NATIONAL_ID,SEX,DOB,SERIES,QUALIFICATION,LEVEL_ENTERED,LEVEL_ACHIEVED,SYLLABUS,SYLLABUS_NAME,GRADE
2+
State,District 01,UX421,1111,STUDENT ONE,00001,,,F,11111,30-May,AS & A Level,AS Level,AS Level,8021,ENGLISH GENERAL PAPER,e
3+
State,District 01,UX421,2222,STUDENT TWO,00002,,,F,11111,30-May,ADIP,ADIP,ADIP,ADIP,AICE DIPLOMA,M
4+
State,District 01,UX421,3333,STUDENT THREE,00003,,,F,11111,1-Jun,AS & A Level,AS Level,AS Level,8021,ENGLISH GENERAL PAPER,a
5+
State,District 01,UX421,4444,STUDENT FOUR,00004,,,M,11111,1-Jun,AS & A Level,AS Level,-,8021,ENGLISH GENERAL PAPER,X
6+
State,District 01,UX421,5555,STUDENT FIVE,00005,,,F,11111,2-Jun,AS & A Level,AS Level,-,8021,ENGLISH GENERAL PAPER,U

‎assessments/Cambridge/earthmover.yaml‎

Lines changed: 137 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,137 @@
1+
version: 2
2+
3+
4+
config:
5+
log_level: INFO
6+
output_dir: ${OUTPUT_DIR}
7+
memory_limit: 1GB
8+
state_file: ${STATE_FILE}
9+
show_graph: False
10+
show_stacktrace: true
11+
parameter_defaults:
12+
STUDENT_ID_NAME: 'edFi_studentUniqueID' # default to the column added by the apply_xwalk package of student ID xwalking feature
13+
POSSIBLE_STUDENT_ID_COLUMNS: 'STUDENT_STATE_ID' # this column will need to be added and populated as a pre-preprocessing step before this bundle can be used
14+
DESCRIPTOR_NAMESPACE: uri://ed-fi.org
15+
EDFI_DS_VERSION: 3
16+
17+
sources:
18+
assessments:
19+
file: ./seeds/assessments.csv
20+
header_rows: 1
21+
assessmentReportingMethodDescriptors:
22+
file: ./seeds/assessmentReportingMethodDescriptors.csv
23+
header_rows: 1
24+
gradeLevelDescriptors:
25+
file: ./seeds/gradeLevelDescriptors.csv
26+
header_rows: 1
27+
performanceLevelDescriptors:
28+
file: ./seeds/performanceLevelDescriptors.csv
29+
header_rows: 1
30+
31+
input:
32+
# THIS FILE DOES NOT EXIST; IT MUST BE SUPPLIED BY THE USER!
33+
file: ${INPUT_FILE}
34+
header_rows: 1
35+
# See the accompanying bundle_metadata.json for a list of required columns for this file
36+
# Or see data/sample_anonymized_file.csv
37+
38+
39+
40+
transformations:
41+
input:
42+
source: $sources.input
43+
operations: []
44+
45+
clean_results:
46+
source: $transformations.input
47+
operations:
48+
- operation: duplicate_columns
49+
columns:
50+
${STUDENT_ID_NAME}: student_id
51+
# governance decision: removing rows without an achieved level because we have no other way to assign their identifier
52+
- operation: filter_rows
53+
query: LEVEL_ACHIEVED == '-'
54+
behavior: exclude
55+
- operation: add_columns
56+
columns:
57+
assessmentIdentifier: "{%raw%}cambridge_{%- if QUALIFICATION == 'ADIP' and LEVEL_ACHIEVED == 'ADIP' -%}aice_diploma{%- elif QUALIFICATION != 'ADIP' and LEVEL_ACHIEVED != '-' -%}{{ LEVEL_ACHIEVED.split(' ')[0] }}_{{ SYLLABUS }}{%- endif -%}{%endraw%}"
58+
namespace: uri://cambridgeinternational.org
59+
schoolYear: ${SCHOOL_YEAR}
60+
testMonth: >
61+
{%raw%}
62+
{%- if 'Apr' in SERIES -%}
63+
04
64+
{%- elif 'May' in SERIES -%}
65+
05
66+
{%- elif 'Jun' in SERIES -%}
67+
06
68+
{%- elif 'Sep' in SERIES -%}
69+
09
70+
{%- elif 'Oct' in SERIES -%}
71+
10
72+
{%- elif 'Nov' in SERIES -%}
73+
11
74+
{%- endif -%}
75+
{%endraw%}
76+
testYear: >
77+
{%raw%}
78+
{%- if testMonth|int < 7 -%}
79+
{{ schoolYear }}
80+
{%- else -%}
81+
{{ schoolYear | int - 1 }}
82+
{%- endif -%}
83+
{%endraw%}
84+
testDay: "{%raw%}{{'%02d' % SERIES.split('-')[0]|int}}{%endraw%}"
85+
administrationDate: "{%raw%}{{ testYear | string }}-{{ testMonth }}-{{ testDay }}{%endraw%}"
86+
edfi_ds_version: ${EDFI_DS_VERSION}
87+
- operation: combine_columns
88+
columns:
89+
- student_id
90+
- administrationDate
91+
- SYLLABUS
92+
new_column: studentAssessmentIdentifier
93+
separator: '-'
94+
- operation: modify_columns
95+
columns:
96+
studentAssessmentIdentifier: "{%raw%}{{ md5(studentAssessmentIdentifier) }}{%endraw%}"
97+
98+
# instead of hardcoding descriptors that typically use ed-fi defaults (like subjects/grades)
99+
# we list those in a csv and aggregate as a transformation so we never hardcode those in the jsont
100+
grade_level_descriptors:
101+
source: $sources.gradeLevelDescriptors
102+
operations:
103+
- operation: group_by
104+
group_by_columns:
105+
- assessmentFamily
106+
create_columns:
107+
grade_descriptors: agg(gradeLevelDescriptor,;)
108+
109+
assessments:
110+
source: $sources.assessments
111+
operations:
112+
- operation: join
113+
sources:
114+
- $transformations.grade_level_descriptors
115+
join_type: inner
116+
left_key: assessmentFamily
117+
right_key: assessmentFamily
118+
119+
120+
121+
destinations:
122+
assessments:
123+
source: $transformations.assessments
124+
template: ./templates/assessments.jsont
125+
extension: jsonl
126+
assessmentReportingMethodDescriptors:
127+
source: $sources.assessmentReportingMethodDescriptors
128+
template: ./templates/descriptors.jsont
129+
extension: jsonl
130+
performanceLevelDescriptors:
131+
source: $sources.performanceLevelDescriptors
132+
template: ./templates/descriptors.jsont
133+
extension: jsonl
134+
studentAssessments:
135+
source: $transformations.clean_results
136+
template: ./templates/studentAssessments.jsont
137+
extension: jsonl

‎assessments/Cambridge/lightbeam.yaml‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
state_dir: ${STATE_DIR}
2+
data_dir: ${DATA_DIR}
3+
edfi_api:
4+
base_url: ${EDFI_API_BASE_URL}
5+
version: 3
6+
mode: district_specific
7+
year: ${API_YEAR}
8+
client_id: ${EDFI_API_CLIENT_ID}
9+
client_secret: ${EDFI_API_CLIENT_SECRET}
10+
connection:
11+
pool_size: 2
12+
timeout: 6000
13+
num_retries: 10
14+
backoff_factor: 1.5
15+
retry_statuses: [429, 500, 502, 503, 504]
16+
verify_ssl: True
17+
log_level: DEBUG
18+
show_stacktrace: True

‎assessments/Cambridge/output/.gitignore‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
**.jsonl

‎assessments/Cambridge/seeds/assessmentReportingMethodDescriptors.csv‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
codeValue,description,namespace,shortDescription
2+
Grade,The result earned on the assessment,uri://cambridgeinternational.org/AssessmentReportingMethodDescriptor,Grade
3+
Level Entered,Level of exam entry reported by the school (will be AS Level (advanced subsidiary) or A Level (advanced),uri://cambridgeinternational.org/AssessmentReportingMethodDescriptor,Level Entered
4+
Level Achieved,The grade result the student achieved on the endorsement where applicable,uri://cambridgeinternational.org/AssessmentReportingMethodDescriptor,Level Achieved
5+
Syllabus,Cambridge’s unique code for the associated syllabus,uri://cambridgeinternational.org/AssessmentReportingMethodDescriptor,Syllabus
6+
Syllabus Name,Cambridge’s name for the associated syllabus,uri://cambridgeinternational.org/AssessmentReportingMethodDescriptor,Syllabus Name
7+
AICE Diploma Level,The Cambridge AICE Diploma is a group certificate that is awarded on a points system. Distinction: 360-420 points; Merit: 250-359 points; Pass: 140-249 points.,uri://cambridgeinternational.org/AssessmentReportingMethodDescriptor,AICE Diploma Level
8+
Series,Exam series candidate complete the exam,uri://cambridgeinternational.org/AssessmentReportingMethodDescriptor,Series

0 commit comments

Comments
 (0)