Skip to content

Commit 462ca13

Browse files
stephenfuquastephenfuqua
authored
[PERF-330] Bulk upload test scripts (#110)
* Documentation fix * Files for bulk uploading Grand Bend or Southridge data * New southridge file name * Copy extra files into bootstrap directory * Extra comments * Rename directory * Readme update * Fix a few errors * Also give direct access to this school * Require exact package version
1 parent 39f62c7 commit 462ca13

File tree

10 files changed

+681
-2
lines changed

10 files changed

+681
-2
lines changed

‎.gitignore‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -78,3 +78,8 @@ override.tf.json
7878
.terraformrc
7979
terraform.rc
8080
**/.ipynb_checkpoints/*
81+
82+
83+
# Bulk load related
84+
.packages/
85+
.working/

‎eng/ods-api/README.md‎

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2,8 +2,7 @@
22

33
This directory contains startup scripts for the ODS/API running in Docker, using
44
versions 5.3, 5.4, 6.1, and 6.2; and another that supports versions 7.1, 7.2,
5-
Docker Compose for orchestration. There are two scripts: one that supports
6-
and 7.3 (prerelease).
5+
Docker Compose for orchestration.
76

87
> [!WARNING]
98
> These are poorly secured environments and should not be exposed over the open

‎eng/ods-api/v7/bootstrap.sql‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,12 @@ from dbo.Applications
2525
where ApplicationName = 'Bootstrap Application'
2626
and not exists (select 1 from dbo.ApplicationEducationOrganizations where EducationOrganizationId = 255901);
2727

28+
insert into dbo.ApplicationEducationOrganizations (EducationOrganizationId, Application_ApplicationId)
29+
select 19255901, ApplicationId
30+
from dbo.Applications
31+
where ApplicationName = 'Bootstrap Application'
32+
and not exists (select 1 from dbo.ApplicationEducationOrganizations where EducationOrganizationId = 19255901);
33+
2834
insert into dbo.ApiClients (Key, Secret, Name, IsApproved, UseSandbox, SandboxType, SecretIsHashed, Application_ApplicationId)
2935
select 'minimalKey', 'minimalSecret', 'Bootstrap', true, false, 0, false, ApplicationId from dbo.Applications
3036
where not exists (select 1 from dbo.ApiClients where Name = 'Bootstrap');

‎src/bulk-load/.gitignore‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
# Something about these scripts sometimes creates a .NET solution file
2+
*.sln
3+

‎src/bulk-load/Invoke-LoadGrandBend.ps1‎

Lines changed: 67 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,67 @@
1+
# SPDX-License-Identifier: Apache-2.0
2+
# Licensed to the Ed-Fi Alliance under one or more agreements.
3+
# The Ed-Fi Alliance licenses this file to you under the Apache License, Version 2.0.
4+
# See the LICENSE and NOTICES files in the project root for more information.
5+
6+
# Runs the complete bulk upload of the Grand Bend dataset, aka "populated template"
7+
8+
param(
9+
[string]
10+
$Key = "minimalKey",
11+
12+
[string]
13+
$Secret = "minimalSecret",
14+
15+
# 8080 is the default docker port. 5198 is the default when running F5 on
16+
# DMS. 8001 for the ODS/API running from the Docker Compose files in this
17+
# performance testing repository.
18+
[string]
19+
$BaseUrl = "http://localhost:8080",
20+
21+
# 5.0.0-dev.1, 5.1.0-dev.3
22+
[string]
23+
$SampleDataVersion = "5.1.0-dev.3",
24+
25+
# When false (default), only loads descriptors
26+
[switch]
27+
$FullDataSet,
28+
29+
# Use with the DMS, do not use with the ODS/API
30+
[switch]
31+
$LoadSchoolYear
32+
)
33+
34+
#Requires -Version 7
35+
$ErrorActionPreference = "Stop"
36+
37+
Import-Module ./modules/Package-Management.psm1 -Force
38+
Import-Module ./modules/Get-XSD.psm1 -Force
39+
Import-Module ./modules/BulkLoad.psm1 -Force
40+
41+
$paths = Initialize-ToolsAndDirectories
42+
$paths.SampleDataDirectory = Import-SampleData -Template "GrandBend" -Version $SampleDataVersion
43+
44+
$parameters = @{
45+
BaseUrl = $BaseUrl
46+
Key = $Key
47+
Secret = $Secret
48+
Paths = $paths
49+
LoadSchoolYear = $LoadSchoolYear
50+
}
51+
52+
$stopwatch = [system.diagnostics.stopwatch]::StartNew()
53+
Write-Bootstrap @parameters
54+
55+
if ($FullDataSet) {
56+
$parameters = @{
57+
BaseUrl = $BaseUrl
58+
Key = $Key
59+
Secret = $Secret
60+
SampleDataDirectory = $Paths.SampleDataDirectory
61+
Paths = $Paths
62+
}
63+
Write-XmlFiles @parameters
64+
}
65+
66+
$stopwatch.Stop()
67+
$stopwatch.Elapsed | Out-Host

‎src/bulk-load/Invoke-LoadSouthridge.ps1‎

Lines changed: 62 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,62 @@
1+
# SPDX-License-Identifier: Apache-2.0
2+
# Licensed to the Ed-Fi Alliance under one or more agreements.
3+
# The Ed-Fi Alliance licenses this file to you under the Apache License, Version 2.0.
4+
# See the LICENSE and NOTICES files in the project root for more information.
5+
6+
7+
param(
8+
[string]
9+
$Key = "minimalKey",
10+
11+
[string]
12+
$Secret = "minimalSecret",
13+
14+
# 8080 is the default docker port. 5198 is the default when running F5 on
15+
# DMS. 8001 for the ODS/API running from the Docker Compose files in this
16+
# performance testing repository.
17+
[string]
18+
$BaseUrl = "http://localhost:8080",
19+
20+
# When false (default), only loads descriptors
21+
[switch]
22+
$FullDataSet,
23+
24+
# Use with the DMS, do not use with the ODS/API
25+
[switch]
26+
$LoadSchoolYear
27+
)
28+
29+
#Requires -Version 7
30+
$ErrorActionPreference = "Stop"
31+
32+
Import-Module ./modules/Package-Management.psm1 -Force
33+
Import-Module ./modules/Get-XSD.psm1 -Force
34+
Import-Module ./modules/BulkLoad.psm1 -Force
35+
36+
$paths = Initialize-ToolsAndDirectories
37+
$paths.SampleDataDirectory = Import-SampleData -Template "Southridge"
38+
39+
$parameters = @{
40+
BaseUrl = $BaseUrl
41+
Key = $Key
42+
Secret = $Secret
43+
Paths = $paths
44+
LoadSchoolYear = $LoadSchoolYear
45+
}
46+
47+
$stopwatch = [system.diagnostics.stopwatch]::StartNew()
48+
Write-Bootstrap @parameters
49+
50+
if ($FullDataSet) {
51+
$parameters = @{
52+
BaseUrl = $BaseUrl
53+
Key = $Key
54+
Secret = $Secret
55+
SampleDataDirectory = $Paths.SampleDataDirectory
56+
Paths = $Paths
57+
}
58+
Write-XmlFiles @parameters
59+
}
60+
61+
$stopwatch.Stop()
62+
$stopwatch.Elapsed | Out-Host

‎src/bulk-load/README.MD‎

Lines changed: 68 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,68 @@
1+
# Bulk Upload Testing
2+
3+
These scripts can be used with the Data Management Service or with the ODS/API
4+
Platform, version 7.1 or 72. There are two data set options: Grand Bend (small)
5+
and Southridge (large); in each case, it is possible to run a bootstrap data
6+
load only (descriptors, education organizations, learning standards), or to turn
7+
on a full data load.
8+
9+
## Testing with the Data Management Service
10+
11+
Recommended approach for the DMS: run either `start-local-dms.ps1` or
12+
`start-published-dms.ps1` from the `eng/docker-compose` directory. By default,
13+
this will run the API on `http://localhost:8080`.
14+
15+
Then run one of the following commands:
16+
17+
```shell
18+
# Only the bootstrap data
19+
./Invoke-LoadGrandBend.ps1 -LoadSchoolYear
20+
21+
# ... or a larger data set
22+
./Invoke-LoadSouthridge.ps1 -LoadSchoolYear
23+
24+
# Run the entire data set
25+
./Invoke-LoadSouthridge.ps1 -FullDataSet -LoadSchoolYear
26+
27+
# ... or a larger data set
28+
./Invoke-LoadGrandBend.ps1 -FullDataSet -LoadSchoolYear
29+
```
30+
31+
## Testing with the ODS/API
32+
33+
Recommended approach for the ODS/API: use the `bootstrap.ps1 -Hub` the script in
34+
`Ed-Fi-ODS-Implementation/Docker`. Run without any arguments, this will startup
35+
a single local instance of the ODS/API, running on PostgreSQL. By default, this will
36+
run the API on `http://localhost:8001`.
37+
38+
Compared to the DMS instructions: don't use `LoadSchoolYear`, change the
39+
`BaseUrl`, and optionally change the key and secret (default values are shown:
40+
`minimalKey` and `minimalSecret`).
41+
42+
```shell
43+
# Only the bootstrap data
44+
./Invoke-LoadGrandBend.ps1 -BaseUrl http://localhost:8001 -Key minimalKey -Secret minimalSecret
45+
46+
# ... or a larger data set
47+
./Invoke-LoadSouthridge.ps1 -BaseUrl http://localhost:8001 -Key minimalKey -Secret minimalSecret
48+
49+
# Run the entire data set
50+
./Invoke-LoadSouthridge.ps1 -FullDataSet -BaseUrl http://localhost:8001 -Key minimalKey -Secret minimalSecret
51+
52+
# ... or a larger data set
53+
./Invoke-LoadGrandBend.ps1 -FullDataSet -BaseUrl http://localhost:8001 -Key minimalKey -Secret minimalSecret
54+
```
55+
56+
## Correct Education Organization Id
57+
58+
If running Southridge against the ODS/API using, then make sure that client
59+
application is setup to work with education organization ID 255901. The
60+
following script might be useful _if you have only one API client in the
61+
`EdFi_Admin` database_.
62+
63+
```sql
64+
update dbo.applicationeducationorganizations set educationorganizationid=255901;
65+
```
66+
67+
This is not necessary when using the bootstrap startup process in this
68+
repository's [eng/ods-api](../../eng/ods-api/) directory.

0 commit comments

Comments
 (0)