[DMS-243] Database configuration changes

docs/CONFIGURATION.md

@@ -1,6 +1,6 @@
 # AppSettings Configuration
 
-The sections below describe custom configuration options in the `appSettings` file.
+The sections below describe custom configuration options in the `appSettings.json` file.
 
 ## AppSettings
 
@@ -10,6 +10,13 @@ The sections below describe custom configuration options in the `appSettings` fi
 | DeployDatabaseOnStartup  | When `true` the database in `ConnectionStrings:DatabaseConnection` will be created and initialized on startup.                                                                                          |
 | BypassStringTypeCoercion | String type coercion attempts to coerce boolean and numeric strings to their proper type on `POST` and `PUT` requests. For example `"true"` becomes `true`. This setting bypasses that for performance. |
 
+## DatabaseOptions
+
+| Parameter      | Description                                                                                                                                                              |
+| -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
+| IsolationLevel | The `System.Data.IsolationLevel` to use for transaction locks. See [documentation](https://learn.microsoft.com/en-us/dotnet/api/system.data.isolationlevel?view=net-8.0) |
+
+
 ## RateLimit
 
 Basic rate limiting can be applied by supplying a `RateLimit` object in the
@@ -21,6 +28,6 @@ The `RateLimit` object should have the following parameters.
 
 | Parameter     | Description                                                                                                                                                                                                                                                                |
 | ------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| `PermitLimit` | The number of requests permitted within the window time span. This will be the number of requests, per hostname permitted per timeframe (`Window`). Must be > 0.                                                                                                           |
-| `Window`      | The number of seconds before the `PermitLimit` is reset. Must be > 0.                                                                                                                                                                                                      |
-| `QueueLimit`  | The maximum number of requests that can be Queued once `PermitLimit`s are exhausted. These requests will wait until the `Window` expires and will be processed FIFO. When the queue is exhausted, clients will receive a `429` `Too Many Requests` response. Must be >= 0. |
+| PermitLimit | The number of requests permitted within the window time span. This will be the number of requests, per hostname permitted per timeframe (`Window`). Must be > 0.                                                                                                           |
+| Window      | The number of seconds before the `PermitLimit` is reset. Must be > 0.                                                                                                                                                                                                      |
+| QueueLimit  | The maximum number of requests that can be Queued once `PermitLimit`s are exhausted. These requests will wait until the `Window` expires and will be processed FIFO. When the queue is exhausted, clients will receive a `429` `Too Many Requests` response. Must be >= 0. |

docs/DOCKER.md

@@ -30,13 +30,13 @@ Service container.
 NEED_DATABASE_SETUP=<Flag (true or false) to decide whether the DMS database setup needs to be executed as part of the container setup>
 POSTGRES_ADMIN_USER=<Admin user to use with database setup>
 POSTGRES_ADMIN_PASSWORD=<Admin password to use with database setup>
-POSTGRES_USER=<Non-admin user to use for accessing the database from the DMS application>
-POSTGRES_PASSWORD=<Non-admin user password>
 POSTGRES_PORT=<Port for postgres server Eg. 5432>
 POSTGRES_HOST=<DNS or IP address of the PostgreSQL Server, i.e. sql.somedns.org Eg. 172.25.32.1>
 LOG_LEVEL=<serilog log level i.e. Information>
 OAUTH_TOKEN_ENDPOINT=<Authentication service url>
 BYPASS_STRING_COERCION=<Boolean whether to bypass coercion of boolean and numeric values represented as strings to their natural type. Eg. "true" = true>
+DATABASE_ISOLATION_LEVEL=<The System.Data.IsolationLevel to use for transaction locking. Eg. RepeatableRead>
+DATABASE_CONNECTION_STRING=<The non-admin database connection string>
 ```
 
 For example, you might have a `.env` file like the following:
@@ -45,13 +45,13 @@ For example, you might have a `.env` file like the following:
 NEED_DATABASE_SETUP=true
 POSTGRES_ADMIN_USER=postgres
 POSTGRES_ADMIN_PASSWORD=P@ssw0rd53
-POSTGRES_USER=dms
-POSTGRES_PASSWORD=P@ssw0rd
 POSTGRES_PORT=5432
 POSTGRES_HOST=localhost
 LOG_LEVEL=Information
 OAUTH_TOKEN_ENDPOINT=http://localhost:8080/oauth/token
 BYPASS_STRING_COERCION=false
+DATABASE_ISOLATION_LEVEL=RepeatableRead
+DATABASE_CONNECTION_STRING=host=localhost;port=5432;username=dms;password=P@ssw0rd;database=edfi_datamanagementservice;
 ```
 
 ## Orchestration

eng/kubernetes/development/app-configmap.yaml

@@ -12,3 +12,4 @@ data:
     log-level: "Information"
     oauth-token-endpoint: "http://localhost:8080/oauth/token"
     bypass-string-coercion: "false"
+    database-isolation-level: "RepeatableRead"

eng/kubernetes/development/app-secret.yaml.example

@@ -4,5 +4,5 @@ metadata:
     name: app-secret
 type: Opaque
 data:
-    postgres-password: dG9wc2VjdXJl
     postgres-admin-password: dG9wc2VjdXJl
+    database-connection-string: aG9zdD1kbXNkYjtwb3J0PTU0MzI7dXNlcm5hbWU9cG9zdGdyZXM7cGFzc3dvcmQ9dG9wc2VjdXJlO2RhdGFiYXNlPWVkZmlfZGF0YW1hbmFnZW1lbnRzZXJ2aWNlOw==

eng/kubernetes/development/data-management-service-deployment.yaml

@@ -25,16 +25,11 @@ spec:
                 configMapKeyRef:
                   name: app-configmap
                   key: need-db-setup
-            - name: POSTGRES_USER
-              valueFrom:
-                configMapKeyRef:
-                  name: app-configmap
-                  key: postgres-user
-            - name: POSTGRES_PASSWORD
+            - name: DATABASE_CONNECTION_STRING
               valueFrom:
                 secretKeyRef:
                   name: app-secret
-                  key: postgres-password
+                  key: database-connection-string
             - name: POSTGRES_ADMIN_USER
               valueFrom:
                 configMapKeyRef:
@@ -70,6 +65,11 @@ spec:
                 configMapKeyRef:
                   name: app-configmap
                   key: bypass-string-coercion
+            - name: DATABASE_ISOLATION_LEVEL
+              valueFrom:
+                configMapKeyRef:
+                  name: app-configmap
+                  key: database-isolation-level
           livenessProbe:
             exec:
               command:

eng/kubernetes/production/app-configmap.yaml

@@ -12,3 +12,4 @@ data:
     log-level: "Information"
     oauth-token-endpoint: "http://localhost:8080/oauth/token"
     bypass-string-coercion: "false"
+    database-isolation-level: "RepeatableRead"

eng/kubernetes/production/app-secret.yaml.example

@@ -4,5 +4,5 @@ metadata:
     name: app-secret
 type: Opaque
 data:
-    postgres-password: dG9wc2VjdXJl
     postgres-admin-password: dG9wc2VjdXJl
+    database-connection-string: aG9zdD1kbXNkYjtwb3J0PTU0MzI7dXNlcm5hbWU9cG9zdGdyZXM7cGFzc3dvcmQ9dG9wc2VjdXJlO2RhdGFiYXNlPWVkZmlfZGF0YW1hbmFnZW1lbnRzZXJ2aWNlOw==

eng/kubernetes/production/data-management-service-deployment.yaml

@@ -31,16 +31,11 @@ spec:
                 configMapKeyRef:
                   name: app-configmap
                   key: need-db-setup
-            - name: POSTGRES_USER
-              valueFrom:
-                configMapKeyRef:
-                  name: app-configmap
-                  key: postgres-user
-            - name: POSTGRES_PASSWORD
+            - name: DATABASE_CONNECTION_STRING
               valueFrom:
                 secretKeyRef:
                   name: app-secret
-                  key: postgres-password
+                  key: database-connection-string
             - name: POSTGRES_ADMIN_USER
               valueFrom:
                 configMapKeyRef:
@@ -76,6 +71,11 @@ spec:
                 configMapKeyRef:
                   name: app-configmap
                   key: bypass-string-coercion
+            - name: DATABASE_ISOLATION_LEVEL
+              valueFrom:
+                configMapKeyRef:
+                  name: app-configmap
+                  key: database-isolation-level
           livenessProbe:
             exec:
               command:

eng/kubernetes/production/postgres-deployment.yaml

@@ -22,12 +22,12 @@ spec:
         ports:
             - containerPort: 5432
         env:
-          - name: POSTGRES_USER
+          - name: POSTGRES_ADMIN_USER
             valueFrom:
               configMapKeyRef:
                 name: app-configmap
                 key: postgres-admin-user
-          - name: POSTGRES_PASSWORD
+          - name: POSTGRES_ADMIN_PASSWORD
             valueFrom:
               secretKeyRef:
                 name: app-secret

eng/postgresql-compose.yml

@@ -10,8 +10,8 @@ services:
     image: postgres:16.3-alpine
     restart: always
     environment:
-      - POSTGRES_USER=${POSTGRES_USER:-postgres}
-      - POSTGRES_PASSWORD=${POSTGRES_PASSWORD:-P@ssW0rd}
+      - POSTGRES_USER=${POSTGRES_ADMIN_USER:-postgres}
+      - POSTGRES_PASSWORD=${POSTGRES_ADMIN_PASSWORD:-P@ssW0rd}
     ports:
       - ${POSTGRES_PORT:-5432}:5432
     volumes:

src/.env.example

@@ -3,11 +3,11 @@
 ASPNETCORE_HTTP_PORTS=8080
 OAUTH_TOKEN_ENDPOINT=http://host.docker.internal:8000/oauth/token
 NEED_DATABASE_SETUP=true
+DATABASE_CONNECTION_STRING=localhost;port=5432;username=dms;password=P@ssw0rd;database=edfi_datamanagementservice;
 BYPASS_STRING_COERCION=false
 POSTGRES_ADMIN_USER=postgres
 POSTGRES_ADMIN_PASSWORD=ENTER PASSWORD
-POSTGRES_USER=postgres
-POSTGRES_PASSWORD=ENTER PASSWORD
 POSTGRES_PORT=5432
 POSTGRES_HOST=host.docker.internal
+DATABASE_ISOLATION_LEVEL=RepeatableRead
 LOG_LEVEL=DEBUG

src/Directory.Packages.props

@@ -16,6 +16,7 @@
     <PackageVersion Include="Microsoft.Extensions.Logging.Abstractions" Version="8.0.1" />
     <PackageVersion Include="Microsoft.Extensions.Logging.Console" Version="8.0.0" />
     <PackageVersion Include="Microsoft.Extensions.Logging.Debug" Version="8.0.0" />
+    <PackageVersion Include="Microsoft.Extensions.Options" Version="8.0.2" />
     <PackageVersion Include="Npgsql" Version="8.0.3" />
     <PackageVersion Include="Npgsql.DependencyInjection" Version="8.0.3" />
     <PackageVersion Include="Respawn" Version="6.2.1" />
@@ -50,4 +51,4 @@
     <PackageVersion Include="Reqnroll.NUnit" Version="2.0.3" />
     <PackageVersion Include="Testcontainers" Version="3.9.0" />
   </ItemGroup>
-</Project>
+</Project>

src/Dockerfile

@@ -39,12 +39,11 @@ ENV ASPNETCORE_HTTP_PORTS=8080
 ENV OAUTH_TOKEN_ENDPOINT=${OAUTH_TOKEN_ENDPOINT}
 ENV NEED_DATABASE_SETUP=${NEED_DATABASE_SETUP}
 ENV BYPASS_STRING_COERCION=${BYPASS_STRING_COERCION}
-
+ENV DATABASE_ISOLATION_LEVEL=${DATABASE_ISOLATION_LEVEL}
 ENV POSTGRES_ADMIN_USER=${POSTGRES_ADMIN_USER}
 ENV POSTGRES_ADMIN_PASSWORD=${POSTGRES_ADMIN_PASSWORD}
 
-ENV POSTGRES_USER=${POSTGRES_USER}
-ENV POSTGRES_PASSWORD=${POSTGRES_PASSWORD}
+ENV DATABASE_CONNECTION_STRING=${DATABASE_CONNECTION_STRING}
 
 ENV POSTGRES_PORT=${POSTGRES_PORT}
 ENV POSTGRES_HOST=${POSTGRES_HOST}

src/Nuget.Dockerfile

@@ -16,9 +16,9 @@ ARG VERSION=0.0.0
 ENV ASPNETCORE_HTTP_PORTS=8080
 ENV OAUTH_TOKEN_ENDPOINT=${OAUTH_TOKEN_ENDPOINT}
 ENV BYPASS_STRING_COERCION=${BYPASS_STRING_COERCION}
+ENV DATABASE_ISOLATION_LEVEL=${DATABASE_ISOLATION_LEVEL}
 
-ENV POSTGRES_USER=${POSTGRES_USER}
-ENV POSTGRES_PASSWORD=${POSTGRES_PASSWORD}
+ENV DATABASE_CONNECTION_STRING=${DATABASE_CONNECTION_STRING}
 
 ENV POSTGRES_PORT=${POSTGRES_PORT}
 ENV POSTGRES_HOST=${POSTGRES_HOST}

src/appsettings.template.json

@@ -6,7 +6,10 @@
         "BypassStringTypeCoercion": "${BYPASS_STRING_COERCION}"
     },
     "ConnectionStrings": {
-        "DatabaseConnection": "host=${POSTGRES_HOST};port=${POSTGRES_PORT};username=${POSTGRES_USER};password=${POSTGRES_PASSWORD};database=edfi_datamanagementservice;"
+        "DatabaseConnection": "${DATABASE_CONNECTION_STRING}"
+    },
+    "DatabaseOptions": {
+        "IsolationLevel": "${DATABASE_ISOLATION_LEVEL}"
     },
     "AllowedHosts": "*",
     "Serilog": {

src/backend/EdFi.DataManagementService.Backend.Postgresql.Test.Integration/Configuration.cs

@@ -4,6 +4,7 @@
 // See the LICENSE and NOTICES files in the project root for more information.
 
 
+using System.Data;
 using Microsoft.Extensions.Configuration;
 
 namespace EdFi.DataManagementService.Backend.Postgresql.Test.Integration
@@ -28,5 +29,8 @@ public static IConfiguration Config()
         }
 
         public static string? DatabaseConnectionString => Config().GetConnectionString("DatabaseConnection");
+
+        public static IsolationLevel IsolationLevel =>
+            Enum.Parse<IsolationLevel>(Config().GetSection("IsolationLevel").Value!);
     }
 }

src/backend/EdFi.DataManagementService.Backend.Postgresql.Test.Integration/DatabaseTest.cs

@@ -24,7 +24,7 @@ public abstract class DatabaseTest : DatabaseTestBase
     public async Task ConnectionSetup()
     {
         Connection = await DataSource!.OpenConnectionAsync();
-        Transaction = await Connection.BeginTransactionAsync(IsolationLevel.RepeatableRead);
+        Transaction = await Connection.BeginTransactionAsync(ConfiguredIsolationLevel);
     }
 
     [TearDown]
@@ -268,7 +268,7 @@ Func<NpgsqlConnection, NpgsqlTransaction, Task<U>> dbOperation2
 
         // Connection and transaction managed in this method for DB transaction 1
         await using var connection1 = await DataSource!.OpenConnectionAsync();
-        await using var transaction1 = await connection1.BeginTransactionAsync(IsolationLevel.RepeatableRead);
+        await using var transaction1 = await connection1.BeginTransactionAsync(ConfiguredIsolationLevel);
 
         // Use these for threads to signal each other for coordination
         using EventWaitHandle Transaction1Go = new AutoResetEvent(false);
@@ -289,7 +289,7 @@ Func<NpgsqlConnection, NpgsqlTransaction, Task<U>> dbOperation2
 
             // Step #3: Create new connection and begin DB transaction 2
             connection2 = await DataSource!.OpenConnectionAsync();
-            transaction2 = await connection2.BeginTransactionAsync(IsolationLevel.RepeatableRead);
+            transaction2 = await connection2.BeginTransactionAsync(ConfiguredIsolationLevel);
 
             // Step #4: Signal to transaction 1 thread to continue in parallel
             Transaction1Go?.Set();

src/backend/EdFi.DataManagementService.Backend.Postgresql.Test.Integration/DatabaseTestBase.cs

@@ -3,6 +3,7 @@
 // The Ed-Fi Alliance licenses this file to you under the Apache License, Version 2.0.
 // See the LICENSE and NOTICES files in the project root for more information.
 
+using System.Data;
 using Npgsql;
 using NUnit.Framework;
 using Respawn;
@@ -17,6 +18,7 @@ public abstract class DatabaseTestBase
     private Respawner? _respawner;
 
     public NpgsqlDataSource? DataSource { get; set; }
+    public static readonly IsolationLevel ConfiguredIsolationLevel = Configuration.IsolationLevel;
 
     [OneTimeSetUp]
     public void OneTimeSetup()

src/backend/EdFi.DataManagementService.Backend.Postgresql.Test.Integration/DeleteTests.cs

@@ -3,7 +3,6 @@
 // The Ed-Fi Alliance licenses this file to you under the Apache License, Version 2.0.
 // See the LICENSE and NOTICES files in the project root for more information.
 
-using System.Data;
 using EdFi.DataManagementService.Core.External.Backend;
 using FluentAssertions;
 using Npgsql;
@@ -419,7 +418,7 @@ public async Task Setup()
             _upsertResults.Add(await CreateUpsert().Upsert(upsertRequest, Connection!, Transaction!));
 
             await Transaction!.CommitAsync();
-            Transaction = await Connection!.BeginTransactionAsync(IsolationLevel.RepeatableRead);
+            Transaction = await Connection!.BeginTransactionAsync(ConfiguredIsolationLevel);
 
             _deleteResult = await CreateDeleteById()
                 .DeleteById(
@@ -494,7 +493,7 @@ public async Task Setup()
             _upsertResults.Add(await CreateUpsert().Upsert(upsertRequest, Connection!, Transaction!));
 
             await Transaction!.CommitAsync();
-            Transaction = await Connection!.BeginTransactionAsync(IsolationLevel.RepeatableRead);
+            Transaction = await Connection!.BeginTransactionAsync(ConfiguredIsolationLevel);
 
             _deleteResult = await CreateDeleteById()
                 .DeleteById(
@@ -568,7 +567,7 @@ public async Task Setup()
 
             await Transaction!.CommitAsync();
 
-            Transaction = await Connection!.BeginTransactionAsync(IsolationLevel.RepeatableRead);
+            Transaction = await Connection!.BeginTransactionAsync(ConfiguredIsolationLevel);
 
             _deleteResult = await CreateDeleteById()
                 .DeleteById(

src/backend/EdFi.DataManagementService.Backend.Postgresql.Test.Integration/appsettings.json

@@ -1,5 +1,6 @@
 {
     "ConnectionStrings": {
         "DatabaseConnection": "host=localhost;port=5432;username=postgres;database=edfi_datamanagementservice;pooling=true;minimum pool size=10;maximum pool size=50;Application Name=EdFi.DataManagementService"
-    }
+    },
+    "IsolationLevel": "RepeatableRead"
 }

src/backend/EdFi.DataManagementService.Backend.Postgresql/EdFi.DataManagementService.Backend.Postgresql.csproj

@@ -35,6 +35,7 @@
           <PrivateAssets>all</PrivateAssets>
       </PackageReference>
       <PackageReference Include="Microsoft.Extensions.Logging.Abstractions" />
+      <PackageReference Include="Microsoft.Extensions.Options" />
       <PackageReference Include="Npgsql" />
       <PackageReference Include="Npgsql.DependencyInjection" />
       <PackageReference Include="SonarAnalyzer.CSharp">