[SQL-264] JWT and Routing Enhancement Suite

doc/dev.md

@@ -4,6 +4,15 @@
 
 The SQL LRS is a Clojure Web Application built on the Pedestal Framework.
 
+### Testing
+
+Development is primarily test-driven, which uses an exhaustive suite of unit tests. To run them locally, run a `make test-[database]` command. In addition, all tests are run for all versions in GitHub Actions CI.
+
+However, in some situations, such as UI development, relying on the unit tests may be inadequate. In these cases, in addition to performing visual tests on the UI, one may need to test these specific scenarios:
+- Login with OIDC ([demo](oidc.md#keycloak-demo))
+- Proxy paths ([demo](other_demos.md#proxied-lrs-demo))
+- JWT override ([JWT config vars](env_vars.md#jwt-config))
+
 ### Build
 
 The SQL LRS can be built or run with the following Makefile targets. They can be executed with `make [target]`.

doc/endpoints.md

@@ -27,10 +27,13 @@ The following examples use `http://example.org` as the URL body. All methods ret
   * `password` must contain at least one of the following special characters: `!@#$%^&*_-+=?`.
 
 The response body contains a newly generated JSON Web Token (JWT) on success. A `401 UNAUTHORIZED` status code is returned if the credentials are incorrect.
+- `POST http://example.org/admin/account/logout`: Log out of the current account. This will revoke any unexpired JWTs associated with the user. (NOTE: This endpoint will return a `400 BAD REQUEST` error if `LRSQL_JWT_NO_VAL` is set to `true`.)
+- `GET http://example.org/admin/account/renew`: Renew the current account's login session by issuing a new JWT. For a given JWT, the renewal is only granted if the current time is less than the `ref` timestamp (which is determined by `LRSQL_JWT_REFRESH_EXP_TIME`).
 - `POST http://example.org/admin/account/create`: Create a new admin account. The request body must be a JSON object that contains `username` and `password` strings. The endpoint returns a JSON object with the ID (UUID) of the newly created user on success, and returns a `409 CONFLICT` if the account already exists.
 - `DELETE http://example.org/admin/account`: Delete an existing account. The JSON request body must contain a UUID `account-id` value. The endpoint returns a JSON object with the ID of the deleted account on success and returns a `404 NOT FOUND` error if the account does not exist.
 - `GET http://example.org/admin/account`: Return an array of all admin accounts in the system on success.
 - `GET http://example.org/admin/me`: Returns the currently authenticated admin accounts on success.
+- `GET http://example.org/admin/verify`: Returns a `204 No Content` response, without a body, on success (the success conditions are the same as the `/admin/me` endpoint).
 
 #### Admin Credential Routes
 
@@ -42,6 +45,8 @@ The response body contains a newly generated JSON Web Token (JWT) on success. A
 #### Misc Admin Routes
 
 - `GET http://example.org/admin/env`: Get select environment variables about the configuration which may aid in client-side operations.
+- `GET http://example.org/admin/openapi`: Get an OpenAPI JSON spec of the endpoint API, which can then be visualized using an OpenAPI viewer like Swagger.
+- `GET http://example.org/admin/status`: Get LRS status information, such as the number of statements in the LRS.
 - `DELETE http://example.org/admin/agents`: Runs a *hard delete* of all records of an actor, and associated records (statements, attachments, etc).  Intended for privacy purposes like GDPR.  Body should be a JSON object of form `{"actor-ifi":<actor-ifi>}`.  Disabled unless the configuration variable enableAdminDeleteActor to be set to `true`.
 
 ### Reaction Management Routes

doc/env_vars.md

@@ -123,7 +123,7 @@ _NOTE:_ `LRSQL_STMT_RETRY_LIMIT` and `LRSQL_STMT_RETRY_BUDGET` are used to mitig
 | `LRSQL_HTTP_HOST`                 | `httpHost`               | The host that the webserver will run on.                                                                                                                                                                                                               | `0.0.0.0` |
 | `LRSQL_HTTP_PORT`                 | `httpPort`               | The HTTP port that the webserver will be open on.                                                                                                                                                                                                      | `8080`    |
 | `LRSQL_SSL_PORT`                  | `sslPort`                | The HTTPS port that the webserver will be open on.                                                                                                                                                                                                     | `8443`    |
-| `LRSQL_URL_PREFIX`                | `urlPrefix`              | The prefix of the webserver URL path, e.g. the prefix in `http://0.0.0.0:8080/xapi` is `/xapi`. Used when constructing the `more` value for multi-statement queries. *(Note: Only applies to LRS xapi endpoints, not admin/ui endpoints)*              | `/xapi`   |
+| `LRSQL_URL_PREFIX`                | `urlPrefix`              | The prefix of the webserver URL path, e.g. the prefix in `http://0.0.0.0:8080/xapi` is `/xapi`. Used when constructing the `more` value for multi-statement queries. Cannot start with `/admin`. *(Note: Only applies to LRS xapi endpoints, not admin/ui endpoints)*              | `/xapi`   |
 | `LRSQL_PROXY_PATH`                | `proxyPath`              | This path modification is exclusively for use with a proxy, such as apache or nginx or a load balancer, where a path is added to prefix the entire application (such as `https://www.mysystem.com/mylrs/xapi/statements`). This does not actually change the routes of the application, it informs the admin frontend where to look for the server endpoints based on the proxied setup, and thus must be used in conjunction with a third party proxy. If used, the value must start with a leading `/` but not end with one (e.g. `/mylrs` is valid, as is `/mylrs/b` but `/mylrs/` is not). Use with caution. | Not Set |
 
 #### TLS/SSL Certificate
@@ -141,8 +141,11 @@ _NOTE:_ `LRSQL_STMT_RETRY_LIMIT` and `LRSQL_STMT_RETRY_BUDGET` are used to mitig
 
 | Env Var                           | Config                   | Description                                                                                                                                                                                   | Default           |
 | --------------------------------- | ------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------- |
-| `LRSQL_JWT_EXP_TIME`              | `jwtExpTime`             | The amount of time, in seconds, after a JWT is created when it expires. Since JWTs are not revocable, **this this time should be short** (i.e. one hour or less).                             | `3600` (one hour) |
+| `LRSQL_JWT_EXP_TIME`              | `jwtExpTime`             | The amount of time, in seconds, after a JWT is created when it expires. **This this time should be short** (i.e. one hour or less).                             | `3600` (one hour) |
 | `LRSQL_JWT_EXP_LEEWAY`            | `jwtExpLeeway`           | The amount of time, in seconds, before or after the expiration instant when a JWT should still count as un-expired. Used to compensate for clock desync. Applied to both LRS and OIDC tokens. | `1` (one second)  |
+| `LRSQL_JWT_REFRESH_EXP_TIME`      | `jwtRefreshExpTime`      | The amount of time, in seconds, after a JWT is issued upon an initial login (_not_ a login renewal), after which login renewal can no longer be performed. Note: this is unaffected by expiration leeway. | `86400` (one day) |
+| `LRSQL_JWT_REFRESH_INTERVAL`      | `jwtRefreshInterval`     | The amount of time that the client should poll the server in order to refresh the JWT. This **must** be less than `LRSQL_JWT_EXP_TIME`. | `3540` (59 minutes) |
+| `LRSQL_JWT_INTERACTION_WINDOW`    | `jwtInteractionWindow`   | The amount of time before a potential JWT refresh that the client checks for interaction. This **must** be less than or equal to `LRSQL_JWT_REFRESH_INTERVAL`. | `600` (10 minutes) |
 | `LRSQL_JWT_NO_VAL`                | `jwtNoVal`               | (**DANGEROUS!**) This flag removes JWT Token Validation and simply accepts token claims as configured by the associated variables below. It is extremely unlikely that you need this as it is for very specific proxy-overwrite authentication scenarios, and it poses a serious threat to system security if enabled. | `false` |
 | `LRSQL_JWT_NO_VAL_UNAME`          | `jwtNoValUname`          | (**DANGEROUS!** See `LRSQL_JWT_NO_VAL`) This variable configures which claim key to use for the username when token validation is turned off.                     | Not Set |
 | `LRSQL_JWT_NO_VAL_ISSUER`         | `jwtNoValIssuer`         | (**DANGEROUS!** See `LRSQL_JWT_NO_VAL`) This variable configures which claim key to use for the issuer when token validation is turned off.                       | Not Set |

resources/lrsql/config/prod/default/webserver.edn

@@ -6,6 +6,9 @@
  :key-enable-selfie         #boolean #or [#env LRSQL_KEY_ENABLE_SELFIE true]
  :jwt-exp-time              #long #or [#env LRSQL_JWT_EXP_TIME 3600]
  :jwt-exp-leeway            #long #or [#env LRSQL_JWT_EXP_LEEWAY 1]
+ :jwt-refresh-exp-time      #long #or [#env LRSQL_JWT_REFRESH_EXP_TIME 86400]
+ :jwt-refresh-interval      #long #or [#env LRSQL_JWT_REFRESH_INTERVAL 3540]
+ :jwt-interaction-window    #long #or [#env LRSQL_JWT_INTERACTION_WINDOW 600]
  :jwt-no-val                #boolean #or [#env LRSQL_JWT_NO_VAL false]
  :jwt-no-val-uname          #or [#env LRSQL_JWT_NO_VAL_UNAME nil]
  :jwt-no-val-issuer         #or [#env LRSQL_JWT_NO_VAL_ISSUER nil]

resources/lrsql/config/test/default/webserver.edn

@@ -4,6 +4,9 @@
  :key-enable-selfie         true
  :jwt-exp-time              3600
  :jwt-exp-leeway            1
+ :jwt-refresh-exp-time      86400
+ :jwt-refresh-interval      3540
+ :jwt-interaction-window    600
  :jwt-no-val                false
  :jwt-no-val-uname          nil
  :jwt-no-val-issuer         nil

src/db/postgres/lrsql/postgres/record.clj

@@ -76,7 +76,8 @@
     (when (nil? (check-statement-to-actor-cascading-delete tx))
       (add-statement-to-actor-cascading-delete! tx))
     (when (some? (query-varchar-exists tx))
-      (convert-varchars-to-text! tx)))
+      (convert-varchars-to-text! tx))
+    (create-blocked-jwt-table! tx))
 
   bp/BackendUtil
   (-txn-retry? [_ ex]
@@ -197,6 +198,14 @@
   (-query-account-count-local [_ tx]
     (query-account-count-local tx))
 
+  bp/JWTBlocklistBackend
+  (-insert-blocked-jwt! [_ tx input]
+    (insert-blocked-jwt! tx input))
+  (-delete-blocked-jwt-by-time! [_ tx input]
+    (delete-blocked-jwt-by-time! tx input))
+  (-query-blocked-jwt [_ tx input]
+    (query-blocked-jwt-exists tx input))
+
   bp/CredentialBackend
   (-insert-credential! [_ tx input]
     (insert-credential! tx input))

src/db/postgres/lrsql/postgres/sql/ddl.sql

@@ -488,3 +488,14 @@ ALTER TABLE lrs_credential ALTER COLUMN secret_key TYPE TEXT;
 ALTER TABLE credential_to_scope ALTER COLUMN api_key TYPE TEXT;
 ALTER TABLE credential_to_scope ALTER COLUMN secret_key TYPE TEXT;
 ALTER TABLE reaction ALTER COLUMN title TYPE TEXT;
+
+/* Migration 2024-10-31 - Add JWT Blocklist Table */
+
+-- :name create-blocked-jwt-table!
+-- :command :execute
+-- :doc Create the `blocked_jwt` table and associated indexes if they do not exist yet.
+CREATE TABLE IF NOT EXISTS blocked_jwt (
+  jwt        TEXT PRIMARY KEY,
+  evict_time TIMESTAMP WITH TIME ZONE
+);
+CREATE INDEX IF NOT EXISTS blocked_jwt_evict_time_idx ON blocked_jwt(evict_time);

src/db/postgres/lrsql/postgres/sql/delete.sql

@@ -108,3 +108,12 @@ DELETE FROM state_document WHERE agent_ifi = :actor-ifi;
 
 DELETE FROM actor WHERE actor_ifi = :actor-ifi;
 ------------------end delete-actor--------------------
+
+/* JWT Blocklist */
+
+-- :name delete-blocked-jwt-by-time!
+-- :command :execute
+-- :result :affected
+-- :doc Delete all blocked JWTs where `:current-time` is past the eviction time.
+DELETE FROM blocked_jwt
+WHERE evict_time <= :current-time;

src/db/postgres/lrsql/postgres/sql/insert.sql

@@ -159,3 +159,15 @@ INSERT INTO reaction (
 ) VALUES (
   :primary-key, :title, :ruleset, :active, :created, :modified
 );
+
+/* JWT Blocklist */
+
+-- :name insert-blocked-jwt!
+-- :command :insert
+-- :result :affected
+-- :doc Insert a `:jwt` and a `:eviction-time` into the blocklist.
+INSERT INTO blocked_jwt (
+  jwt, evict_time
+) VALUES (
+  :jwt, :eviction-time
+);

src/db/postgres/lrsql/postgres/sql/query.sql

@@ -427,3 +427,12 @@ WITH RECURSIVE trigger_history (statement_id, reaction_id, trigger_id) AS (
 SELECT reaction_id
 FROM trigger_history
 WHERE reaction_id IS NOT NULL;
+
+/* JWT Blocklist */
+
+-- :name query-blocked-jwt-exists
+-- :command :query
+-- :result :one
+-- :doc Query that `:jwt` is in the blocklist.
+SELECT 1 FROM blocked_jwt
+WHERE jwt = :jwt;

src/db/sqlite/lrsql/sqlite/record.clj

@@ -112,6 +112,8 @@
       (xapi-statement-add-trigger-id! tx))
     (when-not (some? (query-statement-to-actor-has-cascade-delete tx))
       (update-schema-simple! tx alter-statement-to-actor-add-cascade-delete!))
+    (create-blocked-jwt-table! tx)
+    (create-blocked-jwt-evict-time-idx! tx)
     (log/infof "sqlite schema_version: %d"
                (:schema_version (query-schema-version tx))))
 
@@ -237,6 +239,14 @@
   (-query-account-count-local [_ tx]
     (query-account-count-local tx))
 
+  bp/JWTBlocklistBackend
+  (-insert-blocked-jwt! [_ tx input]
+    (insert-blocked-jwt! tx input))
+  (-delete-blocked-jwt-by-time! [_ tx input]
+    (delete-blocked-jwt-by-time! tx input))
+  (-query-blocked-jwt [_ tx input]
+    (query-blocked-jwt-exists tx input))
+
   bp/CredentialBackend
   (-insert-credential! [_ tx input]
     (insert-credential! tx input))

src/db/sqlite/lrsql/sqlite/sql/ddl.sql

@@ -149,8 +149,6 @@ CREATE INDEX IF NOT EXISTS sts_ancestor_id_idx ON statement_to_statement(ancesto
 -- :doc Create an index on the `statement_to_statement.descendant_id` column.
 CREATE INDEX IF NOT EXISTS sts_descendant_id_idx ON statement_to_statement(descendant_id)
 
-
-
 /* Document Tables */
 
 -- :name create-state-document-table!
@@ -524,3 +522,19 @@ SET sql = 'CREATE TABLE statement_to_actor (
   FOREIGN KEY (actor_ifi, actor_type) REFERENCES actor(actor_ifi, actor_type)
 )'
 WHERE type = 'table' AND name = 'statement_to_actor'
+;
+
+/* Migration 2024-10-31 - Add JWT Blocklist Table */
+
+-- :name create-blocked-jwt-table!
+-- :command :execute
+-- :doc Create the `blocked_jwt` table if it does not exist yet.
+CREATE TABLE IF NOT EXISTS blocked_jwt (
+  jwt        TEXT PRIMARY KEY,
+  evict_time TIMESTAMP
+);
+
+-- :name create-blocked-jwt-evict-time-idx!
+-- :command :execute
+-- :doc Create the `blocked_jwt_evict_time_idx` table if it does not exist yet.
+CREATE INDEX IF NOT EXISTS blocked_jwt_evict_time_idx ON blocked_jwt(evict_time);

src/db/sqlite/lrsql/sqlite/sql/delete.sql

@@ -123,3 +123,12 @@ DELETE FROM state_document WHERE agent_ifi = :actor-ifi
 -- :command :execute
 -- :result :affected
 DELETE FROM actor WHERE actor_ifi = :actor-ifi
+
+/* JWT Blocklist */
+
+-- :name delete-blocked-jwt-by-time!
+-- :command :execute
+-- :result :affected
+-- :doc Delete all blocked JWTs where `:current-time` is past the eviction time.
+DELETE FROM blocked_jwt
+WHERE evict_time <= :current-time;

src/db/sqlite/lrsql/sqlite/sql/insert.sql

@@ -159,3 +159,15 @@ INSERT INTO reaction (
 ) VALUES (
   :primary-key, :title, :ruleset, :active, :created, :modified
 );
+
+/* JWT Blocklist */
+
+-- :name insert-blocked-jwt!
+-- :command :insert
+-- :result :affected
+-- :doc Insert a `:jwt` and a `:eviction-time` into the blocklist.
+INSERT INTO blocked_jwt (
+  jwt, evict_time
+) VALUES (
+  :jwt, :eviction-time
+);

src/db/sqlite/lrsql/sqlite/sql/query.sql

@@ -395,3 +395,12 @@ WITH RECURSIVE trigger_history (statement_id, reaction_id, trigger_id) AS (
 SELECT reaction_id
 FROM trigger_history
 WHERE reaction_id IS NOT NULL;
+
+/* JWT Blocklist */
+
+-- :name query-blocked-jwt-exists
+-- :command :query
+-- :result :one
+-- :doc Query that `:jwt` is in the blocklist.
+SELECT 1 FROM blocked_jwt
+WHERE jwt = :jwt

src/main/lrsql/admin/interceptors/account.clj

@@ -1,5 +1,6 @@
 (ns lrsql.admin.interceptors.account
   (:require [clojure.spec.alpha :as s]
+            [java-time.api :as jt]
             [io.pedestal.interceptor :refer [interceptor]]
             [io.pedestal.interceptor.chain :as chain]
             [lrsql.admin.protocol :as adp]
@@ -250,19 +251,80 @@
                :response
                {:status 200 :body result})))}))
 
+(def no-content
+  "Return a 204 No Content response, without a body."
+  (interceptor
+   {:name ::get-no-content
+    :enter
+    (fn get-account [ctx]
+      (assoc ctx :response {:status 204}))}))
+
+;; JWT interceptors for admin
+
 (defn generate-jwt
   "Upon account login, generate a new JSON web token."
-  [secret exp]
+  [secret exp ref leeway]
   (interceptor
    {:name ::generate-jwt
     :enter
     (fn generate-jwt [ctx]
-      (let [{{:keys [account-id]} ::data}
+      (let [{lrs :com.yetanalytics/lrs
+             {:keys [account-id]} ::data}
             ctx
             json-web-token
-            (admin-u/account-id->jwt account-id secret exp)]
+            (admin-u/account-id->jwt account-id secret exp ref)]
+        (adp/-purge-blocklist lrs leeway) ; Update blocklist upon login
         (assoc ctx
                :response
                {:status 200
                 :body   {:account-id     account-id
                          :json-web-token json-web-token}})))}))
+
+(defn renew-admin-jwt
+  [secret exp]
+  (interceptor
+   {:name ::renew-jwt
+    :enter
+    (fn renew-jwt [ctx]
+      (let [{{:keys [account-id refresh-exp]} ::jwt/data} ctx
+            curr-time (u/current-time)]
+        (if (jt/before? curr-time refresh-exp)
+          (let [json-web-token (admin-u/account-id->jwt* account-id
+                                                         secret
+                                                         exp
+                                                         refresh-exp)]
+            (assoc ctx
+                   :response
+                   {:status 200
+                    :body   {:account-id     account-id
+                             :json-web-token json-web-token}}))
+          (assoc (chain/terminate ctx)
+                 :response
+                 {:status 401
+                  :body   {:error "Attempting JWT login after refresh expiry."}}))))}))
+
+(def ^:private block-admin-jwt-error-msg
+  "This operation is unsupported when `LRSQL_JWT_NO_VAL` is set to `true`.")
+
+(defn block-admin-jwt
+  "Add the current JWT to the blocklist. Return an error if we are in
+   no-val mode."
+  [exp leeway no-val?]
+  (interceptor
+   {:name ::add-jwt-to-blocklist
+    :enter
+    (fn add-jwt-to-blocklist [ctx]
+      (if-not no-val?
+        (let [{lrs :com.yetanalytics/lrs
+               {:keys [jwt account-id]} :lrsql.admin.interceptors.jwt/data}
+              ctx]
+          (adp/-purge-blocklist lrs leeway) ; Update blocklist upon logout
+          (adp/-block-jwt lrs jwt exp)
+          (assoc (chain/terminate ctx)
+                 :response
+                 {:status 200
+                  :body   {:account-id account-id}}))
+        (assoc (chain/terminate ctx)
+               :response
+               {:status 400
+                :body   {:error block-admin-jwt-error-msg}})))}))