Reactions (#306)

* initial spec * simplify spec * simple trigger sequence * WIP reaction backend * working basic query * added trigger statement id * add ret spec * rudimentary test * reaction query takes whole trigger statement * introduce concept of identity * delint * remove test data since there is a test * allow reaction query with empty identity * comments on test return * implement LIKE * refactor and improve naming for col snip * use all denormalized fields * escape string keys in paths * Add a tiny sleep in between statements so timestamps are ordered * Implement and test contains * add numeric comp test * remove identity logic from query-reaction * make query input a single map * SQL-197 Reaction Persistence (#307) * SQL-197 input->ruleset * SQL-197 add reaction table * SQL-197 SQL-198 add persistence for reactions and fk on xapi_statement * SQL-197 a migration that actually works * use fake timestamp types by convention and add active/inactive/soft delete bool * ReactionQueryBackend -> ReactionBackend * SQL-197 CRUD queries/commands * SQL-197 stubbed out reaction values for statement insert * SQL-197 function for storing reaction exts info on statement * SQL-197 standardize spec ns a bit * SQL-197 more spec refactor * SQL-197 remove comment * SQL-197 add reaction and trigger id as metadata * SQL-197 inputs for reaction crud sql fns * SQL-197 test for add meta * SQL-197 specs for persistence ops * SQL-197 get rid of keywords in values and path segments * SQL-197 serde for reaction ruleset * SQL-197 ruleset commands * SQL-197 return active rulesets query * SQL-197 reaction CRUD protocol and test * SQL-197 remove forward declaration in spec * SQL-197 delint * SQL-197 use :as-alias and remove kondo exception * SQL-193: adds lrsql.ops.query.generate, which takes a map of key->statement and a statement template, returning a new statement based on template * SQL-198 Reaction Function (#312) * SQL-198 move low-level queries to util and stub out reaction fn * SQL-198 dedupe test reaction * SQL-198 use new test const in reaction util test * SQL-198 add template to ruleset * SQL-198 add instrumentation fixture, now some spec-violating tests fail * SQL-198 instrument and ensure valid specs in reaction util tests * SQL-198 instrument and fix specs for ops/util/reaction * SQL-198 type new id fields on statement * SQL-198 query-active-reactions test * SQL-198 query-all-reactions test * SQL-198 correct aliasing in reaction util query test ns * SQL-198 query-statement-for-reaction * SQL-198 query-reaction-history * SQL-198 remove query-statement-for-reaction * SQL-198 refactor some more test consts * SQL-198 formatting * SQL-198 actual reaction function * SQL-198 SQL-200 pass errors * SQL-200 Reaction TX Function & Errors (#315) * SQL-200 json-friendly error shape and slight refactor * SQL-200 add error column * SQL-200 add delete mod time * SQL-200 error-reaction! command * SQL-200 clear error on ruleset update * SQL-200 react-to-statement * SQL-200 SQL-198 SQL-193 Add authority to query output based on ruleset or trigger * SQL-200 delint * SQL-200 correct testing statement * SQL-200 react-to-statement error storage test * SQL-200 delint * SQL-200 derive custom authority from template * SQL-195 Async Reaction Processing (#316) * SQL-195 config toggle for enabling reactions * SQL-195 moved react-to-statement to new reactor system component * SQL-195 reaction buffer size param * SQL-195 correct default for buffer size * SQL-195 submit new statements to reaction channel * SQL-195 working async reaction processing * SQL-195 add shutdown hook for graceful shutdown * SQL-195 added note about what happens when reaction buffer is full * SQL-202 Implicit DB serialization (#318) * SQL-202 relax json spec to allow keywords and don't eagerly serialize ruleset * SQL-202 delint * SQL-202 allow keyword keys from json parse fn * SQL-202 remove unused sql blob JSON read fn * SQL-202 implicit json reading for ruleset and error * SQL-202 auto deserialize ids and timestamps for reaction * Coerce active to boolean at lower level * some operations working. failing some tests still * WIP eod commit json extraction. Working with basic types i think * SQL-196 Reaction Management API (#321) * SQL-196 Reaction Mgmt API * format * SQL-196 minor test refactor * SQL-196 additional tests * SQL-196 new json coercion fns for reaction * SQL-196 deal in camel reaction ids * SQL-196 API accepts camel everything * SQL-206 camelize identityPaths (#323) * SQL-204 Reaction Documentation (#322) * SQL-204 initial stub out docs * SQL-204 more detailed doc for reactions * SQL-204 added index links * query now working, but parsing in core is not, needs another approach * Minor optimization in query and slight refactor * move backend specific code for sqlite result parse * contains working with dynamic casting * accounting for jsonb * stmt react_id and trigger_id insert fix * fixed spec * test fixes * linting * universal extension native comparison * added another condition to make sure only statements prior to trigger are used, in case a batch insert or another thread inserts while trigger is waiting to be processed * fixed type issue and spec * SQL-211 provide reactions toggle to FE env (#333) * SQL-212 Reaction Title (#337) * SQL-212 add title * SQL-212 handle conflict errors on reaction titles * SQL-211 handle title-conflict-error * format * SQL-211 handle title conflict on update * SQL-211 use shared lib for ruleset + descendant specs * SQL-211 remove incomplete path, now forbidden by spec * SQL-211 make identityPaths fuzzier (#356) * SQL-211 make identityPaths fuzzier * SQL-211 please linter * SQL-211 fix keyword chopping bug (#360) * SQL-224 Reactions: encode condition names (#361) * SQL-224 encode condition names * SQL-224 slight test improvement * SQL-211 release of lrs-reactions * document like op * update template desc * crib cliffs tooltip for identity path doc * maps in function position for clarity * move timestamp coercions to ops ns * Reactions Refactoring (#362) * partial refactor of big reaction query op * further break down reaction q * further refactor and commenting * remove extra spaces * def'd out test-specific constants * remove extra spacing * replace more reaction test constants * removed ugly text to its own def * reaction doc example (#363) * release version of ui --------- Co-authored-by: Daniel Bell <[email protected]> Co-authored-by: Cliff Casey <[email protected]> Co-authored-by: Cliff Casey <[email protected]>
yetanalytics · Jan 23, 2024 · 113ffda · 113ffda
1 parent fad0428
commit 113ffda
Show file tree

Hide file tree

Showing 59 changed files with 3,607 additions and 298 deletions.
diff --git a/Makefile b/Makefile
@@ -2,7 +2,7 @@
 
 # Version of LRS Admin UI to use
 
-LRS_ADMIN_UI_VERSION ?= v0.1.12
+LRS_ADMIN_UI_VERSION ?= v0.1.14
 LRS_ADMIN_UI_LOCATION ?= https://github.com/yetanalytics/lrs-admin-ui/releases/download/${LRS_ADMIN_UI_VERSION}/lrs-admin-ui.zip
 LRS_ADMIN_ZIPFILE ?= lrs-admin-ui-${LRS_ADMIN_UI_VERSION}.zip
 

diff --git a/README.md b/README.md
@@ -41,6 +41,7 @@ For releases and release notes, see the [Releases](https://github.com/yetanalyti
 - [HTTP Endpoints](doc/endpoints.md)
 - [Developer Documentation](doc/dev.md)
 - [Example AWS Deployment](doc/aws.md)
+- [Reactions](doc/reactions.md)
 
 ### Demos
 

diff --git a/deps.edn b/deps.edn
@@ -5,6 +5,7 @@
   org.clojure/tools.logging                {:mvn/version "1.1.0"}
   org.clojure/core.memoize                 {:mvn/version "1.0.250"}
   clojure-interop/java.security            {:mvn/version "1.0.5"}
+  org.clojure/core.async                   {:mvn/version "1.6.681"}
   ;; Util deps
 
   camel-snake-kebab/camel-snake-kebab      {:mvn/version "0.4.2"}
@@ -53,7 +54,10 @@
   com.yetanalytics/pedestal-oidc
   {:mvn/version "0.0.8"
    :exclusions  [org.clojure/clojure
-                 buddy/buddy-sign]}}
+                 buddy/buddy-sign]}
+  com.yetanalytics/lrs-reactions
+  {:mvn/version "0.0.1"
+   :exclusions [org.clojure/clojure]}}
  :aliases
  {:db-sqlite
   {:extra-paths ["src/db/sqlite"]

diff --git a/doc/endpoints.md b/doc/endpoints.md
@@ -36,4 +36,13 @@ The following examples use `http://example.org` as the URL body. All methods ret
 - `GET http://example.org/admin/env`: Get select environment variables about the configuration which may aid in client-side operations. Currently returns a map containing the configuration variables `urlPrefix` and `enableStmtHtml`.
 - `DELETE http://example.org/admin/agents`: Runs a *hard delete* of all records of an actor, and associated records (statements, attachments, etc).  Intended for privacy purposes like GDPR.  Body should be a JSON object of form `{"actor-ifi":<actor-ifi>}`.  Disabled unless the configuration variable enableAdminDeleteActor to be set to `true`.
 
+### Reaction Management Routes
+
+If [Reactions](reactions.md) are enabled, the following routes can be used to manage them:
+
+- `POST http://example.org/admin/reaction`: Create a new reaction by providing a JSON `ruleset` and `active` boolean. On success returns 200 with the `reactionId` of the new reaction.
+- `GET http://example.org/admin/reaction`: List all reactions, active and inactive.
+- `PUT http://example.org/admin/reaction`: Given a `reactionId` and either a JSON `ruleset` or boolean `active` parameter, update the given reaction. On success returns 200 with the `reactionId` of the updated reaction. Returns 404 if the reaction is not found.
+- `DELETE http://example.org/admin/reaction`: Delete a reaction specified by `reactionId`. Returns 200 with the `reactionId` of the deleted reaction on success or 404 if not found.
+
 [<- Back to Index](index.md)
diff --git a/doc/env_vars.md b/doc/env_vars.md
@@ -107,6 +107,8 @@ The following options are used for advanced database performance tuning and may
 | `LRSQL_OIDC_SCOPE_PREFIX`       | `oidcScopePrefix`       | An optional prefix prepended to OIDC scope. For example, setting this to `lrs:` would change the expected `all` scope to `lrs:all`                                                                                | `""`                                                                                  |
 | `LRSQL_STMT_RETRY_LIMIT`        | `stmtRetryLimit`        | The number of times to retry a statement post transaction before failing.                                                                                                                                         | `10`                                                                                  |
 | `LRSQL_STMT_RETRY_BUDGET`       | `stmtRetryBudget`       | The max amount of time allowed for statement POST transaction retries before failing (ms).                                                                                                                        | `1000`                                                                                |
+| `LRSQL_ENABLE_REACTIONS`        | `enableReactions`       | Whether or not to enable statement reactions.                                                                                                                                                                     | `false`                                                                               |
+| `LRSQL_REACTION_BUFFER_SIZE`    | `reactionBufferSize`    | Number of pending reactions to allow. Additional reactions will be dropped with a warning message.                                                                                                                | `10000`                                                                               |
 
 _NOTE:_ `LRSQL_STMT_RETRY_LIMIT` and `LRSQL_STMT_RETRY_BUDGET` are used to mitigate a rare scenario where specific Actors or Activities are updated many times in large concurrent batches. In this situation the DBMS can encounter locking and these settings are used to allow retries that eventually write all the conflicting transactions, but may incur performance degradation. If you are experiencing this situation the first step would be to look at why your data needs to rewrite specific Actors or Activities rapidly with different values, which could potentially solve it at the source. If the issue cannot be avoided by data design alone, another possible solution is reducing batch sizes to decrease or eliminate locks. As a last resort, increasing these settings will at least ensure the statements get written but as mentioned may incur a slowdown in concurrent throughput.
 

diff --git a/doc/index.md b/doc/index.md
@@ -22,6 +22,7 @@
 - [HTTP Endpoints](endpoints.md)
 - [Developer Documentation](dev.md)
 - [Example AWS Deployment](aws.md)
+- [Reactions](reactions.md)
 
 ### Releases
 

diff --git a/doc/reactions.md b/doc/reactions.md
@@ -0,0 +1,232 @@
+[<- Back to Index](index.md)
+
+# Reactions
+
+Reactions allow SQL LRS to watch for patterns in submitted xAPI data and dynamically generate new statements in response.
+
+## Usage
+
+To use Reactions the `LRSQL_ENABLE_REACTIONS` environment variable or the `enableReactions` LRS configuration property must be set to `true`. Reactions are disabled by default.
+
+Reaction "rulesets" are defined in JSON:
+
+``` json
+
+{
+  "identityPaths": [
+    [
+      "actor",
+      "mbox"
+    ],
+    [
+      "actor",
+      "mbox_sha1sum"
+    ],
+    [
+      "actor",
+      "openid"
+    ],
+    [
+      "actor",
+      "account",
+      "homePage"
+    ],
+    [
+      "actor",
+      "account",
+      "name"
+    ]
+  ],
+  "conditions": {
+    "a": {
+      "and": [
+        {
+          "path": [
+            "object",
+            "id"
+          ],
+          "op": "eq",
+          "val": "https://example.com/activities/a"
+        },
+        {
+          "path": [
+            "verb",
+            "id"
+          ],
+          "op": "eq",
+          "val": "https://example.com/verbs/completed"
+        },
+        {
+          "path": [
+            "result",
+            "success"
+          ],
+          "op": "eq",
+          "val": true
+        }
+      ]
+    },
+    "b": {
+      "and": [
+        {
+          "path": [
+            "object",
+            "id"
+          ],
+          "op": "eq",
+          "val": "https://example.com/activities/b"
+        },
+        {
+          "path": [
+            "verb",
+            "id"
+          ],
+          "op": "eq",
+          "val": "https://example.com/verbs/completed"
+        },
+        {
+          "path": [
+            "result",
+            "success"
+          ],
+          "op": "eq",
+          "val": true
+        },
+        {
+          "path": [
+            "timestamp"
+          ],
+          "op": "gt",
+          "ref": {
+            "condition": "a",
+            "path": [
+              "timestamp"
+            ]
+          }
+        }
+      ]
+    }
+  },
+  "template": {
+    "actor": {
+      "mbox": {
+        "$templatePath": [
+          "a",
+          "actor",
+          "mbox"
+        ]
+      }
+    },
+    "verb": {
+      "id": "https://example.com/verbs/completed"
+    },
+    "object": {
+      "id": "https://example.com/activities/a-and-b",
+      "objectType": "Activity"
+    }
+  }
+}
+
+
+```
+
+### Identity Paths
+
+Identity Paths (`identityPaths` in the ruleset JSON) are a method of grouping statements for which you are attempting to match conditions. Typically, Reactions may revolve around actor Inverse Functional Identifiers (IFIs), e.g. `["actor", "mbox"]` or `["actor", "account", "name"]` which is equivalent to saying "For a given Actor, look for statements that share IFI values".
+
+Alternative approaches to Identity Path may be used by modifying `identityPaths`, for instance `["context", "registration"]` to group statements by learning session.
+
+### Conditions
+
+`conditions` is a mapping of names to rules for finding significant statements. Rules can be composed with boolean logic.
+
+In the example given above statement `a` must have an object id equal to `https://example.com/activities/a`, a verb id equal to `https://example.com/verbs/completed`, and a result success equal to `true`. Statement `b` must have the same verb and result success but an object id equal to `https://example.com/activities/b` and a timestamp greater than that of `a`.
+
+#### Rules
+
+All rules have a `path` array that indicates a path in an xAPI statement and an `op` that is one of the following operators:
+
+* `gt` - Greater than
+* `lt` - Less than
+* `gte` - Greater than or equal
+* `lte` - Less than or equal
+* `eq` - Equal
+* `noteq` - Not equal
+* `like` - Fuzzy match using SQL `%` syntax. For example, `bo%` matches `bob` and `boz`.
+* `contains` - Array contains
+
+Rules either have a `val` literal value or a `ref` which is a path into a statement found for another condition.
+
+#### Booleans
+
+Booleans compose multiple rules together. Booleans are objects with a single key:
+
+* `and` - Array of rules which must all be true
+* `or` - Array of rules of which one must be true
+* `not` - Rule to nullify
+
+### Template
+
+`template` describes the xAPI statement the reaction will produce. It is identical to an xAPI statement, except that object properties may be substituted with `$templatePath`. This is a path that points to a value in a statement matched by `conditions`, using the same syntax as an `identityPaths` path. In the above example, the `$templatePath` points to the actor `mbox` for the actor matched by condition `a`.
+
+## Example
+
+Given the reaction specified above, if the following statements are posted to the LRS:
+
+``` json
+[
+  {
+    "actor": {
+      "mbox": "mailto:[email protected]"
+    },
+    "verb": {
+      "id": "https://example.com/verbs/completed"
+    },
+    "object": {
+      "id": "https://example.com/activities/a",
+      "objectType": "Activity"
+    },
+    "result": {
+      "success": true
+    },
+    "timestamp": "2024-01-23T01:00:00.000Z"
+  },
+  {
+    "actor": {
+      "mbox": "mailto:[email protected]"
+    },
+    "verb": {
+      "id": "https://example.com/verbs/completed"
+    },
+    "object": {
+      "id": "https://example.com/activities/b",
+      "objectType": "Activity"
+    },
+    "result": {
+      "success": true
+    },
+    "timestamp": "2024-01-23T02:00:00.000Z"
+  }
+]
+
+```
+
+Then the following statement will be added subsequently (note that some unrelated fields are removed for clarity):
+
+``` json
+{
+  "actor": {
+    "mbox": "mailto:[email protected]"
+  },
+  "verb": {
+    "id": "https://example.com/verbs/completed"
+  },
+  "object": {
+    "id": "https://example.com/activities/a-and-b",
+    "objectType": "Activity"
+  }
+}
+
+```
+
+[<- Back to Index](index.md)
diff --git a/resources/lrsql/config/prod/default/lrs.edn b/resources/lrsql/config/prod/default/lrs.edn
@@ -10,4 +10,6 @@
  :oidc-authority-template #or [#env LRSQL_OIDC_AUTHORITY_TEMPLATE "config/oidc_authority.json.template"]
  :oidc-scope-prefix       #or [#env LRSQL_OIDC_SCOPE_PREFIX ""]
  :stmt-retry-limit        #or [#env LRSQL_STMT_RETRY_LIMIT 10]
- :stmt-retry-budget       #or [#env LRSQL_STMT_RETRY_BUDGET 1000]}
+ :stmt-retry-budget       #or [#env LRSQL_STMT_RETRY_BUDGET 1000]
+ :enable-reactions        #boolean #or [#env LRSQL_ENABLE_REACTIONS false]
+ :reaction-buffer-size    #long #or [#env LRSQL_REACTION_BUFFER_SIZE 10000]}
diff --git a/resources/lrsql/config/test/default/lrs.edn b/resources/lrsql/config/test/default/lrs.edn
@@ -10,4 +10,6 @@
  :oidc-authority-template "config/oidc_authority.json.template"
  :oidc-scope-prefix       ""
  :stmt-retry-limit        20
- :stmt-retry-budget       10000}
+ :stmt-retry-budget       10000
+ :enable-reactions        true
+ :reaction-buffer-size    10000}
diff --git a/src/db/postgres/lrsql/postgres/data.clj b/src/db/postgres/lrsql/postgres/data.clj
@@ -4,7 +4,7 @@
             [lrsql.util :as u])
   (:import [clojure.lang IPersistentMap]
            [org.postgresql.util PGobject]
-           [java.sql PreparedStatement]))
+           [java.sql PreparedStatement ResultSetMetaData]))
 
 ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
 ;; PGObject
@@ -17,11 +17,11 @@
     (.setValue (u/write-json-str jsn))))
 
 (defn- pg-object->json
-  [^PGobject pg-obj]
+  [kw-labels label ^PGobject pg-obj]
   (let [type  (.getType pg-obj)
         value (.getValue pg-obj)]
     (if (#{"jsonb" "json"} type)
-      (u/parse-json value)
+      (u/parse-json value :keyword-keys? (some? (kw-labels label)))
       (throw (ex-info "Invalid PostgreSQL JSON type"
                       {:type       ::invalid-postgres-json
                        :json-type  type
@@ -32,13 +32,13 @@
 ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
 
 (defn set-read-pgobject->json!
-  []
+  [kw-labels]
   (extend-protocol ReadableColumn
     PGobject
-    (read-column-by-label [^PGobject pg-obj _]
-      (pg-object->json pg-obj))
-    (read-column-by-index [^PGobject pg-obj _2 _3]
-      (pg-object->json pg-obj))))
+    (read-column-by-label [^PGobject pg-obj ^String label]
+      (pg-object->json kw-labels label pg-obj))
+    (read-column-by-index [^PGobject pg-obj ^ResultSetMetaData rsmeta ^long i]
+      (pg-object->json kw-labels (.getColumnLabel rsmeta i) pg-obj))))
 
 ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
 ;; Write
@@ -59,3 +59,14 @@
   "Returns a properly formatted hug input map to inject a timezone id into a
   query needing a timezone id"
   {:tz-id (str "'" u/local-zone-id "'")})
+
+;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
+;; JSON Field Coercion
+;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
+
+(def type->pg-type
+  {:bool   "BOOLEAN"
+   :int    "INTEGER"
+   :dec    "DECIMAL"
+   :string "TEXT"
+   :json   "JSONB"})
diff --git a/src/db/postgres/lrsql/postgres/main.clj b/src/db/postgres/lrsql/postgres/main.clj
@@ -1,6 +1,7 @@
 (ns lrsql.postgres.main
   (:require [com.stuartsierra.component :as component]
             [lrsql.system :as system]
+            [lrsql.system.util :as su]
             [lrsql.postgres.record :as pr])
   (:gen-class))
 
@@ -10,8 +11,11 @@
   "Run a Postgres-backed LRSQL instance based on the `:test-postgres`
    config profile. For use with `clojure -X:db-postgres`."
   [_] ; Need to pass in a map for -X
-  (component/start (system/system postgres-backend :test-postgres)))
+  (-> (system/system postgres-backend :test-postgres)
+      component/start
+      su/add-shutdown-hook!))
 
 (defn -main [& _args]
   (-> (system/system postgres-backend :prod-postgres)
-      component/start))
+      component/start
+      su/add-shutdown-hook!))