Automated Action/Snapshot testing framework

[CarlosNZ]

NOTE: This includes changes from #922 (as I needed the functionality), so probably wait till that's merged to review this.

Okay, I've got the basics of this working pretty well, I think it will be worth adding detailed test scripts to our core snapshots (Angola, Fiji, etc.)

How it works:

The main functionality is in eventTesting.ts. What it does is load a particular snapshot (passed through command line), then run an associated "test script" to test a bunch of events.

A "test script" file is formatted like so:

{
  "snapshot": <name of snapshot to load>,
  "tests": [
    {
      "name": "Org Reg -- Application create",  \\ Test name
      "input": { "templateCode": "U2-OrgRegistration", "trigger": "create" }, \\ The input for "testTrigger" function
      "output": { ...<expected output> }
    },
    ...more tests...
  ]
}

The output is matched using Jest's toMatchObject matcher, which matches a result that contains at least the properties defined in the test specification -- this means we can omit a lot of redundant stuff, like the responses, most of the "finalApplicationData", and also timestamps and other values that can change each time.

To run:

yarn test_snapshot <name-of-snapshot-or-test-script-file>

The first place the script will look is in a (new) snapshot_test_scripts folder (.git-ignored) for a .json file matching the name. If it finds one, it'll load the snapshot named in the json file, then run the tests on it.

If no matching file exists in snapshot_test_scripts, it'll assume the "name" is the name of the snapshot itself and look for a file called tests.json in that snapshot folder (and then it'll load that snapshot).

The reason for this is that while developing tests, we'll work on a "test script" file in the snapshot_test_scripts folder, and keep running it from there until we're happy with it. Once it's good, we can export a new snapshot, and the test script .json file will be automatically saved with the snapshot. In order to export a test script along with the snapshot, you need to make sure the new snapshot name matches the script file. E.g. saving a snapshot called "test_snapshot_angola" will look for a file called test_snapshot_angola.json in the test scripts folder and export that if it exists.

To test

I've made one snapshot with an embedded test script and put it in the templates repo:
conforma-templates/testing_suites/angola_test_suite_2022_10_27.zip

Import the snapshot, then run it with yarn test_snapshot angola_test_suite_2022_10_27.

The current tests just run through the OrgReg process, with one LOQ and re-submission. So fairly simple, but it'll give you the idea.

Notes

I know we talked about making a global test suite for actions rather than individual snapshots. However we still need a snapshot to base it off. So initially, we can just use an Angola one to build from and make custom ones later on.

It'll be fairly tedious to set up a full test suite for Product Reg and test all the possible pathways, so it's something we can do incrementally. But I think it'll give us a lot more peace of mind in the long run when doing configurations. In future, we should add tests scripts at the same time as doing the config - since we should be testing it manually anyway, it shouldn't be too much more effort to just copy-paste the results into a test script.

Improvements

I'd like to take advantage or more dynamic matchers in Jest. Currently I'm just ignoring fields that might change each run (like timestamps and randomly generated strings), but it'd be good to match them as well. Also, arrays -- with the current setup, any array in the result must be exactly the same, or we have to omit it completely from the test. There are ways to do partial array matching, as well as checking the contents but not the order.

The reason I didn't do that yet is that the script file is a static JSON, so can't have additional matching functions defined in it. In order to do this, we'd need to invent a syntax to represent the different types of matchers in the JSON, then run a mapping function over it to turn them into actual matcher objects. Obviously that's a bit involved for now, and in most cases won't add that much extra value, so leaving it for the time being.

What we should do -- include this in our "release" script, so we can't make release builds without passing tests. -- This is done: when you run yarn release, it will test whatever test script you have defined in your .env file: BUILD_TEST_SNAPSHOT

So to test the above snapshot before building a release, you would add BUILD_TEST_SNAPSHOT='angola_test_suite_2022_10_27' to .env.

---

Edit: one further improvement I'd like to do: when you run the test on a specific snapshot, at the end of the test (if it passes), we re-export the snapshot with an additional file in it -- a "certificate" that confirms that the test suite was passed (with the date/time and build/branch data).

[CarlosNZ]

I added the sorting here, as the array order was coming out different each time. As mentioned in the original comment, it'd be good to do more nuanced array matching (check elements but not order), but that's a bit more involved for now.