Automated migration setup

[nsarrazin]

This PR adds a check on startup that runs migrations if needed.

How it works

Each migration is defined by it's. _id field which is unique and hardcoded. When the migration is applied we add it to the "migrationResults" collections, so we can check if it has already been run. Each migration is wrapped in a transaction to ensure consistency.

The migration check runs at the top-level of hooks.server.ts to ensure it only runs once on startup.

DB Locks

Because we can have multiple instances of chat-ui running against a single DB we need to ensure we have a lock before running migrations.

The code for lock related operations is under lock.ts with the associated unit-tests.

How it works:

1. We have a semaphore collection with a unique index on keys
2. We can set the lock status by using insertOne, if it fails to insert then a lock must already exist (see acquireLock)
3. the lock has an updatedAt field with a TTL index for mongo to discard if it times out
4. On server startup, all instances try to acquire the lock, the one that gets it can run migrations and the other will wait until the lock is released to start the rest of chat-ui.

Migrations

You can find an example migration here

Migrations can have both up (required) and down (optional) methods (though we don't use the down method for now) and optionally they can also set:

	runForHuggingChat?: "only" | "never"; // leave unspecified to run for both

We should port the migration from #841 when merging this PR.

[nsarrazin]

cc @coyotte508 might be worth taking a look if you have time, especially the following files:

• lock.ts has all the functions used to create/acquire/release locks
• This part of migrations.ts where we run the migrations
• the indexes and document definitions (semaphore, MigrationResult)

[coyotte508]

A bit of an issue with spaces with continued availability - the old app can run for a bit after migrations are run.

So ideally, migrations happen in two steps: a first PR is deployed that fills in the new data (eg search tokens), and a second PR is deployed with the migration + the feature visible for users.

Just a bit of a process to think of

[nsarrazin]

So ideally, migrations happen in two steps: a first PR is deployed that fills in the new data (eg search tokens), and a second PR is deployed with the migration + the feature visible for users.

If we deploy the data generation before deploying the feature, isn't there a gap between the two deployments where new documents would be missed ? Not sure if I got it correctly 😅

[coyotte508]

So ideally, migrations happen in two steps: a first PR is deployed that fills in the new data (eg search tokens), and a second PR is deployed with the migration + the feature visible for users.

If we deploy the data generation before deploying the feature, isn't there a gap between the two deployments where new documents would be missed ? Not sure if I got it correctly 😅

What I mean:

• First PR fills in searchTokens when assistants are created / updated, and does nothing else
• Second PR has the migration + the rest of the feature

So:

• Deploy first PR
• When users update assistants, the searchTokens fields is updated for them
• Deploy second PR & run migration
• Even if old code runs for a bit, no problems, as searchTokens are properly populated by both the migration and the first PR

[nsarrazin]

Oh my bad 🤦 That makes a lot of sense, will keep in mind for deploying #841 cc @mishig25

[mishig25]

lgtm !