[Assistants] trending feature

[mishig25]

Description

There is a new field assistant.last24HoursCount that will be used to sort assistants in trending manner.

last24HoursCount:{
	count: 3356, // total number of requests an assistant received over the last 24 hours
	byHour: {
		0: 567, // number of requests an assistant received in the last 24 hours from 01:00-02:00
		1: 928, // number of requests an assistant received in the last 24 hours from 02:00-03:00
		2: 734,
		...
		23: 374, // number of requests an assistant received in the last 24 hours from 23:00-24:00
	}
}

The trick is: we created a 24-hour window rolling counter for assistants by running the function below every hour:

const hourExpired = new Date().getUTCHours() + 1;
// then in mongo database
assistant.last24HoursCount.count -= assistant.last24HoursCount[hourExpired]; // 24-hour window rolling counter
assistant.last24HoursCount[hourExpired] = 0; // reset

using semaphore/lock/migration helpers from #897 to make sure that this periodic function runs correctly & runs only on one instance (no double running)

Screenshots

Desktop	Mobile

Deployment plan

1. Get this PR approved
2. Hide the sort assistants UX/frontend while the data is being collected
3. Deploy
4. Enable/show sort assistants UX/frontend

[coyotte508]

A few comments:

• You can use a dedicated collection for assistant analytics, with a unique key on date (truncated to the hour) + assistant's _id

await collections.assistantStats.updateOne({
  // or `"date.at": startOfHour(new Date()) - with `date-fns` 
  "date.at": {$dateTrunc: {date: new Date(), unit: "hour"}},
  assistantId: ...
}, {
  $inc: {count: 1}
}, {upsert: true});

This will be more robust

And periodically you can run a $merge aggregation query:

await collections.assistantStats.aggregate([..., {$merge: {into: "assistants"}})
await collections.assistants.updateMany({
  "last24HoursCount.count": {$gt: 0},
  "last24HoursCount.updatedAt": ...
}, {$set: {"last24HoursCount.count": 0}});

• The "cron" shouldn't be in "migrations.ts". There should maybe be an env var to enable it. And you can reuse the collections exported from src/lib/server/database.ts, no need to create a separate DB connection.

eg

setInterval(refreshAssistantsStats, 3600_000)

async function refreshAssistantsStats() {
  if (!cantLockDb()) {
    return;
  }

  // run DB queries
}

[gary149]

Ok for the UI

[coyotte508]

I don't think you need to put it in a Migration anymore?

Just to be clear, let's not mix up cron jobs & migrations in the migrations collection

I don't think we need to store cron job results in DB, but if we do, probably a separate collection

[mishig25]

I was in assumption that multiple machines are running chat-ui (and the call below might be called my multiple machines within a same period). therefore, we would need to use Migration & locks. No ?

chat-ui/src/hooks.server.ts

Line 26 in 58f3eac

refreshAssistantsCounts();

[coyotte508]

locks yes, but no need for Migration

We can create a lock on refresh-assistant-counts

[mishig25]

handled in 0803740

1. Removed usage of Migration
2. Also set those lines below to make sure that different machines will try to run the refresh "cron" job at the same time (i.e. synced even if the machines start at different times)

chat-ui/src/lib/assistantStats/refresh-assistants-counts.ts

Lines 70 to 79 in 0803740

	const now = new Date();
	const nextHour = startOfHour(addHours(now, 1));
	const delay = nextHour.getTime() - now.getTime();
	const ONE_HOUR_MS = 3_600_000;
	// wait until the next hour to start the hourly interval
	setTimeout(() => {
	refreshAssistantsCountsHelper();
	setInterval(refreshAssistantsCountsHelper, ONE_HOUR_MS);

[mishig25]

based on #938 (review), I've committed 93c9ebc

Description:

1. Created a new collection called AssistantStats. Fields are: _id: Assistant["_id"], dateWithHour: Date, count: number
2. dateWithHour is a date field. However, the value should have minutes, seconds, ms truncated. let date = new Date(); date.setUTCMinutes(0, 0, 0) so that we can use this field dateWithHour to increment count field everytime an assistant gets a visit within same hour
3. Every hour, there is a "cron-like" aggregation query that does: for the last 24 hours, group on _id: Assistant["_id"] & sum count: number and afterwards using $merge.whenMatched pipeline update assistant.last24HoursCount field

[coyotte508]

Did several backend changes

• For stats collection, the _id shouldn't be assistantId (otherwise we can create only one document per assistant, while we should have 1 / hour / assistant)
• added date.at and date.span fields to be consistent with other stat collections - also in case we want to be more granular in the future (instead of 1 doc/hour, 1 doc/10 minutes for example)
• Reworked the locking mechanism
  • Made lockKey mandatory
  • Return the lockId, to make sure we return / release a lock we own
• Maintain the lock for the cron job during the whole process's lifespan - no need to sync job start
• Changed the aggregation to use $unionWith - a nice trick originating from @Kakulukian I think. No need for a lastUpdatedAt field with this!
• Fixed a bug on usage limits for assistant creation

Maybe the lock mechanism (maintainLock) can be more standardized if we have more cron jobs

[mishig25]

@nsarrazin wdyt ?

[nsarrazin]

Nice! Doing one last pass

[nsarrazin]

Looks good to me! If I can help with the deployment let me know 😁

[mishig25]

checking one more time from my end & will ping you on deployment

[mishig25]

as written in the dscription

Deployment plan

1. Get this PR approved
2. Hide the sort assistants UX/frontend while the data is being collected
3. Deploy
4. Enable/show sort assistants UX/frontend

I've hidden the UI 4dd7e65

Ready to merge & deploy @nsarrazin