Fix multiple data sources for a single conversation #11509
Conversation
Fraggle
added
the
migration-ack
| @@ -1,45 +1,37 @@ | |||
| export class Lock { | |||
| private static locks = new Map<string, Promise<void>>(); | |||
There was a problem hiding this comment.
So wait the issue stems from the fact that the lock was in memory rather than on Redis?
| @@ -0,0 +1,6 @@ | |||
| DROP INDEX CONCURRENTLY "data_sources_workspace_id_conversation_id"; | |||
There was a problem hiding this comment.
So we drop the index before creating the new one?
There was a problem hiding this comment.
Yep, seems that we can't update.
| "Successfully processed file" | ||
| ); | ||
| } catch (error) { | ||
| logger.error( |
There was a problem hiding this comment.
Don't want to throw/exit if we hit those errors? I guess if we have a lot of volume, those errors might end up being "unnoticed".
| continue; | ||
| } | ||
|
|
||
| await hardDeleteDataSource(tempAuth, ds); |
There was a problem hiding this comment.
You need to soft delete them first!
There was a problem hiding this comment.
Really ? Why ? (it was working just fine in my test)
There was a problem hiding this comment.
IIRC, we had a check to ensure data sources were soft deleted first.
There was a problem hiding this comment.
If you look at the soft delete, it does delete directly the data source view, to ensure it's not visible anymore in the product:
There was a problem hiding this comment.
Also it's probably best, to simply use the softDeleteDataSourceAndLaunchScrubWorkflow.
Fraggle
deleted the
2428-multiple-data-sources-for-a-single-conversation
branch
Description
Enforce unique index, add distributed lock, create backfill script.
Tests
Run locally (i had duplicates)
Risk
Low
Deploy Plan
Run backfill, apply migration, deploy
front