feat(Gong) - add an optional retention policy #11264

aubin-tchoi · 2025-03-06T16:17:16Z

Description

This PR implements an optional retention policy.

When disabled, all transcripts are synced (no change here).
When enabled, only the transcripts that are no older than the retention period are synced.

This behavior is implemented using two things:

We fetch transcripts starting from the retention period at minimum.
We garbage collect outdated transcripts.

There is currently no UI for configuring it, it will be added to the GongOptionComponent in a follow up PR (get/set already there on the connector manager).

The main workflow is updated to add an optional step of garbage collection:

We store the lastGarbageCollectionTimestamp in the gong_configuration alongside the lastSyncTimestamp.
After completing the sync, we check whether the garbage collection should be run (using retentionPeriodDays, current date and lastGarbageCollectionTimestamp).
If so, we run it and update lastGarbageCollectionTimestamp

Tests

Ongoing.

Risk

N/A.

Deploy Plan

Apply migration_62.sql.
Deploy connectors.

github-actions · 2025-03-06T16:18:58Z

	Warnings
⚠️	Files in `**/lib/models/` have been modified and the PR has the `migration-ack` label. Don't forget to run the migration from prodbox.

Generated by 🚫 dangerJS against 2982782

This reverts commit da48897.

…ingle activity

# Conflicts: # connectors/src/connectors/gong/temporal/activities.ts

flvndvd · 2025-03-14T14:30:47Z

connectors/src/connectors/gong/index.ts

+    switch (configKey) {
+      case RETENTION_PERIOD_CONFIG_KEY: {
+        const configuration = await fetchGongConfiguration(connector);
+        const retentionPeriodDays = configValue


You might want to check the result of this with a Number.isFinite() to avoid saving NaN.

flvndvd · 2025-03-14T14:33:37Z

connectors/src/connectors/gong/temporal/workflows.ts

+  garbageCollectionStartTs: number;
+}) {
+  let hasMoreTranscripts = true;
+  while (hasMoreTranscripts) {


Let's use a do/while to align with code above. Also do/while runs at least once 😉

flvndvd · 2025-03-14T14:34:59Z

connectors/src/resources/gong_resources.ts

+        return Date.now() - this.retentionPeriodDays * 24 * 60 * 60 * 1000;
+      }
+      return Math.max(
+        this.lastSyncTimestamp,


Not sure to see the value of this field? In both cases we add an SQL clause on a data field, right?

not sure I understand, does this make sense?

if (this.retentionPeriodDays && this.lastSyncTimestamp === null) { return Date.now() - daysToMs(this.retentionPeriodDays); } return this.lastSyncTimestamp;

My question is more, why do we store the last GC timestamp? I don't see the value it brings?

oh ok the comment line number bugged out

I use it to prevent the gc from running every time: currentTimestamp - this.lastGarbageCollectionTimestamp > GC_FREQUENCY_MS, it could be replaced with a modulo on the hour number but this seemed more simple to understand

we could also run the GC every time, wdyt?

flvndvd · 2025-03-14T14:35:21Z

connectors/src/resources/gong_resources.ts

+
+    const retentionPeriodStart =
+      garbageCollectionStartTs -
+      configuration.retentionPeriodDays * 24 * 60 * 60 * 1000;


nit helper for days to ms.

* add retentionPeriodDays to GongConfigurationModel * add migration file * set lastSyncTimestamp based on the retention period * make lastSyncTimestamp non nullable * make lastSyncTimestamp non nullable in db * merge both sql migration scripts * add an activity that deleted outdated transcripts * add a column callDate * add migration script to create the new column * make retentionPeriodDays optional (keeps all transcripts) * add a garbage collect workflow * fix * keep lastSyncTimestamp and retentionPeriodDays separate * Revert "allow selecting the Transcripts folder" This reverts commit da48897. * add get/set configuration key methods * whitelist the new key * add the garbage collection as a child workflow of the main workflow * add baseUrl and retentionPeriodDays to the resource's toJSON * add a column lastGarbageCollectTimestamp to GongConfigurationModel * add the column lastGarbageCollectTimestamp to the db migration * rename the column * prevent the GC from running more than once a day * move the pagination on GC at the activity level instead of within a single activity * sort activity list * add lastGarbageCollectionTimestamp to toJSON * refactor: simplify checkGarbageCollectionState * pass along the garbageCollectionStartTs * rename * rename * correctly use the callDate * remove rebase artefact * refactor: uniformize a while into a do/while * add a Number.isFinite check on the parsed value for the retention period * add an util daysToMs

aubin-tchoi force-pushed the gong-retention-policy branch 2 times, most recently from ce6bcc7 to 584667b Compare March 6, 2025 16:23

aubin-tchoi added the migration-ack label Mar 6, 2025

aubin-tchoi force-pushed the gong-retention-policy branch from 19da437 to 2c40deb Compare March 6, 2025 22:08

aubin-tchoi and others added 16 commits March 14, 2025 11:44

add retentionPeriodDays to GongConfigurationModel

2a9fe6d

add migration file

3872068

set lastSyncTimestamp based on the retention period

1720173

make lastSyncTimestamp non nullable

3e42d3c

make lastSyncTimestamp non nullable in db

34eb39b

merge both sql migration scripts

ea5cea1

add an activity that deleted outdated transcripts

2c22a80

add a column callDate

977c20c

add migration script to create the new column

755940a

make retentionPeriodDays optional (keeps all transcripts)

11279b4

add a garbage collect workflow

cfaada9

fix

2432d12

keep lastSyncTimestamp and retentionPeriodDays separate

a3856f6

Revert "allow selecting the Transcripts folder"

ce79ee7

This reverts commit da48897.

add get/set configuration key methods

5f92d30

whitelist the new key

373d4f4

aubin-tchoi force-pushed the gong-retention-policy branch from 2c40deb to 373d4f4 Compare March 14, 2025 10:44

aubin-tchoi added 2 commits March 14, 2025 13:56

add the garbage collection as a child workflow of the main workflow

c850468

add baseUrl and retentionPeriodDays to the resource's toJSON

ef487c2

aubin-tchoi force-pushed the gong-retention-policy branch from d92f873 to ef487c2 Compare March 14, 2025 13:03

aubin-tchoi added 5 commits March 14, 2025 14:20

add a column lastGarbageCollectTimestamp to GongConfigurationModel

79c6cc6

add the column lastGarbageCollectTimestamp to the db migration

ed3003b

rename the column

20f144d

prevent the GC from running more than once a day

0e2c0e9

move the pagination on GC at the activity level instead of within a s…

71c2296

…ingle activity

aubin-tchoi added 3 commits March 14, 2025 14:54

sort activity list

21b5b01

add lastGarbageCollectionTimestamp to toJSON

36bd8b7

refactor: simplify checkGarbageCollectionState

4a1f733

aubin-tchoi force-pushed the gong-retention-policy branch from b2b0402 to 4a1f733 Compare March 14, 2025 13:58

aubin-tchoi added 4 commits March 14, 2025 15:02

pass along the garbageCollectionStartTs

3c878c4

rename

459086f

rename

df284c1

correctly use the callDate

207b2fc

aubin-tchoi requested a review from flvndvd March 14, 2025 14:24

aubin-tchoi added 2 commits March 14, 2025 15:25

remove rebase artefact

540977d

Merge branch 'main' into gong-retention-policy

6ebf718

# Conflicts: # connectors/src/connectors/gong/temporal/activities.ts

flvndvd approved these changes Mar 14, 2025

View reviewed changes

aubin-tchoi added 3 commits March 14, 2025 15:40

refactor: uniformize a while into a do/while

3017651

add a Number.isFinite check on the parsed value for the retention period

b05b5c1

add an util daysToMs

2d17413

aubin-tchoi mentioned this pull request Mar 14, 2025

feat(Gong) - allow setting the retention period in GongOptionComponent #11384

Merged

Merge branch 'main' into gong-retention-policy

2982782

aubin-tchoi merged commit 947027f into main Mar 17, 2025
9 checks passed

aubin-tchoi deleted the gong-retention-policy branch March 17, 2025 09:00

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(Gong) - add an optional retention policy #11264

feat(Gong) - add an optional retention policy #11264

aubin-tchoi commented Mar 6, 2025 •

edited

Loading

github-actions bot commented Mar 6, 2025 •

edited

Loading

flvndvd Mar 14, 2025

flvndvd Mar 14, 2025

flvndvd Mar 14, 2025

aubin-tchoi Mar 14, 2025

flvndvd Mar 14, 2025

aubin-tchoi Mar 14, 2025

aubin-tchoi Mar 14, 2025

aubin-tchoi Mar 14, 2025

flvndvd Mar 14, 2025

feat(Gong) - add an optional retention policy #11264

feat(Gong) - add an optional retention policy #11264

Conversation

aubin-tchoi commented Mar 6, 2025 • edited Loading

Description

Tests

Risk

Deploy Plan

github-actions bot commented Mar 6, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aubin-tchoi commented Mar 6, 2025 •

edited

Loading

github-actions bot commented Mar 6, 2025 •

edited

Loading