Source Intercom: Update with latest CDK features, remove custom incremental sync components

[btkcodedev]

What

Closes https://github.com/airbytehq/airbyte-internal-issues/issues/11532

How

• source-intercom is running on the latest version of CDK v6 (6.33.0+)
• The companies stream does not use the IncrementalSingleSliceCursor
• The contacts stream does not use the IncrementalSingleSliceCursor
• The conversations stream does not use the IncrementalSingleSliceCursor
• The segments stream does not use the IncrementalSingleSliceCursor
• The company_segments stream does not use the IncrementalSubstreamSlicerCursor
• The conversation_parts stream does not use the IncrementalSubstreamSlicerCursor
• IncrementalSingleSliceCursor and IncrementalSubstreamSlicerCursor is deleted from components.py
• All tests in /unit_tests pass.
• All CI checks are passing ✅
• Live traffic regression test suite run with passing results
• Progressive rollout starting with 20%
• Progressive rollout starting with 50%
• Progressive rollout as main release of the candidate version

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Feb 17, 2025 2:12am

[btkcodedev]

/bump-version type="minor" changelog="Update with latest CDK features, remove custom incremental sync components"

Bump Version job started... Check job output.

✅ Changes applied successfully. (1b27557)

[btkcodedev]

/format-fix

Format-fix job started... Check job output.

✅ Changes applied successfully. (2584cfa)

[btkcodedev]

Removed parameters field, and other redundant fields for more compatibility with builder

[btkcodedev]

Removed the incremental custom components

[btkcodedev]

Tested with builder
Streams are fetching data, hashed within manifest

[btkcodedev]

Hashed at manifest: https://github.com/airbytehq/airbyte/pull/53187/files#diff-06642ea1714b19eda2e04aebb09a6c67f87162abc441a9548a9f5e144640e26fR948

[brianjlai]

@DanyloGL let's get a regression test of intercom going when you have a moment

[DanyloGL]

All regression runs were not successful. Some were cut off due to time out, some failed with error: No connection objects could be fetched. Created Cloud connection with dev image on our sandbox. All affected streams retrieve expected records except Segments. Investigating.

[btkcodedev]

Yes @DanyloGL, I've experienced that too during test, 3 records were not fetched from segments stream and thus I explicitly removed it from expected_records.jsonl
I thought it was deleted from cloud account due to some reason, Please ping if any changes are needed

[brianjlai]

Hi @btkcodedev,

I was doing a quick review of the code and I actually noticed one more change that will be required. This is not on you, but I missed this when I wrote the original issue for you up.

Specifically for the the conversations.retriever.requester.request_body_json.query should have this field modified to no longer make any reference to stream_state or stream_slice. It should instead look as such:

query: >-
  { 'operator': 'OR', 'value': [ { 'field': 'updated_at',
  'operator': '>', 'value': {{ stream_interval.get('start_time', {}) or
  format_datetime(config['start_date'], '%s') }} }, { 'field':
  'updated_at', 'operator': '=', 'value': {{
  stream_interval.get('start_time', {}) or format_datetime(config['start_date'],
  '%s') }} }, ], }

The same needs to also be done for contacts.retriever.requester.request_body_json.query. Both of these streams have the same query object so they can be replaced with the same text blob.

Let me know if you have any questions, but otherwise this should be the last thing before final validations

[DanyloGL]

@brianjlai @btkcodedev it looks like segments stream does not retrieve default segments which are created automatically when account is created. Current connector version returns such segments. Analyzing live connections in most cases there is only 1 segment. In this run there is connection with 3 records on segments stream, all of them were retrieved as in prod version.

[btkcodedev]

@brianjlai I've done the query change as you suggested.
I've checked segments again, but saw something weird in builder, If the is_client_side_incremental: true is applied, responses could not be extracted from builder, and the extracted_path is the same, but if is_client_side_incremental: true is removed, it extracts 9 records normally,
What should we do?