Facebook marketing data sync issue

[darsh9292]

I have been exploring airbyte and planning to use for one of my product if things goes well. However, I have been struggling to sync Facebook marketing data quickly.

It takes hours to sync or complete one data to store in either postgres/Mongo.

From the logs, I have observed following failures :

2024-12-04 20:08:36 replication-orchestrator INFO failures: [ {
"failureOrigin" : "source",
"failureType" : "config_error",
"internalMessage" : "Please reduce the amount of data you're asking for, then retry your request",
"externalMessage" : "Please reduce the number of fields requested. Go to the schema tab, select your source, and unselect the fields you do not need.",
"metadata" : {
"attemptNumber" : 0,
"jobId" : 23282747,
"from_trace_message" : true,
"connector_command" : "read"
},
"stacktrace" : "Traceback (most recent call last):\n File "/usr/local/lib/python3.10/site-packages/backoff/_sync.py", line 105, in retry\n ret = target(args, kwargs)\n File "/airbyte/integration_code/source_facebook_marketing/api.py", line 169, in call\n response = super().call(method, path, params, headers, files, url_override, api_version)\n File "/usr/local/lib/python3.10/site-packages/facebook_business/api.py", line 336, in call\n raise fb_response.error()\nfacebook_business.exceptions.FacebookRequestError: \n\n Message: Call was not successful\n Method: GET\n Path: https://graph.facebook.com/v19.0/act_1384927272/insights\n Params: {'level': 'campaign', 'action_breakdowns': '["action_type"]', 'action_report_time': 'mixed', 'breakdowns': '[]', 'fields': '["campaign_id"]', 'action_attribution_windows': '["1d_click","7d_click","28d_click","1d_view","7d_view","28d_view"]', 'filtering': '[{"field":"ad.effective_status","operator":"IN","value":["ACTIVE","ADSET_PAUSED","ARCHIVED","CAMPAIGN_PAUSED","DELETED","DISAPPROVED","IN_PROCESS","PAUSED","PENDING_BILLING_INFO","PENDING_REVIEW","PREAPPROVED","WITH_ISSUES"]}]', 'time_range': '{"since":"2024-10-28","until":"2024-11-26"}'}\n\n Status: 500\n Response:\n {\n "error": {\n "code": 1,\n "message": "Please reduce the amount of data you're asking for, then retry your request"\n }\n }\n\n",
"timestamp" : 1733264936504,
"streamDescriptor" : {
"name" : "ads_insights_action_type"
}
}, {
"failureOrigin" : "source",
"failureType" : "config_error",
"internalMessage" : "Please reduce the amount of data you're asking for, then retry your request",
"externalMessage" : "Please reduce the number of fields requested. Go to the schema tab, select your source, and unselect the fields you do not need.",
"metadata" : {
"attemptNumber" : 0,
"jobId" : 23282747,
"from_trace_message" : true,
"connector_command" : "read"
},
"stacktrace" : "Traceback (most recent call last):\n File "/usr/local/lib/python3.10/site-packages/backoff/_sync.py", line 105, in retry\n ret = target(args, kwargs)\n File "/airbyte/integration_code/source_facebook_marketing/api.py", line 169, in call\n response = super().call(method, path, params, headers, files, url_override, api_version)\n File "/usr/local/lib/python3.10/site-packages/facebook_business/api.py", line 336, in call\n raise fb_response.error()\nfacebook_business.exceptions.FacebookRequestError: \n\n Message: Call was not successful\n Method: GET\n Path: https://graph.facebook.com/v19.0/act_13849284192/insights\n Params: {'level': 'campaign', 'action_breakdowns': '["action_type","action_target_id","action_destination"]', 'action_report_time': 'mixed', 'breakdowns': '[]', 'fields': '["campaign_id"]', 'action_attribution_windows': '["1d_click","7d_click","28d_click","1d_view","7d_view","28d_view"]', 'filtering': '[{"field":"ad.effective_status","operator":"IN","value":["ACTIVE","ADSET_PAUSED","ARCHIVED","CAMPAIGN_PAUSED","DELETED","DISAPPROVED","IN_PROCESS","PAUSED","PENDING_BILLING_INFO","PENDING_REVIEW","PREAPPROVED","WITH_ISSUES"]}]', 'time_range': '{"since":"2024-10-31","until":"2024-11-29"}'}\n\n Status: 500\n Response:\n {\n "error": {\n "code": 1,\n "message": "Please reduce the amount of data you're asking for, then retry your request"\n }\n }\n\n",
"timestamp" : 1733319871112,
"streamDescriptor" : {
"name" : "ads_insights"
}
}, {
"failureOrigin" : "source",
"failureType" : "config_error",
"internalMessage" : "Please reduce the amount of data you're asking for, then retry your request",
"externalMessage" : "Please reduce the number of fields requested. Go to the schema tab, select your source, and unselect the fields you do not need.",
"metadata" : {
"attemptNumber" : 0,
"jobId" : 23282747,
"from_trace_message" : true,
"connector_command" : "read"
},
"stacktrace" : "Traceback (most recent call last):\n File "/usr/local/lib/python3.10/site-packages/backoff/_sync.py", line 105, in retry\n ret = target(args, kwargs)\n File "/airbyte/integration_code/source_facebook_marketing/api.py", line 169, in call\n response = super().call(method, path, params, headers, files, url_override, api_version)\n File "/usr/local/lib/python3.10/site-packages/facebook_business/api.py", line 336, in call\n raise fb_response.error()\nfacebook_business.exceptions.FacebookRequestError: \n\n Message: Call was not successful\n Method: GET\n Path: https://graph.facebook.com/v19.0/act_13849284192*/insights\n Params: {'level': 'campaign', 'action_breakdowns': '["action_type","action_target_id","action_destination"]', 'action_report_time': 'mixed', 'breakdowns': '["region"]', 'fields': '["campaign_id"]', 'action_attribution_windows': '["1d_click","7d_click","28d_click","1d_view","7d_view","28d_view"]', 'filtering': '[{"field":"ad.effective_status","operator":"IN","value":["ACTIVE","ADSET_PAUSED","ARCHIVED","CAMPAIGN_PAUSED","DELETED","DISAPPROVED","IN_PROCESS","PAUSED","PENDING_BILLING_INFO","PENDING_REVIEW","PREAPPROVED","WITH_ISSUES"]}]', 'time_range': '{"since":"2024-10-31","until":"2024-11-29"}'}\n\n Status: 500\n Response:\n {\n "error": {\n "code": 1,\n "message": "Please reduce the amount of data you're asking for, then retry your request"\n }\n }\n\n",
"timestamp" : 1733323040205,
"streamDescriptor" : {
"name" : "ads_insights_region"
}
}, {
"failureOrigin" : "source",
"failureType" : "config_error",
"internalMessage" : "(#200) Requires business_management permission to manage the object",
"externalMessage" : "Credentials don't have enough permissions. Re-authenticate if FB oauth is used or refresh access token with all required permissions.",
"metadata" : {
"attemptNumber" : 0,
"jobId" : 23282747,
"from_trace_message" : true,
"connector_command" : "read"
},
"stacktrace" : "Traceback (most recent call last):\n File "/usr/local/lib/python3.10/site-packages/backoff/_sync.py", line 105, in retry\n ret = target(args, kwargs)\n File "/airbyte/integration_code/source_facebook_marketing/api.py", line 169, in call\n response = super().call(method, path, params, headers, files, url_override, api_version)\n File "/usr/local/lib/python3.10/site-packages/facebook_business/api.py", line 336, in call\n raise fb_response.error()\nfacebook_business.exceptions.FacebookRequestError: \n\n Message: Call was not successful\n Method: GET\n Path: https://graph.facebook.com/v19.0/act_13849284192**/\n Params: {'fields': 'id,age,name,owner,tax_id,balance,partner,rf_spec,business,currency,fb_entity,io_number,spend_cap,account_id,user_tasks,is_personal,tax_id_type,timezone_id,amount_spent,business_zip,capabilities,created_time,line_numbers,media_agency,tos_accepted,business_city,business_name,tax_id_status,timezone_name,account_status,business_state,disable_reason,end_advertiser,funding_source,business_street,business_street2,min_daily_budget,user_tos_accepted,is_tax_id_required,end_advertiser_name,business_country_code,failed_delivery_checks,is_direct_deals_enabled,has_migrated_permissions,is_notifications_enabled,timezone_offset_hours_utc,can_create_brand_lift_study,offsite_pixels_tos_accepted,min_campaign_group_spend_cap,extended_credit_invoice_group,is_attribution_spec_system_default,is_in_3ds_authorization_enabled_market'}\n\n Status: 403\n Response:\n {\n "error": {\n "message": "(#200) Requires business_management permission to manage the object",\n "type": "OAuthException",\n "code": 200,\n "fbtrace_id": "AyPwP3P6HjSlaGVexyoemob"\n }\n }\n\n",
"timestamp" : 1733323042531,
"streamDescriptor" : {
"name" : "ad_account"
}
}, {
"failureOrigin" : "source",
"failureType" : "config_error",
"externalMessage" : "During the sync, the following streams did not sync successfully: ads_insights_action_type: AirbyteTracedException("Please reduce the amount of data you're asking for, then retry your request")\nads_insights: AirbyteTracedException("Please reduce the amount of data you're asking for, then retry your request")\nads_insights_region: AirbyteTracedException("Please reduce the amount of data you're asking for, then retry your request")\nad_account: AirbyteTracedException('(#200) Requires business_management permission to manage the object')",
"metadata" : {
"attemptNumber" : 0,
"jobId" : 23282747,
"from_trace_message" : true,
"connector_command" : "read"
},
"stacktrace" : "Traceback (most recent call last):\n File "/airbyte/integration_code/main.py", line 9, in \n run()\n File "/airbyte/integration_code/source_facebook_marketing/run.py", line 19, in run\n launch(source, sys.argv[1:])\n File "/usr/local/lib/python3.10/site-packages/airbyte_cdk/entrypoint.py", line 263, in launch\n for message in source_entrypoint.run(parsed_args):\n File "/usr/local/lib/python3.10/site-packages/airbyte_cdk/entrypoint.py", line 133, in run\n yield from map(AirbyteEntrypoint.airbyte_message_to_string, self.read(source_spec, config, config_catalog, state))\n File "/usr/local/lib/python3.10/site-packages/airbyte_cdk/entrypoint.py", line 191, in read\n for message in self.source.read(self.logger, config, catalog, state):\n File "/usr/local/lib/python3.10/site-packages/airbyte_cdk/sources/abstract_source.py", line 177, in read\n raise AirbyteTracedException(message=error_message, failure_type=FailureType.config_error)\nairbyte_cdk.utils.traced_exception.AirbyteTracedException: None\n",
"timestamp" : 1733323101479
}, {
"failureOrigin" : "destination",
"failureType" : "transient_error",
"internalMessage" : "Some streams either received an INCOMPLETE stream status, or did not receive a stream status at all: null.ad_account, null.images, null.ads_insights_region, null.ads_insights_action_type, null.campaigns, null.videos, null.ads, null.ads_insights, null.ad_sets, null.ad_creatives\nio.airbyte.commons.exceptions.TransientErrorException: Some streams were unsuccessful due to a source error. See logs for details.",
"externalMessage" : "Some streams were unsuccessful due to a source error. See logs for details.",
"metadata" : {
"attemptNumber" : 0,
"jobId" : 23282747,
"from_trace_message" : true,
"connector_command" : "write"
},
"stacktrace" : "io.airbyte.commons.exceptions.TransientErrorException: Some streams were unsuccessful due to a source error. See logs for details.\n\tat io.airbyte.cdk.integrations.destination.async.AsyncStreamConsumer.close(AsyncStreamConsumer.kt:215)\n\tat io.airbyte.cdk.integrations.base.SerializedAirbyteMessageConsumer$Companion$appendOnClose$1.close(SerializedAirbyteMessageConsumer.kt:70)\n\tat kotlin.jdk7.AutoCloseableKt.closeFinally(AutoCloseableJVM.kt:48)\n\tat io.airbyte.cdk.integrations.base.IntegrationRunner.runInternal(IntegrationRunner.kt:215)\n\tat io.airbyte.cdk.integrations.base.IntegrationRunner.run(IntegrationRunner.kt:119)\n\tat io.airbyte.cdk.integrations.base.IntegrationRunner.run$default(IntegrationRunner.kt:113)\n\tat io.airbyte.integrations.destination.postgres.PostgresDestinationStrictEncrypt$Companion.main(PostgresDestinationStrictEncrypt.kt:74)\n\tat io.airbyte.integrations.destination.postgres.PostgresDestinationStrictEncrypt.main(PostgresDestinationStrictEncrypt.kt)\n",
"timestamp" : 1733323113808
}, {
"failureOrigin" : "source",
"internalMessage" : "Source process exited with non-zero exit code 1",
"externalMessage" : "Something went wrong within the source connector",
"metadata" : {
"attemptNumber" : 0,
"jobId" : 23282747,
"connector_command" : "read"
},
"stacktrace" : "io.airbyte.workers.internal.exception.SourceException: Source process exited with non-zero exit code 1\n\tat io.airbyte.workers.general.BufferedReplicationWorker.readFromSource(BufferedReplicationWorker.java:364)\n\tat io.airbyte.workers.general.BufferedReplicationWorker.lambda$runAsyncWithHeartbeatCheck$3(BufferedReplicationWorker.java:222)\n\tat java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n\tat java.base/java.lang.Thread.run(Thread.java:1583)\n",
"timestamp" : 1733323101805
}, {
"failureOrigin" : "destination",
"internalMessage" : "Destination process exited with non-zero exit code 1",
"externalMessage" : "Something went wrong within the destination connector",
"metadata" : {
"attemptNumber" : 0,
"jobId" : 23282747,
"connector_command" : "write"
},
"stacktrace" : "io.airbyte.workers.internal.exception.DestinationException: Destination process exited with non-zero exit code 1\n\tat io.airbyte.workers.general.BufferedReplicationWorker.readFromDestination(BufferedReplicationWorker.java:502)\n\tat io.airbyte.workers.general.BufferedReplicationWorker.lambda$runAsync$2(BufferedReplicationWorker.java:215)\n\tat java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n\tat java.base/java.lang.Thread.run(Thread.java:1583)\n",
"timestamp" : 1733323113871
}, {
"failureOrigin" : "replication",
"internalMessage" : "io.airbyte.workers.exception.WorkerException: Destination process exit with code 1. This warning is normal if the job was cancelled.",
"externalMessage" : "Something went wrong during replication",
"metadata" : {
"attemptNumber" : 0,
"jobId" : 23282747
},
"stacktrace" : "java.lang.RuntimeException: io.airbyte.workers.exception.WorkerException: Destination process exit with code 1. This warning is normal if the job was cancelled.\n\tat io.airbyte.workers.general.BufferedReplicationWorker$CloseableWithTimeout.lambda$close$0(BufferedReplicationWorker.java:547)\n\tat io.airbyte.workers.general.BufferedReplicationWorker.lambda$runAsyncWithTimeout$5(BufferedReplicationWorker.java:243)\n\tat java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n\tat java.base/java.lang.Thread.run(Thread.java:1583)\nCaused by: io.airbyte.workers.exception.WorkerException: Destination process exit with code 1. This warning is normal if the job was cancelled.\n\tat io.airbyte.workers.internal.LocalContainerAirbyteDestination.close(LocalContainerAirbyteDestination.kt:60)\n\tat io.airbyte.workers.general.BufferedReplicationWorker$CloseableWithTimeout.lambda$close$0(BufferedReplicationWorker.java:545)\n\t... 5 more\n",
"timestamp" : 1733323113894
} ]
2024-12-04 20:08:36 replication-orchestrator INFO
2024-12-04 20:08:36 replication-orchestrator INFO ----- END REPLICATION -----
2024-12-04 20:08:36 replication-orchestrator INFO
2024-12-04 20:08:37 replication-orchestrator INFO Returning output...
2024-12-04 20:08:37 platform INFO Closing Segment analytics client...
2024-12-04 20:08:37 platform INFO Waiting for Segment analytic client to flush enqueued messages...
2024-12-04 20:08:37 platform INFO Segment analytic client flush complete.
2024-12-04 20:08:37 platform INFO Segment analytics client closed. No new events will be accepted.
2024-12-04 20:08:37 platform INFO

Couple of questions :

1. Why too many fields error if default reports has that many fields by default?
2. Why syncing takes hours to complete?
3. Why everything is stored in postgres as jsonb? is this normal for facebook data? Does postres is right option for storage or I should be choosing Mongo instead?