You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: content/bigquery.textile
+8-7Lines changed: 8 additions & 7 deletions
Original file line number
Diff line number
Diff line change
@@ -30,6 +30,8 @@ Set up the necessary BigQuery resources, permissions, and authentication to enab
30
30
31
31
h3(#settings). BigQuery rule settings
32
32
33
+
The following explains the components of the BigQuery rule settings:
34
+
33
35
|_. Section |_. Purpose |
34
36
| *Source* | Defines the type of event(s) for delivery. |
35
37
| *Channel filter* | A regular expression to filter which channels to capture. Only events on channels matching this regex are streamed into BigQuery. |
| *Partitioning* | _(Optional)_ The table must be created with the desired partitioning settings in BigQuery before making the rule in Ably. |
39
41
| *Advanced settings* | Any additional configuration or custom fields relevant to your BigQuery setup (for future enhancements). |
40
42
41
-
h4(#dashboard). Create a BigQuery rule in the Dashboard
43
+
h4(#dashboard). Create a BigQuery rule in the dashboard
42
44
43
45
The following steps to create a BigQuery rule using the Ably dashboard:
44
46
@@ -51,7 +53,7 @@ The following steps to create a BigQuery rule using the Ably dashboard:
51
53
52
54
h4(#api-rule). Create a BigQuery rule using the Control API
53
55
54
-
The following steps to create a BigQuery rule using the Control API:
56
+
The following steps to create a BigQuery rule using the "Control API:":https://ably.com/docs/api#control-api
55
57
56
58
* Using the required "rules":/control-api#examples-rules to specify the following parameters:
57
59
** @ruleType@: Set this to "bigquery" to define the rule as a BigQuery integration.
@@ -61,7 +63,6 @@ The following steps to create a BigQuery rule using the Control API:
61
63
** @format@ (optional): Define the data format based on how you want messages to be structured.
62
64
* Make an HTTP request to the Control API to create the rule.
63
65
64
-
65
66
h3(#schema). JSON Schema
66
67
67
68
You can run queries directly against the Ably-managed BigQuery table. For example, if the message payloads are stored as raw JSON in the data column, you can parse them using the following query:
@@ -88,10 +89,10 @@ WHERE channel = “my-channel”
88
89
89
90
The following explains the components of the query:
90
91
91
-
|. Query Function |. Purpose |
92
-
| CAST(data AS STRING) | Converts the data column from BYTES (if applicable) into a STRING format. |
93
-
| PARSE_JSON(…) | Parses the string into a structured JSON object for easier querying. |
94
-
| WHERE channel = “my-channel” | Filters results to retrieve messages only from a specific Ably channel. |
92
+
|_. Query Function |_. Purpose |
93
+
| @CAST(data AS STRING)@ | Converts the data column from BYTES (if applicable) into a STRING format. |
94
+
| @PARSE_JSON(…)@ | Parses the string into a structured JSON object for easier querying. |
95
+
| @WHERE channel = “my-channel”@ | Filters results to retrieve messages only from a specific Ably channel. |
95
96
96
97
<aside data-type='note'>
97
98
<p>Parsing JSON at query time can be computationally expensive for large datasets. If your queries need frequent JSON parsing, consider pre-processing and storing structured fields in a secondary table using an ETL pipeline for better performance.</p>
0 commit comments