Optimize the gcp_logging_log_entry table result or result timing by applying a timestamp filter and added the columns json_payload, proto_payload, metadata, resource, operation and tags Closes #507

[ParthaI]

Integration test logs

Logs

N/A

Example query results

Results

> select log_name, receive_timestamp,timestamp, resource_type from gcp_logging_log_entry where log_name = 'projects/isldke-aaa/logs/diagnostic-log' and timestamp = '2023-09-29T10:09:08+05:30'
+----------+-------------------+-----------+---------------+
| log_name | receive_timestamp | timestamp | resource_type |
+----------+-------------------+-----------+---------------+
+----------+-------------------+-----------+---------------+

> select log_name, receive_timestamp,timestamp, resource_type from gcp_logging_log_entry where log_name = 'projects/isldke-aaa/logs/diagnostic-log' and timestamp <= '2023-09-29T10:09:08+05:30'
+-----------------------------------------+---------------------------+---------------------------+---------------+
| log_name                                | receive_timestamp         | timestamp                 | resource_type |
+-----------------------------------------+---------------------------+---------------------------+---------------+
| projects/isldke-aaa/logs/diagnostic-log | 2023-09-29T09:45:09+05:30 | 2023-09-29T09:45:06+05:30 | k8s_container |
| projects/isldke-aaa/logs/diagnostic-log | 2023-09-29T09:45:12+05:30 | 2023-09-29T09:45:09+05:30 | k8s_container |
+-----------------------------------------+---------------------------+---------------------------+---------------+

Time: 63.4s. Rows fetched: 2. Hydrate calls: 0.

> select log_name, receive_timestamp,timestamp, resource_type from gcp_logging_log_entry where log_name = 'projects/isldke-aaa/logs/diagnostic-log' and timestamp >= '2023-09-29T10:09:08+05:30'
+-----------------------------------------+---------------------------+---------------------------+---------------+
| log_name                                | receive_timestamp         | timestamp                 | resource_type |
+-----------------------------------------+---------------------------+---------------------------+---------------+
| projects/isldke-aaa/logs/diagnostic-log | 2023-09-29T10:09:18+05:30 | 2023-09-29T10:09:16+05:30 | k8s_container |
| projects/isldke-aaa/logs/diagnostic-log | 2023-09-29T10:09:11+05:30 | 2023-09-29T10:09:08+05:30 | k8s_container |
+-----------------------------------------+---------------------------+---------------------------+---------------+

Time: 5.8s. Rows fetched: 2. Hydrate calls: 0.

[cbruno10]

Other suggestions:

• Remove "log_entry_" prefix from columns.
• Consolidate JSON columns so each property has only 1 column, we can always add more columns later on. This will be a breaking change but I think will make the table columns more consistent and easy to maintain in the future.

[cbruno10]

Suggested change

	// After conducting tests with maximum page sizes of 1000, 5000, 17000, 20000, and 30000, we decided to stick with a page size of 10000 for performance improvement.
	// 10000 seems to be a balanced limit, based on initial tests for retrieving 140k log entries: 5000 (124s), 10000 (88s), 20000 (84s)

[github-actions]

This PR is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 30 days.

[cbruno10]

@ParthaI Please see comments, thanks!