feat!: allow retrieving all billing accounts and add billing_information column to gcp_project and gcp_organization_project tables

[pdecat]

This is a breaking change, submitted as draft to initiate a discussion.

The rationale of this PR is that a billing account is not tied to a single project.
Without this change, all existing billing accounts cannot be discovered.

Edit: rebased on main so removed comment about what commits to review.

Integration test logs

Logs

Add passing integration test logs here

Example query results

Results

> select name, master_billing_account from gcp_billing_account order by master_billing_account, name
+----------------------+--------------------------------------+
| name                 | master_billing_account               |
+----------------------+--------------------------------------+
| 00D010-012345-000000 |                                      |
| 011C92-012345-111111 |                                      |
| 016C27-012345-222222 |                                      |
| 01BDFD-012345-333333 |                                      |
| 01C328-012345-444444 |                                      |
| 0024B1-012345-012345 | billingAccounts/00D010-012345-000000 |
| 010892-012345-012345 | billingAccounts/00D010-012345-000000 |
| 0109E7-012345-012345 | billingAccounts/00D010-012345-000000 |
| 010FA8-012345-012345 | billingAccounts/00D010-012345-000000 |
| 0110A7-012345-012345 | billingAccounts/00D010-012345-000000 |
| 0112CD-012345-012345 | billingAccounts/00D010-012345-000000 |
| 01245D-012345-012345 | billingAccounts/00D010-012345-000000 |
| 0128B9-012345-012345 | billingAccounts/00D010-012345-000000 |
| 012DD3-012345-012345 | billingAccounts/00D010-012345-000000 |
| 0130FA-012345-012345 | billingAccounts/00D010-012345-000000 |
| 013522-012345-012345 | billingAccounts/00D010-012345-000000 |
| 0145D3-012345-012345 | billingAccounts/00D010-012345-000000 |
| 014F17-012345-012345 | billingAccounts/00D010-012345-000000 |
| 015116-012345-012345 | billingAccounts/00D010-012345-000000 |
| 0151FE-012345-012345 | billingAccounts/00D010-012345-000000 |
| 015574-012345-012345 | billingAccounts/00D010-012345-000000 |
| 016C91-012345-012345 | billingAccounts/00D010-012345-000000 |
| 017047-012345-012345 | billingAccounts/00D010-012345-000000 |
| 0180B0-012345-012345 | billingAccounts/00D010-012345-000000 |
| 018C5C-012345-012345 | billingAccounts/00D010-012345-000000 |
| 018D0A-012345-012345 | billingAccounts/00D010-012345-000000 |
| 01A1D3-012345-012345 | billingAccounts/00D010-012345-000000 |
| 01BAE2-012345-012345 | billingAccounts/00D010-012345-000000 |
| 01C9B8-012345-012345 | billingAccounts/00D010-012345-000000 |
| 01C9CA-012345-012345 | billingAccounts/00D010-012345-000000 |
| 01DE4B-012345-012345 | billingAccounts/00D010-012345-000000 |
| 01E25A-012345-012345 | billingAccounts/00D010-012345-000000 |
| 01E62F-012345-012345 | billingAccounts/00D010-012345-000000 |
| 014D64-012345-012345 | billingAccounts/01F621-012345-555555 |
| 016524-012345-012345 | billingAccounts/01F621-012345-555555 |
| 01DE7B-012345-012345 | billingAccounts/01F621-012345-555555 |
| 01FD31-012345-012345 | billingAccounts/01F621-012345-555555 |
+----------------------+--------------------------------------+

[misraved]

Hey @pdecat, could you please resolve the merge conflicts in this PR?

Also, is the PR waiting on some discussion? Apologies if I missed any thread.

[pdecat]

Rebased on main and marked as ready.

Also, is the PR waiting on some discussion? Apologies if I missed any thread.

Not from my point of view, I just wanted to emphasize that this is somewhat a breaking change.

[cbruno10]

@pdecat Can you please describe a bit more in detail what the breaking changes are? What types of queries would be affected?

[pdecat]

This PR changes the table behavior by returning all billing accounts accessible by the connection's credentials (multiple rows).
Previously, only the billing account attached to the project that is configured for the connection was returned (always at most one row).

[cbruno10]

@pdecat So if I understand correctly, this PR:

• Updates the gcp_billing_account table so it returns all billing accounts the connection/credentials has access to, instead of just the current project in the connection (breaking change).
• Removes the project column from the gcp_billing_account table (breaking change).
• Adds the billing_information column to the gcp_organization_project and gcp_project tables (new feature).

Some questions:

• Does the https://cloud.google.com/billing/docs/reference/rest/v1/organizations.billingAccounts/list API call return a subset (limited by organization) of https://cloud.google.com/billing/docs/reference/rest/v1/billingAccounts/list (which is the API call this PR is using I believe)? It seems like using https://cloud.google.com/billing/docs/reference/rest/v1/billingAccounts/list is the correct list API call, but just double checking.
• Are billing subaccounts returned by the gcp_billing_account table by default? The docs say you can set these in the filter(https://cloud.google.com/billing/docs/reference/rest/v1/billingAccounts/list) or there's a separate API call for them (https://cloud.google.com/billing/docs/reference/rest/v1/billingAccounts.subAccounts/list), but if they're just billing accounts in shape, then I'm not sure we'd need to use the separate API call.

Also, for a future addition, I think we could add a gcp_billing_account_project table that uses https://cloud.google.com/billing/docs/reference/rest/v1/billingAccounts.projects/list, which would allow users to view all projects associated with a particular billing account. This would be a separate table as opposed to a projects column in the gcp_billing_account_project table since it has paging. But, I'm not suggesting we add that as part of this PR and we can wait until someone has a particular use case.

[pdecat]

@pdecat So if I understand correctly, this PR:

• Updates the gcp_billing_account table so it returns all billing accounts the connection/credentials has access to, instead of just the current project in the connection (breaking change).
• Removes the project column from the gcp_billing_account table (breaking change).
• Adds the billing_information column to the gcp_organization_project and gcp_project tables (new feature).

Exactly. Forgot about the second breaking change in my previous message.

Some questions:

• Does the https://cloud.google.com/billing/docs/reference/rest/v1/organizations.billingAccounts/list API call return a subset (limited by organization) of https://cloud.google.com/billing/docs/reference/rest/v1/billingAccounts/list (which is the API call this PR is using I believe)? It seems like using https://cloud.google.com/billing/docs/reference/rest/v1/billingAccounts/list is the correct list API call, but just double checking.

Good question. The only apparent difference I could find between the two endpoints is that the optional parent parameter is passed in the path in the former instead of a query parameter in the latter.

• Are billing subaccounts returned by the gcp_billing_account table by default? The docs say you can set these in the filter(https://cloud.google.com/billing/docs/reference/rest/v1/billingAccounts/list) or there's a separate API call for them (https://cloud.google.com/billing/docs/reference/rest/v1/billingAccounts.subAccounts/list), but if they're just billing accounts in shape, then I'm not sure we'd need to use the separate API call.

With this PR, yes. I've just added an example query with results in this PR's description. No need for a separate API call I believe.

Also, for a future addition, I think we could add a gcp_billing_account_project table that uses https://cloud.google.com/billing/docs/reference/rest/v1/billingAccounts.projects/list, which would allow users to view all projects associated with a particular billing account. This would be a separate table as opposed to a projects column in the gcp_billing_account_project table since it has paging. But, I'm not suggesting we add that as part of this PR and we can wait until someone has a particular use case.

👍