Provide a single source of truth for wether a community's GCP cloud costs is paid by us or not

[yuvipanda]

Context

There's a gcp.billing.paid_by_us boolean field in our cluster.yaml that tells us whether a GCP project is being paid for with the 2i2c credit card or the account is provided to us by the community. While this is documented, it's not well documented. Partnerships has some automation that looks at various hubs we run and scrapes it into airtable, but that doesn't pick up this field.

Proposal

• Go through all current clusters, and validate the gcp.billing.paid_by_us field is accurate (see Provide a single source of truth for wether a community's GCP cloud costs is paid by us or not #3711 (comment) for broad information on how)
  • Done via Add aws.billing.paid_by_us and initialize AWS based cluster.yaml files #3872 and leap: change cluster.yaml's gcp.billing.paid_by_us to false #3873
• Add an item in new hub turnup to verify this information is accurate Document setting paid_by_us when creating cluster.yaml file #3853
  • Done via docs: ensure docs capture details about paid_by_us field #3946
• Document this as the single source of truth #4068
  • Done via docs: indicate .billing.paid_by_us as source of truth #3959

Edit: Added to proposal

• Ensure our "phase 1" about cluster creation ensures we have info about who should pay etc so that we set this up correctly initially.
  • Until there is a version controlled place to update "phase 1" which is a work in progress, this item is hard to get started with.

Edit: Removed from proposal

• Modify the automation that partnerships has to look for this boolean value and represent it in airtable

Updates and actions

No response

[damianavila]

@haroldcampbell, I think this one is well-specified and I see a clear definition of done, do you agree?
If that is not the case, can you drop your thoughts/requests so we can specify it better?
Thanks!

[yuvipanda]

Just a note on how someone can verify if a particular project if being paid for by 2i2c or not.

1. On GCP, if the billing account attached to the project is https://console.cloud.google.com/billing/0157F7-E3EA8C-25AC3C?project=two-eye-two-see, we are paying for it. If not, we are not paying for it.
2. On AWS, if it's not part of our SSO (and hence accessible by starting at https://2i2c.awsapps.com/), it's not paid for by us.

[haroldcampbell]

@damianavila this feels well-defined

[consideRatio]

On AWS, if it's not part of our SSO (and hence accessible by starting at https://2i2c.awsapps.com/), it's not paid for by us.

@yuvipanda there is a gcp.billing.paid_by_us, but there is no aws equivalent. Did you have something in mind about this? Is the single source of truth for GCP projects gcp.billing.paid_by_us and then for AWS projects its something else?

Currently, the only information about a AWS account is via a comment like this: provider: aws # https://2i2c.awsapps.com/start#/.

For reference, it can look like this in cluster.yaml files:

GCP clusters

gcp:
  key: enc-deployer-credentials.secret.json
  project: hhmi-398911
  cluster: hhmi-cluster
  zone: us-west2
  billing:
    paid_by_us: true
    bigquery:
      project: two-eye-two-see
      dataset: cloud_costs
      billing_id: 0157F7-E3EA8C-25AC3C

AWS clusters

aws:
  key: enc-deployer-credentials.secret.json
  clusterType: eks
  clusterName: 2i2c-aws-us
  region: us-west-2

Conclusion about cluster.yaml metadata

• We lack non-comment metadata about the AWS account currently.
• We lack metadata on AWS billing - but we currently have it tied to being managed by our management account we can sign into via https://2i2c.awsapps.com/start#/

[consideRatio]

On AWS, if it's not part of our SSO (and hence accessible by starting at https://2i2c.awsapps.com/), it's not paid for by us.

If and only if they are part of our SSO - they are paid by us? Is there / could there be a account paid by someone else part of our SSO/management account?

[yuvipanda]

Great questions, @consideRatio.

paid_by_us should be added to AWS clusters as well, if it currently is only present for GCP clusters

Complete way to check if we're paying for an AWS account:

Way 1:

1. In our AWS SSO, select 2i2c-sandbox. Click "Administrator Access" to get web console access.
2. Go to https://us-east-1.console.aws.amazon.com/organizations/v2/home/accounts ('AWS Accounts' in the search bar)
3. If the account is here, we are paying for it.

Way 2:

1. If it's not in our SSO, login to the account. You should have credentials. I'll describe a way to find the account id later in the comment
2. Go to payment preferences -> payment methods. If this is not accessible to you, we aren't paying for it
3. Look at the credit cards present, and see if it has "Chris Holdgraf" as name. If it is, we are paying for it.

My current understanding is that Way 1 should confirm everything. But we can do a round of way 2 as well to make 100% sure. After that, we can amend our documentation to set this information correctly when we create the initial cluster file. I'll take this as an action item for myself.

A consistent way to find the AWS account id for a cluster.yaml file is:

1. Run deployer use-cluster-credentials <cluster-name>
2. Run aws sts get-caller-identity. It should output a small JSON object, with an "Account" key that shows account id.

[yuvipanda]

I've opened #3853.

[consideRatio]

@yuvipanda for LEAP, it said we paid for it - but the VISA card registered didn't seem like CH's - no name were provided by the final digits didn't match either of two cards tied to CH in the AWS 2i2c-sandbox project. So - I'm switching that from true to false. Is this OK? See related PR #3873 to change it.

No other cluster's existing paid_by_us entry had to be changed.

[yuvipanda]

I think this just needs documentation now to be marked as complete?

[consideRatio]

I think this just needs documentation now to be marked as complete?

I'm not sure @yuvipanda - I've failed to figure out how to figure out the details of the last two checkboxes in the PR.

• Modify the automation that partnerships has to look for this boolean value and represent it in airtable

I found airtable automations that I've not before onboarded to what they are about, but I figure they maybe needs to be updated?

Airtable has the following data tables:

• Document this as the single source of truth

I'm not sure for whom and where this should be documented =/

[yuvipanda]

@consideRatio hmm, good question - I too am unaware of how to do anything in Airtable. My suggestion now is:

1. We document this in our engineering documentation as the source of truth. I think this should be in the billing section, and also mentioned wherever the instructions for creating cluster.yaml are
2. For the airtable itself, create a separate issue, and post that in the #partnerships channel. My current understanding is that partnerships is responsible for Airtable automations, and we shouldn't touch that. Posting it there allows them to prioritize it based on their capacity and processes.

I think once these two are done, this can be marked as dover.

[consideRatio]

1. We document this in our engineering documentation as the source of truth. I think this should be in the billing section, and also mentioned wherever the instructions for creating cluster.yaml are

I've opened #3959 about this

2. For the airtable itself, create a separate issue, and post that in the #partnerships channel. My current understanding is that partnerships is responsible for Airtable automations, and we shouldn't touch that. Posting it there allows them to prioritize it based on their capacity and processes.

I didn't end up actionable here, and instead pivoting slightly.

I'm thinking about three places where "paid by us" may be declared:

1. Community contract (actual source of truth)
2. Using community contract info: request to setup community cluster/hub -> we update billing.paid_by_us=true (practical source of truth, needs to mirror the actual source of truth)
3. (optional) automation to update airtable based on a communities associated cluster.yaml and its x.billing.paid_by_us field

I assume a community contract is the actual source of truth, and any community having a hub in a cluster needs that cluster to have ...billing.paid_by_us=true - as we rely on that information as the source of truth - but really its a "mirror" or "secondary replica" of the source of truth being in a binding contract. We could then mirror info in cluster.yaml via airtable automation to some column in an airtable table - but is that helping us ensure there isn't confusion about this?

I perceive the crucial point to be that we ensure the partnership->engineering process to updated so new cluster/hub's always leads to declaring paid_by_us in alignment with the community contract.

I looked at the issue templates, but they are outdated right now - where you have a WIP for the issue template to setup new cluster/hubs? I'm not sure how to ensure that the partnership->engineering is given the information about paid_by_us except to update that.

---

@yuvipanda could we settle for the action point of ensuring partnership's handover to engineering to setup a cluster/hub is providing info that determines paid_by_us definitively?

[consideRatio]

What remains in this issue IMO is:

[...] ensuring partnership's handover to engineering to setup a cluster/hub is providing info that determines paid_by_us definitively

So, I figure this should be a checkbox thing for the docs/issue-template on setup a new cluster aka. phase 1.

Currently, that docs/issue-template is a WIP though and I'm not sure where to patch it.

[consideRatio]

Seeing this, I'll go for a close - that this is part of the phase 1 was the only action point I think remained here: