Clarify what the requestor_pays flag does #3431

yuvipanda · 2023-11-16T14:52:10Z

Reference: https://2i2c.slack.com/archives/C055A1J1DRP/p1700050896750479

GeorgianaElena · 2024-02-14T14:10:26Z

I just pushed a small commit updating the docs as well and believe it might be ready for review. Did you want to do add anything else to this @yuvipanda ?

yuvipanda

One documentation change, but otherwise this can be merged!

docs/topic/features.md

GeorgianaElena · 2024-02-15T10:33:08Z

docs/howto/features/buckets.md

+      "public_access": true,
+      "requester_pays": true


@yuvipanda, do you think requester_pays has sense outside of a public access? Should these two flags be documented togheter?

Ah, great question @GeorgianaElena.

Right now, we don't actually have a terraform variable called requstor_pays (see list for gcp and aws for what we do have). So we currently don't have a way to mark a bucket as requestor pays via our terraform.

It's not super hard to implement (pass through to gcp config or this aws resource), so we should implement it if / when a community asks to make one of their buckets into requestor_pays.

It may also be helpful to rename public_access to public_anonymous_access - basically, it currently means that anyone can access that bucket, without needing to provide any authentication at all. But since requestor_pays requires that the entity requesting this be part of an AWS or GCP project, it by definition is authenticated - so GCP or AWS knows who to charge! This 'authentication' is what our hubs provide when we turn on the allow_access_to_requestor_pays_buckets flag.

So public_access (or public_anonymous_access) is actually mutually exclusive with requestor_pays. The first is fully unauthenticated, while the second requires authentication.

I hope this is helpful!

Extremely helpful @yuvipanda! Thank you very much for taking the time to clarify this <3

docs/topic/features.md

sgibson91 · 2024-02-19T12:35:17Z

docs/howto/features/buckets.md

+Some hubs want to expose a particular bucket to the broad internet
+but not be billed for the charges associated with making and executing the
+requests on this bucket.
+
+By enabling the [Requester Pays flag](https://cloud.google.com/storage/docs/using-requester-pays#using),
+the requesters are required to include a billing project in their requests,
+which will mean that the billing will happen on the requester's project.
+
+Enabling Requester Pays is useful, for example, if the communities have a lot
+of data that they want to make available to others, but don't want to be
+charged for their access to that data.


... but this section sounds like the host hub is providing access to buckets inside their hub. Which is it? Can it be both? In what scenario is the bucket inside and what scenario is the bucket outside?

Yes, it can be both. Requestor Pays is a feature that a bucket can have.

If buckets outside the project have this flag, then we need to:

set hub_cloud_permissions.allow_access_to_requestor_pays_buckets in the terraform config of a cluster (this config is simply called requestor_pays without this PR, which was the source of confusion for me).

This is to allow them to be charged on their project for access of such outside buckets

But buckets that we set for communities, inside their projects can also have this feature enabled on them, which means that other people outside will be charged for their usage.

This can be enabled by setting user_buckets.<bucket_name>.requester_pays to true in the terraform config of the cluster.

I don't think we have any buckets like this yet, but was recently requested by leap in https://2i2c.freshdesk.com/a/tickets/1316

Does the above has more sense @sgibson91? I will update the language in the docs as well to make the split and differences more clear.

Yeah, thank you - though I'm wondering if it should be hub_cloud_permissions.allow_access_to_external_requester_pays_buckets to be crystal clear? Though I realise it's getting kinda long.

I just want to say that https://github.com/2i2c-org/infrastructure/pull/3431/files#r1495651158 is an accurate description of the situation! Only difference is that right now,

This can be enabled by setting user_buckets.<bucket_name>.requester_pays to true in the terraform config of the cluster.

is not true because we don't have that implemented.

yuvipanda

Let's simplify this PR to:

Renaming the existing requestor_pays to allow_access_to_external_requestor_pays_buckets (as @sgibson91 suggested in https://github.com/2i2c-org/infrastructure/pull/3431/files#r1495801966) for clarity
Document what exactly that does in clearer terms. I think your words here (https://github.com/2i2c-org/infrastructure/pull/3431/files#diff-022e962ded687634ecdad02e21584d886b7e0ed8002cd91e2e9b94f5637c289cR58) are great, and maybe can be ported to the features page?
Open an issue to track the new feature request of marking a bucket we manage as requestor_pays to the outside world.

I hope this helps! I can't 'request changes' because this is my own PR, but thanks for picking this up, @GeorgianaElena

terraform/aws/variables.tf

docs/howto/features/buckets.md

yuvipanda · 2024-02-22T22:34:07Z

docs/howto/features/buckets.md

+Some hubs want to expose a particular bucket to the broad internet
+but not be billed for the charges associated with making and executing the
+requests on this bucket.
+
+By enabling the [Requester Pays flag](https://cloud.google.com/storage/docs/using-requester-pays#using),
+the requesters are required to include a billing project in their requests,
+which will mean that the billing will happen on the requester's project.
+
+Enabling Requester Pays is useful, for example, if the communities have a lot
+of data that they want to make available to others, but don't want to be
+charged for their access to that data.


I just want to say that https://github.com/2i2c-org/infrastructure/pull/3431/files#r1495651158 is an accurate description of the situation! Only difference is that right now,

This can be enabled by setting user_buckets.<bucket_name>.requester_pays to true in the terraform config of the cluster.

is not true because we don't have that implemented.

…ate docs language for clarity

… being supported on aws

GeorgianaElena · 2024-02-28T12:52:05Z

I've updated the PR per instructions and the terraform config of the clusters. I've also opened the follow-up issue, so this should be ready for review again 🚀

docs/howto/features/cloud-access.md

terraform/gcp/variables.tf

sgibson91

LGTM!

Co-authored-by: Sarah Gibson <[email protected]>

GeorgianaElena · 2024-02-28T14:18:57Z

Yuhuuu! Thank you @sgibson91!

GeorgianaElena · 2024-02-28T14:19:54Z

And thank you @yuvipanda for shining light over this confusing topic 🎉

yuvipanda · 2024-02-29T00:50:52Z

This is great work @GeorgianaElena!

github-actions bot assigned yuvipanda Nov 16, 2023

yuvipanda and others added 2 commits February 14, 2024 11:59

Clarify what the requestor_pays flag does

5e8825d

Update requester_pays docs with clarifications

38e0d89

GeorgianaElena force-pushed the req-pay branch from 274c57d to 38e0d89 Compare February 14, 2024 14:08

GeorgianaElena marked this pull request as ready for review February 14, 2024 14:09

GeorgianaElena requested a review from a team as a code owner February 14, 2024 14:09

yuvipanda commented Feb 14, 2024

View reviewed changes

docs/topic/features.md Outdated Show resolved Hide resolved

GeorgianaElena added 2 commits February 15, 2024 12:28

Update the enable/disable instructure to not be overwritable by an apply

fec6d4d

Also provide a technical example of the setup

8cf8947

GeorgianaElena reviewed Feb 15, 2024

View reviewed changes

GeorgianaElena requested a review from a team February 15, 2024 11:57

sgibson91 reviewed Feb 19, 2024

View reviewed changes

GeorgianaElena self-assigned this Feb 20, 2024

yuvipanda commented Feb 22, 2024

View reviewed changes

GeorgianaElena added 10 commits February 27, 2024 13:47

Rename requester_pays again and rm it from aws

90b9ec1

Rename requestor to requester to match google's documentation and upd…

b7fd9a5

…ate docs language for clarity

Rm unexistent feature docs

e9023d7

Reference howto docs

df35174

Rename var in one more place

92d1bba

Rm quote

59fc657

Rename docs ref

60b58e8

Rename requester in one more terraform config

2a04cbc

Rm undifinded requestor_pays var from aws terraform config

f8c23e0

Rename terraform var in projects

d703d8b

GeorgianaElena mentioned this pull request Feb 28, 2024

[Support] Implement Requester Pays access on buckets that we manage #3746

Open

4 tasks

GeorgianaElena added 3 commits February 28, 2024 14:08

Link to issue

2483813

Merge branch 'master' into req-pay

ee4a7e7

Add warning about allow_access_to_external_requester_pays_buckets not…

7bdb407

… being supported on aws

sgibson91 reviewed Feb 28, 2024

View reviewed changes

docs/howto/features/cloud-access.md Outdated Show resolved Hide resolved

sgibson91 reviewed Feb 28, 2024

View reviewed changes

terraform/gcp/variables.tf Outdated Show resolved Hide resolved

sgibson91 approved these changes Feb 28, 2024

View reviewed changes

GeorgianaElena and others added 2 commits February 28, 2024 16:08

Move the warning higher-up

80143da

Rephrase for clarity

c544d97

Co-authored-by: Sarah Gibson <[email protected]>

GeorgianaElena merged commit 31796fa into 2i2c-org:master Feb 28, 2024
3 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Clarify what the requestor_pays flag does #3431

Clarify what the requestor_pays flag does #3431

yuvipanda commented Nov 16, 2023 •

edited by GeorgianaElena

Loading

GeorgianaElena commented Feb 14, 2024

yuvipanda left a comment

GeorgianaElena Feb 15, 2024

yuvipanda Feb 21, 2024

GeorgianaElena Feb 28, 2024

sgibson91 Feb 19, 2024

GeorgianaElena Feb 20, 2024

GeorgianaElena Feb 20, 2024

sgibson91 Feb 20, 2024 •

edited

Loading

yuvipanda Feb 22, 2024

yuvipanda left a comment •

edited by GeorgianaElena

Loading

yuvipanda Feb 22, 2024

GeorgianaElena commented Feb 28, 2024

sgibson91 left a comment

GeorgianaElena commented Feb 28, 2024

GeorgianaElena commented Feb 28, 2024

yuvipanda commented Feb 29, 2024

Clarify what the requestor_pays flag does #3431

Clarify what the requestor_pays flag does #3431

Conversation

yuvipanda commented Nov 16, 2023 • edited by GeorgianaElena Loading

GeorgianaElena commented Feb 14, 2024

yuvipanda left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sgibson91 Feb 20, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yuvipanda left a comment • edited by GeorgianaElena Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

GeorgianaElena commented Feb 28, 2024

sgibson91 left a comment

Choose a reason for hiding this comment

GeorgianaElena commented Feb 28, 2024

GeorgianaElena commented Feb 28, 2024

yuvipanda commented Feb 29, 2024

yuvipanda commented Nov 16, 2023 •

edited by GeorgianaElena

Loading

sgibson91 Feb 20, 2024 •

edited

Loading

yuvipanda left a comment •

edited by GeorgianaElena

Loading