diff --git a/external-storage-uri.md b/external-storage-uri.md index cb99bcb77018f..e44238d72c030 100644 --- a/external-storage-uri.md +++ b/external-storage-uri.md @@ -15,6 +15,8 @@ The basic format of the URI is as follows: ## Amazon S3 URI format + + - `scheme`: `s3` - `host`: `bucket name` - `parameters`: @@ -48,12 +50,42 @@ tiup cdc:v7.5.0 cli changefeed create \ --config=cdc_csv.toml ``` -The following is an example of an Amazon S3 URI for [`IMPORT INTO`](/sql-statements/sql-statement-import-into.md). In this example, you need to specify a specific filename `test.csv`. + + + + +- `scheme`: `s3` +- `host`: `bucket name` +- `parameters`: + + - `access-key`: Specifies the access key. + - `secret-access-key`: Specifies the secret access key. + - `session-token`: Specifies the temporary session token. + - `use-accelerate-endpoint`: Specifies whether to use the accelerate endpoint on Amazon S3 (defaults to `false`). + - `endpoint`: Specifies the URL of custom endpoint for S3-compatible services (for example, ``). + - `force-path-style`: Use path style access rather than virtual hosted style access (defaults to `true`). + - `storage-class`: Specifies the storage class of the uploaded objects (for example, `STANDARD` or `STANDARD_IA`). + - `sse`: Specifies the server-side encryption algorithm used to encrypt the uploaded objects (value options: empty, `AES256`, or `aws:kms`). + - `sse-kms-key-id`: Specifies the KMS ID if `sse` is set to `aws:kms`. + - `acl`: Specifies the canned ACL of the uploaded objects (for example, `private` or `authenticated-read`). + - `role-arn`: To allow TiDB Cloud to access Amazon S3 data using a specified [IAM role](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles.html), specify the corresponding [Amazon Resource Name (ARN)](https://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html) of the IAM role using the `role-arn` URL query parameter, such as `arn:aws:iam::888888888888:role/my-role`. + + > **Note:** + > + > - To automatically create an IAM role, navigate to the **Import Data from Amazon S3** page of your cluster in the [TiDB Cloud console](https://tidbcloud.com/), fill in the **Folder URI** field, click **Click here to create new one with AWS CloudFormation** under the **Role ARN** field, and then follow the on-screen instructions in the **Add New Role ARN** dialog. + > - If you have any trouble creating the IAM role using AWS CloudFormation, click **click Having trouble? Create Role ARN manually** in the **Add New Role ARN** dialog to get the TiDB Cloud Account ID and TiDB Cloud External ID, and then follow the steps in [Configure Amazon S3 access using a Role ARN](/tidb-cloud/dedicated-external-storage.md#configure-amazon-s3-access-using-a-role-arn) to create the role manually. When configuring the IAM role, make sure to enter the TiDB Cloud account ID in the **Account ID** field and select **Require external ID** to protect against [confused deputy attacks](https://docs.aws.amazon.com/IAM/latest/UserGuide/confused-deputy.html). + > - To enhance security, you can reduce the valid duration of the IAM role by configuring a shorter **Max session duration**. For more information, see [Update the maximum session duration for a role](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_update-role-settings.html#id_roles_update-session-duration) in AWS documentation. + + - `external-id`: To allow TiDB Cloud to access Amazon S3 data, you must specify the TiDB Cloud External ID using the `external-id` parameter. You can get the TiDB Cloud External ID from the **Add New Role ARN** dialog in the [TiDB Cloud console](https://tidbcloud.com/). For more information, see [Configure Amazon S3 access using a Role ARN](/tidb-cloud/dedicated-external-storage.md#configure-amazon-s3-access-using-a-role-arn). + +The following is an example of an Amazon S3 URI for [`BACKUP`](/sql-statements/sql-statement-backup.md) and [`RESTORE`](/sql-statements/sql-statement-restore.md). In this example, you need to specify a specific file path `testfolder`. ```shell -s3://external/test.csv?access-key=${access-key}&secret-access-key=${secret-access-key} +s3://external/testfolder?access-key=${access-key}&secret-access-key=${secret-access-key} ``` + + ## GCS URI format - `scheme`: `gcs` or `gs` @@ -64,12 +96,16 @@ s3://external/test.csv?access-key=${access-key}&secret-access-key=${secret-acces - `storage-class`: Specifies the storage class of the uploaded objects (for example, `STANDARD` or `COLDLINE`) - `predefined-acl`: Specifies the predefined ACL of the uploaded objects (for example, `private` or `project-private`) + + The following is an example of a GCS URI for TiDB Lightning and BR. In this example, you need to specify a specific file path `testfolder`. ```shell gcs://external/testfolder?credentials-file=${credentials-file-path} ``` + + The following is an example of a GCS URI for [`IMPORT INTO`](/sql-statements/sql-statement-import-into.md). In this example, you need to specify a specific filename `test.csv`. ```shell diff --git a/sql-statements/sql-statement-backup.md b/sql-statements/sql-statement-backup.md index f7316eb078f1b..c58bbf71092c9 100644 --- a/sql-statements/sql-statement-backup.md +++ b/sql-statements/sql-statement-backup.md @@ -112,18 +112,8 @@ BR supports backing up data to S3 or GCS: BACKUP DATABASE `test` TO 's3://example-bucket-2020/backup-05/?access-key={YOUR_ACCESS_KEY}&secret-access-key={YOUR_SECRET_KEY}'; ``` - - The URL syntax is further explained in [URI Formats of External Storage Services](/external-storage-uri.md). - - - - -The URL syntax is further explained in [external storage URI](https://docs.pingcap.com/tidb/stable/external-storage-uri). - - - When running on cloud environment where credentials should not be distributed, set the `SEND_CREDENTIALS_TO_TIKV` option to `FALSE`: {{< copyable "sql" >}} diff --git a/sql-statements/sql-statement-restore.md b/sql-statements/sql-statement-restore.md index 415c3d2efe358..f3dbbad72dc45 100644 --- a/sql-statements/sql-statement-restore.md +++ b/sql-statements/sql-statement-restore.md @@ -103,18 +103,8 @@ BR supports restoring data from S3 or GCS: RESTORE DATABASE * FROM 's3://example-bucket-2020/backup-05/'; ``` - - The URL syntax is further explained in [URI Formats of External Storage Services](/external-storage-uri.md). - - - - -The URL syntax is further explained in [external storage URI](https://docs.pingcap.com/tidb/stable/external-storage-uri). - - - When running on cloud environment where credentials should not be distributed, set the `SEND_CREDENTIALS_TO_TIKV` option to `FALSE`: {{< copyable "sql" >}}