Skip to content

external storage url in tidb cloud #21058

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Conversation

dulao5
Copy link

@dulao5 dulao5 commented May 27, 2025

First-time contributors' checklist

What is changed, added or deleted? (Required)

Append external storage url document to tidb cloud.

(only for TiDB Cloud)

  • TOC-tidb-cloud.md
  • external-storage-uri.md
  • sql-statements/sql-statement-backup.md
  • sql-statements/sql-statement-restore.md

Which TiDB version(s) do your changes apply to? (Required)

Tips for choosing the affected version(s):

By default, CHOOSE MASTER ONLY so your changes will be applied to the next TiDB major or minor releases. If your PR involves a product feature behavior change or a compatibility change, CHOOSE THE AFFECTED RELEASE BRANCH(ES) AND MASTER.

For details, see tips for choosing the affected versions.

  • master (the latest development version)
  • v9.0 (TiDB 9.0 versions)
  • v8.5 (TiDB 8.5 versions)
  • v8.4 (TiDB 8.4 versions)
  • v8.3 (TiDB 8.3 versions)
  • v8.1 (TiDB 8.1 versions)
  • v7.5 (TiDB 7.5 versions)
  • v7.1 (TiDB 7.1 versions)
  • v6.5 (TiDB 6.5 versions)
  • v6.1 (TiDB 6.1 versions)
  • v5.4 (TiDB 5.4 versions)

What is the related PR or file link(s)?

  • This PR is translated from:
  • Other reference link(s):

Do your changes match any of the following descriptions?

  • Delete files
  • Change aliases
  • Need modification after applied to another branch
  • Might cause conflicts after applied to another branch

@CLAassistant
Copy link

CLAassistant commented May 27, 2025

CLA assistant check
All committers have signed the CLA.

Copy link

ti-chi-bot bot commented May 27, 2025

Welcome @dulao5!

It looks like this is your first PR to pingcap/docs 🎉.

I'm the bot to help you request reviewers, add labels and more, See available commands.

We want to make sure your contribution gets all the attention it needs!



Thank you, and welcome to pingcap/docs. 😃

@ti-chi-bot ti-chi-bot bot added contribution This PR is from a community contributor. first-time-contributor Indicates that the PR was contributed by an external member and is a first-time contributor. needs-ok-to-test Indicates a PR created by contributors and need ORG member send '/ok-to-test' to start testing. labels May 27, 2025
Copy link

ti-chi-bot bot commented May 27, 2025

Hi @dulao5. Thanks for your PR.

I'm waiting for a pingcap member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@ti-chi-bot ti-chi-bot bot added missing-translation-status This PR does not have translation status info. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels May 27, 2025
Copy link

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hello @dulao5, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

Summary of Changes

Hello team, gemini-code-assist here to provide a summary of this pull request. This PR focuses on enhancing the documentation related to external storage URIs, specifically tailoring content for TiDB Cloud users and integrating the new documentation into the relevant sections of the TiDB Cloud documentation.

Highlights

  • TiDB Cloud Documentation: Adds a new document (external-storage-uri.md) specifically covering URI formats for external storage services, with content tailored for TiDB Cloud.
  • Table of Contents Update: Integrates the new external storage URI document into the TiDB Cloud Table of Contents (TOC-tidb-cloud.md).
  • Platform-Specific Content: Uses <CustomContent> tags in external-storage-uri.md to differentiate between information relevant to standard TiDB and TiDB Cloud, particularly regarding S3 parameters like session-token, role-arn, and external-id.
  • Backup/Restore Documentation Links: Simplifies the links to the external storage URI documentation within the BACKUP and RESTORE SQL statement documents by removing platform-specific wrappers.

Changelog

Click here to see the changelog
  • TOC-tidb-cloud.md
    • Added a link to the new 'URI Formats of External Storage Services' document under the 'Best Practices' section (line 695).
  • external-storage-uri.md
    • Wrapped the description of the session-token parameter for BR in <CustomContent platform="tidb"> tags (lines 24-26).
    • Added a description of the session-token parameter specifically for TiDB Cloud, wrapped in <CustomContent platform="tidb-cloud"> tags (lines 28-30).
    • Wrapped the descriptions of role-arn and external-id parameters for standard TiDB in <CustomContent platform="tidb"> tags (lines 38-41).
    • Added a new section for TiDB Cloud specific S3 parameters (role-arn, external-id) including important notes on IAM role configuration and a link to the configuration guide, wrapped in <CustomContent platform="tidb-cloud"> tags (lines 60-67).
    • Removed a general S3 example URI for IMPORT INTO (lines 51-55 in original).
    • Added an S3 example URI for BACKUP and RESTORE specifically for TiDB Cloud, wrapped in <CustomContent platform="tidb-cloud"> tags (lines 69-73).
  • sql-statements/sql-statement-backup.md
    • Removed the <CustomContent platform="tidb"> and <CustomContent platform="tidb-cloud"> wrappers around the link to the external storage URI document (lines 115-125 in original, now lines 115-116).
  • sql-statements/sql-statement-restore.md
    • Removed the <CustomContent platform="tidb"> and <CustomContent platform="tidb-cloud"> wrappers around the link to the external storage URI document (lines 106-116 in original, now lines 106-107).
Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in issue comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist is currently in preview and may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments to provide feedback.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution.

Copy link

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request effectively adds documentation for external storage URI formats specific to TiDB Cloud, particularly for Amazon S3. The introduction of <CustomContent> tags to differentiate between TiDB self-hosted and TiDB Cloud is a good approach for clarity. The security note regarding IAM role configuration for S3 access in TiDB Cloud is a valuable addition.

The refactoring in sql-statement-backup.md and sql-statement-restore.md to use a single, now platform-aware, link to external-storage-uri.md simplifies those documents and improves maintainability.

Overall, the changes enhance the documentation's accuracy and usefulness for TiDB Cloud users. I have a couple of minor suggestions for further improvement.

Summary of Findings

  • Wording conciseness: In external-storage-uri.md, the phrase "must need to specify" can be shortened to "must specify" for better conciseness.
  • Example Clarity for TiDB Cloud S3 URI: In external-storage-uri.md, the S3 URI example for TiDB Cloud uses access keys. Consider if an example using role-arn and external-id would be beneficial, given the preceding text emphasizes this method for security.
  • Documentation Structure Improvement: The changes in sql-statement-backup.md and sql-statement-restore.md to consolidate platform-specific URI information into external-storage-uri.md is a good refactoring that improves maintainability and clarity.

Merge Readiness

The pull request is in good shape and significantly improves the documentation for external storage URIs, especially for TiDB Cloud users. The changes are clear, and the added security information is valuable. The minor suggestions provided are for enhancing conciseness and example clarity, and can be addressed at the author's discretion. I am not authorized to approve pull requests, so please ensure it undergoes the standard review and approval process. Based on my review, after considering the minor points, the PR should be ready for merging.

Comment on lines 69 to 73
The following is an example of an Amazon S3 URI for [`BACKUP`](/sql-statements/sql-statement-backup.md) and [`RESTORE`](/sql-statements/sql-statement-restore.md). In this example, you need to specify a specific file path `testfolder`.

```shell
gcs://external/testfolder?credentials-file=${credentials-file-path}
s3://external/testfolder?access-key=${access-key}&secret-access-key=${secret-access-key}
```

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

low

The example S3 URI for BACKUP and RESTORE in the TiDB Cloud context (lines 71-73) uses access-key and secret-access-key. Given the strong emphasis on using role-arn and external-id for TiDB Cloud security in the preceding paragraphs (lines 61-67), would it be beneficial to also provide an example URI that showcases the role-arn and external-id parameters? This could help reinforce the recommended practice for TiDB Cloud users.

Alternatively, if the intention is to show that both methods are supported, perhaps a brief note clarifying this could be added. The linked document /config-s3-and-gcs-access.md does cover both methods, so this is more of a suggestion for enhancing direct clarity within this page.

qiancai and others added 4 commits May 28, 2025 09:27
@qiancai qiancai force-pushed the release-8.1-external-storage-url-in-tidb-cloud branch from e0cec8b to 91859c1 Compare May 28, 2025 02:20
@qiancai qiancai self-assigned this May 28, 2025
@BornChanger
Copy link
Contributor

/cc @benmeadowcroft

@ti-chi-bot ti-chi-bot bot requested a review from benmeadowcroft May 28, 2025 05:45
@qiancai qiancai force-pushed the release-8.1-external-storage-url-in-tidb-cloud branch from 94f9b5e to c862f91 Compare May 28, 2025 08:26
@qiancai qiancai added area/tidb-cloud This PR relates to the area of TiDB Cloud. translation/no-need No need to translate this PR. needs-cherry-pick-master Should cherry pick this PR to master branch. needs-cherry-pick-release-8.5 Should cherry pick this PR to release-8.5 branch. and removed missing-translation-status This PR does not have translation status info. labels May 28, 2025
@qiancai
Copy link
Collaborator

qiancai commented Jun 3, 2025

Hi @benmeadowcroft, I've removed the unnecessary sentence "For backup and restore, this parameter is supported starting from v7.6.0." for TiDB Cloud in the following commits. Would you please check and approve this PR if no further change is needed? Thanks.

https://github.com/pingcap/docs/pull/21058/files/c862f916a953083fec4bf101e495438e6600b9b9..62a4fd8d25b086010f68a3e4cce053de0d1b9d34

Copy link

ti-chi-bot bot commented Jun 4, 2025

@benmeadowcroft: adding LGTM is restricted to approvers and reviewers in OWNERS files.

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@qiancai
Copy link
Collaborator

qiancai commented Jun 6, 2025

/ok-to-test

@ti-chi-bot ti-chi-bot bot added ok-to-test Indicates a PR is ready to be tested. and removed needs-ok-to-test Indicates a PR created by contributors and need ORG member send '/ok-to-test' to start testing. labels Jun 6, 2025
@ti-chi-bot ti-chi-bot bot added the needs-1-more-lgtm Indicates a PR needs 1 more LGTM. label Jun 6, 2025
Copy link

ti-chi-bot bot commented Jun 6, 2025

[LGTM Timeline notifier]

Timeline:

  • 2025-06-06 01:21:52.440973232 +0000 UTC m=+577598.643318498: ☑️ agreed by qiancai.

@qiancai
Copy link
Collaborator

qiancai commented Jun 6, 2025

/approve

@qiancai qiancai added the lgtm label Jun 6, 2025
@ti-chi-bot ti-chi-bot bot added the approved label Jun 6, 2025
@qiancai
Copy link
Collaborator

qiancai commented Jun 6, 2025

/approve

Copy link

ti-chi-bot bot commented Jun 6, 2025

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: qiancai

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ti-chi-bot ti-chi-bot bot merged commit 3af1e10 into pingcap:release-8.1 Jun 6, 2025
9 checks passed
@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created to branch master: #21133.

ti-chi-bot pushed a commit to ti-chi-bot/docs that referenced this pull request Jun 6, 2025
ti-chi-bot pushed a commit to ti-chi-bot/docs that referenced this pull request Jun 6, 2025
@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created to branch release-8.5: #21134.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved area/tidb-cloud This PR relates to the area of TiDB Cloud. contribution This PR is from a community contributor. first-time-contributor Indicates that the PR was contributed by an external member and is a first-time contributor. lgtm needs-1-more-lgtm Indicates a PR needs 1 more LGTM. needs-cherry-pick-master Should cherry pick this PR to master branch. needs-cherry-pick-release-8.5 Should cherry pick this PR to release-8.5 branch. ok-to-test Indicates a PR is ready to be tested. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. translation/no-need No need to translate this PR.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants