bbr-backup-tkgi.html.md.erb

---
title: Backing Up TKGI Management Plane Components
owner: TKGI
---

This topic describes how to use BOSH Backup and Restore (BBR) to back up VMware Tanzu Kubernetes Grid Integrated Edition (TKGI) Managment Plane components.  

##<a id="overview"></a> Overview

The BOSH Director, Tanzu Kubernetes Grid Integrated Edition Control Plane, and cluster deployments include custom back up and restore scripts which encapsulate the correct procedure
for backing up and restoring the Director and Control Plane.

BBR orchestrates running the back up and restore scripts and transferring the generated backup artifacts to and from a backup directory.
If configured correctly, BBR can use TLS to communicate securely with back up targets.

* To perform a restore of the BOSH Director, see [Restore the BOSH Director](bbr-restore.html#redeploy-restore-director).
* To perform a restore of the TKGI Control Plane, see [Restore the Tanzu Kubernetes Grid Integrated Edition Control Plane](bbr-restore.html#redeploy-restore-control-plane).

To view the BBR release notes, see the Cloud Foundry documentation, [BOSH Backup and Restore Release Notes](https://docs.cloudfoundry.org/bbr/bbr-rn.html).

## <a id='recs'></a> Recommendations

<%= vars.recommended_by %> recommends:

* Follow the full procedure documented in this topic when creating a backup. This ensures that you always have a consistent backup of Ops Manager and Tanzu Kubernetes Grid Integrated Edition to restore from.

* Back up frequently, especially before upgrading your Tanzu Kubernetes Grid Integrated Edition deployment.

* For BOSH v270.0 and above (currently in <%= vars.platform_name %> 2.7), prune the BOSH blobstore by running `bosh clean-up --all` prior to running a backup of the BOSH director. This removes all unused resources, including packages compiled against older stemcell versions, which can result in a smaller, faster backup of the BOSH Director. For more information see the [`clean-Up`](https://bosh.io/docs/cli-v2/#clean-up) command.

<p class="note"><strong>Note:</strong>The command <code>bosh clean-up --all</code> is a destructive operation and can remove resources that are unused but needed. For example, if an On-Demand Service Broker such as Tanzu Kubernetes Grid Integrated Edition is deployed <strong>and</strong> no service instances have been created, the releases needed to create a service instance will be categorized as unused and removed.</p>

## <a id="prepare"></a> Prepare to Back Up

<%= partial 'preparing-for-bbr' %>

## <a id='backup'></a> Back Up Tanzu Kubernetes Grid Integrated Edition

To back up your Tanzu Kubernetes Grid Integrated Edition environment you must first connect to your jump box before executing `bbr` back up commands.

### <a id='connect-to-jumpbox'></a> Connect to Your Jump Box

You can establish a connection to your jump box in one of the following ways:

* [Connect with SSH](#ssh)
* [Connect with BOSH&#95;ALL&#95;PROXY](#bosh-all-proxy)

For general information about the jump box, see [Installing BOSH Backup and Restore](bbr-install.html).

#### <a id='ssh'></a> Connect with SSH

To connect to your jump box with SSH, do one of the following:

* To use the Ops Manager VM as your jump box:
    1. Log in to the Ops Manager VM. For more information, see
    [Log in to the Ops Manager VM with SSH](https://docs.vmware.com/en/VMware-Tanzu-Operations-Manager/3.0/vmware-tanzu-ops-manager/install-trouble-advanced.html#ssh) 
    in _Advanced Troubleshooting with the BOSH CLI_.  

* To connect to your jump box using the command line:
    1. Run the following command:  

        ```
        ssh -i PATH-TO-KEY JUMP-BOX-USERNAME@JUMP-BOX-ADDRESS
        ```
        Where:  

        * `PATH-TO-KEY` is the local path to your private key for the jump box host.  
        * `JUMP-BOX-USERNAME` is your jump box user name.  
        * `JUMP-BOX-ADDRESS` is the address of the jump box.  

    <p class="note"><strong>Note:</strong> If you connect to your jump box with SSH, you must run the BBR commands
    in the following sections from within your jump box.</p>


#### <a id='bosh-all-proxy'></a> Connect with BOSH&#95;ALL&#95;PROXY

You can use the `BOSH_ALL_PROXY` environment variable to open an SSH tunnel with SOCKS5 to your jump box.
This tunnel enables you to forward requests from your local machine to the BOSH Director through the jump box.
When `BOSH_ALL_PROXY` is set, BBR always uses its value to forward requests to the BOSH Director.

<p class="note"><strong>Note:</strong>
  For the following procedures to work, ensure the SOCKS port is not already in use by a different tunnel or process.</p>

To connect with `BOSH_ALL_PROXY`, do one of the following:

* To establish a tunnel separate from the BOSH CLI:  
    1. Establish the tunnel and make it available on a local port by running the following command:  

        ```
        ssh -4 -D SOCKS-PORT -fNC JUMP-BOX-USERNAME@JUMP-BOX-ADDRESS -i JUMP-BOX-KEY-FILE -o ServerAliveInterval=60
        ```
        Where:  

        * `SOCKS-PORT` is the local SOCKS port.  
        * `JUMP-BOX-USERNAME` is your jump box user name.  
        * `JUMP-BOX-ADDRESS` is the address of the jump box.  
        * `JUMP-BOX-KEY-FILE` is the local SSH private key for accessing the jump box.  

        For example:
        ```console
        $ ssh -4 -D 12345 -fNC jumpbox@203.0.113.0 -i jumpbox.key -o ServerAliveInterval=60
        ```
    1. Provide the BOSH CLI with access to the tunnel through `BOSH_ALL_PROXY` by running the following command:

        ```
        export BOSH_ALL_PROXY=socks5://localhost:SOCKS-PORT
        ```
        Where is `SOCKS-PORT` is your local SOCKS port.  

* To establish a tunnel using the BOSH CLI:  
    1. Provide the BOSH CLI with the necessary SSH credentials to create the
    tunnel by running the following command:  

        ```
        export BOSH_ALL_PROXY=ssh+socks5://JUMP-BOX-USERNAME@JUMP-BOX-ADDRESS:SOCKS-PORT?private_key=JUMP-BOX-KEY-FILE
        ```

        Where:  

        * `JUMP-BOX-USERNAME` is your jump box user name.  
        * `JUMP-BOX-ADDRESS` is the address of the jump box.  
        * `SOCKS-PORT` is your local SOCKS port.  
        * `JUMP-BOX-KEY-FILE` is the local SSH private key for accessing the jump box.  

        For example:
        ```console
        $ export BOSH_ALL_PROXY=ssh+socks5://jumpbox@203.0.113.0:12345?private_key=jumpbox.key
        ```


<p class="note"><strong>Note:</strong> Using <code>BOSH_ALL_PROXY</code> can result in longer
back up and restore times because of network performance degradation. All operations must pass
through the proxy which means moving backup artifacts can be significantly slower.</p>

<div class="note warning"><strong>Warning:</strong> In BBR v1.5.0 and earlier,
  the tunnel created by the BOSH CLI does not include the <code>ServerAliveInterval</code> flag.
  This might result in your SSH connection timing out when transferring large artifacts.
  In BBR v1.5.1, the <code>ServerAliveInterval</code> flag is included.
  For more information,
  see  <a href="https://github.com/cloudfoundry-incubator/bosh-backup-and-restore/releases/tag/v1.5.1">bosh-backup-and-restore v1.5.1</a> on GitHub.
</div>

### <a id='export-opsman-settings'></a> Back Up Ops Manager Installation Settings

To ensure your BBR backup is reliable, frequently export your Ops Manager installation settings as a backup.

There are two ways to export Ops Manager installation settings:

* [Export settings using the Ops Manager UI](#export-via-ui)
* [Export settings using the Ops Manager API](#export-via-api)

<p class="note"><strong>Note</strong>: If you want to automate the back up process,
you can use the Ops Manager API to export your installation settings.</p>

When exporting your installation settings, keep in mind the following:

* Always export your installation settings before following the steps in the
[Restore the BOSH Director](bbr-restore.html#redeploy-restore-director)
section of the *Restoring Tanzu Kubernetes Grid Integrated Edition* topic.

* You can only export Ops Manager installation settings after you have deployed at least once.

* Your Ops Manager settings export is only a backup of Ops Manager configuration settings.
The export is not a backup of your VMs or any external MySQL databases.

* Your Ops Manager settings export is encrypted. Make sure you keep track of your Decryption Passphrase
because this is needed to restore the Ops Manager settings.

#### <a id='export-via-ui'></a> Export Settings Using the Ops Manager UI

To export your Ops Manager installation settings using the Ops Manager UI, perform the following steps:

1. From the **Installation Dashboard** in the Ops Manager interface, click your user name at the top right navigation.
1. Select **Settings**.
1. Select **Export Installation Settings**.
1. Click **Export Installation Settings**.

#### <a id='export-via-api'></a> Export Settings Using the Ops Manager API
To export your Ops Manager installation settings using the Ops Manager API, perform the following steps:

1. To export your installation settings using the Ops Manager API, run the following command:

    ```
    curl https://OPS-MAN-FQDN/api/v0/installation_asset_collection \
    -H "Authorization: Bearer UAA-ACCESS-TOKEN" > installation.zip
    ```
    Where:  

    * `OPS-MAN-FQDN` is the fully-qualified domain name (FQDN) for your Ops Manager deployment.  
    * `UAA-ACCESS-TOKEN` is your UAA access token. For more information, see Access the API.  

### <a id='back-up-director'></a> Back Up the Tanzu Kubernetes Grid Integrated Edition BOSH Director

To back up BOSH Director you will validate your current configuration, then execute the `bbr` back up command.

#### <a id='back-up-director-validate'></a> Validate the Tanzu Kubernetes Grid Integrated Edition BOSH Director

1. To confirm that your BOSH Director is reachable and has the correct BBR scripts, run the following command:

    ```
    bbr director  --host BOSH-DIRECTOR-IP  --username bbr \
    --private-key-path PRIVATE-KEY-FILE  pre-backup-check
    ```
    Where:  

    * `BOSH-DIRECTOR-IP` is the address of the BOSH Director. If the BOSH Director is public, `BOSH-DIRECTOR-IP` is a URL, such as
    `https://my-bosh.xxx.cf-app.com`. Otherwise, this is the internal IP `BOSH-DIRECTOR-IP` which you can retrieve as show in
    [Retrieve the BOSH Director Address](#bosh-address).  
    * `PRIVATE-KEY-FILE` is the path to the private key file that you can create from `Bbr Ssh Credentials` as shown in
    [Download the BBR SSH Credentials](#bbr-ssh-creds).  

    For example:  
    ```console
    $ bbr director --host 10.0.0.5 --username bbr \
    --private-key-path private-key.pem pre-backup-check
    ```

1. If the pre-backup check command fails, perform the following actions:
    1. Run the command again, adding the `--debug` flag to enable debug logs. For more information,
    see [BBR Logging](bbr-logging.html).  
    1. Make any correction suggested in the output and run the pre-backup check again.  

#### <a id='back-up-director-back-up'></a> Back Up the Tanzu Kubernetes Grid Integrated Edition BOSH Director

1. If the pre-backup check succeeds, run the BBR back up command from your jump box to back up the
TKGI BOSH Director:  

    ```
    bbr director --host BOSH-DIRECTOR-IP --username bbr \
    --private-key-path PRIVATE-KEY-FILE  backup
    ```
    Where:  

    * `BOSH-DIRECTOR-IP` is the address of the BOSH Director. If the BOSH Director is public, `BOSH-DIRECTOR-IP`
    is a URL, such as `https://my-bosh.xxx.cf-app.com`. Otherwise, this is the internal IP.
    See [Retrieve the BOSH Director Address](#bosh-address) for more information.  
    * `PRIVATE-KEY-FILE` is the path to the private key file that you can create from `Bbr Ssh Credentials` as shown in
    [Download the BBR SSH Credentials](#bbr-ssh-creds).  

    For example:  
    ```console
    $ bbr director  --host 10.0.0.5 --username bbr \
    --private-key-path private-key.pem  backup
    ```

    <p class="note"><strong>Note</strong>: The BBR back up command can take a long time to complete.
    You can run it independently of the SSH session so that the process can continue running even
    if your connection to the jump box fails. The command above uses <code>nohup</code>, but you can
    run the command in a <code>screen</code> or <code>tmux</code> session instead.</p>

1. If the command completes successfully, follow the steps in [Manage Your Backup Artifact](#good-practices) below.

1. If the back up command fails, perform the following actions:
    * Run the command again, adding the `--debug` flag to enable debug logs. For more information,
    see [BBR Logging](bbr-logging.html).  
    * Follow the steps in [Recover from a Failing Command](#recover-from-failing-command).  

### <a id='back-up-control-plane'></a> Back Up the Tanzu Kubernetes Grid Integrated Edition Control Plane

To back up your Tanzu Kubernetes Grid Integrated Edition Control Plane you will validate the Control Plane, then execute the `bbr` back up command.

#### <a id='locate-deploy-name'></a> Locate the Tanzu Kubernetes Grid Integrated Edition Deployment Name
Locate and record your Tanzu Kubernetes Grid Integrated Edition BOSH deployment name as follows:

1. Open an SSH connection to either your jump box, as described in the previous section, or the Ops Manager VM.
For instructions on how to SSH into the Ops Manager VM, see
[Log in to the Ops Manager VM with SSH](https://docs.vmware.com/en/VMware-Tanzu-Operations-Manager/3.0/vmware-tanzu-ops-manager/install-trouble-advanced.html#ssh)
in _Advanced Troubleshooting with the BOSH CLI_.  
1. On the command line, run the following command to retrieve your Tanzu Kubernetes Grid Integrated Edition BOSH deployment name.  

    ```
    BOSH-CLI-CREDENTIALS deployments | grep pivotal-container-service
    ```

    Where `BOSH-CLI-CREDENTIALS` is the full value that you copied from the BOSH Director tile in 
    [Retrieve the BOSH Command Line Credentials](#bosh-cli-creds).  
<br>
    For example:  
    ```console
    $ BOSH_CLIENT=ops_manager BOSH_CLIENT_SECRET=p455w0rd BOSH_CA_CERT=/var/tempest/workspaces/default/root_ca_certificate BOSH_ENVIRONMENT=10.0.0.5 bosh deployments | grep pivotal-container-service  
    pivotal-container-service-51f08f6402aaa960f041           backup-and-restore-sdk/1.9.0    bosh-google-kvm-ubuntu-jammy-go_agent/1.75  
    service-instance_4ffeb5b5-5182-4faa-9d92-696d97cc9ae1    bosh-dns/1.10.0                 bosh-google-kvm-ubuntu-jammy-go_agent/1.75  
    pivotal-container-service-51f08f6402aaa960f041  
    ```

1. Review the returned output. The Tanzu Kubernetes Grid Integrated Edition BOSH deployment name begins with
  `pivotal-container-service` and includes a unique identifier.
   In the example output above, the BOSH deployment name is `pivotal-container-service-51f08f6402aaa960f041`.  

#### <a id='back-up-control-plane-check'></a> Validate the Tanzu Kubernetes Grid Integrated Edition Control Plane

1. To confirm that your TKGI control plane is reachable and has a deployment that can be backed up, run the BBR pre-backup check command:  

    ```
    BOSH_CLIENT_SECRET=BOSH-CLIENT-SECRET bbr deployment \
    --target BOSH-TARGET  --username BOSH-CLIENT --deployment DEPLOYMENT-NAME \
    --ca-cert PATH-TO-BOSH-SERVER-CERT \
    pre-backup-check
    ```
    Where:  

    * `BOSH-CLIENT-SECRET` is your BOSH client secret. If you do not know your BOSH Client Secret, open your BOSH Director tile,
    navigate to **Credentials > Bosh Commandline Credentials** and record the value for `BOSH_CLIENT_SECRET`.  
    * `BOSH-TARGET` is your BOSH Environment setting. If you do not know your BOSH Environment setting, open your BOSH Director tile,
    navigate to **Credentials > Bosh Commandline Credentials** and record the value for `BOSH_ENVIRONMENT`. You must be able to
    reach the target address from the workstation where you run `bbr` commands.  
    * `BOSH-CLIENT` is your BOSH Client Name. If you do not know your BOSH Client Name, open your BOSH Director tile,
    navigate to **Credentials > Bosh Commandline Credentials** and record the value for `BOSH_CLIENT`.  
    * `DEPLOYMENT-NAME` is the Tanzu Kubernetes Grid Integrated Edition BOSH deployment name that you located in
    the [Locate the Tanzu Kubernetes Grid Integrated Edition Deployment Name](#locate-deploy-name) section above.  
    * `PATH-TO-BOSH-CA-CERT` is the path to the root CA certificate that you downloaded in [Download the Root CA Certificate](#root-ca-cert) above.  

    For example:  
    ```console
    $ BOSH_CLIENT_SECRET=p455w0rd bbr deployment \
    --target bosh.example.com --username admin --deployment cf-acceptance-0 \
    --ca-cert bosh.ca.cert \
    pre-backup-check
    ```

1. If the pre-backup check command fails, perform the following actions:  
    1. Run the command again, adding the `--debug` flag to enable debug logs. For more information,
    see [BBR Logging](bbr-logging.html).  
    1. Make any correction suggested in the output and run the pre-backup check again. For example,
    the deployment that you selected might not have the correct back up scripts, or the connection
    to the BOSH Director failed.  

#### <a id='back-up-control-plane-backup'></a> Back Up the Tanzu Kubernetes Grid Integrated Edition Control Plane

If the pre-backup check succeeds, run the BBR back up command.

1. To back up the TKGI control plane, run the following BBR back up command from your jump box:  

    ```
    BOSH_CLIENT_SECRET=BOSH-CLIENT-SECRET nohup bbr deployment \
    --target BOSH-TARGET --username BOSH-CLIENT --deployment DEPLOYMENT-NAME \
    --ca-cert PATH-TO-BOSH-SERVER-CERT \
    backup --with-manifest [--artifact-path]
    ```
    Where:  

    * `BOSH-CLIENT-SECRET` is your BOSH client secret. If you do not know your BOSH Client Secret, open your BOSH Director tile,
    navigate to **Credentials > Bosh Commandline Credentials** and record the value for `BOSH_CLIENT_SECRET`.  
    * `BOSH-TARGET` is your BOSH Environment setting. If you do not know your BOSH Environment setting, open your BOSH Director tile,
    navigate to **Credentials > Bosh Commandline Credentials** and record the value for `BOSH_ENVIRONMENT`. You must be able to
    reach the target address from the workstation where you run <code>bbr</code> commands.  
    * `BOSH-CLIENT` is your BOSH Client Name. If you do not know your BOSH Client Name, open your BOSH Director tile,
    navigate to **Credentials > Bosh Commandline Credentials** and record the value for `BOSH_CLIENT`.  
    * `DEPLOYMENT-NAME` is the Tanzu Kubernetes Grid Integrated Edition BOSH deployment name that you located in
    the [Locate the Tanzu Kubernetes Grid Integrated Edition Deployment Name](#locate-deploy-name) section above.  
    * `PATH-TO-BOSH-CA-CERT` is the path to the root CA certificate that you downloaded in [Download the Root CA Certificate](#root-ca-cert) above.  
    * `--with-manifest` is necessary in order to redeploy your TKGI Control Plane in the case of its loss.
    `--with-manifest` is an optional `backup` parameter to include the manifest in the backup artifact.  
    * `--artifact-path` is an optional `backup` parameter to specify the output path for the backup artifact.  

    For example:  
    ```console
    $ BOSH_CLIENT_SECRET=p455w0rd nohup bbr deployment \
    --target bosh.example.com  --username admin --deployment cf-acceptance-0 \
    --ca-cert bosh.ca.cert \
    backup --with-manifest
    ```

    <p class="note"><strong>Note</strong>: The <code>--with-manifest</code> flag is necessary in order to redeploy your TKGI Control Plane in the case of its loss.
    Secure the backup artifact created by this process because it contains secret credentials.</p>    
    
1. Wait for the BBR back up command to complete:  
    * If the command completes successfully, follow the steps in [Manage Your Backup Artifact](#good-practices) below.
    * If the back up command fails, perform the following actions:
        1. Run the command again, adding the `--debug` flag to enable debug logs. For more information,
        see [BBR Logging](bbr-logging.html).  
        1. Follow the steps in [Recover from a Failing Command](#recover-from-failing-command).  

    <p class="note"><strong>Note</strong>: The BBR back up command can take a long time to complete.
    You can run it independently of the SSH session so that the process can continue running even
    if your connection to the jump box fails. The command above uses <code>nohup</code>, but you can
    run the command in a <code>screen</code> or <code>tmux</code> session instead.</p>

### <a id='cancel-backup'></a> Cancel a Back Up

Backing up can take a long time. If you realize that the back up is going to fail or that
your developers need to push an app immediately, you might need to cancel the back up.

To cancel a back up, perform the following steps:

1. Terminate the BBR process by pressing Ctrl-C and typing `yes` to confirm.  
1. Because stopping a back up can leave the system in an unusable state and prevent additional
back ups, follow the procedures in [Clean up After a Failed Back Up](#manual-clean) below.  

## <a id="after-backup"></a> After Backing Up Tanzu Kubernetes Grid Integrated Edition

After the back up has completed, review and manage the generated backup artifacts.

### <a id="good-practices"></a> Manage Your Backup Artifact
The BBR-created backup consists of a directory containing the backup artifacts and metadata files.
BBR stores each completed backup directory within the current working directory.

<p class="note"><strong>Note</strong>: The optional <code>--with-manifest</code> flag directs BBR to create a backup
containing credentials. Manage the generated backup artifact knowing it contains secrets for administering
your environment.</p>

BBR backup artifact directories are named using the following formats:

* `DIRECTOR-IP-TIMESTAMP` for the BOSH Director backups.
* `DEPLOYMENT-TIMESTAMP` for the Control Plane backup.
* `DEPLOYMENT-TIMESTAMP` for the cluster deployment backups.

Keep your backup artifacts safe by following these steps:

1. Move the backup artifacts off the jump box to your storage space.  

1. Compress and encrypt the backup artifacts when storing them.  

1. Make redundant copies of your backup and store them in multiple locations. This minimizes the
risk of losing your backups in the event of a disaster.  

1. Each time you redeploy Tanzu Kubernetes Grid Integrated Edition, test your backup artifact by following the procedures in:  
    * [Restore the Tanzu Kubernetes Grid Integrated Edition BOSH Director](bbr-restore.html#redeploy-restore-director)  
    * [Restore the Tanzu Kubernetes Grid Integrated Edition Control Plane](bbr-restore.html#redeploy-restore-control-plane)  
    * [Restore Tanzu Kubernetes Grid Integrated Edition Clusters](bbr-restore.html#redeploy-restore-clusters)  

### <a id="recover-from-failing-command"></a> Recover from a Failing Command

If the back up fails, follow these steps:  

1. Ensure that you set all the parameters in the back up command.
1. Ensure the credentials previously obtained are valid.
1. Ensure the deployment that you specify in the BBR command exists.
1. Ensure that the jump box can reach the BOSH Director.
1. Consult [BBR Logging](bbr-logging.html).
1. If you see the error message: `Directory /var/vcap/store/bbr-backup already exists on instance`,
   run the appropriate cleanup command. See [Clean Up After a Failed Backup](#manual-clean) below for more information.
1. If the backup artifact is corrupted, discard the failing artifacts and run the back up again.


### <a id="manual-clean"></a> Clean Up after a Failed Back Up

If your back up process fails, use the BBR cleanup script to clean up the failed run.

<p class="note warning"><strong>Warning</strong>: It is important to run the BBR cleanup script after a
failed BBR back up run. A failed back up run might leave the BBR backup directory on the instance,
causing any subsequent attempts to back up to fail. In addition, BBR might not have run the post-backup scripts,
leaving the instance in a locked state.</p>

* If the TKGI BOSH Director back up failed:  
    1. Run the following BBR cleanup script command to clean up:  

        ```
        bbr director --host BOSH-DIRECTOR-IP \
        --username bbr  --private-key-path PRIVATE-KEY-FILE \
        backup-cleanup
        ```
        Where:  

        * `BOSH-DIRECTOR-IP` is the address of the BOSH Director. If the BOSH Director is public,
        `BOSH-DIRECTOR-IP` is a URL, such as `https://my-bosh.xxx.cf-app.com`. Otherwise, this is the internal
        IP `BOSH-DIRECTOR-IP` which you can retrieve as show in [Retrieve the BOSH Director Address](#bosh-address) above.  
        * `PRIVATE-KEY-FILE` is the path to the private key file that you can create from `Bbr Ssh Credentials` as shown in
        [Download the BBR SSH Credentials](#bbr-ssh-creds) above. Replace the placeholder text using the
        information in the following table.  

        For example:  
        ```console
        $ bbr director  --host 10.0.0.5  --username bbr \
        --private-key-path private-key.pem \
        backup-cleanup
        ```

* If the TKGI control plane or TKGI clusters back ups fail:  
    1. Run the following BBR cleanup script command to clean up:

        ```
        BOSH_CLIENT_SECRET=BOSH-CLIENT-SECRET \
        bbr deployment \
        --target BOSH-TARGET \
        --username BOSH-CLIENT \
        --deployment DEPLOYMENT-NAME \
        --ca-cert PATH-TO-BOSH-CA-CERT \
        backup-cleanup
        ```
        Where:  

        * `BOSH-CLIENT-SECRET` is your BOSH client secret. If you do not know your BOSH Client Secret,
        open your BOSH Director tile, navigate to **Credentials > Bosh Commandline Credentials** and
        record the value for `BOSH_CLIENT_SECRET`.  
        * `BOSH-TARGET` is your BOSH Environment setting. If you do not know your BOSH Environment setting,
        open your BOSH Director tile,     navigate to **Credentials > Bosh Commandline Credentials** and
        record the value for `BOSH_ENVIRONMENT`. You must be able to reach the target address from the
        workstation where you run `bbr` commands.  
        * `BOSH-CLIENT` is your BOSH Client Name. If you do not know your BOSH Client Name, open your BOSH Director tile,
        navigate to **Credentials > Bosh Commandline Credentials** and record the value for `BOSH_CLIENT`.  
        * `DEPLOYMENT-NAME` is the Tanzu Kubernetes Grid Integrated Edition BOSH deployment name that you located in
        the [Locate the Tanzu Kubernetes Grid Integrated Edition Deployment Names](#locate-deploy-name) section above.  
        * `PATH-TO-BOSH-CA-CERT` is the path to the root CA certificate that you downloaded in
        [Download the Root CA Certificate](#root-ca-cert) above.  

        For example:  
        ```console
        $ BOSH_CLIENT_SECRET=p455w0rd bbr deployment \
        --target bosh.example.com --username admin --deployment cf-acceptance-0 \
        --ca-cert bosh.ca.crt \
        backup-cleanup
        ```
    1. If the cleanup script fails, consult the following table to match the exit codes to an error
    message:
        <table>
          <tr>
            <th>Value</th>
            <th>Error</th>
          </tr>
          <tr>
            <td>0</td>
            <td>Success</td>
          </tr>
          <tr>
            <td>1</td>
            <td>General failure</td>
          </tr>
          <tr>
            <td>8</td>
            <td>The post-backup unlock failed. One of your deployments might be in a bad state and require
              attention.</td>
          </tr>
          <tr>
            <td>16</td>
            <td>The cleanup failed. This is a non-fatal error indicating that the utility has been unable
              to clean up open BOSH SSH connections to a deployment's VMs. Manual cleanup might be required
              to clear any hanging BOSH users and connections.</td>
          </tr>
        </table>