forked from aquarist-labs/s3gw-website
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
website: upgrade, adjust for s3gw-tech org
Signed-off-by: Joao Eduardo Luis <[email protected]>
- Loading branch information
Showing
40 changed files
with
15,966 additions
and
2,749 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -20,7 +20,7 @@ This is the first publicly available iteration of the s3gw project. We expect (a | |
|
|
||
| We would love to hear from you about what you'd like to see on our roadmap. What would enable you best to use s3gw in your environment? | ||
|
|
||
| Reach out to us at <[email protected]> or our [Slack channel](https://aquaristlabs.slack.com/archives/C03RFG0BES0). You can also join [our mailing list](https://lists.suse.com/mailman/listinfo/s3gw) or have a look at our [GitHub repository](https://github.com/aquarist-labs/s3gw) -- feature requests are welcome! 🙂 | ||
| Reach out to us at [[email protected]](mailto:[email protected]) or have a look at our [GitHub repository][1] -- feature requests are welcome! 🙂 | ||
|
|
||
| [1]:https://github.com/aquarist-labs/s3gw | ||
| [1]:https://github.com/s3gw-tech/s3gw | ||
| [2]:https://docs.ceph.com/en/quincy/radosgw/ | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -12,41 +12,49 @@ Increased demand for cloud storage solutions has become a crucial topic in recen | |
|
|
||
| <!--truncate--> | ||
|
|
||
| :::warning | ||
|
|
||
| This content was adapted from the original `aquarist-labs` organization to the | ||
| new `s3gw-tech` organization. Some of it, especially screenshots, may contain | ||
| `aquarist-labs` references. Keep that in mind when following tutorials. | ||
|
|
||
| ::: | ||
|
|
||
| Increased demand for cloud storage solutions has become a crucial topic in recent years: companies are requiring data to be made more readily available for their cloud-native applications. | ||
|
|
||
| In a cloud-native environment, it is important to offer storage systems that can interact with clients using a standard protocol. | ||
|
|
||
| ## Simple Storage Service | ||
|
|
||
|  | ||
|  | ||
|
|
||
| [Simple Storage Service,](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html) or S3, is a protocol designed by Amazon that launched in the United States market in 2006. S3 is a vast protocol that covers key concepts such as *buckets*, *objects*, *keys*, *versioning*, *ACLs* and *regions*. | ||
| [Simple Storage Service](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html), or S3, is a protocol designed by Amazon that launched in the United States market in 2006. S3 is a vast protocol that covers key concepts such as *buckets*, *objects*, *keys*, *versioning*, *ACLs* and *regions*. | ||
|
|
||
| To continue with this article, you need to know that the S3 API can be invoked with REST calls and that you can just store your objects inside holders called buckets. For more information, there are other resources available online. | ||
|
|
||
| ## K3s and Rancher | ||
|
|
||
| This article explores the use of [K3s](https://k3s.io/) and [Rancher](https://rancher.com/) as foundations for experiments with an S3 gateway. | ||
|
|
||
|  | ||
|  | ||
|
|
||
| K3s is a lightweight Kubernetes distribution that runs smoothly on Edge low resource devices. Rancher is a graphical user manager that simplifies the underlying complexity of a Kubernetes cluster. | ||
|
|
||
|  | ||
|  | ||
|
|
||
| With Rancher, you can manage a cluster in a user-friendly fashion, regardless of the Kubernetes version being used. | ||
|
|
||
| ## Longhorn | ||
|
|
||
|  | ||
|  | ||
|
|
||
| A Kubernetes cluster and manager alone are not sufficient when dealing with cloud storage. You could use the primitive resources offered by a standard Kubernetes cluster such as the basic persistent volume types, but we recommend installing a component that takes care of providing your pods with some advanced storage resource. | ||
|
|
||
| It is desirable to have a system that can take care of your data securely and in a redundant fasion and expose volumes with the standard Kubernetes interfaces. [Longhorn](https://longhorn.io/) is the right system for this kind of need. Built from scratch to work natively with Kubernetes, Longhorn allows pods to obtain highly available persistent volumes. The portion of storage managed by Longhorn is replicated so that a hardware failure does not compromise user's data. | ||
|
|
||
| ## S3 Gateway | ||
|
|
||
|  | ||
|  | ||
|
|
||
| Having Longhorn deployed on your cluster allows the persistent volumes to be consumed by internal applications deployed on Kubernetes. If you want to give access to the data to external clients, you need an S3 gateway. | ||
|
|
||
|
|
@@ -100,7 +108,7 @@ After this, you will be able to operate on K3s with your user. | |
|
|
||
| ### Install Helm | ||
|
|
||
|  | ||
|  | ||
|
|
||
| We are going to install Rancher using an [Helm](https://helm.sh/) chart, so you must first install Helm on the system: | ||
|
|
||
|
|
@@ -163,44 +171,49 @@ helm install rancher rancher-latest/rancher\ | |
| --set bootstrapPassword=admin | ||
| ``` | ||
|
|
||
| When Rancher's pods have booted up, with your browser you can navigate to: <https://rancher.local> and complete the initial setup: | ||
| When Rancher's pods have booted up, with your browser you can navigate to: [https://rancher.local](https://rancher.local) and complete the initial setup: | ||
|
|
||
|  | ||
|  | ||
|
|
||
| Once you have completed the step, and saved the password, you can start exploring your local cluster with the graphical manager: | ||
|
|
||
|  | ||
|  | ||
|
|
||
| Depending on what you have deployed on the cluster, you could see more or less resource consumption. | ||
|
|
||
| ### Deploy Longhorn | ||
|
|
||
| You can now deploy Longhorn using the Charts chooser under the Apps section on the left of the Rancher's dashboard: | ||
|
|
||
|  | ||
|  | ||
|
|
||
| The installation is pretty straightforward and you don't need to change any | ||
| default value of the chart. | ||
|
|
||
| The installation is pretty straightforward and you don't need to change any default value of the chart.\ | ||
| In the end, if everything has gone well, you should see the Rancher's console showing: | ||
|
|
||
|  | ||
|  | ||
|
|
||
| After Longhorn has been installed, you can simply click on the Longhorn entry on the left Rancher's menu to be redirected to the Longhorn's dashboard: | ||
|
|
||
|  | ||
|  | ||
|
|
||
| A freshly installation of Longhorn shows that still no application is using a persistent volume. | ||
|
|
||
| ### Deploy s3gw | ||
|
|
||
| You are now ready to add the last ingredient to your system: s3gw, the S3 gateway.\ | ||
| Rancher does not bundle with the [Helm repository of s3gw](https://github.com/aquarist-labs/s3gw-charts) set by default, so you must add it from the dashboard: | ||
| You are now ready to add the last ingredient to your system: s3gw, the S3 | ||
| gateway. | ||
|
|
||
| Rancher does not bundle with the [Helm repository of s3gw](https://github.com/s3gw-tech/s3gw-charts) set by default, so you must add it from the dashboard: | ||
|
|
||
|  | ||
|
|
||
|  | ||
| You can choose an unique name: for example: s3gw. | ||
|
|
||
| You can choose an unique name: for example: s3gw.\ | ||
| For the Target field you have to choose Git repository and the repository URL: | ||
|
|
||
| <https://github.com/aquarist-labs/s3gw-charts> | ||
| [https://github.com/s3gw-tech/s3gw-charts](https://github.com/s3gw-tech/s3gw-charts) | ||
|
|
||
| In the Git Branch field put the latest available release: | ||
|
|
||
|
|
@@ -209,31 +222,43 @@ v0.2.0 | |
| You can now click on Create button.\ | ||
| On Apps section, you can now find the s3gw chart: | ||
|
|
||
|  | ||
|  | ||
|
|
||
| So let's proceed with installation; you can choose a namespace and a name for s3gw: | ||
|
|
||
|  | ||
|  | ||
|
|
||
| You don't need to customize the chart, so you can leave the bottom checkbox | ||
| unchecked. | ||
|
|
||
| You don't need to customize the chart, so you can leave the bottom checkbox unchecked.\ | ||
| Once the installation completed, if everything has gone well, you should see the Rancher's console showing: | ||
|
|
||
|  | ||
|  | ||
|
|
||
| On Longhorn dashboard, you can verify that the application is using a Longhorn persistent volume: | ||
|
|
||
|  | ||
|  | ||
|
|
||
| ### Test the S3 gateway | ||
|
|
||
| By default the s3gw chart configures an ingress resource pointing to the S3 gateway with the FQDN: s3gw.local .\ | ||
| By default the s3gw chart configures an ingress resource pointing to the S3 | ||
| gateway with the FQDN: s3gw.local . | ||
|
|
||
| Thus, you must define s3gw.local in /etc/hosts pointing to the ip address of one of the host's physical interfaces, for example: | ||
|
|
||
| 10.0.0.2 s3gw.local | ||
|
|
||
| For testing the S3 gateway you can rely on [s3cmd](https://github.com/s3tools/s3cmd) that is a popular command line S3 client.\ | ||
| You can install it choosing a method listed [here](https://github.com/s3tools/s3cmd/blob/master/INSTALL.md).\ | ||
| Once you have installed it, you can take the s3cmd configuration file from [here](https://raw.githubusercontent.com/aquarist-labs/s3gw-core/main/env/s3cmd.cfg) and use that as it is against s3gw.\ | ||
| For testing the S3 gateway you can rely | ||
| on [s3cmd](https://github.com/s3tools/s3cmd) that is a popular command line S3 | ||
| client. | ||
|
|
||
| You can install it choosing a method | ||
| listed [here](https://github.com/s3tools/s3cmd/blob/master/INSTALL.md). | ||
|
|
||
| Once you have installed it, you can take the s3cmd configuration file | ||
| from [here](https://raw.githubusercontent.com/s3gw-tech/s3gw/main/tools/env/s3cmd.cfg) and | ||
| use that as it is against s3gw. | ||
|
|
||
| All you need to to is to create a directory, put s3cmd.cfg inside that and finally invoke s3cmd. | ||
|
|
||
| #### Create a bucket | ||
|
|
@@ -244,14 +269,14 @@ $ s3cmd -c s3cmd.cfg mb s3://foo | |
|
|
||
| #### Put some objects in the bucket | ||
|
|
||
| Let's create a 1mb file filled with random data and put it in the bucket: | ||
| Let's create a 1 MB file filled with random data and put it in the bucket: | ||
|
|
||
| ```bash | ||
| $ dd if=/dev/random bs=1k count=1k of=obj.1mb.bin | ||
| $ s3cmd -c s3cmd.cfg put obj.1mb.bin s3://foo | ||
| ``` | ||
|
|
||
| Let's create a 10mb file filled with random data and put it in the bucket: | ||
| Let's create a 10 MB file filled with random data and put it in the bucket: | ||
|
|
||
| ```bash | ||
| $ dd if=/dev/random bs=1k count=10k of=obj.10mb.bin | ||
|
|
@@ -278,6 +303,6 @@ In this tutorial, you've seen how to set up a K3s cluster, manage it with Ranche | |
|
|
||
| ## Call to action | ||
|
|
||
| We would love to hear from you about what you'd like to see on our roadmap. What would enable you best to use s3gw in your environment? | ||
| We would love to hear from you about what you'd like to see on our roadmap. What would enable you best to use s3gw in your environment? | ||
|
|
||
| Reach out to us at <[email protected]> or our [Slack channel](https://aquaristlabs.slack.com/archives/C03RFG0BES0). You can also join [our mailing list](https://lists.suse.com/mailman/listinfo/s3gw) or have a look at our [GitHub repository](https://github.com/aquarist-labs/s3gw) -- feature requests are welcome! 🙂 | ||
| Reach out to us at [[email protected]](mailto:[email protected]) or have a look at our [GitHub repository](https://github.com/s3gw-tech/s3gw) -- feature requests are welcome! 🙂 | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -14,15 +14,23 @@ s3gw is an S3-compatible service focused on deployments in a Kubernetes environm | |
|
|
||
| <!--truncate--> | ||
|
|
||
| s3gw is based on Ceph's RADOSGW (RGW) but runs as a stand--alone service without the RADOS cluster and relies on a storage backend still under heavy development by the storage team at SUSE. The s3gw team is also developing a web-based UI for management and an object explorer. | ||
| s3gw is based on Ceph's RADOSGW (RGW) but runs as a stand--alone service without the RADOS cluster and relies on a storage backend still under heavy development by the storage team at SUSE. The s3gw team is also developing a web-based UI for management and an object explorer. | ||
|
|
||
| :::warning | ||
|
|
||
| This content was adapted from the original `aquarist-labs` organization to the | ||
| new `s3gw-tech` organization. Some of it, especially screenshots, may contain | ||
| `aquarist-labs` references. Keep that in mind when following tutorials. | ||
|
|
||
| ::: | ||
|
|
||
| ## The s3gw service | ||
|
|
||
| Distributed as a small container, the s3gw service runs RGW and exposes an S3-compatible API. Instead of requiring a full Ceph cluster deployment, we leverage RGW's standalone capabilities and keep data on a local storage volume. Although the focus is on running within a Kubernetes environment with on-premise storage provided to the container, s3gw can consume any storage type with a filesystem on it. This can be PVC in Kubernetes or a local directory on a development machine. | ||
| Distributed as a small container, the s3gw service runs RGW and exposes an S3-compatible API. Instead of requiring a full Ceph cluster deployment, we leverage RGW's standalone capabilities and keep data on a local storage volume. Although the focus is on running within a Kubernetes environment with on-premise storage provided to the container, s3gw can consume any storage type with a filesystem on it. This can be PVC in Kubernetes or a local directory on a development machine. | ||
|
|
||
| As the container consumes the storage volume, the object data is kept in a hash tree of directories and the metadata is kept in an SQLite database. This allows us to leverage the ACID properties of SQLite to ensure the state is committed atomically while keeping large blobs of data on the filesystem and away from SQLite's path. | ||
|
|
||
| In the future, we will release a blog post describing the s3gw service's data store in more depth. | ||
| In the future, we will release a blog post describing the s3gw service's data store in more depth. | ||
|
|
||
| However, it should be noted that we don't support all RGW's S3 APIs. Some components are still under development and other features not yet included that are required for proper operation. | ||
|
|
||
|
|
@@ -34,22 +42,22 @@ Also distributed as a small container, the s3gw web UI provides an intuitive way | |
|
|
||
| We have a few screenshots of the current UI version below, but please keep in mind that we are still actively developing it and it is not feature-complete. | ||
|
|
||
|  | ||
|  | ||
|  | ||
|  | ||
|  | ||
|  | ||
|  | ||
|  | ||
|  | ||
|  | ||
|
|
||
| ## Installing | ||
|
|
||
| You may find our Helm chart helpful if you have a Kubernetes cluster, whichever flavor that might be. You'll be able to find it on [ArtifactHub](https://artifacthub.io/packages/helm/s3gw/s3gw), and our [documentation](https://s3gw-docs.readthedocs.io/en/latest/helm-charts/) can provide important insights as to available configuration values. | ||
|
|
||
| Alternatively, if you are using Rancher, you may find s3gw available in the Partner repository, as depicted below: | ||
|
|
||
|  | ||
|  | ||
|
|
||
| ## Call for action | ||
|
|
||
| We would love to hear from you about what you'd like to see on our roadmap. What would enable you best to use s3gw in your environment? | ||
|
|
||
| Reach out to us at <[email protected]> or our [Slack channel](https://aquaristlabs.slack.com/archives/C03RFG0BES0). You can also join [our mailing list](https://lists.suse.com/mailman/listinfo/s3gw) or have a look at our [GitHub repository](https://github.com/aquarist-labs/s3gw) -- feature requests are welcome! 🙂 | ||
| Reach out to us at [[email protected]](mailto:[email protected]) or have a look at our [GitHub repository](https://github.com/s3gw-tech/s3gw) -- feature requests are welcome! 🙂 | ||
Oops, something went wrong.