Merge pull request #279 from umccr/docs/simplify

docs: re-word and simplify, add quick starts where applicable

CONTRIBUTING.md

@@ -2,7 +2,7 @@
 
 Thank you for your interest in contributing. We greatly value feedback and contributions, whether that's
 an issue, bug fix, new feature or document change. All contributions are welcome, and no change is too small, even if
-its just a typo fix.
+it's just a typo fix.
 
 To get familiar with the project, have a look at the READMEs of each crate.
 
@@ -17,13 +17,13 @@ Have a look at existing [issues] to see if an issue has already been discussed.
 
 ## Pull Requests
 
-We welcome you to open up a pull request 
-to suggest a change, even if it's a small one line change. If the change is large, it is a good idea to first open an 
-issue to discuss the change in order gain feedback and guidance.
+We welcome you to open up a pull request to suggest a change, even if it's a small one line change.
+If the change is large, it is a good idea to first open an issue to discuss the change in order gain feedback and
+guidance.
 
 ### Tests and formatting
 
-If the proposed change alters the code, tests should updated to ensure that no regressions are made. Any new features 
+If the proposed change alters the code, tests should update to ensure that no regressions are made. Any new features 
 need to have thorough testing before they are merged. 
 
 We also use [clippy] and [rustfmt] for code style, linting and formatting.

README.md

@@ -12,18 +12,35 @@
 A **server** implementation of the [htsget protocol][htsget-protocol] for bioinformatics in Rust. It is:
 * **Fully-featured**: supports BAM and CRAM for reads, and VCF and BCF for variants, as well as other aspects of the protocol such as TLS, and CORS.
 * **Serverless**: supports local server instances using [Axum][axum] and [Actix Web][actix-web], and serverless instances using [AWS Lambda Rust Runtime][aws-lambda-rust-runtime].
-* **Storage interchangeable**: supports local filesystem storage as well as objects via [Minio][minio] and AWS S3.
+* **Storage interchangeable**: supports local filesystem storage as well as objects via [Minio][minio] and [AWS S3][aws-s3].
 * **Thoroughly tested and benchmarked**: tested using a purpose-built [test suite][htsget-test] and benchmarked using [criterion-rs].
 
-To get started, see [Usage].
-
-**Note**: htsget-rs is still experimental, and subject to change.
-
 [actix-web]: https://github.com/actix/actix-web
 [criterion-rs]: https://github.com/bheisler/criterion.rs
-[Usage]: #usage
 
-## Overview
+## Quickstart
+
+To run a local instance htsget-rs, run [htsget-axum]:
+
+```sh
+cargo run -p htsget-axum
+```
+
+And fetch tickets from `127.0.0.1:8080`, which serves data from [data]:
+
+```sh
+curl 'http://127.0.0.1:8080/variants/data/vcf/sample1-bcbio-cancer'
+```
+
+### Configuration
+
+Htsget-rs is configured using environment variables or config files, see [htsget-config] for details.
+
+### Cloud
+
+Cloud-based htsget-rs uses [htsget-lambda]. For an example deployment of this crate see [deploy].
+
+## Protocol
 
 Htsget-rs implements the [htsget protocol][htsget-protocol], which is an HTTP-based protocol for querying bioinformatics files. 
 The htsget protocol outlines how a htsget server should behave, and it is an effective way to fetch regions of large bioinformatics files. 
@@ -51,22 +68,6 @@ htsget-rs implements the following components of the protocol:
 [htsget-diagram-png]: https://samtools.github.io/hts-specs/pub/htsget-ticket.png
 [tokio]: https://github.com/tokio-rs/tokio
 
-## Usage
-
-Htsget-rs is configured using environment variables, for details on how to set them, see [htsget-config].
-
-### Local
-To run a local instance htsget-rs, run [htsget-axum] by executing the following:
-```sh
-cargo run -p htsget-axum
-```
-Using the default configuration, this will start a ticket server on `127.0.0.1:8080` and a data block server on `127.0.0.1:8081`
-with data accessible from the [data] directory. See [htsget-axum] for more information.
-
-### Cloud
-Cloud based htsget-rs uses [htsget-lambda]. For more information and an example deployment of this crate see 
-[deploy].
-
 ### Tests
 
 Tests can be run tests by executing:
@@ -77,53 +78,41 @@ cargo test --all-features
 
 To run benchmarks, see the benchmark sections of [htsget-actix][htsget-actix-benches] and [htsget-search][htsget-search-benches].
 
-[htsget-actix-benches]: htsget-actix/README.md#Benchmarks
-[htsget-search-benches]: htsget-search/README.md#Benchmarks
+[htsget-actix-benches]: htsget-actix/README.md#benchmarks
+[htsget-search-benches]: htsget-search/README.md#benchmarks
 
 ## Project Layout
 
-This repository consists of a workspace composed of the following crates:
+This repository is a workspace of crates:
 
 - [htsget-config]: Configuration of the server.
 - [htsget-actix]: Local instance of the htsget server. Contains framework dependent code using [Actix Web][actix-web].
 - [htsget-axum]: Local instance of the htsget server. Contains framework dependent code using [Axum][axum].
 - [htsget-http]: Handling of htsget HTTP requests. Framework independent code.
-- [htsget-lambda]: Cloud based instance of the htsget server. Contains framework dependent
+- [htsget-lambda]: Cloud-based instance of the htsget server. Contains framework dependent
 code using the [Rust Runtime for AWS Lambda][aws-lambda-rust-runtime].
 - [htsget-search]: Core logic needed to search bioinformatics files based on htsget queries.
+- [htsget-storage]: Storage interfaces for local and cloud-based files.
 - [htsget-test]: Test suite used by other crates in the project.
 
 Other directories contain further applications or data:
-- [data]: Contains example data files which can be used by htsget-rs, in folders denoting the file type.
-This directory also contains example events used by a cloud instance of htsget-rs in the [`events`][data-events] subdirectory.
-- [deploy]: An example deployment of [htsget-lambda].
-
-In htsget-rs the ticket server handled by [htsget-axum], [htsget-actix] or [htsget-lambda], and the data
-block server is handled by the [storage backend][storage-backend], either [locally][local-storage], or using [AWS S3][s3-storage].
-This project layout is structured to allow for extensibility and modularity. For example, a new ticket server and data server could 
-be implemented using Cloudflare Workers in a `htsget-http-workers` crate and Cloudflare R2 in [htsget-search].
-
-See the [htsget-search overview][htsget-search-overview] for more information on the storage backend.
+- [data]: Contains example data files used by htsget-rs and in tests.
+- [deploy]: Deployments for htsget-rs.
 
 [axum]: https://github.com/tokio-rs/axum
 [htsget-config]: htsget-config
 [htsget-actix]: htsget-actix
 [htsget-http]: htsget-http
 [htsget-lambda]: htsget-lambda
 [htsget-search]: htsget-search
-[htsget-search-overview]: htsget-search/README.md#Overview
+[htsget-storage]: htsget-storage
 [htsget-test]: htsget-test
 
-[storage-backend]: htsget-search/src/storage
-[local-storage]: htsget-search/src/storage/local.rs
-[s3-storage]: htsget-search/src/storage/s3.rs
-
 [data]: data
 [deploy]: deploy
 
 [actix-web]: https://actix.rs/
 [aws-lambda-rust-runtime]: https://github.com/awslabs/aws-lambda-rust-runtime
-[data-events]: data/events
 
 ## Contributing
 
@@ -140,4 +129,5 @@ This project is licensed under the [MIT license][license].
 [htsget-lambda]: htsget-lambda
 [license]: LICENSE
 [aws-lambda-rust-runtime]: https://github.com/awslabs/aws-lambda-rust-runtime
+[aws-s3]: https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html
 [minio]: https://min.io/

SECURITY.md

@@ -2,4 +2,4 @@
 
 ## Reporting a Vulnerability
 
-Please report vulnerabilities by opening an issue or sending an email to [email protected]
+Please report vulnerabilities by opening an issue or sending an email to [email protected].

htsget-actix/README.md

@@ -9,7 +9,7 @@
 [actions-url]: https://github.com/umccr/htsget-rs/actions?query=workflow%3Atests+branch%3Amain
 
 > [!IMPORTANT]  
-> The functionality of [htsget-axum] is identical to this crate and it is recommended for all
+> The functionality of [htsget-axum] is identical to this crate, and it is recommended for all
 > projects to use [htsget-axum] instead.
 > 
 > This crate will be maintained to preserve backwards compatibility, however [htsget-axum] is
@@ -29,16 +29,21 @@ This crate is used for running a local instance of htsget-rs. It is based on:
 
 [htsget-http]: ../htsget-http
 
-## Usage
+## Quickstart
 
-This application has the same functionality as [htsget-axum]. To use it, following the [htsget-axum][htsget-axum] instructions, and
-replace any calls to `htsget-axum` with `htsget-actix`.
+Launch a server instance:
 
-It is recommended to use [htsget-axum] because it better fits with the rest of [htsget-rs]. For example [htsget-actix]
-uses the actix-web framework for the ticket server, however it depends on [htsget-axum] for the data server. Also, components
-in [htsget-lambda] use Axum dependencies.
+```sh
+cargo run -p htsget-actix
+```
+
+And fetch tickets from `localhost:8080`:
 
-[htsget-lambda]: ../htsget-lambda
+```sh
+curl 'http://localhost:8080/variants/data/vcf/sample1-bcbio-cancer'
+```
+
+This crate uses [htsget-config] for configuration. All options supported in [htsget-axum] are also supported here.
 
 ### As a library
 
@@ -53,12 +58,13 @@ This crate has the following features:
 * `experimental`: used to enable experimental features that aren't necessarily part of the htsget spec, such as Crypt4GH support through `C4GHStorage`.
 
 ## Benchmarks
+
 Benchmarks for this crate written using [Criterion.rs][criterion-rs], and aim to compare the performance of this crate with the
 [htsget Reference Server][htsget-refserver].
-There are a set of light benchmarks, and one heavy benchmark. Light benchmarks can be performed by executing:
+There are a set of light benchmarks, and one heavy benchmark. For light benchmarks run:
 
 ```
-cargo bench -p htsget-axum -- LIGHT
+cargo bench -p htsget-actix -- LIGHT
 ```
 
 To run the heavy benchmark, an additional vcf file needs to be downloaded, and placed in the [`data/vcf`][data-vcf] directory:
@@ -67,16 +73,17 @@ To run the heavy benchmark, an additional vcf file needs to be downloaded, and p
 curl ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/data_collections/1000_genomes_project/release/20190312_biallelic_SNV_and_INDEL/ALL.chr14.shapeit2_integrated_snvindels_v2a_27022019.GRCh38.phased.vcf.gz > data/vcf/internationalgenomesample.vcf.gz
 ```
 
-Then to run the heavy benchmark:
+Then run the heavy benchmark:
 
 ```
-cargo bench -p htsget-axum -- HEAVY
+cargo bench -p htsget-actix -- HEAVY
 ```
 
 [criterion-rs]: https://github.com/bheisler/criterion.rs
 [htsget-refserver]: https://github.com/ga4gh/htsget-refserver
 [data-vcf]: ../data/vcf
-[htsget-axum]: ../htsget-axum/README.md#usage
+[htsget-axum]: ../htsget-axum/README.md
+[htsget-config]: ../htsget-config/README.md
 
 ## License
 

htsget-axum/README.md

@@ -21,23 +21,32 @@ This crate is used for running a server instance of htsget-rs. It is based on:
 
 [htsget-http]: ../htsget-http
 
-## Usage
+## Quickstart
 
-### For running htsget-rs as an application
+Launch a server instance:
 
-This crate uses [htsget-config] for configuration. See [htsget-config] for details on how to configure this crate.
-
-To run an instance of this crate, execute the following command:
 ```sh
 cargo run -p htsget-axum
 ```
+
+And fetch tickets from `localhost:8080`:
+
+```sh
+curl 'http://localhost:8080/variants/data/vcf/sample1-bcbio-cancer'
+```
+
+This crate uses [htsget-config] for configuration.
+
+### Storage backends
+
 Using the default configuration, this will start a ticket server on `127.0.0.1:8080` and a data block server on `127.0.0.1:8081`
 with data accessible from the [`data`][data] directory. This application supports storage backends defined in [htsget-storage].
 
 To use `S3Storage`, compile with the `s3-storage` feature:
 ```sh
 cargo run -p htsget-axum --features s3-storage
 ```
+
 This will start a ticket server with `S3Storage` using a bucket called `"data"`.
 
 To use `UrlStorage`, compile with the `url-storage` feature.
@@ -51,19 +60,18 @@ See [htsget-search] for details on how to structure files.
 
 #### Using TLS
 
-There two server instances that are launched when running this crate. The ticket server, which returns a list of ticket URLs that a client must fetch.
-And the data block server, which responds to the URLs in the tickets. By default, the data block server runs without TLS. 
-To run the data block server with TLS, pem formatted X.509 certificates are required.
+By default, htsget-rs runs without TLS. To use TLS, pem formatted X.509 certificates are required.
 
-For development and testing purposes, self-signed certificates can be used.
-For example, to generate self-signed certificates run:
+For development and testing purposes, self-signed certificates can be used. For example, to generate self-signed certificates run:
 
 ```sh
 openssl req -x509 -newkey rsa:4096 -keyout key.pem -out cert.pem -sha256 -days 365 -nodes -subj '/CN=localhost'
 ```
 
-It is not recommended to use self-signed certificates in a production environment 
-as this is considered insecure.
+It is not recommended to use self-signed certificates in a production environment as this is considered insecure.
+
+There two server instances that are launched when running this crate, the ticket server and data block server. TLS
+is specified separately for both servers.
 
 #### Example requests
 
@@ -73,39 +81,39 @@ Some example requests using `curl` are shown below:
 * GET
 
 ```sh
-curl '127.0.0.1:8080/variants/data/vcf/sample1-bcbio-cancer'
+curl 'http://localhost:8080/variants/data/vcf/sample1-bcbio-cancer'
 ```
 
 * POST
 
 ```sh
-curl --header "Content-Type: application/json" -d '{}' '127.0.0.1:8080/variants/data/vcf/sample1-bcbio-cancer'
+curl --header "Content-Type: application/json" -d '{}' 'http://localhost:8080/variants/data/vcf/sample1-bcbio-cancer'
 ```
 
 * Parametrised GET
 
 ```sh
-curl '127.0.0.1:8080/variants/data/vcf/sample1-bcbio-cancer?format=VCF&class=header'
+curl 'http://localhost:8080/variants/data/vcf/sample1-bcbio-cancer?format=VCF&class=header'
 ```
 
 * Parametrised POST
 
 ```sh
-curl --header "Content-Type: application/json" -d '{"format": "VCF", "regions": [{"referenceName": "chrM"}]}' '127.0.0.1:8080/variants/data/vcf/sample1-bcbio-cancer'
+curl --header "Content-Type: application/json" -d '{"format": "VCF", "regions": [{"referenceName": "chrM"}]}' 'http://localhost:8080/variants/data/vcf/sample1-bcbio-cancer'
 ```
 
 * Service info
 
 ```sh
-curl '127.0.0.1:8080/variants/service-info'
+curl 'http://localhost:8080/variants/service-info'
 ```
 
 ### Crypt4GH
 
-The htsget-rs server experimentally supports serving [Crypt4GH][c4gh] encrypted files to clients. See the [Crypt4GH section][config-c4gh] in the configuration
-for more details on how to configure this.
+The htsget-rs server experimentally supports serving [Crypt4GH][c4gh] encrypted files to clients. See the [Crypt4GH section][config-c4gh]
+in the configuration for more details on how to configure this.
 
-Run the server with the following to enable Crypt4GH support using the [example config][example-config]:
+To use Crypt4GH run the server using the [example config][example-config] and the `experimental` flag:
 
 ```sh
 cargo run -p htsget-axum --features experimental -- --config htsget-config/examples/config-files/c4gh.toml
@@ -119,6 +127,7 @@ curl 'http://localhost:8080/reads/data/c4gh/htsnexus_test_NA12878?referenceName=
 
 The output consists of the Crypt4GH header, which includes the original header, the edit lists, and the re-encrypted header that
 the recipient can use to decrypt bytes:
+
 ```json
 {
   "htsget": {