CHANGELOG.md

Changelog

All notable changes to this project will be documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

[Unreleased]

0.9.1 - 2024-10-16

Added

• allow retrieving c4gh keys from secrets manager

Other

• (search) remove unnecessary Arcs and unwraps in test code

0.9.0 - 2024-09-30

Added

• (config) add configuration values for C4GH S3 and Url storage
• Crypt4GH support using LocalStorage

Other

• (deps) bump noodles and tower
• remove ObjectType in favour of a more flattened config hierarchy
• rename type to backend and clarify experimental feature flag
• [breaking] make storage config more explicit
• bump noodles and other dependencies
• [breaking] rename c4gh-experimental to experimental
• [breaking] split out storage lib.rs into another types.rs module
• (storage) use preprocess and postprocess for C4GH storage instead of Arc<Mutex<..>>
• [breaking] allow mutable search trait, use way less Arcs and clones

0.8.1 - 2024-09-03

Other

• release

0.8.0 - 2024-08-04

Other

• update rust msrv
• bump Lambda and noodles dependencies
• (axum) add server tests for axum ticket server
• add routers for data and ticket servers
• move storage module from htsget-search to htsget-storage

0.7.1 - 2024-05-22

Other

• major dep updates, rustls 0.21 -> 0.23, http 0.2 -> 1, reqwest 0.11 -> 0.12, noodles 0.65 -> 0.74 + minor bumps for other crates depending on these.
• Merge branch 'main' of https://github.com/umccr/htsget-rs into mio_h2_bump

0.7.0 - 2024-05-19

Added

• remove blacklisted_headers also from UrlStorage GET and HEAD requests.
• UrlStorage forwarded header blacklist

Other

• cargo fmt
• update MSRV
• [breaking] swap out hyper for reqwest to support redirects
• (refactor) convert unwraps to Results inside concat module
• (search) additional chr11 and chr20 byte ranges tests for bam and cram
• all range tests now concatenate bytes, update CRAM files
• Merge branch 'main' of https://github.com/umccr/htsget-rs into test/concat-responses
• (deps) update noodles to 0.65

0.6.6 - 2024-01-02

Other

• (deps) further minor dependency changes
• (deps) update noodles to 0.60, new clippy warnings

0.6.5 - 2023-11-02

Added

• (search) remove response scheme because response url can specify it
• (search) add response url option to url storage

Fixed

• (s3) properly handle first capture group bucket and add warnings for s3 errors

Other

• Merge pull request #216 from umccr/feat/response_url

0.6.4 - 2023-10-30

Other

• Cargo fmt, brrrr
• Fix clippy to newest 1.73 rust release and use secrets.GITHUB_TOKEN for the release workflow

0.6.3 - 2023-10-02

Fixed

• (search) use bcf test for bcf search
• (search) return empty response when reference name is in header but not in index instead of error

Other

• Merge branch 'main' of https://github.com/umccr/htsget-rs into fix/error-no-index-pos
• (search) formatting

0.6.2 - 2023-09-06

Other

• (deps) fix htsget-test feature flags in htsget-search
• (deps) properly update noodles and aws dependencies
• revert htsget-test change to a dev dependency
• (deps) update msrv and attempt using htsget-test as a dev dependency
• bump deps including rustls 0.21
• Merge branch 'main' of https://github.com/umccr/htsget-rs into feat/htsget-elsa

0.6.1 - 2023-09-05

Other

• updated the following local packages: htsget-test

0.6.0 - 2023-07-11

Added

• move hyper client construction to config and copy it to url storage
• [breaking] implement client tls config
• [breaking] add server config to certificate key pair
• [breaking] add stronger types for certificate key pairs
• introduce cert and key parsing into config

Other

• Merge branch 'main' of https://github.com/umccr/htsget-rs into fix/http1

0.5.3 - 2023-06-25

Fixed

• (search) enable http/1.1 ALPN on UrlStorage client

0.5.2 - 2023-06-20

Other

• bump deps

0.5.1 - 2023-06-19

Fixed

• avoid overwriting forwarded headers in url when formatting response

Other

• add tests for extending urls and headers
• (search) allow http or https url storage client

0.5.0 - 2023-06-08

Other

• nightly clippy suggestions
• remove unused dependencies and update msrv
• use http::Uri directly in config
• (search) swap out reqwest for hyper
• update remaining dependencies, hold back tokio-rustls due to conflicting versions with actix-web
• update noodles, remove repeated code in search

0.4.1 - 2023-06-02

Other

• updated the following local packages: htsget-config

0.3.0 - 2023-05-29

Added

• implement url formatting for url storage
• (search) [breaking] implement get request and streamable type for url storage
• (search) implement head request for url storage
• (search) get url from key
• (search) add url storage struct
• (config) use proper url parsing in config
• (search) include request headers to storage options
• (search) re-export some config types
• (config) add url-storage feature flag
• add error type to config
• (search) add pub rustls server config function
• (test) add multiple resolvers for server tests and test resolution
• use serve_at in data server rather than a constant
• (config) add automatic config inference for local and s3 storage, and rearrange modules
• [breaking] simplify storage config by allowing untagged enum representation

Fixed

• inserting keys with the same name multiple times into headers serializes correctly
• local storage on windows no longer returns urls with incorrect paths
• (search) use url directly instead of converting to string first
• use correct help context for a crate using htsget-config
• (search) return error instead of panicking when a TLS key is not found
• (release) Bump all crates to 0.1.2 as explored in release-plz/release-plz#507 (comment)
• fixes for byte range ends
• fix IOError name
• fixes & model improvements & bam search progress
• fixes & model improvements & bam search progress
• fix docs

Other

• update for UrlStorage
• Merge branch 'main' of https://github.com/umccr/htsget-rs into feat/url_storage
• (search) add authorization header check to test server
• (search) fix head content-length behaviour and add remaining storage tests
• (search) add mock server and request tests
• (search) add url format tests
• (search) add get url from key test
• [breaking] rename AwsS3Storage to S3Storage in search
• [breaking] http refactor, pass request with query
• remove s3-storage as default
• (config) rename ResolveResponse functions
• (config) add logic for url storage in resolvers
• Merge branch 'main' of https://github.com/umccr/htsget-rs into feat/actix-tls
• (search) return plain ServerConfig from rustls load function rather than Arc
• (search) convert match to if let
• (search) add warning when a non-valid PL read group header is found
• (search) add additional tests for searching resolvers and from storage
• (search) implement ResolveResponse on HtsGetFromStorage
• [breaking] rename HttpTicketFormatter and remove UrlFormatter implementation for it
• [breaking] move htsget structs to config, and resolve storage type in config
• manually bump crate versions to 0.1.4
• make htsget-test a regular dependency
• bump crate versions to 0.1.3 manually
• allow htsget-test to be published and bump deps
• specify htsget-test version
• [breaking] move CertificateKeyPair to config to simplify data server logic
• (search) apply rustfmt
• Merge pull request #154 from umccr/release-plz/2023-02-07T21-45-42Z
• release
• Downgrade release after fixing cargo publish circular dep issues as discussed in release-plz/release-plz#507 (comment)
• Update cargo files (#152)
• Remove s3-server and dependencies (#150)
• (deps) bump tokio from 1.24.0 to 1.24.2 (#151)
• (fix) Remove version from htsget-test and mark it for publish=false to avoid circular dependency as recommended by @Marcoleni in release-plz/release-plz#452 (comment)
• release (#148)
• Set MSRV on all sub-crates (#146)
• Better CI (#98)
• Merge branch 'main' of https://github.com/umccr/htsget-rs into more-flexible-config-rename
• add test for long resolvers from environment variable config
• clippy and fmt
• allow specifying tags, reference names, fields with an 'All' value
• reduce some options for cors, remove repeated code when configuring cors
• fix tests affected by config, change some default values and move around config options
• apply changes to other crates from reworked config
• deserialize empty string as None value
• move cors config to separate file
• fix errors relating to new config
• add tests for checking for contained value in interval
• add safe cast for conversion between i64 and u64
• move fields, tags, no tags, query, and interval to config
• Make search structs public
• Remove ReferenceSequenceInfo
• Remove unused code and logic
• Add tests for no end position
• Remove requirement for default end position when converting to noodles interval
• Merge branch 'main' of https://github.com/umccr/htsget-rs into exports
• Export some functions for use as a dependency
• Re-export htsget-config as a dependency from other crates
• Update non-noodles dependencies
• Http and tls server test uses test utils
• Convert preflight server test to test utils
• Convert data server test to use test utils
• Implement http test traits for data server
• Add CorsLayer responses to htsget-http-lambda
• Move configure_cors to module file
• Bump deps for noodles and simplify logic around maximum reference sequence length using new noodles types
• List out allowed methods rather than sending wildcard
• Add ticket server cors tests
• Add cors options request test
• Fix tests implementation
• Can't use base directory function in test
• Implement additional cors test
• Add base path only function in tests for code reuse
• Add cors tests
• Layer should go after merge
• Implement cors for data server
• Rename ticket server to data server
• Move data server config to separate struct
• Implement cors for htsget-http-actix.
• Changes to deployment (#116)
• Remove some trace log details to avoid making them overly long.
• Add more detail to gzi traces.
• Add more spans and tracing calls.
• Add buffered reading to bai and gzi.
• Add buffered reading to cram search.
• Remove logging chunks as it is too noisy.
• Emit trace logs from functions.
• Add some more instrument targets, use span in_scope.
• Remove sleep call.
• Add a few more tracing span targets.
• Add span tracing to test timings.
• Small changes related to indices (#114)
• Bump dependencies, fix clippy warnings.
• Avoid reading index unless it is required.
• Remove some unnecessary unwraps.
• Improve errors so that they are more informative.
• Remove RangeBounds on BytesPosition as its use is less readable with classes.
• Fix tests related to response class.
• Remove headers from response if empty.
• Simplify JsonUrl logic.
• Responses should contain a class for all ranges, or no ranges.
• Fix tests.
• Implement RangeBounds for BytesPosition.
• Perform byte position merging when creating data blocks.
• Byte position records class, Header for only header bytes, Body for only body bytes, and None if there is a mix of bytes.
• Allow BytesPosition to record its own class.
• Fix unneseccary storage queries (#107)
• Simplify querying for all records by determining file size.
• Server benchmarks should use non-tls ticket server as this is a fairer comparison to the htsget-refserver.
• Clean up code, format, update dependencies.
• Tests run independently by using dynamic port allocation.
• The GC info field should have a Float type rather than an Integer type.
• Implement non-tls ticket server alongside tls ticket server.
• Rename some traits and structs to clarify their purpose.
• Bump many deps (except querymap) and avoid pulling full tokio in, we just need macros and rt-multi-thread (#96)
• Out of order urls (#95)
• Pinning to noodles-tabix =0.9.0 as suggested in zaeleus/noodles#90 (comment) as a result of getting CI errors on https://github.com/umccr/htsget-rs/runs/6803593182?check_suite_focus=true#step:6:90
• Fix eof errors (#87)
• Add benchmarks (#59)
• Fix localstorage path (#86)
• Fix tests and errors (#83)
• Deploy htsget-http-lambda. (#81)
• Enable choosing between storage types. (#80)
• Remove file from localstorage (#79)
• Spawn s3-server once so that tests don't have to be run on one thread. (#78)
• Remove blocking (#77)
• Htsget http lambda (#76)
• Storage class for s3 (#74)
• Decouple File struct from Search trait. (#70)
• Fix runtime panics from curl (#69)
• Bump all tokio versions and stay on track with Noodles versioning instead of working from its git /cc @andrewpatto
• Implement id resolver (#60)
• Convert Storage and HtsGet traits to use async/await (#56)
• Bump up noodles across crates, otherwise several versions get mixed up
• Add the service info endpoints (#54)
• Add the htsget-http-core and htsget-http-actix crates (#45)
• Track crates.io version of noodles (#53)
• Use file size for end bytes ranges.
• Refactor commonalities across all formats.
• Providing the file size through the Storage abstraction. (#49)
• Implement CRAM search backend. (#44)
• Add BCF support (#43)
• Improve the bytes ranges for the BAM header (#42)
• VCF search interface implementation (/variants endpoint) (#37)
• Adapt tests to noodles changes (#41)
• Htsget tests (#40)
• Calculate BAM byte ranges more accurately (#35)
• Remove duplicity in Query, UrlOptions and Url (#29)
• Fix Local Storage always adding the Range header (#31)
• Storage model tests (#28)
• Merge pull request #25 from chris-zen/vcf_bcf_test_data
• Add some tests for LocalStorage::url (#22)
• Add the rest of the builder methods to Query (#21)
• Implement class attribute for reads. (#19)
• Update references and README (#18)
• add tests for BytesRange (#14)
• merge byte ranges
• some renames
• Fix BAM search for unmapped & clippy errors
• add search by reference name and range
• add TODO for BamSearch::url tests
• add TODO for BamSearch::url tests
• added test for HtsGetFromStorage
• added 2 tests for BamSearch
• preparing tests for BamSearch
• add tests and fixes for LocalStorage
• reorganized + rustdocs
• introduced the concept of Storage
• failed attempt to use a BAI index
• work in progress

0.2.0 - 2023-04-28

Added

• (test) add multiple resolvers for server tests and test resolution
• use serve_at in data server rather than a constant
• (config) add automatic config inference for local and s3 storage, and rearrange modules
• [breaking] simplify storage config by allowing untagged enum representation

Other

• (search) convert match to if let
• (search) add warning when a non-valid PL read group header is found
• (search) add additional tests for searching resolvers and from storage
• (search) implement ResolveResponse on HtsGetFromStorage
• [breaking] rename HttpTicketFormatter and remove UrlFormatter implementation for it
• [breaking] move htsget structs to config, and resolve storage type in config

0.1.1 - 2023-01-27

Fixed

• fixes for byte range ends
• fix IOError name
• fixes & model improvements & bam search progress
• fixes & model improvements & bam search progress
• fix docs

Other

• Set MSRV on all sub-crates (#146)
• Better CI (#98)
• Merge branch 'main' of https://github.com/umccr/htsget-rs into more-flexible-config-rename
• add test for long resolvers from environment variable config
• clippy and fmt
• allow specifying tags, reference names, fields with an 'All' value
• reduce some options for cors, remove repeated code when configuring cors
• fix tests affected by config, change some default values and move around config options
• apply changes to other crates from reworked config
• deserialize empty string as None value
• move cors config to separate file
• fix errors relating to new config
• add tests for checking for contained value in interval
• add safe cast for conversion between i64 and u64
• move fields, tags, no tags, query, and interval to config
• Make search structs public
• Remove ReferenceSequenceInfo
• Remove unused code and logic
• Add tests for no end position
• Remove requirement for default end position when converting to noodles interval
• Merge branch 'main' of https://github.com/umccr/htsget-rs into exports
• Export some functions for use as a dependency
• Re-export htsget-config as a dependency from other crates
• Update non-noodles dependencies
• Http and tls server test uses test utils
• Convert preflight server test to test utils
• Convert data server test to use test utils
• Implement http test traits for data server
• Add CorsLayer responses to htsget-http-lambda
• Move configure_cors to module file
• Bump deps for noodles and simplify logic around maximum reference sequence length using new noodles types
• List out allowed methods rather than sending wildcard
• Add ticket server cors tests
• Add cors options request test
• Fix tests implementation
• Can't use base directory function in test
• Implement additional cors test
• Add base path only function in tests for code reuse
• Add cors tests
• Layer should go after merge
• Implement cors for data server
• Rename ticket server to data server
• Move data server config to separate struct
• Implement cors for htsget-http-actix.
• Changes to deployment (#116)
• Remove some trace log details to avoid making them overly long.
• Add more detail to gzi traces.
• Add more spans and tracing calls.
• Add buffered reading to bai and gzi.
• Add buffered reading to cram search.
• Remove logging chunks as it is too noisy.
• Emit trace logs from functions.
• Add some more instrument targets, use span in_scope.
• Remove sleep call.
• Add a few more tracing span targets.
• Add span tracing to test timings.
• Small changes related to indices (#114)
• Bump dependencies, fix clippy warnings.
• Avoid reading index unless it is required.
• Remove some unnecessary unwraps.
• Improve errors so that they are more informative.
• Remove RangeBounds on BytesPosition as its use is less readable with classes.
• Fix tests related to response class.
• Remove headers from response if empty.
• Simplify JsonUrl logic.
• Responses should contain a class for all ranges, or no ranges.
• Fix tests.
• Implement RangeBounds for BytesPosition.
• Perform byte position merging when creating data blocks.
• Byte position records class, Header for only header bytes, Body for only body bytes, and None if there is a mix of bytes.
• Allow BytesPosition to record its own class.
• Fix unneseccary storage queries (#107)
• Simplify querying for all records by determining file size.
• Server benchmarks should use non-tls ticket server as this is a fairer comparison to the htsget-refserver.
• Clean up code, format, update dependencies.
• Tests run independently by using dynamic port allocation.
• The GC info field should have a Float type rather than an Integer type.
• Implement non-tls ticket server alongside tls ticket server.
• Rename some traits and structs to clarify their purpose.
• Bump many deps (except querymap) and avoid pulling full tokio in, we just need macros and rt-multi-thread (#96)
• Out of order urls (#95)
• Pinning to noodles-tabix =0.9.0 as suggested in zaeleus/noodles#90 (comment) as a result of getting CI errors on https://github.com/umccr/htsget-rs/runs/6803593182?check_suite_focus=true#step:6:90
• Fix eof errors (#87)
• Add benchmarks (#59)
• Fix localstorage path (#86)
• Fix tests and errors (#83)
• Deploy htsget-http-lambda. (#81)
• Enable choosing between storage types. (#80)
• Remove file from localstorage (#79)
• Spawn s3-server once so that tests don't have to be run on one thread. (#78)
• Remove blocking (#77)
• Htsget http lambda (#76)
• Storage class for s3 (#74)
• Decouple File struct from Search trait. (#70)
• Fix runtime panics from curl (#69)
• Bump all tokio versions and stay on track with Noodles versioning instead of working from its git /cc @andrewpatto
• Implement id resolver (#60)
• Convert Storage and HtsGet traits to use async/await (#56)
• Bump up noodles across crates, otherwise several versions get mixed up
• Add the service info endpoints (#54)
• Add the htsget-http-core and htsget-http-actix crates (#45)
• Track crates.io version of noodles (#53)
• Use file size for end bytes ranges.
• Refactor commonalities across all formats.
• Providing the file size through the Storage abstraction. (#49)
• Implement CRAM search backend. (#44)
• Add BCF support (#43)
• Improve the bytes ranges for the BAM header (#42)
• VCF search interface implementation (/variants endpoint) (#37)
• Adapt tests to noodles changes (#41)
• Htsget tests (#40)
• Calculate BAM byte ranges more accurately (#35)
• Remove duplicity in Query, UrlOptions and Url (#29)
• Fix Local Storage always adding the Range header (#31)
• Storage model tests (#28)
• Merge pull request #25 from chris-zen/vcf_bcf_test_data
• Add some tests for LocalStorage::url (#22)
• Add the rest of the builder methods to Query (#21)
• Implement class attribute for reads. (#19)
• Update references and README (#18)
• add tests for BytesRange (#14)
• merge byte ranges
• some renames
• Fix BAM search for unmapped & clippy errors
• add search by reference name and range
• add TODO for BamSearch::url tests
• add TODO for BamSearch::url tests
• added test for HtsGetFromStorage
• added 2 tests for BamSearch
• preparing tests for BamSearch
• add tests and fixes for LocalStorage
• reorganized + rustdocs
• introduced the concept of Storage
• failed attempt to use a BAI index
• work in progress