Feature: Storage backend for ID-based APIs (DRS-style resolution)

[jhagberg]

Feature

A new storage backend that supports APIs where files are addressed by opaque
IDs rather than direct paths. The backend performs a resolution step —
translating the htsget request path into a file ID — before constructing
ticket URLs.

Motivation

htsget and DRS (Data Repository Service)
are both GA4GH standards, but htsget-rs currently has no way to serve data
from DRS-style repositories where file access requires an ID lookup.

The existing backends all assume a direct mapping from the htsget request ID
to a storage location:

• FileStorage — ID maps to filesystem path
• S3Storage — ID maps to S3 key
• UrlStorage — ID maps to {base_url}/{key}

This works when the data backend uses the same addressing scheme as htsget.
But a growing number of genomic data repositories use ID-based APIs where the
relationship between a human-readable path and the download endpoint requires
a lookup step:

htsget request:    GET /reads/{dataset}/{filepath}
                         ↓
resolution step:   dataset + filepath  →  fileId     (via API call)
                         ↓
ticket URLs:       GET /files/{fileId}/content  (with Range header)

UrlStorage cannot do this — it constructs URLs by concatenating a base URL
with the key, with no intermediate resolution.

Note: the ID-based data endpoint (/files/{fileId}/content) already provides
streaming and Range support. The missing piece is the {dataset}/{filepath}
→ fileId resolution step before URL construction.

Concrete use case: The NeIC Sensitive Data Archive
(SDA) is a federated genomic archive used by Nordic research institutions.
We are building a new download API (v2 spec)
with a DRS-inspired design — ID-based file access, split header/content
endpoints, and GA4GH service-info. The API is not a standalone DRS service
today but is designed to be easily separable into one in the future.

The v2 API endpoints relevant for htsget:

• GET /datasets/{datasetId}/files → list files (returns fileId per file)
• GET /files/{fileId}/content → encrypted data segments (Range-capable)
• GET /files/{fileId}/header → Crypt4GH header

htsget-rs is already used with SDA via UrlStorage pointed at an internal
path-based endpoint (/s3-encrypted/{dataset}/{filepath}). This path-based
endpoint is being retired in the v2 API. An ID-resolving backend would let
htsget-rs work with the new API without maintaining a legacy endpoint.

Proposed approach

A new backend (working name: DrsStorage or ResolverStorage) that:

1. Resolves the key to a file ID via a configurable API call
   (e.g. GET /datasets/{datasetId}/files?filePath={filepath} → returns fileId)
2. Caches the resolution (genomic archives typically treat files as immutable)
3. Constructs ticket URLs using the resolved ID
   (e.g. {response_url}/files/{fileId}/content)
4. Implements get()/head() by proxying to the ID-based data endpoint

It would reuse the existing response_url pattern and forward_headers
mechanism (forwarding request headers to backend/ticket fetch path per config)
from UrlStorage, and could be feature-gated like the S3 and URL backends.

Example config

[[locations]]
regex = "^(?P<dataset>[^/]+)/(?P<filepath>.+)$"
substitution_string = "$dataset/$filepath"

backend.kind = "Drs"
backend.api_url = "http://download-internal:8080"
backend.response_url = "https://download.example.org"
backend.resolve_endpoint = "/datasets/{dataset}/files?filePath={filepath}"
backend.content_endpoint = "/files/{fileId}/content"
backend.forward_headers = true
backend.header_blacklist = ["Host"]

Alternatives considered

1. Keep a path-based endpoint in the data repository — works but forces
   repositories to maintain legacy endpoints just for htsget compatibility
2. External sidecar that pre-resolves paths — operationally complex,
   static mapping breaks when files are added
3. Regex/substitution in UrlStorage — cannot do HTTP lookups, only
   string transformations

Scope

Happy to contribute an implementation if there's interest. The SDA team has
experience with htsget-rs (we maintain a deployment using UrlStorage + C4GH)
and can provide integration testing against a real archive.

Would like to hear your thoughts on:

• Whether this fits htsget-rs scope or is better as an external plugin
• Naming: DrsStorage vs ResolverStorage vs something else
• Any architectural preferences for the resolution/caching layer

References

• SDA repository
• SDA Download API v2 spec
• GA4GH DRS spec

[mmalenic]

Hi @jhagberg

I think this a great idea to add to htsget-rs and is in-scope. We were already discussing having DRS support and integration for htsget-rs so I'd love to have this in.

Design

In terms of design, we could keep it like you suggested, although I'd also like to propose the possibility of having this stick close to the DRS spec. I think matching that is beneficial as we wouldn't have to use custom schemes for how objects are listed and returned. It also gives htsget-rs a GA4GH interoperability bonus.

Given a htsget request:
GET /reads/<dataset>/<filePath>

Could we assume a DrsObject present at?
GET /objects/<dataset>/<filePath>

We could have a separate backend.content_endpoint option, but I feel like the DRS spirit would be more in-line with getting the access_methods[].access_url field from the returned DrsObject. E.g. this field could have the the fileId already resolved in a url at /files/{fileId}/content. This could then be used to construct the URL tickets (and internally by htsget to fetch what it needs to calculate ranges).

I'm happy for this to be kept as simple as possible for now. E.g. only assuming access_methods[].access_url exists and is of type https, and ignoring complexities around authorization and passports by just using forward_headers.

Comments

I think the main difference between this approach and the one you suggested (other than endpoint naming conventions) is that it pushes the resolution step from fetching <fileIds> directly to fetching complete access urls pre-formatted (which would include the <fileId> in your case).

It also doesn't use a list-based request with a query parameter, and instead directly accesses the DrsObject. We could use the POST /objects DRS request if that's easier as well.

I'm not sure if these differences would be feasible from your end, or if you'd like to keep it as you suggested.

I'd love to hear your thoughts on this and if there is anything I misunderstood or missed.

Implementation

For naming I'd go with DrsStorage. If we are going with a DRS-based approach, you might not need to have resolve_endpoint, content_endpoint in the config, as the DRS spec would define those.

E.g.

• resolve_endpoint could be /objects/<dataset>/<filePath>.
• content_endpoint could be access_methods[].access_url from the DrsObject.

You might not even need response_url, depending on how the client should fetch the data, and whether that is the same as htsget-rs.

For feature gating, yes, I'd implement this feature-gated using #[cfg(feature = "drs")].

For caching, there is a caching layer for UrlStorage already which uses standard http caching headers, which could be used:

htsget-rs/htsget-config/src/config/advanced/mod.rs

Lines 104 to 110 in e57a3f9

	reqwest_middleware::ClientBuilder::new(inner_client)
	.with(Cache(HttpCache {
	mode: CacheMode::Default,
	manager: CACacheManager::new(client_cache, false),
	options: HttpCacheOptions::default(),
	}))
	.build()

More than happy for you guys to implement this feature if you'd like. I think it would be fairly big as it would touch many htsget-rs crates (likely all of them if using feature-gating, but mostly htsget-config, htsget-storage and possibly htsget-test), so I'm also happy to take it given the size.

Let me know how you'd like to proceed.

[jhagberg]

Hi @mmalenic,

Thanks for the detailed feedback — we've aligned with your suggestion to stick close to the DRS spec.

I've suggested that we add a minimal GET /objects/{datasetId}/{filePath} endpoint to our download API that returns a DRS 1.5 DrsObject with access_methods[].access_url pointing to the resolved download URL. The resolution step would happen server-side, so htsget-rs gets a standard DRS object with everything it needs. That work is tracked in neicnordic/sensitive-data-archive#2246.

On the htsget-rs side, we've written up a design for a DrsStorage backend on a branch here: feat/drs-storage-backend.

We're happy to contribute the implementation. Would be great to hear if this direction looks right before we start writing code.

[mmalenic]

Hi @jhagberg

After some discussions internally about how best to approach this, I think it would be best to keep this slightly less dependent on DRS. Apologies for leading it the wrong way with my initial comment.

I think it's best to implement it as a configurable storage backend that can set the desired endpoint and path, similar to your original suggestion. This can still be compatible with DRS, however I'd like to make it compatible with your original suggestion too.

There shouldn't be any requirement to change the endpoints on the sda (although you're welcome to do that if you were already planning to).

Looking at the endpoint API docs, I think there's no need to have a separate content_endpoint, because the resolve_endpoint already returns the downloadUrl?

Here's what I think the config could look like:

[[locations]]
regex = "^(?P<dataset>[^/]+)/(?P<filepath>.+)$"
substitution_string = "$dataset/$filepath"

backend.kind = "Resolve"
backend.api_url = "http://download-internal:8080"
backend.response_url = "https://download.example.org"
backend.resolve_endpoint = "/datasets/{dataset}/files?filePath={filepath}"
# Json path to get download location from resolve endpoint response
backend.content_location = "$.files[0].downloadUrl"
backend.forward_headers = true
backend.header_blacklist = ["Host"]

Let me know if that proposal works for you.

To progress, I'll take over and implement it, as there's a few things in relation to caching I'd like to investigate. Namely, I think it should be possible to integrate the caching with the existing HTTP cache rather than having a separate HashMap on the storage struct.

[jhagberg]

Hi @mmalenic,

No worries at all — the generic Resolve approach makes a lot of sense. It covers our use case and is more flexible for others.

Looking at the config you proposed, it would work with our existing /datasets/{dataset}/files?filePath={filepath} endpoint out of the box using $.files[0].downloadUrl. We've also opened a PR for a minimal DRS /objects endpoint on the SDA side (neicnordic/sensitive-data-archive#2296) — that would give a second option with a simpler JSONPath ($.access_methods[0].access_url.url), but either way works.

Happy to let you take the lead on implementation. If there's anything you'd like to delegate or if you want someone to help test against a real SDA deployment, just let me know.