Add encryptionScheme parameter to htsget

[brainstorm]

This is a non-breaking, optional change for htsget.

This is already supported as experimental feature in htsget-rs since umccr/htsget-rs#298. It specifically allows for Crypt4GH requests for each underlying bioinformatics format.

There's also the corresponding work in progress in htsget-compliance testsuite to support this extra parameter/schemes in ga4gh/htsget-compliance#11.

/cc @ohofmann @mmalenic @mlin @jppay

[daviesrob]

How exactly are you implementing crypt4gh in htsget-rs?

There's basically two ways you could do it - either encrypt each segment linked in the htsget response separately, or have them all join together to make a single encrypted file. HTSlib has historically supported the latter, and also looks for the magic number so it can tell an encrypted payload apart from a plain one without having to make any changes to the htsget specification.

[mmalenic]

How exactly are you implementing crypt4gh in htsget-rs?

It's implemented by joining segments together into a single encrypted file. It calculates the bytes to return to the client, and then re-encrypts the Crypt4GH header and adds an edit list packet using a configured public/private key pair.

and also looks for the magic number so it can tell an encrypted payload apart from a plain one without having to make any changes to the htsget specification.

This works, however the encryptionScheme parameter is non-breaking and optional, and I think it would clearer and more practical for the client.

The advantage of adding an additional parameter is that it allows the client to directly request encrypted or non-encrypted files without having to inspect the response header type. I think this lines up with the logic of specifying CRAM/BAM or BCF/VCF in the format parameter. It also allows the client to request encrypted or non-encrypted data for the same <id>.

Similar discussion to #581, where clients can give hints on the data they are expecting to make their logic easier.

[brainstorm]

@daviesrob If needed, shall we discuss this over a the 4-weekly htsget APAC call?:

GA4GH htsget videoconference taking place at an Asia-Pacific
friendly time of 10pm BST / 2pm PDT / 5pm EDT and 
next day 7am Melbourne AEST.

[jmarshall]

I agree with @daviesrob that detecting what you got back is and should be the primary way of selecting how to decode the response.

However it does seem like there would be some benefit in enabling the client to say it wants encryption and having a way for the client and server to negotiate that. Somewhat unclear that that should be in the initial request URL rather than something negotiated via the ticket… 🤔

[jmarshall]

If encryptionScheme is specified a 4 letter code […] When c4gh is specified […]

Where do these 4-character codes come from? Are they in common with some other protocol, or have they been invented for this PR? Are letters really in such short supply that this has to be specifically 4 characters?

[jmarshall]

This doesn't make sense and appears to have been copy/pasted from the referenceName section. Do you mean if the server does not recognise a requested encryption code?

[brainstorm]

Good catch, I meant (encrypted) object Id, fixed in 4e6fa1a

[jmarshall]

If this is about the resource's <id>, i.e., what forms the bulk of the path part of the URL, then it should be in the URL parameters section (in fact, it seems this has always been missing there!). That would be a separate PR.

If this is about the 4-letter code designating the encryption method, then it should be here and use the same code terminology.

[brainstorm]

If encryptionScheme is specified a 4 letter code […] When c4gh is specified […]

Where do these 4-character codes come from? Are they in common with some other protocol, or have they been invented for this PR? Are letters really in such short supply that this has to be specifically 4 characters?

I was hesitant to go for free-form string and I borrowed the idea of restricting format names to (at most) a 4 letter enum: BAM, CRAM, VCF, BCF.

Admittedly a lot of encryption schemes might require more than 4 letters to declare, I wish there was a standard that defined this (IETF? W3C? OpenSSL source code?) that enumerates the most common encryption scheme strings. Anyways, very good point, what would you suggest @jmarshall?

[brainstorm]

@jmarshall:

(...) is and should be the primary way of selecting how to decode the response.

Why?