Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP: GPG remote trusted key updating #2260

Draft
wants to merge 15 commits into
base: main
Choose a base branch
from

Conversation

dbnicholson
Copy link
Member

This is a work in progress that I started a while back and would like some comments on. In order to securely update the trusted GPG keys for a remote, it adds an implementation of the OpenPGP Web Key Directory client side. This lets a remote publish updated GPG keys at a known location and have ostree fetch them to update its local keyring.

There are still parts that need some polish, but I believe it mostly works. I'm mostly posting this to get feedback. The impetus for this work is that at Endless we had one of our trusted keys expire, which would have prevented people from updating. We had a workaround by having a 2nd key in the trusted keyring, but we would have preferred a reliable way to push an update for the expiring key.

Currently the verifier only imports all the GPG keys when verifying
data, but it would also be useful for inspecting the trusted keys.
In order to use the GPG verifier, it needs to be seeded with GPG keys
after instantation. Currently this is only used for verifying data, but
it will also be used for getting a list of trusted GPG keys in a
subsequent commit.
This function enumerates the trusted GPG keys for a remote and returns
an array of `GVariant`s describing them. This is useful to see which
keys are collected by ostree for a particular remote. The same
information can be gathered with `gpg`. However, since ostree allows
multiple keyring locations, that's only really useful if you have
knowledge of how ostree collects GPG keyrings.

The format of the variants is documented in
`OSTREE_GPG_KEY_GVARIANT_FORMAT`. This format is primarily a copy of
selected fields within `gpgme_key_t` and its subtypes. The fields are
placed within vardicts rather than using a more efficient tuple of
concrete types. This will allow flexibility if more components of
`gpgme_key_t` are desired in the future.
This provides a wrapper for the `ostree_repo_remote_get_gpg_keys`
function to show the GPG keys associated with a remote. This is
particularly useful for validating the GPG key updates have been
applied.
This will be used to implement the PGP Web Key Directory (WKD) URL
generation. This is a slightly cleaned up implementation[1] taken from
the zbase32 author's original implementation[2]. It provides a single
zbase32_encode API to convert a set of bytes to the zbase32 encoding.

I believe this should be acceptable for inclusion in ostree. The license
in the source files is BSD style while the original repo LICENSE file
claims the Creative Commons CC0 1.0 Universal license, which is public
domain.

1. https://github.com/dbnicholson/libbase32/tree/for-ostree
2. https://github.com/zooko/libbase32
Calculate the advanced and direct update URLs for the key discovery
portion[1] of the OpenPGP Web Key Directory specification, and include
the URLs in the key listing in ostree_repo_remote_get_gpg_keys(). These
URLs can be used to locate updated GPG keys for the remote.

1. https://tools.ietf.org/html/draft-koch-openpgp-webkey-service-08#section-3.1
If the key UID contains a valid email address, include the GPG WKD
update URLs in GVariant returned by ostree_repo_remote_get_gpg_keys().
This provides a wrapper for the `ostree_repo_remote_update_gpg_keys` API
to update a remote's GPG trusted keys using the PGP Web Key Directory
protocol.
In order to test `ostree_remote_update_gpg_keys`, we need to be able to
fetch the keys from a local test server. This inherently requires
introducing a backdoor to the update process. If the
_OSTREE_GPG_UPDATE_LOCAL_PORT environment variable is set, change the
server to http://127.0.0.1:<port> after validating that the port is
numerical. This should keep any attack local by not allowing the URL to
be changed to an arbitrary remote server.
@openshift-ci-robot
Copy link
Collaborator

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: dbnicholson
To complete the pull request process, please assign cgwalters after the PR has been reviewed.
You can assign the PR to them by writing /assign @cgwalters in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci-robot
Copy link
Collaborator

@dbnicholson: PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@dbnicholson dbnicholson marked this pull request as draft January 9, 2021 16:12
@dbnicholson dbnicholson changed the title GPG remote trusted key updating WIP: GPG remote trusted key updating Jan 9, 2021
@cgwalters
Copy link
Member

For Fedora(host systems) we use gpgkeypath=/etc/pki/rpm-gpg/ - so keys are delivered as part of the ostree commit itself.
For Fedora Flatpak it's currently via OCI which...hm, I don't think we're GPG verifying right now.

For flatpak in general, wouldn't it be a lot simpler to sign .flatpakrepo files?

@cgwalters
Copy link
Member

To be clear I'm not opposed to this, I just want to be sure that the simple solutions aren't sufficient.

@dbnicholson
Copy link
Member Author

For Fedora(host systems) we use gpgkeypath=/etc/pki/rpm-gpg/ - so keys are delivered as part of the ostree commit itself.

This is the same as Endless - key updates are delivered in OS updates via /usr/share/ostree/trusted.gpg.d (which is safer than something in /etc that might not survive the merge, but I digress). The trusted path to key updates is the key itself since you can't receive new keys without the key that verifies the update.

What happens when the OS update is no longer signed by that trusted key because it's expired or is compromised? There needs to be an out of band mechanism to update the local trusted key. Currently that out of band mechanism would be manual and technical - you'd send out a bunch of emails and make a web page explaining the situation with the new trusted key and instructions on how to add it to the local key store. This is a pretty high barrier. It requires the user to be looking for that type of information and to know how to handle a security sensitive process correctly. Turning that manual process into an automated process is the goal of this work.

For flatpak, there's already a mechanism for delivering keys via the xa.gpg-keys metadata key in the summary file. However, since flatpak remotes by default have summary GPG verification, I think it suffers from the same situation as above. If the remote summary needs to be signed by a new key because the existing one can no longer be trusted, then the client won't be able to verify it to receive updated keys and manual intervention will be needed. It felt to me more appropriate to deliver the keys out of band than through the repository itself. I.e., use a different trusted path to update this trusted path. I also was interested in a neutral method that could live in libostree to be used by any ostree application.

@dbnicholson
Copy link
Member Author

Just for some other background, here's a page on the gnupg wiki discussing different key update methods - https://wiki.gnupg.org/OpenPGPEmailSummit201607/KeyDiscoveryComparison. There are some other pages on the wiki discussing these types of things.

@dbnicholson
Copy link
Member Author

Lastly, I sent an email to the mailing list in https://mail.gnome.org/archives/ostree-list/2021-January/msg00000.html if you want to have a more general discussion there.

@cgwalters
Copy link
Member

What happens when the OS update is no longer signed by that trusted key because it's expired or is compromised?

If it's compromised...I don't see how we can do anything? I need to dig into this a bit but what's the trust model for the web service? Is it just "TLS with keys from ca-certificates"?

This WKD stuff seems to mostly be about authenticating the email address in GPG keys, which seems orthogonal?

I guess let me say it this way: You've added a lot of new code here but I am missing a brief document that describes how one might set this up and use it. And a specific question: if ostree had had this support originally when your key expired, would it have saved you? How?

I feel like this key expired case is actually a special case of a more general one where one wants an "out of band hotfix" process for an ostree-based system. I believe Firefox has something like this - a mechanism they can use for truly critical fires.

A "hotfix system" wouldn't need to be implemented in ostree at all really, it could be a separate systemd service that pulls content however it wants and validates it integrity however and basically supports live-updating the system.

Of course, the question is how the integrity of those patches is validated...and if you're signing those correctly then presumably you're also signing your ostree commits correctly.

@cgwalters
Copy link
Member

Can you split out the non-WKD patches to a separate PR?

@dbnicholson
Copy link
Member Author

Can you split out the non-WKD patches to a separate PR?

Sure. No promises on when I'll get to that, but there are some things in here that are not specifically for WKD.

@dbnicholson
Copy link
Member Author

What happens when the OS update is no longer signed by that trusted key because it's expired or is compromised?

If it's compromised...I don't see how we can do anything? I need to dig into this a bit but what's the trust model for the web service? Is it just "TLS with keys from ca-certificates"?

If it's compromised but you can easily (or automatically) fetch an updated key from a trusted location, then you can carry on. For example, suppose my ostree signing key got compromised. I can generate a new key that has the same ID (email address) and publish it on a web server in a WKD compliant location. The client checks for an updated key at that location, sees there's something new, updates the local keyring and we carry on. It's basically the same process as if I'd written an email saying where to get my new key from and what to do with it except that it can happen automatically because the URL is predictable.

The trust model is TLS here and the fact that the key can only be fetched from a fixed URL within the domain it's email address is a part of. To spoof that process, an attacker would need to send a fake DNS reply and provide a valid TLS certificate.

This WKD stuff seems to mostly be about authenticating the email address in GPG keys, which seems orthogonal?

The email address determines the URL where the update can come from, so I was trying to be a bit paranoid about ensuring that was done correctly. This goes in 2 directions - determining the URL to make the request to and then only taking the key for that desired email address in the received keyring.

I guess let me say it this way: You've added a lot of new code here but I am missing a brief document that describes how one might set this up and use it. And a specific question: if ostree had had this support originally when your key expired, would it have saved you? How?

Sure, I can write something up on how you'd do it. https://wiki.gnupg.org/WKDHosting basically explains how you'd populate the server side, but I'll follow up with something more concrete.

It definitely would have saved us. After updating the expiration date on our key, we would have published the new key on our WKD server. Then the clients would fetch the updated key and signatures for our commits would be valid again.

$ /usr/lib/gnupg/gpg-wks-client --print-wkd-url "EOS OSTree Signing Key 1 (EOSK1) <[email protected]>"
https://openpgpkey.endlessm.com/.well-known/openpgpkey/endlessm.com/hu/e5t1hbkongpswdkafkdzkxwwxokkammg?l=maintainers

I feel like this key expired case is actually a special case of a more general one where one wants an "out of band hotfix" process for an ostree-based system. I believe Firefox has something like this - a mechanism they can use for truly critical fires.

A "hotfix system" wouldn't need to be implemented in ostree at all really, it could be a separate systemd service that pulls content however it wants and validates it integrity however and basically supports live-updating the system.

They feel a little different to me. Being able to provide a hotfix because the OS we published is itself broken or compromised sounds very useful. We have more than once released commits that could no longer boot or update and the only thing we could resort to was publishing some instructions for people to run manually. But this issue isn't with the OS itself, it's with the trust path. Hotfixing the payload vs hotfixing the delivery mechanism seems different to me.

That said, there isn't any requirement for this to be in ostree. You could have a service that occasionally checks for and updates the trusted keys. The nice thing about having it in ostree is that any ostree application can make use of it without depending on an outside service.

@cgwalters
Copy link
Member

The trust model is TLS here and the fact that the key can only be fetched from a fixed URL within the domain it's email address is a part of. To spoof that process, an attacker would need to send a fake DNS reply and provide a valid TLS certificate.

I wouldn't assume any security with DNS in general, so this is basically TLS. And if you're trusting that absolutely...what's the point of GPG signing in the first place over just fetching the commit over that same TLS connection?

@dbnicholson
Copy link
Member Author

I wouldn't assume any security with DNS in general, so this is basically TLS. And if you're trusting that absolutely...what's the point of GPG signing in the first place over just fetching the commit over that same TLS connection?

Because they validate different things. TLS validates the transport while GPG validates the payload. Having separate validation of the payload means that you can use an untrusted transport such as any non-TLS network protocol or a local filesystem. Like would happen for peer to peer distribution. Likewise, payload verification means that if someone can insert a rogue commit into the remote repo it would still fail to be validated by clients.

Why do you sign the ostree git tags? Why did you develop git-evtag? Presumably we're all using a secure transport to access the remote repo (HTTPS or SSH), so by the same logic we can assume that any object in the git repo is valid. How do I get your public key to validate the signature on those tags? I wouldn't take it from an object in the repo and then use it to validate a tag on the repo. I would get it from some other trusted source.

TLS is also the security mechanism used for the transport of the trusted keys here, but I don't think that TLS also being the typical repo object transport security mechanism really has any bearing. They're trying to secure and validate access to different things even though they happen to use the same mechanism in this work. Indeed I am trusting TLS here just like we all do every day on the internet and just like the WKD developers determined would be acceptable.

@cgwalters
Copy link
Member

cgwalters commented Feb 3, 2021

I'm not arguing against GPG signing, you are right transport and offline integrity are different things.

I've glanced though the WKD stuff but haven't done a deep dive - it may be somewhere there and I'm just missing it. It seems to mostly be a protocol about discovery, leaving the "how to choose to trust this key" outside of the scope.

You are here basically automatically trusting keys found this way, right? So that to me seems to negate the whole point of using GPG in the first place if we unconditionally load new keys over TLS.

Perhaps one middle ground is that we only try to fetch keys from WKD in the case of expired keys by default? And in that case, require the new key be cross-signed from the old one?

@dbnicholson
Copy link
Member Author

Sorry for taking so long to respond.

I've glanced though the WKD stuff but haven't done a deep dive - it may be somewhere there and I'm just missing it. It seems to mostly be a protocol about discovery, leaving the "how to choose to trust this key" outside of the scope.

You are here basically automatically trusting keys found this way, right? So that to me seems to negate the whole point of using GPG in the first place if we unconditionally load new keys over TLS.

In the case of ostree, we're saying that we already trust the IDs in the keys for the remote's trusted keyring. You or someone on your behalf has already imported my key with [email protected] to trust my repo at the remote URL. What WKD does is give you a URL to find a new version of my key that's at a fixed URL tied to my domain. So, in case my key expires or I need to revoke it there's a known place to get it from that can't really be spoofed unless I lose control of my domain or TLS breaks down.

$ /usr/lib/gnupg/gpg-wks-client --print-wkd-url [email protected]
https://openpgpkey.endlessos.org/.well-known/openpgpkey/endlessos.org/hu/9q46ttg995y1xz6dq5fsheb18u177pz4?l=dbn

So, it's not totally arbitrary, but it does preclude that you trust the IDs that are in your trusted keyrings. Perhaps there could be a remote flag that says whether the key can be updated.

Perhaps one middle ground is that we only try to fetch keys from WKD in the case of expired keys by default? And in that case, require the new key be cross-signed from the old one?

I think revoked is the bigger issue, which you can't tell by looking at the key you currently have. I think cross signing could work. Another option would be that the primary key fingerprint is unchanged. In which case you'd do all your key management on subkeys.

@cgwalters
Copy link
Member

Perhaps there could be a remote flag that says whether the key can be updated.

That's really what I'm focusing on - in exactly what circumstances can a key be updated via this mechanism, and what does it require?

If we're focusing on the expired key problem, then requiring new keys to be cross signed seems like a good default.

Another option would be that the primary key fingerprint is unchanged. In which case you'd do all your key management on subkeys.

Right; this is a common pattern in the GPG world for this.

Here's another random idea though, what if we patched ostree to warn loudly if a key it verifies is e.g. going to expire in a month or even 3 months?
I mean honestly the "time bomb" aspect of this is a huge problem with all long term keys. Fixing this problem is really a lot of the benefit of Let's Encrypt - they force everyone to rotate keys frequently. This seems like a good thing to recommend in general even for GPG. Fedora generates a new key for each Fedora major every ~6 months, though we currently still trust even quite old ones.

@pemensik
Copy link

pemensik commented Mar 11, 2021

Is it okay to rely GPG keys lookup on SHA-1 algorithm used in protocol? Shouldn't it be updated to more secure algorithms? It does not have support for different hash algorithms it seems to me.

@dbnicholson
Copy link
Member Author

Sorry I keep forgetting to reply on this.

For the expiration warnings, I think they could be a little useful. On the server side I probably would have noticed that before the expiration and done something about it. On the client side I can't see it really being useful, though. The warnings wouldn't be seen unless a CLI was in use, and even if they were users can't do anything to fix the issue. Still wouldn't do anything about a key that needed to be revoked, though.

I think that the really scary part here is the fully automated part. So, maybe let's not do that right now. If ostree can just help you see what keys are there and what the WKD URLs are, then anyone could essentially write their own automation around it. Providing a way to fetch and filter the returned key would be nice, but not really necessary. That's really not different than what someone could do now manually - figure out the email address from a remote's key, turn it into a WKD URL, download it and import it back into ostree. What I'm proposing now is having ostree just make parts of that easier to do. I think your initial "hey can you split out these general parts" would cover most of it. Whether someone wants to trust the WKD URL itself or what's returned from downloading it can be someone else's business.

@dbnicholson
Copy link
Member Author

I cleaned up and broke out the non-updating parts to #2401. I think with that in place the actual key updating and associated policy and security decisions can be made by someone else.

@cgwalters cgwalters added difficulty/hard hard complexity/difficutly issue reward/medium Fixing this will be notably useful triaged This issue has been evaluated and is valid labels May 2, 2023
@openshift-ci
Copy link

openshift-ci bot commented Jun 29, 2023

@dbnicholson: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/sanity 6d46edf link true /test sanity
ci/prow/fcos-e2e 6d46edf link true /test fcos-e2e
ci/prow/images 6d46edf link true /test images

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
difficulty/hard hard complexity/difficutly issue do-not-merge/work-in-progress needs-rebase reward/medium Fixing this will be notably useful triaged This issue has been evaluated and is valid
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants