Making CSP more usable for organizations at scale

[swijckmans]

Dear W3C,

As we dive into deep technical conversations on missing features and support of evolving needs in CSP3 I would like to take a step back.

CSP is a challenging thing to implement, and mostly, maintain.

By design, CSP is an allow list. Anything that does not meet that allowed behaviour is either reported upon or stopped. This works for static or self managed assets. However, most 3rd party scripts will change without notice. This is a normal and intended behaviour for client-side scripts.

Today client-side scripts are used for dynamic conditional use cases. A/B testing, but also serving different script versions based on user agents, time of day, location… A chatbot widget may add more assets or tooling.

Depending on the policy these changes would not be allowed and this breaks features or user flows.

As a result, people do not want to use CSP. The complexity of writing a good CSP is definitely part of the problem but the bigger issue is that once a CSP is set to blocking, a week later something would change and in a quick response CSP would be thrown out again because it broke parts of the site and cost the business money. The result: CSP gets a bad name, it gets kicked out. CSP? Never again.

There is more to say here: in many companies the development environment does not have CSP setup (at all or in the same way) as the production environment. So when things get pushed to production the CSP kicks in unexpectedly.

Now this is the problem: in an organization of decent size, the companies that are a prime target for client-side attacks, the people that vouch for client-side security are more often than not people within the security or GRC team.

They have a hard time getting engineering resources, they have a hard time moving quickly and in many companies culturally struggle to get the broader engineering team to allow them to do things. Culturally, they are often not seen as core revenue driving roles instead they are there to cover the business. Those people have to pick their fights. There are rarely second chances…

CSP is a leading cause of emergency production releases. Does a specification work if it causes emergency production releases routinely?

The fact that CSP is a hard coded HTTP header makes it a hard to adopt security feature.

CSP, because of the fact that it needs to be adjusted quickly needs to be easier to adjust.

Today, full web proxies are at a major unfair advantage as they have the ability to change headers at the proxy. While proxies are often compensating for imperfections in specifications, we should strive to improve specifications so that a full proxy is not necessary.

A discussion we need to have: we are doing a good job at discussing CSP directives. Adding directives with dynamicness in mind. But let’s go a level deeper, should CSP be hard coded or can it be a call to an external endpoint or a file? Allowing dynamic management, allowing faster adjustments.

I know about implementing via meta tags, but that is not really what we want here…

There is of course the latency concern, but there are aways to make this work. I'd love to start the thread on how we can make CSP a lot more usable in real world circumstances.

Would love to hear your thoughts.

Simon - CEO cside.dev

[swijckmans]

Adding @mikewest for visibility.

[annevk]

There is of course the latency concern, but there are aways to make this work.

There's quite a few assertions in your post without citation, but this one in particular I'd like to see elaborated upon.

[mikewest]

Thanks for the feedback, @swijckmans. Extracting a few points rather than going line-by-line so if I've missed anything, please do call it out:

1. I agree that it would be nice to be able to support dynamic scripts (and their potentially unknown dependencies). That's something I believe CSP will be able to support via a combination of the existing 'strict-dynamic' behavior and the hopefully-soon-to-be-added https://wicg.github.io/signature-based-sri/ (and in particular https://wicg.github.io/signature-based-sri/#monkey-patch-csp). It would be helpful to understand if there are additional requirements in this area that those mechanisms wouldn't support.

2. It's unclear to me why header-based delivery is something you see as a root cause of problems. And, if it is indeed a blocker, it would be helpful to understand why <meta> isn't a satisfactory replacement.

3. I'm not sure I understand the relevance of proxies, or what CSP should be doing differently in order to reduce their relevance.

4. Pointing to some other resource to define a page's policy is something we've explored at least twice: once as the policy-uri directive in very early drafts of CSP (and, I think, in Mozilla's original implementation?), and again as part of the Origin Policy proposal. Adding a navigation-blocking HTTP request was certainly one concern in both cases, as were more fundamental concerns around caching. As @annevk notes, it would be helpful to understand what ways to make this work might be useful to explore.

[swijckmans]

Thanks for the response @mikewest and @mikewest @annevk.

@annevk - Latency of a 3rd party CSP endpoint.

By design an externally fetch navigation blocking mechanism would make things slower than a direct first party HTTP header. But what I meant by that line is that there are faster and slower ways to do things and that a potential performance impact is not enough to block the existence of a feature that would allow a dynamic 3rd party tool, service or file to define CSP altogether. CSP in its own right adds CPU cycles, security more often than not is extra overhead, but that on its own is not a reason to stop it from being possible.

Ways to make things faster, to Mike's point: caching but perhaps a mechanism similar to early-hints can be helpful. Also worth pointing out that CSP can be layered anyway. I never built a browser and I am likely under-qualified here but just sharing ideas.

A CSP header can be implemented for specific high performance requirements and a request can be made to other another endpoint for more rules that impacts 3rd party fetches specifically.

More complex idea: a setting to allow fetching of first party assets before the external CSP response comes back.

Just throwing out some ideas here. Ultimately it is up to the user whether they wish to pay the performance price to gain security, it not being possible at all is worse.

@mikewest Thanks for the helpful references.

1. I may be wrong or getting lost in the wording, but would strict-dynamic cover this example: 3rd party script X makes sub requests to Y and Z. But they change things up and in subrequest Y a further subrequest to A is being made.

2. Header-based delivery at many companies - depending on their frameworks - means a webserver level adjustment. That often requires a different team from the front-end team.

Another thing I called out on this subject was regular emergency release: when a company with a strict production release process uses a 3rd party client-side script and CSP ends up blocking some change that dynamic 3rd party made this will trigger an emergency response as customers are actively impacted by the CSP no longer being valid. The result is that those who maintain CSP have to act quickly and often use the emergency release process to make CSP changes. That is of course not ideal and gives CSP a bad reputation.

Meta tag based injection has a couple of issues that make it less desirable. Some directives are not supported, I think many can get over that but it is a real limitation. Cross browser support is also a lot less reliable.

People implement Meta tags in the wrong place all the time. Meaning scripts loaded before are a regular occurrence and that window is Christmas to a bad actor. In some JS frameworks the order of injection is also hard to force. Another element is that client-side injections can originate from various places, having the HTTP header separation from the front-end code is a good security feature in its own right. An NPM package can force a client-side injection to be the first script on the page.

3. Proxies: Akamai, Cloudflare, Fastly... full website proxies use their position in the flow of the request to inject CSP headers in the HTTP response. Allowing them to offer nice dashboard to their customers to manage CSP. The current spec really only allows full-web proxies to do that in a managed way. That is majorly limiting but the fact that people use those proxies to manage CSP is an indicator that the subject we are discussing now is a relevant one.

A good question to ask: of the sites that adopted CSP, how many of those use the proxy to inject them? Not easy to find out but my gut feelings is that a substantial amount of CSP usage on sites is proxy injected.

Through checking reporting endpoints that are vendor owned I could probably figure out a way to do that if those stats are relevant to the discussion.

4. I think I jumped on this a few times in the above answers. Good to see other flagged this as well. It is a valid request and idea IMO.

I hope this helps.

[swijckmans]

PS: At my tiny company we have to update CSP 2 times per week because of marketing tools adopting new subdomains or domains entirely. CSP requires constant attention, we need to dig deeper into why adoption is low. I strongly believe that the constant work it requires is a factor.

[mikewest]

Pulling out some specifics below. The dynamic nature of applications seems like the core of the concern you're raising, and I think 'strict-dynamic' and/or nonces are likely going to be a reasonable recommendation for you to explore (see https://web.dev/articles/strict-csp for some thoughts from relevant teams at Google and the quick demo I linked at the bottom for an example).

By design an externally fetch navigation blocking mechanism would make things slower than a direct first party HTTP header. But what I meant by that line is that there are faster and slower ways to do things and that a potential performance impact is not enough to block the existence of a feature that would allow a dynamic 3rd party tool, service or file to define CSP altogether.

First, I find it somewhat difficult to reconcile the embrace of a dynamic third-party tool here with the rejection of proxy-injected headers below. Isn't the latter (e.g. reverse proxies like https://helmetjs.github.io/ or Cloudflare or etc) a perfectly reasonably way of providing the former?

Second, I likewise find it difficult to reconcile the embrace of dynamic third-party tools here with the suggestion that a separation between front-end and server teams makes CSP harder to deploy. It seems easier to imagine collaboration between application teams and server teams within a company than collaboration between application teams and folks entirely outside the company that aim to manage a page's policy. If the latter can work, surely the former can work as well?

Third, from the browser's perspective, we need to know about some specific aspects of the page's policy before we commit a document and start parsing it: frame-ancestors might prevent us from committing the document at all, sandbox might change the document's origin (and therefore the process into which it's committed), etc. This means that we'd end up making this a blocking request in the middle of a navigation, which would add some complexity that we'd need to justify. Certainly not impossible to do, but there are real tradeoffs here that have made this approach difficult to pursue in the past.

to Mike's point: caching but perhaps a mechanism similar to early-hints can be helpful.

Mike's point, unfortunately, was that caching is complicated and that some aspects of CSP (nonces in particular) are directly in conflict with cachability.

Meta tag based injection has a couple of issues that make it less desirable. Some directives are not supported, I think many can get over that but it is a real limitation. Cross browser support is also a lot less reliable.

The non-supported directives are report-uri, frame-ancestors, and sandbox. Which of those do you see as an important barrier to adoption?

I'm also interested in what you see as the unreliable cross-vendor support. Ideally, vendors would align on behavior.

I may be wrong or getting lost in the wording, but would strict-dynamic cover this example: 3rd party script X makes sub requests to Y and Z. But they change things up and in subrequest Y a further subrequest to A is being made.

I put together a quick demo at https://mikewest.github.io/scratchpad/strict-dynamic/. TL;DR: 'strict-dynamic' aims to handle exactly this case.

[simon-friedberger]

In your presentation you also stated that strict-dynamic is not a solution because customers don't understand why they would first apply a CSP and then allow dependencies of their scripts and that it was "not a solution".

Could you elaborate on that? I don't understand why this doesn't satisfy the requirement of managing dependency changes.

[swijckmans]

Hey team,

CSP - practical issues in the real world.pdf

Sorry, time flies! I promised to recap what we discussed on the app-sec meeting in August.

Presentation is attached.

My presentation in short:

1. I conducted a series of interviews with some of cside's customers and combined that with some of my own experiences of working in the client-side security space at a range of companies before.

2. Who pushes for CSP is often not the one with ability to maintain or control it. GRCs, security teams… requiring attention from front-end teams or even infra if the HTTP headers are managed by them.

• GRC teams (Governance Risk Compliance)
• Security teams
• Not the front-end engineers…
• Ofcourse never the marketeers in Google Tag Manager

=> CSP adoption is low and that is not because people didn't try, those that want it have to pick their fights. Those that had it rebound back to removing it.

3. Security teams need to pick their battles. If something they pushed for becomes a headache, requires contestant attention or negatively impacts the business their lives get harder.

• The general rule of thumb, per 50 employees (maybe) 1 security person
• Often do not touch code themselves
• Or at least need to work with another team to get something done
• Have to beg for engineering resources
• Pushback is the norm
• not considered core team because hard to route back revenue or business wins to them
• "Wait, this things need constant maintenance? We don't have capacity for that. Not doing it"
• Often chasing compliance and internal existential risk projects.
• Any security engineer that brings CSP into a company gets fried.

4. I also presented an accurate representation of what happens outside the door of the security team after CSP is implemented in a tight and secure manner (and it blocks a 3rd party resource that makes a new subrequest to something).

https://www.youtube.com/embed/EWfN3YNIqKM

5. CSP management is a disaster. Many 3rd party assets change a lot. And while their core functionality usually stays the same, nothing is stopping blocking dependencies from being moved to a new subrequest.

• No notice is given on changes
• Problem is noticed when it is already too late
• Difficulty to make fast changes

6. Companies struggle to make fast changes to CSP because it is hard coded.

• Normal release cycles 2-5 days
• Emergency changes 'couple of hours'
• What if it's my payment provider's client-side script?

If they use a proxy, changes can* happen a lot faster. But…
=> If a proxy is required for a specification to be actionable, the specification is wrong.
A proxy should NEVER be the standard, it's a patch to imperfection at the spec level.

*IaC is an exception to this rule.

7. How do I even know what I should put in a CSP?

• Report-only => noisy ugly and alarming badly worded alerts
• [Report Only] Refused to load => still loads ?!?!?!?!
• By the time I have my reports, it is already outdated

8. How CSP is implemented
   • Who built a reporting endpoint from scratch recently? It's a disaster. Examples of that everywhere: https://w3c.github.io/webappsec-csp/#issue-d43ce829
   • The spec is interpreted as broadly as the recipe of Bolognese sauce (I love Bolognese - sorry to bring it in here)

9. CSP is a headache.

And the worst part, it still does not protect. Bad actors use googletagmanager.com all the time to bypass CSP, because so many of us use googletagmanager and it allows anyone to inject payloads.
Sure, if you wrote your CSP strict enough this wouldn't be possible but we are blamning the user again for something that is just not userfriendly.

And even if it was perfectly implemented, why can't I see script payloads beyond the first 40 characters? Why can't I get the script hash sent to me so I know what is loading?

10. In comes strict-dynamic...

When I conducted customer feedback on this concept, it took them a while to get it. But when they did...

Feedback:
"Wow, wait? So I am now explicitly allowing any subrequest from a script? So I have the trust the entire supply chain based on the parent? Why use CSP in the first place? I am closing the gate to then open it again?"
On sharing you can still define an allowlist:

"Content-Security-Policy: script-src 'nonce-abcdefg' 'strict-dynamic', script-src https://example.cdn"

"Yea but what is the point then in the first place?"

• Wait so this would only work for dynamic sites? (nonce)
• Literal quote "a very magento take on things"
• Internal pushback: Bad actors inject client-side fetches inside NPM scripts. 1st party scripts will need strict-dynamic. This proposal will result in more negative impact.
• Internal pushback: Bad-actors already use googletagmanager.com to bypass CSP. It is a matter of time before Google recommends Strict-dynamic on GTM and at that point we are much worse off.

Making sense of Strict-dynamic was a lot in the first place - expect this to be misused and abused.

Also, remember the Polyfill.io attack? That was a subrequest. Strict-dyanmic would be cool with that.

11. CSP was not built with

• dynamicness in mind
• CSR in mind
• NPM in mind
• how it would actually go down in an average company (not google, not meta…)
• Script-dynamic - by all means, support it. But this is not adding security.
  • It sure adds convenience in some cases but it is not a real solution to the problem at hand.

12. What people really want and asked for

• Ability to silence report-only console logs
• Fix the shouty wording of a non-blocking report-only console logs
• Instead of having to import a hashing engine into a web application to get the hashes of scripts, the browser does it anyway, allow us to relay the hashes back to an endpoint.
• Allow to set CSP's elsewhere, not hard coded, not in a proxy, not in a meta tag and a janky server side dynamic element.

13. Questions I was asked before

But what about CSP in elements?

• Logical separation between code and headers is a feature, not a bug. Sure, some frameworks nowadays can set CSP. but thats not the case for everything and separation can prevent a single point of failure in an exploit.
• Limitations of features is a thing.
• Does not solve most of the problems I raised before.
• Sure but can't you use and make a customer server-side component?
• Yes, but I could also build Doom in a PDF or build a back-end in CSS.
• Is a specification good if you need to get creative to achieve basic daily usability?
• How many people do this?

14. The objections I expect

• But the added latency of doing a render blocking 3rd party fetch
  • People add literal proxies in front of their website to make CSP work => performance issue too
  • It is not because something could be slow, it should not be an option
  • => it can be done fast too
  • => security features often add latency - we all want SSL right?
  • If we let this argument carry any weight, we might as well all go home and revert everything we did in the last 25 years.
• Your sample size is not big enough.
  • CSP adoption is low. We're a vendor with meaningful customers. Customers that come to us because CSP failed them. We are not the only one. We need to do the creativity for our customers because it is such a burden. Sorry but this is not a helpful take. What are we really trying to achieve here? For me it is simple: make good security practices more accessible to everyone.

After this we held a Q&A.

Some previous proposals were brought forward like the ability for a 3rd party assets on the page to set its own CSP.

https://www.usenix.org/conference/usenixsecurity17/technical-sessions/presentation/calzavara

I flagged that this is an interesting idea but that a surprisingly large amount of attacks happen through expired domains or s3 buckets and that such a mechanism will result in major security incidents and create a false sense of security.
If this was a feature, I am sure googletagmanager would want to use it. But maybe their first attention should go to stopping malicious code from being distributed on their tag manager.

Later on during the call, one of us (I believe our friend from Mozilla) flagged that CSP was never built for what it is being used for today. The original reason why it was scoped was to prevent inline scripts.

There was sense of nodding in the room on that point. So it sounds like we are mostly if not all in agreement that we are making something work for a purpose it was never intended for by substituting it with more directives, features and bypasses.

Especially with PCI DSS now calling it out in the way that it does etc, this is becoming a more pressing issue.
With React adoption being as high as it is and how it works, we're even further from what CSP originally was built to do.

People are forced to adopt something that either breaks sites, creates a false sense of security because the rules are scope too widley or even when implemented fully correctly can be bypassed.

So perhaps, while maintaining CSP, we should accept our losses and start from scratch again.

There are a few clear next steps that came up:

1. Chrome to fix the wording of their CSP report-only error
2. Review the feature requests above:
   1. CSP report-only silenced
   2. Expose script hashes
   3. Allow CSP to be set on a dynamic endpoint
3. Start specing out what the ideal world here would look like, outside of CSP. And on that, I hope to have a great discussion in Japan with y'all in the same room.

My final thought:

Our responsibility is to write specifications that if adopted, allow people and businesses to be safer.
Design the mechanisms used to stop the bad guy. Protecting everyone who uses the internet. While they bank online, watch videos, shop online etc... Your friends, your family, all of us.
If we make mistakes or fail to detect defects in our design or thinking, we pave the path for those who seek to destabalize the free world we live in.
We have a responsibility here, this is not some weekend project.

Did this recap do the job?

[quantumpacket]

And even if it was perfectly implemented, why can't I see script payloads beyond the first 50 characters? Why can't I get the script hash sent to me so I know what is loading?

The spec actually says 40-chars, which is even worse. If someone injects code into my application I want the entire payload send to me in my violation report, so I can see what was done. Perhaps this hard-limit can be removed and the payload sent in the report "sanitized" as a Base64 string?

[swijckmans]

Correcting to 40, thanks for flagging!