Corporate Proxy Support for Verify

[zac-poe]

I am in the process of updating to the latest version of pact-js and I am encountering issues configuring a web proxy for use with the Verifier.

My approach with pact 9.17 (and pact-node 10.17) was to use global-agent to leverage our https proxy and this worked successfully when my providerBaseUrl for the pact Verifier would otherwise be blocked by our web proxy.

When updating to pact 12.3.0, I noticed several issues with the above approach. Initially I had to add NO_PROXY configuration for 127.0.0.1. Subsequently, I saw runtime issues with the pact parseBody.js behavior as the proxy request was already in a finished/headersSent state when this event hook was attempting to update the request. However, even after manually modifying this behavior I was observing the global agent proxy on https was still not actually being leveraged and my provider requests were not going through the web proxy.

Similary, setting environment variables for HTTPS_PROXY or HTTP_PROXY does not seem to impact the underlying pact internal proxy forwarding from local to the provider URL.

What is the proper way to configure use of a corporate proxy with the latest Pact Verifier? Is this a feature that would need to be requested as an enhancement (e.g. adding an optional arg for an http Agent to the Verifier)?

[mefellows]

Good question. I thought this was supported on the latest and it would be surprising to me that we haven't bumped into this already.

There are potentially 2 places this is impacted:

1. In the proxy code (probably here)
2. In the core verifier

I suspect if you couldn't resolve it in JS, it's in the core verifier. Were you able to find the source of where the requests were stuck?

Looking at the core, it uses the Reqwest library and that has code to detect the system proxies and use them.

[zac-poe]

If I'm understanding the architecture properly, it looks like the call sequence is: native core verifier -> 127.0.0.1 (express app) -> request handler proxies to providerBaseUrl

It looks like the native verifier reaches the express app proxy behavior just fine. Also given that when I define HTTP_PROXY I have to then define NO_PROXY, that seems like good evidence that the Rust code is properly applying those proxy configs.

From within the js code proxy behavior - the link to the proxy.web call you shared - that request does not seem to be affected when I define an HTTPS_PROXY value. That makes sense since the used http-proxy wouldn't natively support those environment configs. This also makes sense since previously we had to use global-agent to get corporate proxy behavior with the verifier's proxy.

I think what this means is there isn't Pact Verifier provided proxy support for providerBaseUrl values and we were just backing into it on our own through the use of a global agent.

What I am observing is that if I revert this change locally: feat: modify request body inside of verifier v3, then the http-proxy behavior with the global agent proceeds as expected. I'll keep digging deeper, but somehow there is a conflict when the http-proxy is also using a https global agent (proxy) with the middleware and event handlers for the body parser behavior.

[zac-poe]

It looks like others encounter this issue with body-parser and proxy middlewares - this is the same manifestation I get if I either use a global agent, or explicitly provide an agent to the proxy:

• app.use(bodyParser.json()) breaks http-proxy-middleware for HTTP POST JSON with express chimurai/http-proxy-middleware#320
• https://stackoverflow.com/questions/28371641/how-can-i-use-express-http-proxy-after-bodyparser-json-has-been-called

It seems like the request buffer gets read to completion by the parser middleware with the expectation that the proxyReq event will re-write just for the proxy request. But it seems like this may break any piping behavior used by http agents in between this sequence.

I think this might need to be an enhancement request to allow the Verifier to accept an option for proxy support.

Here's what I tried that seemed to work:

• added an option for the Verifier to accept an http.Agent
  • (in my code, I pass an HttpsProxyAgent)
• inside the proxy.web call
  • include the provided agent as a config option
  • added a buffer option - I just inlined this for testing: require('stream').Readable.from(Buffer.from(JSON.stringify(req.body)))
• removed the .on('proxyReq' event handler entirely since the write back behavior is no longer needed to forward the proxy request

Could something like the above be added as enhanced support for corporate proxies from the Verifier?

[zac-poe]

Alternatively, the replacement of forwarding the buffer to the proxy request and removal of the writeback event handler mentioned above would be sufficient to support the use of global agents again and users could configure that external of any API changes to the Verifier

[mefellows]

It seems like the request buffer gets read to completion by the parser middleware with the expectation that the proxyReq event will re-write just for the proxy request. But it seems like this may break any piping behavior used by http agents in between this sequence.

This is probably true. I seem to think it was working previously but perhaps that was 9.x.x or prior.

Alternatively, the replacement of forwarding the buffer to the proxy request and removal of the writeback event handler mentioned above would be sufficient to support the use of global agents again and users could configure that external of any API changes to the Verifier

What we need is

1. The ability to intercept and log the request/response details (this helps with debugging)
2. The ability for users to modify the incoming request / outgoing response
3. The ability to use native npm global agents or de-facto environment variables to configure behaviour (e.g. https_proxy etc.)
4. Ideally, backwards compatibility in behaviour (but that's not necessarily required if we can't make it)

It sounds like your first proposal would allow for this, and I'd welcome a PR to address this problem.

[zac-poe]

Thanks @mefellows!

I'll see about getting a PR together for that ticket in the near future

[mefellows]

I've created #1188 to track this