EPIPE write, ECONNRESET issues when switch to nodejs 20

[roman-kupriyanov]

Hello,

After the migration to nodejs 20.16.0 from 18.20.4 some of our tests that use Mockttp server started to fail with EPIPE write or ECONNRESET read or ECONNRESET socket hang up errors. We have around 200 tests in general and only some of them are affected. A weird part is that if to run failed tests individually they are passing, which makes me think that it may be related to some parallel running issues or race conditions. We use Jest as a test runner.

Any help would be appreciated. Thanks!

[pimterry]

Oooh, that sounds very interesting! I haven't seen this before myself, and Mockttp's own tests seem to run fine in 20.16.0 so I'm not sure exactly what would cause this.

Is there any pattern to which tests fail? Is there anything notable about those tests?

If you try using other Node versions (older Node v20 versions, or Node v22) do those fail in the same way? If not, it would be really interesting to try a few different versions and see if you can work out exactly which version is breaking this for you.

[roman-kupriyanov]

Hi @pimterry,

Indeed a very interesting issue it is 🙂 Regarding your questions:

If you try using other Node versions (older Node v20 versions, or Node v22) do those fail in the same way? If not, it would be really interesting to try a few different versions and see if you can work out exactly which version is breaking this for you.

I tried version 20.0.0 and it is not working right away, so seems that the issue is appearing from very beginning of the node 20 version. I also tried 22.6.0 and it fails the tests in the same way. And again the latest 18.x version fixes everything immediately.

I also use the latest to date mockttp version: 3.15.1.

Is there any pattern to which tests fail? Is there anything notable about those tests?

There are two patterns usually:

• When we expect some specific response from API, but instead getting the socket errors:

    expect(received).toEqual(expected) // deep equality

    - Expected  - 3
    + Received  + 7

      Object {
        "details": Object {
    -     "messages": Array [],
    -     "returnCode": 0,
    +     "connectionCode": "ECONNRESET",
    +     "httpStatusCode": undefined,
    +     "messages": Array [
    +       "read ECONNRESET",
    +     ],
        },
    -   "status": "OK",
    +   "status": "ERROR",
    +   "type": "CONNECTION_ERROR",
      }

      10244 |       )(packageName)(packageOptions);
      10245 |       // assert
    > 10246 |       expect(createPackageResult).toEqual({
            |                                   ^
      10247 |         status: ResponseStatus.OK,
      10248 |         details: {
      10249 |           returnCode: 0,

These tests usually use forAnyRequest().thenJson() mock:

      await mockServer.forAnyRequest().thenJson(
        200,
        {
          data: [],
          messages,
          reports: {},
          returnCode,
        },
        {
          version: '2.5',
          'content-type': 'application/json',
        }
      );

• And the second case is during the check if API was even called:

    expect(received).toBe(expected) // Object.is equality

    Expected: true
    Received: false

      6726 |         const seenRequests = await endevorEndpoint.getSeenRequests();
      6727 |         const calledOnce = seenRequests.length === 1;
    > 6728 |         expect(calledOnce).toBe(true);
           |                            ^
      6729 |
      6730 |         expect(isErrorEndevorResponse(generateResult)).toBe(false);
      6731 |       });

This one happens usually with the forPut() or forGet() mock.

[pimterry]

Ok, that's very useful thanks. I think it's likely that this is a Node bug (or intentional breaking change that's causing issues), but unfortunately I'm also a Node maintainer, so I can't really pass the buck here 😆, and whether the fix is in Node or Mockttp it'd be useful to know exactly what's going on.

Can you share more detail about how exactly are you're sending the requests, and any kinds of client configuration there? One thing that has changed a few times in Node during major version bumps is a slow tightening of which kinds of subtly invalid connections or unusual requests are accepted, so it might be that there's something unusual there triggering this.

If you test with some v19 versions, does this fail the same way there too? If we can pin this down to a specific version where it breaks then that would reduce the set of possibly related changes in Node significantly, which would be very helpful to understand the issue.

Is there any chance you can put together a reproduction of the issue, or share enough code that I can run the tests myself? That would make it very dramatically easier to investigate and fix.

[jaramir]

I think I managed to replicate the same problem on a simpler codebase. Pushed the code here: https://github.com/codurance/mockttp-test

This project has two tests, reusing the same server which will start on the same port. The first one passes and the second one fails.

I think this also depends on the specific version of Jest used but still looking into that. [update: nope, upgraded everything to latest and it still fails]

[pimterry]

Thanks @jaramir, that's definitely an interesting example. It looks like node-fetch (via cross-fetch) is the real culprit here. If you comment out that import entirely and just use Node's fetch built-in instead then the test passes fine with Node v20 or v22. Meanwhile if you use fetch from node-fetch or cross-fetch, the 2nd test fails.

Any idea why that might be? I know Jest has some weird & wonderful hooks into various internals, I wonder if that might interact with fetch in some unexpected ways somewhere. Adding a bunch of debug logging, it looks like Mockttp doesn't even receive a TCP connection at all in the 2nd test. If you try using a simpler test runner (Node's built-in test runner, or mocha or something) does that work?

[vokrik]

I am not sure if it helps, but to me it seems to be happening on every other test. My setup is node-fetch, jest as a runner. I have a completely wild theory without any proof, but I wonder if the node-fetch is not using some persistent tcp connection. I remember that node in certain versions was not sending the closing message and just ended the connection on it's end. If the node-fetch somehow tried to re-use the connection, it would make sense for it to fail. (and it would explain why it would fail every other test)

Don't want to add a red herring here, cause I am really just a user of both of the libs, just wanted to add a theory to test if you run out of ideas

EDIT:
Found this mention in the node-fetch docs:
https://github.com/node-fetch/node-fetch/blob/8b3320d2a7c07bce4afc6b2bf6c3bbddda85b01f/README.md?plain=1#L560

I was using node 20, when I downgrade to 16 (just picked random installed prior v19), all tests pass. So now I am quite confident this might be the problem.

Not sure how to fix it though :D