Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[CI] Netty4HttpServerTransportTests testChannelAcceptorCannotTamperThreadContext failing #108193

Closed
slobodanadamovic opened this issue May 2, 2024 · 8 comments · Fixed by #108362 or #108374
Closed

[CI] Netty4HttpServerTransportTests testChannelAcceptorCannotTamperThreadContext failing #108193

slobodanadamovic opened this issue May 2, 2024 · 8 comments · Fixed by #108362 or #108374
Assignees
@slobodanadamovic
Labels
low-risk An open issue or test failure that is a low risk to future releases :Security/Security Security issues without another label Team:Security Meta label for security team >test-failure Triaged test failures from CI

Comments

@slobodanadamovic
Copy link
Contributor

Build scan:
https://gradle-enterprise.elastic.co/s/yqf7kpf3qc4jg/tests/:modules:transport-netty4:test/org.elasticsearch.http.netty4.Netty4HttpServerTransportTests/testChannelAcceptorCannotTamperThreadContext

Reproduction line:

gradlew ':modules:transport-netty4:test' --tests "org.elasticsearch.http.netty4.Netty4HttpServerTransportTests.testChannelAcceptorCannotTamperThreadContext" -Dtests.seed=F5D355E8719865F5 -Dtests.locale=hu -Dtests.timezone=Asia/Atyrau -Druntime.java=21

Applicable branches:
main

Reproduces locally?:
No

Failure history:
Failure dashboard for org.elasticsearch.http.netty4.Netty4HttpServerTransportTests#testChannelAcceptorCannotTamperThreadContext

Failure excerpt:

java.lang.AssertionError: java.io.IOException: An established connection was aborted by the software in your host machine

  at __randomizedtesting.SeedInfo.seed([F5D355E8719865F5]:0)
  at org.elasticsearch.http.netty4.Netty4HttpClient$CountDownLatchHandler$1.exceptionCaught(Netty4HttpClient.java:210)
  at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346)
  at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:325)
  at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:317)
  at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireExceptionCaught(CombinedChannelDuplexHandler.java:424)
  at io.netty.channel.ChannelHandlerAdapter.exceptionCaught(ChannelHandlerAdapter.java:92)
  at io.netty.channel.CombinedChannelDuplexHandler$1.fireExceptionCaught(CombinedChannelDuplexHandler.java:145)
  at io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:143)
  at io.netty.channel.CombinedChannelDuplexHandler.exceptionCaught(CombinedChannelDuplexHandler.java:231)
  at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346)
  at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:325)
  at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:317)
  at io.netty.channel.DefaultChannelPipeline$HeadContext.exceptionCaught(DefaultChannelPipeline.java:1377)
  at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346)
  at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:325)
  at io.netty.channel.DefaultChannelPipeline.fireExceptionCaught(DefaultChannelPipeline.java:907)
  at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.handleReadException(AbstractNioByteChannel.java:125)
  at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:177)
  at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
  at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689)
  at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652)
  at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
  at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
  at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
  at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
  at java.lang.Thread.run(Thread.java:1583)

  Caused by: java.io.IOException: An established connection was aborted by the software in your host machine

    at sun.nio.ch.SocketDispatcher.read0(SocketDispatcher.java:-2)
    at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:46)
    at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:340)
    at sun.nio.ch.IOUtil.read(IOUtil.java:294)
    at sun.nio.ch.IOUtil.read(IOUtil.java:269)
    at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:425)
    at org.elasticsearch.transport.netty4.CopyBytesSocketChannel.readFromSocketChannel(CopyBytesSocketChannel.java:131)
    at org.elasticsearch.transport.netty4.CopyBytesSocketChannel.doReadBytes(CopyBytesSocketChannel.java:116)
    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:151)
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.lang.Thread.run(Thread.java:1583)
@slobodanadamovic slobodanadamovic added :Security/Security Security issues without another label >test-failure Triaged test failures from CI Team:Security Meta label for security team labels May 2, 2024
@elasticsearchmachine elasticsearchmachine added the needs:risk Requires assignment of a risk label (low, medium, blocker) label May 2, 2024
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/es-security (Team:Security)

@slobodanadamovic
Copy link
Contributor Author

slobodanadamovic commented May 2, 2024 •
edited

Seems to fail consistently on Windows hosts. Will try to look closer.

@slobodanadamovic slobodanadamovic self-assigned this May 3, 2024
@ywangd
Copy link
Member

ywangd commented May 7, 2024

This fails quite often on Windows

slobodanadamovic added a commit to slobodanadamovic/elasticsearch that referenced this issue May 7, 2024
@slobodanadamovic
[Test] Ignore closed connections on Windows hosts
0b76e01 
Added handling of the following `IOException` in `Netty4HttpClient`:

```
  Caused by: java.io.IOException: An established connection was aborted by the software in your host machine
```

Resolves: elastic#108193
@slobodanadamovic slobodanadamovic mentioned this issue May 7, 2024
Merged
@slobodanadamovic slobodanadamovic added low-risk An open issue or test failure that is a low risk to future releases and removed needs:risk Requires assignment of a risk label (low, medium, blocker) labels May 7, 2024
@albertzaharovits albertzaharovits mentioned this issue May 7, 2024
Merged
@luigidellaquila
Copy link
Contributor

It failed again https://gradle-enterprise.elastic.co/s/2ufy4uu7btph6/tests/task/:modules:transport-netty4:pooledTest/details/org.elasticsearch.http.netty4.Netty4HttpServerTransportTests/testChannelAcceptorCannotTamperThreadContext?top-execution=1

@albertzaharovits
Copy link
Contributor

@Tim-Brooks do you have any tips on this one please?

@Tim-Brooks
Copy link
Contributor

I imagine it is because the way a rejected channel is closed on windows breaks the assertion for caught exceptions in the client.

                @Override
                public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause) {
                    if (cause instanceof PrematureChannelClosureException || cause instanceof SocketException) {
                        // no more requests coming, so fast-forward the latch
                        fastForward();
                    } else {
                        ExceptionsHelper.maybeDieOnAnotherThread(new AssertionError(cause));
                    }
                }

Windows appears to be returning this as java.io.IOException. Which seems fine. Just an issue with the test expectations.

@slobodanadamovic
Copy link
Contributor Author

slobodanadamovic commented May 13, 2024 •
edited

I've re-opened #108362, which handles this java.io.IOException in Netty4HttpClient.

@valeriy42
Copy link
Contributor

Another failure : https://gradle-enterprise.elastic.co/s/gdq6e3zg6ggea

slobodanadamovic added a commit that referenced this issue May 13, 2024
@slobodanadamovic
[Test] Ignore closed connections on Windows hosts (#108362)
93ec9d6 
This commit adds special handling for the  `java.io.IOException: An established connection was aborted by the software in your host machine` in `Netty4HttpClient#exceptionCaught ` method. This exception only occurs when running tests on Windows hosts

Resolves: #108193
slobodanadamovic added a commit to slobodanadamovic/elasticsearch that referenced this issue May 13, 2024
@slobodanadamovic
[Test] Ignore closed connections on Windows hosts (elastic#108362)
dfb43f3 
This commit adds special handling for the  `java.io.IOException: An established connection was aborted by the software in your host machine` in `Netty4HttpClient#exceptionCaught ` method. This exception only occurs when running tests on Windows hosts

Resolves: elastic#108193
(cherry picked from commit 93ec9d6)
elasticsearchmachine pushed a commit that referenced this issue May 13, 2024
@slobodanadamovic
[Test] Ignore closed connections on Windows hosts (#108362) (#108586)
3d1ea25 
This commit adds special handling for the  `java.io.IOException: An established connection was aborted by the software in your host machine` in `Netty4HttpClient#exceptionCaught ` method. This exception only occurs when running tests on Windows hosts

Resolves: #108193
(cherry picked from commit 93ec9d6)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@slobodanadamovic slobodanadamovic

Labels
low-risk An open issue or test failure that is a low risk to future releases :Security/Security Security issues without another label Team:Security Meta label for security team >test-failure Triaged test failures from CI
Projects
None yet
Milestone
No milestone
7 participants
@luigidellaquila @Tim-Brooks @valeriy42 @ywangd @slobodanadamovic @albertzaharovits @elasticsearchmachine