wso2mi 4.2.0 : Continous Socket connection errors in logs when pod count >1

[Sukeerth148]

[I am encountering an issue within my WSO2 Micro Integrator (WSO2 MI) 4.2.0 pods deployed on OpenShift. Specifically, this problem arises when there are multiple pods present. The error occurs when I configure the route.openshift.io/termination property in the Ingress resource to reencrypt. However, I've observed that this issue does not manifest when the property is set to edge.]

Version

Client Version: 4.12.0-202304070941.p0.g31aa3e8.assembly.stream-31aa3e8
Kustomize Version: v4.5.7
Server Version: 4.14.0
Kubernetes Version: v1.27.6+f67aeb3

Steps To Reproduce

1. Deploy an application pod with ingress.
2. Keep the ingress termination as reencrypt.

Current Result

[2024-`03-07 09:23:19,362] ERROR {SourceHandler} - I/O error: Connection reset java.net.SocketException: Connection reset
at java.base/sun.nio.ch.SocketChannelImpl.throwConnectionReset(SocketChannelImpl.java:394)
at java.base/sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:426)
at org.apache.http.nio.reactor.ssl.SSLIOSession.receiveEncryptedData(SSLIOSession.java:482)
at org.apache.http.nio.reactor.ssl.SSLIOSession.isAppInputReady(SSLIOSession.java:544)
at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:120)
at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
at java.base/java.lang.Thread.run(Thread.java:840)

Expected Result

No error with in the logs.

Additional Information

Debug logs:

[2024-03-06 18:12:20,444] DEBUG {SourceConnections} - Shutting down connection forcefully http-incoming-6
[2024-03-06 18:12:20,444] DEBUG {LoggingNHttpServerConnection} - http-incoming-6: Shutdown connection
[2024-03-06 18:12:20,444] DEBUG {SSLIOSession} - I/O session http-incoming-6-4 10.255.6.92:8253<->10.255.14.1:59748[ACTIVE][r:r][ACTIVE][r][NOT_HANDSHAKING][0][0][0]: Shutdown
[2024-03-06 18:12:20,444] DEBUG {SourceHandler} - http-incoming-6: Keep-Alive connection was closed: Remote Address : /10.255.14.1:59748
[2024-03-06 18:12:20,444] DEBUG {SourceConnections} - Shutting down connection forcefully http-incoming-6
[2024-03-06 18:12:20,444] DEBUG {LoggingNHttpServerConnection} - http-incoming-6: Shutdown connection
[2024-03-06 18:12:20,444] DEBUG {SSLIOSession} - I/O session http-incoming-6-4 10.255.6.92:8253<->10.255.14.1:59748[CLOSED][][CLOSED][r][NOT_HANDSHAKING][0][0][0]: Shutdown
[2024-03-06 18:12:25,442] DEBUG {SSLIOSession} - I/O session http-incoming-7-5 10.255.6.92:8253<->10.255.18.1:47144[ACTIVE][r:][ACTIVE][r][NOT_HANDSHAKING][0][0][0]: Set timeout 180000
[2024-03-06 18:12:25,442] DEBUG {SSLIOSession} - I/O session http-incoming-7-5 10.255.6.92:8253<->10.255.18.1:47144[ACTIVE][r:][ACTIVE][r][NOT_HANDSHAKING][0][0][0]: Set attribute CONNECTION_INFORMATION
[2024-03-06 18:12:25,446] DEBUG {SSLIOSession} - I/O session http-incoming-8-6 10.255.6.92:8253<->10.255.14.1:59758[ACTIVE][r:][ACTIVE][r][NOT_HANDSHAKING][0][0][0]: Set timeout 180000
[2024-03-06 18:12:25,446] DEBUG {SSLIOSession} - I/O session http-incoming-8-6 10.255.6.92:8253<->10.255.14.1:59758[ACTIVE][r:][ACTIVE][r][NOT_HANDSHAKING][0][0][0]: Set attribute CONNECTION_INFORMATION
[2024-03-06 18:12:25,448] ERROR {SourceHandler} - I/O error: Connection reset java.net.SocketException: Connection reset
at java.base/sun.nio.ch.SocketChannelImpl.throwConnectionReset(SocketChannelImpl.java:394)
at java.base/sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:426)
at org.apache.http.nio.reactor.ssl.SSLIOSession.receiveEncryptedData(SSLIOSession.java:482)
at org.apache.http.nio.reactor.ssl.SSLIOSession.isAppInputReady(SSLIOSession.java:544)
at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:120)
at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
at java.base/java.lang.Thread.run(Thread.java:840)