Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Hazelcast Session Tests unable to start on CI / K8s #8877

Closed
joakime opened this issue Nov 9, 2022 · 4 comments
Closed

Hazelcast Session Tests unable to start on CI / K8s #8877

joakime opened this issue Nov 9, 2022 · 4 comments
Assignees
@olamy
Labels
Build Test

Comments

@joakime
Copy link
Contributor

joakime commented Nov 9, 2022 •
edited
Loading

Jetty version(s)
Jetty 10 + Jetty 11 + Jetty 12

Description
The Hazelcast Tests are unable to start with the new k8s configuration on CI.

The tests report that the nodes are unable to start ...

Running org.eclipse.jetty.hazelcast.session.ClusteredOrphanedSessionTest.testOrphanedSession()
[WARNING] Can not connect to Ryuk at 10.0.1.1:49168
java.net.SocketTimeoutException: Connect timed out
    at sun.nio.ch.NioSocketImpl.timedFinishConnect (NioSocketImpl.java:546)
    at sun.nio.ch.NioSocketImpl.connect (NioSocketImpl.java:597)
    at java.net.SocksSocketImpl.connect (SocksSocketImpl.java:327)
    at java.net.Socket.connect (Socket.java:633)
    at org.testcontainers.utility.RyukResourceReaper.lambda$null$0 (RyukResourceReaper.java:92)
    at org.rnorth.ducttape.ratelimits.RateLimiter.doWhenReady (RateLimiter.java:27)
    at org.testcontainers.utility.RyukResourceReaper.lambda$maybeStart$1 (RyukResourceReaper.java:88)
    at java.lang.Thread.run (Thread.java:833)
2022-11-09 12:21:06.053:WARN :com.hazelcast.cp.CPSubsystem:main: [10.1.145.125]:5701 [dev] [4.2.5] CP Subsystem is not enabled. CP data structures will operate in UNSAFE mode! Please note that UNSAFE mode will not provide strong consistency guarantees.
2022-11-09 12:21:06.857:WARN :com.hazelcast.kubernetes.KubernetesClient:main: Cannot fetch public IPs of Hazelcast Member PODs, you won't be able to use Hazelcast Smart Client from outside of the Kubernetes network

and

2022-11-09 12:24:57.398:WARN :com.hazelcast.internal.server.tcp.TcpServerConnection:hz.SESSION_TEST_20227606451311.IO.thread-in-1: [10.1.145.125]:5701 [dev] [4.2.5] Connection[id=18220, /10.1.145.125:35261->/10.1.232.54:22, qualifier=null, endpoint=[10.1.232.54]:22, alive=false, connectionType=NONE, planeIndex=-1] closed. Reason: Exception in Connection[id=18220, /10.1.145.125:35261->/10.1.232.54:22, qualifier=null, endpoint=[10.1.232.54]:22, alive=true, connectionType=NONE, planeIndex=-1], thread=hz.SESSION_TEST_20227606451311.IO.thread-in-1
java.lang.IllegalStateException: Unknown protocol: SSH
	at com.hazelcast.internal.server.tcp.UnifiedProtocolDecoder.onRead(UnifiedProtocolDecoder.java:117)
	at com.hazelcast.internal.networking.nio.NioInboundPipeline.process(NioInboundPipeline.java:138)
	at com.hazelcast.internal.networking.nio.NioThread.processSelectionKey(NioThread.java:383)
	at com.hazelcast.internal.networking.nio.NioThread.processSelectionKeys(NioThread.java:368)
	at com.hazelcast.internal.networking.nio.NioThread.selectLoop(NioThread.java:294)
	at com.hazelcast.internal.networking.nio.NioThread.executeRun(NioThread.java:249)
	at com.hazelcast.internal.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:102)

and

[WARNING] Can not connect to Ryuk at 10.0.1.1:49168
java.net.NoRouteToHostException: No route to host
    at sun.nio.ch.Net.pollConnect (Native Method)
    at sun.nio.ch.Net.pollConnectNow (Net.java:672)
    at sun.nio.ch.NioSocketImpl.timedFinishConnect (NioSocketImpl.java:549)
    at sun.nio.ch.NioSocketImpl.connect (NioSocketImpl.java:597)
    at java.net.SocksSocketImpl.connect (SocksSocketImpl.java:327)
    at java.net.Socket.connect (Socket.java:633)
    at org.testcontainers.utility.RyukResourceReaper.lambda$null$0 (RyukResourceReaper.java:92)
    at org.rnorth.ducttape.ratelimits.RateLimiter.doWhenReady (RateLimiter.java:27)
    at org.testcontainers.utility.RyukResourceReaper.lambda$maybeStart$1 (RyukResourceReaper.java:88)
    at java.lang.Thread.run (Thread.java:833)
[WARNING] Can not connect to Ryuk at 10.0.1.1:49168
java.net.NoRouteToHostException: No route to host
    at sun.nio.ch.Net.pollConnect (Native Method)
    at sun.nio.ch.Net.pollConnectNow (Net.java:672)
    at sun.nio.ch.NioSocketImpl.timedFinishConnect (NioSocketImpl.java:549)
    at sun.nio.ch.NioSocketImpl.connect (NioSocketImpl.java:597)
    at java.net.SocksSocketImpl.connect (SocksSocketImpl.java:327)
    at java.net.Socket.connect (Socket.java:633)
    at org.testcontainers.utility.RyukResourceReaper.lambda$null$0 (RyukResourceReaper.java:92)
    at org.rnorth.ducttape.ratelimits.RateLimiter.doWhenReady (RateLimiter.java:27)
    at org.testcontainers.utility.RyukResourceReaper.lambda$maybeStart$1 (RyukResourceReaper.java:88)
    at java.lang.Thread.run (Thread.java:833)
[WARNING] Can not connect to Ryuk at 10.0.1.1:49168
java.net.NoRouteToHostException: No route to host
    at sun.nio.ch.Net.pollConnect (Native Method)
    at sun.nio.ch.Net.pollConnectNow (Net.java:672)
    at sun.nio.ch.NioSocketImpl.timedFinishConnect (NioSocketImpl.java:549)
    at sun.nio.ch.NioSocketImpl.connect (NioSocketImpl.java:597)
    at java.net.SocksSocketImpl.connect (SocksSocketImpl.java:327)
    at java.net.Socket.connect (Socket.java:633)
    at org.testcontainers.utility.RyukResourceReaper.lambda$null$0 (RyukResourceReaper.java:92)
    at org.rnorth.ducttape.ratelimits.RateLimiter.doWhenReady (RateLimiter.java:27)
    at org.testcontainers.utility.RyukResourceReaper.lambda$maybeStart$1 (RyukResourceReaper.java:88)
    at java.lang.Thread.run (Thread.java:833)
2022-11-09 12:26:12.109:ERROR:com.hazelcast.instance.impl.Node:main: [10.1.145.125]:5701 [dev] [4.2.5] Could not join cluster. Shutting down now!
2022-11-09 12:26:12.114:WARN :com.hazelcast.instance.impl.Node:main: [10.1.145.125]:5701 [dev] [4.2.5] Terminating forcefully...
2022-11-09 12:26:12.161:WARN :com.hazelcast.internal.util.phonehome.PhoneHome:main: [10.1.145.125]:5701 [dev] [4.2.5] Could not schedule phone home task! Most probably Hazelcast failed to start.

and

[ERROR] org.eclipse.jetty.hazelcast.session.ClusteredOrphanedSessionTest.testOrphanedSession  Time elapsed: 306.631 s  <<< ERROR!
java.lang.ExceptionInInitializerError
	at org.eclipse.jetty.hazelcast.session.ClusteredOrphanedSessionTest.setUp(ClusteredOrphanedSessionTest.java:31)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:727)
	at org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
	at org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:156)
	at org.junit.jupiter.engine.extension.TimeoutExtension.interceptLifecycleMethod(TimeoutExtension.java:128)
	at org.junit.jupiter.engine.extension.TimeoutExtension.interceptBeforeEachMethod(TimeoutExtension.java:78)
	at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(InterceptingExecutableInvoker.java:103)
	at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.lambda$invoke$0(InterceptingExecutableInvoker.java:93)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
	at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:92)
	at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:86)
	at org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.invokeMethodInExtensionContext(ClassBasedTestDescriptor.java:520)
	at org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.lambda$synthesizeBeforeEachMethodAdapter$23(ClassBasedTestDescriptor.java:505)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeBeforeEachMethods$3(TestMethodTestDescriptor.java:174)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeBeforeMethodsOrCallbacksUntilExceptionOccurs$6(TestMethodTestDescriptor.java:202)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeBeforeMethodsOrCallbacksUntilExceptionOccurs(TestMethodTestDescriptor.java:202)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeBeforeEachMethods(TestMethodTestDescriptor.java:171)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:134)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:68)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:151)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
	at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
	at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
	at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:35)
	at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor.execute(HierarchicalTestExecutor.java:57)
	at org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute(HierarchicalTestEngine.java:54)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:147)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:127)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:90)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.lambda$execute$0(EngineExecutionOrchestrator.java:55)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.withInterceptedStreams(EngineExecutionOrchestrator.java:102)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:54)
	at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:114)
	at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:86)
	at org.junit.platform.launcher.core.DefaultLauncherSession$DelegatingLauncher.execute(DefaultLauncherSession.java:86)
	at org.apache.maven.surefire.junitplatform.LazyLauncher.execute(LazyLauncher.java:55)
	at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.execute(JUnitPlatformProvider.java:223)
	at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invokeAllTests(JUnitPlatformProvider.java:175)
	at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invoke(JUnitPlatformProvider.java:139)
	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:456)
	at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:169)
	at org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:595)
	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:581)
	Suppressed: java.lang.NullPointerException: Cannot invoke "org.eclipse.jetty.hazelcast.session.HazelcastTestHelper.tearDown()" because "this._testHelper" is null
		at org.eclipse.jetty.hazelcast.session.ClusteredOrphanedSessionTest.shutdown(ClusteredOrphanedSessionTest.java:36)
		at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
		at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
		at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
		at java.base/java.lang.reflect.Method.invoke(Method.java:568)
		at org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:727)
		at org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
		at org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
		at org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:156)
		at org.junit.jupiter.engine.extension.TimeoutExtension.interceptLifecycleMethod(TimeoutExtension.java:128)
		at org.junit.jupiter.engine.extension.TimeoutExtension.interceptAfterEachMethod(TimeoutExtension.java:110)
		at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(InterceptingExecutableInvoker.java:103)
		at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.lambda$invoke$0(InterceptingExecutableInvoker.java:93)
		at org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
		at org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
		at org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
		at org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
		at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:92)
		at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:86)
		at org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.invokeMethodInExtensionContext(ClassBasedTestDescriptor.java:520)
		at org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.lambda$synthesizeAfterEachMethodAdapter$24(ClassBasedTestDescriptor.java:510)
		at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeAfterEachMethods$10(TestMethodTestDescriptor.java:243)
		at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeAllAfterMethodsOrCallbacks$13(TestMethodTestDescriptor.java:276)
		at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
		at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeAllAfterMethodsOrCallbacks$14(TestMethodTestDescriptor.java:276)
		at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
		at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeAllAfterMethodsOrCallbacks(TestMethodTestDescriptor.java:275)
		at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeAfterEachMethods(TestMethodTestDescriptor.java:241)
		at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:142)
		... 49 more
Caused by: java.lang.IllegalStateException: Node failed to start!
	at com.hazelcast.instance.impl.HazelcastInstanceImpl.<init>(HazelcastInstanceImpl.java:125)
	at com.hazelcast.instance.impl.HazelcastInstanceFactory.constructHazelcastInstance(HazelcastInstanceFactory.java:211)
	at com.hazelcast.instance.impl.HazelcastInstanceFactory.getOrCreateHazelcastInstance(HazelcastInstanceFactory.java:108)
	at com.hazelcast.core.Hazelcast.getOrCreateHazelcastInstance(Hazelcast.java:165)
	at org.eclipse.jetty.hazelcast.session.HazelcastTestHelper.<clinit>(HazelcastTestHelper.java:56)
	... 76 more
joakime added a commit that referenced this issue Nov 9, 2022
@joakime
Issue #8877 - disabling hazelcast testing until k8s issue can be solv…
a34e45a 
…ed (in a branch)
joakime added a commit that referenced this issue Nov 9, 2022
@joakime
Issue #8877 - disabling hazelcast testing until k8s issue can be solv…
27abda6 
…ed (in a branch)
@joakime
Copy link
Contributor Author

joakime commented Nov 9, 2022

@olamy this could be related to hazelcast/hazelcast-kubernetes#207

joakime added a commit that referenced this issue Nov 9, 2022
@joakime
Issue #8877 - enabling hazelcast testing in branch
430b765
joakime added a commit that referenced this issue Nov 9, 2022
@joakime
Issue #8877 - downgrading testcontainers to 1.17.4 to see if more stable
047789a
@joakime joakime mentioned this issue Nov 9, 2022
Closed
@joakime
Copy link
Contributor Author

joakime commented Nov 9, 2022

@olamy this is pretty important to fix.

I've started a draft PR at #8878 to attempt to address this

joakime added a commit that referenced this issue Nov 9, 2022
@joakime
Issue #8877 - bumping hazelcast from 4.2.5 to 5.2.0
952bcf3 
+ Not using hazelcast-all uber-jar
joakime added a commit that referenced this issue Nov 9, 2022
@joakime
Revert "Issue #8877 - downgrading testcontainers to 1.17.4 to see if …
352f70d 
…more stable"

This reverts commit 047789a.
@joakime
Copy link
Contributor Author

joakime commented Nov 9, 2022

This started failing in all branches about 6 hours ago.

@joakime
Copy link
Contributor Author

joakime commented Nov 14, 2022

Recent k9s changes have made this stable again. Dropping my branch.

@joakime joakime closed this as completed Nov 14, 2022
@olamy olamy mentioned this issue Dec 26, 2022
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@olamy olamy

Labels
Build Test
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@olamy @joakime