Arthas诊断抛ConcurrentModificationException异常

[xiaoyunwen]

• 我已经在 issues 里搜索，没有重复的issue。

环境信息

• Arthas 版本: 3.5.6
• 操作系统版本: CentOS 7.9
• 目标进程的JVM版本: 1.8

重现问题的步骤

阿里云 arms 应用监控系统，"应用诊断" --> "Arthas诊断"，问题偶现

实际运行的结果

2024-11-19|ERROR|DubboServerHandler-192.168.123.123:20984-thread-419|f4dc4121911eab70|org.apache.dubbo.rpc.filter.ExceptionFilter
.onResponse():79 -  [DUBBO] Got unchecked and undeclared exception which called by 172.16.123.123. service: com.abc.api.service.XXXService, 
method: queryInfo, exception: java.util.ConcurrentModificationException: null, dubbo version: 2.7.22, current host: 192.168.123.123
java.util.ConcurrentModificationException: null
        at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:911)
        at java.util.ArrayList$Itr.next(ArrayList.java:861)
        at com.taobao.arthas.core.advisor.SpyImpl.atEnter(SpyImpl.java:37)
        at java.arthas.SpyAPI.atEnter(SpyAPI.java:59)
        at com.abc.service.remote.XXXServiceImpl$$EnhancerBySpringCGLIB$$6b519d46.queryInfo(<generated>)
        at org.apache.dubbo.common.bytecode.Wrapper28.invokeMethod(Wrapper28.java)
        at org.apache.dubbo.rpc.proxy.javassist.JavassistProxyFactory$1.doInvoke(JavassistProxyFactory.java:47)
        at org.apache.dubbo.rpc.proxy.AbstractProxyInvoker.invoke(AbstractProxyInvoker.java:84)
        at org.apache.dubbo.config.invoker.DelegateProviderMetaDataInvoker.invoke(DelegateProviderMetaDataInvoker.java:56)
        at org.apache.dubbo.rpc.protocol.InvokerWrapper.invoke(InvokerWrapper.java:56)
        at org.apache.dubbo.rpc.protocol.dubbo.filter.TraceFilter.invoke(TraceFilter.java:77)
        at org.apache.dubbo.rpc.protocol.FilterNode.invoke(FilterNode.java:61)
        at org.apache.dubbo.rpc.filter.TimeoutFilter.invoke(TimeoutFilter.java:44)
        at org.apache.dubbo.rpc.protocol.FilterNode.invoke(FilterNode.java:61)
        at com.abc.framework.ndubbo.filter.TaskProviderFilter.invoke(TaskProviderFilter.java:25)
        at org.apache.dubbo.rpc.protocol.FilterNode.invoke(FilterNode.java:61)
        at com.alibaba.csp.sentinel.adapter.dubbo.SentinelDubboProviderFilter.invoke(SentinelDubboProviderFilter.java:80)
        at org.apache.dubbo.rpc.protocol.FilterNode.invoke(FilterNode.java:61)
        at org.apache.dubbo.monitor.support.MonitorFilter.invoke(MonitorFilter.java:91)
        at org.apache.dubbo.rpc.protocol.FilterNode.invoke(FilterNode.java:61)
        at org.apache.dubbo.rpc.filter.ExceptionFilter.invoke(ExceptionFilter.java:52)
        at org.apache.dubbo.rpc.protocol.FilterNode.invoke(FilterNode.java:61)
        at com.abc.framework.ndubbo.filter.DubboFilter.doProvider(DubboFilter.java:113)
        at com.abc.framework.ndubbo.filter.DubboFilter.invoke(DubboFilter.java:44)
        at org.apache.dubbo.rpc.protocol.FilterNode.invoke(FilterNode.java:61)
        at com.abc.framework.dubbo.filter.DubboExceptionFilter.invoke(DubboExceptionFilter.java:60)
        at org.apache.dubbo.rpc.protocol.FilterNode.invoke(FilterNode.java:61)
        at com.abc.framework.dubbo.filter.PermitsFilter.invoke(PermitsFilter.java:52)
        at org.apache.dubbo.rpc.protocol.FilterNode.invoke(FilterNode.java:61)
        at com.alibaba.dubbo.rpc.Invoker$CompatibleInvoker.invoke(Invoker.java:55)
        at net.dubboclub.catmonitor.CatTransaction.invoke(CatTransaction.java:37)
        at com.alibaba.dubbo.rpc.Filter.invoke(Filter.java:29)
        at org.apache.dubbo.rpc.protocol.FilterNode.invoke(FilterNode.java:61)
        at org.apache.dubbo.rpc.filter.GenericFilter.invoke(GenericFilter.java:192)
        at org.apache.dubbo.rpc.protocol.FilterNode.invoke(FilterNode.java:61)
        at org.apache.dubbo.rpc.filter.ClassLoaderFilter.invoke(ClassLoaderFilter.java:38)
        at org.apache.dubbo.rpc.protocol.FilterNode.invoke(FilterNode.java:61)
        at org.apache.dubbo.rpc.filter.EchoFilter.invoke(EchoFilter.java:41)
        at org.apache.dubbo.rpc.protocol.FilterNode.invoke(FilterNode.java:61)
        at org.apache.dubbo.rpc.filter.ContextFilter.invoke(ContextFilter.java:129)
        at org.apache.dubbo.rpc.protocol.FilterNode.invoke(FilterNode.java:61)
        at org.apache.dubbo.rpc.protocol.dubbo.DubboProtocol$1.reply(DubboProtocol.java:148)
        at org.apache.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.handleRequest(HeaderExchangeHandler.java:100)
        at org.apache.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.received(HeaderExchangeHandler.java:175)
        at org.apache.dubbo.remoting.transport.DecodeHandler.received(DecodeHandler.java:51)
        at org.apache.dubbo.remoting.transport.dispatcher.ChannelEventRunnable.run(ChannelEventRunnable.java:57)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at org.apache.dubbo.common.threadlocal.InternalRunnable.run(InternalRunnable.java:41)
        at java.lang.Thread.run(Thread.java:748)

[sheikmca]

Use ConcurrentHashMap Instead of HashMap
If you are modifying a map, use ConcurrentHashMap:
Map<String, String> map = new ConcurrentHashMap<>();
This avoids ConcurrentModificationException in multi-threaded environments.

The Dubbo service might be accessing a non-thread-safe collection in a multi-threaded environment.
If multiple requests call queryInfo() simultaneously, and it modifies a shared ArrayList, a race condition can occur.
Solution: Use a Thread-Safe Collection
Use CopyOnWriteArrayList for thread-safe list modifications:
List list = new CopyOnWriteArrayList<>();
Use ConcurrentHashMap for maps instead of HashMap
Map<String, String> map = new ConcurrentHashMap<>();

[hengyunabc]

#2952 4.0.5 版本已修复。