hosen-lhs opened a new issue, #15111: URL: https://github.com/apache/dubbo/issues/15111
### Pre-check - [x] I am sure that all the content I provide is in English. ### Search before asking - [x] I had searched in the [issues](https://github.com/apache/dubbo/issues?q=is%3Aissue) and found no similar issues. ### Apache Dubbo Component Java SDK (apache/dubbo) ### Dubbo Version Env： Dubbo + Nacos Dubbo： 3.3.2 Nacos： 2.4.0 我们采用采用Nacos + dubbo的服务框架，由于我们原先采用的是REST服务进行内部调用， 升级Nacos + Dubbo以后， 为了保证能够全局使用Dubbo服务治理， 我们采用uri到类名的映射关系进行泛化调用。 在生产环境， 偶发性出现Invoke remote method timeout。 我们目前采用k8s进行部署， 出现问题的时候， 不会全部有问题， 会集中在某几个pod里面。 有下面几种场景都遇到过： 1. 多个consumer应用的pod 调用同一个provider大批量的接口出现Invoke remote method timeout。 2. 单个consumer调用所有provider应用， 大批量接口有问题。 3. 单个consumer调用单个provider应用有问题， 其他没问题。 具体堆栈： org.apache.dubbo.rpc.RpcException: Invoke remote method timeout. method: $invoke, provider: DefaultServiceInstance{serviceName='da', host='172.22.2.83', port=50051, enabled=true, healthy=true, metadata={dubbo.metadata-service.url-params={"serialization":"hessian2","prefer.serialization":"hessian2,fastjson2","version":"2.0.0","dubbo":"2.0.2","release":"3.3.2","side":"provider","port":"50051","protocol":"tri"}, sentinel-transport-port=8719, dubbo.endpoints=[{"port":50051,"protocol":"tri"}], dubbo.metadata.revision=a7d4f6cb5616b15d3a88f548a88f181b, dubbo.metadata.storage-type=local, meta-v=2.0.0, timestamp=1738810956783}}, service{name='com.jooan.qacloud.da.controller.api.InnerDaService',group='',version='',protocol='tri',port='50051',params={check.serializable=false, side=provider, release=3.3.2, methods=alilvRegister,batchMarkDevice,clearDeviceMark,deviceCheck,deviceMark,deviceRegister,getBatchSizeByBatchNo,getDeviceInfoByDnPk,getDeviceInfoBySn,logout,queryDeviceInfo,queryDeviceInf oList,queryDeviceProductInfo,queryMqttOnline,query_pc_url,query_product_info,set_device_properties,set_device_region,statisticalUserBehavior,updateDeviceCsGiveStatus, logger=slf4j, deprecated=false, dubbo=2.0.2, interface=com.jooan.qacloud.da.controller.api.InnerDaService, service-name-mapping=true, generic=false, revision=3.0.0, serialize.check.status=WARN, serialization=hessian2, application=da, prefer.serialization=hessian2,fastjson2, dynamic=true, dispatcher=message, REGISTRY_CLUSTER=default:6a3d2225-a2e4-4fb0-bacc-e218dd3fe7ad},}, cause: Timeout after 30000ms waiting for result. at org.apache.dubbo.rpc.protocol.AbstractInvoker.waitForResultIfSync(AbstractInvoker.java:331) at org.apache.dubbo.rpc.protocol.AbstractInvoker.invoke(AbstractInvoker.java:195) at org.apache.dubbo.rpc.listener.ListenerInvokerWrapper.invoke(ListenerInvokerWrapper.java:71) at org.apache.dubbo.rpc.filter.GenericImplFilter.invoke(GenericImplFilter.java:139) at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349) at org.apache.dubbo.validation.filter.ValidationFilter.invoke(ValidationFilter.java:102) at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349) at org.apache.dubbo.rpc.filter.RpcExceptionFilter.invoke(RpcExceptionFilter.java:40) at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349) at io.opentelemetry.javaagent.shaded.instrumentation.apachedubbo.v2_7.TracingFilter.invoke(TracingFilter.java:56) at io.opentelemetry.javaagent.instrumentation.apachedubbo.v2_7.OpenTelemetryFilter.invoke(OpenTelemetryFilter.java:40) at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349) at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CallbackRegistrationInvoker.invoke(FilterChainBuilder.java:197) at org.apache.dubbo.rpc.protocol.ReferenceCountInvokerWrapper.invoke(ReferenceCountInvokerWrapper.java:106) at org.apache.dubbo.registry.client.ServiceDiscoveryRegistryDirectory$InstanceWrappedInvoker.invoke(ServiceDiscoveryRegistryDirectory.java:800) at org.apache.dubbo.rpc.cluster.support.AbstractClusterInvoker.invokeWithContext(AbstractClusterInvoker.java:412) at org.apache.dubbo.rpc.cluster.support.FailoverClusterInvoker.doInvoke(FailoverClusterInvoker.java:82) at org.apache.dubbo.rpc.cluster.support.AbstractClusterInvoker.invoke(AbstractClusterInvoker.java:366) at com.alibaba.csp.sentinel.adapter.dubbo3.SentinelDubboConsumerFilter.syncInvoke(SentinelDubboConsumerFilter.java:83) at com.alibaba.csp.sentinel.adapter.dubbo3.SentinelDubboConsumerFilter.invoke(SentinelDubboConsumerFilter.java:67) at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349) at org.apache.dubbo.rpc.cluster.router.RouterSnapshotFilter.invoke(RouterSnapshotFilter.java:46) at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349) at org.apache.dubbo.monitor.support.MonitorFilter.invoke(MonitorFilter.java:109) at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349) at org.apache.dubbo.rpc.cluster.filter.support.MetricsClusterFilter.invoke(MetricsClusterFilter.java:57) at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349) at org.apache.dubbo.rpc.protocol.dubbo.filter.FutureFilter.invoke(FutureFilter.java:53) at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349) at com.alibaba.csp.sentinel.adapter.dubbo3.DubboAppContextFilter.invoke(DubboAppContextFilter.java:47) at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349) at org.apache.dubbo.spring.security.filter.ContextHolderParametersSelectedTransferFilter.invoke(ContextHolderParametersSelectedTransferFilter.java:40) at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349) at org.apache.dubbo.metrics.filter.MetricsFilter.invoke(MetricsFilter.java:86) at org.apache.dubbo.rpc.cluster.filter.support.MetricsConsumerFilter.invoke(MetricsConsumerFilter.java:38) at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349) at org.apache.dubbo.rpc.cluster.filter.support.ConsumerClassLoaderFilter.invoke(ConsumerClassLoaderFilter.java:40) at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349) at org.apache.dubbo.tracing.filter.ObservationSenderFilter.invoke(ObservationSenderFilter.java:60) at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349) at org.apache.dubbo.rpc.cluster.filter.support.ConsumerContextFilter.invoke(ConsumerContextFilter.java:119) at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349) at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CallbackRegistrationInvoker.invoke(FilterChainBuilder.java:197) at org.apache.dubbo.rpc.cluster.support.wrapper.AbstractCluster$ClusterFilterInvoker.invoke(AbstractCluster.java:101) at org.apache.dubbo.rpc.cluster.support.wrapper.MockClusterInvoker.invoke(MockClusterInvoker.java:107) at org.apache.dubbo.rpc.cluster.support.wrapper.ScopeClusterInvoker.invoke(ScopeClusterInvoker.java:171) at org.apache.dubbo.registry.client.migration.MigrationInvoker.invoke(MigrationInvoker.java:294) at org.apache.dubbo.rpc.proxy.InvocationUtil.invoke(InvocationUtil.java:64) at org.apache.dubbo.rpc.proxy.InvokerInvocationHandler.invoke(InvokerInvocationHandler.java:81) at com.alibaba.dubbo.rpc.service.GenericServiceDubboProxy0.$invoke(GenericServiceDubboProxy0.java) at com.jooan.qacloud.commons.utils.CseHttpUtil.postByDubbo(CseHttpUtil.java:255) at com.jooan.qacloud.commons.utils.CseHttpUtil.postForEntity(CseHttpUtil.java:225) at com.jooan.qacloud.commons.utils.CseHttpUtil.lambda$postForEntity$2(CseHttpUtil.java:164) at com.jooan.qacloud.commons.utils.CseHttpUtil.exceptionToNullWrapper(CseHttpUtil.java:177) at com.jooan.qacloud.commons.utils.CseHttpUtil.postForEntity(CseHttpUtil.java:164) at com.jooan.qacloud.iot.IotClient.request(IotClient.java:41) at com.jooan.qacloud.iot.IotClient.getDeviceInfo(IotClient.java:117) at com.jooan.qacloud.vas.service.VasProvisionService.queryDeviceInfo(VasProvisionService.java:53) at com.jooan.qacloud.vas.action.VasProvisionAction.vasProvision(VasProvisionAction.java:162) at com.jooan.qacloud.vas.controller.PayController.vasProvision(PayController.java:75) at com.jooan.qacloud.vas.controller.PayControllerDubboWrap5.invokeMethod(PayControllerDubboWrap5.java) at org.apache.dubbo.rpc.proxy.javassist.JavassistProxyFactory$1.doInvoke(JavassistProxyFactory.java:89) at org.apache.dubbo.rpc.proxy.AbstractProxyInvoker.invoke(AbstractProxyInvoker.java:100) at org.apache.dubbo.config.invoker.DelegateProviderMetaDataInvoker.invoke(DelegateProviderMetaDataInvoker.java:55) at org.apache.dubbo.rpc.protocol.InvokerWrapper.invoke(InvokerWrapper.java:56) at org.apache.dubbo.rpc.filter.ClassLoaderCallbackFilter.invoke(ClassLoaderCallbackFilter.java:38) at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349) at org.apache.dubbo.validation.filter.ValidationFilter.invoke(ValidationFilter.java:102) at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349) at org.apache.dubbo.rpc.protocol.tri.rest.filter.RestFilterAdapter.invoke(RestFilterAdapter.java:38) at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349) at org.apache.dubbo.rpc.protocol.tri.h12.HttpContextCallbackFilter.invoke(HttpContextCallbackFilter.java:37) at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349) at org.apache.dubbo.rpc.protocol.dubbo.filter.TraceFilter.invoke(TraceFilter.java:80) at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349) at org.apache.dubbo.rpc.filter.TimeoutFilter.invoke(TimeoutFilter.java:45) at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349) at com.alibaba.csp.sentinel.adapter.dubbo3.SentinelDubboProviderFilter.invoke(SentinelDubboProviderFilter.java:82) at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349) at io.opentelemetry.javaagent.shaded.instrumentation.apachedubbo.v2_7.TracingFilter.invoke(TracingFilter.java:56) at io.opentelemetry.javaagent.instrumentation.apachedubbo.v2_7.OpenTelemetryFilter.invoke(OpenTelemetryFilter.java:40) at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349) at org.apache.dubbo.monitor.support.MonitorFilter.invoke(MonitorFilter.java:109) at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349) at org.apache.dubbo.rpc.filter.ExceptionFilter.invoke(ExceptionFilter.java:55) at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349) at org.apache.dubbo.rpc.filter.AccessLogFilter.invoke(AccessLogFilter.java:120) at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349) at org.apache.dubbo.rpc.filter.GenericFilter.invoke(GenericFilter.java:222) at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349) at com.jooan.qacloud.dubbo.rpc.generic.rest.filter.RestGenericFilter.invoke(RestGenericFilter.java:187) at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349) at org.apache.dubbo.rpc.protocol.tri.h12.HttpContextFilter.invoke(HttpContextFilter.java:50) at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349) at org.apache.dubbo.rpc.filter.ClassLoaderFilter.invoke(ClassLoaderFilter.java:54) at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349) at org.apache.dubbo.rpc.filter.EchoFilter.invoke(EchoFilter.java:41) at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349) at org.apache.dubbo.metrics.filter.MetricsFilter.invoke(MetricsFilter.java:86) at org.apache.dubbo.metrics.filter.MetricsProviderFilter.invoke(MetricsProviderFilter.java:37) at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349) at org.apache.dubbo.tracing.filter.ObservationReceiverFilter.invoke(ObservationReceiverFilter.java:59) at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349) at org.apache.dubbo.rpc.filter.ProfilerServerFilter.invoke(ProfilerServerFilter.java:66) at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349) at org.apache.dubbo.rpc.filter.ContextFilter.invoke(ContextFilter.java:191) at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349) at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CallbackRegistrationInvoker.invoke(FilterChainBuilder.java:197) at org.apache.dubbo.rpc.protocol.tri.h12.AbstractServerCallListener.invoke(AbstractServerCallListener.java:69) at org.apache.dubbo.rpc.protocol.tri.h12.UnaryServerCallListener.onComplete(UnaryServerCallListener.java:50) at org.apache.dubbo.rpc.protocol.tri.h12.http2.GenericHttp2ServerTransportListener$Http2StreamingDecodeListener.onClose(GenericHttp2ServerTransportListener.java:206) at org.apache.dubbo.remoting.http12.message.DefaultListeningDecoder.close(DefaultListeningDecoder.java:47) at org.apache.dubbo.remoting.http12.message.StreamingDecoder$DefaultFragmentListener.onClose(StreamingDecoder.java:60) at org.apache.dubbo.remoting.http12.message.LengthFieldStreamingDecoder.deliver(LengthFieldStreamingDecoder.java:135) at org.apache.dubbo.remoting.http12.message.LengthFieldStreamingDecoder.close(LengthFieldStreamingDecoder.java:81) at org.apache.dubbo.rpc.protocol.tri.h12.http2.GenericHttp2ServerTransportListener.onDataCompletion(GenericHttp2ServerTransportListener.java:148) at org.apache.dubbo.rpc.protocol.tri.h12.http2.GenericHttp2ServerTransportListener.onDataCompletion(GenericHttp2ServerTransportListener.java:50) at org.apache.dubbo.rpc.protocol.tri.h12.AbstractServerTransportListener.doOnData(AbstractServerTransportListener.java:188) at org.apache.dubbo.rpc.protocol.tri.h12.AbstractServerTransportListener.lambda$onData$1(AbstractServerTransportListener.java:172) at org.apache.dubbo.common.threadpool.serial.SerializingExecutor.run(SerializingExecutor.java:105) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) at org.apache.dubbo.common.threadlocal.InternalRunnable.run(InternalRunnable.java:39) at java.base/java.lang.Thread.run(Thread.java:1583) Caused by: java.util.concurrent.TimeoutException: Timeout after 30000ms waiting for result. at org.apache.dubbo.rpc.AsyncRpcResult.get(AsyncRpcResult.java:223) at org.apache.dubbo.rpc.protocol.AbstractInvoker.waitForResultIfSync(AbstractInvoker.java:293) ... 123 more ### Steps to reproduce this issue 概率性出现， 出现以后就一直有问题。 大概率无法自动恢复。 ### What you expected to happen 给出问题的解决方案。 ### Anything else _No response_ ### Are you willing to submit a pull request to fix on your own? - [ ] Yes I am willing to submit a pull request on my own! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
