Sotatek-HungNguyenThanh opened a new issue, #11471:
URL: https://github.com/apache/pinot/issues/11471
I create a pinot cluster include 1 controller and many server (6 server). I
am setup stream table with mode upsert table size 1B records. I run a time I
see pinot server offten crash and controller lag when view UI. I received some
error below:
2023/08/30 23:12:55.906 ERROR [StatusUpdateUtil]
[HelixTaskExecutor-message_handle_thread_7] Exception while logging status
update
```
org.apache.helix.HelixException: HelixManager is not connected within retry
timeout for cluster PinotCluster
at
org.apache.helix.manager.zk.ZKHelixManager.checkConnected(ZKHelixManager.java:417)
~[pinot-all-0.12.0-jar-with-dependencies.jar:0.12.0-118f5e065cb258c171d97a586183759fbc61e2bf]
at
org.apache.helix.manager.zk.ZKHelixManager.getHelixDataAccessor(ZKHelixManager.java:681)
~[pinot-all-0.12.0-jar-with-dependencies.jar:0.12.0-118f5e065cb258c171d97a586183759fbc61e2bf]
at
org.apache.helix.util.StatusUpdateUtil.logMessageStatusUpdateRecord(StatusUpdateUtil.java:353)
[pinot-all-0.12.0-jar-with-dependencies.jar:0.12.0-118f5e065cb258c171d97a586183759fbc61e2bf]
at
org.apache.helix.util.StatusUpdateUtil.logError(StatusUpdateUtil.java:405)
[pinot-all-0.12.0-jar-with-dependencies.jar:0.12.0-118f5e065cb258c171d97a586183759fbc61e2bf]
at
org.apache.helix.messaging.handling.HelixTask.call(HelixTask.java:119)
[pinot-all-0.12.0-jar-with-dependencies.jar:0.12.0-118f5e065cb258c171d97a586183759fbc61e2bf]
at
org.apache.helix.messaging.handling.HelixTask.call(HelixTask.java:49)
[pinot-all-0.12.0-jar-with-dependencies.jar:0.12.0-118f5e065cb258c171d97a586183759fbc61e2bf]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
[?:?]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
[?:?]
at java.lang.Thread.run(Thread.java:829) [?:?]
2023/08/30 23:12:59.776 ERROR [HelixTask]
[HelixTaskExecutor-message_handle_thread_7] Message execution failed. msgId:
3cc8c5cf-140a-4c93-b0e4-6695d3761868, errorMsg: HelixManager is not connected
within retry timeout for cluster PinotCluster
2023/08/30 23:13:11.376 ERROR [ZKHelixManager]
[HelixTaskExecutor-message_handle_thread_7] zkClient is not connected after
waiting 10000ms., clusterName: PinotCluster, zkAddress: 172.16.198.43:2181
```
```
Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread "Start a Pinot
[SERVER]-SendThread(172.16.198.43:2181)"
Exception in thread "transactions__8__69__20230831T0534Z" Exception in
thread "HttpServer-0" java.lang.OutOfMemoryError: Java heap space
at java.base/java.util.HashMap.values(HashMap.java:955)
at
org.apache.pinot.segment.local.segment.readers.PinotSegmentRecordReader.close(PinotSegmentRecordReader.java:243)
at
org.apache.pinot.segment.local.segment.creator.impl.SegmentIndexCreationDriverImpl.build(SegmentIndexCreationDriverImpl.java:250)
at
org.apache.pinot.segment.local.realtime.converter.RealtimeSegmentConverter.build(RealtimeSegmentConverter.java:110)
at
org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.buildSegmentInternal(LLRealtimeSegmentDataManager.java:895)
at
org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.buildSegmentForCommit(LLRealtimeSegmentDataManager.java:806)
at
org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager$PartitionConsumer.run(LLRealtimeSegmentDataManager.java:705)
at java.base/java.lang.Thread.run(Thread.java:829)
java.lang.OutOfMemoryError: Java heap space
Exception in thread "metrics-meter-tick-thread-1"
java.lang.OutOfMemoryError: Java heap space
2023/08/31 06:04:02.458 ERROR [SegmentBuildTimeLeaseExtender]
[pool-4-thread-1] Failed to send lease extension for
transactions__8__69__20230831T0534Z
Exception in thread "transactions__2__74__20230831T0538Z"
java.lang.OutOfMemoryError: Java heap space
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
