shwin opened a new issue, #11520: URL: https://github.com/apache/pinot/issues/11520
Full table config here: https://gist.github.com/shwin/7817b6d96a9facf6687f749bbe1a1507 When building a segment on a minion, native indices are not built; that is, there are no `.nativetext.idx` present in the segments pushed to the segment output directory. This is unlike lucene indices, which are present: ``` columns.psf creation.meta index_map toProtocolAttributes.lucene.index transactionToProtocolAttributes.lucene.index contractAddressProtocolAttributes.lucene.index fromProtocolAttributes.lucene.index metadata.properties transactionFromProtocolAttributes.lucene.index ``` I suspect this is not a problem with our table config, since the servers do attempt to build the native text indices: ``` Creating new text index for column: chain in segment: 6ace1858-ec7e-4bc8-acea-abbeb48f16d2-test-one-segment_OFFLINE_1641487292000_1641509420000_0_a42ac055-2275-425d-b623-492f08bfa591, hasDictionary: true 83 Created text index for column: chain in segment: 6ace1858-ec7e-4bc8-acea-abbeb48f16d2-test-one-segment_OFFLINE_1641487292000_1641509420000_0_a42ac055-2275-425d-b623-492f08bfa591 79 ``` This is bad, since they eventually fail with stack traces like the following: ``` Exception while executing a state transition task 6ace1858-ec7e-4bc8-acea-abbeb48f16d2_OFFLINE_1641487292000_1641509420000_1481_15c79469-d635-45c1-a0e1-805701aab041 java.lang.reflect.InvocationTargetException: null at jdk.internal.reflect.GeneratedMethodAccessor73.invoke(Unknown Source) ~[?:?] at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?] at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?] at org.apache.helix.messaging.handling.HelixStateTransitionHandler.invoke(HelixStateTransitionHandler.java:350) ~[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-c68b3be360fafac1c38f3322ed5fa64201d08daf] at org.apache.helix.messaging.handling.HelixStateTransitionHandler.handleMessage(HelixStateTransitionHandler.java:278) [pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-c68b3be360fafac1c38f3322ed5fa64201d08daf] at org.apache.helix.messaging.handling.HelixTask.call(HelixTask.java:97) [pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-c68b3be360fafac1c38f3322ed5fa64201d08daf] at org.apache.helix.messaging.handling.HelixTask.call(HelixTask.java:49) [pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-c68b3be360fafac1c38f3322ed5fa64201d08daf] at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:829) [?:?] Caused by: java.lang.OutOfMemoryError: Direct buffer memory at java.nio.Bits.reserveMemory(Bits.java:175) ~[?:?] at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:118) ~[?:?] at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:317) ~[?:?] at org.apache.pinot.segment.spi.memory.PinotByteBuffer.allocateDirect(PinotByteBuffer.java:38) ~[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-c68b3be360fafac1c38f3322ed5fa64201d08daf] at org.apache.pinot.segment.spi.memory.ByteBufferPinotBufferFactory.allocateDirect(ByteBufferPinotBufferFactory.java:32) ~[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-c68b3be360fafac1c38f3322ed5fa64201d08daf] at org.apache.pinot.segment.spi.memory.SmallWithFallbackPinotBufferFactory.allocateDirect(SmallWithFallbackPinotBufferFactory.java:47) ~[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-c68b3be360fafac1c38f3322ed5fa64201d08daf] at org.apache.pinot.segment.spi.memory.PinotDataBuffer.allocateDirect(PinotDataBuffer.java:216) ~[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-c68b3be360fafac1c38f3322ed5fa64201d08daf] at org.apache.pinot.segment.local.io.writer.impl.DirectMemoryManager.allocateInternal(DirectMemoryManager.java:53) ~[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-c68b3be360fafac1c38f3322ed5fa64201d08daf] at org.apache.pinot.segment.local.io.readerwriter.RealtimeIndexOffHeapMemoryManager.allocate(RealtimeIndexOffHeapMemoryManager.java:85) ~[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-c68b3be360fafac1c38f3322ed5fa64201d08daf] at org.apache.pinot.segment.local.realtime.impl.dictionary.OffHeapMutableBytesStore.add(OffHeapMutableBytesStore.java:119) ~[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-c68b3be360fafac1c38f3322ed5fa64201d08daf] at org.apache.pinot.segment.local.utils.nativefst.ImmutableFST.readRemaining(ImmutableFST.java:207) ~[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-c68b3be360fafac1c38f3322ed5fa64201d08daf] at org.apache.pinot.segment.local.utils.nativefst.ImmutableFST.<init>(ImmutableFST.java:200) ~[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-c68b3be360fafac1c38f3322ed5fa64201d08daf] at org.apache.pinot.segment.local.utils.nativefst.FST.read(FST.java:91) ~[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-c68b3be360fafac1c38f3322ed5fa64201d08daf] at org.apache.pinot.segment.local.utils.nativefst.FST.read(FST.java:114) ~[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-c68b3be360fafac1c38f3322ed5fa64201d08daf] at org.apache.pinot.segment.local.segment.index.readers.text.NativeTextIndexReader.populateIndexes(NativeTextIndexReader.java:92) ~[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-c68b3be360fafac1c38f3322ed5fa64201d08daf] at org.apache.pinot.segment.local.segment.index.readers.text.NativeTextIndexReader.<init>(NativeTextIndexReader.java:60) ~[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-c68b3be360fafac1c38f3322ed5fa64201d08daf] at org.apache.pinot.segment.local.segment.index.text.TextIndexType$ReaderFactory.createIndexReader(TextIndexType.java:173) ~[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-c68b3be360fafac1c38f3322ed5fa64201d08daf] at org.apache.pinot.segment.local.segment.index.text.TextIndexType$ReaderFactory.createIndexReader(TextIndexType.java:153) ~[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-c68b3be360fafac1c38f3322ed5fa64201d08daf] at org.apache.pinot.segment.local.segment.index.column.PhysicalColumnIndexContainer.<init>(PhysicalColumnIndexContainer.java:59) ~[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-c68b3be360fafac1c38f3322ed5fa64201d08daf] at org.apache.pinot.segment.local.indexsegment.immutable.ImmutableSegmentLoader.load(ImmutableSegmentLoader.java:184) ~[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-c68b3be360fafac1c38f3322ed5fa64201d08daf] at org.apache.pinot.core.data.manager.BaseTableDataManager.tryLoadExistingSegment(BaseTableDataManager.java:815) ~[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-c68b3be360fafac1c38f3322ed5fa64201d08daf] at org.apache.pinot.core.data.manager.BaseTableDataManager.addOrReplaceSegment(BaseTableDataManager.java:463) ~[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-c68b3be360fafac1c38f3322ed5fa64201d08daf] at org.apache.pinot.server.starter.helix.HelixInstanceDataManager.addOrReplaceSegment(HelixInstanceDataManager.java:504) ~[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-c68b3be360fafac1c38f3322ed5fa64201d08daf] at org.apache.pinot.server.starter.helix.SegmentOnlineOfflineStateModelFactory$SegmentOnlineOfflineStateModel.onBecomeOnlineFromOffline(SegmentOnlineOfflineStateModelFactory.java:166) ~[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-c68b3be360fafac1c38f3322ed5fa64201d08daf] ... 11 more Default rollback method invoked on error. Error Code: ERROR ``` If this happened on minions, that would be one thing; this happening on our servers is a lot more impactful. See https://apache-pinot.slack.com/archives/C011C9JHN7R/p1693783364237779 for more discussion, but I think this is a decent summary. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
