PovilasV1 opened a new issue, #16426:
URL: https://github.com/apache/pinot/issues/16426
## Summary
`BufferUnderflowException` occurs during DataTable deserialization in
broker-server communication when using Apache Pinot `1.4.0-SNAPSHOT`. The issue
is resolved by using stable version `1.3.0`.
## Environment
- **Pinot Version**: `1.4.0-SNAPSHOT` (jar:
`pinot-all-1.4.0-SNAPSHOT-jar-with-dependencies.jar`)
- **Deployment**: Kubernetes StatefulSet
- **Image**: `apachepinot/pinot:latest-21-ms-openjdk`
- **Tables**: Both OFFLINE and REALTIME tables affected
## Stack Trace
```
[pinot-broker-3 broker] 2025/07/25 10:37:59.229 ERROR [DataTableHandler]
[nioEventLoopGroup-2-1] Caught exception while deserializing data table of
size: 1687 from server: pinot-server-3_O
[pinot-broker-3 broker] java.nio.BufferUnderflowException
[pinot-broker-3 broker] at
java.base/java.nio.ByteBuffer.get(ByteBuffer.java:826)
[pinot-broker-3 broker] at
java.base/java.nio.ByteBuffer.get(ByteBuffer.java:855)
[pinot-broker-3 broker] at
org.apache.pinot.common.datatable.DataTableUtils.decodeString(DataTableUtils.java:88)
[pinot-broker-3 broker] at
org.apache.pinot.common.datatable.DataTableImplV4.deserializeMetadata(DataTableImplV4.java:595)
[pinot-broker-3 broker] at
org.apache.pinot.common.datatable.DataTableImplV4.<init>(DataTableImplV4.java:195)
[pinot-broker-3 broker] at
org.apache.pinot.common.datatable.DataTableFactory.getDataTable(DataTableFactory.java:36)
[pinot-broker-3 broker] at
org.apache.pinot.core.transport.DataTableHandler.channelRead0(DataTableHandler.java:71)
[pinot-broker-3 broker] at
org.apache.pinot.core.transport.DataTableHandler.channelRead0(DataTableHandler.java:38)
[pinot-broker-3 broker] at
io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
[pinot-broker-3 broker] at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
[pinot-broker-3 broker] at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
[pinot-broker-3 broker] at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
[pinot-broker-3 broker] at
io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
[pinot-broker-3 broker] at
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
[pinot-broker-3 broker] at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
[pinot-broker-3 broker] at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
[pinot-broker-3 broker] at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
[pinot-broker-3 broker] at
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1357)
[pinot-broker-3 broker] at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
[pinot-broker-3 broker] at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
[pinot-broker-3 broker] at
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:868)
[pinot-broker-3 broker] at
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
[pinot-broker-3 broker] at
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:796)
[pinot-broker-3 broker] at
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:732)
[pinot-broker-3 broker] at
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:658)
[pinot-broker-3 broker] at
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
[pinot-broker-3 broker] at
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:998)
[pinot-broker-3 broker] at
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
[pinot-broker-3 broker] at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
[pinot-broker-3 broker] at
java.base/java.lang.Thread.run(Thread.java:1583)
[pinot-broker-3 broker] 2025/07/25 10:38:09.224 ERROR [PinotClientRequest]
[jersey-server-managed-async-executor-3] Query processing exceptions: {427=1
servers [pinot-server-3_O] not responded}
```
## Detailed Description
### Symptoms
- `BufferUnderflowException` occurs during DataTable deserialization on
broker side
- Only affects **one specific server** (`pinot-server-3`) consistently
- Error persists across pod restarts and fresh data downloads from deep
storage (GCS)
- Both OFFLINE and REALTIME tables affected
### Investigation Results
#### Server-Specific Behavior
- 6 servers total, only `pinot-server-3` consistently affected
- Issue appears related to **new segments** being assigned to the
problematic server
#### Resource Analysis
- Server resources adequate: 29Gi memory limit, 18Gi usage, 4 CPU limit,
367m usage
- JVM settings: `-Xms6G -Xmx16G -XX:+UseG1GC -XX:MaxGCPauseMillis=200`
- No memory pressure, GC issues, or resource constraints identified
#### Version Analysis
- All components running identical `1.4.0-SNAPSHOT` version
- Issue resolved immediately after upgrading to stable `1.3.0`
- Confirms version-specific bug in SNAPSHOT build
## Root Cause
The issue appears to be a **bug in `1.4.0-SNAPSHOT`** affecting DataTable
serialization/deserialization, specifically:
1. **New segments** contain data/metadata patterns that trigger the bug
2. **DataTable metadata serialization** creates malformed byte streams
3. **Broker-side deserialization** attempts to read more bytes than available
4. **String decoding** in metadata fails with `BufferUnderflowException`
## Reproduction Steps
1. Deploy Pinot `1.4.0-SNAPSHOT` in production environment
2. Ingest data into both OFFLINE and REALTIME tables
3. Wait for new segments to be created and assigned to servers
4. Execute queries that route to servers with newly created segments
5. Observe `BufferUnderflowException` in broker logs
## Resolution
Upgrading from `1.4.0-SNAPSHOT` to `1.3.0` completely resolved the issue
with no data loss or additional configuration changes required.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
