wirybeaver opened a new pull request, #12242: URL: https://github.com/apache/pinot/pull/12242
For easer explanation, let's assume the server holds exactly one table and the table has only one column. The column is a playload column and the larget byte size of the entry is 2MB. The segment has 1000 immutable segments and each segment has at least one entry with 2MB payload column. The thread number of the query worker is 48cores * 2 ~= 100. All immutable segments' java instances would never be deleted once loaded unless the segment deletion happens. It indicated the IndexReader would also not be deleted. Once a segment is queried by a thread from the query worker thread pool (the thread never dies), then the associated byte array would be hold by local thread local map forever. As time goes by, each segment can be queried by all threads coming from query worker thread pool. Finally, the total memory usage of byte array = 1000 segments * 100 thread * 2MB (longest byte size of the entry of a column) * 1 column = 200GB. The OOM happened. Restart won't work since query continues in the real world. We can make the thread local to static because each thread can only touch one reader at a time so that the memory usage will be 100 thread * 2MB = 200MB. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
