OOM can occurs in any place, if most of memory is used by some of the `defect`, the exception stack probably doesn’t show the real problem.
Most of the this occurs in ThriftServer as I know, people are trying to collect a huge result set, can you confirm that? If it fall into this category, probably you can set the “spark.sql.thriftServer.incrementalCollect” to false; Hao From: Yana Kadiyska [mailto:yana.kadiy...@gmail.com] Sent: Friday, November 13, 2015 8:30 AM To: user@spark.apache.org Subject: HiveServer2 Thrift OOM Hi folks, I'm starting a HiveServer2 from a HiveContext (HiveThriftServer2.startWithContext(hiveContext)) and then connecting to it via beenline. On the server side, I see the below error which I think is related to https://issues.apache.org/jira/browse/HIVE-6468 But I'd like to know: 1. why I see it (I'm using bin/beeline from spark to connect, not http) 2. should I be dropping any hive-site or hive-default files in conf/ -- Hive-6468 talks about hive.server2.sasl.message.limit but I can't see any documentation on where this setting would go or what's a reasonable value (im trying to do a light-weight deployment and have not needed hive-site.xml so far...) Advice on how to get rid of the below exception much appreciated Exception in thread "pool-17-thread-2" java.lang.OutOfMemoryError: Java heap space at org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:181) at org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:125) at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:253) at org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41) at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:189) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) E
