Hey, > Trying to benchmark with Hive on Tez causes the following error. >Admittedly these are some very large looking records .. the same job runs >fine on MR2. ... > I'v attached the query explain tree. It fails in the very last reducer >phase ..
Can you attach the explain plan with hive.execution.engine=tez (even better would be the JSON output of "explain formatted"). > Error: Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: >org.apache.tez.runtime.library.common.sort.impl.ExternalSorter$MapBufferTo >oSmallException: Record too large for in-memory buffer. Exceeded buffer >overflow limit, bufferOverflowRecursion=2, bufferList.size=1, >blockSize=268435456 You have a single row which is > 200Mb? The tez 0.8.2 out there should already have this issue fixed, but single spill record mode is likely to spin your disks all wrong (unless you have SSDs). I'm guessing there's a UDTF + PTF producing humongous rows & that it's being planned wrong because that's not a standard pattern. Cheers, Gopal
