Here's the json version. On Wed, Feb 10, 2016 at 7:44 PM, Gautam <gautamkows...@gmail.com> wrote:
> Whoops.. meant to send the tez explain earlier. Here's the Tez query plan. > Good to know there's a fix .. Is there a jira that talks about this issue? Coz > I couldn't find one. Maybe I can alter the query a bit to filter these > out. > > Cheers, > -Gautam. > > > On Wed, Feb 10, 2016 at 7:32 PM, Gopal Vijayaraghavan <gop...@apache.org> > wrote: > >> Hey, >> >> > Trying to benchmark with Hive on Tez causes the following error. >> >Admittedly these are some very large looking records .. the same job runs >> >fine on MR2. >> ... >> > I'v attached the query explain tree. It fails in the very last reducer >> >phase .. >> >> Can you attach the explain plan with hive.execution.engine=tez (even >> better would be the JSON output of "explain formatted"). >> >> > Error: Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: >> >> >org.apache.tez.runtime.library.common.sort.impl.ExternalSorter$MapBufferTo >> >oSmallException: Record too large for in-memory buffer. Exceeded buffer >> >overflow limit, bufferOverflowRecursion=2, bufferList.size=1, >> >blockSize=268435456 >> >> You have a single row which is > 200Mb? >> >> The tez 0.8.2 out there should already have this issue fixed, but single >> spill record mode is likely to spin your disks all wrong (unless you have >> SSDs). >> >> I'm guessing there's a UDTF + PTF producing humongous rows & that it's >> being planned wrong because that's not a standard pattern. >> >> Cheers, >> Gopal >> >> >> > > > -- > "If you really want something in this life, you have to work for it. Now, > quiet! They're about to announce the lottery numbers..." > -- "If you really want something in this life, you have to work for it. Now, quiet! They're about to announce the lottery numbers..."
explain_tez.json
Description: application/json
