Thanks Gopal! I'l look at options provided. On Wed, Feb 10, 2016 at 7:46 PM, Gautam <gautamkows...@gmail.com> wrote:
> Here's the json version. > > On Wed, Feb 10, 2016 at 7:44 PM, Gautam <gautamkows...@gmail.com> wrote: > >> Whoops.. meant to send the tez explain earlier. Here's the Tez query >> plan. Good to know there's a fix .. Is there a jira that talks about this >> issue? Coz I couldn't find one. Maybe I can alter the query a bit to >> filter these out. >> >> Cheers, >> -Gautam. >> >> >> On Wed, Feb 10, 2016 at 7:32 PM, Gopal Vijayaraghavan <gop...@apache.org> >> wrote: >> >>> Hey, >>> >>> > Trying to benchmark with Hive on Tez causes the following error. >>> >Admittedly these are some very large looking records .. the same job >>> runs >>> >fine on MR2. >>> ... >>> > I'v attached the query explain tree. It fails in the very last reducer >>> >phase .. >>> >>> Can you attach the explain plan with hive.execution.engine=tez (even >>> better would be the JSON output of "explain formatted"). >>> >>> > Error: Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: >>> >>> >org.apache.tez.runtime.library.common.sort.impl.ExternalSorter$MapBufferTo >>> >oSmallException: Record too large for in-memory buffer. Exceeded buffer >>> >overflow limit, bufferOverflowRecursion=2, bufferList.size=1, >>> >blockSize=268435456 >>> >>> You have a single row which is > 200Mb? >>> >>> The tez 0.8.2 out there should already have this issue fixed, but single >>> spill record mode is likely to spin your disks all wrong (unless you have >>> SSDs). >>> >>> I'm guessing there's a UDTF + PTF producing humongous rows & that it's >>> being planned wrong because that's not a standard pattern. >>> >>> Cheers, >>> Gopal >>> >>> >>> >> >> >> -- >> "If you really want something in this life, you have to work for it. Now, >> quiet! They're about to announce the lottery numbers..." >> > > > > -- > "If you really want something in this life, you have to work for it. Now, > quiet! They're about to announce the lottery numbers..." > -- "If you really want something in this life, you have to work for it. Now, quiet! They're about to announce the lottery numbers..."
