;>>
>>>>
>>>> On 29 May 2016 at 00:26, heri wijayanto wrote:
>>>>
>>>>> I implement spark with join function for processing in around 250
>>>>> million rows of text.
>>>>>
>>>>> When I just used
hen I just used several hundred of rows, it could run, but when I use
>>>> the large data, it is failed.
>>>>
>>>> My spark version in 1.6.1, run above yarn-cluster mode, and we have 5
>>>> node computers.
>>>>
>>>> Thank you very much
r mode, and we have 5
>>> node computers.
>>>
>>> Thank you very much, Ted Yu
>>>
>>> On Sun, May 29, 2016 at 6:48 AM, Ted Yu wrote:
>>>
>>>> Can you let us know your case ?
>>>>
>>>> When the join failed, what was the err
your case ?
>>>
>>> When the join failed, what was the error (consider pastebin) ?
>>>
>>> Which release of Spark are you using ?
>>>
>>> Thanks
>>>
>>> > On May 28, 2016, at 3:27 PM, heri wijayanto
>>> wrote:
>&
park are you using ?
>>
>> Thanks
>>
>> > On May 28, 2016, at 3:27 PM, heri wijayanto wrote:
>> >
>> > Hi everyone,
>> > I perform join function in a loop, and it is failed. I found a tutorial
>> from the web, it says that I should use a bro
te:
> >
> > Hi everyone,
> > I perform join function in a loop, and it is failed. I found a tutorial
> from the web, it says that I should use a broadcast variable but it is not
> a good choice for doing it on the loop.
> > I need your suggestion to address this problem,
Can you let us know your case ?
When the join failed, what was the error (consider pastebin) ?
Which release of Spark are you using ?
Thanks
> On May 28, 2016, at 3:27 PM, heri wijayanto wrote:
>
> Hi everyone,
> I perform join function in a loop, and it is failed. I found a t
Hi everyone,
I perform join function in a loop, and it is failed. I found a tutorial
from the web, it says that I should use a broadcast variable but it is not
a good choice for doing it on the loop.
I need your suggestion to address this problem, thank you very much.
and I am sorry, I am a
