I agree, it was by mistake.
I just updated so that the next person looking for torrent broadcast issues
will have a hint :)
Thank you.
Daniel
On Sun, Jun 19, 2016 at 5:26 PM, Ted Yu wrote:
> I think good practice is not to hold on to SparkContext in mapFunction.
>
> On Sun, Jun 19, 2016 at 7:10
I think good practice is not to hold on to SparkContext in mapFunction.
On Sun, Jun 19, 2016 at 7:10 AM, Takeshi Yamamuro
wrote:
> How about using `transient` annotations?
>
> // maropu
>
> On Sun, Jun 19, 2016 at 10:51 PM, Daniel Haviv <
> daniel.ha...@veracity-group.com> wrote:
>
>> Hi,
>> Jus
How about using `transient` annotations?
// maropu
On Sun, Jun 19, 2016 at 10:51 PM, Daniel Haviv <
daniel.ha...@veracity-group.com> wrote:
> Hi,
> Just updating on my findings for future reference.
> The problem was that after refactoring my code I ended up with a scala
> object which held Spar
Hi,
Just updating on my findings for future reference.
The problem was that after refactoring my code I ended up with a scala
object which held SparkContext as a member, eg:
object A {
sc: SparkContext = new SparkContext
def mapFunction {}
}
and when I called rdd.map(A.mapFunction) it
Hi,
Since `HttpBroadcastFactory` has already been removed in master, so
you cannot use the broadcast mechanism in future releases.
Anyway, I couldn't find a root cause only from the stacktraces...
// maropu
On Mon, Jun 6, 2016 at 2:14 AM, Daniel Haviv <
daniel.ha...@veracity-group.com> wrote
Hi,
I've set spark.broadcast.factory to
org.apache.spark.broadcast.HttpBroadcastFactory and it indeed resolve my
issue.
I'm creating a dataframe which creates a broadcast variable internally and
then fails due to the torrent broadcast with the following stacktrace:
Caused by: org.apache.spark.Spa
I found spark.broadcast.blockSize but no parameter to switch broadcast
method.
Can you describe the issues with torrent broadcast in more detail ?
Which version of Spark are you using ?
Thanks
On Wed, Jun 1, 2016 at 7:48 AM, Daniel Haviv <
daniel.ha...@veracity-group.com> wrote:
> Hi,
> Our ap
