Yeah, after destroy, accessing the broadcast variable results in an
error. Accessing it after it's unpersisted (on an executor) causes it
to be rebroadcast.
On Tue, Aug 30, 2016 at 5:12 PM, Jerry Lam wrote:
> Hi Sean,
>
> Thank you for sharing the knowledge between unpersist and destroy.
> Does t
Hi Sean,
Thank you for sharing the knowledge between unpersist and destroy.
Does that mean unpersist keeps the broadcast variable in the driver whereas
destroy will delete everything about the broadcast variable like it has
never existed?
Best Regards,
Jerry
On Tue, Aug 30, 2016 at 11:58 AM, S
Yes, although there's a difference between unpersist and destroy,
you'll hit the same type of question either way. You do indeed have to
reason about when you know the broadcast variable is no longer needed
in the face of lazy evaluation, and that's hard.
Sometimes it's obvious and you can take ad
Hi Sean,
Thank you for the response. The only problem is that actively managing
broadcast variables require to return the broadcast variables to the caller
if the function that creates the broadcast variables does not contain any
action. That is the scope that uses the broadcast variables cannot d
Yes you want to actively unpersist() or destroy() broadcast variables
when they're no longer needed. They can eventually be removed when the
reference on the driver is garbage collected, but you usually would
not want to rely on that.
On Mon, Aug 29, 2016 at 4:30 PM, Jerry Lam wrote:
> Hello spar
Hello spark developers,
Anyone can shed some lights on the life cycle of the broadcast variables?
Basically, if I have a broadcast variable defined in a loop and for each
iteration, I provide a different value.
// For example:
for(i< 1 to 10) {
val bc = sc.broadcast(i)
sc.parallelize(Seq(1
Hello spark developers,
Can someone explain to me what is the lifecycle of a broadcast variable?
When a broadcast variable will be garbage-collected at the driver-side and
at the executor-side? Does a spark application need to actively manage the
broadcast variables to ensure that it will not run
