It's was done 2014 by yours truly https://github.com/apache/spark/pull/1498
so any modern version would have it.
On Mon, Sep 23, 2019 at 9:04 PM, Dhrubajyoti Hati < dhruba.w...@gmail.com >
wrote:
>
> Thanks. Could you please let me know which version of spark its changed.
> We are still at 2.2
Thanks. Could you please let me know which version of spark its changed. We
are still at 2.2.
On Tue, 24 Sep, 2019, 9:17 AM Reynold Xin, wrote:
> A while ago we changed it so the task gets broadcasted too, so I think the
> two are fairly similar.
>
>
>
> On Mon, Sep 23, 2019 at 8:17 PM, Dhrubajy
A while ago we changed it so the task gets broadcasted too, so I think the two
are fairly similar.
On Mon, Sep 23, 2019 at 8:17 PM, Dhrubajyoti Hati < dhruba.w...@gmail.com >
wrote:
>
> I was wondering if anyone could help with this question.
>
> On Fri, 20 Sep, 2019, 11:52 AM Dhrubajyoti Hat
I was wondering if anyone could help with this question.
On Fri, 20 Sep, 2019, 11:52 AM Dhrubajyoti Hati,
wrote:
> Hi,
>
> I have a question regarding passing a dictionary from driver to executors
> in spark on yarn. This dictionary is needed in an udf. I am using pyspark.
>
> As I understand th
Hi,
I have a question regarding passing a dictionary from driver to executors
in spark on yarn. This dictionary is needed in an udf. I am using pyspark.
As I understand this can be passed in two ways:
1. Broadcast the variable and then use it in the udfs
2. Pass the dictionary in the udf itself
