have to say sorry. I check the code again, Broadcast is serializable and
should be able to use within lambdas/inner classes. actually according to
the javadoc it should be used in this way to avoid the large contained
value object's serialization.
so what's wrong with the first approach?
On Sat,
ok, this one is doing what I want
SparkConf conf = new SparkConf()
.set("spark.sql.warehouse.dir",
"hdfs://localhost:9000/user/hive/warehouse")
.setMaster("local[*]")
.setAppName("TestApp");
JavaSparkContext sc = new JavaSparkContext(conf);
SparkSession session = SparkSes
I don't think Broadcast itself can be serialized. you can get the value out
on the driver side and refer to it in foreach, then the value would be
serialized with the lambda expr and sent to workers.
On Fri, Jun 16, 2017 at 2:29 AM, Anton Kravchenko <
kravchenko.anto...@gmail.com> wrote:
> How on
