Yes it is, you have some more customizable options over here http://spark.apache.org/docs/1.2.0/configuration.html#compression-and-serialization
Thanks Best Regards On Sun, Feb 22, 2015 at 11:47 PM, Tassilo Klein <[email protected]> wrote: > Hi Akhil, > > thanks for your reply. I am using the latest version of Spark 1.2.1 (also > tried 1.3 developer branch). If I am not mistaken the TorrentBroadcast is > the default there, isn't it? > > Thanks, > Tassilo > > On Sun, Feb 22, 2015 at 10:59 AM, Akhil Das <[email protected]> > wrote: > >> Did you try with torrent broadcast factory? >> >> Thanks >> Best Regards >> >> On Sun, Feb 22, 2015 at 3:29 PM, TJ Klein <[email protected]> wrote: >> >>> Hi, >>> >>> I am trying to broadcast large objects (order of a couple of 100 MBs). >>> However, I keep getting errors when trying to do so: >>> >>> Traceback (most recent call last): >>> File "/LORM_experiment.py", line 510, in <module> >>> broadcast_gradient_function = sc.broadcast(gradient_function) >>> File "/scratch/users/213444/spark/python/pyspark/context.py", line >>> 643, in >>> broadcast >>> return Broadcast(self, value, self._pickled_broadcast_vars) >>> File "/scratch/users/213444/spark/python/pyspark/broadcast.py", line >>> 65, >>> in __init__ >>> self._path = self.dump(value, f) >>> File "/scratch/users/213444/spark/python/pyspark/broadcast.py", line >>> 82, >>> in dump >>> cPickle.dump(value, f, 2) >>> SystemError: error return without exception set >>> 15/02/22 04:52:14 ERROR Utils: Uncaught exception in thread delete Spark >>> local dirs >>> java.lang.IllegalStateException: Shutdown in progress >>> >>> Any idea how to prevent that? I got plenty of RAM, so there shouldn't be >>> any >>> problem with that. >>> >>> Thanks, >>> Tassilo >>> >>> >>> >>> -- >>> View this message in context: >>> http://apache-spark-user-list.1001560.n3.nabble.com/Broadcasting-Large-Objects-Fails-tp21752.html >>> Sent from the Apache Spark User List mailing list archive at Nabble.com. >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: [email protected] >>> For additional commands, e-mail: [email protected] >>> >>> >> >
