Re: Running the BroadcastTest.scala with TorrentBroadcastFactory in a standalone cluster

2014-07-07 Thread jackxucs
Hi Mosharaf, Thanks a lot for the detailed reply. The reason I am using Torrent is mainly because I would like to have something different to be a comparison with the IP multicast/broadcast. I will need to transmit larger data size as well. I tried to increase spark.executor.memory to 2g and spar

Re: Running the BroadcastTest.scala with TorrentBroadcastFactory in a standalone cluster

2014-07-03 Thread Mosharaf Chowdhury
Hi Jack, 1. Several previous instances of "key not valid?" error had been attributed to memory issues, either memory allocated per executor or per task, depending on the context. You can google it to see some examples. 2. I think your case is similar, even though its happening due to

Running the BroadcastTest.scala with TorrentBroadcastFactory in a standalone cluster

2014-07-03 Thread jackxucs
Hello, I am running the BroadcastTest example in a standalone cluster using spark-submit. I have 8 host machines and made Host1 the master. Host2 to Host8 act as 7 workers to connect to the master. The connection was fine as I could see all 7 hosts on the master web ui. The BroadcastTest example w