Yes of course it works.
[image: Inline image 1]
Thanks
Best Regards
On Tue, Sep 22, 2015 at 4:53 PM, Priya Ch <learnings.chitt...@gmail.com>
wrote:
> Parallelzing some collection (array of strings). Infact in our product we
> are reading data from kafka using KafkaUtils.createStream and applying some
> transformations.
>
> Is creating sparContext at object level instead of creating in main
> doesn't work ????
>
> On Tue, Sep 22, 2015 at 2:59 PM, Akhil Das <ak...@sigmoidanalytics.com>
> wrote:
>
>> Its a "value" not a variable, and what are you parallelizing here?
>>
>> Thanks
>> Best Regards
>>
>> On Fri, Sep 18, 2015 at 11:21 PM, Priya Ch <learnings.chitt...@gmail.com>
>> wrote:
>>
>>> Hello All,
>>>
>>> Instead of declaring sparkContext in main, declared as object variable
>>> as -
>>>
>>> object sparkDemo
>>> {
>>>
>>> val conf = new SparkConf
>>> val sc = new SparkContext(conf)
>>>
>>> def main(args:Array[String])
>>> {
>>>
>>> val baseRdd = sc.parallelize()
>>> .
>>> .
>>> .
>>> }
>>>
>>> }
>>>
>>> But this piece of code is giving :
>>> java.io.IOException: org.apache.spark.SparkException: Failed to get
>>> broadcast_5_piece0 of broadcast_5
>>> at
>>> org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1155)
>>> at
>>> org.apache.spark.broadcast.TorrentBroadcast.readBroadcastBlock(TorrentBroadcast.scala:164)
>>> at
>>> org.apache.spark.broadcast.TorrentBroadcast._value$lzycompute(TorrentBroadcast.scala:64)
>>> at
>>> org.apache.spark.broadcast.TorrentBroadcast._value(TorrentBroadcast.scala:64)
>>> at
>>> org.apache.spark.broadcast.TorrentBroadcast.getValue(TorrentBroadcast.scala:87)
>>> at org.apache.spark.broadcast.Broadcast.value(Broadcast.scala:70)
>>> at
>>> org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:58)
>>> at org.apache.spark.scheduler.Task.run(Task.scala:64)
>>> at
>>> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>> at java.lang.Thread.run(Thread.java:745)
>>> Caused by: org.apache.spark.SparkException: Failed to get
>>> broadcast_5_piece0 of broadcast_5
>>> at
>>> org.apache.spark.broadcast.TorrentBroadcast$$anonfun$org$apache$spark$broadcast$TorrentBroadcast$$readBlocks$1$$anonfun$2.apply(TorrentBroadcast.scala:137)
>>> at
>>> org.apache.spark.broadcast.TorrentBroadcast$$anonfun$org$apache$spark$broadcast$TorrentBroadcast$$readBlocks$1$$anonfun$2.apply(TorrentBroadcast.scala:137)
>>> at scala.Option.getOrElse(Option.scala:120)
>>> at
>>> org.apache.spark.broadcast.TorrentBroadcast$$anonfun$org$apache$spark$broadcast$TorrentBroadcast$$readBlocks$1.apply$mcVI$sp(TorrentBroadcast.scala:136)
>>> at
>>> org.apache.spark.broadcast.TorrentBroadcast$$anonfun$org$apache$spark$broadcast$TorrentBroadcast$$readBlocks$1.apply(TorrentBroadcast.scala:119)
>>> at
>>> org.apache.spark.broadcast.TorrentBroadcast$$anonfun$org$apache$spark$broadcast$TorrentBroadcast$$readBlocks$1.apply(TorrentBroadcast.scala:119)
>>> at scala.collection.immutable.List.foreach(List.scala:318)
>>> at org.apache.spark.broadcast.TorrentBroadcast.org
>>> <http://org.apache.spark.broadcast.torrentbroadcast.org/>
>>> $apache$spark$broadcast$TorrentBroadcast$$readBlocks(TorrentBroadcast.scala:119)
>>> at
>>> org.apache.spark.broadcast.TorrentBroadcast$$anonfun$readBroadcastBlock$1.apply(TorrentBroadcast.scala:174)
>>> at
>>> org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1152)
>>>
>>> Why should't we declare sc as object variable ???
>>>
>>> Regards,
>>> Padma Ch
>>>
>>
>>
>
