@Chaoran Yu I finally got it working. here is my code. I usually code in
Java but tried to convert it into scala below.
import spark.implicits._
import org.apache.spark.streaming._
SparkConf sparkConf = sc.getConf();
sparkConf.setJars(JavaSparkContext.jarOfClass(Hello.class));
val streamingContext = new StreamingContext(sc, 1000)
val jsonDStream = streamingContext.receiverStream(receiver); // indirect
receiver
jsonDStream.foreachRDD{rdd =>
val jsonDF = spark.read.json(rdd)
jsonDF.createOrReplaceTempView("jsondf")
}
streamingContext.start()
%sql select * from jsondf
On Sun, Apr 23, 2017 at 12:11 PM, kant kodali <[email protected]> wrote:
> @Chaoran Yu Yeah I don't think its dependency issue. you wouldn't be able
> to call methods if you are missing dependencies.
>
> I am also in a similar boat though I am trying to get Streaming and
> Zeppelin to work except I have my own indirect receiver (not the direct
> stream). That twitter example is pretty old. I am using Spark 2.1.0. I
> wonder if I should call streamingContext.awaitTermination from zeppelin?
> I only do the following and streamingContext is stopped every time after
> execute the following lines. still debugging and trying to see what is
> going on. can let you know once I have something working.
>
> jsonDStream.foreachRDD(rdd => println(rdd.count()));
> streamingContext.start();
>
> On Thu, Mar 30, 2017 at 6:47 AM, Chaoran Yu <[email protected]>
> wrote:
>
>>
>> I think I’ve added required artifacts in interpreter dependency settings.
>> If there are artifacts missing, wouldn’t I see errors either in Zeppelin
>> notebook UI or /logs folder? But I didn’t. For example, if I missed a
>> Twitter artifact, I would have gotten a ClassNotFound error. Instead, what
>> I saw was the notebook started running but never made any progress.
>>
>>
>> On Mar 30, 2017, at 3:59 AM, Raffaele S <[email protected]> wrote:
>>
>> You might have to add the relative artifacts manually in the interpreter
>> page.
>>
>> Raffaele
>>
>>
>>
>> 2017-03-30 4:12 GMT+02:00 Chaoran Yu <[email protected]>:
>>
>>> Hello,
>>>
>>> Has anybody got Spark Streaming Twitter example to work in Zeppelin?
>>> When I started the streaming context with ssc.start(), the Zeppelin
>>> paragraph seemed to have started but it got stuck there. The top right
>>> corner of the paragraph says “RUNNING 0%”
>>>
>>> I think this is a problem with Spark Streaming + Zeppelin, rather than
>>> one with Twitter example in particular. Because I’ve tried my own simple
>>> streaming tests but got the same thing: Stuck in “RUNNING 0%” status
>>> forever.
>>>
>>> I also tried to stop the streaming context with ssc.stop() in a new
>>> paragraph but it won’t execute i.e. remain in “PENDING” status. In fact,
>>> any new code won’t execute in a new paragraph. I had to restart Zeppelin to
>>> get me out of this situation. Zeppelin logs didn’t reveal any errors either.
>>>
>>> Could anyone help me here?
>>>
>>> Thank you,
>>> Chaoran Yu
>>
>>
>>
>>
>
