Re: Which kafka client to use with spark streaming

2017-12-25 Thread Diogo Munaro Vieira
Hey Serkan, it depends of your Kafka version... Is it 0.8.2? Em 25 de dez de 2017 06:17, "Serkan TAS" escreveu: > Hi, > > > > Working on spark 2.2.0 cluster and 1.0 kafka brokers. > > > > I was using the library > > "org.apache.spark" % "spark-streaming-kafka-0-10_2.11" % "2.2.0" > > > > and had

Re: Apache Spark - (2.2.0) - window function for DataSet

2017-12-25 Thread Diogo Munaro Vieira
Window function requires a timestamp column because you will apply a function for each window (like an aggregation). You still can use UDF for customized tasks Em 25 de dez de 2017 20:15, "M Singh" escreveu: > Hi: > I would like to use window function on a DataSet stream (Spark 2.2.0) > The wind

Re: Apache Spark - Structured Streaming from file - checkpointing

2017-12-25 Thread Diogo Munaro Vieira
Can you please post here your code? Em 25 de dez de 2017 19:24, "M Singh" escreveu: > Hi: > > I am using spark structured streaming (v 2.2.0) to read data from files. I > have configured checkpoint location. On stopping and restarting the > application, it looks like it is reading the previously

Re: Apache Spark - Structured Streaming graceful shutdown

2017-12-25 Thread Diogo Munaro Vieira
Hi M Singh! Here I'm using query.stop() Em 25 de dez de 2017 19:19, "M Singh" escreveu: > Hi: > Are there any patterns/recommendations for gracefully stopping a > structured streaming application ? > Thanks > > >