Re: Spark-shell doesn't see changes coming from Kafka topic

2016-12-05 Thread Otávio Carvalho
ntinuously running but not finding latest offsets? > > > On Thu, Dec 1, 2016 at 6:31 AM, Otávio Carvalho wrote: > >> Hello hivemind, >> >> I am trying to connect my Spark 2.0.2 cluster to an Apache Kafka 0.10 >> cluster via spark-shell. >> >> The co

Spark-shell doesn't see changes coming from Kafka topic

2016-12-01 Thread Otávio Carvalho
* *Processing rate: 0.0 rows/sec* *Trigger details:* *triggerId: -1* *Sink status - org.apache.spark.sql.execution.streaming.ConsoleSink@54d5b6cb* *Committed offsets: [-]* I am starting the spark-shell as follows: /root/spark/bin/spark-shell --packages org.apache.spark:spark-sql-kafka-

Re: Use SparkListener to get overall progress of an action

2014-05-23 Thread Otávio Carvalho
Mayur, I'm interested on it as well. Can you send me? Cheers, Otávio Carvalho. Undergrad. Student at Federal University of Rio Grande do Sul Porto Alegre, Brazil. 2014-05-23 11:00 GMT-03:00 Pierre Borckmans < pierre.borckm...@realimpactanalytics.com>: > That would be great,

Re: Spark Performace Comparison Spark on YARN vs Spark Standalone

2014-05-23 Thread Otávio Carvalho
I have been analyzing Storm performance and there's no significant overhead added to the processing nodes. I'm interested in those results over Spark as well. Thanks in advance, Otávio Carvalho. Undergrad. CompSci Student at UFRGS Porto Alegre, Brazil. 2014-05-20 18:46 GMT-03