So somehow Spark Streaming doesn't support display of named accumulators in
the WebUI?
On Tue, Feb 24, 2015 at 7:58 AM, Petar Zecevic
wrote:
>
> Interesting. Accumulators are shown on Web UI if you are using the
> ordinary SparkContext (Spark 1.2). It just has to be named (and that's what
> you
Interesting. Accumulators are shown on Web UI if you are using the
ordinary SparkContext (Spark 1.2). It just has to be named (and that's
what you did).
scala> val acc = sc.accumulator(0, "test accumulator")
acc: org.apache.spark.Accumulator[Int] = 0
scala> val rdd = sc.parallelize(1 to 1000)
Unless I am unaware some latest changes, the SparkUI shows stages, and
jobs, not accumulator results. And the UI not designed to be pluggable for
showing user-defined stuff.
TD
On Fri, Feb 20, 2015 at 12:25 AM, Tim Smith wrote:
> On Spark 1.2:
>
> I am trying to capture # records read from a ka
On Spark 1.2:
I am trying to capture # records read from a kafka topic:
val inRecords = ssc.sparkContext.accumulator(0, "InRecords")
..
kInStreams.foreach( k =>
{
k.foreachRDD ( rdd => inRecords += rdd.count().toInt )
inRecords.value
Questi
