RE: Maintaining overall cumulative data in Spark Streaming

Yes, update state by key worked. Though there are some more complications. On Oct 30, 2015 8:27 AM, "skaarthik oss" wrote: > Did you consider UpdateStateByKey operation? > > > > *From:* Sandeep Giri [mailto:sand...@knowbigdata.com] > *Sent:* Thursday, October 29, 2015 3:09 PM > *To:* user ; dev

RE: Maintaining overall cumulative data in Spark Streaming

Did you consider UpdateStateByKey operation? From: Sandeep Giri [mailto:sand...@knowbigdata.com] Sent: Thursday, October 29, 2015 3:09 PM To: user ; dev Subject: Maintaining overall cumulative data in Spark Streaming Dear All, If a continuous stream of text is coming in and you have t

Re: Maintaining overall cumulative data in Spark Streaming

-dev +user Hi Sandeep, Perhaps (flat)mapping values and using an accumulator? > El 29/10/2015, a las 23:08, Sandeep Giri escribió: > > Dear All, > > If a continuous stream of text is coming in and you have to keep publishing > the overall word count so far since 0:00 today, what would you d

Maintaining overall cumulative data in Spark Streaming

Dear All, If a continuous stream of text is coming in and you have to keep publishing the overall word count so far since 0:00 today, what would you do? Publishing the results for a window is easy but if we have to keep aggregating the results, how to go about it? I have tried to keep an StreamR

[VOTE] Release Apache Spark 1.5.2 (RC1)

+1 (non binding) I tested several of the examples on mesos latest version (fine and coarse-grained modes) and they work fine. Hope not too late...though.. -- Stavros Kontopoulos

Spark streaming - failed recovery from checkpoint

Hi guys, (apologize for the huge font, reposting), I’ve encountered some problems with a crashed Spark Streaming job, when restoring from checkpoint. I’m runnning spark 1.5.1 on Yarn (hadoop 2.6) in cluster mode, reading from Kafka with the direct consumer and a few updateStateByKey stateful

Getting Started

Hello, my name is Saurabh Shah and I am a second year undergraduate student at DA-IICT, Gandhinagar, India. I have quite lately been contributing towards the open source organizations and I find your organization the most appropriate one to work on. I request you to please guide me through the

Spark streaming - failed recovery from checkpoint

Hi guys, I’ve encountered some problems with a crashed Spark Streaming job, when restoring from checkpoint. I’m runnning spark 1.5.1 on Yarn (hadoop 2.6) in cluster mode, reading from Kafka with the direct consumer and a few updateStateByKey stateful transformations. After investigating, I thi

Re: want to contribute

You both can check out following links :- https://cwiki.apache.org/confluence/display/SPARK/Contributing+to+Spark http://spark.apache.org/docs/latest/building-spark.html Thanks -Nitin On Thu, Oct 29, 2015 at 4:13 PM, Aadi Thakar wrote: > Hello, my name is Aaditya Thakkar and I am a second yea

want to contribute

Hello, my name is Aaditya Thakkar and I am a second year undergraduate ICT student at DA-IICT, Gandhinagar, India. I have quite lately been interested in contributing towards the open source organization and I find your organization the most appropriate one. I request you to please guide me throug

Fwd: [jira] [Created] (HADOOP-12527) Upgrade Avro dependency to 1.7.7

Coming from AWS dev team. he's probably got a point: even if you cut avro from the hadoop POM transitives, if you run in a Hadoop cluster with the hadoop classpaths getting in, there could be conflict. 1. has anyone seen this? 2. when spark when up to avro 1.7.7 —what broke? It's that risk of b

Guidance to get started

Hello,my name is Aaska Shah and I am a second year undergrad student at DAIICT,Gandhinagar,India. I have quite lately been interested in contributing towards the open source organization and I find your organization the most appropriate one. I request you to please guide me through how to install