You could use foreachRDD to do the operations and then inside the
foreach create an accumulator to gather all the errors together
dstream.foreachRDD { rdd =>
val accumulator = new Accumulator[]
rdd.map { . }.count // whatever operation that is error prone
// gather all errors
Hi,
On Wed, Jan 28, 2015 at 1:45 PM, Soumitra Kumar
wrote:
> It is a Streaming application, so how/when do you plan to access the
> accumulator on driver?
>
Well... maybe there would be some user command or web interface showing the
errors that have happened during processing...?
Thanks
Tobias
It is a Streaming application, so how/when do you plan to access the
accumulator on driver?
On Tue, Jan 27, 2015 at 6:48 PM, Tobias Pfeiffer wrote:
> Hi,
>
> thanks for your mail!
>
> On Wed, Jan 28, 2015 at 11:44 AM, Tathagata Das <
> tathagata.das1...@gmail.com> wrote:
>
>> That seems reasonab
Hi,
thanks for your mail!
On Wed, Jan 28, 2015 at 11:44 AM, Tathagata Das wrote:
> That seems reasonable to me. Are you having any problems doing it this way?
>
Well, actually I haven't done that yet. The idea of using accumulators to
collect errors just came while writing the email, but I tho
