Re: Tracking deserialization errors

2018-04-23 Thread Alexander Smirnov
That's absolutely no problem Tzu-Li. Either of them would work. Thank you! On Thu, Apr 19, 2018 at 4:56 PM Tzu-Li (Gordon) Tai wrote: > @Alexander > Sorry about that, that would be my mistake. I’ll close FLINK-9204 as a > duplicate and leave my thoughts on FLINK-9155. Thanks for pointing out! >

Re: Tracking deserialization errors

2018-04-19 Thread Tzu-Li (Gordon) Tai
@Alexander Sorry about that, that would be my mistake. I’ll close FLINK-9204 as a duplicate and leave my thoughts on FLINK-9155. Thanks for pointing out! On 19 April 2018 at 2:00:51 AM, Elias Levy (fearsome.lucid...@gmail.com) wrote: Either proposal would work.  In the later case, at a minimum

Re: Tracking deserialization errors

2018-04-18 Thread Elias Levy
Either proposal would work. In the later case, at a minimum we'd need a way to identify the source within the metric. The basic error metric would then allow us to go into the logs to determine the cause of the error, as we already record the message causing trouble in the log. On Mon, Apr 16,

Re: Tracking deserialization errors

2018-04-18 Thread Alexander Smirnov
;> I have the same question. In case of kafka source, it would be good to >> know topic name and offset of the corrupted message for further >> investigation. >> Looks like the only option is to write messages into a log file >> >> On Fri, Apr 6, 2018 at 9:12 PM

Re: Tracking deserialization errors

2018-04-18 Thread Tzu-Li (Gordon) Tai
essage for further investigation. Looks like the only option is to write messages into a log file On Fri, Apr 6, 2018 at 9:12 PM Elias Levy wrote: I was wondering how are folks tracking deserialization errors.  The  AbstractDeserializationSchema interface provides no mechanism for the deser

Re: Tracking deserialization errors

2018-04-16 Thread Fabian Hueske
case of kafka source, it would be good to > know topic name and offset of the corrupted message for further > investigation. > Looks like the only option is to write messages into a log file > > On Fri, Apr 6, 2018 at 9:12 PM Elias Levy > wrote: > >> I was wondering how a

Re: Tracking deserialization errors

2018-04-08 Thread Alexander Smirnov
lks tracking deserialization errors. > The AbstractDeserializationSchema interface provides no mechanism for the > deserializer to instantiate a metric counter, and "deserialize" must return > a null instead of raising an exception in case of error if you want your > job to cont

Tracking deserialization errors

2018-04-06 Thread Elias Levy
I was wondering how are folks tracking deserialization errors. The AbstractDeserializationSchema interface provides no mechanism for the deserializer to instantiate a metric counter, and "deserialize" must return a null instead of raising an exception in case of error if you want y