In that case, though, every access to that key is doomed to failure as the database is corrupted. So i think it should probably die in a steaming heap at that point!
On Fri, 26 May 2017 at 17:33 Eno Thereska <eno.there...@gmail.com> wrote: > Hi Damian, > > I was thinking of cases when there is bit-rot on the storage itself and we > get a malformed record that cannot be de-serialized. There is an > interesting intersection here with CRCs in both Kafka (already there, they > throw on deserialization) and potentially local storage (we don't have CRCs > here on the data files, though RocksDB has them on its write-ahead log > records). > > Basically in a nutshell, I'm saying that every deserialization exception > should go through this new path. The user can decide to fail or continue. > We could start with just poison pills from Kafka though and punt the > storage one to later. > > Eno > > > On 26 May 2017, at 16:59, Damian Guy <damian....@gmail.com> wrote: > > > > Eno, > > > > Under what circumstances would you get a deserialization exception from > the > > state store? I can only think of the case where someone has provided a > bad > > deserializer to a method that creates a state store. In which case it > would > > be a user error and probably should just abort? > > > > Thanks, > > Damian > > > > On Fri, 26 May 2017 at 16:32 Eno Thereska <eno.there...@gmail.com> > wrote: > > > >> See latest reply to Jan's note. I think I unnecessarily broadened the > >> scope of this KIP to the point where it sounded like it handles all > sorts > >> of exceptions. The scope should be strictly limited to "poison pill" > >> records for now. Will update KIP, > >> > >> Thanks > >> Eno > >>> On 26 May 2017, at 16:16, Matthias J. Sax <matth...@confluent.io> > wrote: > >>> > >>> "bad" for this case would mean, that we got an > >>> `DeserializationException`. I am not sure if any other processing error > >>> should be covered? > >>> > >>> @Eno: this raises one one question. Might it be better to allow for two > >>> handlers instead of one? One for deserialization exception and one for > >>> all other exceptions from user code? > >>> > >>> Just a thought. > >>> > >>> > >>> -Matthias > >>> > >>> On 5/26/17 7:49 AM, Jim Jagielski wrote: > >>>> > >>>>> On May 26, 2017, at 5:13 AM, Eno Thereska <eno.there...@gmail.com> > >> wrote: > >>>>> > >>>>> > >>>>>> > >>>>>> > >>>>>> With regard to `DeserializationException`, do you thing it might > make > >>>>>> sense to have a "dead letter queue" as a feature to provide > >> out-of-the-box? > >>>>> > >>>>> We could provide a special topic where bad messages go to, and then > >> we'd have to add a config option for the user to provide a topic. Is > that > >> what you're thinking? > >>>>> > >>>> > >>>> For various definitions of "bad"?? > >>>> > >>> > >> > >> > >
