I like this "fancy idea" to just flip the to/from bytes but I think there
are some practical limitations to implementing this. In particular
I'm thinking about this issue
<https://issues.apache.org/jira/browse/KAFKA-8159> with the built-in signed
number serdes.

This trick would actually fix the problem for negative-negative queries
(ie where to & from are negative) but would cause undetectable
incorrect results for negative-positive queries. For example, say you
call #range with from = -1 and to = 1, using the Short serdes. The
serialized bytes for that are

from = 1111111111111111
to = 0000000000000001

so we would end up flipping those and iterating over all keys from
0000000000000001 to 1111111111111111. Iterating in lexicographical
order means we would iterate over every key in the space *except* for
0, but 0 is actually the *only* other key we meant to be included in the
range query.

Currently we just log a warning and return an empty iterator when
to < from, which is obviously also incorrect but feels slightly more
palatable. If we start automatically converting to reverse queries we
can't even log a warning in this case unless we wanted to log a warning
every time, which would be weird to do for a valid usage of a new
feature.

All that said, I still like the idea overall. Off the top of my head I guess
we could add a store config to enable/disable automatic reverse iteration,
which is off by default?

Thanks for the KIP! This will be a nice addition

Sophie


On Tue, May 19, 2020 at 3:21 PM John Roesler <vvcep...@apache.org> wrote:

> Hi there Jorge,
>
> Thanks for the KIP!
>
> I think this feature sounds very reasonable.
>
> I'm not 100% sure if this is "too fancy", but what do you think
> about avoiding the enum by instead allowing people to flip
> the "from" and "to" endpoints? I.e., reading from "A" to "Z" would
> be a forward scan, and from "Z" to "A" would be a backward one?
>
> Thanks,
> -John
>
> On Tue, May 19, 2020, at 16:20, Jorge Quilcate wrote:
> > Hi everyone,
> >
> > I would like to start the discussion for KIP-617:
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-617%3A+Allow+Kafka+Streams+State+Stores+to+be+iterated+backwards
> >
> > Looking forward to your feedback.
> >
> > Thanks!
> > Jorge.
> >
> >
> > Attachments:
> > * 0x5F2C6E22064982DF.asc
>

Reply via email to