My current implementation: https://gist.github.com/nfo/eaf350afb5667a3516593da4d48e757a . I just appended the window `end` at the end of the byte array. Comments and suggestions are welcome !
2017-01-16 15:48 GMT+01:00 Nicolas Fouché <nfou...@onfocus.io>: > Hi Damian, > > I recall now that I copied the `WindowedSerde` class [1] from Confluent > examples by Confluent, which uses the internal `WindowedSerializer` class. > Better write my own Serde them. You're right, I should not rely on > internal classes, especially for data written outside Kafka Streams > topologies. > > Thanks for the insights on KAFKA-4468. > > https://github.com/confluentinc/examples/blob/ > 89db45c6890cf757b8e18565bdf7bc23f119a2ff/kafka-streams/src/ > main/java/io/confluent/examples/streams/utils/WindowedSerde.java > > Nicolas. > > 2017-01-16 12:31 GMT+01:00 Damian Guy <damian....@gmail.com>: > >> Hi Nicolas, >> >> I guess you are using the Processor API for your topology? The >> WindowedSerializer is an internal class that is used as part of the DSL. >> In >> the DSL a topic will be created for each window operation, so we don't >> need >> the end time as it can be calculated from the window size. >> However, there is an open jira for this: >> https://issues.apache.org/jira/browse/KAFKA-4468 >> >> Thanks, >> Damian >> >> On Mon, 16 Jan 2017 at 11:18 Nicolas Fouché <nfou...@onfocus.io> wrote: >> >> > Hi, >> > >> > In the same topology, I generate aggregates with 1-day windows and >> 1-week >> > windows and write them in one single topic. On Mondays, these windows >> have >> > the same start time. The effect: these aggregates overrides each other. >> > >> > That happens because WindowedSerializer [1] only serializes the window >> > start time. I'm a bit surprised, a window has by definition a start and >> an >> > end. I suppose one wanted save on key sizes ? And/or one would consider >> > that topics should not contain aggregates with different granularities ? >> > >> > I have two choices then, either create as many output topics as I have >> > granularities, or create my own serializer which also includes the >> window >> > end time. What would the community recommend ? >> > >> > Getting back to the core problem: >> > I could understand that it's not "right" to store different >> granularities >> > in one topic, and I thought it would save resources (less topic to >> manage >> > by Kafka). But, I'm really not sure about this default serializer: it >> does >> > not serialize all instance variables of the `Window` class, and more >> > generally does comply to the definition of a window. >> > >> > [1] >> > >> > https://github.com/apache/kafka/blob/0.10.1/streams/src/main >> /java/org/apache/kafka/streams/kstream/internals/WindowedSerializer.java >> > >> > Thanks. >> > Nicolas >> > >> > >
