Nicolas, I'm checking with Bill who originally was interested in KAFKA-4468. If he isn't actively working on it, why don't you give it a go and create a pull request (PR) for it? That way your contribution is properly acknowledged etc. We can help you through with that.
Thanks Eno > On 16 Jan 2017, at 18:46, Nicolas Fouché <nfou...@onfocus.io> wrote: > > My current implementation: > https://gist.github.com/nfo/eaf350afb5667a3516593da4d48e757a . I just > appended the window `end` at the end of the byte array. > Comments and suggestions are welcome ! > > > 2017-01-16 15:48 GMT+01:00 Nicolas Fouché <nfou...@onfocus.io>: > >> Hi Damian, >> >> I recall now that I copied the `WindowedSerde` class [1] from Confluent >> examples by Confluent, which uses the internal `WindowedSerializer` class. >> Better write my own Serde them. You're right, I should not rely on >> internal classes, especially for data written outside Kafka Streams >> topologies. >> >> Thanks for the insights on KAFKA-4468. >> >> https://github.com/confluentinc/examples/blob/ >> 89db45c6890cf757b8e18565bdf7bc23f119a2ff/kafka-streams/src/ >> main/java/io/confluent/examples/streams/utils/WindowedSerde.java >> >> Nicolas. >> >> 2017-01-16 12:31 GMT+01:00 Damian Guy <damian....@gmail.com>: >> >>> Hi Nicolas, >>> >>> I guess you are using the Processor API for your topology? The >>> WindowedSerializer is an internal class that is used as part of the DSL. >>> In >>> the DSL a topic will be created for each window operation, so we don't >>> need >>> the end time as it can be calculated from the window size. >>> However, there is an open jira for this: >>> https://issues.apache.org/jira/browse/KAFKA-4468 >>> >>> Thanks, >>> Damian >>> >>> On Mon, 16 Jan 2017 at 11:18 Nicolas Fouché <nfou...@onfocus.io> wrote: >>> >>>> Hi, >>>> >>>> In the same topology, I generate aggregates with 1-day windows and >>> 1-week >>>> windows and write them in one single topic. On Mondays, these windows >>> have >>>> the same start time. The effect: these aggregates overrides each other. >>>> >>>> That happens because WindowedSerializer [1] only serializes the window >>>> start time. I'm a bit surprised, a window has by definition a start and >>> an >>>> end. I suppose one wanted save on key sizes ? And/or one would consider >>>> that topics should not contain aggregates with different granularities ? >>>> >>>> I have two choices then, either create as many output topics as I have >>>> granularities, or create my own serializer which also includes the >>> window >>>> end time. What would the community recommend ? >>>> >>>> Getting back to the core problem: >>>> I could understand that it's not "right" to store different >>> granularities >>>> in one topic, and I thought it would save resources (less topic to >>> manage >>>> by Kafka). But, I'm really not sure about this default serializer: it >>> does >>>> not serialize all instance variables of the `Window` class, and more >>>> generally does comply to the definition of a window. >>>> >>>> [1] >>>> >>>> https://github.com/apache/kafka/blob/0.10.1/streams/src/main >>> /java/org/apache/kafka/streams/kstream/internals/WindowedSerializer.java >>>> >>>> Thanks. >>>> Nicolas >>>> >>> >> >>
