Nicolas,

I'm checking with Bill who originally was interested in KAFKA-4468. If he isn't 
actively working on it, why don't you give it a go and create a pull request 
(PR) for it? That way your contribution is properly acknowledged etc. We can 
help you through with that.

Thanks
Eno
> On 16 Jan 2017, at 18:46, Nicolas Fouché <nfou...@onfocus.io> wrote:
> 
> My current implementation:
> https://gist.github.com/nfo/eaf350afb5667a3516593da4d48e757a . I just
> appended the window `end` at the end of the byte array.
> Comments and suggestions are welcome !
> 
> 
> 2017-01-16 15:48 GMT+01:00 Nicolas Fouché <nfou...@onfocus.io>:
> 
>> Hi Damian,
>> 
>> I recall now that I copied the `WindowedSerde` class [1] from Confluent
>> examples by Confluent, which uses the internal `WindowedSerializer` class.
>> Better write my own Serde them. You're right, I should not rely on
>> internal classes, especially for data written outside Kafka Streams
>> topologies.
>> 
>> Thanks for the insights on KAFKA-4468.
>> 
>> https://github.com/confluentinc/examples/blob/
>> 89db45c6890cf757b8e18565bdf7bc23f119a2ff/kafka-streams/src/
>> main/java/io/confluent/examples/streams/utils/WindowedSerde.java
>> 
>> Nicolas.
>> 
>> 2017-01-16 12:31 GMT+01:00 Damian Guy <damian....@gmail.com>:
>> 
>>> Hi Nicolas,
>>> 
>>> I guess you are using the Processor API for your topology? The
>>> WindowedSerializer is an internal class that is used as part of the DSL.
>>> In
>>> the DSL a topic will be created for each window operation, so we don't
>>> need
>>> the end time as it can be calculated from the window size.
>>> However, there is an open jira for this:
>>> https://issues.apache.org/jira/browse/KAFKA-4468
>>> 
>>> Thanks,
>>> Damian
>>> 
>>> On Mon, 16 Jan 2017 at 11:18 Nicolas Fouché <nfou...@onfocus.io> wrote:
>>> 
>>>> Hi,
>>>> 
>>>> In the same topology, I generate aggregates with 1-day windows and
>>> 1-week
>>>> windows and write them in one single topic. On Mondays, these windows
>>> have
>>>> the same start time. The effect: these aggregates overrides each other.
>>>> 
>>>> That happens because WindowedSerializer [1] only serializes the window
>>>> start time. I'm a bit surprised, a window has by definition a start and
>>> an
>>>> end. I suppose one wanted save on key sizes ? And/or one would consider
>>>> that topics should not contain aggregates with different granularities ?
>>>> 
>>>> I have two choices then, either create as many output topics as I have
>>>> granularities, or create my own serializer which also includes the
>>> window
>>>> end time. What would the community recommend ?
>>>> 
>>>> Getting back to the core problem:
>>>> I could understand that it's not "right" to store different
>>> granularities
>>>> in one topic, and I thought it would save resources (less topic to
>>> manage
>>>> by Kafka). But, I'm really not sure about this default serializer: it
>>> does
>>>> not serialize all instance variables of the `Window` class, and more
>>>> generally does comply to the definition of a window.
>>>> 
>>>> [1]
>>>> 
>>>> https://github.com/apache/kafka/blob/0.10.1/streams/src/main
>>> /java/org/apache/kafka/streams/kstream/internals/WindowedSerializer.java
>>>> 
>>>> Thanks.
>>>> Nicolas
>>>> 
>>> 
>> 
>> 

Reply via email to