Hi Damian,

I recall now that I copied the `WindowedSerde` class [1] from Confluent
examples by Confluent, which uses the internal `WindowedSerializer` class.
Better write my own Serde them. You're right, I should not rely on internal
classes, especially for data written outside Kafka Streams topologies.

Thanks for the insights on KAFKA-4468.

https://github.com/confluentinc/examples/blob/89db45c6890cf757b8e18565bdf7bc23f119a2ff/kafka-streams/src/main/java/io/confluent/examples/streams/utils/WindowedSerde.java

Nicolas.

2017-01-16 12:31 GMT+01:00 Damian Guy <damian....@gmail.com>:

> Hi Nicolas,
>
> I guess you are using the Processor API for your topology? The
> WindowedSerializer is an internal class that is used as part of the DSL. In
> the DSL a topic will be created for each window operation, so we don't need
> the end time as it can be calculated from the window size.
> However, there is an open jira for this:
> https://issues.apache.org/jira/browse/KAFKA-4468
>
> Thanks,
> Damian
>
> On Mon, 16 Jan 2017 at 11:18 Nicolas Fouché <nfou...@onfocus.io> wrote:
>
> > Hi,
> >
> > In the same topology, I generate aggregates with 1-day windows and 1-week
> > windows and write them in one single topic. On Mondays, these windows
> have
> > the same start time. The effect: these aggregates overrides each other.
> >
> > That happens because WindowedSerializer [1] only serializes the window
> > start time. I'm a bit surprised, a window has by definition a start and
> an
> > end. I suppose one wanted save on key sizes ? And/or one would consider
> > that topics should not contain aggregates with different granularities ?
> >
> > I have two choices then, either create as many output topics as I have
> > granularities, or create my own serializer which also includes the window
> > end time. What would the community recommend ?
> >
> > Getting back to the core problem:
> > I could understand that it's not "right" to store different granularities
> > in one topic, and I thought it would save resources (less topic to manage
> > by Kafka). But, I'm really not sure about this default serializer: it
> does
> > not serialize all instance variables of the `Window` class, and more
> > generally does comply to the definition of a window.
> >
> > [1]
> >
> > https://github.com/apache/kafka/blob/0.10.1/streams/src/
> main/java/org/apache/kafka/streams/kstream/internals/
> WindowedSerializer.java
> >
> > Thanks.
> > Nicolas
> >
>

Reply via email to