RE: Restructure data in GlobakKTable

ta out of a GlobalKTable (other than the join which is what it's there for in the first place). (This only seems practical because the amount of data I'm handling here is quite small.) But now the requirements have changed and I'm struggling with handling deletions in the new req

RE: Kafka Streams incorrect aggregation results when re-balancing occurs

pplication changed when I was part way through writing it).) Tim Ward -Original Message- From: Alex Brekken Sent: 19 August 2019 20:24 To: users@kafka.apache.org Subject: Kafka Streams incorrect aggregation results when re-balancing occurs Hi all, I have a (relatively) simple streams topolo

RE: How do I tell Kafka Streams not to repartition?

Thanks. Tim Ward -Original Message- From: Matthias J. Sax Sent: 13 August 2019 08:23 To: users@kafka.apache.org Subject: Re: How do I tell Kafka Streams not to repartition? Atm, it's not possible to tell Kafka Streams that repartitioning is not necessary after a key-changing oper

RE: KSTREAM-AGGREGATE-STATE-STORE persistence?

Thanks! Tim Ward -Original Message- From: Bruno Cadonna Sent: 12 August 2019 14:18 To: users@kafka.apache.org Subject: Re: KSTREAM-AGGREGATE-STATE-STORE persistence? Hi Tim, Kafka Streams guarantees at-least-once processing semantics by default. That means, a record is processed (e.g

KSTREAM-AGGREGATE-STATE-STORE persistence?

e the aggregation process idempotent? I don't see this explained in the JavaDoc for either Aggregator (or Reducer, where presumably it also applies). Tim Ward This email is from Origami Energy Limited. The contents of this email and any attachment are confidential to the intended recipient(s). If y

RE: How do I tell Kafka Streams not to repartition?

or take a problem I've discovered with persistence following a crash in the middle of aggregation, which I'll post separately. Tim Ward -Original Message- From: Boyang Chen Sent: 09 August 2019 23:31 To: users@kafka.apache.org Subject: Re: How do I tell Kafka Streams not to r

RE: How to start a stream from only new records?

s of what offset it committed last time it ran. Tim Ward -Original Message- From: Boyang Chen Sent: 09 August 2019 17:23 To: users@kafka.apache.org Subject: Re: How to start a stream from only new records? Hey Tim, if you are talking about avoid re-processing data and start consumption

How do I tell Kafka Streams not to repartition?

at there is no need to repartition in this case, because all the data that should remain together in the same instance of the application will do so without repartitioning? (I appreciate that Streams can't know about the parent-child relationship unless I *do* tell it in some way.) Tim Ward

How to start a stream from only new records?

ocess any backlog? Tim Ward This email is from Origami Energy Limited. The contents of this email and any attachment are confidential to the intended recipient(s). If you are not an intended recipient: (i) do not use, disclose, distribute, copy or publish this email or its contents; (ii) please co

Restructure data in GlobakKTable

do this, but I don't know how to do this if using StreamsBuilder.stream().) What am I missing please? Tim Ward This email is from Origami Energy Limited. The contents of this email and any attachment are confidential to the intended recipient(s). If you are not an intended recipient: (i)

Java consumer error handling on DNS lookup failure

ChannelImpl.java:622) ~[?:1.8.0_212] at org.apache.kafka.common.network.Selector.doConnect(Selector.java:233) ~[kafka-clients-2.0.0.jar:?] ... 19 more Tim Ward This email is from Origami Energy Limited. The contents of this email and any attachment are confiden

RE: Prioritized Topics for Kafka

ve a look at it when they get bored"), of which there could be a long queue. Urgent alerts (a phone call telling someone "you need to do this now") should be processed before non-urgent alerts (a phone call telling someone "FYI, such and such is going to happen in a couple o

RE: Kafka running on AWS - how to retain broker.id on new instance spun-up in-place of instance/broker failed

Isn't that what stateful sets are for, giving you a predictable pod name from which you can derive the ID? Tim Ward -Original Message- From: Srinivas Rapolu Sent: 15 November 2018 17:42 To: users@kafka.apache.org Subject: Re: Kafka running on AWS - how to retain broker.id o

RE: Terminate Streams application from within Transformer?

That works, thanks. Tim Ward -Original Message- From: Bill Bejeck Sent: 21 September 2018 15:06 To: users@kafka.apache.org Subject: Re: Terminate Streams application from within Transformer? Hi Tim, I wouldn't recommend System.exit(), as it won't give streams a chance to go

Terminate Streams application from within Transformer?

record currently being processed, so that on application restart it re-processes this message and everything works fine. How should I do this? System.exit() or something more sophisticated? Tim Ward The contents of this email and any attachment are confidential to the intended recipient(s). I

RE: Understanding default.deserialization.exception.handler

Thanks. No, it's not that big a deal now that I understand it, but as I'd had to spend a fair amount of time working out what was going on I thought I'd flag it up in case it wasn't known about. Tim Ward -Original Message- From: Matthias J. Sax Sent: 14 September

RE: Timing state changes?

ous magic going on there, in relation to how I can crash the application and restart it and all the state is preserved with my having to write 0 characters of code to achieve this - I think I'm going to like Kafka Streams. Thanks. Tim Ward The contents of this email and any attachment are

Understanding default.deserialization.exception.handler

t any good messages having been read after the bad ones. Have I understood this correctly? If so, is this correct behaviour as designed? Is it documented to that level of detail? Tim Ward The contents of this email and any attachment are confidential to the intended recipient(s). If you are not a

RE: Timing state changes?

cally so I don't have to worry about it? (I'll be deploying to Kubernetes, and applications going away and restarting at random seems to be a fact of life there.) Tim Ward The contents of this email and any attachment are confidential to the intended recipient(s). If you are not

RE: Timing state changes?

Thanks - I've been diverted to other things but will come back to this shortly. Tim Ward -Original Message- From: John Roesler Sent: 06 September 2018 19:38 To: users@kafka.apache.org Subject: Re: Timing state changes? Hi Tim, From your spec, I think that Kafka Streams has se

Timing state changes?

out separately for each widget. (Solutions involving updating a database table and then polling it on a cron job are really the sort of thing we're trying to get away from here.) Tim Ward The contents of this email and any attachment are confidential to the intended recipient(s). If yo