Processor API in 2.7

Hi, I wanted to provide some feedback about the new API introduced in Kafka streaming 2.7 for adding a Processor to the Topology (using PAPI). Prior to 2.7, the Processor was typed only by the input key values , and the ProcessorContext was untyped. In 2.7 ProcessorSupplier & Processor now have

Emit events that are NOT joined

Hi, I am trying to find the best pattern to solve a specific problem using Kafka streaming. All of our current processing uses the Kafka streaming API (using multiple joins, windows, repartitions etc) so I already think I have a decent grasp of the fundamentals. We have 2 streams of events: - pr

Re: global state store

ate store (that is internally also just > a HashMap). > > > -Matthias > > On 11/4/20 6:15 AM, Ross Black wrote: > > I think I figured it out after digging through the source code ... > > > > > >- The Processor is called for each new message in the globa

Re: global state store

, and the StateRestoreCallback. Any better ideas / suggestions ? Thanks, Ross On Wed, 4 Nov 2020 at 20:41, Ross Black wrote: > Hi, > > I am trying to implement some functionality using a global state store, > and would like some help so that I can understand it. > > I am using

global state store

Hi, I am trying to implement some functionality using a global state store, and would like some help so that I can understand it. I am using the Processor API with Kafka 2.6. The code I have so far is : final StoreBuilder> storeBuilder = Stores.keyValueStoreBuilder(Stores.inMemoryKeyValueStore(

Sending messages within ProducerInterceptor callback?

Hi, I could not find any documentation or posts about the behaviour of a ProducerInterceptor if the callback '*onSend*' is used to send additional events to Kafka, so I was hoping somebody here could help me understand its behaviour. I would like to send a subset of outgoing messages (being sent

Topic labels / metadata

Hi, I looked through documentation etc but could not find anything... Is it possible to attach arbitrary labels / metadata to individual topics? Is there a plan to do this? Is this a feature that others would be interested in, or is there an alternative that I am missing? I am looking at it from

Re: New Producer Public API

Hi Jay, - Just to add some more info/confusion about possibly using Future ... If Kafka uses a JDK future, it plays nicely with other frameworks as well. Google Guava has a ListenableFuture that allows callback handling to be added via the returned future, and allows the callbacks to be passed

Re: Producer message ordering problem

he time (I have not yet played with Trident stuff > myself, but Storm in general, yes). Coupling Storm to Kafka is a very > popular thing to do. Even without Trident, and just using Storm in a > simpler mode, may save you from writing a ton of code. > > Philip > > On Thu, Aug 22, 2

Re: Producer message ordering problem

make produce calls > idempotent, enforce strong ordering in the case of retries, as well as fix > a number of other corner cases. I think it would handle this issue as well. > But it's not a quick patch. > > I will try to get a design proposal up by next week so we have something

Re: Producer message ordering problem

der > so tightly? Maybe there is another way to meet your needs instead of > relying on Kafka to do it. > > Philip > > On Aug 22, 2013, at 9:32 PM, Ross Black wrote: > > > Hi, > > > > I am using Kafka 0.7.1, and using the low-level SyncProducer to send >

Producer message ordering problem

Hi, I am using Kafka 0.7.1, and using the low-level SyncProducer to send messages to a *single* partition from a *single* thread. The client sends messages that contain sequential numbers so it is obvious at the consumer when message order is shuffled. I have noticed that messages can be saved out

Re: message ordering guarantees

Thanks for the explanation. Ross On 23 May 2013 07:19, Neha Narkhede wrote: > Thanks, > Neha > On May 21, 2013 5:42 PM, "Ross Black" wrote: > > > Hi, > > > > I am using Kafka 0.7.1, and using SyncProducer and SimpleConsumer with a > > single bro

Re: message ordering guarantees

in a topic. > > Thanks, > > Jun > > > On Tue, May 21, 2013 at 5:42 PM, Ross Black > wrote: > > > Hi, > > > > I am using Kafka 0.7.1, and using SyncProducer and SimpleConsumer with a > > single broker service process. > > > > I am occa

message ordering guarantees

Hi, I am using Kafka 0.7.1, and using SyncProducer and SimpleConsumer with a single broker service process. I am occasionally seeing messages (from a *single* partition) being processed out of order to what I expect and I am trying to find where the problem lies. The problem may well be in my co

Re: Uncompress / re-compress of messages in the message server

ks, > Neha > > On Monday, March 18, 2013, Ross Black wrote: > > > Hi, > > > > I have just started looking at moving from 0.7 to 0.8 and wanted to > confirm > > my understanding of code in the message server/broker. > > > > In the code for 0.8, KafkaA

Uncompress / re-compress of messages in the message server

Hi, I have just started looking at moving from 0.7 to 0.8 and wanted to confirm my understanding of code in the message server/broker. In the code for 0.8, KafkaApis.appendToLocalLog calls log.append(..., assignOffsets = true), which then calls ByteBufferMessageSet.assignOffsets. This method seem