IllegalStateException :Tried to lookup lag for unknown task 1_0

Hi Team, I am facing error Exception in thread “ topic_name-0ce8394a-2394-4919-e46ce630c6ad-StreamThread-4” Java.lang.IllegalStateException :Tried to lookup lag for unknown task 1_0. Please suggest a solution. Thanks, Hima

IBM MQ Connector capacity planning

Hello, Can anyone share their experiences if they have use cases ? I am specifically looking for these and any other details you may have. 1. How did you measure the throughput when sourcing MQ messages ? 2. Did you configure some type of buffering in MQ ? What happens if the conne

Kafka Streams aggregation cost

size of the state that applications deal with today ? Are there any guidelines available ? Having some standard benchmark numbers published would help with making such decisions. Thanks Mohan

Re: Kafka Stream: State replication seems unpredictable.

Guozhang, Thanks for the clarification. It makes sense. As long as the output hits the broker, zombies can be detected. -mohan On 5/3/21, 2:17 PM, "Guozhang Wang" wrote: Hello Mohan, Sorry for getting late on the thread. Just to revive your concerns here: if

Re: Spark Streams vs Kafka Streams

Matthias, I will create a KIP or ticket for tracking this issue. -thanks Mohan On 4/28/21, 1:01 PM, "Matthias J. Sax" wrote: Feel free to do a KIP and contribute to Kafka! https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals O

Re: Spark Streams vs Kafka Streams

Andrew, I am not sure I understand. We have built several analytics applications. We typically use custom aggregations as they are not available directly in the library. -mohan On 4/28/21, 12:12 PM, "Andrew Otto" wrote: I'd assume this is because Kafka Streams is

Re: Spark Streams vs Kafka Streams

it very easy. Thanks Mohan On 4/28/21, 12:00 PM, "Matthias J. Sax" wrote: I am not familiar with all the details about Spark, however, the link you shared is for Spark SQL. I thought Spark SQL is for batch processing only? Personally, I would be open to add mor

Spark Streams vs Kafka Streams

library ? Thanks Mohan

Re: Kafka Stream: State replication seems unpredictable.

changelog topic can ever be disabled. Thanks Mohan On 4/21/21, 10:22 PM, "Guozhang Wang" wrote: Hello Mangat, I think using persistent store that relies on in-memory stores could help if the threads are from the same instance. Guozhang On Tue, Ap

Re: Redis as state store

Thanks for the responses. In the worst case, I might have to keep both rocksdb for local store and keep an external store like Redis. -mohan On 3/13/21, 8:53 PM, "Pushkar Deole" wrote: Another issue with 3rd party state stores could be violation of exactly-once guarantee p

Redis as state store

anyone tried this ? Any caveats. Thanks Mohan

Re: Is this a valid use case for reading local store ?

Thanks. I was curious about the other real world use cases i.e what do people use it for ? Is this widely used or mostly for debugging purposes ? Any caveats ? Thanks Mohan On 10/1/20, 5:55 PM, "Guozhang Wang" wrote: Mohan, I think you can build a REST API on t

Re: Is this a valid use case for reading local store ?

people do if a kafka streams application (App1) has to offer REST interface also ? -thanks Mohan On 9/30/20, 5:01 PM, "Guozhang Wang" wrote: Hello Mohan, If I understand correctly, your async event trigger process runs out of the streams application, that reads

Is this a valid use case for reading local store ?

first place. Are there any caveats doing it this way ? If not, any other suggestions ? Thanks Mohan

Re: Event time semantics across windows

any guidelines on what the associated overhead is with the grace period and is it reasonable for it to be in hours ? I guess there is state in memory, rocksdb, Anything else to worry about ? Thanks Mohan On 9/25/19, 11:25 PM, "Matthias J. Sax" wrote: Time is currently t

Event time semantics across windows

“groupBy”. If the event_time was associated with the window (where there is one per “client_id”), then it would have worked well in this case. Is there any reason for the current design ? Is there any way to solve this problem ? Thanks Mohan

Re: Can kafka internal state be purged ?

. Let me know how I can help. Thanks -mohan On 6/28/19, 1:21 PM, "John Roesler" wrote: Ok, good, that's what I was hoping. I think that's a good strategy, at the end of the "real" data, just write a dummy record with the same keys with a high timest

Re: Can kafka internal state be purged ?

flush. How does one solve this issue in production ? I am wondering why the design did not accommodate a timer to flush the windows ? Thanks Mohan On 6/26/19, 8:18 AM, "John Roesler" wrote: Hi Mohan, I see where you're going with this, and it might indeed be a c

Re: Can kafka internal state be purged ?

in but post-map operation, the partitions in the repartitioned topic does not see the same thing. Thanks Mohan On 6/24/19, 7:49 AM, "John Roesler" wrote: Hey, this is a very apt question. GroupByKey isn't a great example because it doesn't actually change

Re: Can kafka internal state be purged ?

across all the application instances ? Is there a designated instance for a particular key ? In my case, there was only one instance processing records from all partitions and it is kind of odd that windows did not expire even though I understand why now. Thanks Mohan On 6/21/19, 2:25 PM

Re: Can kafka internal state be purged ?

That change "In the same partition" must explain what we are seeing. Unless you see one message per partition, all windows will not expire. That is an interesting twist. Thanks for the correction ( I will go back and confirm this. -mohan On 6/21/19, 12:40 PM, "John Roesler&quo

Re: Can kafka internal state be purged ?

ime and not on the key Are these two statements correct ? Thanks Mohan On 6/20/19, 9:17 AM, "John Roesler" wrote: Hi! In addition to setting the grace period to zero (or some small number), you should also consider the delays introduced by record caches up

Re: Can kafka internal state be purged ?

We do explicitly set the grace period to zero. I am going to try the new version -mohan On 6/19/19, 12:50 PM, "Parthasarathy, Mohan" wrote: Thanks. We will give it a shot. On 6/19/19, 12:42 PM, "Bruno Cadonna" wrote: Hi Mohan,

Re: Can kafka internal state be purged ?

Thanks. We will give it a shot. On 6/19/19, 12:42 PM, "Bruno Cadonna" wrote: Hi Mohan, I realized that my previous statement was not clear. With a grace period of 12 hour, suppress would wait for late events until stream time has advanced 12 hours before a resul

Re: Can kafka internal state be purged ?

No, I have not set any grace period. Is that mandatory ? Have you seen problems with suppress and windows expiring ? Thanks Mohan On 6/19/19, 12:41 AM, "Bruno Cadonna" wrote: Hi Mohan, Did you set a grace period on the window? Best, Bruno On T

Re: Can kafka internal state be purged ?

t there an order in the way windows are closed ? As event time progresses by the new messages arriving, the older ones should expire. Is that right understanding or not ? Thanks Mohan On 6/17/19, 3:43 PM, "Parthasarathy, Mohan" wrote: Hi, We are using suppress in the appl

Can kafka internal state be purged ?

next stage. It is as though the state created earlier was purged. Is this possible ? Thanks Mohan

Retrieve open windows

Hi, Is there a way to know about the currently open windows that is currently not expired yet ? We are trying to track down a problem where some messages are read by the application but then we don’t see the actual processing of the message by the application. Thanks Mohan

Re: Kafka streams in Kubernetes

ompaction affects the rebuilding time ? Sorry, I am not sure I understand what you meant by this in the current context. -mohan On 6/10/19, 10:22 AM, "Parthasarathy, Mohan" wrote: Thanks. That helps me understand why recreating state might take time. -mohan

Re: Kafka streams in Kubernetes

Thanks. That helps me understand why recreating state might take time. -mohan On 6/9/19, 11:50 PM, "Matthias J. Sax" wrote: By default, Kafka Streams does not "close" windows. To handle out-of-order data, windows are maintained until their retention

Re: Kafka streams in Kubernetes

Pavel, Thanks for the pointer. I will take a look. -mohan On 6/8/19, 4:29 PM, "Pavel Sapozhnikov" wrote: I suggest take a look at Strimzi project https://strimzi.io/ Kafka operator deployed in Kubernetes environment. On Sat, Jun 8, 2019, 6:09 PM Parthasara

Re: Kafka streams in Kubernetes

the whole store retention time ? Could you clarify ? Thanks Mohan On 6/8/19, 11:18 PM, "Matthias J. Sax" wrote: If depends how much state you need to restore and how much restore-time you can accept in your application. The amount of data that needs to be restore

Kafka streams in Kubernetes

that it was handling before. I had an instance of such a restart where it was running a long time in REBALANCE which makes me think that my understanding is incorrect. In this case, the state store was available during the restart. Can someone clarify ? Thanks Mohan

Re: Streams reprocessing whole topic when deployed but not locally

I am a little confused by what you say. I can see how it has to build the state when it is not available on restart but i don’t think it will process old messages from input topics. It should start from the last committed offset whatever that is before the crash. Could you confirm ? I thought th

Re: RecordCollectorImpl: task [1_1] Error sending records

? It seems to be okay but not sure about what we are missing From: Bruno Cadonna Sent: Tuesday, June 4, 2019 11:53 PM To: users@kafka.apache.org Subject: Re: RecordCollectorImpl: task [1_1] Error sending records Hi Mohan, Could you post the log messages you see

RecordCollectorImpl: task [1_1] Error sending records

amount of binary data. How can I turn this off ? Thanks Mohan

What happens in Rebalancing state ?

Hi, When applications are restarted, I see it go into rebalancing state. Is it trying to recreate state from the state store files ? I am seeing an application stuck for a long time and wondering how to debug the application in this state ? Thanks Mohan

Re: Multiple state store files

Thanks. Unfortunately, this leads to lot of open file descriptors in the application. Anyone else faced that situation ? I will increase the limit for now. Thanks Mohan On 5/16/19, 5:18 AM, "Matthias J. Sax" wrote: This is how RocksDB works, and it's normal.

Multiple state store files

Hi, I am seeing multiple state store files (.sst) created under each directory. Is this normal ? If an application crashes and comes back, would it cause this ? Thanks Mohan

Intermittent NoLeaderForPartition exceptions

;:"host1:4443,host2:4443,host3:4443,host4:4443,host5:4443"} Could you please comment on what could be causing the issue? -- Atul Mohan

Reg: Read External API Response Kafka Producer

Hi, I am investigating Kafka as a message bus service for my MANO Monitoring module. I have a python producer that will take the message from a file and will put it in the message bus in json format. I have a consumer that reads the message and does some operation in the external OpenStack API.

Re: Mirror maker Configs 0.9.0

Hello Experts, I am trying to replicate data between On Prem Kafka Cluster(Source) and another Kafka cluster (Target?) set up in a Cloud provider environment. The On-Prem to cloud is connected via IPSec VPN and Mirror maker tool is used on 0.10.2.x version. The consumer configs are as fo

Kafka and spark integration

Any body know the end to end hadoop data flow which has Kafka - spark integration. I am primarily concerned on how kafka messages written to kafka partition gets captured into files?