Possible to Disable Offset Checkpointing for an In-Memory Global Store?

2018-04-17 Thread David Chu
I have a custom in-memory state store which I’d like to configure as a global store for my Kafka Streams application. However, even though my state store is not persistent (i.e. the implementation of StateStore.persistent() returns false) it appears that the GlobalStateManager still continues t

Producer#commitTransaction() Not Being Called if New Records Aren't Processed by StreamTask

2018-05-10 Thread David Chu
I have a simple Kafka Streams topology of the format “Source Topic -> Processor -> Sink Topic” that is configured to use exactly-once processing guarantees. The Processor performs a reduce operation on the incoming messages and stores the results to a key-value state store (with logging enabled

Kafka Streams Application Failing to Start Due to State Store Recovery Time Exceeding Producer Transaction Timeout

2018-07-09 Thread David Chu
I have a Kafka Streams application which is currently failing to start due to the following ProducerFencedException: "Caused by: org.apache.kafka.common.errors.ProducerFencedException: task [0_57] Abort sending since producer got fenced with a previous record (key ABCD value [B@4debf146 timesta

Re: Kafka Streams Application Failing to Start Due to State Store Recovery Time Exceeding Producer Transaction Timeout

2018-07-10 Thread David Chu
t 9:38 AM, Matthias J. Sax wrote: > > Can it be, that you hit: https://issues.apache.org/jira/browse/KAFKA-6634 > > -Matthias > > On 7/9/18 7:58 PM, David Chu wrote: >> I have a Kafka Streams application which is currently failing to start due >> to the following Pro

Frequent SIGSEGV Errors on Kafka Brokers

2019-01-10 Thread David Chu
We are running Kafka 2.0.1 and lately the brokers have been crashing with several different SIGSEGV errors as listed below. I was wondering if anybody has encountered similar errors or might have advice on what the problem could be? Thanks, David 1. # # A fatal error has been detected by the