Re: Improving Kafka State Store performance

In addition to the measurements that you are doing yourself, Kafka Streams also has its own metrics. They are exposed via JMX, if you have that enabled: http://kafka.apache.org/documentation/#monitoring If you set metrics.recording.level="debu

Re: Improving Kafka State Store performance

Hi Bill. Thank you pointing out, But in actual code I am calling iter.close() in the finally block if the iterator is not null. I don't see any issues when I am running it on light traffic. As soon as I switch to production traffic I start seeing these issues. Below I have provided additional det

Re: Kafka Connect Sink Connector for multiple JDBC sinks

Hi, entirely depends on how you want to serialize. You should be able to get everything running on Windows anyhow. Nothing expect the broker is really extensively using OS support for operating. To answer your initial question: You would simply start multiple sinks and give each sink a diffe

Re: Kafka Streams application Unable to Horizontally scale and the application on other instances refusing to start.

I searched for 'stream-thread failed to' in StreamThread100.log but didn't find any occurrence: StreamThread100.log:org.apache.kafka.streams.errors.StreamsException: stream-thread failed to suspend stream tasks Can you double check ? On Fri, Sep 15, 2017 at 8:09 PM, dev loper wrote: > Hi All ,

Re: Kafka Connect Sink Connector for multiple JDBC sinks

Yes I have, I do need to build and run Schema Registry as a pre-requisite isn't that correct? because the QuickStart seems to start AVRO - without AVRO you need your own implementation of transformer/serdes etc. I am only asking since my deployment platform is Windows Server 2012 - and Confluent p

Re: Kafka Connect Sink Connector for multiple JDBC sinks

Have you looked at https://github.com/confluentinc/kafka-connect-jdbc ? On Sat, Sep 16, 2017 at 1:39 PM, M. Manna wrote: > Sure. But all these are not available via Kafka open source (requires > manual coding), correct? Only Confluence seems to provide some > off-the-shelf connector but Confluen

Re: Kafka Connect Sink Connector for multiple JDBC sinks

Sure. But all these are not available via Kafka open source (requires manual coding), correct? Only Confluence seems to provide some off-the-shelf connector but Confluent isn't compatible on Windows (yet), also correct? On 13 September 2017 at 18:11, Sreejith S wrote: > This is possible. Once

Re: Improving Kafka State Store performance

Hi, It's hard to say exactly without a little more information. On a side note, I don't see where you are closing the KeyValueIterator in the code above. Not closing a KeyValueIterator on a Permanent State Store can cause a resource leak over time, so I'd add `iter.close()` right before your `log

Re: Kafka Streams application Unable to Horizontally scale and the application on other instances refusing to start.

If you were asking whether Kafka brokers should be upgraded, I would say yes. See also https://archive.apache.org/dist/kafka/0.11.0.1/RELEASE_NOTES.html On Sat, Sep 16, 2017 at 8:21 AM, dev loper wrote: > Hi Ted, > > I will try out 0.11.0.1 version. Is there any upgrade which needs to be > don

Re: Kafka Streams application Unable to Horizontally scale and the application on other instances refusing to start.

Hi Ted, I will try out 0.11.0.1 version. Is there any upgrade which needs to be done on kafka broker side as well ? On Sat, Sep 16, 2017 at 5:18 AM, Ted Yu wrote: > Hi, > Were you using 0.11.0.0 ? > > I ask this because some related fixes, KAFKA-5167 and KAFKA-5152, are only > in 0.11.0.1 > >

Re: Improving Kafka State Store performance

Hi Ted, I am using 0.11.0.0 . I am not using external state store. I am using the persistent store that comes with kafka stream 0.11.0.0. My assumption is put just inserts my object into the state store. I guess the state store is manged by rocksdb internally by kafkastreams. I am not sure why

Re: Kafka Streams application Unable to Horizontally scale and the application on other instances refusing to start.

Hi, Were you using 0.11.0.0 ? I ask this because some related fixes, KAFKA-5167 and KAFKA-5152, are only in 0.11.0.1 Mind trying out 0.11.0.1 release and see whether the problem persists ? On Fri, Sep 15, 2017 at 12:52 PM, Ted Yu wrote: > bq. 1) Reduced MAX_POLL_RECORDS_CONFIG to 5000 (prev

Re: Improving Kafka State Store performance

Looking at the calculation of totalTimeTakenToStoreRecords, it covers store.put() call. Can you tell us more about what the put() does ? Does it involve external key value store ? Are you using 0.11.0.0 ? Thanks On Sat, Sep 16, 2017 at 6:14 AM, dev loper wrote: > Hi Kafka Streams Users, > > I

Improving Kafka State Store performance

Hi Kafka Streams Users, I am trying to improve the performance of Kafka Streams State Store Persistent Store. In our application we are using Kafka Streams Processor API and using Persistent State Store.. My application when starts up it performing well but over a period of time the performance d

Re: Kafka Streams application Unable to Horizontally scale and the application on other instances refusing to start.

Hi All, Sometimes I used to see below exception before Consumer getting disconnected. I am not sure whether it has any relevance with this issue I am facing. DEBUG | 05:16:24 | clients.NetworkClient (NetworkClient.java:702) - Node 0 disconnected. DEBUG | 05:16:24 | clients.NetworkClient (Network

Re: Kafka Streams application Unable to Horizontally scale and the application on other instances refusing to start.

Thank you Ted for your inputs. I have also observed the similarity in logs. But I am not sure how to resolve the issue. @Bill, I repeated my tests by reducing the max MAX_POLL_RECORDS to 500. I could see the consumer getting started with 500. But the same problem exits even after the configuration