Re: Kafka Monitoring

2017-12-12 Thread Irtiza Ali
Ok thank you Michal On Tue, Dec 12, 2017 at 9:30 PM, Michal Michalski < michal.michal...@zalando.ie> wrote: > Hi Irtiza, > > Unfortunately I don't know what could be the issue in your case - in my > case all these metrics simply appear under appropriate keys > (e.g. "kafka.controller") in the JSO

Re: Strange rebalancing exception in Kafka 1.0.0

2017-12-12 Thread Sameer Kumar
Hi All, Any pointers to the above issue that I can explore further. -Sameer. On Tue, Dec 12, 2017 at 5:40 PM, Sameer Kumar wrote: > HI Ismael, > > This is what I see in the logs, I tried this twice and got the same > exception. > > This is only reproducible when I have 3 nodes in my cluster ev

Using Bootstrap Server List for data center failover

2017-12-12 Thread Behrang Saeedzadeh
Hi, If we have two data centers, one with a passive Kafka cluster and one with an active Kafka cluster, would listing the Kafka servers for both clusters as the bootstrap servers for our consumer apps cause any trouble? We were thinking about doing this to avoid the need to reconfigure out consum

Connectors Information from Microsoft SQL Server to Kafka

2017-12-12 Thread harish reddy m
Hi Team, We have a requirement of replicating the data from MSSQL Source database to MSSQL target database. We are using SQL Server 2014 Web edition as the source database and target database as well. We want to replicate the data from Source database to target database realtime. Enabled the cha

Re: ConcurrentModificationException

2017-12-12 Thread Guozhang Wang
Since you're running Storm to read data from Kafka, I'd expect it is not due to 6349. Could you provide a more detailed stack trace? The current provided one is really small and not informative at all. Guozhang On Tue, Dec 12, 2017 at 6:42 PM, Ted Yu wrote: > From the stack trace, it seems you

Re: ConcurrentModificationException

2017-12-12 Thread Ted Yu
>From the stack trace, it seems you hit KAFKA-6349 (where Damian provided a PR). FYI On Mon, Dec 11, 2017 at 11:29 PM, Mr.Wang <1282183...@qq.com> wrote: > Hi~ > We found this error when we read kafka data using storm: > java.util.ConcurrentModificationException at > java.util.LinkedHash

ConcurrentModificationException

2017-12-12 Thread Mr.Wang
Hi~ We found this error when we read kafka data using storm: java.util.ConcurrentModificationException at java.util.LinkedHashMap$LinkedHashIterator.nextNode(LinkedHashMap.java:719) at java.util.LinkedHashMap$LinkedKeyIterator.next(LinkedHashMap.java:742) at java.util.Abs

Re: Why is compression disabled by default?

2017-12-12 Thread Dmitry Minkovsky
Thank you Ismael On Tue, Dec 12, 2017 at 11:49 AM, Ismael Juma wrote: > Yes, the broker will store the batches it receives from the producer. By > default, it won't recompress, but it can be configured to recompress. > Either way, it won't merge multiple batches into a single one. > > Ismael > >

Re: Why is compression disabled by default?

2017-12-12 Thread Ismael Juma
Yes, the broker will store the batches it receives from the producer. By default, it won't recompress, but it can be configured to recompress. Either way, it won't merge multiple batches into a single one. Ismael On Tue, Dec 12, 2017 at 6:45 PM, Dmitry Minkovsky wrote: > I would like to follow

Re: Why is compression disabled by default?

2017-12-12 Thread Dmitry Minkovsky
I would like to follow up with this more concrete question: For the purpose of achieving decent compression ratios, I am having difficulty finding information on whether the broker can perform additional record batching beyond the batching received from a producer. It seems like in a low-latency a

Re: Kafka Monitoring

2017-12-12 Thread Michal Michalski
Hi Irtiza, Unfortunately I don't know what could be the issue in your case - in my case all these metrics simply appear under appropriate keys (e.g. "kafka.controller") in the JSON response of the /list request - the same as the one you're making (I checked it right now to be sure). I also checked

Re: Strange rebalancing exception in Kafka 1.0.0

2017-12-12 Thread Sameer Kumar
HI Ismael, This is what I see in the logs, I tried this twice and got the same exception. This is only reproducible when I have 3 nodes in my cluster even in the start of the app, everything works fine on 2 nodes. [2017-12-12 17:28:36,822] INFO [GroupCoordinator 90]: Preparing to rebalance group

Re: Strange rebalancing exception in Kafka 1.0.0

2017-12-12 Thread Ismael Juma
Can you please check the broker logs for errors? Ismael On Tue, Dec 12, 2017 at 12:10 PM, Sameer Kumar wrote: > Hi All, > > Facing an strange exception while running Kafka Streams. I am reading from > a topic of 60 partitions. I am using exactly once in Kafka 1.0.0. > > Now, this error has star

Strange rebalancing exception in Kafka 1.0.0

2017-12-12 Thread Sameer Kumar
Hi All, Facing an strange exception while running Kafka Streams. I am reading from a topic of 60 partitions. I am using exactly once in Kafka 1.0.0. Now, this error has started appearing recently. The application runs fine on 2 nodes, but as soon as a 3rd node is added, it starts throwing excepti