Re: Potential socket leak in kafka sync producer

Looking at that heap dump, this probably is a database connection/resource leak (298 connections?) than anything to do with Kafka. Have you investigated if there's any DB resource leak in the application and ruled out that part? -Jaikiran On Friday 30 January 2015 01:08 PM, ankit tyagi wrote:

Using Only one partition to store object change log

HI Group: I'm currently working to push object changes into external services. Now we are trying to append the change record into a Kafka. **My problem** Only one partition can be created for one kind of data, so that the sequence of change can be guaranteed. If I do like that, I guess I will lo

Re: Potential socket leak in kafka sync producer

Jaikiran, I have already investigated that this is kafka related. I made a small application which is used only for publishing messages to kafka. If I use dynamic thread pool means where number of maxPoolSize is very large comparative to corePoolSize and I publish each batch of messages only aft

RE: per-topic configuration names... unit suffix ?

Well, in lack of a response, I rolled up my sleeves and took to the code. It appears that these ARE indeed in Milliseconds! So, while you set the log.retention.minutes in increments of minutes, you must set the topic-level configuration in milliseconds. Pretty odd. Though it does appear that yo

Re: [VOTE] 0.8.2.0 Candidate 3

This is a reminder that the vote will close tomorrow night. Please test RC3 out and vote before the deadline. Thanks, Jun On Wed, Jan 28, 2015 at 11:22 PM, Jun Rao wrote: > This is the third candidate for release of Apache Kafka 0.8.2.0. > > Release Notes for the 0.8.2.0 release > > https://pe

java.nio.channels.ClosedChannelException

Hi, All I send messages from one VM to production, but getting such error [2015-01-30 18:43:44,810] WARN Failed to send producer request with correlation id 126 to broker 101 with data for partitions [test-rep-three,5],[test-rep-three,2] (kafka.producer.async.DefaultEventHandler) java.nio.channel

Re: Resilient Producer

Fernando, have a look - http://blog.sematext.com/2014/10/06/top-5-most-popular-log-shippers/ Otis -- Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr & Elasticsearch Support * http://sematext.com/ On Wed, Jan 28, 2015 at 1:39 PM, Fernando O. wrote: > Hi all, > I'

KafkaWordCount

Hi Guys, I would like to put in the kafkawordcount scala code the kafka parameter: val kafkaParams = Map(“fetch.message.max.bytes” -> “400”). I’ve put this variable like this val KafkaDStreams = (1 to numStreams) map {_ =>

Re: Using Only one partition to store object change log

do you need total ordering among all events? or you just need ordering by some partitionKey (e.g. events regarding one particular database key or user id)? if it's the later, you can create multiple partitions and just partition your events using the key to different kafka partitions. On Fri, Jan

Re: One or multiple instances of MM to aggregate kafka data to one hadoop

Really appreciate you guys' recommendations. On Thu, Jan 29, 2015 at 9:22 AM, Jon Bringhurst < jbringhu...@linkedin.com.invalid> wrote: > Hey Mingjie, > > Here's how we have our mirror makers configured. For some context, let me > try to describe this using the example datacenter layout as descri

Re: [kafka-clients] Re: [VOTE] 0.8.2.0 Candidate 3

+1 (binding) verified signatures, ran quick start, tests all passed. - Joe Stein On Fri, Jan 30, 2015 at 12:04 PM, Jun Rao wrote: > This is a reminder that the vote will close tomorrow night. Please test > RC3 out and vote before the deadline. > > Thanks, > > Jun > > On Wed, Jan 28, 2015 at 11

Re: Potential socket leak in kafka sync producer

Hi Ankit, Would you be able to share the trimmed down application code which reproduces this issue (maybe as a repo on github)? That way, some of us will get more context about the issue and probably be able to try it out ourselves and see what's wrong. On a related note, have you tried this