Hi Jaikiran,
I am using ubuntu and was able to reproduce on redhat too. Please find the
more information below.
*DISTRIB_ID=Ubuntu*
*DISTRIB_RELEASE=12.04*
*DISTRIB_CODENAME=precise*
*DISTRIB_DESCRIPTION="Ubuntu 12.04.5 LTS"*
*java version "1.7.0_72"*
This is happening on client side. Output of lsof was showing that maximum
fd were FIFO and anon. But after GC FD count was reduced significantly.
Below is my Client Code which i am using for publishing message.
* private Producer<KafkaPartitionKey, KafkaEventWrapper> myProducer;*
* myProducer = new Producer<>(new
ProducerConfig(myProducerProperties));*
* public void send(*
* List<KeyedMessage<KafkaPartitionKey, KafkaEventWrapper>> msgs)*
* {*
* myProducer.send(msgs);*
* }*
we are using sync producer. I am attaching object histo before GC(histo_1)
and after GC(histo_2) in my application.
On Fri, Jan 30, 2015 at 9:34 AM, Jaikiran Pai <jai.forums2...@gmail.com>
wrote:
>
> Which operating system are you on and what Java version? Depending on the
> OS, you could get tools (like lsof) to show which file descriptors are
> being held on to. Is it the client JVM which ends up with these leaks?
>
> Also, would it be possible to post a snippet of your application code
> which shows how you are using the Kafka APIs?
>
> -Jaikiran
> On Thursday 29 January 2015 04:36 PM, ankit tyagi wrote:
>
>> Hi,
>>
>> Currently we are using sync producer client of 0.8.1 version in our
>> production box . we are getting the following exception while publishing
>> kafka message
>>
>> *[2015-01-29
>> 13:21:45.505][ThreadPoolTaskExecutor-603][WARN][ClientUtils$:89] Fetching
>> topic metadata with correlation id 10808 for topics [Set(*
>> *kafka_topic_coms_FD_test1)] from broker [id:0,host:localhost,port:9092]
>> failed*
>> *java.net.ConnectException: Connection refused*
>> * at sun.nio.ch.Net.connect0(Native Method)*
>> * at sun.nio.ch.Net.connect(Net.java:465)*
>> * at sun.nio.ch.Net.connect(Net.java:457)*
>> * at
>> sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670)*
>> at kafka.network.BlockingChannel.connect(BlockingChannel.scala:
>> 57)
>> at kafka.producer.SyncProducer.connect(SyncProducer.scala:141)
>> at
>> kafka.producer.SyncProducer.getOrMakeConnection(SyncProducer.scala:156)
>> at
>> kafka.producer.SyncProducer.kafka$producer$SyncProducer$$
>> doSend(SyncProducer.scala:68)
>> at kafka.producer.SyncProducer.send(SyncProducer.scala:112)
>> at
>> kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:53)
>> at
>> kafka.producer.BrokerPartitionInfo.updateInfo(
>> BrokerPartitionInfo.scala:82)
>>
>>
>> we are using dynamic thread pool to publish message to kafka. My
>> observation is when after keep alive time when threads in my executor gets
>> destroyed, somehow file descriptor is not getting cleared but when i did
>> explicitly ran the full gc, fd count got reduced by a signification amout.
>>
>>
>
