Re: Potential socket leak in kafka sync producer

attaching my producer whole code. Creating kafkaProducer Bean in xml * * ** ** ** *${KAFKA_PRODUCER_BROKER_LIST}* ** ** ** public class KafkaProducer { private static final Logger LOG = LoggerFactory.getLogger(Kafk

Re: Potential socket leak in kafka sync producer

I have shared object histogram after and before gc on gist https://gist.github.com/ankit1987/f4a04a1350fdd609096d On Fri, Jan 30, 2015 at 12:43 PM, Jaikiran Pai wrote: > What kind of a (managed) component is that which has the @PreDestroy? > Looking at the previous snippet you added, it looks li

Re: Potential socket leak in kafka sync producer

What kind of a (managed) component is that which has the @PreDestroy? Looking at the previous snippet you added, it looks like you are creating the Producer in some method? If you are going to close the producer in a @PreDestroy of the component, then you should be creating the producer in the

Re: Potential socket leak in kafka sync producer

Hi, I am closing my producer at the time of shutting down my application. @PreDestroy public void stop() { LOG.info("Stopping Kafka Producer for topic: {}", myTopic); if (myProducer != null) { myProducer.close(); } } On Fri, Jan 30, 2015 at 11:22

Re: Potential socket leak in kafka sync producer

Hope you are closing the producers. can you share the attachment through gist/patebin On Fri, Jan 30, 2015 at 11:11 AM, ankit tyagi wrote: > Hi Jaikiran, > > I am using ubuntu and was able to reproduce on redhat too. Please find the > more information below. > > > *DISTRIB_ID=Ubuntu* > *DISTRIB_

Re: Potential socket leak in kafka sync producer

Hi Jaikiran, I am using ubuntu and was able to reproduce on redhat too. Please find the more information below. *DISTRIB_ID=Ubuntu* *DISTRIB_RELEASE=12.04* *DISTRIB_CODENAME=precise* *DISTRIB_DESCRIPTION="Ubuntu 12.04.5 LTS"* *java version "1.7.0_72"* This is happening on client side. Output o

Re: Potential socket leak in kafka sync producer

Which operating system are you on and what Java version? Depending on the OS, you could get tools (like lsof) to show which file descriptors are being held on to. Is it the client JVM which ends up with these leaks? Also, would it be possible to post a snippet of your application code which

Question on ETL while replau

What's the best way to add two 'fields' to my kafka messages once they are stored? Can I just do a replay on all of them and add the field? How would I throw out the old kafka messages that don't have the field then? I am using Druid to process the data, but not sure of how to propagate data cha

Re: [VOTE] 0.8.2.0 Candidate 3

I think we were just confirming whether that issue was fixed in 0.8.2 not. Given that this issue only happens in unclean shutdown, I don't think it's a blocker for 0.8.2. Also, the patch is not trivial and it's better to test it out a bit longer in trunk. Thanks, Jun On Thu, Jan 29, 2015 at 5:36

Re: [VOTE] 0.8.2.0 Candidate 3

In meetup we said that KAFKA-1650 and follow up patches is included in 0.8.2, but it seems not on the list. On 1/29/15, 1:01 AM, "Magnus Edenhill" wrote: >+1 on librdkafka interop > >Minor nitpick: > KAFKA-1781 (state required gradle version in README) is included in the >Release notes but is

RE: LinkedIn Engineering Blog Post - Current and Future

Yep, thanks guys. -Original Message- From: Aditya Auradkar [mailto:aaurad...@linkedin.com.INVALID] Sent: Thursday, January 29, 2015 2:46 PM To: users@kafka.apache.org Subject: RE: LinkedIn Engineering Blog Post - Current and Future This should work. http://engineering.linkedin.com/kafka

RE: LinkedIn Engineering Blog Post - Current and Future

This should work. http://engineering.linkedin.com/kafka/kafka-linkedin-current-and-future Aditya From: Thunder Stumpges [tstump...@ntent.com] Sent: Thursday, January 29, 2015 2:43 PM To: users@kafka.apache.org Subject: RE: LinkedIn Engineering Blog Post -

Re: LinkedIn Engineering Blog Post - Current and Future

https://engineering.linkedin.com/kafka/kafka-linkedin-current-and-future On Thu, Jan 29, 2015 at 5:43 PM, Thunder Stumpges wrote: > Thanks Jon. That link isn't working for me though: > > "Page not found > > The requested page "/kafka/kafka-linkedin-%E2%80%93-current-and-future" > could not be fo

RE: LinkedIn Engineering Blog Post - Current and Future

Thanks Jon. That link isn't working for me though: "Page not found The requested page "/kafka/kafka-linkedin-%E2%80%93-current-and-future" could not be found." -Original Message- From: Jon Bringhurst [mailto:jbringhu...@linkedin.com.INVALID] Sent: Thursday, January 29, 2015 10:23 AM To

per-topic configuration names... unit suffix ?

Hi all, I am reading about the per-topic configs and the unit suffixes on a couple don't seem to match to the global/server ones. Specifically: retention.ms => log.retention.minutes segment.ms => log.roll.hours Can someone clear this up

Re: Errors from ReassignPartitionsCommand

Each broker should have a controller log, and at one period of time only one of them will host the controller, while others' controller logs will be almost empty. If you found some entries like "controller start-up" one multiple controllers or if more than one controller log has large amount of dat

Kafka ETL Camus Question

Hi Kafka Team or Linked-In Team, I would like to know if you guys run Camus ETL job with speculative execution true or false. Does it make sense to set this to false ? Having true, it creates additional load on brokers for each map task (create a map task to pull same partition twice). Is there

Re: Errors from ReassignPartitionsCommand

We are using 0.8.1.1. How do we identify controller migration? Is it in logs or some metrics? Allen On Tue, Jan 27, 2015 at 9:35 AM, Guozhang Wang wrote: > Allen, which version of Kafka are you using? And if you have multiple > brokers, is there a controller migration happened before? > > Guoz

Re: Consuming Kafka Messages Inside of EC2 Instances

Thanks Jonathon! Ahh got it, I connected to a different port. Also it just happened to be that the CM zookeeper was running on the one of 10 instances that I could have manually started zookeeper on -_-. Thanks for the help! On Thu, Jan 29, 2015 at 12:51 PM, Jonathan Natkins wrote: > Responses

Re: Consuming Kafka Messages Inside of EC2 Instances

Responses inline Jonathan "Natty" Natkins StreamSets | Customer Engagement Engineer mobile: 609.577.1600 | linkedin On Thu, Jan 29, 2015 at 12:47 PM, Su She wrote: > Thanks Jonathon! > > 1) How can I get the whole command string? What i c/p was the whole o

Re: Consuming Kafka Messages Inside of EC2 Instances

Thanks Jonathon! 1) How can I get the whole command string? What i c/p was the whole output 2) I do have Zookeeper as a service from Cloudera Manager, except I used to have this before as well and was never a problem. Is there a way I can keep the Zookeeper from CM as well as manually start it up

Re: Consuming Kafka Messages Inside of EC2 Instances

Hey Su, It's hard to say, because you didn't copy the entire command string, but it looks like it's probably a process that's being managed by Cloudera Manager. Do you have a Cloudera Manager instance that is running a Zookeeper quorum? If so, that's where the conflict is. You can either reconfigu

Re: Consuming Kafka Messages Inside of EC2 Instances

Thanks Jonathan! This was the result, would it be okay for me to kill 3544? Or do I try to access a new port?: *sudo netstat -tulpn | grep 2181* tcp0 0 0.0.0.0:21810.0.0.0:* LISTEN 3544/java *ps 3544* PID TTY STAT TIME COMMAND 3544 ?

Re: Consuming Kafka Messages Inside of EC2 Instances

Hey Su, That exception indicates that there's something else already running on the port that you're trying to start up Zookeeper on. The quickest way to figure out what's causing the conflict is to run netstat and look for the port: [root@ip-10-0-0-45 ~]# *netstat -tulpn | grep 2181* tcp

Re: Routing modifications at runtime

Thank you all. I'll have a look at flume and also at akka-http and akka-streams since the MACs will send the data to a REST endpoint. El 29/01/2015 16:10, "Jeff Holoman" escribió: > Yeah if you're into Flume you can definitely do per event > modification/routing in an interceptor with relative

Can't start Zookeeper on a EC2 instance in a public subnet

Hello Everyone, I previously had my EC2 instances in a private subnet, but I spun up a new cluster in a public subnet. However, it seems to have taken me a step back as now I can't even start the zookeeper. I am getting this error: ERROR Unexpected exception, exiting abnormally (org.apache.zookee

Re: Consuming Kafka Messages Inside of EC2 Instances

I tried a new approach and have started ec2 instances in a public subnet rather than a private subnet. However, now when I try to start zookeeper I get this error. How can I go about solving this issue? Thank you. ERROR Unexpected exception, exiting abnormally (org.apache.zookeeper.server.ZooKeepe

LinkedIn Engineering Blog Post - Current and Future

Here's an overview of what LinkedIn plans to concentrate on in the upcoming year. https://engineering.linkedin.com/kafka/kafka-linkedin-%E2%80%93-current-and-future -Jon signature.asc Description: Message signed with OpenPGP using GPGMail

Re: Resilient Producer

Thanks David. This looks to be interesting. Will definitely test this out to see whether this solves our problem. On Thu, Jan 29, 2015 at 8:29 AM, David Morales wrote: > Existing "tail" source is not the best choice in your scenario, as you have > pointed out. > > SpoolDir could be a solution if

Re: Consuming Kafka Messages Inside of EC2 Instances

Thank you Guozhang for an updated link. I read the answer for "Why can't my consumers/producers connect to the brokers?". I am confused on a couple things. I can ping the private IP of A (Cluster Launcher) from B (EC2 instance). I can't ping the public ip of A (or google.com) from B which makes s

Re: One or multiple instances of MM to aggregate kafka data to one hadoop

Hey Mingjie, Here's how we have our mirror makers configured. For some context, let me try to describe this using the example datacenter layout as described in: https://engineering.linkedin.com/samza/operating-apache-samza-scale In that example, there are four data centers (A, B, C, and D). How

Re: Resilient Producer

Existing "tail" source is not the best choice in your scenario, as you have pointed out. SpoolDir could be a solution if your log file rotation policy is very low (5 minutes, for example), but then you have to deal with a huge number of files in the folder (slower listings). There is a proposal f

Re: Consuming Kafka Messages Inside of EC2 Instances

Sorry my previous link was not complete: https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Whycan%27tmyconsumers/producersconnecttothebrokers ? On Wed, Jan 28, 2015 at 10:56 PM, Su She wrote: > Thank you Dillian and Guozhang for the responses. > > Yes, Dillian you are understanding my

Detecting lost connection in high level consumer

Hi I am using the 0811 Kafka High level consumer and I have configured " consumer.timeout.ms" to a value that is not -1, say 5000ms. I create the consumer iterator and invoke hasNext() method on it. Irrespective of whether kafka broker was shutdown or there was no message written to kafka, I see

Re: Routing modifications at runtime

Yeah if you're into Flume you can definitely do per event modification/routing in an interceptor with relative ease. I don't know the size of the total MAC addresses to look up (or actually why a hash partitioning scheme wouldn't just work, but w/e I assume you have your reasons). There's kind of a

Potential socket leak in kafka sync producer

Hi, Currently we are using sync producer client of 0.8.1 version in our production box . we are getting the following exception while publishing kafka message *[2015-01-29 13:21:45.505][ThreadPoolTaskExecutor-603][WARN][ClientUtils$:89] Fetching topic metadata with correlation id 10808 for topics

Re: Can't create a topic; can't delete it either

> If you can tell me where the find the logs I can check. I haven't restarted > my brokers since the issue. This will be specified in the log4j properties that you are using. On Wed, Jan 28, 2015 at 12:01:01PM -0800, Sumit Rangwala wrote: > On Tue, Jan 27, 2015 at 10:54 PM, Joel Koshy wrote: >

Re: Routing modifications at runtime

Hi Toni, 1. Kafka can create topics on the fly, in case you need it. https://kafka.apache.org/08/configuration.html auto.create.topics.enabletrueEnable auto creation of topic on the server. If this is set to true then attempts to produce, consume, or fetch metadata for a non-existent topic will

Re: [VOTE] 0.8.2.0 Candidate 3

+1 on librdkafka interop Minor nitpick: KAFKA-1781 (state required gradle version in README) is included in the Release notes but is not actually fixed 2015-01-29 6:22 GMT+01:00 Jun Rao : > This is the third candidate for release of Apache Kafka 0.8.2.0. > > Release Notes for the 0.8.2.0 rele