How to keep-alive connection from producer to broker?

Hi, I would like to know if there is a way/configuration to keep-alive connections from producer/client to broker? I observed that connection is getting closed every time, which takes considerable time. Can you please suggest ? Thanks & Regards, Neeraja

Re: 0.8.2 producer with 0.8.1.1 cluster?

10x Christian On Thu, Nov 13, 2014 at 9:50 AM, cac...@gmail.com wrote: > I used the 0.8.2 producer in a 0.8.1 cluster in a nonproduction > environment. No problems to report it worked great, but my testing at that > time was not particularly extensive for failure scenarios. > > Christian > > On

Re: Broker keeps rebalancing

@Neha, Can you share suggested consumer side GC settings? On Wed, Nov 12, 2014 at 5:31 PM, Neha Narkhede wrote: > Does this help? > > https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Whyaretheremanyrebalancesinmyconsumerlog > ? > > On Wed, Nov 12, 2014 at 3:53 PM, Chen Wang > wrote: >

Kafka to transport binary files

I'm thinking of using Kafka for transporting binary files (tiff, jpeg, pdf). These files are anywhere between 10 KB to 5MB. Thought behind considering Kafka is - It serves as a staging area for the files and facilitates asynchronous ingestion in near-real-time. Any thoughts on using Kafka for bina

Subscibe

Hello there: I'd like to be added to the mailing list Thanks, -- Rohit Pujari Solutions Engineer, Hortonworks rpuj...@hortonworks.com 716-430-6899 -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain inform

Re: Indefinite growth of FetchRequestPurgatory

Thanks a lot Guozhang, I've now upgraded to 0.8.2-beta and the issue seems to be gone. András On 11/3/2014 4:45 PM, Guozhang Wang wrote: Hi Andras, Could you try 0.8.2-beta and see if this issue comes out again? We fixed a couple of the purgatory issues (e.g. KAFKA-1616

how to configure kafka over firewall

Hi, I would like to know the way to configure kafka over firewall. We have firewall where it filters the connections based on protocol and there are security checks in place. Our broker 0.8.1 resides over firewall when producers sending data to broker, have to go through checks. Please suggest o

Re: Subscibe

Rohit, Please send a mail to users-subscr...@kafka.apache.org. more info here http://kafka.apache.org/contact.html. -Harsha On Wed, Nov 12, 2014, at 09:20 PM, Rohit Pujari wrote: > Hello there: > > I'd like to be added to the mailing list > > Thanks, > -- > Rohit Pujari > Solutions Engin

Re: Broker keeps rebalancing

Hey Chen, As Neha suggested, typical reason of too many rebalances is that your consumers kept being timed out from ZK, and you can verify this by checking in your consumer logs for sth. like "session timeout" entries (these are not ERROR entries). Guozhang Guozhang On Wed, Nov 12, 2014 at 5:31

Re: how to configure kafka over firewall

Kafka uses a binary protocol over tcp. To allow kafka traffic, you should open all the brokerIP:port in firewall. kumar On Thu, Nov 13, 2014 at 9:47 PM, Kothakota, Neeraja (HP Software) < neeraj...@hp.com> wrote: > Hi, > > I would like to know the way to configure kafka over firewall. > We h

Re: Broker keeps rebalancing

Thanks for the info. It makes sense, however, I didn't see any "session timeout"/"expired" entries in consumer log.. but do see lots of connection closed entry in zookeeper log: 2014-11-13 10:07:53,132 [myid:1] - INFO [NIOServerCxn.Factory: 0.0.0.0/0.0.0.0:2182:NIOServerCnxn@1007] - Closed socket

Re: Broker keeps rebalancing

@Neha, Can you share suggested consumer side GC settings? Consumer side GC settings are not standard since it is a function of your application that embeds the consumer. Your consumer application's memory patterns will dictate your GC settings. Sorry, I know that's not very helpful, but GC tuning

Re: Broker keeps rebalancing

Chen, >From ZK logs it sounds like ZK kept timed out consumers which triggers rebalance. What is the zk session timeout config value in your consumers? Guozhang On Thu, Nov 13, 2014 at 10:15 AM, Chen Wang wrote: > Thanks for the info. > It makes sense, however, I didn't see any "session timeo

Re: How to keep-alive connection from producer to broker?

Neeraja, Producer does use keep-alive connections to the brokers, and a recent change is introduced in broker which will actively close connections if it has not got any requests from the producer for some time. The default period is 10 min, you can set it to INT_MAX if you do not want this featur

Re: Broker keeps rebalancing

@Guozhang: In server.properties we have : zookeeper.connection.timeout.ms=100 In zoo.cfg we have tickTime=2000 initLimit=10 syncLimit=5 dataDir=/opt/zookeeper/data dataLogDir=/opt/zookeeper/logs clientPort=2182 server.1=.com:2888:3888 server.2=.com:2888:3888 server.3=.

Re: Broker keeps rebalancing

I was originally asking about consumer configs, which should contain the following: http://kafka.apache.org/documentation.html#consumerconfigs zookeeper.session.timeout.ms zookeeper.connection.timeout.ms On Thu, Nov 13, 2014 at 10:40 AM, Manish wrote: > @Guozhang: > > In server.properties we

Re: Broker keeps rebalancing

kafka.config.zookeeper.session.timeout.ms6 kafka.config.rebalance.backoff.ms6000kafka.config.rebalance.max.retries6 On Thu, Nov 13, 2014 at 10:56 AM, Guozhang Wang wrote: > I was originally asking about consumer configs, which should contain the > following: > > http://kafka.apache.org/docum

Re: Getting Simple consumer details using MBean

I tried running kafka-simple-consumer-shell. I can see the following mbean. "kafka.consumer":type="FetchRequestAndResponseMetrics",name="SimpleConsumerShell-AllBrokersFetchRequestRateAndTimeMs" Thanks, Jun On Wed, Nov 12, 2014 at 9:57 PM, Madhukar Bharti wrote: > Hi Jun Rao, > > Thanks for yo

Re: Broker keeps rebalancing

>From your zk logs: 2014-11-13 10:07:53,132 [myid:1] - INFO [NIOServerCxn.Factory: .. Closed socket connection for 2014-11-13 10:08:08,746 [myid:1] - INFO [NIOServerCxn.Factory: .. Closed socket connection for It does seem to kick out one consumer every 6ms, and you would probably check you

One of two consumers is always Idle though I have 2 partitions

Dear Developers, I am 2nd year masters student at IIT. I am using Kafka for one of my research projects.My question is the following: 1. I have a producer, consumer and a broker(that contains 1st partition of my topic) on node1 2. I have a producer, consumer, zookeeper and a broker(that contains

Is AWS Kinesis Kafka?

Oh man they look similar. Any comments?

Re: Is AWS Kinesis Kafka?

I've wondered that about Azure Event Hubs as well. They both use a different consumer offset tracking mechanism than the one in 0.8 for their higher level consumers. Christian On Thu, Nov 13, 2014 at 2:32 PM, Joseph Lawson wrote: > Oh man they look similar. Any comments? >

Re: Is AWS Kinesis Kafka?

Perhaps humanity just hit that inevitable point where we needed streaming event queues. Sort of like how Darwin and Alfred Russel Wallace both thought of evolution at the same time. From: cac...@gmail.com Sent: Thursday, November 13, 2014 2:36:01 PM To:

Re: One of two consumers is always Idle though I have 2 partitions

Are both of them in same Consumer Group? On Fri, Nov 14, 2014 at 9:12 AM, Palur Sandeep wrote: > Dear Developers, > > I am 2nd year masters student at IIT. I am using Kafka for one of my > research projects.My question is the following: > > 1. I have a producer, consumer and a broker(that contai

Re: One of two consumers is always Idle though I have 2 partitions

Yes, they are on the same consumer group, but I have two partitions. On Thu, Nov 13, 2014 at 5:04 PM, Jagat Singh wrote: > Are both of them in same Consumer Group? > > On Fri, Nov 14, 2014 at 9:12 AM, Palur Sandeep > wrote: > > > Dear Developers, > > > > I am 2nd year masters student at IIT. I

Re: Is AWS Kinesis Kafka?

Many similarities. For Kinesis right now: * only a 1 day max retention * max 50KB message size * guaranteed throughput based on MB/sec in and out. * servers hosting the shards abstracted away by SaaS For collaborative consumption, Kinesis uses DynamoDB whereas Kafka uses Zookeeper. Until recent

Re: One of two consumers is always Idle though I have 2 partitions

It would be worth reading once the consumer section from the documentation. https://kafka.apache.org/documentation.html On Fri, Nov 14, 2014 at 10:09 AM, Palur Sandeep wrote: > Yes, they are on the same consumer group, but I have two partitions. > > On Thu, Nov 13, 2014 at 5:04 PM, Jagat Sing

Re: Is AWS Kinesis Kafka?

Yeah the real question is really are the products built on top of Kafka (Kafka with a hat on). The last place I worked we ended up using Kinesis rather than Kafka basically for the reason Niek mentions, it seemed easier to accept the limitations and pay Amazon rather than run Kafka (small company <

Re: Is AWS Kinesis Kafka?

Do you think things will change now , with Kafta makers setting up own company and will provide commercial support? On Fri, Nov 14, 2014 at 10:28 AM, cac...@gmail.com wrote: > Yeah the real question is really are the products built on top of Kafka > (Kafka with a hat on). The last place I worke

log4j dir?

Hi guys, Just notice kafka.logs.dir in log4j.properties doesn't take effect It's always set to *$base_dir/logs* in kafka-run-class.sh LOG_DIR=$base_dir/logs KAFKA_LOG4J_OPTS="-Dkafka.logs.dir=$LOG_DIR $KAFKA_LOG4J_OPTS" Best, Siyuan

Re: Broker keeps rebalancing

Which version of ZK are you using? Thanks, Jun On Thu, Nov 13, 2014 at 10:15 AM, Chen Wang wrote: > Thanks for the info. > It makes sense, however, I didn't see any "session timeout"/"expired" > entries in consumer log.. > but do see lots of connection closed entry in zookeeper log: > > 2014-1

Re: One of two consumers is always Idle though I have 2 partitions

Hi Palur, When producing messages, did you specify a key in your KeyedMessage? If not, producer will send all messages to ONE randomly selected partition and stick to this partition for 10 minutes by default. regards, Chia-Chun 2014-11-14 7:19 GMT+08:00 Jagat Singh : > It would be worth reading

Re: Kafka to transport binary files

Both the Kafka client and broker need to allocate memory for the whole message. So, the larger the message, the more memory fragmentation it may cause, which can lead to GC/OOME issues. Thanks, Jun On Wed, Nov 12, 2014 at 9:23 PM, Rohit Pujari wrote: > I'm thinking of using Kafka for transport

RE: How to keep-alive connection from producer to broker?

@Guozhang, Thanks for your reply. Is is available as part of broker configuration? If yes, how to configure. --Neeraja -Original Message- From: Guozhang Wang [mailto:wangg...@gmail.com] Sent: Friday, November 14, 2014 12:02 AM To: users@kafka.apache.org Subject: Re: How to keep-alive c

Re: How to keep-alive connection from producer to broker?

I think there is no docs yet for this feature since it is only included in trunk (are you using a released version? If yes then you should not hit this); but here is the ticket that you can take a look: https://issues.apache.org/jira/browse/KAFKA-1282 On Thu, Nov 13, 2014 at 6:37 PM, Kothakota, N

Re: One of two consumers is always Idle though I have 2 partitions

Yup, sounds like https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Whyisdatanotevenlydistributedamongpartitionswhenapartitioningkeyisnotspecified ? This should go away with 0.8.2 with the default partions now being 1 =8^) with auto create topics. /*

Re: One of two consumers is always Idle though I have 2 partitions

Thank you Chia-chun,Joe and Jagat. I am not using any custom partitioner logic. Here is what I observed when I ran kafka on 4 nodes with the following structure: 1. Each node has a producer, consumer and a broker (that contains one partition of my topic) and one of the machine has the Zookeeper t

Re: Location of Logging Files/How To Turn On Logging For Kafka Components

Hi Jun, These are the two lines of log4j-related warnings I get when I try to run my producer: log4j:WARN No appenders could be found for logger (kafka.utils.VerifiableProperties). log4j:WARN Please initialize the log4j system properly. I have searched extensively online and have so far not fo

Re: Getting Simple consumer details using MBean

Hi Jun Rao, Sorry to disturb you. But I my Kafka setup it is not showing. I am attaching screen shot taken from all brokers. In kafka.consumer it is listing only "ReplicaFetcherThread". As I said earlier I am using "2.10-0.8.1.1" version. Do i need to configure any extra parameter for this? I am

Re: One of two consumers is always Idle though I have 2 partitions

If you're not using your own partitioning logic, messages are partitioned randomly. This is the current default behavior I believe. On Fri, Nov 14, 2014 at 12:01 PM, Palur Sandeep wrote: > Thank you Chia-chun,Joe and Jagat. > > I am not using any custom partitioner logic. Here is what I observe