Re: Metadata Refresh and TimeoutException when MAX_BLOCK_MS_CONFIG set 0

refresh is not available. I hope there is enough interest to make this producer broker down vs metadata not available. Thanks, Bhavesh On Mon, Oct 10, 2022 at 4:04 PM Bhavesh Mistry wrote: > Hi Luke, > > Thanks for the pointers. > > Sorry for being late I was out. > >

Re: Metadata Refresh and TimeoutException when MAX_BLOCK_MS_CONFIG set 0

this idea? > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-286%3A+producer.send%28%29+should+not+block+on+metadata+update > > Thank you. > Luke > > On Sat, Sep 24, 2022 at 6:36 AM Bhavesh Mistry > > wrote: > > > Hello Kafka Team, > > > > I would

Re: Metadata Refresh and TimeoutException when MAX_BLOCK_MS_CONFIG set 0

Hello Kafka Team, I would appreciate any insight into how to distinguish between Brocker Down vs Metadata Refresh not available due to timing issues. Thanks, Bhavesh On Mon, Sep 19, 2022 at 12:50 PM Bhavesh Mistry wrote: > Hello Kafka Team, > > > > We have an environment whe

Metadata Refresh and TimeoutException when MAX_BLOCK_MS_CONFIG set 0

Hello Kafka Team, We have an environment where Kafka Broker can go down for whatever reason. Hence, we had configured MAX_BLOCK_MS_CONFIG=0 because we wanted to drop messages when brokers were NOT available. Now the issue is we get data loss due to METADATA not being available and get this

Re: Authorization Engine For Kafka Related to KPI-11

+ Kafka Dev team to see if Kafka Dev team know or recommend any Auth engine for Producers/Consumers. Thanks, Bhavesh Please pardon me, I accidentally send previous blank email. On Tue, Nov 3, 2015 at 9:52 PM, Bhavesh Mistry wrote: > On Sun, Nov 1, 2015 at 11:15 PM, Bhavesh Mistry >

Re: Authorization Engine For Kafka Related to KPI-11

On Sun, Nov 1, 2015 at 11:15 PM, Bhavesh Mistry wrote: > HI All, > > Have any one used Apache Ranger as Authorization Engine for Kafka Topic > creation, consumption (read) and write operation on a topic. I am looking > at having audit log and regulating consumption/ write to p

Authorization Engine For Kafka Related to KPI-11

HI All, Have any one used Apache Ranger as Authorization Engine for Kafka Topic creation, consumption (read) and write operation on a topic. I am looking at having audit log and regulating consumption/ write to particular topic (for example, having production

Re: New Consumer API and Range Consumption with Fail-over

bes its design: > > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+0.9+Consumer+Rewrite+Design > > -Jason > > On Tue, Aug 4, 2015 at 12:01 AM, Bhavesh Mistry < > mistry.p.bhav...@gmail.com> > wrote: > > > Hi Jason and Kafka Dev Team, > > > >

Gobblin Data Ingestion Framework

Camus will be phased out according to their git hub update. Has anyone tried Gobblin to load data into Hadoop (1.x) from Kafka (0.8.x.x) Thanks, Bhavesh

Re: New Consumer API and Range Consumption with Fail-over

t's not yet exposed in KafkaConsumer. I assume it > will be eventually, but I'm not sure whether that will be part of the > initial release. > > > Hope that helps! > > Jason > > On Thu, Jul 30, 2015 at 7:54 AM, Bhavesh Mistry < > mistry.p.bhav...@gmail.com&g

New Consumer API and Range Consumption with Fail-over

Hello Kafka Dev Team, With new Consumer API redesign ( https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java ), is there a capability to consume given the topic and partition start/ end position. How would I achieve following use

Mirror Maker Copy Data over Public Network

Hi All, Does anyone have set-up mirror maker to copy data over target cluster via SSH tunnel or ipsec for encrypting data ? I cannot use application encryption because existing consumer expect non-encrypted data (already running in prod). Any feedback SSH tunnel vs Ipsec or any issues you may

Re: querying messages based on timestamps

We had similar requirement to re-load the data based on timestamp (range between 1PM to 2PM) etc. We store the relationship between timestamp and largest offset number in Time Series Database using jmxtrans (LogEndOffset JMX bean vs current time.). You can setup polling interval to be 60 minutes

Re: At-least-once guarantees with high-level consumer

HI Carl, Produce side retry can produce duplicated message being sent to brokers with different offset with same message. Also, you may get duplicated when the High Level Consumer offset is not being saved or commit but you have processed data and your server restart etc... To guaranteed at-lea

Re: How Producer handles Network Connectivity Issues

Hi Kamal, In order to monitor each instance of producer, you will need to have alternative network monitoring channel (e.g Flume or Another Kafka Cluster for just monitoring a producers at large scale). Here is detail: 1) Add Custom Appender for Log4J and intercept all logs of Kafka Producer jav

Re: Is there a way to know when I've reached the end of a partition (consumed all messages) when using the high-level consumer?

I have used what Gwen has suggested but to avoid false positive: While consuming records keep track of *last* consumed offset and compare with latest offset on broker for consumed topic when you get "TimeOut Exception" for that particular partition for given topic (e.g JMX Bean *LogEndOffset *for

Re: Kafka Monitoring using JMX

You can use this https://github.com/Stackdriver/jmxtrans-config-stackdriver/blob/master/jmxtrans/stackdriver/json-specify-instance/kafka.json as an example of how Mbean are named and how " is being escaped with \", and just use different output writer for any thing prior to 0.8.2.1 version. After

Producer Behavior When one or more Brokers' Disk is Full.

Hello Kafka Community, What is expected behavior on Producer side when one or more Brokers’ disk is full, but have not reached retention period for topics (by size or by time limit). Does producer send data to that particular brokers and/or Producer Queue gets full and always throws Queue F

New Offset Management API Question

Hi Joel Koshy, I have following questions: 1) Does Offset Commit/Fetch API works with Simple Consumer ? Use Case: Camus dual commit offset on hdfs as well as to Kafka offset topic. It will be only committed once when FILE Is committed to HDFS (Not like auto commit) 2) With MM, can you selecti

Re: integrate Camus and Hive?

I Thanks, Bhavesh On Wed, Mar 11, 2015 at 10:42 AM, Andrew Otto wrote: > Thanks, > > Do you have this partitioner implemented? Perhaps it would be good to try > to get this into Camus as a build in option. HivePartitioner? :) > > -Ao > > > > On Mar 11, 2015

Re: integrate Camus and Hive?

artition values from your messages' timestamps? > > > > On Mar 11, 2015, at 11:29, Bhavesh Mistry > wrote: > > > > HI Yang, > > > > We do this today camus to hive (without the Avro) just plain old tab > > separated log line. > > > > W

Re: integrate Camus and Hive?

HI Yang, We do this today camus to hive (without the Avro) just plain old tab separated log line. We use the hive -f command to add dynamic partition to hive table: Bash Shell Scripts add time buckets into HIVE table before camus job runs: for partition in "${@//\//,}"; do echo "ALTER TABLE

Re: Camus Issue about Output File EOF Issue

-issues-end-of-file-javaioioexception-file-i.html for more info. We will have to patch Camus to copy to tmp directory then move to final destination as work around for now to make rename or file rename a more reliable. Thanks, Bhavesh On Monday, March 2, 2015, Bhavesh Mistry wrote: > > I s

Re: Camus Issue about Output File EOF Issue

hows that its trying to read a TEXT file > as if it was Seq. Thats why I suspected a misconfiguration of some > sort. > > Why do you suspect a race condition? > > On Mon, Mar 2, 2015 at 5:19 PM, Bhavesh Mistry > wrote: > > Hi Gwen, > > > > We are using Map

Re: Camus Issue about Output File EOF Issue

aprfs - you may want to check with > your vendor... I doubt Camus was extensively tested on that particular > FS. > > On Mon, Mar 2, 2015 at 3:59 PM, Bhavesh Mistry > wrote: > > Hi Kakfa User Team, > > > > I have been encountering two issues with Camus Kafka E

Camus Issue about Output File EOF Issue

Hi Kakfa User Team, I have been encountering two issues with Camus Kafka ETL Job: 1) End Of File (unclosed files) 2) Not SequenceFile Error The details of issues can be found at https://groups.google.com/forum/#!topic/camus_etl/RHS3ASy7Eqc. If you guys have faced similar issue, please let me kn

Re: After Leadership Election and "kafka.log" JMXBean Registration Process

HI Jun, Thanks for info. Thanks, Bhavesh On Tue, Feb 24, 2015 at 2:45 PM, Jun Rao wrote: > These two metrics are always registered, whether the replica is the leader > or the follower. > > Thanks, > > Jun > > On Mon, Feb 23, 2015 at 6:40 PM, Bhavesh Mistry < &

After Leadership Election and "kafka.log" JMXBean Registration Process

Hi Kafka Team or User Community , After leadership election or switch between follower/leader of partition for given topic, does following metrics JMX bean gets registered (on leader) and de-registered (on follower). LogEndOffset Size LogStartOffset eg: "kafka.log":type="Log",name="TOPIC-17-*Si

Re: Kafka New(Java) Producer Connection reset by peer error and LB

HI Kafka Team, Please confirm if you would like to open Jira issue to track this ? Thanks, Bhavesh On Mon, Feb 9, 2015 at 12:39 PM, Bhavesh Mistry wrote: > Hi Kakfa Team, > > We are getting this connection reset by pears after couple of minute aster > start-up of pro

Re: Get Latest Offset for Specific Topic for All Partition

5 PM, Gwen Shapira wrote: > You can use the metrics Kafka publishes. I think the relevant metrics are: > Log.LogEndOffset > Log.LogStartOffset > Log.size > > > Gwen > > On Thu, Feb 5, 2015 at 11:54 AM, Bhavesh Mistry > wrote: > > HI All, > > > >

Kafka New(Java) Producer Connection reset by peer error and LB

Hi Kakfa Team, We are getting this connection reset by pears after couple of minute aster start-up of producer due to infrastructure deployment strategies we have adopted from LinkedIn. We have LB hostname and port as seed server, and all producers are getting following exception because of TCP i

Get Latest Offset for Specific Topic for All Partition

HI All, I just need to get the latest offset # for topic (not for consumer group). Which API to get this info ? My use case is to analyze the data injection rate to each of partition is uniform or not (close). For this, I am planing to dump the latest offset into graphite for each partition an

Re: high cpu and network traffic when cluster has no topic

Hi Steven, Can you please try to see if io thread is indeed a problem ? The following on works on Linux: ps -p "$java_pid" -L -o tid,pcpu jstack -F "$java_pid" Then compare the thread # (may have to Hex # to decimal) between the Jstack and ps command. This will tell you which thread is consu

Re: Kafka ETL Camus Question

e cluster level and typically don't need it for most of our > > jobs. Especially with something like Camus, I don't see any need to run > > parallel copies of the same task. > > > > On Mon, Feb 2, 2015 at 10:36 PM, Bhavesh Mistry < > mistry.p.bhav...@gmail.com>

Re: Kafka ETL Camus Question

Hi Jun, Thanks for info. I did not get answer to my question there so I thought I try my luck here :) Thanks, Bhavesh On Mon, Feb 2, 2015 at 9:46 PM, Jun Rao wrote: > You can probably ask the Camus mailing list. > > Thanks, > > Jun > > On Thu, Jan 29, 2015 at 1:

Kafka ETL Camus Question

Hi Kafka Team or Linked-In Team, I would like to know if you guys run Camus ETL job with speculative execution true or false. Does it make sense to set this to false ? Having true, it creates additional load on brokers for each map task (create a map task to pull same partition twice). Is there

Re: [kafka-clients] Re: [VOTE] 0.8.2.0 Candidate 2 (with the correct links)

Hi Kafka Team, I just wanted to bring this to your attention regarding Java New Producer limitation compare to old producer. a) Partition Increasing is limited to configured memory allocation. buffer.memory batch.size The maximum partition you could have before impacting (New Java Producers)

Counting # of Message Brokers Receive Per Minute Per Topic

Hi Kafka Team, I need to count message received by entire Kafka Broker Cluster for a particular topic. I have 3 brokers, so do I need to sum the COUNT metric or just one server count reflect all server count. It seems that count is always increasing (although metric name is *MessagesInPerSec* so

Kafka Cluster Monitoring and Documentation of Internals (JMX Metrics) of Rejected Events

Hi Kafka Team, I am trying to find out Kafka Internal and how a message can be corrupted or lost at brokers side. I have refer to following documentations for monitoring: https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Internals http://kafka.apache.org/documentation.html#monitoring I am

Re: Follow-up On Important Issues for 0.8.2

Adding User Community to see if any one knows behavior of Producer for issue #1) and status of 2). Thanks, Bhavesh On Fri, Jan 2, 2015 at 12:37 PM, Bhavesh Mistry wrote: > Hi Kafka Dev Team, > > I am following-up with you guys regarding New (Java) Producer behavior in > event o

Latency Tracking Across All Kafka Component

Hi Kafka Team/Users, We are using Linked-in Kafka data pipe-line end-to-end. Producer(s) ->Local DC Brokers -> MM -> Central brokers -> Camus Job -> HDFS This is working out very well for us, but we need to have visibility of latency at each layer (Local DC Brokers -> MM -> Central brokers -> Ca

Re: Kafka 0.8.2 new producer blocking on metadata

Hi Paul, I have faced similar issue, which you have faced. Our use case was bit different and we needed to aggregate events and publish to same partition for same topic. Occasionally, I have run into blocked application threads (not because of metadata but sync block for each batch). When you

Re: [DISCUSSION] adding the serializer api back to the new java producer

Hi All, This is very likely when you have large site such as Linked-in and you have thousand of servers producing data. You will mixed bag of producer and serialization or deserialization because of incremental code deployment. So, it is best to keep the API as generic as possible and each org /

Re: [DISCUSSION] adding the serializer api back to the new java producer

How will mix bag will work with Consumer side ? Entire site can not be rolled at once so Consumer will have to deals with New and Old Serialize Bytes ? This could be app team responsibility. Are you guys targeting 0.8.2 release, which may break customer who are already using new producer API (be

Re: How to recover from ConsumerRebalanceFailedException ?

alternative proposal for this for new and old high-level consumer API. Thanks, Bhavesh On Tue, Nov 18, 2014 at 9:53 PM, Bhavesh Mistry wrote: > Hi Jun, > > ZK cluster are up and running. What is best way to programmatically > recover and I would try to exponential recovery proces

Re: How to recover from ConsumerRebalanceFailedException ?

2014 at 9:34 PM, Jun Rao wrote: > Is your ZK service alive at that point? If not, you just need to monitor > the ZK server properly. > > Thanks, > > Jun > > On Mon, Nov 17, 2014 at 2:30 PM, Bhavesh Mistry < > mistry.p.bhav...@gmail.com> > wrote: > > > H

Re: How to recover from ConsumerRebalanceFailedException ?

.sv.walmartlabs.com:9091/>* dies thread dies, then restart the sources. Is there any alter approach or life cycle method that so api consumer can attached to Consumer life cycle that it is dying and get notified so we can take some action. Thanks, Bhavesh On Mon, Nov 17, 2014 at 2:30 PM,

Re: Enforcing Network Bandwidth Quote with New Java Producer

e producer api. > > Thanks, > > Jun > > On Fri, Nov 14, 2014 at 10:34 AM, Bhavesh Mistry < > mistry.p.bhav...@gmail.com > > wrote: > > > HI Kafka Team, > > > > We like to enforce a network bandwidth quota limit per minute on producer > > side.

How to recover from ConsumerRebalanceFailedException ?

Hi Kafka Team, I get following exception due to ZK/Network issues intermittently. How do I recover from consumer thread dying *programmatically* and restart source because we have alerts that due to this error we have partition OWNERSHIP is *none* ? Please let me know how to restart source and

Enforcing Network Bandwidth Quote with New Java Producer

HI Kafka Team, We like to enforce a network bandwidth quota limit per minute on producer side. How can I do this ? I need some way to count compressed bytes on producer ? I know there is callback does not give this ability ? Let me know the best way. Thanks, Bhavesh

Re: Programmatic Kafka version detection/extraction?

If is maven artifact then you will get following pre-build property file from maven build called pom.properties under /META-INF/maven/groupid/artifactId/pom.properties folder. Here is sample: #Generated by Maven #Mon Oct 10 10:44:31 EDT 2011 version=10.0.1 groupId=com.google.guava artifactId=guava

Re: expanding cluster and reassigning parititions without restarting producer

I had different experience with expanding partition for new producer and its impact. I only tried for non-key message.I would always advice to keep batch size relatively low or plan for expansion with new java producer in advance or since inception otherwise running producer code is impacted.

Re: Announcing Confluent

HI Guys, Thanks for your awesome support. I wish you good luck !! Thanks for open sources Kafka !! Thanks, Bhavesh On Thu, Nov 6, 2014 at 10:52 AM, Rajasekar Elango wrote: > Congrats. Wish you all the very best and success. > > Thanks, > Raja. > > On Thu, Nov 6, 2014 at 1:36 PM, Niek Sande

Re: Spark Kafka Performance

Hi Eduardo, Can you please take thread dump and see if there are blocking issues on producer side ? Do you have single instance of Producers and Multiple treads ? Are you using Scala Producer or New Java Producer ? Also, what is your producer property ? Thanks, Bhavesh On Tue, Nov 4, 2014 a

Re: queued.max.message.chunks impact and consumer tuning

be sufficient. There is > little reason to buffer more than that. If you increase it to 2000 you > will most likely run into memory issues. E.g., if your fetch size is > 1MB you would enqueue 1MB*2000 chunks in each queue. > > On Tue, Nov 04, 2014 at 09:05:44AM -0800, Bhavesh Mistry w

Re: High Level Consumer Iterator IllegalStateException Issue

ov 3, 2014 at 4:35 PM, Jun Rao wrote: > Bhavesh, > > That example has a lot of code. Could you provide a simpler test that > demonstrates the problem? > > Thanks, > > Jun > > On Fri, Oct 31, 2014 at 10:07 PM, Bhavesh Mistry < > mistry.p.bhav...@gmail.com >

queued.max.message.chunks impact and consumer tuning

Hi Kafka Dev Team, It seems that Maximum buffer size is set to 2 default. What is impact of changing this to 2000 or so ? This will improve the consumer thread performance ? More event will be buffered in memory. Or Is there any other recommendation to tune High Level Consumers ? Here is co

Re: High Level Consumer Iterator IllegalStateException Issue

: > Do you have a simple test that can reproduce this issue? > > Thanks, > > Jun > > On Thu, Oct 30, 2014 at 8:34 PM, Bhavesh Mistry < > mistry.p.bhav...@gmail.com> > wrote: > > > HI Jun, > > > > Consumer Connector is not closed because I can see th

Re: High Level Consumer Iterator IllegalStateException Issue

the consumer connector is already closed and > then you call hasNext() on the iterator. > > Thanks, > > > Jun > > On Wed, Oct 29, 2014 at 9:06 PM, Bhavesh Mistry < > mistry.p.bhav...@gmail.com> > wrote: > > > Hi Jun, > > > > The hasNext() its

Re: partitions stealing & balancing consumer threads across servers

Hi Joel, Yes, I am on Kafka Trunk branch. In my scenario, if you have back-up threads does that impact the allocation. If I have 24 threads (6 thread for each JVM total of 4 JVMS) in above example , does partition allocation gets evenly distributed (3 on each JVM) ? is this supported use case ?

Re: partitions stealing & balancing consumer threads across servers

HI Joel, Correction to my previous question: What is expected behavior of *roundrobin *policy above scenario ? Thanks, Bhavesh On Thu, Oct 30, 2014 at 1:39 PM, Bhavesh Mistry wrote: > Hi Joel, > > I have similar issue. I have tried *partition.assignment.strategy=* > *"

Re: partitions stealing & balancing consumer threads across servers

Hi Joel, I have similar issue. I have tried *partition.assignment.strategy=* *"roundrobin"*, but how do you accept this accept to work ? We have a topic with 32 partitions and 4 JVM with 10 threads each ( 8 is backup if one of JVM goes down). The roundrobin does not select all the JVM only 3 J

Re: High Level Consumer Iterator IllegalStateException Issue

ll next() before > hasNext() on the iterator. > > Thanks, > > Jun > > On Tue, Oct 28, 2014 at 10:50 AM, Bhavesh Mistry < > mistry.p.bhav...@gmail.com > > wrote: > > > Hi Neha, > > > > Thanks for your answer. Can you please let me know how I can r

Re: High Level Consumer and Close with Auto Commit On

onnector.scala (currently the only > implementation of ConsumerConnector) you'll see shutdown() includes > the following: > > if (config.autoCommitEnable) > commitOffsets() > > Gwen > > On Tue, Oct 28, 2014 at 11:44 AM, Bhavesh Mistry > wrote: > >

High Level Consumer and Close with Auto Commit On

Hi Kafka Team, What is expected behavior when you close *ConsumerConnector* and auto commit is on ? Basically, when auto commit interval is set to 5 seconds and shutdown is called (before 5 seconds elapses) does ConsumerConnector commit the offset of message consumed by (next()) method or consume

Re: High Level Consumer Iterator IllegalStateException Issue

: > queued.max.message.chunks controls the consumer's fetcher queue. > > On Mon, Oct 27, 2014 at 9:32 PM, Bhavesh Mistry < > mistry.p.bhav...@gmail.com> > wrote: > > > HI Neha, > > > > If I solved the problem number 1 think and number 2 will be solved

Where Compression/Decompression happens

Hi Kafka Team, Is Compression happening on Producer Side (on application thread meaning thread that call send method or background Kafka thread ) and where does decompression Consumer side ? Is there any Compression/Decompression happening on Brokers Side when receiving message from producer and

Re: High Level Consumer Iterator IllegalStateException Issue

ue > can fill up if your consumer thread dies or slows down. I'd recommend you > ensure that all your consumer threads are alive. You can take a thread dump > to verify this. > > Thanks, > Neha > > On Mon, Oct 27, 2014 at 2:14 PM, Bhavesh Mistry < > mistry.p.bhav..

Re: High Level Consumer Iterator IllegalStateException Issue

2014 at 3:14 PM, Neha Narkhede wrote: > Can you provide the steps to reproduce this issue? > > On Fri, Oct 24, 2014 at 6:11 PM, Bhavesh Mistry < > mistry.p.bhav...@gmail.com> > wrote: > > > I am using one from the Kafka Trunk branch. > > > > Thanks, > &

Re: High Level Consumer Iterator IllegalStateException Issue

I am using one from the Kafka Trunk branch. Thanks, Bhavesh On Fri, Oct 24, 2014 at 5:24 PM, Neha Narkhede wrote: > Which version of Kafka are you using on the consumer? > > On Fri, Oct 24, 2014 at 4:14 PM, Bhavesh Mistry < > mistry.p.bhav...@gmail.com> > wrote: >

High Level Consumer Iterator IllegalStateException Issue

HI Kafka Community , I am using kafka trunk source code and I get following exception. What could cause the iterator to have FAILED state. Please let me know how I can fix this issue. *java.lang.IllegalStateException: Iterator is in failed stateat kafka.utils.IteratorTemplate.hasNext(It

Re: Sending Same Message to Two Topics on Same Broker Cluster

e.org/~nehanarkhede/kafka-0.9-producer-javadoc/doc/org/apache/kafka/clients/producer/ProducerRecord.html > > > > Thanks for your quick response and I appreciate your help. > > > > Thanks, > > > > Bhavesh > > > > > > On Mon, Oct 20, 2014 at 9:10 PM

Re: Sending Same Message to Two Topics on Same Broker Cluster

PM, Neha Narkhede wrote: > Not really. You need producers to send data to Kafka. > > On Mon, Oct 20, 2014 at 9:05 PM, Bhavesh Mistry < > mistry.p.bhav...@gmail.com> > wrote: > > > Hi Kakfa Team, > > > > > > I would like to send a single message

Sending Same Message to Two Topics on Same Broker Cluster

Hi Kakfa Team, I would like to send a single message to multiple topics (two for now) without re-transmitting the message from producer to brokers. Is this possible? Both Producers Scala and Java does not allow this. I do not have to do this all the time only based on application condition.

Re: Auto Purging Consumer Group Configuration [Especially Kafka Console Group]

easy to write a script to detect and > clean them directly on ZK. > > BTW. Why do you need to clean them? What issue do you see with just > letting them hang around? > > Gwen > > On Thu, Oct 9, 2014 at 9:18 PM, Bhavesh Mistry > wrote: > > Hi Kafka, > > >

Auto Purging Consumer Group Configuration [Especially Kafka Console Group]

Hi Kafka, We have lots of lingering console consumer group people have created for testing or debugging purpose for one time use via bin/kafka-console-consumer.sh. Is there auto purging that clean script that Kafka provide ? Is three any API to find out inactive Consumer group and delete consume

Re: [Java New Producer] CPU Usage Spike to 100% when network connection is lost

ugs > similar to this against trunk. > > -Jay > > On Wed, Sep 17, 2014 at 2:14 PM, Bhavesh Mistry > wrote: > > Hi Kafka Dev team, > > > > I see my CPU spike to 100% when network connection is lost for while. It > > seems network IO thread are very busy l

[Java New Producer] CPU Usage Spike to 100% when network connection is lost

Hi Kafka Dev team, I see my CPU spike to 100% when network connection is lost for while. It seems network IO thread are very busy logging following error message. Is this expected behavior ? 2014-09-17 14:06:16.830 [kafka-producer-network-thread] ERROR org.apache.kafka.clients.producer.interna

Re: MBeans, dashes, underscores, and KAFKA-1481

ientid". That seems to be a much > cleaner way than having all parts included in a single string separated by > '|'. > > Thanks, > > Jun > > > > > On Tue, Sep 16, 2014 at 5:15 PM, Bhavesh Mistry < > mistry.p.bhav...@gmail.com> > wrote: > >

Re: MBeans, dashes, underscores, and KAFKA-1481

HI Otis, What is migration path ? If topic with special chars exists already( ".","-","|" etc) in previous version of producer/consumer of Kafka, what happens after the upgrade new producer or consumer (kafka version) ? Also, in new producer API (Kafka Trunk), does this enforce the rule about c

Re: Need Document and Explanation Of New Metrics Name in New Java Producer on Kafka Trunk

is thread this ties back into > > >> the JMX/MBean issues related the usage of "-"/dashes and > > >> "_"/underscores, which are unfortunately the only non-alphanumeric > > >> characters that are at your disposal to add reasonable "structure" to >

[Java New Producer Configuration] Maximum time spent in Queue in Async mode

Hi Kafka team, How do I configure a max amount a message spend in Queue ? In old producer, there is property called queue.buffering.max.ms and it is not present in new one. Basically, if I just send one message that is less than batch size, what is amount of time message will be in local Produc

Re: message size limit

e in its own batch. Then, only one > message will be rejected by the broker. > > Thanks, > > Jun > > On Tue, Sep 9, 2014 at 5:51 PM, Bhavesh Mistry > > wrote: > > > Hi Jun, > > > > Is there any plug-ability that Developer can customize batching logi

Re: Need Document and Explanation Of New Metrics Name in New Java Producer on Kafka Trunk

Jay Kreps wrote: > Hi Bhavesh, > > Each of those JMX attributes comes with documentation. If you open up > jconsole and attach to a jvm running the consumer you should be able > to read the descriptions for each attribute. > > -Jay > > On Tue, Sep 9, 2014 at 2:07 PM, Bh

Re: message size limit

> Jun > > On Tue, Sep 9, 2014 at 12:59 PM, Bhavesh Mistry < > mistry.p.bhav...@gmail.com> > wrote: > > > HI Jun, > > > > Thanks for clarification. Follow up questions, does new producer solve > the > > issues highlight. In event of compression

Need Document and Explanation Of New Metrics Name in New Java Producer on Kafka Trunk

Kafka Team, Can you please let me know what each of following Metrics means ? Some of them are obvious, but some are hard to understand. My Topic name is *TOPIC_NAME*. can we enforce a Topic Name Convention or Metric Name Convention. Because in previous version of Kafka, we have similar issue

Re: message size limit

so, message.max.bytes applies > to the compressed size of a batch of messages. Otherwise, message.max.bytes > applies to the size of each individual message. > > Thanks, > > Jun > > On Wed, Sep 3, 2014 at 3:25 PM, Bhavesh Mistry > > wrote: > > > I am referring to wiki http://

Re: High Level Consumer and Commit

ssing the messages. > > > > Does that solve the problem, Bhavesh? > > > > Gwen > > > > On Tue, Sep 2, 2014 at 5:47 PM, Philip O'Toole > wrote: > >> Yeah, from reading that I suspect you need the SimpleConsumer. Try it > out and see. > >> > >> Philip > >> > >> >

Re: message size limit

essage.max.bytes` is applied per message not per > batch. > is there another limit I should be aware of? > > thanks > > > On Wed, Sep 3, 2014 at 2:07 PM, Bhavesh Mistry > > wrote: > > > Hi Jun, > > > > We have similar problem. We have variable lengt

Re: message size limit

Hi Jun, We have similar problem. We have variable length of messages. So when we have fixed size of Batch sometime the batch exceed the limit set on the brokers (2MB). So can Producer have some extra logic to determine the optimal batch size by looking at configured message.max.bytes value. D

Re: High Level Consumer and Commit

evel Consumer > should be able to do what you want if you disable auto-commit. I'm not sure > what else you're asking. > > > Philip > > > - > http://www.philipotoole.com > > > On Tuesday, September 2, 2014 5:15

Re: High Level Consumer and Commit

o-commit. > > Philip > > > - > http://www.philipotoole.com > > > On Tuesday, September 2, 2014 4:38 PM, Bhavesh Mistry < > mistry.p.bhav...@gmail.com> wrote: > > > > Hi Kafka Group, > > I have to pull the data from the Topic and index into El

High Level Consumer and Commit

Hi Kafka Group, I have to pull the data from the Topic and index into Elastic Search with Bulk API and wanted to commit only batch that has been committed and still continue to read from topic further on same topic. I have auto commit to be off. List batch . while (iterator.hasNext()) {

[Kafka MirrorMaker] Message with Custom Partition Logic

HI Kafka Dev Team, We have to aggregate events (count) per DC and across DCs for one of topic. We have standard Linked-in data pipe line producers --> Local Brokers --> MM --> Center Brokers. So I would like to know How MM handles messages when custom partitioning logic is used as below and

Re: [New Feature Request] Ability to Inject Queue Implementation Async Mode

- > http://www.philipotoole.com > > > On Aug 7, 2014, at 2:44 PM, Bhavesh Mistry > wrote: > > > > Basically, requirement is to support message dropping policy in event > when > > queue is full. When you get storm of data (in our case logging

Re: [New Feature Request] Ability to Inject Queue Implementation Async Mode

op <http://www.twitter.com/allthingshadoop> > ********/ > > > On Mon, Aug 4, 2014 at 8:52 PM, Bhavesh Mistry > > wrote: > > > Kafka Version: 0.8.x > > > > 1) Ability to define which messages get drooped (least recently instead > of > &g

Re: Uniform Distribution of Messages for Topic Across Partitions Without Effecting Performance

**********/ > > > On Mon, Aug 4, 2014 at 9:12 PM, Bhavesh Mistry > > wrote: > > > How to achieve uniform distribution of non-keyed messages per topic > across > > all partitions? > > > > We have tried to do this unifor

Uniform Distribution of Messages for Topic Across Partitions Without Effecting Performance

How to achieve uniform distribution of non-keyed messages per topic across all partitions? We have tried to do this uniform distribution across partition using custom partitioning from each producer instance using round robing ( count(messages) % number of partition for topic). This strategy resul

[New Feature Request] Ability to Inject Queue Implementation Async Mode

Kafka Version: 0.8.x 1) Ability to define which messages get drooped (least recently instead of most recent in queue) 2) Try Unbounded Queue to find out the Upper Limit without drooping any messages for application (use case Stress test) 3) Priority Blocking Queue ( meaning a single Producer can

Re: Monitoring Producers at Large Scale

c. for log handling) > > Otis > -- > Performance Monitoring * Log Analytics * Search Analytics > Solr & Elasticsearch Support * http://sematext.com/ > > > On Thu, Jun 26, 2014 at 3:09 PM, Bhavesh Mistry < > mistry.p.bhav...@gmail.com> > wrote: > > > Hi

  1   2   >