Re: Missing Per-Topic BrokerTopicMetrics in v0.8.2.0

I can confirm that the per topic metrics are not coming through to the yammer metrics registry. I do see them in jmx (via jconsole), but the MetricsRegistry does not have them. All the other metrics are coming through that appear in jmx. This is with single node instance running locally. Jason

Can't create a topic; can't delete it either

While running kafka in production I found an issue where a topic wasn't getting created even with auto topic enabled. I then went ahead and created the topic manually (from the command line). I then delete the topic, again manually. Now my broker won't allow me to either create *the* topic or delet

Re: kafka production server test

Hmm, so you just want to start producer / consumer clients which points to the existing brokers in production. I think what you described here should work, but I may be wrong. Please let me know if that does not work. Guozhang On Mon, Jan 26, 2015 at 1:50 PM, Sa Li wrote: > Thank you for repl

Kafka consumer connection error

Hi all, I am currently working on logstash to mongodb.logstash's input (producer) is kafka and output (consumer) is mongodb.It is worked fine for past two days.Today it is not working and i got the following error.i googled this error,still didnt find any correct solution.kindly help me to

Re: Kafka Out of Memory error

Thanks a lot Gwen. I bumped up the JVM to 1g on the consumer side and it works :) All the consumer belong to the same group and I am using the High level group API to consume from the kafka. It seems there is some initial meta data exchange or something about all the partitions are sent to all the

Re: Missing Per-Topic BrokerTopicMetrics in v0.8.2.0

If you are using multi-node cluster, then metrics may be reported from other servers. pl check all the servers in the cluster. On Tue, Jan 27, 2015 at 4:12 AM, Kyle Banker wrote: > I've been using a custom KafkaMetricsReporter to report Kafka broker > metrics to Graphite. In v0.8.1.1, Kafka was

Re: [VOTE] 0.8.2.0 Candidate 2 (with the correct links)

Hi, Don't use Graphite, so I don't know. Kyle, maybe you can share more info? What do you mean by "reported to Yammer" for example? And when you say Yammer/Graphite, are you trying to say that you are using the Graphite Reporter? If so, can you try other Yammer Reporters and see if there is a m

Re: [VOTE] 0.8.2.0 Candidate 2 (with the correct links)

Kyle, Also, which reporter are you using for Graphite? Thanks, Jun On Mon, Jan 26, 2015 at 4:08 PM, Kyle Banker wrote: > This is still preliminary, but it looks as if the change to metric names > for per-topic metrics (bytes/messages in/out) is preventing these metrics > from being reported t

Re: [DISCUSSION] Boot dependency in the new producer

Jay, I don't think this line will bootstrap full metadata (for all topics). it will just construct the cluster object with bootstrap host. you need to do "metadata.add(topic)" to set interest of a topic's partition metadata. Guozhang, I personally think this is ok. it just do a few DNS lookup or

Re: [VOTE] 0.8.2.0 Candidate 2 (with the correct links)

Hmm, that's not the intention. The per-topic mbeans are definitely registered by Yammer. So, not sure why it's not reported to Graphite. Otis, Vladimir, Do you guys know? Thanks, Jun On Mon, Jan 26, 2015 at 4:08 PM, Kyle Banker wrote: > This is still preliminary, but it looks as if the cha

Re: [VOTE] 0.8.2.0 Candidate 2 (with the correct links)

This is still preliminary, but it looks as if the change to metric names for per-topic metrics (bytes/messages in/out) is preventing these metrics from being reported to Yammer/Graphite. If this isn't intentional, it should probably be addressed before release. On Wed, Jan 21, 2015 at 9:28 AM, Jun

Re: Missing Per-Topic BrokerTopicMetrics in v0.8.2.0

The jmx beans created by Yammer have been changed to the new names. However, the global mbeans have different names from the topic level mbeans. The new mbeans have completely different names from the old ones since they no longer have quotes. Does that answer your question? Thanks, Jun On Mon,

Re: Missing Per-Topic BrokerTopicMetrics in v0.8.2.0

Thanks, Jun. I'm pretty sure, though not 100% confident, that this has caused a change in the way that these metrics are reported to Yammer metrics (I believe that they are stomping on each other). Is that intended as well? On Mon, Jan 26, 2015 at 4:13 PM, Jun Rao wrote: > Yes, we refactored the

Re: Missing Per-Topic BrokerTopicMetrics in v0.8.2.0

Yes, we refactored the mbean names in 0.8.2.0 to make them more standard. Those metrics are now registered under the following names. I did some quick test and the values do get updated. kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec

Missing Per-Topic BrokerTopicMetrics in v0.8.2.0

I've been using a custom KafkaMetricsReporter to report Kafka broker metrics to Graphite. In v0.8.1.1, Kafka was reporting bytes and messages in and out for all topics together and for each individual topic. After upgrading to v0.8.2.0, these metrics are no longer being reported. I'm only seeing

Re: SimpleConsumer leaks sockets on an UnresolvedAddressException

Meant to write a run loop. void run() { while (running) { if (simpleConsumer == null) { simpleConsumer = new SimpleConsumer(host, port, (int) kafkaSocketTimeout, kafkaRExeiveBufferSize, clientName); } try { // Do stuff with simpleConsumer. } catch (Exception e) {

Re: Kafka sending messages with zero copy

Hi Guozhang, I am a bit busy at work. When I get the change I'll definitely try to get a proof of concept going. Not the kafka protocol, but just the buffering and threading structures, maybe just write to another socket. I think it would be useful just to get the queueing and buffer management go

SimpleConsumer leaks sockets on an UnresolvedAddressException

Here is my typical flow: void run() { if (simpleConsumer == null) { simpleConsumer = new SimpleConsumer(host, port, (int) kafkaSocketTimeout, kafkaRExeiveBufferSize, clientName); } try { // Do stuff with simpleConsumer. } catch (Exception e) { if (consumer != null) { si

Re: does kafka support "COMMIT" of a batch ?

thanks for the info yang On Mon, Jan 26, 2015 at 2:09 PM, Jay Kreps wrote: > We did a relatively complete prototype but it isn't integrated into the > mainline code yet and there isn't a target release date. There is rather a > lot of testing and compatability work that would have to be done to

Re: [DISCUSSION] Boot dependency in the new producer

Oh, yes, I guess I thought you meant that construction of the client would block on the metadata request. I don't personally think that is a problem because if it fails it will retry in the background, right? But actually I think this is probably violating another desirable criteria we had talked

Re: does kafka support "COMMIT" of a batch ?

We did a relatively complete prototype but it isn't integrated into the mainline code yet and there isn't a target release date. There is rather a lot of testing and compatability work that would have to be done to fully productionize it. I suspect someone will pick it up in 2015 but I wouldn't blo

Re: unable to delete topic with 0.8.2 rc2

I was planning to start a new thread by my experience is similar to the one described here. I am currently using 0.8.2-beta and my setup has a producer producing to topic "DC-DATA", kafka brokers with auto-topic-creation enabled, and mirrormaker set to replicate any ".*-DATA" topics. After the prod

Re: unable to delete topic with 0.8.2 rc2

Hmm.. that's right. completely forgot about that. On Mon, Jan 26, 2015 at 01:49:33PM -0800, Jun Rao wrote: > Joel, > > That's probably because console consumer always uses wildcard for > consumption. > > Thanks, > > Jun > > On Mon, Jan 26, 2015 at 1:44 PM, Joel Koshy wrote: > > > I think thi

Re: does kafka support "COMMIT" of a batch ?

well I guess this is what I need https://cwiki.apache.org/confluence/display/KAFKA/Transactional+Messaging+in+Kafka anyone knows how far this effort has gone? thanks Yang On Mon, Jan 26, 2015 at 12:06 PM, Yang wrote: > we are pulling from a DB, maybe 100k rows at a time. for each row we send >

Re: kafka production server test

Thank you for reply Guozhang, right now I can get it work out of box, run testcase_1 on VM and access production. However, from my point of view, we really like to test the existing configs on productions, which means for example, replica_basic_test.py won't start zookeeper&kafka since I want to te

Re: unable to delete topic with 0.8.2 rc2

Joel, That's probably because console consumer always uses wildcard for consumption. Thanks, Jun On Mon, Jan 26, 2015 at 1:44 PM, Joel Koshy wrote: > I think this should work even on non-wildcard consumers. (i.e., I just > tried it locally and it appears to work fine). > > Here is what I did:

Re: unable to delete topic with 0.8.2 rc2

I think this should work even on non-wildcard consumers. (i.e., I just tried it locally and it appears to work fine). Here is what I did: - Leave a console consumer up and running, bounce the broker to turn off auto-create - Delete the topic - The consumer should rebalance and stop consuming th

Re: [DISCUSSION] Boot dependency in the new producer

It will set the needUpdate flag to true and hence the background Sender will try to talk to the bootstrap servers. Guozhang On Mon, Jan 26, 2015 at 1:12 PM, Jay Kreps wrote: > Hey Guozhang, > > That line shouldn't cause any connections to Kafka to be established, does > it? All that is doing is

Re: [DISCUSSION] Boot dependency in the new producer

Hey Guozhang, That line shouldn't cause any connections to Kafka to be established, does it? All that is doing is creating the Cluster pojo using the supplied addresses. The use of InetSocketAddress may cause some dns stuff to happen, though... -Jay On Mon, Jan 26, 2015 at 10:50 AM, Guozhang Wan

Re: unable to delete topic with 0.8.2 rc2

Actually, I think the deletion of a topic will trigger a rebalance on all wildcard subscribers. The time to complete the rebalance depends on the # of topic/partitions. Thanks, Jun On Mon, Jan 26, 2015 at 12:26 PM, Jason Rosenberg wrote: > Yeah Joel, > > I just thought of that idea too (e.g. t

Kafka under replicated partitions..

Hi All, I would like to know the factors that can cause followers to fall behind and under-replicated replica getting created? In my production system , last Friday one of the partitions fell behind and before i could check my operation team had restarted the follower broker... Restart did fixed

Re: unable to delete topic with 0.8.2 rc2

Yeah Jun, I realize it's not worth blocking things, since it's not trivial to solve. Thanks for all the work on this. Jason On Mon, Jan 26, 2015 at 3:01 PM, Jun Rao wrote: > Hi, Jason, > > I am not sure that we should delay the 0.8.2 release. The reasons are (1) > There are other features suc

Re: unable to delete topic with 0.8.2 rc2

Yeah Joel, I just thought of that idea too (e.g. temporarily disable auto topic creation). I can probably make that work (it may result in unexpected lack of topic creation, but I can message this as a maintenance downtime, etc.). So, that was my next question, so a consumer rebalance will trigg

Re: kafka production server test

Sa, I believe your questions have mostly been answered by Ewen, and sorry for getting late on this. As you notice the current system test's out-of-the-box experience is not very good, and we are proposing ways to improve that situation: KAFKA-1748

does kafka support "COMMIT" of a batch ?

we are pulling from a DB, maybe 100k rows at a time. for each row we send it to kafka. now the problem is , if any errors happen during the db pulling (such as an earthquake), we stop. next time we wake up. we can't write the same record again into kafka, otherwise there might be over counting.

Re: unable to delete topic with 0.8.2 rc2

Hey Jason, Is it an option for you to do the following: - Bounce in a config change to the brokers to turn off auto-create - (Batch)-delete the topic(s) - Wait long enough for consumers to rebalance (after which they will no longer consume the topic(s)) - Bounce in a config change to the broker

Re: [kafka-clients] Re: [VOTE] 0.8.2.0 Candidate 2 (with the correct links)

Hi Kafka Team, I just wanted to bring this to your attention regarding Java New Producer limitation compare to old producer. a) Partition Increasing is limited to configured memory allocation. buffer.memory batch.size The maximum partition you could have before impacting (New Java Producers)

Re: Regarding Kafka release 0.8.2-beta

The new consumer api is actually excluded from the javadoc that we generate. Thanks, Jun On Mon, Jan 26, 2015 at 11:54 AM, Jason Rosenberg wrote: > shouldn't the new consumer api be removed from the 0.8.2 code base then? > > On Fri, Jan 23, 2015 at 10:30 AM, Joe Stein wrote: > > > The new con

Re: unable to delete topic with 0.8.2 rc2

Hi, Jason, I am not sure that we should delay the 0.8.2 release. The reasons are (1) There are other features such as the new java producer and Kafka-based offset management that are potentially useful to people. (2) It may take some time to completely fix the issue with deleting topic since it in

Re: Regarding Kafka release 0.8.2-beta

Matve wr should add to the documentation experimental so folks that don't know understand. /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop

Re: Regarding Kafka release 0.8.2-beta

shouldn't the new consumer api be removed from the 0.8.2 code base then? On Fri, Jan 23, 2015 at 10:30 AM, Joe Stein wrote: > The new consumer is scheduled for 0.9.0. > > Currently Kafka release candidate 2 for 0.8.2.0 is being voted on. > > There is an in progress patch to the new consumer that

Re: kafka deleted old logs but not released

hi All, I am facing the same issue.. i have got Kafka_2.9.2-0.8.1.1.jar deployed in my application. My Operation team had reported significant increase in the directory where Kafak-log folder exists . when i checked the size of parent directory is it 99GB (100%) utilized.. This was very confusin

Re: unable to delete topic with 0.8.2 rc2

I think this shortcoming should probably delay release of 0.8.2 until resolved, no? On Mon, Jan 26, 2015 at 2:01 PM, Guozhang Wang wrote: > It then seems to me that delete-topic will not actually work "smoothly" > until create topic request is added since it is too much to require people > to tu

Re: unable to delete topic with 0.8.2 rc2

So, this is rather disappointing, especially since topic deletion is really the primary feature in 0.8.2 I'm interested in. The topic I was trying to delete above had no data for many months. The consumer which is triggering recreation of that topic has been restarted several times since that top

Re: unable to delete topic with 0.8.2 rc2

It then seems to me that delete-topic will not actually work "smoothly" until create topic request is added since it is too much to require people to turn off their clients while deleting topics. In this case shall we make it clear in the release docs of in 0.8.2, or even still mark it as not-suppo

[DISCUSSION] Boot dependency in the new producer

Hi all, I am not sure if we have discussed about this before, but recently I realized that we have introduced boot dependency of the kafka-server specified by the "bootstrap.servers" config in the new producer. More specifically, although in the old producer we also have a similar config for speci

MQTT and Kafka Integration

Hello Everyone, I recently watched this video by Tim Kellogg: https://www.youtube.com/watch?v=VoTclkxSago. 1) Has anyone integrated MQTT and Kafka? If so how did the architecture look like? I am confused as to how to get messages from the MQTT broker to the Kafka broker. 2) Has anyone used this

Re: [kafka-clients] Re: [VOTE] 0.8.2.0 Candidate 2 (with the correct links)

+1 (binding) artifacts and quick start look good. I ran in some client code, minor edits from 0-8.2-beta https://github.com/stealthly/scala-kafka/pull/26 On Mon, Jan 26, 2015 at 3:38 AM, Manikumar Reddy wrote: > +1 (Non-binding) > Verified source package, unit tests, release build, topic deleti

Re: unable to delete topic with 0.8.2 rc2

unsubscribe On Mon, Jan 26, 2015 at 8:14 AM, Harsha wrote: > Jun, > I made an attempt at fixing that issue as part of this JIRA > https://issues.apache.org/jira/browse/KAFKA-1507 . > As Jay pointed out there should be admin api if there is more info on > this api I am interested in a

Re: Increased CPU usage with 0.8.2-beta

Hi Neha, I sent an e-mail earlier today, but noticed now that it didn't actually go through. Anyhow, I've attached two files, one with output from a 10 minute run and one with output from a 30 minute run. Realized that maybe I should've done one or two runs with 0.8.1.1 as well, but nevertheless.

Re: [VOTE] 0.8.2.0 Candidate 2 (with the correct links)

+1 (binding) Verified quick start and unit tests. Thanks, Jun On Wed, Jan 21, 2015 at 8:28 AM, Jun Rao wrote: > This is the second candidate for release of Apache Kafka 0.8.2.0. There > has been some changes since the 0.8.2 beta release, especially in the new > java producer api and jmx mbean

Re: unable to delete topic with 0.8.2 rc2

Jun, I made an attempt at fixing that issue as part of this JIRA https://issues.apache.org/jira/browse/KAFKA-1507 . As Jay pointed out there should be admin api if there is more info on this api I am interested in adding/fixing this issue. Thanks, Harsha On Mon, Jan 26, 2015, at 07:28

Re: unable to delete topic with 0.8.2 rc2

Yes, that's the issue. Currently, topics can be auto-created on TopicMetadataRequest, which can be issued from both the producer and the consumer. To prevent that, you would need to stop the producer and the consumer before deleting a topic. We plan to address this issue once we have a separate req

Re: unable to delete topic with 0.8.2 rc2

There could be another case where if you have auto.create.topics.enable to set to true ( its true by default) . Any TopicMetadataRequest can recreate topics. So if you issued a delete topic command and you have producers running or consumers? too which is issuing a TopicMetadataRequest than the top

Re: no brokers found when trying to rebalance

Yeah the issue was mostly that literally ALL of the docs use / as the chroot. People kept getting tripped up with the ZK URL with /kafka, it's a bit funky eg: host1.domain.com:2181,host2.domain.com:2181,host3.domain.com:2181/kafka. So we opted to change it back to what everyone else is doing by d

Re: [kafka-clients] Re: [VOTE] 0.8.2.0 Candidate 2 (with the correct links)

+1 (Non-binding) Verified source package, unit tests, release build, topic deletion, compaction and random testing On Mon, Jan 26, 2015 at 6:14 AM, Neha Narkhede wrote: > +1 (binding) > Verified keys, quick start, unit tests. > > On Sat, Jan 24, 2015 at 4:26 PM, Joe Stein wrote: > > > That make