Streams Processing meetup on Wednesday, February 5, 2020 at LinkedIn, Sunnyvale

*[bcc: (users,dev)@kafka.apache.org ]* Hello, The Streams Infra team invites you to attend the Streams Processing meetup to be held on Wednesday, February 5, 2020. This meetup will focus on Apache Kafka, Apache Samza and related streaming technologies. *Where*: Unify Con

Streams meetup at LinkedIn Sunnyvale, 6pm, Thursday, October 3, 2019

*[bcc: (users,dev)@kafka.apache.org ]* Hi everyone, The Streams Infra team invites you to attend a Streams Processing meetup on Thursday, October 3, 2019 at LinkedIn's Sunnyvale campus. (This meetup focuses on Apache Kafka, Apache Samza, and related streaming technologies

Streams Meetup at LinkedIn Sunnyvale, 6pm, Wednesday, March 20, 2019

*[bcc: (users,dev)@kafka.apache.org ]* Hi everyone, The Streams Infrastructure team at LinkedIn invites you to attend a Streams Processing meetup on Wednesday, March 20 at LinkedIn’s Sunnyvale campus. (This meetup focuses on Apache Kafka, Apache Samza, and related streami

Stream processing meetup at LinkedIn (Sunnyvale) on Thursday, February 16 at 6pm

Hi everyone, We would like to invite you to a Stream Processing Meetup at LinkedIn’s Sunnyvale campus on Thursday, February 16 at 6pm. Please RSVP here (*only if you intend to attend in person*): https://www.meetup.com/Stream-Processing-Meetup-LinkedIn/events/237171557/

Re: [ANNOUNCE] New committer: Grant Henke

Hey Grant - congrats! On Thu, Jan 12, 2017 at 10:00 AM, Neha Narkhede wrote: > Congratulations, Grant. Well deserved! > > On Thu, Jan 12, 2017 at 7:51 AM Grant Henke wrote: > > > Thanks everyone! > > > > On Thu, Jan 12, 2017 at 2:58 AM, Damian Guy > wrote: > > > > > Congratulations! > > > > >

Re: [VOTE] Vote for KIP-101 - Leader Epochs

(adding the dev list back - as it seems to have gotten dropped earlier in this thread) On Thu, Jan 5, 2017 at 6:36 PM, Joel Koshy wrote: > +1 > > This is a very well-written KIP! > Minor: there is still a mix of terms in the doc that references the > earlier LeaderGenerationRe

Re: [VOTE] Vote for KIP-101 - Leader Epochs

+1 This is a very well-written KIP! Minor: there is still a mix of terms in the doc that references the earlier LeaderGenerationRequest (which is what I'm assuming what it was called in previous versions of the wiki). Same for the diagrams which I'm guessing are a little harder to make consistent

[ANNOUNCE] New committer: Jiangjie (Becket) Qin

The PMC for Apache Kafka has invited Jiangjie (Becket) Qin to join as a committer and we are pleased to announce that he has accepted! Becket has made significant contributions to Kafka over the last two years. He has been deeply involved in a broad range of KIP discussions and has contributed sev

Re: Stream processing meetup at LinkedIn (Sunnyvale) on Wednesday, November 2 at 6pm

rs, > João Reis > > ____ > From: Joel Koshy > Sent: Monday, October 17, 2016 10:25:10 PM > Cc: eyakabo...@linkedin.com > Subject: Stream processing meetup at LinkedIn (Sunnyvale) on Wednesday, > November 2 at 6pm > > Hi everyone, > > We wou

Stream processing meetup at LinkedIn (Sunnyvale) on Wednesday, November 2 at 6pm

Hi everyone, We would like to invite you to a Stream Processing Meetup at LinkedIn’s Sunnyvale campus on Wednesday, November 2 at 6pm. Please RSVP here (if you intend to attend in person): http://www.meetup.com/Stream-Processing-Meetup-LinkedIn/events/234454163 We have the following three talks

Stream processing meetup at LinkedIn (Mountain View) on Tuesday, August 23 at 6pm

Hi everyone, We would like to invite you to a Stream Processing Meetup at LinkedIn’s *Mountain View campus on Tuesday, August 23 at 6pm*. Please RSVP here (only if you intend to attend in person): https://www.meetup.com/Stream-Processing-Meetup-LinkedIn/events/232864129 We have three great talks

Re: [kafka-clients] [VOTE] 0.10.0.1 RC2

+1 (binding) Thanks Ismael! On Thu, Aug 4, 2016 at 6:54 AM, Ismael Juma wrote: > Hello Kafka users, developers and client-developers, > > This is the third candidate for the release of Apache Kafka 0.10.0.1. This > is a bug fix release and it includes fixes and improvements from 53 JIRAs > (inc

Kafka/Samza meetup at LinkedIn, June 15, 6pm

[bcc: users@kafka.apache.org, d...@kafka.apache.org] Hi everyone, We would like to invite you to our first Stream Processing Meetup at LinkedIn on June 15 at 6pm. Please RSVP here: http://www.meetup.com/Stream-Processing-Meetup-LinkedIn/events/231454378 Going forward (at LinkedIn) we will host m

Re: Wiki Karma

You should have access now. On Thu, Feb 18, 2016 at 12:09 PM, Christian Posta wrote: > Can someone add Karma to my user id for contributing to the wiki/docs? > userid is 'ceposta' > > thanks! > > -- > *Christian Posta* > twitter: @christianposta > http://www.christianposta.com/blog > http://fabr

Re: Kafka response ordering guarantees

> > Does this mean that when a client is sending more than one in-flight >> request per connection, the server does not guarantee that responses will >> be sent in the same order as requests? > > > No - the server does provide this guarantee - i.e., responses will always > be sent in the same order

Re: Kafka response ordering guarantees

> Does this mean that when a client is sending more than one in-flight > request per connection, the server does not guarantee that responses will > be sent in the same order as requests? No - the server does provide this guarantee - i.e., responses will always be sent in the same order as reques

Re: [kafka-clients] 0.9.0.1 RC1

+1 On Thu, Feb 11, 2016 at 6:55 PM, Jun Rao wrote: > This is the first candidate for release of Apache Kafka 0.9.0.1. This a > bug fix release that fixes 70 issues. > > Release Notes for the 0.9.0.1 release > https://home.apache.org/~junrao/kafka-0.9.0.1-candidate1/RELEASE_NOTES.html > > *** Ple

Re: Question about offset expiration

Hi Matvey, I have a question about the config value offsets.retention.minutes in kafka > 0.9.0.0. > Is this the timeout for when offsets get compacted in the topic or actually > deleted (as it appears from a cursory reading of > > https://github.com/apache/kafka/blob/0.9.0/core/src/main/scala/kaf

Re: trouble upgrading from 0.8.2.1 to 0.9.0.0: invalid message

Hi Dave, This change was introduced in https://issues.apache.org/jira/browse/KAFKA-1755 for compacted topics. > > Interestingly, none of the messages currently going to the topic use > message > compaction (i.e. they all have empty keys), although at some time in the > past > I may have sent a fe

Re: Kafka APIs version

That is definitely no longer true. A number of requests are at version 1 or higher. I will file a jira for this. On Wed, Jan 20, 2016 at 8:30 AM, tao xiao wrote: > Hi team, > > In the Kafka protocol wiki it states that version 0 is the only supported > version in all APIs. I want to know if this

Re: Create Kafka Topic Programatically

https://issues.apache.org/jira/browse/KAFKA-2945 and related jiras is probably what you are looking for. That is planned for the next release. On Wed, Jan 20, 2016 at 8:41 AM, Tommy Becker wrote: > This works, but it's clumsy and has limitations. Unfortunately, I'm not > aware of any alternative

Re: KAFKA-1499 compression.type

Yes you are right - thanks for pointing it out. We will get that fixed. Joel On Fri, Jan 15, 2016 at 10:25 AM, Elias Levy wrote: > Anyone? > > On Thu, Jan 14, 2016 at 8:42 PM, Elias Levy > wrote: > > > The description of the compression.type config property in the > > documentation is somewhat

Re: Does quota requires 0.9.X clients?

I'm pretty sure it should work - you may want to give it a try locally though. We did add a throttle-time field in the responses but that will only be included in responses for requests from 0.9.x clients. 0.8.x requests will just get throttled at the broker and will get an 0.8.x format response wi

Re: log compaction scaling with ~100m messages

Using log compaction is well-suited for applications that use Kafka directly and need to persist some state associated with its processing. So something like offset management for consumers is a good fit. Another good use-case is for st

Re: Dealing with large messages

The best practice I think is to just put large objects in a blob store and have messages embed references to those blobs. Interestingly we ended up having to implement large-message-support at LinkedIn but for various reasons were forced to put messages inline (i.e., against the above recommendatio

Re: [kafka-clients] [VOTE] 0.8.2.2 Candidate 1

+1 binding On Thu, Sep 3, 2015 at 9:22 AM, Jun Rao wrote: > This is the first candidate for release of Apache Kafka 0.8.2.2. This only > fixes two critical issues (KAFKA-2189 and KAFKA-2308) related to snappy in > 0.8.2.1. > > Release Notes for the 0.8.2.2 release > https://people.apache.org/~jun

Re: New producer in production

It is pretty safe :) and adopting it now will save you the trouble of migrating later. At LinkedIn, we are using the new producer for all the producers that the Kafka team directly own - this means mirror-maker pipelines, REST proxies and some auditing tools. We have not yet migrated other produce

Re: Offset not committed

e: > >> > >> >just caught this error again. I issue commitOffsets - no error but no > >> >committng offsets either. __consumer_offsets watching shows no new > >> >messages > >> >either. Then in a few minutes I issue commitOffsets again - all

Re: Fwd: Offset not committed

hen another question - why does it seem to > depend on the number of consumed messages? > > On Tue, Jul 14, 2015 at 11:36 AM, Joel Koshy wrote: > > > Can you take a look at the kafka commit rate mbean on your consumer? > > Also, can you consume the offsets topic while you

Re: Fwd: Offset not committed

Can you take a look at the kafka commit rate mbean on your consumer? Also, can you consume the offsets topic while you are committing offsets and see if/what offsets are getting committed? (http://www.slideshare.net/jjkoshy/offset-management-in-kafka/32) Thanks, Joel On Tue, Jul 14, 2015 at 11:1

Re: some problem abort isrshrinkrate

I'm assuming you mean you shut down the follower and the leader did not reflect this in isr shrink-rate. I think the reason for this is that the controller would have sent a leaderAndIsr request to the (existing leader) with the shrunk ISR. On the (redundant) leader transition the leader updates t

Re: Indication in FetchResponse that fetch size is too small

A somewhat related request came up in another thread and I think it is reasonable to provide this. However, there are already some indicators that you can use: - The consumer iterator throws a MessageSizeTooLargeException if it cannot extract any messages out of the next chunk. - If you are using

Re: EOL JDK 1.6 for Kafka

+1 On Wednesday, July 1, 2015, Harsha wrote: > Hi, > During our SSL Patch KAFKA-1690. Some of the reviewers/users > asked for support this config > > https://docs.oracle.com/javase/8/docs/api/javax/net/ssl/SSLParameters.html#setEndpointIdentificationAlgorithm-java.lang.String- >

Re: data loss - replicas

ze: 431 > magic: 0 compresscodec: NoCompressionCodec crc: 895349554 > > Thanks > > > On 06/23/2015 04:52 AM, Joel Koshy wrote: > >The replicas do not have to decompress/recompress so I don't think > >that would contribute to this. > > > >There may be so

Re: Is trunk safe for production?

Yes new features are a big part of it and sometimes bug fixes/improvements. Bug fixes are mostly due to being on trunk, but some aren't necessarily introduced on trunk. For e.g., we would like to do a broader roll-out of the new producer, but KAFKA-2121 (adding a request timeout to NetworkClient)

Re: data loss - replicas

The replicas do not have to decompress/recompress so I don't think that would contribute to this. There may be some corner cases such as: - Multiple unclean leadership elections in sequence - Changing the compression codec for a topic on the fly - different brokers may see this config change at

Re: Increased replication factor. Replication didn't happen!

This may help: http://kafka.apache.org/documentation.html#basic_ops_increase_replication_factor On Thu, Jun 11, 2015 at 11:20:05AM +0800, Shady Xu wrote: > Right now, Kafka topics do not support changing replication factor or > partition number after creation. The kafka-reassign-partitions.sh too

Re: offset storage as kafka with zookeeper 3.4.6

> Is it mandatory to use the zookeeper that comes with kafka for offset > storage to be migrated to kafka? If you want to "move" offsets from zookeeper to Kafka then yes you need to have a phase where all consumers in your group set dual commit to true. If you are starting a fresh consumer group th

Re: Consumer lag lies - orphaned offsets?

On Fri, Jun 05, 2015 at 12:53:00AM -0400, Otis Gospodnetić wrote: > Hi Joel, > > On Thu, Jun 4, 2015 at 8:52 PM, Joel Koshy wrote: > > > Hi Otis, > > > > Yes this is a limitation in the old consumer. i.e., a number of > > per-topic/partition mbeans remai

Re: Consumer lag lies - orphaned offsets?

Hi Otis, Yes this is a limitation in the old consumer. i.e., a number of per-topic/partition mbeans remain even on a rebalance. Those need to be de-registered. So if you stop consuming from some partition after a rebalance that lag mbean currently remain which is why it remains flat. This is a kn

Re: Kafka Not Commiting Messages

I think you can also set this dynamically via an mbean (kafka.Log4jController) On Thu, May 28, 2015 at 08:26:00PM +, Jiangjie Qin wrote: > Actually the name should be log4j.logger.kafka.network.RequestChannel$ > It should be there in 0.8.2.1. Can you check it again? > > From: Charlie Mason ma

Re: Waiting for replication factor to take effect...?

There are also various mbeans you can poke to check the replica fetcher lag but from your description it appears that the fetcher quit for some reason (since you mentioned the topic is 'pretty much' empty) BTW, the below is only for partition 0 > Do you have state change logs available? You shoul

Re: Waiting for replication factor to take effect...?

Do you have state change logs available? You should have a become follower state transition on 9255216 and a leader state transition at around the same time on 925537. The server log should also show 9255216 starting a replica fetcher thread to the leader. Do you have request logging available? Yo

Re: Is fetching from in-sync replicas possible?

well? > > On Tue, May 26, 2015 at 10:44 PM, Joel Koshy wrote: > > > > Apologies if this question has been asked before. If I understand things > > > correctly a client can only fetch from the leader of a partition, not > > from > > > an (in-sync) replic

Re: Is fetching from in-sync replicas possible?

ednesday, May 27, 2015 10:50 AM > To: users@kafka.apache.org > Subject: Re: Is fetching from in-sync replicas possible? > > On May 26, 2015, at 1:44 PM, Joel Koshy wrote: > > >> Apologies if this question has been asked before. If I understand things > >> correctly a cli

Re: Is fetching from in-sync replicas possible?

> Apologies if this question has been asked before. If I understand things > correctly a client can only fetch from the leader of a partition, not from > an (in-sync) replica. I have a use case where it would be very beneficial > if it were possible to fetch from a replica instead of just the leade

Re: Replica manager exception in broker

ollower didn't fetch any data at all from leader the log file size in > follower didn't grow for several hours > > On Sat, May 23, 2015 at 12:40 AM, Joel Koshy wrote: > > > When you say "keeps getting below exception" I'm assuming that the > > error of

Re: Mirrormaker stops consuming

ngle data channel shared between all > producer and consumer threads..? > > Thanks, > Raja. > > > On Fri, May 22, 2015 at 12:12 PM, Joel Koshy wrote: > > > The threaddump suggests that one of the producers > > (mirrormaker-producer-6) is blocked on write for

Re: Replica manager exception in broker

When you say "keeps getting below exception" I'm assuming that the error offset (206845418) keeps changing - right? We saw a similar issue in the past and it turned out to be due to a NIC issue - i.e., it negotiated at a low speed. So the replica fetcher couldn't keep up with the leader. i.e., whil

Re: Mirrormaker stops consuming

The threaddump suggests that one of the producers (mirrormaker-producer-6) is blocked on write for some reason. So the data-channel for that producer (which sits between the consumers and the producer) is full which blocks the consumers from progressing. This appears to be in your (custom) SSLSock

Re: New Producer API - batched sync mode support

> Fine grained tracking of status of individual events is quite painful in > contrast to simply blocking on every batch. Old style Batched-sync mode > has great advantages in terms of simplicity and performance. I may be missing something, but I'm not so convinced that it is that painful/very d

Re: New Producer API - batched sync mode support

e if I am wrong). > > -roshan > > > > On 4/27/15 1:36 PM, "Joel Koshy" wrote: > > >This sounds like flush: > >https://cwiki.apache.org/confluence/display/KAFKA/KIP-8+-+Add+a+flush+meth > >od+to+the+producer+API > > > >which was recently imple

Re: New Producer API - batched sync mode support

This sounds like flush: https://cwiki.apache.org/confluence/display/KAFKA/KIP-8+-+Add+a+flush+method+to+the+producer+API which was recently implemented in trunk. Joel On Mon, Apr 27, 2015 at 08:19:40PM +, Roshan Naik wrote: > Been evaluating the perf of old and new Produce APIs for reliable

Re: New Offset Management API Question

> 1) Does Offset Commit/Fetch API works with Simple Consumer ? Yes - in 0.8.2.1. There is an example given at https://cwiki.apache.org/confluence/display/KAFKA/Committing+and+fetching+consumer+offsets+in+Kafka > 2) With MM, can you selectively MM offset topic. > > Use Case: > Let's suppose you

Re: High Replica Max Lag

o them, others vary > between 5 to 14). > > Thanks > Zakee > > > > > On Mar 13, 2015, at 3:25 PM, Joel Koshy wrote: > > > > I think what people have observed in the past is that increasing > > num-replica-fetcher-threads has diminishing returns f

Re: High Replica Max Lag

I think what people have observed in the past is that increasing num-replica-fetcher-threads has diminishing returns fairly quickly. You may want to instead increase the number of partitions in the topic you are producing to. (How many do you have right now?) On Fri, Mar 13, 2015 at 02:48:17PM -07

Re: Log cleaner patch (KAFKA-1641) on 0.8.2.1

+1 - if you have a way to reproduce that would be ideal. We don't know the root cause of this yet. Our guess is a corner case around shutdowns, but not sure. On Fri, Mar 13, 2015 at 03:13:45PM -0700, Jun Rao wrote: > Is there a way that you can reproduce this easily? > > Thanks, > > Jun > > On

Re: moving replications

I think what you may be looking for is being discussed here: https://cwiki.apache.org/confluence/display/KAFKA/KIP-6+-+New+reassignment+partition+logic+for+rebalancing On Wed, Mar 04, 2015 at 12:34:30PM +0530, sunil kalva wrote: > Is there any way to automate > On Mar 3, 2015 11:57 AM, "sunil kalv

Re: high level consumer rollback

This is not possible with the current high-level consumer without a restart, but the new consumer (under development) does have support for this. On Wed, Mar 04, 2015 at 03:04:57PM -0500, Luiz Geovani Vier wrote: > Hello, > > I'm using the high level consumer with auto-commit disabled and a > sin

Re: Trying to get kafka data to Hadoop

I think the camus mailing list would be more suitable for this question. Thanks, Joel On Wed, Mar 04, 2015 at 11:00:51AM -0500, max square wrote: > Hi all, > > I have browsed through different conversations around Camus, and bring this > as a kinda Kafka question. I know is not the most orthodo

Re: Why The Division Between Scala And Java

We will eventually only have Java clients. For your specific question: javaapi.SimpleConsumer and consumer.SimpleConsumer - there are some arguments that contain scala-specific constructs. E.g., scala maps which cannot be created in Java. This is why we expose a javaapi variant which takes Java co

Re: Simple Consumer and offsets

on restart > > was a viable option, while continuing to use the High Level Consumer for > > our normal operations. Not sure if there is a better way that is compatible > > across 0.8.1 and 0.8.2. > > -Suren > > > > > > On Thursday, February 19, 2015

Re: Commit offset with SimpleConsumer in 0.8.2

Can you add yourself as a watcher on KAFKA-1729? I will update that when I fix the example on the wiki. On Sun, Feb 22, 2015 at 10:16:44PM +0100, Jochen Mader wrote: > I have a hard time figuring out how to do a commit using API 0.8.2 on JDK 8. > > I tried using the examples from 0.8.1.1. > > Fi

Re: ability to delete inactive queues?

We do support delete topic. However, this is a client(admin) operation that is done via zookeeper. It would be useful to do this automatically on the broker-side. Can you file a jira for this? It is not very straightforward to implement this since you would want to check across all partitions whic

Re: Simple Consumer and offsets

Yeah that is a good point - will do the update as part of the doc changes in KAFKA-1729 On Thu, Feb 19, 2015 at 09:26:30PM -0500, Evan Huus wrote: > On Thu, Feb 19, 2015 at 8:43 PM, Joel Koshy wrote: > > > If you are using v0 of OffsetCommit/FetchRequest then you can issue >

Re: Simple Consumer and offsets

; Joel/All, > The SimpleConsumer constructor requires a specific host and port. > > Can this be any broker? > If it needs to be a specific broker, for 0.8.2, should this be the offset > coordinator? For 0.8.1, does it matter? > -Suren > > > On Thursday, Febru

Re: Consuming a snapshot from log compacted topic

February 2015 at 18:47, Joel Koshy wrote: > > > > If I consumed up to the log end offset and log compaction happens in > > > between, I would have missed some messages. > > > > Compaction actually only runs on the rolled over segments (not the > > active - i.e

Re: New Consumer Offset management in 0.8.2

Yes it is supported in 0.8.2-beta. It is documented on the site - you will need to set offsets.storage to kafka. On Thu, Feb 19, 2015 at 03:57:31PM -0500, Matthew Butt wrote: > I'm having a hard time figuring out if the new Kafka-based offset > management in the high-level Scala Consumer is implem

Re: Consuming a snapshot from log compacted topic

gt; To confirm then, the log-end-offset is the same as the cleaner point? > > > > > > > > > > > > On 19 February 2015 at 03:10, Jay Kreps wrote: > > > > > > > Yeah I was thinking either along the lines Joel was suggesting or else > >

Re: Simple Consumer and offsets

> On Thursday, February 19, 2015 10:25 AM, Joel Koshy > wrote: > > > Not sure what you mean by using the SimpleConsumer on failure > recovery. Can you elaborate on this? > > On Thu, Feb 19, 2015 at 03:04:47PM +, Suren wrote: > > Haven't used eithe

Re: Simple Consumer and offsets

to use the > SimpleConsumer on failure recovery to set the offsets. > Is that the recommended approach for this use case? > Thanks. > -Suren > > > On Thursday, February 19, 2015 9:40 AM, Joel Koshy > wrote: > > > Are you using it from Java or Scala? i.e.

Re: Simple Consumer and offsets

On Tuesday, February 17, 2015 12:22 PM, Joel Koshy > wrote: > > > Hi Chris, > > In 0.8.2, the simple consumer Java API supports committing/fetching > offsets that are stored in ZooKeeper. You don't need to issue any > ConsumerMetadataRequest for this. Unfortu

Re: Consuming a snapshot from log compacted topic

consumer timeout set to -1, it takes some time > to query the max offset values, which is still long enough for more > messages to arrive. Got it - thanks for clarifying. > > > > On 18 February 2015 at 23:16, Joel Koshy wrote: > > > > You are also correct and pe

Re: Consuming a snapshot from log compacted topic

> You are also correct and perceptive to notice that if you check the end of > the log then begin consuming and read up to that point compaction may have > already kicked in (if the reading takes a while) and hence you might have > an incomplete snapshot. Isn't it sufficient to just repeat the che

Re: Simple Consumer and offsets

Hi Chris, In 0.8.2, the simple consumer Java API supports committing/fetching offsets that are stored in ZooKeeper. You don't need to issue any ConsumerMetadataRequest for this. Unfortunately, the API currently does not support fetching offsets that are stored in Kafka. Thanks, Joel On Mon, Feb

Re: consumer lag metric

There are FetcherLagMetrics that you can take a look at. However, it is probably easiest to just monitor MaxLag as that reports the maximum of all the lag metrics. On Fri, Feb 13, 2015 at 05:03:28PM +0800, tao xiao wrote: > Hi team, > > Is there a metric that shows the consumer lag of a particula

Re: offset migration from kafka to zookeeper

led=false. > >2. After consuming messages for a while shutdown the consumer and change > >setting dual.commit.enabled=true > >3. bounce the consumer and run for while. The lag looks good > >4. change setting offsets.storage=zookeeper and bounce the consumer. > >Starting from

Re: offset migration from kafka to zookeeper

12:18 AM, Joel Koshy wrote: > > > Actually I meant to say check that is not increasing. > > > > On Thu, Feb 12, 2015 at 08:15:01AM -0800, Joel Koshy wrote: > > > Possibly a bug - can you also look at the MaxLag mbean in the consumer > > > to verify that the

Re: offset migration from kafka to zookeeper

Actually I meant to say check that is not increasing. On Thu, Feb 12, 2015 at 08:15:01AM -0800, Joel Koshy wrote: > Possibly a bug - can you also look at the MaxLag mbean in the consumer > to verify that the maxlag is zero? > > On Thu, Feb 12, 2015 at 11:24:42PM +0800, tao xiao w

Re: offset migration from kafka to zookeeper

r == ErrorMapping.NoError) > > offsetMap.put(topicAndPartition, offsetAndMetadata.offset) > > else { > > println("Could not fetch offset for %s due to %s.".format( > topicAndPartition, ErrorMapping.exceptionFor(offsetAndMetadata

Re: offset migration from kafka to zookeeper

There are mbeans named KafkaCommitsPerSec and ZooKeeperCommitsPerSec - can you look those up and see what they report? On Thu, Feb 12, 2015 at 07:32:39PM +0800, tao xiao wrote: > Hi team, > > I was trying to migrate my consumer offset from kafka to zookeeper. > > Here is the original settings of

Re: Java APIs - which one to use?

You should use the producer under o.a.k.c. The new consumer implementation is not available in 0.8.2 (although the APIs are there) so you would need to use the kafka.javaapi classes for the consumer. We plan to deprecate kafka.javaapi eventually. Thanks, Joel On Wed, Feb 11, 2015 at 11:42:04AM +

Re: Lack of JMX LogCleaner and LogCleanerManager metrics

+1 On Tue, Feb 10, 2015 at 01:32:13PM -0800, Jay Kreps wrote: > I agree that would be a better name. We could rename it if everyone likes > Compactor better. > > -Jay > > On Tue, Feb 10, 2015 at 9:33 AM, Gwen Shapira wrote: > > > btw. the name LogCleaner is seriously misleading. Its more of a

Re: question about new consumer offset management in 0.8.2

On Thu, Feb 05, 2015 at 11:57:15PM -0800, Joel Koshy wrote: > On Fri, Feb 06, 2015 at 12:43:37AM -0500, Jason Rosenberg wrote: > > I'm not sure what you mean by 'default' behavior 'only if' offset.storage > > is kafka. Does that mean the 'default&

Re: question about new consumer offset management in 0.8.2

; Finally, why is section 5.6 titled "Distribution"? Seems to be a grab-bag > of mostly consumer related topics? Yes this was prior structure that can be improved. > > > > > On Thu, Feb 5, 2015 at 2:21 PM, Joel Koshy wrote: > > > > > > > This is do

Re: How to fetch old messages from kafka

> We can reset the offset and get first 10 messages, but since we need to back > in reverse sequence, suppose user has consumed messages upto 100 offset , > currently there are only last 10 messages are visible, from 100 -90, now I > want to retrieve messages from 80 to 90, how can we do that?

Re: How to delete defunct topics

There are mbeans (http://kafka.apache.org/documentation.html#monitoring) that you can poke for incoming message rate - if you look at those over a period of time you can figure out which of those are likely to be defunct and then delete those topics. On Thu, Feb 05, 2015 at 02:38:27PM -0800, Jagbi

Re: Issue with topic deletion

with the default > >> num.partitions and > >> > >> > replication.factor. Did you try stopping the consumer first and > >> issue > >> > >> > the topic delete. > >> > >> > -Harsha > >> > >> > > >> > >> > On Tue, Feb 3, 2015, at 08:37 P

Re: question about new consumer offset management in 0.8.2

dirtiness threshold has been met). The compaction policy is also documented on the site. Thanks, Joel > On Thu, Feb 5, 2015 at 2:21 PM, Joel Koshy wrote: > > > This is documented in the official docs: > > http://kafka.apache.org/documentation.html#distributionimpl > > &g

Re: generics type for Producer and Consumer do not need to match?

There has to be an implicit contract between the producer and consumer. The K, V pairs don't _need_ to match but generally _should_. If producer sends with the consumer may receive as long as it knows how to convert those raw bytes to . In the example if CK == byte[] and CV == byte[] it is effect

Re: Get Latest Offset for Specific Topic for All Partition

https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-HowdoIaccuratelygetoffsetsofmessagesforacertaintimestampusingOffsetRequest? However, you will need to issue a TopicMetadataRequest first to discover the leaders for all the partitions and then issue the offset request. On Thu, Feb 05, 2015

Re: question about new consumer offset management in 0.8.2

This is documented in the official docs: http://kafka.apache.org/documentation.html#distributionimpl On Thu, Feb 05, 2015 at 01:23:01PM -0500, Jason Rosenberg wrote: > What are the defaults for those settings (I assume it will be to continue > using only zookeeper by default)? > > Also, if I hav

Re: Issue with topic deletion

y for the topic "test1" while > >> > > >deletion of topic going on. > >> > > > > >> > > > >> > > Yes it is the case. However, after a small period of time (say few > >> > > minutes) > >> > &

Re: Issue with topic deletion

Thanks for the logs - will take a look tomorrow unless someone else gets a chance to get to it today. Joel On Tue, Feb 03, 2015 at 04:11:57PM -0800, Sumit Rangwala wrote: > On Tue, Feb 3, 2015 at 3:37 PM, Joel Koshy wrote: > > > Hey Sumit, > > > > I thought you would

Re: Turning on cleanup.policy=compact for a topic => not starting cleanup ?

re exactly how to do that or what thread > I'd be looking for specifically... Found a suggestion to run > > Jstack -l > jstack.out > > So I did that, and looked for anything containing "Clean" or "clean" and no > matches. > > I wi

Re: Turning on cleanup.policy=compact for a topic => not starting cleanup ?

stacktrace in the broker log if this is the case. > > > -Original Message- > From: Joel Koshy [mailto:jjkosh...@gmail.com] > Sent: Tuesday, February 03, 2015 3:07 PM > To: users@kafka.apache.org > Subject: Re: Turning on cleanup.policy=compact for a topic => not starting &

Re: Issue with topic deletion

Hey Sumit, I thought you would be providing the actual steps to reproduce :) Nevertheless, can you get all the relevant logs: state change logs and controller logs at the very least and if possible server logs and send those over? Joel On Tue, Feb 03, 2015 at 03:27:43PM -0800, Sumit Rangwala wro

Re: Turning on cleanup.policy=compact for a topic => not starting cleanup ?

- Can you check the log cleaner logs? - Do you have any compressed messages in your log? Or messages without a key? - By default it is in a log-cleaner.log file unless you modified that. - Can you take a thread-dump to see if the log cleaner is still alive? - Also, there is an mbean that you can

Re: kafka-web-console goes down regularly

Can you contact the maintainer directly? http://github.com/claudemamo/kafka-web-console/issues On Tue, Feb 03, 2015 at 12:09:46PM -0800, Sa Li wrote: > Hi, All > > I am currently using kafka-web-console to monitor the kafka system, it get > down regularly, so I have to restart it every few hours

Re: Can't create a topic; can't delete it either

creation. Since the exact setup is > different I will a start another thread will the the information. > > Sumit > > On Thu, Jan 29, 2015 at 1:29 AM, Joel Koshy wrote: > > > > If you can tell me where the find the logs I can check. I haven't > > restarted &

  1   2   3   4   >