Re: Is re-partition hitless process?

By re-partition you mean adding partitions to an existing topics? There are two things to note in that case: 1. It is "hitless" because all it does is create new partitions where future records can go, it does not actually move data around. 2. You could be "hit" if your consumer code assumes that

Re: All brokers are running but some partitions' leader is -1

We fixed many many bugs since August. Since we are about to release 0.9.0 (with SSL!), maybe wait a day and go with a released and tested version. On Mon, Nov 23, 2015 at 3:01 PM, Qi Xu wrote: > Forgot to mention is that the Kafka version we're using is from Aug's > Trunk branch---which has the

Re: Fetching Offsets Stored in Kafka in 0.9.0

Are you using the new consumer API (KafkaConsumer) or the older ZookeeperConnector? KafkaConsumer has seek() API allowing you to replay events from any point. You can also manually commit a specific offset. Gwen On Tue, Nov 24, 2015 at 2:11 PM, Jack Lund wrote: > We’re running Kafka 0.9.0, and

Re: Change kafka broker ids dynamically

You should definitely use the same id if you still have the data - it makes life so much better. There are 3 common ways to do it: 1. Use the last 3 digits of the IP as the broker ID (assuming Docker gives you the same IP when the container relaunches) 2. Use a deployment manager that can register

Re: Producer property to set to enable async data transfer in kafka 8.2.2

The new producer is async by default. You can see few examples of how to use it here: https://github.com/gwenshap/kafka-examples/tree/master/SimpleCounter/src/main/java/com/shapira/examples/producer/simplecounter On Tue, Nov 24, 2015 at 10:40 AM, Amit Karyekar wrote: > Hi folks, > > We are work

Re: Is re-partition hitless process?

this to do very few (maybe one) partition at a time. On Tue, Nov 24, 2015 at 4:42 PM, Dillian Murphey wrote: > Not adding. Taking some of the partitions from one kafka server and > spreading them to another. > > On Mon, Nov 23, 2015 at 5:40 PM, Gwen Shapira wrote: > > >

Re: All brokers are running but some partitions' leader is -1

together in the same Kafka cluster? > Also we currently run spark streaming job (with scala 2.10) against the > cluster. Any known issues of 0.9.0 are you aware of under this scenario? > > Thanks, > Tony > > > On Mon, Nov 23, 2015 at 5:41 PM, Gwen Shapira wrote: > > >

Re: Flush Messages in KafkaProducer Buffer

In 0.9.0, close() has a timeout parameter that allows specifying how long to wait for the in-flight messages to complete (definition of complete depends on value of "acks" parameter). On Wed, Nov 25, 2015 at 3:58 AM, Muqtafi Akhmad wrote: > Hello guys, > > I am using KafkaProducer (org.apache.ka

Re: Kafka broker goes down when consumer is stopped.

It looks like you have a single broker (with id = 0 ) and that topic1 has a single replica and the broker is alive and well. The socket error is our bug (shouldn't be an error) and doesn't indicate that the broker is down. On Wed, Nov 25, 2015 at 3:26 AM, Shaikh, Mazhar A (Mazhar) < mazhar.sha...

Re: Increasing replication factor reliable?

Yeah - it will increase IO and network utilization by a lot while data is replication, but it should be safe. On Tue, Nov 24, 2015 at 4:56 PM, Dillian Murphey wrote: > Is it safe to run this on an active production topic? A topic was created > without a replication factor of 2 and I want to inc

Re: What is the benefit of using acks=all and minover e.g. acks=3

In your scenario, you are receiving acks from 3 replicas while it is possible to have 4 in the ISR. This means that one replica can be up to 4000 messages (by default) behind others. If a leader crashes, there is 33% chance this replica will become the new leader, thereby losing up to 4000 messages

Re: What is the benefit of using acks=all and minover e.g. acks=3

; Thanks, > Prabhjot > On Nov 28, 2015 10:20 AM, "Gwen Shapira" wrote: > > > In your scenario, you are receiving acks from 3 replicas while it is > > possible to have 4 in the ISR. This means that one replica can be up to > > 4000 messages (by default) behin

Re: New Consumer API + Reactive Kafka

On Wed, Dec 2, 2015 at 10:44 PM, Krzysztof Ciesielski < krzysztof.ciesiel...@softwaremill.pl> wrote: > Hello, > > I’m the main maintainer of Reactive Kafka - a wrapper library that > provides Kafka API as Reactive Streams ( > https://github.com/softwaremill/reactive-kafka). > I’m a bit concerned a

Re: Kafka integration with Oracle QA

Hi, Can you explain a bit more what you'd expect this integration to do? Kafka is a queue, just like Oracle AQ is - so I can see how you may replace Oracle AQ with Kafka, but I'm not sure what you are trying to achieve by integrating them. Gwen On Mon, Dec 7, 2015 at 7:52 PM, CY Kuek wrote: >

Re: Unable to connect to AWS Kafka cluster remotely

Sounds like you need to use advertised.host configuration with the external name / ip. This means that the broker will send producers / consumers / zookeeper their external address and they will be able to connect. Gwen On Tue, Dec 8, 2015 at 11:17 AM, Henrik Martin wrote: > Greetings. Apologi

Re: Match Producer and RecordMetadata with Consumer and ConsumerRecord

Correlation ID is for a request (i.e. separate ID for produce request and a fetch request), not a record. So it can't be used in the way you are trying to. On Wed, Dec 9, 2015 at 9:30 AM, John Menke wrote: > Can a correlationID be created from a ConsumerRecord that will allow for > identificatio

Re: Local Storage

Hi, Kafka *is* a data store. It writes data to files on the OS file system. One directory per partition, and a new file every specific amount of time (you can control this with log.roll.ms). The data format is specific to Kafka. Hope this helps, Gwen On Thu, Dec 17, 2015 at 3:32 PM, Heath Ivie

Re: NPE on 0.9.0.0 Consumer during broker restart

Yes, this looks like a bug. Please file a JIRA :) On Wed, Dec 23, 2015 at 1:08 AM, Enrico Olivelli - Diennea < enrico.olive...@diennea.com> wrote: > Hi, > I'm running a brand new Kafka cluster (version 0.9.0.0). During my tests I > noticed this error at Consumer.partitionsFor during a full cluste

Re: Pulling data from remote server

We don't have a built-in functionality for this, but I know few places that implemented the following architecture. 1. Install a small (1-3 nodes) Kafka cluster on the remote environments (AWS, Rackspace, etc) 2. Use KafkaConnect or similar to pull logs into the local Kafka 3. Install Kafka + Mirr

Re: How to reset a consumer-group's offset in kafka 0.9?

Created this to follow up: https://issues.apache.org/jira/browse/KAFKA-3059 On Mon, Jan 4, 2016 at 9:21 AM, Stevo Slavić wrote: > IMO just command/tool to commit offset for given consumer group and topic > should work (maybe partition can be optional parameter, if partition is not > provided co

Re: stable release?

First, I think the reason 0.8.2.2 is stable and 0.9.0.0 is latest is mostly due to oversight. 0.9.0.0 is stable. Some of the new APIs are considered unstable, but this doesn't imply toward a simple upgrade of the brokers. Regarding issues, you can see what we fixed for 0.9.0.1: https://github.com/

Re: Migrate a topic which has no brokers

Stevo pointed you at the correct document for moving topics around. However, if you lost a broker, by far the easiest way to recover is to start a new broker and give it the same ID as the one that went down. On Tue, Jan 5, 2016 at 8:49 AM, Stevo Slavić wrote: > Hello Ben, > > Yes, you can use

Re: Topic Deletion Issues

Does the issue reproduce consistently? On Tue, Jan 5, 2016 at 10:28 AM, Brenden Cobb wrote: > Thanks Alex (and sorry all for the spam) > > I'll give it another go as suggested, however in my tests I was just > using the kafka-topics command and there was definitely a few seconds > between creati

Re: Topic Deletion Issues

s still obsessing about the last failed deletion attempt and > won't move on to subsequent delete requests. > > I've planned downtime today so I can restart Kafka (clearing the topic > info) and test again. > > On Tue, Jan 5, 2016 at 1:30 PM, Gwen Shapira wrote: >

Re: Migrate a topic which has no brokers

ing kafka instance to take on a new > broker ID? > > Thanks, > > Ben > > On Tue, Jan 5, 2016 at 6:19 PM, Gwen Shapira wrote: > > > Stevo pointed you at the correct document for moving topics around. > > However, if you lost a broker, by far the easiest way to rec

Re: Partition rebalancing after broker removal

Hi, There was a Jira to add "remove broker" option to the partition-reassignment tool. I think it died in a long discussion trying to solve a harder problem... To your work-around - it is an acceptable work-around. Few improvements: 1. Manually edit the resulting assignment json to avoid unneces

Re: Partition rebalancing after broker removal

rifiable > step would be nice. > > Thanks, > Luke > > > On Thu, Jan 14, 2016 at 9:36 AM, Gwen Shapira wrote: > > > Hi, > > > > There was a Jira to add "remove broker" option to the > > partition-reassignment tool. I think it died in a long discuss

Re: Partition rebalancing after broker removal

ssues we > experienced. > > 4. Yep, we rely on exactly this behavior when replacing nodes. It's very > helpful :) > > Thanks! > Luke > > > On Thu, Jan 14, 2016 at 10:07 AM, Gwen Shapira wrote: > > > Hi, > > > > 1. If you had problems with con

Re: Controlled shutdown not relinquishing leadership of all partitions

Do you happen to have broker-logs and state-change logs from the controlled shutdown attempt? In theory, the producer should not really see a disconnect - it should get NotALeader exception (because leaders are re-assigned before the shutdown) that will cause it to get the metadata. I am guessing

Re: Partition rebalancing after broker removal

son file > format. > > > On Thu, Jan 14, 2016 at 10:42 AM, Gwen Shapira wrote: > > > Ah, got it! > > > > There's no easy way to transfer leadership on command, but you could use > > the reassignment tool to change the preferred leader (and nothing else) &

Re: Kafka + ZooKeeper on the same hardware?

It depends on load :) As long as there is no contention, you are fine. On Thu, Jan 14, 2016 at 6:06 AM, Erik Forsberg wrote: > Hi! > > Pondering how to configure Kafka clusters and avoid having too many > machines to manage.. Would it be recommended to run say a 3 node kafka > cluster where you

Re: Possible Error Code for Kafka API Calls

I'm wondering if the protocol docs can be auto-generated from our code to a large extent. Or if we can enhance our protocol definition classes a bit to make them self-documenting (the way we did for configuration). Regarding Dana's suggestion: I think you need special wiki-edit privileges. If you

Re: Possible Error Code for Kafka API Calls

I added what I found in the code comments to the wiki. Note that there are some gaps. For example if anyone can fill in the producer error codes, it will be awesome :) On Mon, Jan 18, 2016 at 9:17 AM, Gwen Shapira wrote: > I'm wondering if the protocol docs can be auto-generated from

Re: SimpleConsumer.getOffsetsBefore() in 0.9 KafkaConsumer

Hi Robert! Jason is the expert, and I hope he'll respond soon. Meanwhile: I think that you can do what you are trying to do by: 1. call position() to get the current position you are consuming 2. call seekToEnd() and then position(), which will give you the last position at the point in which you

Re: No Kafka Error when no Server

Producer.send() by itself will not throw anything. You need to either wait on the future: producer.send().get() Or to use it with a callback that logs the error. On Tue, Jan 26, 2016 at 8:50 AM, Joe San wrote: > Is this strange or wierd? I had no Kafka or Zookeeper running on my local > machin

Re: [DISCUSS] KIP-45 Standardize all client sequence interaction on j.u.Collection.

I have a minor preference toward modifying the API. Because it is source-compatible and protocol-compatible, the only case that will break is if you use client code from one version but run with a JAR from a different version, which sounds like a pretty weird setup in general. Its not a strong pre

Re: HELP PLEASE->Kafka 0.9.0.0 create topic throwing ERROR kafka.admin.AdminOperationException: replication factor: 1 larger than available brokers: 0 with zookeeper 3.4.6

Did you check your brokers are running? On Wed, Jan 27, 2016 at 1:30 PM, Sandhu, Dilpreet wrote: > Hi all, > I am using Kafka 0.9.0.0 with Zookeeper 3.4.6. I am not sure if I > am missing anything :( > When I try to create any topic I get the following error:- > > > Error while executing

Re: Kakfa Connect Converter per Connector

Hi Eric, 1. You are correct that the way to handle custom data formats in Kafka is to use a custom convertor. 2. You are also correct that we are currently assuming one converter per Connect instance / cluster that all connectors share (in the hope that each organization has one common data format

Re: Kafka Committed Offset Behavior off by 1

This is the second time I see this complaint, so we could probably make the API docs clearer. Adam, feel like submitting a JIRA? On Mon, Feb 1, 2016 at 3:34 PM, Adam Kunicki wrote: > Thanks, actually found this out per: > > http://www.confluent.io/blog/tutorial-getting-started-with-the-new-apac

Re: Kafka Committed Offset Behavior off by 1

91&signature=7188d1843f83499b > > > Feel free to assign it to me (wasn't able to do that myself) > > On Mon, Feb 1, 2016 at 9:55 PM, Gwen Shapira wrote: > > > This is the second time I see this complaint, so we could probably make > the > > API docs clearer. > >

Re: at-least-once delivery

MAX_INT is a good value if you want to just block until the buffer has some space (and never get an exception). On Tue, Feb 2, 2016 at 8:08 AM, Franco Giacosa wrote: > Thanks for the information James, the slides are really good. > > One question, in the new producer the property block.on.buffer

Re: 0.9.0.1 RC1

Actually, for releases, committers are non-binding. PMC votes are the only binding ones for releases. On Wed, Feb 17, 2016 at 11:57 AM, Jun Rao wrote: > Christian, > > Similar to other Apache projects, a vote from a committer is considered > binding. During the voting process, we encourage non-c

Re: Java Client connection errors with Kafka 0.9.0.0 when SSL is enabled

props.put("ssl.protocal", "SSL"); <- looks like a typo. On Thu, Feb 18, 2016 at 2:49 PM, Srikrishna Alla < srikrishna.a...@aexp.com.invalid> wrote: > Hi, > > We are getting the below error when trying to use a Java new producer > client. Please let us know the reason for this error - > > Err

Re: Kafka Consumer - Java

See below On Tue, Feb 23, 2016 at 11:45 AM, vivek shankar wrote: > Hello All, > > Can you please help with the below : > > I was reading up on Kafka 0.9 API version and came across the below : > > The following is a draft design that uses a high-available consumer > coordinator at the broker sid

Re: migrating the main-page docs to gitbook format

Hey! Yes! We'd love that too! Maybe you want to help us out with https://issues.apache.org/jira/browse/KAFKA-2967 ? Gwen On Wed, Mar 2, 2016 at 2:39 PM, Christian Posta wrote: > Would love to have the docs in gitbook/markdown format so they can easily > be viewed from the source repo (or mirror

Re: Having trouble to connect StreamSets to Kafka with Kerberos authentication

Hi Michal, Can you succesfully connect to the SASL port without StreamSet? For example using the console consumer as explain here? http://www.confluent.io/blog/apache-kafka-security-authorization-authentication-encryption (the end-to-end example is all the way at the end of the blog) This can hel

Re: What is the best way to ensure connectivity to Kafka without polling any messages

What we normally do is consumer.poll(0). This connects to the broker, finds the consumer group, handles partition assignment, gets the metadata - and then doesn't stick around to actually give you any data. Pretty hacky, but we use this all over the place. Gwen On Tue, Mar 8, 2016 at 12:59 PM, A

Re: [ANNOUNCE] New committer: Damian Guy

Congratulations :) On Fri, Jun 9, 2017 at 1:49 PM Vahid S Hashemian wrote: > Great news. > > Congrats Damian! > > --Vahid > > > > From: Guozhang Wang > To: "d...@kafka.apache.org" , > "users@kafka.apache.org" , > "priv...@kafka.apache.org" > Date: 06/09/2017 01:34 PM > Subject:

Re: [VOTE] 0.11.0.0 RC2

Hi, One super minor issue (that can be fixed without a new RC): The big exactly-once stuff (KIP-98) doesn't actually show up as new features in the release notes. Most chunks appear as sub-tasks, but the new feature itself (KAFKA-4815) is marked as 0.11.1.0 so this is missing. I get that this is c

Making sure all of you know about Kafka Summit

Hi, I figured everyone in this list kinda cares about Kafka, so just making sure you all know. Kafka Summit SF happens in about a week: https://kafka-summit.org/events/kafka-summit-sf/ August 28 in San Francisco. It is not too late to register. The talks are pretty great (and very relevant to e

Call for Paper: Kafka Summit London 2018

ications sent: December 20, 2017 - Presentations due for initial review: March 19, 2018 - Presentations due for final approval: April 9, 2018 I hope to see you in London! Registration will open soon! Gwen Shapira Kafka PMC and conference enthusiast

[ANNOUNCE] New Kafka PMC Member: Rajini Sivaram

Dear Kafka Developers, Users and Fans, Rajini Sivaram became a committer in April 2017. Since then, she remained active in the community and contributed major patches, reviews and KIP discussions. I am glad to announce that Rajini is now a member of the Apache Kafka PMC. Congratulations, Rajini

Re: [VOTE] 1.1.0 RC4

ache/kafka/tree/1.1.0-rc4 > > > > * Documentation: > > http://kafka.apache.org/11/documentation.html > > > * Protocol: > > http://kafka.apache.org/11/protocol.html > > > > Thanks, > > > Rajini > -- *Gwen Shapira* Product Manager | Confluent 650.450.2760 | @gwenshap Follow us: Twitter <https://twitter.com/ConfluentInc> | blog <http://www.confluent.io/blog>

[VOTE] 0.10.0.0 RC0

Hello Kafka users, developers and client-developers, This is the first candidate for release of Apache Kafka 0.10.0.0. This is a major release that includes: (1) New message format including timestamps (2) client interceptor API (3) Kafka Streams. Since this is a major release, we will give people

Re: Confluent or not?

I'm obviously a bit biased, but I'm pretty sure there is zero lock-in. 1. You can use whichever components of the platform you want. If you just need Kafka, you don't *have* to use the schema registry (although, you should ;) 2. Schema Registry, REST Proxy and our connectors are all open source (w

Re: Documentation

Awesome summary, Dana. I'd like to fit this into our docs, but I'm not sure where does step-by-step-description of the protocol fits. Maybe in "Design" section? Just one more thing: 8) At any time, the broker can respond to a fetch request with "Rebalancing" error code, at which point the assignme

Re: Log Retention: What gets deleted

I think you got it almost right. The missing part is that we only delete whole partition segments, not individual messages. As you are writing messages, every X bytes or Y milliseconds, a new file gets created for the partition to store new messages in. Those files are called segments. The segment

Re: Log Retention: What gets deleted

s. In that case after every 1 GB, will it > roll out new log segment file ? > > > > > > <http://log.roll.ms> > > On Fri, Apr 8, 2016 at 11:32 AM Heath Ivie wrote: > > > Gwen, > > > > Thanks for the detailed reply. > > > > That mak

Re: Subscribe on a topic that does not exist?

It depends. If auto.topic.create.enable is true, a topic will be created. If its false, you will get some kind of topic doesn't exist exception. Gwen On Thu, Apr 7, 2016 at 11:49 AM, Shravan Ambati wrote: > Hi, > > I could not find answer to this in the documentation. > In Kafka 0.9.X, what hap

Re: Log Retention: What gets deleted

e than anticipated. > > > > > > On Fri, Apr 8, 2016 at 1:07 PM Gwen Shapira wrote: > > > Yes. It is whichever is shorter :) > > > > Another clarification: > > A segment is deleted as a whole, based on the newest event in the > segment. > > So

Re: Kafka Connector for Solr

Thank you, Surendra. I've added your connector to the Connector Hub page: http://www.confluent.io/developers/connectors On Fri, Apr 22, 2016 at 10:11 PM, Surendra , Manchikanti wrote: > Hi Jay, > > Thanks!! Can you please share the contact person to include this in > Confluent Coneector Hub pag

Re: kafka.common.QueueFullException

few more things you can do: * Increase "batch.size" - this will give you a larger queue and usually better throughput * More producers - very often the bottleneck is not in Kafka at all. Maybe its the producer? or the network? * Increate max.inflight.requests for the producer - it will allow sendi

Re: [ANNOUNCE] New committer: Ismael Juma

Congratulations, very well deserved. On Apr 25, 2016 10:53 PM, "Neha Narkhede" wrote: > The PMC for Apache Kafka has invited Ismael Juma to join as a committer and > we are pleased to announce that he has accepted! > > Ismael has contributed 121 commits >

[VOTE] 0.10.0.0 RC2

Hello Kafka users, developers and client-developers, This is the first candidate for release of Apache Kafka 0.10.0.0. This is a major release that includes: (1) New message format including timestamps (2) client interceptor API (3) Kafka Streams. (4) Configurable SASL authentication mechanisms (5

Re: leader election bug

Thats a good version :) On Mon, May 2, 2016 at 11:04 AM, Kane Kim wrote: > We are running Zookeeper version: 3.4.6-1569965, built on 02/20/2014 09:09 > GMT, does it have any known problems? > > On Fri, Apr 29, 2016 at 2:35 PM, James Brown wrote: > >> What version of ZooKeeper are you on? There h

[VOTE] 0.10.0.0 RC3

Hello Kafka users, developers and client-developers, This is the fourth candidate for release of Apache Kafka 0.10.0.0. This is a major release that includes: (1) New message format including timestamps (2) client interceptor API (3) Kafka Streams. Since this is a major release, we will give peop

Re: [VOTE] 0.10.0.0 RC3

Thu, May 5, 2016 at 1:46 PM, Gwen Shapira wrote: > >> Hello Kafka users, developers and client-developers, >> >> This is the fourth candidate for release of Apache Kafka 0.10.0.0. >> >> This is a major release that includes: (1) New message format >> inclu

Re: gobblin error

You are seeing an Hadoop error in Gobblin app, so the Kafka mailing list is probably not your best bet. That said, Gobblin started a MapReduce job which failed. You need to look at the job log and the task logs for MapReduce to find out what happened. Gwen On Sat, May 7, 2016 at 10:51 AM, Mudit

[VOTE] 0.10.0.0 RC4

Hello Kafka users, developers and client-developers, This is the first candidate for release of Apache Kafka 0.10.0.0. This is a major release that includes: (1) New message format including timestamps (2) client interceptor API (3) Kafka Streams. Since this is a major release, we will give people

Re: [VOTE] 0.10.0.0 RC4

sy work around of this and we need a new > RC. > > Thanks, > Liquan > > On Mon, May 9, 2016 at 6:49 PM, Gwen Shapira wrote: > >> Hello Kafka users, developers and client-developers, >> >> This is the first candidate for release of Apache Kafka 0.10.0.0. This >

Re: [DISCUSS] KIP-59 - Proposal for a kafka broker command - kafka-brokers.sh

Hello Jayesh, Thank you for the suggestion. I like the proposal and the new tool seems useful. Do you already have the tool available in a github repository? If you don't, then this would be a good place to start - there are many Kafka utilities in github repositories (Yahoo's Kafka Manager as a

Re: [VOTE] 0.10.0.0 RC4

Just to confirm: You tested both versions with plain text and saw no performance drop? On Thu, May 12, 2016 at 1:26 PM, Tom Crayford wrote: > We've started running our usual suite of performance tests against Kafka > 0.10.0.0 RC. These tests orchestrate multiple consumer/producer machines to > r

Re: [VOTE] 0.10.0.0 RC4

SASL authentication This may help us nail down the issue source of the issue. Gwen On Thu, May 12, 2016 at 1:38 PM, Tom Crayford wrote: > Yep, confirm. > > On Thu, May 12, 2016 at 9:37 PM, Gwen Shapira wrote: > >> Just to confirm: >> You tested both versions wi

Slides and Video from Kafka Summit are now available

Hey Kafka Community, It was great seeing so many of you at the Kafka summit last month. Hope you had fun and learned a lot. I certainly did. Looking forward to meet all of you again at the next summit :) For those of you who missed the event, or those who attended but are sorry they couldn't see

[VOTE] 0.10.0.0 RC5

Hello Kafka users, developers and client-developers, This is the sixth (!) candidate for release of Apache Kafka 0.10.0.0. This is a major release that includes: (1) New message format including timestamps (2) client interceptor API (3) Kafka Streams. Since this is a major release, we will give pe

Re: Mirrormaker schema exception

It looks like you are using MirrorMaker from 0.9.0.1 while the source broker is older. MirrorMaker needs to be older than the older broker involved in the replication. Gwen On Mon, May 16, 2016 at 2:12 PM, Meghana Narasimhan wrote: > Hi, > I came across the following mirrormaker issue today whi

Re: Slides and Video from Kafka Summit are now available

nk > didn't go anywhere. > Real-Time Analytics Visualized w/ Kafka + Streamliner + MemSQL + ZoomData > > Will this be posted anytime? > > Thanks, > > Ben > > On Fri, May 13, 2016 at 8:05 PM, Gwen Shapira wrote: > >> Hey Kafka Community, >> >> It was g

[VOTE] 0.10.0.0 RC6

Hello Kafka users, developers and client-developers, This is the seventh (!) candidate for release of Apache Kafka 0.10.0.0. This is a major release that includes: (1) New message format including timestamps (2) client interceptor API (3) Kafka Streams. This RC was rolled out to fix an issue with

Re: [COMMERCIAL] Re: [COMMERCIAL] Re: download - 0.10.0.0 RC6

RC is never available in the version information, because the RC we vote on is identical to the version we release. This is Apache rules, not mine :) I am not sure about the MBeans - what is the commitID in previous versions? Gwen On Wed, May 18, 2016 at 10:41 AM, Ramanan, Buvana (Nokia - US) w

Re: Question regarding enhancement for Apache Kafka

Or you can use KafkaStreams, which is already available in Kafka :) On Thu, May 19, 2016 at 2:33 AM, Radoslaw Gruchalski wrote: > Hey, you should have a look at Apache Samza. You put Samza on top of Kafka > and you can inject content filtering rules into a Samza system. This will > give you a "c

[RESULTS] [VOTE] Release Kafka version 0.10.0.0

This vote passes with 9 +1 votes (4 bindings) and no 0 or -1 votes. +1 votes PMC Members: * Jay Kreps * Jun Rao * Guozhang Wang * Joe Stein Committers: * Sriharsha Chintalapani * Ewen Cheslack-Postava Community: * Dana Powers * Vahid S. Hashemian * Ashish Singh Vote thread:https://www.mail-arc

[ANNOUCE] Apache Kafka 0.10.0.0 Released

ang, edoardo, Edward Ribeiro, Eno Thereska, Ewen Cheslack-Postava, Flavio Junqueira, Francois Visconte, Frank Scholten, Gabriel Zhang, gaob13, Geoff Anderson, glikson, Grant Henke, Greg Fodor, Guozhang Wang, Gwen Shapira, Igor Stepanov, Ishita Mandhan, Ismael Juma, Jaikiran Pai, Jakub Nowak, James

Re: FW: [ANNOUCE] Apache Kafka 0.10.0.0 Released

. We are working toward improving the compatibility story in the future. On Tue, May 24, 2016 at 4:42 PM, Andy Davidson < a...@santacruzintegration.com> wrote: > Does anyone know if spark plans to upgrade? > > I think the current version is 0.8x? > > Kind regards > &

Re: macbook air and kafka

Well... We added KafkaConnect and KafkaStreams, thats two fairly big features. On Thu, May 26, 2016 at 11:58 AM, S Ahmed wrote: > I just pulled lated on the same old 2010 MPB and the build took over 4 > minutes. > > Have things changed so much since 2013? :) > > I ran: ./gradlew jar > > On Tue,

Re: Is kafka message timestamp preserved in mirror maker

The intent was definitely as you described, but I think we forgot to actually modify the code accordingly. Do you mind opening a JIRA on the issue? Gwen On Wed, Jun 1, 2016 at 4:13 PM, tao xiao wrote: > Hi, > > As per the description in KIP-32 the timestamp of Kafka message is > unchanged mirr

Re: Does the Kafka Streams DSL support non-Kafka sources/sinks?

Currently this is not part of the DSL and needs to be done separately through KafkaConnect. Here's an example: http://www.confluent.io/blog/hello-world-kafka-connect-kafka-streams In the future we want to integrate Connect and Streams better, so you could do something like "builder.stream(..).from

Re: Does the Kafka Streams DSL support non-Kafka sources/sinks?

ount it as "learning a framework". 2) I believe I've seen people implement "writes to database" in KafkaProcessor. Maybe try to google / search the mailing list? Guozhang and Michael can probably add details. Gwen On Wed, Jun 1, 2016 at 11:31 PM, Avi Flax wrote: > O

Re: Does the Kafka Streams DSL support non-Kafka sources/sinks?

Last time I checked (maybe 10 month ago), Camel was using the old async producer, which is not reliable (no callbacks!). Make sure they improved this before using it in a system where reliability is important. On Mon, Jun 6, 2016 at 9:44 PM, Asaf Mesika wrote: > I'd stay off the Camel. It's perfo

Re: Questions about Kafka Scripts

[A] Unfortunately, we only documented this in the code: /** * For verifying the consistency among replicas. * * 1. start a fetcher on every broker. * 2. each fetcher does the following *2.1 issues fetch request *2.2 puts the fetched result in a shared buffer *2.3 waits for al

Re: error: ... protocols are incompatible with those of existing members ??

Actually, this is exactly what Connect is doing. KafkaConnect uses its own "consumer" protocol called "connect" to distribute tasks between the workers. The default group name for this is connect-cluster, but it is possible to override it in the connect-distributed.properties file. SinkTasks also

Re: Introducing Dory

Dory is pretty cool (even though it is named after a somewhat dorky fish). Thank you for sharing :) On Sun, Jun 12, 2016 at 1:24 AM, Dave Peterson wrote: > Hello Kafka users, > > Version 1.1.0 of Dory is now available. See > https://github.com/dspeterson/dory for details. Dory is the successor

Re: error: ... protocols are incompatible with those of existing members ??

More likely that we didn't think of documenting it :) Do you want to open a JIRA? or submit a doc patch? We should obviously document this limitation, but I'm thinking that the REST API could also validate that connector ID doesn't collide with the distributed worker group. On Mon, Jun 13, 2016

Re: Expired messages in kafka topic

Thats a pretty cool feature, if anyone feels like opening a JIRA :) On Thu, Jun 23, 2016 at 8:46 AM, Christian Posta wrote: > Sounds like something a traditional message broker (ie, ActiveMQ) would be > able to do with a TTL setting and expiry. Expired messages get moved to a > DLQ. > > On Thu, J

Re: kafka + autoscaling groups fuckery

Charity, 1. Nothing you do seems crazy to me. Kafka should be able to work with auto-scaling and we should be able to fix the issues you are running into. There are few things you should be careful about when using the method you described though: 1.1 Your life may be a bit simpler if you have a

Re: Question about bootstrap processing in KafkaStreams.

Upgrade :) On Tue, Jun 28, 2016 at 6:49 PM, Rohit Valsakumar wrote: > Hi Jay, > > Thanks for the reply. > > Unfortunately in our case due to legacy reasons we are using > WallclockTimestampExtractor in the application for all the streams and the > existing messages in the stream probably won¹t ha

Re: Kafka Streams/Connect for Persistence?

In addition, our soon-to-be-released JDBC sink connector uses the Connect framework to do things that are kind of annoying to do yourself: * Convert data types * create tables if needed, add columns to tables if needed based on the data in Kafka * support for both insert and upsert * configurable b

Re: Too Many Open Files

woah, it looks like you have 15,000 replicas per broker? You can go into the directory you configured for kafka's log.dir and see how many files you have there. Depending on your segment size and retention policy, you could have hundreds of files per partition there... Make sure you have at least

Re: Kafka 0.9.0.1 failing on new leader election

you know, I ran into those null pointer exceptions when I accidentally tested Kafka with mismatching version of zkclient. Can you share the versions of both? And make sure you have only one zkclient on your classpath? On Tue, Jul 26, 2016 at 6:40 AM, Sean Morris (semorris) wrote: > I have a setu

Re: Chocolatey packages for ZooKeeper, Kafka?

If anyone packages Kafka with Chocolatey, we'll be happy to add this to our ecosystem page. Currently Apache Kafka only publishes tarballs. Gwen On Thu, Jul 28, 2016 at 6:58 PM, Andrew Pennebaker wrote: > Could we please publish Chocolatey packages for ZooKeeper and Kafka, to > make it easier f

Re: Mirrormaker between 0.8.2.1 cluster and 0.10 cluster

You need to use the old mirrormaker (0.8.2.1) to mirror 0.8.2.1 to 0.10.0.0. This is true in general - always use MirrorMaker from the older release. Because new Kafka can talk to old clients and not the other way around. Gwen On Fri, Jul 29, 2016 at 12:04 AM, Yifan Ying wrote: > Hi all, > > I

<    1   2   3   4   5   6   >