Kafka 0.9 producer doesn't work

Hello. I set up the Kafka testbed environment on my VirtualBox, which simply has a Kafka broker. I tested the simple consumer & producer scripts, aka kafka-console-consumer.sh and bin/kafka-console-producer.sh respectively, and both of them worked fine. I could see the output from the consumer si

Re: security: encryption at rest and key rotation idea

Hi Josh, Kafka will/can expire message logs after a certain TTL. You can't simply rely on expiration for key rotation? That is, you start to produce messages with a different key while your consumer temporarily handles the overlap of keys for the duration of the TTL. Just an idea, Jens

Re: Possible Error Code for Kafka API Calls

Thank you guys for filling the details in the wiki, it really helps On Tue, Jan 19, 2016 at 2:07 AM, Ismael Juma wrote: > That would be even better Gwen. > > Ismael > I'm wondering if the protocol docs can be auto-generated from our code to a > large extent. Or if we can enhance our protocol def

Re: Support customized security protocol

Tao, I have granted you the access. Guozhang On Tue, Jan 19, 2016 at 7:56 PM, Connie Yang wrote: > @Ismael, what's the status of the SASL/PLAIN PR, > https://github.com/apache/kafka/pull/341? > > > > On Tue, Jan 19, 2016 at 6:25 PM, tao xiao wrote: > > > The PR provides a new SASL mech but i

Re: Support customized security protocol

@Ismael, what's the status of the SASL/PLAIN PR, https://github.com/apache/kafka/pull/341? On Tue, Jan 19, 2016 at 6:25 PM, tao xiao wrote: > The PR provides a new SASL mech but it doesn't provide a pluggable way to > implement user's own logic to do authentication. So I don't think the PR > w

Re: Support customized security protocol

The PR provides a new SASL mech but it doesn't provide a pluggable way to implement user's own logic to do authentication. So I don't think the PR will meet my need. I will write a KIP to open the discussion. p.s. Ismael, can you grant me the permission to create a KIP in Kafka space? On Wed, 2

Re: Support customized security protocol

Hi Tao, The other way would be to implement a SASL provider: https://docs.oracle.com/javase/8/docs/technotes/guides/security/sasl/sasl-refguide.html#PROV This would still require Kafka to be changed, some of the changes are in the following PR: https://github.com/apache/kafka/pull/341 As per t

Re: Support customized security protocol

Hi Ismael, BTW looks like I don't have the permission to add a KIP in Kafka space. Can you please grant me the permission? On Wed, 20 Jan 2016 at 09:40 tao xiao wrote: > Hi Ismael, > > Thank you for your reply. I am happy to have a writeup on this. > > Can you think of any other ways to make se

Re: Support customized security protocol

Hi Ismael, Thank you for your reply. I am happy to have a writeup on this. Can you think of any other ways to make security protocol pluggable instead of extending ChannelBuilder? On Wed, 20 Jan 2016 at 02:14 Ismael Juma wrote: > Hi Tao, > > As you say, security protocols are not currently plu

security: encryption at rest and key rotation idea

We are trying to deploy kafka into EC2 and one of the requirement from infosec is to have kafka encryption at rest (stored with encrypted value). We also need to be able to rotate encryption keys and re-encrypt all the messages on regular basis since we are a financial company. The re-encryption

MirrorMaker 0.9 as source to 0.8.2.1 as destination issue.

Hi , I am trying to run MirrorMaker from 0.9broker as source and 0.8.2.1 broker as destination. MirrorMaker is run from source(I.e. Using 0.9’s code) I am getting the following error message after it sends out first message. ERROR Uncaught error in kafka producer I/O thread: (org.apache.kaf

kafka version compatibility

is it possible to use the kafka 0.8 java client with kafka brokers running 0.9? is it possible to use the kafka 0.9 client with kafka brokers 0.8.2?

RE: Partitions and consumer assignment

My mistake, I missed the carriage return -Original Message- From: Heath Ivie [mailto:hi...@autoanything.com] Sent: Tuesday, January 19, 2016 2:58 PM To: users@kafka.apache.org Subject: RE: Partitions and consumer assignment I get a 404 for this page. Is there an updated link? -Orig

RE: Partitions and consumer assignment

I get a 404 for this page. Is there an updated link? -Original Message- From: Jason J. W. Williams [mailto:jasonjwwilli...@gmail.com] Sent: Tuesday, January 19, 2016 1:33 PM To: users@kafka.apache.org Subject: Re: Partitions and consumer assignment Thanks Luke. I'll read through that.

Re: Apache Kafka Topic Design

Hi again Joe, I would suggest keeping things simple and using a single topic and a single partition with replication in the beginning if possible. You can split your topic up into multiple topics later if needed. You can split up your topic into more partitions later if needed. If you need strict

Re: Cluster of two nodes?

Hi, Regarding Zookeeper, it will not give you any higher availability running 2 instances instead of 1. You will need three Zookeeper instances to be able to loose one. Two Zookeeper instances will simply make your Zookeeper ensemble run slower without any benefit. Running two nodes/brokers of Ka

Re: kafka version compatibility

Hi Craig, On 19 Jan 2016 20:20, "craig w" wrote: > > is it possible to use the kafka 0.8 java client with kafka brokers running > 0.9? Yes. > is it possible to use the kafka 0.9 client with kafka brokers 0.8.2? No. Ismael

Re: Partitions and consumer assignment

Thanks Luke. I'll read through that. -J On Tue, Jan 19, 2016 at 9:21 AM, Luke Steensen < luke.steen...@braintreepayments.com> wrote: > Hi Jason, > > You might find this blog post useful: > > http://www.confluent.io/blog/how-to-choose-the-number-of-topicspartitions-in-a-kafka-cluster/ > > - Luke

Re: MirrorMaker —new.producer

> Without —new.producer, kafka-console-producer.sh seems to produce byte messages that cause OldProducer to error with java.lang.ClassCastException: [B cannot be cast to java.lang.String Ah sorry, I just realized that’s probably because my producer.properties settings had serializer.class=kafka.se

kafka version compatibility

is it possible to use the kafka 0.8 java client with kafka brokers running 0.9? is it possible to use the kafka 0.9 client with kafka brokers 0.8.2?

Re: Can't save Kafka offset in Zookeeper

0.9 brokers should be backwards compatible, yes. But as with everything -- you should verify in your own environment. -Dana On Tue, Jan 19, 2016 at 9:55 AM, Krzysztof Zarzycki wrote: > Thank you Dana! > I see... > The pity is that Hortonworks claims in their release notes of HDP 2.3.2, > that:

Re: Support customized security protocol

Hi Tao, As you say, security protocols are not currently pluggable. `ChannelBuilder` is already an interface, but `SecurityProtocol` is an enum, which makes it hard for users to add additional security protocols. Changing this would probably require a KIP: https://cwiki.apache.org/confluence/disp

Re: Can't save Kafka offset in Zookeeper

Thank you Dana! I see... The pity is that Hortonworks claims in their release notes of HDP 2.3.2, that: 5.9. Kafka HDP 2.3.2 provides Kafka 0.8.2, with no additional Apache patches. ( https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.2/bk_HDP_RelNotes/content/patch_kafka.html ) So I assume

Re: Apache Kafka Topic Design

Hi Jens, Thanks for your suggestion. So here is a bit more text: I have a List of signals for a specific device: Map[DeviceId, List[Signal]], where Signal is case class Signal(name: String, value: String) Now I will write a producer to connect to the device and read these signals at constant in

Re: Can't save Kafka offset in Zookeeper

Sadly HDP 2.3.2 shipped with a broken OffsetCommit api (the 0.8.2-beta version). You should use the apache releases, or upgrade to HDP 2.3.4.0 or later. -Dana On Tue, Jan 19, 2016 at 12:03 AM, Krzysztof Zarzycki wrote: > Hi Kafka users, > I have an issue with saving Kafka offsets to Zookeeper t

Re: commitSync CommitFailedException

Hey Franco, This time I'll answer briefly ;) 1) Heartbeats also get invoked when you call another blocking operation such as commitSync(). 2) If all consumers in the group die, the coordinator doesn't really do anything other than clean up some group state. In particular, it does not remove offse

Re: Partitions and consumer assignment

Hi Jason, You might find this blog post useful: http://www.confluent.io/blog/how-to-choose-the-number-of-topicspartitions-in-a-kafka-cluster/ - Luke On Sat, Jan 16, 2016 at 1:30 PM, Jason Williams wrote: > Hi Franco, > > Thank you for the info! It is my reading of the docs that adding > parti

MirrorMaker —new.producer

Hi all, I finally have a need to understand MirrorMaker well. I’m running Kafka 0.8.2.2. I see that my version of MirrorMaker has a —new.producer option, which uses NewShinyProducer instead of OldProducer. Without —new.producer, kafka-console-producer.sh seems to produce byte messages that caus

Re: trouble upgrading from 0.8.2.1 to 0.9.0.0: invalid message

I'll try again and look for logged errors. The problem I saw with kafka-console-producer.sh did occur with a mixed version cluster, so that is likely the problem. I'll try again with the 0.8.2.1 version. Thanks, Dave On 1/19/2016 2:50 AM, Ismael Juma wrote: Hi Dave, Do you get any errors lo

Cluster of two nodes?

Hello, is there any value in running zookeeper and kafka in a cluster of two nodes? I.e. one instance of zookeeper and kafka on each node?

Re: Apache Kafka Topic Design

Hi Joe, I think you are leaving out all your requirements to be able to get decent answer from anyone here. On Tue, Jan 19, 2016 at 8:47 AM, Joe San wrote: > I soon realized that this is a doomed approach. > Could you elaborate? Why would it be doomed? > Is there a better way to model my top

Broker Exception: Attempt to read with a maximum offset less than start offset

Hi all, Is KAFKA-725 Broker Exception: Attempt to read with a maximum offset less than start offset still valid ? We are seeing a similar issue when we are carrying out the yahoo's streaming-benchmarks

Kafka cluster performance degradation (Kafka 0.8.2.1)

Hi all, We are using Kafka in production and we have been facing some performance degradation of the cluster, apparently after the cluster is a bit "old". We have our production cluster which is up and running since 31/12/2015 and performance tests on our application measuring a full round trip

Re: trouble upgrading from 0.8.2.1 to 0.9.0.0: invalid message

Hi Dave, Do you get any errors logged in the broker when you get ACK error 2 (InvalidMessage) while producing requests to a mixed version cluster? It would be helpful to see them. With regards to the kafka-console-producer.sh error, did you use the 0.9.0.0 console producer with a mixed version cl

Re: Memory records is not writable in MirrorMaker

Can you please file an issue in JIRA for this? Ismael On Tue, Jan 12, 2016 at 2:40 PM, Meghana Narasimhan < mnarasim...@bandwidth.com> wrote: > Hi, > Came across a similar issue. We are running a 3 node cluster (kafka version > 0.9) and Node 0 also has a few mirror makers running. > When we do a

Can't save Kafka offset in Zookeeper

Hi Kafka users, I have an issue with saving Kafka offsets to Zookeeper through OffsetCommitRequest. It's the same issue I found unanswered on SO, so I kindly borrow the description: http://stackoverflow.com/questions/33897683/cant-save-kafka-offset-in-zookeeper "I've installed Zookeeper and Kafka