Re: Match Producer and RecordMetadata with Consumer and ConsumerRecord

2015-12-09 Thread Ewen Cheslack-Postava
John, Your question was a bit confusing because CorrelationID has a specific meaning in the Kafka protocols, but those are an implementation detail that you, as a user of the API, should not need to worry about. CorrelationIDs as defined by the protocol are not exposed to the user (and do not corr

Re: Error while sending data to kafka producer

2015-12-09 Thread Ritesh Sinha
After editing my server.properties. I didn't start kafka again.That was causing the issue.Silly mistake. Thanks a lot Ben for your replies. On Thu, Dec 10, 2015 at 2:27 AM, Ben Stopford wrote: > Hi Ritesh > > You config on both sides looks fine. There may be something wrong with > your truststor

Re: gradle building error

2015-12-09 Thread Sa Li
Hi, Guozhang I re-install gradle, it works now, thanks a lot. SL > On Dec 9, 2015, at 3:47 PM, Guozhang Wang wrote: > > Sa, > > Which command line did you use under what path? > > Guozhang > > On Wed, Dec 9, 2015 at 1:57 PM, Sa Li wrote: > >> Hi, All >> >> I am having such error to build

Re: gradle building error

2015-12-09 Thread Guozhang Wang
Sa, Which command line did you use under what path? Guozhang On Wed, Dec 9, 2015 at 1:57 PM, Sa Li wrote: > Hi, All > > I am having such error to build Kafka, > > * Where: > Build file '/usr/local/kafka/build.gradle' line: 164 > > * What went wrong: > A problem occurred evaluating root project

gradle building error

2015-12-09 Thread Sa Li
Hi, All I am having such error to build Kafka, * Where: Build file '/usr/local/kafka/build.gradle' line: 164 * What went wrong: A problem occurred evaluating root project 'kafka'. > Could not find property 'ScalaPlugin' on project ':clients’. I try to search online, but can’t even find a solut

Re: Error while sending data to kafka producer

2015-12-09 Thread Ben Stopford
Hi Ritesh You config on both sides looks fine. There may be something wrong with your truststore, although you should see exceptions in either the client or server log files if that is the case. As you appear to be running locally, try creating the JKS files using the shell script included he

RE: Match Producer and RecordMetadata with Consumer and ConsumerRecord

2015-12-09 Thread John Menke
Gwen, thanks for the reply: Just to confirm - are we talking about the combination of the IDs that I layed out as being the "CorrelationID" ? I guess the alternative is to pass our own CorrelationID as part of the data. Would you agree with this? I do see something at this address that tal

Re: Error while sending data to kafka producer

2015-12-09 Thread Ritesh Sinha
This is my server config. On Thu, Dec 10, 2015 at 12:19 AM, Ritesh Sinha < kumarriteshranjansi...@gmail.com> wrote: > # Licensed to the Apache Software Foundation (ASF) under one or more > # contributor license agreements. See the NOTICE file distributed with > # this work for additional informa

Re: Error while sending data to kafka producer

2015-12-09 Thread Ritesh Sinha
# Licensed to the Apache Software Foundation (ASF) under one or more # contributor license agreements. See the NOTICE file distributed with # this work for additional information regarding copyright ownership. # The ASF licenses this file to You under the Apache License, Version 2.0 # (the "Licens

Re: Error while sending data to kafka producer

2015-12-09 Thread Ben Stopford
what is your server config? > On 9 Dec 2015, at 18:21, Ritesh Sinha > wrote: > > Hi, > > I am trying to send message to kafka producer using encryption and > authentication.After creating the key and everything successfully.While > passing the value through console i am getting this error: >

Error while sending data to kafka producer

2015-12-09 Thread Ritesh Sinha
Hi, I am trying to send message to kafka producer using encryption and authentication.After creating the key and everything successfully.While passing the value through console i am getting this error: ERROR Error when sending message to topic test with key: null, value: 2 bytes with error: Faile

Re: Match Producer and RecordMetadata with Consumer and ConsumerRecord

2015-12-09 Thread Gwen Shapira
Correlation ID is for a request (i.e. separate ID for produce request and a fetch request), not a record. So it can't be used in the way you are trying to. On Wed, Dec 9, 2015 at 9:30 AM, John Menke wrote: > Can a correlationID be created from a ConsumerRecord that will allow for > identificatio

Re: Doubt regarding Encryption and Authentication using SSL

2015-12-09 Thread Ritesh Sinha
Solved it . Thanks On Wed, Dec 9, 2015 at 10:39 PM, Ritesh Sinha < kumarriteshranjansi...@gmail.com> wrote: > Thanks Ben for your prompt reply. > > But when I am trying to start the producer it throws this error. > > org.apache.kafka.common.KafkaException: Failed to construct kafka producer > at

Match Producer and RecordMetadata with Consumer and ConsumerRecord

2015-12-09 Thread John Menke
Can a correlationID be created from a ConsumerRecord that will allow for identification of the corresponding RecordMetaData instance that was returned from the Producer.send() method? I am Looking at the JavaDocs and the Producer returns RecordMetadata which has the following signature: Record

Re: Doubt regarding Encryption and Authentication using SSL

2015-12-09 Thread Ritesh Sinha
Thanks Ben for your prompt reply. But when I am trying to start the producer it throws this error. org.apache.kafka.common.KafkaException: Failed to construct kafka producer at org.apache.kafka.clients.producer.KafkaProducer.(KafkaProducer.java:321) at org.apache.kafka.clients.producer.KafkaProdu

Re: Questions reg. Kafka apis and proxy

2015-12-09 Thread Guozhang Wang
Achanta, The Java producer APIs has been running in some production environment and is considered stable; the Java consumer APIs in 0.9 is considered beta and may have light modifications in the future. About the metadata-broker-list: this config is only needed to bootstrap the client, hence does

Re: Doubt regarding Encryption and Authentication using SSL

2015-12-09 Thread Ben Stopford
Hi Ritesh You just need to create yourself a text file called client-ssl.properties or similar in the directory your running from. In that file you put your SSL client information. Something like this: security.protocol = SSL ssl.truststore.location = "/var/private/ssl/kafka.client.truststore.

Kafka new producer not able to use ssh tunnel through proxychains4

2015-12-09 Thread Siddhartha
Hi, I have setup a ssh tunnel for cross dc kafka replication using mirrormaker. The following command succeeds: $ proxychains4 /opt/kafka/bin/kafka-console-producer.sh --broker-list 172.19.19.19:9092 --topic test But using the new producer throws an error without producing any message: $ proxyc

Doubt regarding Encryption and Authentication using SSL

2015-12-09 Thread Ritesh Sinha
Hi, I am following the kafka documentation to create encryption and authentication while sending message to kafka by ssl I got stuck at these commands kafka-console-producer.sh --broker-list localhost:9093 --topic test --producer.config *client-ssl.properties* kafka-console-consumer.sh --boot

Questions reg. Kafka apis and proxy

2015-12-09 Thread Achanta Vamsi Subhash
Hi, We are considering hosting Kafka as a service in our company (with multiple clusters and a common interface) . We have the following questions with regards to hosting Kafka as a service: - The recommended Kafka clients and their api have changed since the last 2 releases - producer in 0.8.2 a