Re: kafka connect(copycat) question

2015-12-08 Thread Ewen Cheslack-Postava
Svante, Just to clarify, the HDFS connector relies on some Avro translation code which is in a separate repository. You need the https://github.com/confluentinc/schema-registry repository built before the kafka-connector-hdfs repository to get that dependency. Confluent has now also released Conf

Re: Unable to connect to AWS Kafka cluster remotely

2015-12-08 Thread Henrik Martin
I am using the external AWS host name for the advertised.host property. See below. Thanks, /Henrik On 12/8/15 12:54 PM, Gwen Shapira wrote: Sounds like you need to use advertised.host configuration with the external name / ip. This means that the broker will send producers / consumers / zook

Re: Unable to connect to AWS Kafka cluster remotely

2015-12-08 Thread Gwen Shapira
Sounds like you need to use advertised.host configuration with the external name / ip. This means that the broker will send producers / consumers / zookeeper their external address and they will be able to connect. Gwen On Tue, Dec 8, 2015 at 11:17 AM, Henrik Martin wrote: > Greetings. Apologi

Unable to connect to AWS Kafka cluster remotely

2015-12-08 Thread Henrik Martin
Greetings. Apologies for the verbose email, but I'm trying to provide as much relevant detail as possible. I have an Amazon AWS server that is running 1 instance of Zookeeper, and 1 instance of Kafka 0.9.0. As all the AWS servers, it has an internal non routable IP address (172.X.X.X), and an e

Offsets not matching with the count

2015-12-08 Thread bigdata hadoop
Hi All I have a topic with encrypted data. I am using spring-xd streams kafka source and my processor which decrypts the message and writes to file. So the issue I am facing is the offsets or throughput isn't matching with the topic offsets. I also made a tap on the stream for counting the message

Re: Kafka integration with Oracle QA

2015-12-08 Thread Gwen Shapira
Hi, Can you explain a bit more what you'd expect this integration to do? Kafka is a queue, just like Oracle AQ is - so I can see how you may replace Oracle AQ with Kafka, but I'm not sure what you are trying to achieve by integrating them. Gwen On Mon, Dec 7, 2015 at 7:52 PM, CY Kuek wrote: >