Creating-kafka-connect

Hi All, I want to create my own kafka connector which will connect multiple data source. Could anyone please help me in doing so...

Re: Need some help in identifying some important metrics to monitor for streams

Son if I am running my stream and across a cluster of different machine each machine should have a different client id. On 4 Mar 2017 12:36 a.m., "Guozhang Wang" wrote: > Sachin, > > The reason that you got metrics name as > > new-part-advice-d1094e71-0f59-45e8-98f4-477f9444aa91-StreamThread-1 >

Re: TimeBasePartitioner for confluent

Gah…nm…looked at source code…use this: schema.generator.class=io.confluent.connect.storage.hive.schema.TimeBasedSchemaGenerator On 3/3/17, 5:36 PM, "David Garcia" wrote: Trying to user s3-loader and am getting this error: org.apache.kafka.common.config.ConfigException: Invalid gene

TimeBasePartitioner for confluent

Trying to user s3-loader and am getting this error: org.apache.kafka.common.config.ConfigException: Invalid generator class: class io.confluent.connect.storage.hive.schema.DefaultSchemaGenerator at io.confluent.connect.storage.partitioner.TimeBasedPartitioner.newSchemaGenerator(T

Setting up Kafka with zookeeper

Hi, In DEV, Kafka and Zookeeper services can be co- located on the same physical hosts In Prod moving forward do we need to set up Zookeeper on its own cluster not sharing with Hadoop cluster? Can these services be shared within the Hadoop cluster? Additionally if there many topics for Kafka wha

Fast way search data in kafka

I have 6 broker kafka setup. I have retention period of 48 hrs. To debug if certain data has reached kafka or not I am using command line consumer to then piping to grep. But it will take huge amount of time and may not succeed as well. Is there an other way to search something in kafka without

Re: Problem connecting on Kafka cluster through a VPN IPSec

Just an update. It works through OpenVPN instead of Openswan. 2017-03-01 17:44 GMT-03:00 Daniel Aparecido Martins Rosa < dan...@daniel.eti.br>: > Hi All, > Does everyone knows if is it possible to connect on Kafka brokers remotely > using IPSec VPN ? > If I connect through Internet it works fine,

Re: Kafka client and ELB (round-robin DNS)

See http://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/java-dg-jvm-ttl.html On Fri, Mar 3, 2017, 20:57 BGCH wrote: > Hi all, > > briefly about the problem, if bootstrap.servers is a domain name and > that domain name eventually starts pointing to different IP, kafka > client will never t

Kafka client and ELB (round-robin DNS)

Hi all, briefly about the problem, if bootstrap.servers is a domain name and that domain name eventually starts pointing to different IP, kafka client will never try to re-resolve the new IP. In our project we use java-based Kafka client 0.9. We configure bootstrap.servers with the domain n

Re: Need some help in identifying some important metrics to monitor for streams

Sachin, The reason that you got metrics name as new-part-advice-d1094e71-0f59-45e8-98f4-477f9444aa91-StreamThread-1 Is that you did not set the "CLIENT_ID_CONFIG" in your app, and KafkaStreams have to use a default combo of "appID: new-part-advice"-"processID: a UUID to guarantee uniqueness acr

Re: kafka streams offsets.retention.minutes

Hello Tom, I think there is a general solution proposed to tackle KAFKA-3806, which is to re-define the semantics of "offset.retention.minutes" such that we will only start timing it with this config value after the consumer group has been removed at the coordinator side (i.e. all members are gone

Re: how to produce/consume kafka through ssh tunnel

Hi wei, I´m having the similar problem, but though VPN IPSec. If you solve this problem, let me know please. I´ll do the same. Thanks, 2017-03-02 19:44 GMT-03:00 wei : > We have kafka setup on staging environment, when debug the consumer we want > to directly listen to the kafka on staging envir

kafka streams offsets.retention.minutes

I recently realised that our consumers, which don’t use auto commit, will suffer from the issue outlined here (https://issues.apache.org/jira/browse/KAFKA-3806 ). Looking through the kafka streams code it will also suffer from this issue. I kn

Re: Need some help in identifying some important metrics to monitor for streams

Hi Sachin, Now that the confluent platform 3.2 is out, we also have some more documentation on this here: http://docs.confluent.io/3.2.0/streams/monitoring.html . We added a note on how to add other metrics. Yeah, your calculation on pol

Re: understanding consumer rebalance trigger(s)

Hi Jon, On 0.10.0.1 it means that your processing is taking longer than the configured session.timeout.ms. "The timeout used to detect failures when using Kafka's " + "group management facilities. When a consumer's heartbeat is not received within the session timeout, " + "the bro