Re: Connect: SourceTask poll & commit interaction

2017-01-05 Thread Shikhar Bhushan
I have created https://issues.apache.org/jira/browse/KAFKA-4598 for this. On Wed, Dec 14, 2016 at 2:58 PM Shikhar Bhushan wrote: > Hi Mathieu, > > I think you are right, there is currently no mutual exclusion between > `task.commit()` and `task.poll()`. The solution you are thin

Re: Connect: SourceTask poll & commit interaction

2016-12-14 Thread Shikhar Bhushan
`commit()` method carrying the offset map (and possibly deprecate the existing one). Best, Shikhar On Sat, Dec 10, 2016 at 7:57 AM Mathieu Fenniak < mathieu.fenn...@replicon.com> wrote: > Hi Kafka Users, > > I'm looking for a bit of clarification on the documentation for > im

Re: Kafka connect jdbc Sink connector issues when moving from MySQL to Oracle DB

2016-09-30 Thread Shikhar Bhushan
the database, you can use the `fields.whitelist` configuration to whitelist the desired fields. Best, Shikhar On Fri, Sep 30, 2016 at 8:38 AM Srikrishna Alla wrote: > Hi, > > I am facing issues with jdbc Sink Connector when working with Oracle DB. > This functionality was working

Re: Schema for jsonConverter

2016-09-19 Thread Shikhar Bhushan
Hi Srikrishna, The issue is that you are using "name" to specify the field name for the struct's fields. The correct key to use is "field". Best, Shikhar On Thu, Sep 15, 2016 at 4:23 PM Gwen Shapira wrote: > ah, never mind - I just noticed you do use a schema...

Re: Sink Connector feature request: SinkTask.putAndReport()

2016-09-13 Thread Shikhar Bhushan
cords. > > In fact, this mechanism is really more general than just volume based > commits; its really about providing sinks a flexible commit capability > (e.g., some sink event requires premature commit, or otherwise requires > modification of the commit interval). > > On Tue, Sep 1

Re: Sink Connector feature request: SinkTask.putAndReport()

2016-09-13 Thread Shikhar Bhushan
askContext`). Another option could be to add something like "offset.flush.record.count" in addition to the existing "offset.flush.interval.ms". Both options could be configured but whichever happens first would reset the other. What do you think? Best, Shikhar On Fri, Sep

Re: Kafka Connect startup issues

2016-09-12 Thread Shikhar Bhushan
initialization. Best, Shikhar On Mon, Sep 12, 2016 at 5:54 AM Jeff Klukas wrote: > I'm doing some local testing on my Mac to get a feel for Kafka Connect, and > I'm running into several issues. > > First, when I untar the Kafka 0.10.0.1 source and run > `./bin/connect-distri

Re: Kafka Source Connector tasks run only in same node

2016-08-31 Thread Shikhar Bhushan
Hi Dhanuka, For something like a TCP connector where you have a dependency on the placement, the standalone mode makes more sense. With distributed mode you can't have guarantees of what instances your connector's tasks will be distributed to. Best, Shikhar On Wed, Aug 31, 2016

Re: How do I remove myself from mailing list?

2016-08-30 Thread Shikhar Bhushan
Just to confirm: after you unsubscribe by sending an email to users-unsubscr...@kafka.apache.org, you should receive an email titled "confirm unsubscribe from users@kafka.apache.org", which you need to reply to in order for the unsubscribe to take effect. Did you folks do that? On Tue, Aug 30, 201

Re: Kafka ETL for Parquet

2016-08-02 Thread Shikhar Bhushan
ementation here, not all methods are needed for the deserialization bits so it need not be complete. Best, Shikhar On Mon, Aug 1, 2016 at 5:44 PM Kidong Lee wrote: > Thanks for your interest Shikhar, > > Actually, I have questioned and discussed in the thread: > > https://mail-arch

Re: Kafka ETL for Parquet

2016-08-01 Thread Shikhar Bhushan
Er, mislinked HDFS connector :) https://github.com/confluentinc/kafka-connect-hdfs On Mon, Aug 1, 2016 at 3:39 PM Shikhar Bhushan wrote: > Hi Kidong, > > That's pretty cool! I'm curious what this offers over the Confluent HDFS > connector <https://github.com/my

Re: Kafka ETL for Parquet

2016-08-01 Thread Shikhar Bhushan
is functionality should actually be pluggable with Connect by implementing a custom `Converter`, e.g. the SR comes with AvroConverter which acts as the glue. Converter classes can be specified with the `key.converter` and `value.converter` configs. Best, Shikhar On Mon, Aug 1, 2016 at 1:56 AM K

Re: Heartbeating during long processing times

2016-07-01 Thread Shikhar Bhushan
Hi Elias, KIP-62 <https://cwiki.apache.org/confluence/display/KAFKA/KIP-62%3A+Allow+consumer+to+send+heartbeats+from+a+background+thread> has a discussion of current options, and the improvements that are coming. Best, Shikhar On Thu, Jun 30, 2016 at 6:02 PM Elias Levy wrote: > Wh

Re: broker randomly shuts down

2016-07-01 Thread Shikhar Bhushan
This is somewhat specific to your runtime environment, you can check out whatever script is getting used for bringing up Kafka, and where the stderr of the java command is being redirected (hopefully not /dev/null!). On Thu, Jun 30, 2016 at 5:24 PM allen chan wrote: > Hi Shikhar, > I do n

Re: broker randomly shuts down

2016-06-30 Thread Shikhar Bhushan
Perhaps it's a JVM crash? You might not see anything in the standard application-level logs, you'd need to look for the stderr. On Thu, Jun 30, 2016 at 5:07 PM allen chan wrote: > Anyone else have ideas? > > This is still happening. I moved off zookeeper from the server to its own > dedicated VM

Re: Membership protocol used in Kafka cluster

2016-05-31 Thread shikhar
Hi Umesh, It's the latter - ephemeral nodes under /brokers/ids Best, Shikhar On Tue, May 31, 2016 at 8:55 PM Unmesh Joshi wrote: > Hi, > > In Kafka cluster, how do brokers find other brokers? Is there a gossip > style protocol used? or does it use zookeeper ephermal no