Re: Kafka connection to start from latest offset

2017-03-15 Thread Stephen Durfey
Yes, it is pretty coarse. I have a pull request open for supporting overriding those settings at the connector level, I'm just waiting for it to be pulled. So, if any committers are interested in reviewing/pulling it for me, that would be great :) https://github.com/apache/kafka/pull/2548 On Wed

Re: Kafka connection to start from latest offset

2017-03-15 Thread Aaron Niskode-Dossett
Thank you Stephen! That's a very coarse setting, as you note, since it's at the worker level, but I'll take it. -Aaron On Tue, Mar 14, 2017 at 8:07 PM Stephen Durfey wrote: > Producer and consumer overrides used by the connect worker can be > overridden by prefixing the specific kafka config w

Re: Kafka connection to start from latest offset

2017-03-14 Thread Stephen Durfey
Producer and consumer overrides used by the connect worker can be overridden by prefixing the specific kafka config with either 'producer.' or 'consumer.'. So, you should be able to set 'consumer.auto.offset.reset=latest' in your worker config to do that. http://docs.confluent.io/3.0.0/connect/use

Kafka connection to start from latest offset

2017-03-14 Thread Aaron Niskode-Dossett
Is it possible to start a kafka connect instance that reads from the *latest* offset as opposed to the earliest? I suppose this would be the equivalent of passing auto.offset.reset=earliest to a kafka consumer. More generally, is this something that specific implementations of the kafka connect A