Re: auto.offset.reset for Kafka streams 0.10.2.0

2017-04-10 Thread Matthias J. Sax
Default for Streams is "earliest" cf. https://github.com/apache/kafka/blob/0.10.2.0/streams/src/main/java/org/apache/kafka/streams/StreamsConfig.java#L405 -Matthias On 4/10/17 9:41 PM, Mahendra Kariya wrote: > This was even my assumption. But I had to explicitly specify > auto.offset.reset=late

Re: auto.offset.reset for Kafka streams 0.10.2.0

2017-04-10 Thread Mahendra Kariya
This was even my assumption. But I had to explicitly specify auto.offset.reset=latest. Without this config, it started from "earliest"! On Tue, Apr 11, 2017 at 10:07 AM, Sachin Mittal wrote: > As far as I know default is latest, if no offsets are found. Otherwise it > starts from the offset. > >

Re: auto.offset.reset for Kafka streams 0.10.2.0

2017-04-10 Thread Sachin Mittal
As far as I know default is latest, if no offsets are found. Otherwise it starts from the offset. On Tue, Apr 11, 2017 at 8:51 AM, Mahendra Kariya wrote: > Hey All, > > Is the auto offset reset set to "earliest" by default in Kafka streams > 0.10.2.0? I thought default was "latest". > > I start

auto.offset.reset for Kafka streams 0.10.2.0

2017-04-10 Thread Mahendra Kariya
Hey All, Is the auto offset reset set to "earliest" by default in Kafka streams 0.10.2.0? I thought default was "latest". I started a new Kafka streams application with a fresh application id and it started consuming messages from the beginning.

Kafka best practice on bare metal hardware

2017-04-10 Thread Ali Nazemian
Hi all, I was wondering if there is any benchmark or any recommendation for having physical HW vs virtual for the Kafka Brokers. I am trying to calculate the HW requirements for a Kafka Cluster with a hard SLA. My questions are as follows. - What is the effect of OS disk caching for a Kafka-Broke

Re: [VOTE] 0.10.2.1 RC0

2017-04-10 Thread Gwen Shapira
Thank you for testing!!! On Mon, Apr 10, 2017 at 7:36 AM, Mathieu Fenniak wrote: > Hi Gwen, > > +1, looks good to me. Tested broker upgrades, and connect & streams > applications. > > Mathieu > > > On Fri, Apr 7, 2017 at 6:12 PM, Gwen Shapira wrote: > >> Hello Kafka users, developers and client

My first producer

2017-04-10 Thread Adaryl Wakefield
I'm getting ready to write my first producer in Java and I need some advice/input. I'm going to simulate sucking in the change log of a transactional database. There are six columns of data. Using ncat, I'm going to broadcast this file on a port. I've been looking at Java networking and multith

UNSUBSCIBE ME

2017-04-10 Thread Zac Harvey
UNSUBSCRIBE ME (BTW this doesn't work)

How to check kafka connection status from producer/ consumer

2017-04-10 Thread Nagaraddi .S S
Hello, I am new to Kafka and I am looking for a solution to check kafka connection status. >From configuration, I have only kafka bootstrap servers, I do not know how to use this information to fetch connection status from producer/consumer application. Best regards Reds

Re: Does Kafka elect new follower when the original follower is down?

2017-04-10 Thread Manikumar
As of now, kafka doesn't have the capability to automatically migrate followers to other brokers. We have to use kafka-reassign-partition.sh script. On Sat, Apr 8, 2017 at 2:15 AM, Ramanan, Buvana (Nokia - US/Murray Hill) < buvana.rama...@nokia-bell-labs.com> wrote: > Hello, > > I have a 6 broker

Re: [VOTE] 0.10.2.1 RC0

2017-04-10 Thread Mathieu Fenniak
Hi Gwen, +1, looks good to me. Tested broker upgrades, and connect & streams applications. Mathieu On Fri, Apr 7, 2017 at 6:12 PM, Gwen Shapira wrote: > Hello Kafka users, developers and client-developers, > > This is the first candidate for the release of Apache Kafka 0.10.2.1. This > is a

Re: In kafka streams consumer seems to hang while retrieving the offsets

2017-04-10 Thread Sachin Mittal
I have done these changes and also set ProducerConfig.REQUEST_TIMEOUT_MS_CONFIG, 18 Now producer sometimes fails after 3 minutes earlier it used to fail at 30 seconds (default value). So I was wondering what would be the reason of the same and how high should this value go. Thanks Sachin On

Re: In kafka streams consumer seems to hang while retrieving the offsets

2017-04-10 Thread Eno Thereska
Hi Sachin, In 0.10.2.1 we've changed the default value of max.poll.interval.ms (to avoid rebalancing during recovery) as well as the default value of the streams producer retries (to retry during a temporary broker failure). I think you are aware of the changes, but just double checking. You do

Missing offset value for a Consumer Group

2017-04-10 Thread Ali Nazemian
Hi all, I have an issue with Kafka 0.10.1. I can produce a message to a specific Kafka topic and I am able to see the message using "Kafka-console-consumer". However, when I am trying to consume that message, it is like there is not any message inside the topic. I suspect maybe there is an issue l