Re: Exactly-once publication behaviour

Hi Andrew There are plans to add exactly once behaviour. This will likely be a little more than Idempotent producers with the motivation being to provide better delivery guarantees for Connect, Streams and Mirror Maker. B > On 19 Feb 2016, at 13:54, Andrew Schofield > wrote: > > When pu

Re: Exactly-once publication behaviour

Andrew, In SDC (https://github.com/streamsets/datacollector ) we do the kind of offset management you mention to achieve this type of behav

Exactly-once publication behaviour

When publishing messages to Kafka, you make a choice between at-most-once and at-least-once delivery, depending on whether you wait for acknowledgments and whether you retry on failures. In most cases, those options are good enough. However, some systems offer exactly-once reliability too. Altho

Exactly-once publication behaviour

When publishing messages to Kafka, you make a choice between at-most-once and at-least-once delivery, depending on whether you wait for acknowledgments and whether you retry on failures. In most cases, those options are good enough. However, some systems offer exactly-once reliability too. Altho

Re: new consumer still classified as beta in 0.9.0.1?

To clarify, the bug I mentioned has been fixed in 0.9.0.1. -Jason On Fri, Feb 19, 2016 at 1:33 PM, Ismael Juma wrote: > Even though we did not remove the beta label, all significant bugs we are > aware of have been fixed (thanks Jason!). I'd say you should try it out. :) > > Ismael > > On Fri,

Re: new consumer still classified as beta in 0.9.0.1?

Even though we did not remove the beta label, all significant bugs we are aware of have been fixed (thanks Jason!). I'd say you should try it out. :) Ismael On Fri, Feb 19, 2016 at 9:06 PM, allen chan wrote: > My company is waiting for the new consumer to move out of "beta" mode > before using

Re: new consumer still classified as beta in 0.9.0.1?

Tough to answer. Definitely the rate of reported bugs has fallen. Other than the one Becket found a few weeks back, I haven't seen anything major since the start of the year. My advice would probably be "proceed with caution." -Jason On Fri, Feb 19, 2016 at 1:06 PM, allen chan wrote: > My compa

new consumer still classified as beta in 0.9.0.1?

My company is waiting for the new consumer to move out of "beta" mode before using it. Does anyone know if it is still considered beta? -- Allen Michael Chan

Re: Kafka Streams: Possible to achieve at-least-once delivery with Streams?

On Thu, Feb 18, 2016 at 8:03 PM, Jason Gustafson wrote: > The consumer is single-threaded, so we only trigger commits in the call to > poll(). As long as you consume all the records returned from each poll > call, the committed offset will never get ahead of the consumed offset, and > you'll have

Re: Partitions not assigned on topic auto creation

Hi Venkatesan, Autocreation of topics happens when the broker receives a topic metadata request. That should mean that both topics get created when the consumer does the initial poll() since that is the first time that topic metadata would be fetched (fetching topic metadata allows the consumer an

Partitions not assigned on topic auto creation

I am running into a curious problem with auto creation of topics and would appreciate any insight into what could be causing it. Some context first: We are using Confluent 2.0 platform and I have an integration test where the following happens: 1. A worker thread is spun up and it creates a

[ANNOUCE] Apache Kafka 0.9.0.1 Released

The Apache Kafka community is pleased to announce the release for Apache Kafka 0.9.0.1. This is a bug fix release that fixes 70 issues reported in 0.9.0.0. All of the changes in this release can be found: https://www.apache.org/dyn/closer.cgi?path=/kafka/0.9.0.1/RELEASE_NOTES.html Apache Kafka is

Re: Java Client connection errors with Kafka 0.9.0.0 when SSL is enabled

We have added the client public certs into broker truststore and vice versa. We removed the keystone related properties from client code and tried with ssl.client.auth as requested and none as well. We are still getting same error. Please let us know what else we can use to try On Fri, Feb 19, 201

Kafka 0.9.0.0 on Solaris 5.10 not running

Hi, I'm desperately trying to run Kafka 0.9.0.0 on Solaris 5.10 (Kafka 0.8.2 is working fine). I use 1 Broker with default configuration just as in the quickstart guide. The Broker is not working and I see SocketTimeoutExceptions in the controller.log. Kafka is running on Java 1.8, Filesyst

Re: ConsumerGroupCommand and ConsumerOffsetChecker

I am not familiar with sarama. You can use zkcli.sh from zookeeper distribution and find if anything exists in path /chroot/consumers where chroot is located in your bootstrap.servers properties if you set it. If you dont find anything in /consumers offset is most likely managed in kafka. Then you

kafka protocol BNF file ?

Hi there! Any one know if there is an official bnf grammar file for download ? --- so that I don't have to do copy-pastes from the wiki page Regards -Zaiming

Re: ConsumerGroupCommand and ConsumerOffsetChecker

I suppose the offset is stored in kafka not in zk, because the newest sarama's offset manager support it, and it shown a topic named __consumer_offsets. Could you please point where is the zk path of /chroot/consumer? I'm a newbie of zk and kafka. On Fri, Feb 19, 2016 at 4:11 PM, tao xiao wrote:

Re: ConsumerGroupCommand and ConsumerOffsetChecker

I'd assume the offset is managed in zk. then kafka-run-class.sh kafka.admin.ConsumerGroupCommand --list --zookeeper localhost:2181 should output the consumer group that is saved in zk. can you check what is stored in zk path /chroot/consumers? On Fri, 19 Feb 2016 at 16:06 Amoxicillin wrote: > I

Re: ConsumerGroupCommand and ConsumerOffsetChecker

I wrote the consumer by myself in golang using Shopify's sarama lib updated of master branch. On Fri, Feb 19, 2016 at 4:03 PM, tao xiao wrote: > which version of consumer do you use? > > > On Fri, 19 Feb 2016 at 15:26 Amoxicillin wrote: > > > I tried as you suggested, but still no output of any

Re: ConsumerGroupCommand and ConsumerOffsetChecker

which version of consumer do you use? On Fri, 19 Feb 2016 at 15:26 Amoxicillin wrote: > I tried as you suggested, but still no output of any group info. > > On Fri, Feb 19, 2016 at 2:45 PM, tao xiao wrote: > > > That is what I mean alive. If you use new consumer connecting to broker > > you s