I was thinking more about this. Successfully writing a block of msgs to HDFS
represents that atomic commit, downstream. However, it is not a 2 or 3-phase
transaction, with rollback. The issue is the difference in scope of a
downstream aggregate commit and an exactly once upstream commit. I s
Yes, our use is not an atomic commit coordinated between kafka and Hadoop, but
best effort. We err on the side of no duplicates and minimal loss, but the
loss is understood and acceptable, so far. How could that aggregate commit be
made atomic - XA? I wish there was a good high-available, p2p
Robert - "exactly once" will be really hard unless you can commit the
offset with the aggregate in an atomic way downstream. That would defend
you against redeliveries (ignore updates from lower offset) while allowing
you to be more failure tolerant on the consuming side.
On Sat, Feb 15, 2014 at
We have this version in prod. It has been fine as we commitOffsets after every
message. After 3 months of this, we rolled out Hadoop, which requires
aggregation. We started commiting every 2 minutes and we saw that the fetchr
would get tangled and stop fetching and the consumer would get stuc
Thanks Bae. I'll report back with our experiences.
On Sat, Feb 15, 2014 at 10:48 AM, Bae, Jae Hyeon wrote:
> Netflix is using kafka 0.7 and 0.8 with zk 3.4.5, very stable.
>
> On Saturday, February 15, 2014, Todd Palino wrote:
>
> > We're not at the moment, but I'd definitely be interested in
Netflix is using kafka 0.7 and 0.8 with zk 3.4.5, very stable.
On Saturday, February 15, 2014, Todd Palino wrote:
> We're not at the moment, but I'd definitely be interested in hearing your
> results if you do. We're going to be experimenting with the latest version
> soon to evaluate it.
>
> -T
We're not at the moment, but I'd definitely be interested in hearing your
results if you do. We're going to be experimenting with the latest version
soon to evaluate it.
-Todd
On 2/14/14 4:32 PM, "Clark Breyman" wrote:
>Is anyone running 0.8 (or pre-0.8.1) with the latest Zookeeper? Any known
>
It should be api compatible. Not sure how stable ZK 3.4.5 is though.
Thanks,
Jun
On Fri, Feb 14, 2014 at 4:32 PM, Clark Breyman wrote:
> Is anyone running 0.8 (or pre-0.8.1) with the latest Zookeeper? Any known
> compatibility issues? I didn't see any in JIRA but thought I'd give a
> shout.
>
Is anyone running 0.8 (or pre-0.8.1) with the latest Zookeeper? Any known
compatibility issues? I didn't see any in JIRA but thought I'd give a
shout.
