When we evaluated MirrorMaker last year we didn't find any risk of data
loss, only duplicate messages in the case of a network partition.
Did you discover data loss in your tests, or were you just looking at the
docs?
On Fri, 1 May 2015 at 4:31 pm Jiangjie Qin
wrote:
> Which mirror maker version
It'll be officially ready only in version 0.9.
Aditya
From: Mohit Gupta [success.mohit.gu...@gmail.com]
Sent: Thursday, April 30, 2015 8:58 PM
To: users@kafka.apache.org
Subject: Java Consumer API
Hello,
Kafka documentation ( http://kafka.apache.org/doc
Which mirror maker version did you look at? The MirrorMaker in trunk
should not have data loss if you just use the default setting.
On 4/30/15, 7:53 PM, "Joong Lee" wrote:
>Hi,
>We are exploring Kafka to keep two data centers (primary and DR) running
>hosts of elastic search nodes in sync. One k
Roshan,
If I understand correctly, you just want to make sure a number of messages
has been sent successfully. Using callback might be easier to do so.
Public class MyCallback implements Callback {
public Set failedSend;
@Override
Public void onCompletion(RecordMetadata me
Hello,
Kafka documentation ( http://kafka.apache.org/documentation.html#producerapi
) suggests using only Producer from kafka-clients ( 0.8.2.0 ) and to use
Consumer from the packaged scala client. I just want to check once if the
Consumer API from this client is ready for production use.
--
Be
Hi,
We are exploring Kafka to keep two data centers (primary and DR) running hosts
of elastic search nodes in sync. One key requirement is that we can't lose any
data. We POC'd use of MirrorMaker and felt it may not meet out data loss
requirement.
I would like ask the community if we should lo
Hey all,
I am attempting to create a topic which uses 8GB log segment sizes, like so:
./kafka-topics.sh --zookeeper localhost:2181 --create --topic perftest6p2r
--partitions 6 --replication-factor 2 --config max.message.bytes=655360
--config segment.bytes=8589934592
And am getting the following e
Which Kafka version are you using?
On Thu, Apr 30, 2015 at 4:11 PM, Dillian Murphey
wrote:
> Scenerio with 1 node broker, and 3 node zookeeper ensemble.
>
> 1) Create topic
> 2) Delete topic
> 3) Re-create with same name
>
> I'm noticing this recreation gives me Leader: non, and Isr: as empty.
>
@Gwen, @Ewen,
While atomicity of a batch is nice to have, it is not essential. I don't
think users always expect such atomicity. Atomicity is not even guaranteed
in many un-batched systems let alone batched systems.
As long as the client gets informed about the ones that failed in the
batch.. th
Scenerio with 1 node broker, and 3 node zookeeper ensemble.
1) Create topic
2) Delete topic
3) Re-create with same name
I'm noticing this recreation gives me Leader: non, and Isr: as empty.
Any ideas what the deal is here?
I googled around and not being an experienced kafka admin, someone said
I feel a need to respond to the Sqoop-killer comment :)
1) Note that most databases have a single transaction log per db and in
order to get the correct view of the DB, you need to read it in order
(otherwise transactions will get messed up). This means you are limited to
a single producer reading
With reties 1 you still see the 3 secs delay? The idea is, you can change these
property to reduce the time to throw exception to 1 secs or below. Does that
help?
Thanks
Zakee
> On Apr 28, 2015, at 10:29 PM, Madhukar Bharti
> wrote:
>
> Hi Zakee,
>
>> message.send.max.retries is 1
>
> Re
Hello Everyone,
I am quite exited about the recent example of replicating PostgresSQL
Changes to Kafka. My view on the log compaction feature always had been
a very sceptical one, but now with its great potential exposed to the
wide public, I think its an awesome feature. Especially when pulli
Why do we think atomicity is expected, if the old API we are emulating here
lacks atomicity?
I don't remember emails to the mailing list saying: "I expected this batch
to be atomic, but instead I got duplicates when retrying after a failed
batch send".
Maybe atomicity isn't as strong requirement a
2015-04-30 8:50 GMT+03:00 Ewen Cheslack-Postava :
> They aren't going to get this anyway (as Jay pointed out) given the current
> broker implementation
>
Is it also incorrect to assume atomicity even if all messages in the batch
go to the same partition?
I am trying to reproduce this. But if I create a topic, then delete it,
then re-create it, no leader is getting assigned.
I can still produce/consume messages (via command line, basic testing).
Is there some additional cleanup I need to do?
Thanks for your time!
Not sure if this is the best way to do this, but my zookeeper.connect is
set to a DNS alias which points to a load balancer for 3 zookeeper nodes.
I was trying this to see if I could have the kafka config dynamic and allow
me to change/scale whatever I wanted with zookeeper and not have to ever
me
Have you changed
zookeeper.connect=
in server.properties.
A better procedure for replacing zookeeper nodes would be to shutdown one
and install the new one with the same ip. This can easily be done to a
running cluster.
/svante
2015-04-30 20:08 GMT+02:00 Dillian Murphey :
> I had 3 zookeeper
I had 3 zookeeper nodes. I added 3 new ones and shut down the old 3.
The server.log shows Closing socket connection error to the old IPs. I
rebooted the kafka server entirely but it still somehow seems aware of
these servers.
Any ideas what's up?
Running a 1 broker system. I had some issues with the system but got it
working. I've deleted the topic I had trouble with and re-created it.
But describing shows no leader, not producer/consumption works on it.
I create a brand new topic with a name I never used before and I get a
leader. I th
You need to first decide the conditions that need to be met for you to
scale to 50 consumers. These can be as simple as the consumer lag. Look at
the console offset checker tool and see if any of those numbers make sense.
Your existing consumers could also produce some metrics based on which
anothe
On Thu, Apr 30, 2015 at 2:15 AM, Nimi Wariboko Jr
wrote:
> My mistake, it seems the Java drivers are a lot more advanced than the
> Shopify's Kafka driver (or I am missing something) - and I haven't used
> Kafka before.
>
> With the Go driver - it seems you have to manage offsets and partitions
>
What I found was 2 problems.
1. The producer wasn't passing in a partition key, so not all partitions
were getting data.
2. After fixing the producer, I could see all threads getting data
consistently then the shutdown method was
clearly killing the threads.
I have removed the shutdown,and with th
That's part of the new consumer API that hasn't been released yet. The API
happens to be included in the 0.8.2.* artifacts because it is under
development, but isn't yet released -- it hasn't been mentioned in the
release notes, nor is it in the official documentation:
http://kafka.apache.org/docum
24 matches
Mail list logo
