Re: Loggly's use of Kafka on AWS

A cheaper approach is to use ephemeral disk and have the replicas across AZs. You can lose up to 2 AZs and still avoid data loss. You mitigate complete region failure using mirror maker to replicate to another region. On Tue, Dec 3, 2013 at 8:41 AM, Philip O'Toole wrote: > Yeah, we use provis

Re: kafka_2.8.0/0.8.0 pom seems invalid

<< Heh, that sounds amazing, considering that's the binary release version you've put up for download :) Not really. The Kafka broker compiled in 2.8.0 Scala is a result of "that is how it runs at LinkedIn" originally (and still I think) and the how it got run everywhere else as a result. So the

How to set kafka path in zk

Hi, If I don't want to register kafka in zk root and I want to make it under a namespace, for example kafka1. If I set only one host in zk property something like 10.237.0.1:2181/kafka, it works. But if I set zk property to 3 zk hosts something like 10.237.0.1:2181/kafka,10.237.0.2:2181/kafka,1

Re: How to set kafka path in zk

ZK namespace is at the every end, not per host. So use 10.237.0.1:2181,10.237.0.2:2181,10.237.0.3:2181/kafka 1 Thanks, Jun On Thu, Dec 5, 2013 at 8:20 AM, Yonghui Zhao wrote: > Hi, > > If I don't want to register kafka

Re: kafka_2.8.0/0.8.0 pom seems invalid

Joe, I expect the vast majority of kafka users are using java for client apps, and in most cases, they will just use the default jars they get in the binary download. I think the confusing thing here, is that there is a single jar file, that includes all the broker, consumer and producer client c

Is there a way to delete partition at runtime?

Hi guys, I found there is a tool to add partition on the fly. My question is, is there a way to delete a partition at runtime? Thanks! Best, Siyuan

Updated kafka client (producer and consumer)

In our environment we use currently use Kafka 0.7.1. The core features I am looking for in a client are this: 1. Provide confirmation of produce requests (or notification of disconnection during requests). 2. Uses asynchronous IO so that: A. Multiple ops can be queued/in-flight at once. B. Fe

Re: Updated kafka client (producer and consumer)

>From the roadmap they published it looks like pipelining as part of the client rewrite is happening post 0.8. Tim On Thu, Dec 5, 2013 at 3:52 PM, Tom Brown wrote: > In our environment we use currently use Kafka 0.7.1. > > The core features I am looking for in a client are this: > > 1. Provide

Preventing topics from appearing on specific brokers

Hello, Say we are using Zookeeper-based Producers, and we specify a topic to be written to. Since we don't specify the actual brokers, is there a way to prevent a topic from appearing on a specific broker? What if we set the topic's partition count to 0 on the broker we don't want it to appear? W

Re: Preventing topics from appearing on specific brokers

We're running 0.72. Thanks, Philip On Thu, Dec 5, 2013 at 4:29 PM, Philip O'Toole wrote: > Hello, > > Say we are using Zookeeper-based Producers, and we specify a topic to be > written to. Since we don't specify the actual brokers, is there a way to > prevent a topic from appearing on a speci

Re: Is there a way to delete partition at runtime?

Hi Siyuan, We do not have a tool to shrink the number of partitions (if that is what you want) for a topic at runtime yet. Could you file a JIRA for this? Guozhang On Thu, Dec 5, 2013 at 2:16 PM, hsy...@gmail.com wrote: > Hi guys, > > I found there is a tool to add partition on the fly. My qu

Trade-off between topics and partitions?

Hello, According to the Kafka FAQ "How do I choose the number of partitions for a topic", clusters with more than 10K partitions are not tested. I am looking for advice on how to scale the number of partitions beyond that. My use case is to publish messages to 1 million users, each with an unique

Fwd: kafka.common.NotLeaderForPartitionException when 1 broker of 2 is down

Hi all, I got kafka.common.NotLeaderForPartitionException when trying to do this: * kafka cluster test1 with 2 brokers: id=1,host=localhost,port=9091; and id=2,host=localhost,port=9092; * create topic:1001, with partition=0/1/2/3; replica=2; * stop broker id=2; * check kafka's zookeeper, topic-100

Re: Preventing topics from appearing on specific brokers

In 0.7, if a topic already exists on at least one broker in a cluster, it won't be created on newly added brokers. Thanks, Jun On Thu, Dec 5, 2013 at 4:29 PM, Philip O'Toole wrote: > Hello, > > Say we are using Zookeeper-based Producers, and we specify a topic to be > written to. Since we don

Re: Trade-off between topics and partitions?

Having 1 million partitions is probably too many. You can still partition by user id with fewer partitions. Events for the same user are still kept in order. It's just that each partition will have multiple users' data. Thanks, Jun On Thu, Dec 5, 2013 at 7:47 PM, mission mission wrote: > Hello

Re: Preventing topics from appearing on specific brokers

Interesting. So if we want to restrict it to a certain set of brokers, we could just create the topic directories by hand on the brokers where we do want the topic, and then bounce the service on those brokers? Philip On Thu, Dec 5, 2013 at 9:18 PM, Jun Rao wrote: > In 0.7, if a topic already

Re: kafka.common.NotLeaderForPartitionException when 1 broker of 2 is down

Any error in the controller and state-change log? Also, 0.8.0 final is just released. Recommend that you try that version if possible. Thanks, Jun On Thu, Dec 5, 2013 at 7:58 PM, 刘洋 wrote: > Hi all, > > I got kafka.common.NotLeaderForPartitionException when trying to do this: > * kafka cluste

Re: Preventing topics from appearing on specific brokers

That's right. Remove the local log dir from brokers that you don't want to have the topic. Thanks, Jun On Thu, Dec 5, 2013 at 9:22 PM, Philip O'Toole wrote: > Interesting. So if we want to restrict it to a certain set of brokers, we > could just create the topic directories by hand on the bro

Re: Preventing topics from appearing on specific brokers

Sweet -- thanks Jun. On Thu, Dec 5, 2013 at 9:25 PM, Jun Rao wrote: > That's right. Remove the local log dir from brokers that you don't want to > have the topic. > > Thanks, > > Jun > > > On Thu, Dec 5, 2013 at 9:22 PM, Philip O'Toole wrote: > > > Interesting. So if we want to restrict it to

Re: Trade-off between topics and partitions?

Deja vu! IMO, what you are describing is a database problem, even though you are talking/thinking about it as a queue problem. I'm sure you could construct something using Kafka (and Samza), but I think you'd have an easier time with a database. The number of pending messages per user and the aver