Kafka Replication Factor

Hi All Just double checking If i have created a topic with replication factor 1 does it mean i dont have replication if one of my disks crashes do i loose data? Thanks Avi

Re: Kafka Replication Factor

Hello Avi, Yes, replication factor of 1 means just one lead replica gets assigned for the topic partitions. To achieve HA of messages there are more things to watch out, not only replication factor - metadata/ZK HA, producer acks, min-in-sync replicas, rack awareness for replica assignment to nam

Re: Topic naming convention and common message envelope.

Thanks Ewen! Do you have an example of your namings and maybe how your envelope looks? Just curious. > On 24 июля 2016 г., at 3:41, Ewen Cheslack-Postava wrote: > > On Tue, Jul 19, 2016 at 12:48 AM, Denis Mikhaylov > wrote: > >> Hi, I plan to use Kafka for event-based integration between serv

I had a problem of kerberos??I try to solve for a long time can not be resolved??Please help me??

HI, My Frends: Sorry,Just send pictures don't show.I change to text. I come from China?? I had a problem, I try to solve for a long time can not be resolved. I also found to a similar article, but did not answer. http://permalink.gmane.org/gmane.comp.apache.kafka.devel/1854

Increasing kafka broker throughput

Hello, Some numbers - There is a single topic with 3 producers, 4 brokers, 80 partitions and replication factor of 2. Average produce rate is 45 MB/sec (cumulative) and max is at around 250 MB/sec (cumulative), the producer goes out of memory during peak times (buffer memory set to 10G, batch s

Re:Kafka on DC/OS

I had a question regarding Kafka on DC/OS because whenever we try to install the Kafka package it always shows that it is deploying in the marathon UI and gives us an error when we search for it using "dcos kafka".We have also tried using "dcos package install kafka" and tried to add the brokers

RE: MirrorMaker replication...

Here is a solution. Each region writes to its own topic. MirrorMaker replicates each region's specific topic to every other region. The consumer in all regions consumes from all three topics. This solution requires one write per producer instead of two. N topics (1 per region) and N(N-1) mirror

MirrorMaker replication...

I need a solution that will mirror a topic to a remote location that avoids cycles and a topic per region issue. This solution would require EXACTLY two topics regardless of how many regions (datacenters) it replicates to. I would like the initial producer (P1) to write to only a local (L1) and

"Powered By" - Kafka in iPinYou

iPinYou (www.ipinyou.com.cn/?defaultLocale=en) is the largest DSP in China which has its HQ in Beijing and offices in Shanghai, Guangzhou, Silicon Valley and Seattle. Kafka clusters are the central data hub in iPinYou. All kinds of Internet display advertising data, such as bid/no-bid, impressi

Re: "Powered By" - Kafka in iPinYou

The incremental data every day is more than 80T and the overall data processed every data is around 30P. On Thu, Jul 21, 2016 at 4:13 PM, Glen Cao wrote: > iPinYou (www.ipinyou.com.cn/?defaultLocale=en > ) is the largest DSP in China which has its HQ in Beijing and offices in > Shanghai, Guangz

Kafka Streams: Merging of partial results

Hello Kafka Users, I've been floating this question around the #apache-kafka IRC channel on Freenode for the last week or two and I still haven't reached a satisfying answer. The basic question is: How does Kafka Steams merge partial results? So let me expand on that a bit... Consider the foll

Rebalances despite frequent heartbeats

I'm trying to track down an issue with one of our consumers. There are 4 threads in the same consumer group, which will run happily for a few hours before one of them crashes with the following exception: org.apache.kafka.clients.consumer.CommitFailedException: Commit cannot be completed due to gr

Re: Maximum number of producers per topic per broker

Just found out what's was causing this. Which is quite dangerous. The additional 100 or 200 server had the wrong ssl certificate on them causing it to creat some kind of DOS on the broker. That was the reason why I was seeing that the thread dump was more pointing at the Ssl side. Sent from so

Re: Kafka on DC/OS

Hi Abhimanyu, What errors are you seeing? And which version of DCOS are you running as well? Tim On Fri, Jul 22, 2016 at 6:14 AM, Chakrabarty, Abhimanyu wrote: > I had a question regarding Kafka on DC/OS because whenever we try to install > the Kafka package it always shows that it is deploying

Re: release of 0.10.1

So this begs the question -- why doesn't Kafka join the mainstream and have a 1.0 release? The 0.x versioning makes one think that this code is, at best, beta. Chris From: Ewen Cheslack-Postava To: "users@kafka.apache.org" Date: 07/23/2016 09:03 PM Subject:Re: release of 0

Re: release of 0.10.1

We basically need the regex(java-util regex) support for specifying source topics. On 7/23/16, 7:41 PM, "Ewen Cheslack-Postava" wrote: 0.10.1.0 is considered a major release. The release 0.10.0.0 might have a follow up 0.10.0.1 for critical bug fixes, but 0.10.1.0 is a "minor" rele

Re: Kafka Streams: Merging of partial results

Hi Michael-Keith, Good question. Two answers: in the default case the same key (e.g., "world") would end up in the same partition, so you wouldn't have the example you describe here where the same key is in two different partitions of the same topic. E.g., this default case applies if you are w

how to write kafka connect hdfs parquet sink.

Hi, I have read confluent kafka connect hdfs but I don't want to use schema registry from confluent. I have produced avro encoded bytes to kafka, at that time, I have written my own avro serializer, not used KafkaAvroSerializer

how to write kafka connect hdfs parquet sink.

Hi, I have read confluent kafka connect hdfs but I don't want to use schema registry from confluent. I have produced avro encoded bytes to kafka, at that time, I have written my own avro serializer, not used KafkaAvroSerializer

Mirror maker higher offset in the mirror.

Am trying to mirror from a production Kafka cluster to a DR cluster. However the offsets between topics (retrieved with GetOffsetShell ) on these two clusters do not always match. While a lesser offset is t