Re: Kafka Stretch Clusters

lo cases. Then in practice, since the cross-colo > usually takes more latency, you'd usually get much fewer round-trips than N > across DC before satisfying the acks. And your average/p99 latencies would > not increase much compared with just one cross-DC replica. > > > Gu

Kafka Stretch Clusters

re the potential issues with partition leaders is worth it! Thanks for any insight y'all have, -Andrew Otto Wikimedia Foundation

Re: Newbie looking for a connector I can configure on my mac

ions are CCL and not FOSS, means that organizations that use purely FOSS software (like the Wikimedia Foundation) makes Kafka Connect effectively unusable. Okay carry on! :) - Andrew Otto On Tue, Mar 29, 2022 at 8:27 PM Liam Clarke-Hutchinson wrote: > Hi Andrew, > > So if you

Re: Spark Streams vs Kafka Streams

2021 at 3:50 PM Parthasarathy, Mohan wrote: > Andrew, > > I am not sure I understand. We have built several analytics applications. > We typically use custom aggregations as they are not available directly in > the library. > > -mohan > > > On 4/28/21, 12:12 PM, &q

Re: Spark Streams vs Kafka Streams

I'd assume this is because Kafka Streams is positioned for building streaming applications, rather than doing analytics, whereas Spark is more often used for analytics purposes.

Re: Standard way to get http POST request into a Kafka topic?

/24/wikimedias-event-data-platform-event-intake/> explains how we use EventGate. -Andrew Otto SRE, Wikimedia Foundation On Tue, Apr 27, 2021 at 11:42 PM Ran Lupovich wrote: > Hi, have a look for Rest Proxy component as part of the kafka eco system > > בתאריך יום ד׳, 28 באפר׳ 2021,

Re: Introducing JulieOps, a tool to help you build your GitOps and self-service with Apache Kafka and related

This looks amazing, thanks Pere! I just filed a ticket for Wikimedia to consider adopting JulieOps. On Sat, Feb 27, 2021 at 2:05 PM Pere Urbón Bayes wrote: > Hi, >I hope this message finds everyone healthy and good. I wanted to reach > out to th

Wikimedia's Event Data Platform

be 2 more posts coming out over the next couple of weeks too. Thanks for reading! :) - Andrew Otto Engineer @ The Wikimedia Foundation

Re: Disk space - sharp increase in usage

WMF recently had an issue where Kafka broker disks were filling up with log segment data. It turned out that Kafka was not deleting old log segments because the oldest log segment had a message with a Kafka timestamp a year in the future. Since

Re: ACLs - How To Allow Anyone To Access of A Topic

If I understand correctly, if your client authenticates, there must be an ACL for that principal, otherwise it will fail authorization. If you are going to allow everything anyway, perhaps you don't need to authenticate?

Re: log.message.timestamp.difference.max.ms and future timestamps?

fference works both ways. > > > -Matthias > > > On 4/16/20 9:39 AM, Andrew Otto wrote: > > log.message.timestamp.difference.max.ms is the "The maximum difference > > allowed between the timestamp when a broker receives a message and the > > timestamp specifie

log.message.timestamp.difference.max.ms and future timestamps?

egative in this case. log.message.timestamp.difference.max.ms - futureTimestamp == -bigNumber Will the message be rejected or accepted in this case? Thanks! -Andrew Otto @Wikimedia Foundation

Re: Where to run MM2? Source or destination DC/region?

ource this event is happing. > > -----Original Message- > From: Andrew Otto > Sent: Thursday, January 9, 2020 8:32 AM > To: users@kafka.apache.org > Subject: Re: Where to run MM2? Source or destination DC/region? > > ---External Email--- > > Hi Peter, > > My understa

Re: Where to run MM2? Source or destination DC/region?

Hi Peter, My understanding here comes from MirrorMaker 1, but I believe it holds for MM2 (someone correct me if I am wrong!) For the most part, if you have no latency or connectivity issues, running MM at the source will be fine. However, the failure scenario is different if something goes wrong.

Re: Exposing Kafka on WAN

The trouble is that the producer and consumer clients need to discover the broker hostnames and address the individual brokers directly. There is an advertised.listeners setting that will allow you to tell clients to connect to external proxy hostnames instead of your internal ones, but those prox

Re: Kafka Connect REST connector with additional logging to kafka

Hiya, this doesn’t help answer your question, but as an FYI, Wikimedia has implemented https://github.com/wikimedia/change-propagation to do what you are trying to do: issue HTTP requests (and other things) triggered by incoming messages in Kafka. On Thu, Aug 23, 2018 at 9:45 AM Andrea Spina wrot

Re: Kafka mirror maker help

Hiya, Saravanan, I saw you emailed my colleague Alex about WMF’s old debian packaging. I’ll reply here. We now use Confluent’s Kafka debian packaging which does not (or did not?) ship with init scripts. We don’t use Sys V init.d scripts anymore either, but use systemd instead. Our systemd serv

Re: Using Kafka CLI without specifying the URLs every single time?

Us too: https://github.com/wikimedia/puppet/blob/production/modules/confluent/files/kafka/kafka.sh This requires that the various kafka-* scrips are in your PATH. And then this gets rendered into /etc/profile.d to set env variables. https://github.com/wikimedia/puppet/blob/production/modules/con

Re: Kafka Mirrormaker issue

t of our mirrormakers is the last > one we will upgrade to 1.x, at which point we can just upgrade the > mirrormakers to 1.x as well. > > On 4/6/18, 1:09 PM, "Jeff Field" wrote: > > I'm hitting the same problem, even with the new consumer, on > MirrorMake

Re: Kafka Mirrormaker issue

sure what the default in 0.10 is. On Fri, Mar 30, 2018 at 11:40 AM, Siva A wrote: > Any other update on this? > > On Mon, Mar 26, 2018, 7:42 PM Andrew Otto wrote: > > > I’ve had similar problems, but I don’t have an explanation for ya :/ > > > > On Sun, Ma

Re: Kafka Mirrormaker issue

I’ve had similar problems, but I don’t have an explanation for ya :/ On Sun, Mar 25, 2018 at 12:19 PM, Siva A wrote: > Hi, > > We have 3 nodes Kafka cluster(0.10.0.1) and its mirroring the data from > another 3 node cluster of same Kafka version. > Both the clusters are Kerberized and we are run

Re: Kafka 0.9 MirrorMaker failing with Batch Expired when producing to Kafka 1.0 cluster

anging MirrorMaker instances, I think looking at stack dumps would > help you get closer to the root cause. > > Best regards, > Andras > > On Mon, Mar 12, 2018 at 7:56 PM, Andrew Otto wrote: > > > Hi all, > > > > I’m troubleshooting a MirrorMaker issue, and

Kafka 0.9 MirrorMaker failing with Batch Expired when producing to Kafka 1.0 cluster

://gist.github.com/ottomata/5324fc3becdd20e9a678d5d37c2db872 Any help is appreciated, thanks! -Andrew Otto Senior Systems Engineer Wikimedia Foundation

Re: Cross-cluster mirror making

by the Kafka cluster. If the buffer gets too full, it will have to start dropping messages. - Andrew Otto On Thu, Feb 8, 2018 at 5:21 PM, Husna Hadi wrote: > Hi, I read on The Definitive Guide to Kafka that when using cross-cluster > kafka mirroring, when possible, consume from a

JSONSchema Kafka Connect Converter

to a ConnectRecord, the messages could be used with any Connector out there, right? I might have space in the next year to work on something like this, but I thought I’d ask here first to see what others thought. Would this be useful? If so, is this something that might be upstreamed into Apache Kafka?

Re: offsetsForTimes API performance

Speaking of, has there been any talk of combining those two requests into a single API call? I’d assume that offsetForTimes + consumer seek is probably the most common use case of offsetForTimes. Maybe a round trip could be avoided if the broker could just auto-assign the consumer to the offset f

super.users or kafka-acls —cluster for broker ACLs?

Hi all, Is there any reason not to list broker principals as super.users? I know there is the —cluster shortcut for adding broker ACL permissions via the kafka-acls CLI. Adding them to super.users would be simpler, as it can more easily be done via configuration management. Thanks! - Andrew

Re: Kafka Monitoring..

We’ve recently started using Prometheus, and use Prometheus JMX Exporter to get Kafka metrics into prometheus. Here’s our JMX Exporter config: https://github.com/wikimedia/puppet/blob/production/modules/profile/files/kafka/broker_prometheus_jmx_exporter

Upgrade by replacing brokers?

anything I’m missing? Are there gotchas related with on disk log file formats that might cause some issues? Thanks! - Andrew Otto Systems Engineer, Wikimedia Foundation

Re: Is there any command can be used to get broker.id on one of broker with command?

Best I got for ya, using https://github.com/r4um/jmx-dump $ broker_hostname=localhost $ jmx_port= $ java -jar jmx-dump-0.4.2-standalone.jar -h $broker_hostname -p $jmx_port -m | grep ‘kafka.server:id=.*type=app-info’ kafka.server:id=12,type=app-info This will only work if a JMX port is config

Re: [VOTE] Add REST Server to Apache Kafka

-1 for http kafka client in core Although a read only management interface, perhaps via http, sounds kinda useful for things like health checks as mentioned. On Wed, Oct 26, 2016 at 2:00 PM, Zakee wrote: > -1 > > Thanks. > > On Oct 25, 2016, at 2:16 PM, Harsha Chintalapani > wrote: > > > > Hi

Re: Brokers changing mtime on data files during startup?

The settings controls the max size in bytes of a partion oft he > specified > > > topic. So you can find a good size by checking the size of a partition > > with > > > du –b and use this value. > > > > > > This way you do not loose ~7 days of data and can

Re: Changing default logger to RollingFileAppender (KAFKA-2394)

+1, this is what Wikimedia uses in production. On Thu, Jun 2, 2016 at 10:38 AM, Tauzell, Dave wrote: > I haven't started using this in production but this is how I will likely > setup the logging as it is easier to manage. > > -Dave > > -Original Message- > From: Dustin Cote [mailto:dus.

Re: Relaying UDP packets into Kafka

Super old, but: https://github.com/atdt/UdpKafka On Wed, May 25, 2016 at 4:20 PM, Joe San wrote: > What about this one: https://github.com/agaoglu/udp-kafka-bridge > > On Wed, May 25, 2016 at 6:48 PM, Sunil Saggar > wrote: > > > Hi All, > > > > I am looking for a kafka producer to receive UDP p

Re: Brokers changing mtime on data files during startup?

“We use the default log retention of 7 *days*" :)* On Wed, May 25, 2016 at 12:34 PM, Andrew Otto wrote: > Hiya, > > We’ve recently upgraded to 0.9. In 0.8, when we restarted a broker, data > log file mtimes were not changed. In 0.9, any data log file that was on > disk b

Brokers changing mtime on data files during startup?

Hiya, We’ve recently upgraded to 0.9. In 0.8, when we restarted a broker, data log file mtimes were not changed. In 0.9, any data log file that was on disk before the broker has it’s mtime modified to the time of the broker restart. This causes problems with log retention, as all the files then

Re: Filter plugins in Kafka

If you want something really simple and hacky, you could use kafkatee[1] and kafkacat[2] together: kafkatee.conf: input [encoding=string] pipe tail -f a.log output pipe 1 grep -v ’not this’ | kafkacat -P -b b1:9092 -t mytopic [1] https://github.com/wikimedia/analytics-kafkatee [2] https://github

Re: Apache Kafka Case Studies

Talk I gave about Kafka at the Wikimedia Foundation at Kafka NYC Meetup in 2014. https://www.hakkalabs.co/articles/apache-kafka-wikimedia On Wed, Feb 3, 2016 at 1:56 PM, Joe San wrote: > The OReilly online training seems to be interesting! Is there anything else > that is similiar to this that

Re: MirrorMaker —new.producer

ass=kafka.serializer.StringEncoder. But still, should I be using —new.producer? On Tue, Jan 19, 2016 at 11:50 AM, Andrew Otto wrote: > Hi all, > > I finally have a need to understand MirrorMaker well. I’m running Kafka > 0.8.2.2. I see that my version of MirrorMaker has a —new.producer

MirrorMaker —new.producer

Hi all, I finally have a need to understand MirrorMaker well. I’m running Kafka 0.8.2.2. I see that my version of MirrorMaker has a —new.producer option, which uses NewShinyProducer instead of OldProducer. Without —new.producer, kafka-console-producer.sh seems to produce byte messages that caus

Re: best python library to use?

idential and may also be privileged. If youare not > the > > intended recipient, please notify the sender immediately, and do > > notdisclose > > the contents to another person, use it for any purpose, or store, or > > copythe > > information in any medium. Please also d

Re: best python library to use?

pykafka’s balanced consumer is very useful. pykafka also has Python bindings to the librdkafka C library that you can optionally enable, which might get you some speed boosts. python-kafka (oh, I just saw this 0.9x version, hm!) was better at producing than pykafka for us, so we am currently using

Replica Fetcher Reset Its Offset to beginning

Hi all, This morning I woke up to see a very high max replica lag on one of my brokers. I looked at logs, and it seems that one of the replica fetchers for a partition just decided that its offset was out of range, so it reset its offset to the beginning of the leader’s log and started replica

Re: API to query cluster metadata on-demand

If you don’t mind doing it with a C CLI: https://github.com/edenhill/kafkacat $ kafkacat -L -b mybroker But, uhhh, you probably want a something in the Java API. :) > On Sep 3, 2015, at 13:58, Gwen Shapira wrote: > > Ah, I wish. > > We are working on it :) > > On Thu, Sep 3, 2015 at 9:10

Re: [DISCUSSION] Kafka 0.8.2.2 release?

I agree: keep it simple :) The latest stable version of Kafka right now has a critical bug in it. Fixing that would be good enough. 0.8.2.2 should probably just a maintenance/bugfix release. > On Aug 18, 2015, at 14:29, Edward Ribeiro wrote: > > I sort of follow Daniel Nelson on this issue

Re: Replica not available...when it is!

running the latest version of Camus? > > -Clark > > Sent from my iPhone > >> On Aug 15, 2015, at 10:25 AM, Andrew Otto wrote: >> >> Hm, interesting. So my real issue is more with Camus than with cluster >> problems? It seems that Camus won’t co

Re: Replica not available...when it is!

eans that there > is a replica that is down. > > If you get Leader not available that means the partition is offline. > > -Clark > > Sent from my iPhone > >> On Aug 15, 2015, at 8:41 AM, Andrew Otto wrote: >> >> Also strange: If I start this bro

Re: Replica not available...when it is!

this broker being offline makes the rest of the cluster think that its none of its replicas are available. > On Aug 15, 2015, at 11:18, Andrew Otto wrote: > > I am having trouble with a single broker causing consumers to lag. As I am > troubleshooting this issue, I have stopped t

Replica not available...when it is!

I am having trouble with a single broker causing consumers to lag. As I am troubleshooting this issue, I have stopped this broker in the hopes that other replicas will take over as leader for this broker’s preferred partitions. However, when I do so, Camus reports: kafka.CamusJob: Skipping

Re: 0.8.2.1 upgrade causes much more IO

s to confirm this). > > > > Thanks, > > Matt Bruce > > > > > > *From:* Andrew Otto [mailto:ao...@wikimedia.org] > *Sent:* Tuesday, August 11, 2015 3:15 PM > *To:* users@kafka.apache.org > *Cc:* Dan Andreescu ; Joseph Allemandou < > jalleman...@wiki

Re: Kafka metadata

Note that broker metadata is not necessarily kept in sync with zookeeper on all brokers at all times: https://issues.apache.org/jira/browse/KAFKA-1367 This looks like it is fixed in the upcoming 0.8.3 > On Aug 8, 2015, at 01:08, Abdoulaye Diallo wrote: > > @Rahul > >> If this is true, why do

Re: Kafka Simple Consumer Replicas versus ISR

Hi, I’m not sure, but it is possible the discrepancy you are seeing is related to this: https://issues.apache.org/jira/browse/KAFKA-1367 I’m pretty sure the CLI talks to Zookeeper directly, whereas likely the SimpleConsumer talks to the Brokers. > On Aug 5, 2015, at 13:13, d...@ariens.ca wro

Re: Decomissioning a broker

eaders should happen for you as well. Though, manual reassignment may be > preferred in your case. > > Here is some extra information on controlled shutdowns: > http://kafka.apache.org/documentation.html#basic_ops_restarting > > Thanks, > Grant > > On Thu, Jul 30, 2015 at

Decomissioning a broker

I know that broker will still be registered in Zookeeper. Should I just delete the znode for that broker once it has been shut down? Thanks! -Andrew Otto

Re: Controlled Shutdown Tool?

oker to tell it to do the controlled shutdown. > I also got confused before and had to look at the code to figure that out. > I think it is better if we can add this to the document. > > -Binh > > On Mon, Jul 27, 2015 at 11:50 AM, Andrew Otto wrote: > >> Thanks! &g

Re: Controlled Shutdown Tool?

x retries using controlled.shutdown.max.retries > defaults to 3 . > > > Thanks, > Harsha > > > On July 27, 2015 at 11:42:32 AM, Andrew Otto (ao...@wikimedia.org > <mailto:ao...@wikimedia.org>) wrote: > >> I’m working on packaging 0.8.2.1 for Wikimedia, and in

Controlled Shutdown Tool?

is supposed to set this for a running broker. Do I need a special JMX tool in order to flick this switch? I’d like to add a command to my kafka bin wrapper script so that I can easily use this when restarting brokers. What is the proper way to set controlled.shutdown.enable? Thanks! -Andrew

Kafka partitions unbalanced

Hi all, I’ve recently noticed that our broker log.dirs are using up different amounts of storage. We use JBOD for our brokers, with 12 log.dirs, 1 on each disk. One of our topics is larger than the others, and has 12 partitions. Replication factor is 3, and we have 4 brokers. Each broker th

Re: Alternative to camus

> We are currently using spark streaming 1.2.1 with kafka and write-ahead log. > I will only say one thing : "a nightmare". ;-) I’d be really interested in hearing about your experience here. I’m exploring streaming frameworks a bit, and Spark Streaming is just so easy to use and set up. I’d be

Re: integrate Camus and Hive?

generatePartitionedPath*(JobContext context, String > topic, String brokerId, int partitionId, String *encodedPartition*) { >StringBuilder sb = new StringBuilder(); >sb.append("Create your HDFS custom path here"); >return sb.toString(); >} > &

Re: integrate Camus and Hive?

api/src/main/java/com/linkedin/camus/etl/Partitioner.java > and use configuration "etl.partitioner.class=CLASSNAME" then you can > organize any way you like. > > I hope this helps. > > > Thanks, > > Bhavesh > > > On Wed, Mar 11, 2015 at 8:36 AM, And

Re: integrate Camus and Hive?

TION '${env:TABLE_LOCATION_CAMUS_OUTPUT}' > ; > > > I hope this will help ! You will have to construct hive query according > to partition define. > > Thanks, > > Bhavesh > > On Wed, Mar 11, 2015 at 7:24 AM, Andrew Otto wrote: > >>> Hiv

Re: integrate Camus and Hive?

> Hive provides the ability to provide custom patterns for partitions. You > can use this in combination with MSCK REPAIR TABLE to automatically detect > and load the partitions into the metastore. I tried this yesterday, and as far as I can tell it doesn’t work with a custom partition layout. A

Re: REST/Proxy Consumer access

BTW, Wikimedia uses varnishkafka to produce http requests to Kafka, and we are pretty happy with it. https://github.com/wikimedia/varnishkafka > On Mar 5, 2015, at 13:09, Ewen Cheslack-Postava wrote: > > Yes, Confluent built a REST proxy that gives access to cluster metadata > (e.g. list to

Re: Announcing the Confluent Platform built on Apache Kafka

Wow, .deb packages. I love you. > On Feb 25, 2015, at 14:48, Joseph Lawson wrote: > > This is really awesome stuff. It's great to see y'all growing! Thank you > and congratulations! > > > From: Neha Narkhede > Sent: Wednesday, February 25, 2015 1:3

Re: kafka jmxtrans config

Hm, I’m sure how long ago the kafka jmxtrans example has been updated on the puppet-jmxtrans module. Wikimedia is currently using Kafka 0.8.1.1 with this puppet module’s jmxtrans setup: https://github.com/wikimedia/puppet-kafka/blob/master/manifests/server/jmxtrans.pp > On Feb 12, 2015, at 10

Re: How to push metrics to graphite - jmxtrans does not work

Maybe also set: -Dcom.sun.management.jmxremote.port= ? > On Dec 2, 2014, at 02:59, David Montgomery wrote: > > Hi, > > I am having a very difficult time trying to report kafka 8 metrics to > Graphite. Nothing is listening on and and no data in graphite. If > this method of graphi

Re: powered by kafka

Oo, add us too! The Wikimedia Foundation (http://wikimediafoundation.org/wiki/Our_projects) uses Kafka as a log transport for analytics data from production webservers and applications. This data is consumed into Hadoop using Camus and to other processors of analytics data. > On Nov 10, 2

Re: Cross-Data-Center Mirroring, and Guaranteed Minimum Time Period on Data

Check out Camus. It was built to do parallel loads from Kafka into time bucketed directories in HDFS. On Oct 16, 2014, at 9:32 AM, Gwen Shapira wrote: > I assume the messages themselves contain the timestamp? > > If you use Flume, you can configure a Kafka source to pull data from > Kafka,

Re: Zookeeper reconnect failed due to 'state changed (Expired)'

ader. So, any unreplicated messages that didn't reach the > new leader are lost. If the old leader rejoins ISR, it will also truncate > it's log to follow the new leader's log. > > Thanks, > Neha > > On Wed, Oct 1, 2014 at 5:48 AM, Andrew Otto wrote: > >>

Re: Zookeeper reconnect failed due to 'state changed (Expired)'

ISR? On Sep 30, 2014, at 7:17 PM, Jun Rao wrote: > With ack=1, acked messages could be lost when the leader fails. > > Thanks, > > Jun > > On Mon, Sep 29, 2014 at 8:04 AM, Andrew Otto wrote: > >> This happened again to me this weekend. I've done some sleuth

Re: Zookeeper reconnect failed due to 'state changed (Expired)'

is case GC threads) to > flush the dirty pages. So, all those time in real was spent on disk I/Os, > rather than real GCs. The fix is to tune dirty_expire_centisecs and > dirty_writeback_centisecs > to flush dirty pages more frequently to avoid such drafting. > > Thanks, > > J

Re: ISR differs between Kafka Metadata and Zookeeper

I am seeing this behavior using librdkafka, as is another user. Listing the topic metadata with the tool provided with Kafka (kafka-topic.sh) shows all replicas in the ISR. However, using kafkacat[1] (built with librdkafka) shows that many ISRs are missing some replicas. I talked with Magnus (au

Re: GenericJMX plugin file for Kafka 0.8.1

We use jmxtrans to pull data out of JMX. https://github.com/wikimedia/puppet-kafka/blob/master/kafka-jmxtrans.json.md On Sep 9, 2014, at 7:54 AM, Navneet Gupta (Tech - BLR) wrote: > Hi, > > We plan to capture various metrics exposed in Kafka via JMX and wanted to > check if anybody has a p

Re: Manual Leader Assignment

For controlled failure, you can reduce the unavailability window by using > controlled shutdown. See > http://kafka.apache.org/documentation.html#basic_ops_restarting > > Thanks, > > Jun > > On Wed, Sep 3, 2014 at 11:48 AM, Andrew Otto wrote: > >> Hiya, >>

Manual Leader Assignment

. Thanks! -Andrew Otto [1] https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools#Replicationtools-Howtousethetool?.1

Re: Interested in contributing to Kafka?

Oh, BTW, I think Yelp is using this .deb packaging (and shell script) too. On Jul 21, 2014, at 10:16 AM, Andrew Otto wrote: > Hm, curious! > > Would this be useful to contribute upstream? > > https://github.com/wikimedia/operations-debs-kafka/blob/debian/debian/bin/kafka >

Re: Interested in contributing to Kafka?

Hm, curious! Would this be useful to contribute upstream? https://github.com/wikimedia/operations-debs-kafka/blob/debian/debian/bin/kafka Wikimedia uses it instead of the myriad of bin/*.sh scripts that come with Kafka. We didn’t want to build a .deb package that installed 16ish short shell s

Re: Zookeeper reconnect failed due to 'state changed (Expired)'

don't see which part of that real time summary makes up the bulk of that GC time This is strange, riight? This broker is identically configured to all its peers, and should be handling on average the exact same amount and type of traffic. Anyone have any advice? Thanks! -Andrew Otto On

Re: Kafka/Zookeeper co-location

Oo, I’m curious about this as well! Wikimedia is considering doing this if/when we install brokers in our web caching data centers. On Apr 24, 2014, at 11:49 AM, Sudarshan Kadambi (BLOOMBERG/ 731 LEXIN) wrote: > Are there any thoughts on running Zookeeper on the same physical nodes that > r

Re: Cluster design distribution and JBOD vs RAID

I >> suggest >>> you try testing it. >>> >>> >>> - Bob >>> >>> >>> -Original Message- >>> From: Andrew Otto [mailto:ao...@wikimedia.org] >>> Sent: Friday, April 18, 2014 8:36 AM >>> To: users@

Re: Cluster design distribution and JBOD vs RAID

> BOB> We are using RAID10. It was a requirement from our Unix guys. The > rationale for this was we didn't want to lose just a disk and to have to > rebuild/re-replicate 20TB of data. We haven't experienced any drive failures > that I am aware of. We have had complete server failures, but the d

Zookeeper reconnect failed due to 'state changed (Expired)'

r for all partitions. I can rebalance the leaders, but I’d prefer if this didn’t happen in the first place. 1. What does 'zookeeper state changed (Expired)’ mean? 2. Has anyone seen issues like this before? Where zookeeper connections are flaky enough to cause leader elections? Thank

Re: Using Kafka Metrics

I’m using jmxtrans to do this for Ganglia, but it should work the same for Graphite: http://www.jmxtrans.org/ Here’s an example Kafka jmxtrans json file. https://github.com/wikimedia/puppet-kafka/blob/master/kafka-jmxtrans.json.md You can change the output writers to use Graphite instead of Gan

Re: Puppet module for deploying Kafka released

). I really > like that your module already supports Kafka mirroring and jmxtrans. :-) > > --Michael > > > On 02/26/2014 03:41 PM, Andrew Otto wrote: >> Oh so many puppet modules! >> >> https://github.com/wikimedia/puppet-kafka >> >> This one requi

Re: Puppet module for deploying Kafka released

Oh so many puppet modules! https://github.com/wikimedia/puppet-kafka This one requires a Kafka .deb built from https://github.com/wikimedia/operations-debs-kafka/tree/debian/debian, which can be found prebuilt here: http://apt.wikimedia.org/wikimedia/pool/universe/k/kafka/ :) On Feb 26,

Re: Unexpected broker election

, > > Jun > > > On Fri, Feb 21, 2014 at 10:22 AM, Andrew Otto wrote: > >> Hi all, >> >> This has happened a couple of times to me now in the past month, and I'm >> not entirely sure of the cause, although I have a suspicion. >> >> E

Unexpected broker election

/ottomata/9139443 Thanks! -Andrew Otto

Re: Description of jmx exposed metrics?

Although that is not all of them, just recommended ones to pay attention to :/ On Feb 11, 2014, at 6:50 AM, Tomas Nunez wrote: > Hi! > > Sorry if this question has already been answered, but I've search the > archives, the project page and the wiki unsuccessfully. > > I'd like to know the mea

Re: Description of jmx exposed metrics?

Here tis! https://kafka.apache.org/documentation.html#monitoring On Feb 11, 2014, at 6:50 AM, Tomas Nunez wrote: > Hi! > > Sorry if this question has already been answered, but I've search the > archives, the project page and the wiki unsuccessfully. > > I'd like to know the meaning of the j

Re: Broker rejoin with big replica lag

> - Increasing num.replica.fetchers (defaults is one) Awesome! I just tried this one, bumped it up to 8 (12 cores on this broker box). It is now catching up at around 17K msgs/sec, which will mean it will finish in about 4 or 5 hours. I’ll check up on it again tomorrow. That should do it, Th

Broker rejoin with big replica lag

topic (kafka data files and in zookeeper), but to do so properly with 0.8.0 I think I’d have to shut down the whole cluster, correct? I’d rather not do this, as another topic does have a consumer and I don’t want to lose messages for it. Thanks! -Andrew Otto

Re: How to design a robust producer?

Thibaud, I wouldn't say this is a 'robust' solution, but the Wikimedia Foundation uses a piece of software we wrote called udp2log. We are in the process of replacing it with more robust direct Kafka producers, but it has worked for us in the intermediary. udp2log is a c++ daemon that listens f

Re: Kafka Cluster Failover

ill need to get it registered in a ZooKeeper cluster right? Also, > where are the events going to be ultimately consumed? I'm assuming in > the main DC - in which case you would anyway need to ship your Kafka > logs from the remote DC to the main DC correct? > > Joel >

Kafka Cluster Failover

producers and their metadata? Would they be able to handle a total switch of cluster metadata? Thanks! -Andrew Otto

Re: How to get monitoring stats

For Kafka 0.8, you could use http://www.jmxtrans.org/ and a variation of this json file: https://github.com/wikimedia/puppet-kafka/blob/master/kafka-jmxtrans.json.md Just change the ganglia output writers to graphite ones. On Nov 19, 2013, at 8:25 PM, Benjamin Black wrote: > https://www.goo

Re: List of topics with JMX?

Hi, > The count in the Mbean > "kafka.server":type="BrokerTopicMetrics",name="AllTopicsMessagesInPerSec" > gives the total no of messages for all the topics on the broker. > > > On Tue, Nov 19, 2013 at 8:13 PM, Andrew Otto wrote: > >>

Re: List of topics with JMX?

Would "kafka.server":type="BrokerTopicMetrics",name="AllTopicsMessagesInPerSec" count be easier? Also, correct me if I am wrong, but I believe that these count values are the total number of messages seen for a topic (or all topics) since the Broker was started, not the total number of message

Re: Kafka IPv6

Jun > > > On Wed, Nov 6, 2013 at 7:43 AM, Andrew Otto wrote: > >> Hm, >> >> Does Kafka support IPv6? I'm trying it now, but I'm just using the >> console-producer, which seems to not be able to read in --broker-list with >> IPv6 addresses.

Re: Kafka 0.8 jmxtrans + puppet

gshadoop <http://www.twitter.com/allthingshadoop> > / > > > On Wed, Nov 6, 2013 at 9:29 AM, Neha Narkhede wrote: > >> Cool, thanks for sharing this! >> >> -Neha >> On Nov 6, 2013 6:16 AM, "Andrew Otto" wrote: >>

  1   2   >