Re: steps path to kafka mastery

First, find a use that makes sense to you. Are you a developer and want to focus on building applications and tools that interact with and through kafka? Are you a platform person, who wants to figure out how to build a data ingest platform and connect various systems and data types through kafk

Re: Kafka advice for small startup

So long as you put some basic monitoring in place, it should run nicely with very little intervention and let you be confident everything is as it should be. Key things to watch: * disk space - a disk filling up really makes things difficult for you. Make sure your retention fits your footprint

RE: Per Topic Metrics

I want to say that the metrics only show up when the first message comes in, but I could be thinking of another tool. Try sending a message to the broker and see if metrics appear? t. -Original Message- From: Wollert, Fabian [mailto:fabian.woll...@zalando.de] Sent: Thursday, December 1

RE: Any gotchas upgrading to 0.9?

The quota page is here: http://kafka.apache.org/documentation.html#design_quotas "By default, each unique client-id receives a fixed quota in bytes/sec as configured by the cluster (quota.producer.default, quota.consumer.default)" I also noticed there's been a change in the replication configur

RE: Fair Usage of Kafka Queues

Hi Debraj, A couple things you could try. Given your design https://chart.googleapis.com/chart?chl=digraph+G+%7B%0D%0A+++rankdir%3DLR%3B%0D%0A+++service1LSFWD+-%3E+LS+-%3E+Kafka+-%3E+LSELK+-%3E+ES+-%3E+Kibana%0D%0A+++service2LSFWD+-%3E+LS%0D%0A+++service3LSFWD+-%3E+LS%0D%0A+++service4LSFWD+-

Re: future of Camus?

Another alternative is to checkout Kaboom ‎ https://github.com/blackberry/KaBoom ‎It uses a pared down kafka consumer library to pull data from Kafka and write it to defined (and somewhat dynamic) hdfs paths in a custom (and changeable) avro schema we call boom. It uses kerberos for authen

Re: 0.8.2.1 upgrade causes much more IO

Hi Andrew, Are you using Snappy Compression by chance? When we tested the 0.8.2.1 upgrade initially we saw similar results and tracked it down to a problem with Snappy version 1.1.1.6 (https://issues.apache.org/jira/browse/KAFKA-2189). We’re running with Snappy 1.1.1.7 now and the performan

RE: How to use kafka for log aggregation

Flume is one of the options to do this, yes. There are a (growing) number of tools you could use to get logs in to Kafka, depending on your environment, needs, level of skill, deployment size, applications, log volume: (in no particular order) Kafkacat : https://github.com/edenhill/ka

RE: Kafka producer input file

You can also checkout Klogger (https://github.com/blackberry/Klogger), which will take input from a TCP port or a file. Todd. -Original Message- From: Jason Gustafson [mailto:ja...@confluent.io] Sent: Monday, July 13, 2015 20:09 To: users@kafka.apache.org Subject: Re: Kafka producer inp

Introducing Klogger

Kafka producer/consumer library which is used by both Klogger and KaBoom. The project is still active internally, with frequent releases. Please have a look if you're interested.  We welcome any feedback or contributions. Cheers, Todd Todd Snyder Senior Infrastructure Specialist, Big Dat

Re: understanding partition key

,#,, Z. Z Sent from the wilds on my BlackBerry smartphone. Original Message From: Gary Ogden Sent: Thursday, February 12, 2015 8:23 AM To: users@kafka.apache.org Reply To: users@kafka.apache.org Subject: Re: understanding partition key Thanks David. Whether Kafka is the right choic