Fwd: kafka TestUtils createBrokerConfig issue

Trying to write a unit test case for Kafka, and stuck with strange createBrokerConfig issue. on TestUtils.createBrokerConfigs(1) it gives me compilation error The method createBrokerConfigs(int, boolean) in the type TestUtils is not applicable for the arguments (int) When I looked into the scala

kafka high level consumer - threads guaranteed to read a single partition?

Hi, For the kafka high level consumer, if I create exactly the number of threads as the number of partitions, is there a guarantee that each thread will be the only thread that reads from a particular partition? I'm following this example

RE: Keep on getting kafka.common.OffsetOutOfRangeException: Random times

Hi Team, Can someone please help me in this? This is really becoming road block to our project we should decide whether to continue to use Kafka or some other project as it is becoming too much of unstable. Thanks, Pradeep Simha Technical Lead -Original Message- From: pradeep.si...@wi

Re: Using kafka in non million users environment

Hi Justin It sounds like Kafka could be a good fit for your environment. Are you able to tell us more about the kinds of applications you will be running? Daniel. > On 19/08/2014, at 10:53 am, Justin Maltat wrote: > > Hello, > > I'm managing a study to explore possibilities for migrating a m

Re: Keep on getting kafka.common.OffsetOutOfRangeException: Random times

Also, what do you have log.retention.hours set to? How often do you publish messages? I can envision a scenario in which you don't publish to a topic often, and in fact publish so infrequently that everything in the topic ages out from log.retention.hours first. I don't know exactly

how is the partitioner working when compression is enabled?

I am using Kafka 0.8 and message compression can only be enabled on producer side. However one batch of messages (decided by the producer configuration batch.num.messages) is compressed as a single message and stored at the broker side. I think one batch compressed message will be stored at one

Re: how is the partitioner working when compression is enabled?

Hi Lex, messages are batched (and compressed) after partitioning. 2014-08-19 15:46 GMT+02:00 Lian, Li : > I am using Kafka 0.8 and message compression can only be enabled on > producer side. > > However one batch of messages (decided by the producer configuration > batch.num.messages) is compr

Re: kafka 0.8.1.1 log.retention.minutes NOT being honored

up? François Langelier Étudiant en génie Logiciel - École de Technologie Supérieure Capitaine Club Capra VP-Communication - CS Games 2014 Jeux de Génie 2011 à 2014 Magistrat Fraternité du Piranha

Re: Improving the Kafka client ecosystem

Did this mailing list ever get created? Was there consensus that it did or didn't need created? -Mark > On Jul 18, 2014, at 14:34, Jay Kreps wrote: > > A question was asked in another thread about what was an effective way > to contribute to the Kafka project for people who weren't very > enth

Re: Improving the Kafka client ecosystem

I created kafka-clie...@groups.google.com https://groups.google.com/forum/m/#!forum/kafka-clients No members and no guidelines yet, but it's a start. Would love to get this going. Dana On Aug 19, 2014 9:03 AM, "Mark Roberts" wrote: > Did this mailing list ever get created? Was there consensu

Re: kafka high level consumer - threads guaranteed to read a single partition?

Hi Josh, Yes. The comsumption distribution at Kafka is at the granularity of partitions, i.e. each partition will only be consumed by one consumer within the group. Guozhang On Tue, Aug 19, 2014 at 2:01 AM, Josh J wrote: > Hi, > > For the kafka high level consumer, if I create exactly the num

Re: how is the partitioner working when compression is enabled?

Magnus is right, in both old and new producer, the data is first partitioned, and within each partition batched and compressed for sending. Guozhang On Tue, Aug 19, 2014 at 7:12 AM, Magnus Edenhill wrote: > Hi Lex, > > messages are batched (and compressed) after partitioning. > > > > 2014-08-1

Re: Keep on getting kafka.common.OffsetOutOfRangeException: Random times

How often do you push messages to the brokers? Guozhang On Tue, Aug 19, 2014 at 2:12 AM, wrote: > Hi Team, > > Can someone please help me in this? This is really becoming road block to > our project we should decide whether to continue to use Kafka or some other > project as it is becoming to

Fwd: Issue in using kafka SimpleConsumer code

Hi Team, I have a requirement of reading real time data using kafka and write to cassandra. For this I am using SimpleConsumer to read data from Kafka topics and writing into Cassandra. I am maintaining offsets of topics in my log files. The issue is that after few days like 3-4 days my cosumer co

Re: Improving the Kafka client ecosystem

Cool. I just joined. I'll add it to the website so others can find it. If someone was willing to ping some of the other client developers and get them to join as well that would probably give us critical mass. -Jay On Tue, Aug 19, 2014 at 9:08 AM, Dana Powers wrote: > I created kafka-clie...@gro

kafka TestUtils createBrokerConfig issue

Trying to write a unit test case for Kafka, and stuck with strange createBrokerConfig issue. on TestUtils.createBrokerConfigs(1) it gives me compilation error The method createBrokerConfigs(int, boolean) in the type TestUtils is not applicable for the arguments (int) When I looked into the scala

Re: Improving the Kafka client ecosystem

I just joined too, and tweeted. /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop / On Tu

Re: Improving the Kafka client ecosystem

I also opened issues on 3 of the clients on github that I frequently use/involved in often enough would be great to get on the README as such. Thanks to the community for driving things along! /*** Joe Stein Founder, Principal Consultant Big Data Ope

Re: Improving the Kafka client ecosystem

Does it make sense to merge the Camus mailing list? (i.e. ask the Camus community to merge?) Its a fairly large and popular client. On Tue, Aug 19, 2014 at 1:27 PM, Joe Stein wrote: > I also opened issues on 3 of the clients on github that I frequently > use/involved in often enough would be

Re: Improving the Kafka client ecosystem

Great idea Gwen! I think it would go a long way to making this work. Issue created :) On Tue, Aug 19, 2014 at 4:33 PM, Gwen Shapira wrote: > Does it make sense to merge the Camus mailing list? (i.e. ask the > Camus community to merge?) Its a fairly large and popular client. > > On Tue, Aug 19,

Re: Using kafka in non million users environment

Hi, As of today, our company IT is mainly composed of domain specific software (proprietary and homemade). We could like to migrate them one after another to a microservice architecture with Kafka as the data pipeline. With the system now in place it's quite difficult to have a common data flow be

Re: Using kafka in non million users environment

>>- we have a low data traffic compared to your figures: around 30 GB a day. Will it be an issue? I have personal experience that Kafka deals extremely well with very low-volumes, as well as very high. I have used Kafka for small integration-test setups, as well as large production systems. Kaf

fsync behaviour

Hi, I wanted to understand the exact behaviour of kafka ( 0.8.1.1 ) w.r.t fsync. I have read the documentation and some related forum posts, but wanted to confirm the understanding here. 1) If we set the log flush to happen on every message, will it happen synchronously with each produce reque

RE: Keep on getting kafka.common.OffsetOutOfRangeException: Random times

Hi, I push maybe once in 15 mins or more like that. Does pushing messages frequently or less frequently have affect on this exception? If so, how do I get rid of this? Can you please guide me? Thanks, Pradeep Simha Technical Lead -Original Message- From: Guozhang Wang [mailto:wangg...@

RE: Keep on getting kafka.common.OffsetOutOfRangeException: Random times

Thank you for your reply. Oh is retention hours have affect on this? I didn't knew this. I have log.retention.hours set to 1, and during development we test this once a 15 mins or hour or 2. So do you think this is causing the issue? Thanks, Pradeep Simha Technical Lead -Original Message---

kafka TestUtils choosePort NoSuchMethodError

I am trying to create unit test case for Kafka with a simple call Properties props =TestUtils.createBrokerConfig(1, TestUtils.choosePort(), true); It fails on > java.lang.NoSuchMethodError: > scala.Predef$.intWrapper(I)Lscala/runtime/RichInt; > at kafka.utils.TestUtils$.choosePorts(Test