Re: Data inputs for Kafka.

there are kafka producers/consumers in a lot of languages. There is an Ecosystem page somewhere in the wiki. Take a look at that. Finally, I think you should experiment with both activemq and kafka and then take a decision. Don't try to find a solution and fit your problem to it, should be the oth

Re: Data inputs for Kafka.

Hi , Thanks for the quick follow up Philip. Also , can you let me know whether the kafka implementation can be as versatile as ActiveMQ with regards to connectivity? I mean to say a web-app generating transactional data in a data base from java webapp, ruby web app or .net web app can kafka t

Re: fsync behaviour

Thanks Jun. If the log flush interval is more than 1, will the unflushed messages be visible to the consumers before being written to disk. Thanks, Manju On 21 Aug 2014 06:03, Jun Rao wrote: 1) if you configure the log flush interval to be 1 message, every produce request will be flushed to dis

Re: Using kafka in non million users environment

Thanks Philip and Anand for the hints. I fill more comfortable going further now. On Wed, Aug 20, 2014 at 6:49 PM, Anand Nalya wrote: > For operating kafka across multiple data centers have a look at > https://kafka.apache.org/08/ops.html and MirrorMaker ( > https://kafka.apache.org/08/tools.h

Re: Keep on getting kafka.common.OffsetOutOfRangeException: Random times

hello, everybody, how can I unsubscribe kafka users emails, Thank you! -- Original -- From: "Philip O'Toole";; Date: Thu, Aug 21, 2014 10:20 AM To: "users@kafka.apache.org"; Subject: Re: Keep on getting kafka.common.OffsetOutOfRangeException: Random times

Re: fsync behaviour

1) if you configure the log flush interval to be 1 message, every produce request will be flushed to disk before the response is sent to the producer. Thanks, Jun On Tue, Aug 19, 2014 at 6:00 PM, Manjunath Shivakumar < manjunath.shivaku...@betfair.com> wrote: > Hi, > > I wanted to understand t

Re: kafka TestUtils createBrokerConfig issue

The second one is actually createBrokerConfig (i.e., w/o the ending s). Thanks, Jun On Tue, Aug 19, 2014 at 1:20 PM, Parin Jogani wrote: > Trying to write a unit test case for Kafka, and stuck with strange > createBrokerConfig issue. > on TestUtils.createBrokerConfigs(1) it gives me compilati

Re: kafka 0.8.1.1 log.retention.minutes NOT being honored

Added to the documentation. Topic level retentions is called retention.ms and can be found in http://kafka.apache.org/documentation.html#configuration Thanks, Jun On Fri, Aug 15, 2014 at 12:32 PM, François Langelier wrote: > UP! Can someone add the log.retention.hours in the documentation? >

Re: Keep on getting kafka.common.OffsetOutOfRangeException: Random times

Steve - thanks for thIs demonstration. As you explained, the way Kafka works makes perfect sense. It uses the next offset. I will say when I first hit this error a couple of years ago (with Storm Spouts) it startled me; it does feel like a bug to the uninitiated. But it's perfectly normal if o

Re: Keep on getting kafka.common.OffsetOutOfRangeException: Random times

OK, so I recreated this, because I wasn't sure if the offsets were preserved when the data was evaporated, or if somehow they reset to zero: /opt/kafka/bin/kafka-topics.sh --zookeeper whatever --topic test.deletion --replication-factor 1 --partitions 1 --config "retention.ms=6" -

Re: Announce: Capillary, a monitor for Kafka 0.8 spout topologies -- and the upgrade to 0.8

Sure! A couple things pushed us in to the upgrade. Our machines were very small in terms of storage, we really wanted replication and we weren't really monitoring our current machines well since JMX was turned off (and without replication/leadership a restart was hard). The first wrinkle was that

Re: Announce: Capillary, a monitor for Kafka 0.8 spout topologies -- and the upgrade to 0.8

Any thoughts on upgrading from 0.7 to 0.8 you'd like to share with the community? How did it go? How has it been? Philip   - http://www.philipotoole.com On Wednesday, August 20, 2014 10:13 AM, Cory Watson wrote: :P  Thanks Philip! On Wed, Aug 2

Re: Announce: Capillary, a monitor for Kafka 0.8 spout topologies

:P Thanks Philip! On Wed, Aug 20, 2014 at 10:04 AM, Philip O'Toole < philip.oto...@yahoo.com.invalid> wrote: > Nice work. That tool I put together was getting a bit old. :-) > > > I updated the Kafka "ecosystem" page with details of both tools. > > https://cwiki.apache.org/confluence/display/KA

Re: Announce: Capillary, a monitor for Kafka 0.8 spout topologies

Nice work. That tool I put together was getting a bit old. :-) I updated the Kafka "ecosystem" page with details of both tools. https://cwiki.apache.org/confluence/display/KAFKA/Ecosystem Philip     - http://www.philipotoole.com On Wednesday, August 2

Re: Keep on getting kafka.common.OffsetOutOfRangeException: Random times

Pradeep, The simple consumer will throw exception upon getting this response by design, for the high-level consumer it will automatically re-set its offset according to the reset policy (either latest or earliest). Guozhang On Wed, Aug 20, 2014 at 1:40 AM, wrote: > Sure, I would try with sett

Announce: Capillary, a monitor for Kafka 0.8 spout topologies

Hello all! If you use Storm and Kafka 0.8 together I may have an interesting project for you. Recently at Keen IO we upgraded from Kafka 0.7 to 0.8 and needed to replace the features of stormkafkamon for monitoring spout offsets since

Re: Keep on getting kafka.common.OffsetOutOfRangeException: Random times

It's not a bug, right? It's the way the system works (if I have been following the thread correctly) -- when the retention time passes, the message is gone. Either consume your messages sooner, or increase your retention time. Kafka is not magic, it can only do what it's told. In practise I hav

Re: Which is better?

If you have studied the docs yet, you should, as this is a broad question which needs background to understand the answer. But in summary, the high-level Consumer does more for you, and importantly, provides balancing between Consumers. The SimpleConsumer does less for you, but gives you more c

Re: Data inputs for Kafka.

Kafka can ingest any kind of data, and connect to many types of systems. Much work exists in this area already, for hooking a wide variety of systems to Kafka. If your system isn't supported, then you write a Kafka Producer to pull (or receive) messages from your system, and write them to Kafka.

Which is better?

Hi All, My requirement is to consume data from kafka using consumer. Which should be better to use? SimpleConsumer or HighLevel Consumer? Thanks, Pankaj Ojha

Data inputs for Kafka.

Hi guys, We are implementing a messaging system in order to perform Real time analytics. However, we are confused on using ActiveMQ or Apache Kafka. The point of worry for us are : 1. What kind of data can be ingested by Apache kafka? If we see , Active MQ uses the JMS which has an API to Java as

RE: Keep on getting kafka.common.OffsetOutOfRangeException: Random times

Sure, I would try with setting longer retention hours. But I feel this would not be good approach? Should we raise it as a bug? Thanks, Pradeep Simha Technical Lead -Original Message- From: Manjunath Shivakumar [mailto:manjunath.shivaku...@betfair.com] Sent: Wednesday, August 20, 2014 1:

RE: Keep on getting kafka.common.OffsetOutOfRangeException: Random times

We had a similar issue in our dev environments, where we had to configure aggressive log retention to save space. And the clients kept failing with this error, on Mondays, because the message from friday had got deleted. Perhaps compaction would help in this scenario too? https://cwiki.apache.or

Re: Using kafka in non million users environment

For operating kafka across multiple data centers have a look at https://kafka.apache.org/08/ops.html and MirrorMaker ( https://kafka.apache.org/08/tools.html) On 20 August 2014 04:09, Justin Maltat wrote: > Hi, > > As of today, our company IT is mainly composed of domain specific > software (pr

Re: Keep on getting kafka.common.OffsetOutOfRangeException: Random times

That seems likely. I'd try either catching the exception and resetting the offset, or upping log.retention.hours. I'd be interested in hearing if that fixes the problem. -Steve > On Aug 19, 2014, at 11:54 PM, wrote: > > Thank you for your reply. Oh is retention hours have affect on this