Re: Consumer sensitive expiration of topic

On Thu, Aug 14, 2014 at 09:56:11PM +, Prunier, Dominique wrote: > Hi, > > I'm playing around with Kafka with the idea to implement a general purpose > message exchanger for a distributed application with high throughput > requirements (multiple hundred thousand messages per sec). > > In th

Re: Consumer sensitive expiration of topic

By design, Kafka stores data independent of the number of publishers or subscribers connecting to it. This provides high performance as the broker does not have to manage consumers and evict data based on the consumers position. This is one of the main reasons why Kafka is much more performance com

Re: Strange topic-corruption issue?

Apache doesn't allow attachments. Could you send maybe a pastebin or something? On Thu, Aug 14, 2014 at 2:11 PM, Steve Miller wrote: >I've attached the full output. The only other thing it produced was > our old favorite: > > Non-secutive offsets in :/home/steve/mytopic-9/0

Re: Writing to Kafka

Hi Telles, 1. One broker is probably fine with the load, though if you want replication for fault tolerance you will need more than one. 2. The host/port you configure is just to discover the full cluster topology. Messages will be partitioned semantically and balanced over all hosts that have par

Consumer sensitive expiration of topic

Hi, I'm playing around with Kafka with the idea to implement a general purpose message exchanger for a distributed application with high throughput requirements (multiple hundred thousand messages per sec). In this context, i would like to be able to use a topic as some form of private mailbox

Writing to Kafka

Hi, I started using kafka with Samza. I’m trying to run a test that is supposed to create messages and write to a kafka topic. In my test I start writing a small amount of messages and it should grow up to 33200/sec. 1. With this amount of messages can one broker handle the messages or shoul

Re: Strange topic-corruption issue?

I've attached the full output. The only other thing it produced was our old favorite: Non-secutive offsets in :/home/steve/mytopic-9/.log 1327 is followed by 1327 For the first time, earlier today, we've seen this happen from one of our other producers; offhand I'm

Re: How to perform a controlled shutdown for rolling bounce?

Yes - you would still issue SIGTERM. Also, right now you will need to use the following process to increase replication factor: http://kafka.apache.org/documentation.html#basic_ops_increase_replication_factor There is a jira open to help reduce the effort to do that. Joel On Thu, Aug 14, 2014 at

Re: How to perform a controlled shutdown for rolling bounce?

Thanks for clarifying. When I increase the replication factor, enable controlled shutdown and want to do a controlled shutdown, do I still issue the same shutdown (SIGTERM)? On Thu, Aug 14, 2014 at 11:40 AM, Joel Koshy wrote: > Controlled shutdown does not really help in your case since your >

Re: How many threads should I use per topic

On Thu, Aug 14, 2014 at 10:39:15AM -0700, Chen Wang wrote: > Suppose I have a topic with 5 partitions, and partition 0, 1 has lag of 0, > while the other 3 all have lags. In this case, should I best start 3 > threads, or 5 threads to read from this topic again to achieve best > performance? > I am

Re: How to perform a controlled shutdown for rolling bounce?

Controlled shutdown does not really help in your case since your replication factor is one. > What does the -1 for Leader and blank Isr indicate? Do I need to run It means the partition is unavailable (since there are no other replicas). So you should either use a higher replication factor or j

Re: Strange topic-corruption issue?

What's the output of the following command? /opt/kafka/bin/kafka-run-class.sh kafka.tools.DumpLogSegments --files .log Thanks, Jun On Wed, Aug 13, 2014 at 11:40 AM, Steve Miller wrote: >Sure. I ran: > > /opt/kafka/bin/kafka-run-class.sh kafka.tools.Du

How to perform a controlled shutdown for rolling bounce?

Running 0.8.1 and am unable to do a controlled shutdown as part of a rolling bounce. Is this the primary reference for this task? https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools#Replicationtools-1.ControlledShutdown I've set the config to enable controlled shutdown. controll

erlang client for 0.8?

anyone aware of any 0.8 compatible erlang client? thanks

How many threads should I use per topic

Hey, Guys, I am using the high level consumer. I have a daemon process that checks the lag for a topic. Suppose I have a topic with 5 partitions, and partition 0, 1 has lag of 0, while the other 3 all have lags. In this case, should I best start 3 threads, or 5 threads to read from this topic again

Re: Correct way to handle ConsumerTimeoutException

Got it. Thanks guys! Chen On Wed, Aug 13, 2014 at 9:35 AM, Neha Narkhede wrote: > I am using consumer.timeout.ms to force a consumer jump out hasNext call, > which will throw ConsumerTimeoutException. > > Yes, this is the downside of the blocking iterator approach. If you want to > pull data in

Re: consumer read from specific partition

You can see an example of using the SimpleConsumer here . On Thu, Aug 14, 2014 at 3:23 AM, Sharninder wrote: > Implement the low level "Simple Consumer". > > -- > Sharninder > > > > On Thu, Aug 14, 2014 at 2:16 PM

Re: consumer read from specific partition

Implement the low level "Simple Consumer". -- Sharninder On Thu, Aug 14, 2014 at 2:16 PM, Josh J wrote: > Hi, > > Suppose I have N partitions. I would like to have X different consumer > threads ( X < N) read from a specified set of partitions. How can I achieve > this? > > Thanks, > > Josh >

consumer read from specific partition

Hi, Suppose I have N partitions. I would like to have X different consumer threads ( X < N) read from a specified set of partitions. How can I achieve this? Thanks, Josh