Re: KAFKA-1362 still open

https://issues.apache.org/jira/browse/KAFKA-1502 On Thu, Aug 28, 2014 at 4:20 PM, Joe Stein wrote: > I am pretty sure there is another JIRA for this issue already open. > > /*** > Joe Stein > Founder, Principal Consultant > Big Data Open Source Security

Re: KAFKA-1362 still open

I am pretty sure there is another JIRA for this issue already open. /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop *

KAFKA-1362 still open

Could someone reopen (KAFKA-1362) https://issues.apache.org/jira/browse/KAFKA-1362 please? It is still not solved but comments on closed issues seem not to be recognized in JIRA. Kind regards Daniel

Re: Request log is 300MB/hour

Hello Edward, You can try to increase the maxWait and minBytes (currently they are 100 ms and 1 byte from your logs) so that the consumer will not frequently try to pull data if there is little. Guozhang On Thu, Aug 28, 2014 at 12:49 PM, Edward Capriolo wrote: > At a certain hour I have seen

Re: Request log is 300MB/hour

Edward, I believe the request log was set to TRACE by default in older versions of Kafka, but has changed to WARN in newer versions. We had the same problem as you, and lowered our log level to WARN with no apparent issues On Thu, Aug 28, 2014 at 8:49 PM, Edward Capriolo wrote: > At a certain h

Request log is 300MB/hour

At a certain hour I have seen a huge up tick in requests. log4j.appender.requestAppender=org.apache.log4j.DailyRollingFileAppender log4j.appender.requestAppender.DatePattern='.'-MM-dd-HH log4j.appender.requestAppender.File=logs/kafka-request.log log4j.appender.requestAppender.layout=org.apache

RE: Consumer sensitive expiration of topic

True, but not different in essence with deleting a topic all together. Basically, this is an administrative operation, not strictly speaking a consumer operation. I'm not expecting this feature to be implemented in the consumers shipped with Kafka, but myself, i could use it in my own code becau

Re: Consumer sensitive expiration of topic

Maybe I misunderstand the proposal, but it sounds like an "irresponsible" consumer can accidentally delete data that others did not consume yet? On Thu, Aug 28, 2014 at 10:06 AM, Prunier, Dominique wrote: > Jay, > > I understand perfectly. I think you have all the reasons in the world to keep >

RE: Consumer sensitive expiration of topic

Jay, I understand perfectly. I think you have all the reasons in the world to keep the broker truly consumer independent, as it is according to me, a very wise principle that differentiate Kafka from pretty much all the other solutions. That is why, instead of the idea of consumer sensitive top

Re: Consumer sensitive expiration of topic

Hey Dominique, What you describe makes sense, and it would certainly be possible for the broker to more aggressively discard data once it sees that the consumer has read it once. The reason we haven't really taken that as a priority is because modern drives are so large relative to their throughp

RE: Consumer sensitive expiration of topic

Yeah, i'm really not worried about performance. Disk space, or more specifically, disk space by duplication of the same data in different topics was my concern. The primary use case would be a special consumer which job would be to partition the messages from a topic into various "private consum

Re: message size limit

That's right. If one message in a batch exceeds the size limit, the whole batch is rejected. When determining message.max.bytes, the most important thing to consider is probably memory since currently we need to allocate memory for a full message in the broker and the producer and the consumer cli