Re: Anyone using log4j Appender for Kafka?

We just configure our logback.xml to have two Appenders, an AsyncAppender -> KafkaAppender, and FileAppender (or ConsoleAppender as appropriate). AsyncAppender removes more failure cases too, e.g. a health check hanging rather than returning rapidly could block you application. On Feb 22, 2015,

Re: Broker w/ high memory due to index file sizes

If you haven't seen it yet, you probably want to look at http://kafka.apache.org/documentation.html#java -Ewen On Thu, Feb 19, 2015 at 10:53 AM, Zakee wrote: > Well are there any measurement techniques for Memory config in brokers. We > do have a large load, with a max throughput 200MB/s. What

Re: Anyone using log4j Appender for Kafka?

Theres also another one here. https://github.com/danielwegener/logback-kafka-appender. It has a fallback appender which might address the issue of Kafka being un-available. On Mon, Feb 23, 2015 at 9:45 AM, Steven Schlansker < sschlans...@opentable.com> wrote: > Here’s my attempt at a Logback v

Re: Anyone using log4j Appender for Kafka?

Here’s my attempt at a Logback version, should be fairly easily ported: https://github.com/opentable/otj-logging/blob/master/kafka/src/main/java/com/opentable/logging/KafkaAppender.java On Feb 22, 2015, at 1:36 PM, Scott Chapman wrote: > I am just starting to use it and could use a little guidan

Re: NetworkProcessorAvgIdlePercent

What kind of load do you have on the brokers? On an idle cluster (just fetch requests from the follower replicas), I saw NetworkProcessorAvgIdlePercent at about 97%. Thanks, Jun On Thu, Feb 19, 2015 at 5:19 PM, Zakee wrote: > Jun, > > I am already using the latest release 0.8.2.1. > > -Zakee >

How to find the number of message in the topic

Hi, Please let me know how to find the total number of messages in a particular topic. Regards, Bhuvana

Re: New Producer - Is the configurable partitioner gone?

yes. this is with the new java client. since it is using non-blocking NIO, sender thread probably was able to scan the buffer very frequently. hence random partitioner won't get much chance to accumulate records for batch or request. Setup * - 3 broker instances (m1.xlarge)- 6 producer instances

Latest offset is frozen

I'm finding that if I continuously produce values to a topic (say, once every 2 seconds), and in another thread, query the head and tail offsets of a topic, then sometimes I see the head offset increasing, sometimes its frozen. What's up with that? I'm using scala client: 0.8.2 and server: 2.9.2-0

earliestOrLatestOffset, getOffsetsBefore

In http://apache.osuosl.org/kafka/0.8.2-beta/scala-doc/index.html#kafka.consumer.SimpleConsumer class SimpleConsumer: def earliestOrLatestOffset(topicAndPartition: TopicAndPartition, earliestOrLatest: Long, consumerId:Int): Long 1) What's the consumerId? It doesn't seem to matter what value

Re: New Producer - Is the configurable partitioner gone?

Interesting, and this was with the new Java client? This sounds like as much an opportunity for improvement in the code as anything. Would you be willing to share the details? -jay On Sunday, February 22, 2015, Steven Wu wrote: > > The low connection partitioner might work for this > by attempt

Re: New Producer - Is the configurable partitioner gone?

> The low connection partitioner might work for this by attempting to reuse recently used nodes whenever possible. That is useful in environments with lots and lots of producers where you don't care about semantic partitioning. In one of the perf test, we found that above "sticky" partitioner impr

Re: Why The Division Between Scala And Java

so there will be both scala and java clients? or will scala users simply import the java libraries (which is after all not too bad) 2015-02-22 16:30 GMT-08:00 Guozhang Wang : > Alex, > > Before 0.8 Kafka is written in Scala, and in 0.8.2 we are re-writing the > clients in Java for better clients

Re: Why The Division Between Scala And Java

Alex, Before 0.8 Kafka is written in Scala, and in 0.8.2 we are re-writing the clients in Java for better clients adoption while the server is still under Scala. The plan after the Java clients also includes migrating the common utils / error code / request formats to Java that will be used for bo

Re: Can Mirroring Preserve Every Topic's Partition?

Hi Alex, What I originally meant is that you probably need to manually modify the MM code in order to achieve your needs. However, MM has been improved a lot since last time we synced up, in the next major release the MM will support exact mirroring (details in KAFKA-1650) with some functional ext

Re: High CPU usage of Crc32 on Kafka broker

Allen, Regarding the two crc computation calls, the first one is used to validate the messages, and the second call is only used if we need to re-compress the data. So logically they are not redundant operations. As Jay said, the re-compression is acutally savable and once it is removed, we will n

Re: [kafka-clients] Re: [VOTE] 0.8.2.1 Candidate 1

Jun, Can we also add https://issues.apache.org/jira/browse/KAFKA-1724 to the next RC please? Thanks! ~ Joe Stein - - - - - - - - - - - - - - - - - http://www.stealth.ly - - - - - - - - - - - - - - - - - On Sun, Feb 22, 2015 at 11:59 AM, Jun Rao wrote: > We identified at least one more bloc

Anyone using log4j Appender for Kafka?

I am just starting to use it and could use a little guidance. I was able to get it working with 0.8.2 but am not clear on best practices for using it. Anyway willing to help me out a bit? Got a few questions, like how to protect applications from when kafka is down or unreachable. It seems like a

Commit offset with SimpleConsumer in 0.8.2

I have a hard time figuring out how to do a commit using API 0.8.2 on JDK 8. I tried using the examples from 0.8.1.1. First of all: I can't use OffsetMetadataAndError inside the offsets-map as it was possible in 0.8.1. I can't really find a difference, but builds break. I'm also unable to figure

Re: kafka-web-console goes down regularly

I saw performance issues with the web console whenever using a large number of partitions (1000 partitions). On 2/3/15 12:09 PM, Sa Li wrote: Hi, All I am currently using kafka-web-console to monitor the kafka system, it get down regularly, so I have to restart it every few hours which is kind

ability to delete inactive queues?

I’m considering porting an app from ActiveMQ to Kafka so I’m new here. Apologizes if this has been asked before. Is it possible to delete inactive queues? Specifically queues with no messages on them which haven’t received any new messages in say 5 minutes. We use a lot of ephemeral queues which

Re: High CPU usage of Crc32 on Kafka broker

Here's my summary of the state of the compression discussion: 1. We all agree that current compression performance isn't very good and it would be nice to improve it. 2. This is not entirely due to actual (de)compression, in large part it is inefficiencies in the current implementation

Re: New Producer - Is the configurable partitioner gone?

Hey Daniel, Yeah I think that would be doable. If you want to pursue it you would need to do a quick KIP just to get everyone on the same page since this would be a public interface we would have to support over a long time: https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Propo

Re: [kafka-clients] Re: [VOTE] 0.8.2.1 Candidate 1

We identified at least one more blocker issue KAFKA-1971 during testing. So, we will have to roll another RC for 0.8.2.1. Thanks, Jun On Sat, Feb 21, 2015 at 6:04 PM, Joe Stein wrote: > Source verified, tests pass, quick start ok. > > Binaries verified, tests on scala > https://github.com/stea

Re: New Producer - Is the configurable partitioner gone?

Jay Kreps writes: > > Hey Daniel, > > partitionsFor() will block the very first time it sees a new topic that it > doesn't have metadata for yet. If you want to ensure you don't block even > that one time, call it prior to your regular usage so it initializes then. > > The rationale for add

Re: High CPU usage of Crc32 on Kafka broker

I just want to bring up that idea of no server side de/recompression again. Features like KAFKA-1499 seem to steer the project into a different direction and the fact that tickets like KAFKA-845