Re: slow log recovery

2013-05-06 Thread Jason Rosenberg
Will producers also be able to start sending new messages to a replica, while one broker is taking a long time to startup? On Mon, May 6, 2013 at 9:31 PM, Jun Rao wrote: > In 0.8, if you turn on replication, it may not matter too much if a broker > takes long to start up since data can still be

Re: download 0.8?

2013-05-06 Thread Jun Rao
Github is an external mirror. Thanks, Jun On Mon, May 6, 2013 at 9:28 PM, Jason Rosenberg wrote: > Sorry, I thought I had read in posting last week, that we now have an > official beta release. Not true? > > Ok > > So, what's the difference between repos: > > https://git-wip-us.apache.or

Re: Kafka producer sends messages efficiency

2013-05-06 Thread Jun Rao
Also, what's the flush.interval on the broker? Thanks, Jun On Mon, May 6, 2013 at 2:48 PM, Sining Ma wrote: > Hi, > I tried some simple performance tests for kafka producer. > I read 1000 lines of text from one file into memory and write these data > to a file, regular socket and using kafka

Re: LeaderNotAvailable Exception

2013-05-06 Thread Jun Rao
If you see it only once when the topic was auto-created, it is fine. Thanks, Jun On Mon, May 6, 2013 at 4:06 PM, Jason Rosenberg wrote: > I just tried to go through the quickstart, step by step on a Mac. I got > the same thing (LeaderNotAvailableException). > > > On Thu, Apr 25, 2013 at 9:30

Re: slow log recovery

2013-05-06 Thread Jun Rao
In 0.8, if you turn on replication, it may not matter too much if a broker takes long to start up since data can still be served from the replicas. It may be possible to improve this by maintaining a flush checkpoint file on disk. We can then use that info to reduce the amount of the data to be rec

Re: download 0.8?

2013-05-06 Thread Jason Rosenberg
Sorry, I thought I had read in posting last week, that we now have an official beta release. Not true? Ok So, what's the difference between repos: https://git-wip-us.apache.org/repos/asf/kafka.git and https://github.com/apache/kafka.git (which has an 0.8 branch)? Jason On Mon, May 6,

Re: maven integration in 0.8

2013-05-06 Thread rektide
There are a lot of release-oriented tickets still about- here's what I'm following pertaining to packaging & getting shipped- Maven- https://issues.apache.org/jira/browse/KAFKA-133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel Scala 2.10- https://issues.apache.org/jira/browse/K

Re: download 0.8?

2013-05-06 Thread Joe Stein
take a look at the quick start https://cwiki.apache.org/KAFKA/kafka-08-quick-start.html for 0.8 you need to build from the branch until there is an official release > git clone https://git-wip-us.apache.org/repos/asf/kafka.git> cd kafka > git checkout -b 0.8 remotes/origin/0.8 > ./sbt update > ./

Re: maven integration in 0.8

2013-05-06 Thread Joe Stein
0.8 is not released yet I need to start catching up on tickets starting here. On Mon, May 6, 2013 at 7:31 PM, Jason Rosenberg wrote: > Is there a maven repo we can point to, to just depend on the kafka 0.8 > core, and have all the dependencies get pulled in as needed? > > For some reason, I ha

maven integration in 0.8

2013-05-06 Thread Jason Rosenberg
Is there a maven repo we can point to, to just depend on the kafka 0.8 core, and have all the dependencies get pulled in as needed? For some reason, I had thought this would be available as part of the 0.8 release Or do I need to manually create a pom.xml for the core, and host it on my nexxu

Re: LeaderNotAvailable Exception

2013-05-06 Thread Jason Rosenberg
Message did go through however (I suspect the exception is just it complaining initially). On Mon, May 6, 2013 at 4:06 PM, Jason Rosenberg wrote: > I just tried to go through the quickstart, step by step on a Mac. I got > the same thing (LeaderNotAvailableException). > > > On Thu, Apr 25, 2013

Re: LeaderNotAvailable Exception

2013-05-06 Thread Jason Rosenberg
I just tried to go through the quickstart, step by step on a Mac. I got the same thing (LeaderNotAvailableException). On Thu, Apr 25, 2013 at 9:30 PM, Jun Rao wrote: > Thanks. Is anyone able to run the 0.8 quickstart without this issue on > Windows? > > Jun > > > On Thu, Apr 25, 2013 at 10:27

download 0.8?

2013-05-06 Thread Jason Rosenberg
There's still no download link for 0.8 on this page: http://kafka.apache.org/downloads.html What's the preferred link? Jason

Re: Kafka producer sends messages efficiency

2013-05-06 Thread Philip O'Toole
Have you tried compression? How powerful is the Kafka broker? Philip On Mon, May 6, 2013 at 2:48 PM, Sining Ma wrote: > Hi, > I tried some simple performance tests for kafka producer. > I read 1000 lines of text from one file into memory and write these data to a > file, regular socket and usin

slow log recovery

2013-05-06 Thread Jason Rosenberg
Recently, we had an issue where our kafka brokers were shut down hard (and so did not write out the clean shutdown file). Thus on restart, it went through all logs and ran a recovery on them. Unfortunately, this took a long time (on the order of 30 minutes). We have a lot of topics (e.g. ~1000 o

Kafka producer sends messages efficiency

2013-05-06 Thread Sining Ma
Hi, I tried some simple performance tests for kafka producer. I read 1000 lines of text from one file into memory and write these data to a file, regular socket and using kafka producer. The results is as follows: When I use BufferedOutputStream writing the data to a file, it takes 17 millisec