kafka.javaapi.producer.SyncProducer Confusion

2013-04-18 Thread Samir Madhavan
Hi, I'm using kafka-0.8 Snapshot. In the quick start, there is an example producer code in which SyncProducer class has been mentioned but when I checked the kafka.javaapi.producer, there is only Producer class and no SyncProducer or ProducerData class. I went through the apache github source cod

Re: How to create the initial zookeeper chroot path for zk.connect?

2013-04-18 Thread Jonathan Creasy
I made the patch to create the chroot and it doesn't handle multiple zk addresses. We fixed it but I guess that patch didn't get submitted. I will make a ticket here to get that done. On Apr 18, 2013 10:47 PM, "Ryan Chan" wrote: > Yes, using the latest Kafka 0.7.2, just tried to reproduce again

Re: How to create the initial zookeeper chroot path for zk.connect?

2013-04-18 Thread Ryan Chan
Yes, using the latest Kafka 0.7.2, just tried to reproduce again 1. Install a single node Kafka, three nodes zookeeper instances kafka1 zookeeper1 zookeeper2 zookeeper3 2. Using a simple Kafka config, able to start without error in the log brokerid=1 log.dir=/data/kafka

Re: Is the information at https://cwiki.apache.org/confluence/display/KAFKA/Operations up to date ?

2013-04-18 Thread Eric Sites
Jun, I have been trying to test 0.8 using two consumers on the same group ID. When I start the second consumer the first stops receiving messages. Can you help? Single Kafka server, single topic, 3 partitions, single Zookeeper. 1 console consumer running in one session. 1 console consumer run

Re: Is the information at https://cwiki.apache.org/confluence/display/KAFKA/Operations up to date ?

2013-04-18 Thread Jun Rao
We have been testing 0.8 at LinkedIn for some time. We plan to release an 0.8 beta version next week. Thanks, Jun On Thu, Apr 18, 2013 at 12:29 PM, Itai Frenkel wrote: > Hello, > > We're considering to use Kafka, and would like to know the Java/ > Scala/Kafka version combination in which it i

Re: How to create the initial zookeeper chroot path for zk.connect?

2013-04-18 Thread Neha Narkhede
That is odd. Is it reproducible ? On Wed, Apr 17, 2013 at 8:03 PM, Ryan Chan wrote: > Hi, > > Yes, I can see the new path exists by using the command "ls /" > > > > On Thu, Apr 18, 2013 at 1:29 AM, Neha Narkhede wrote: > >> After creating the path, did you get a chance to confirm that it got >> c

Is the information at https://cwiki.apache.org/confluence/display/KAFKA/Operations up to date ?

2013-04-18 Thread Itai Frenkel
Hello, We're considering to use Kafka, and would like to know the Java/ Scala/Kafka version combination in which it is being used in production. The wiki "Operations" page states that "We are running Kafka 0.7 right now but may move to trunk as we fix bugs." While I read here that 0.8 was sched

Re: What is the way to produce events from windows OS

2013-04-18 Thread Oleg Ruchovets
My windows client is C#. And As I understand there is a legacy C# client. I never try to execute java from C# code but it looks for me bad solution. Does C# client https://github.com/kafka-dev/kafka/tree/master/clients/csharp/src/Kafka/Kafka.Clientsupports Kafka 0.7.2 ??? In order I want to mainta

Re: What is the way to produce events from windows OS

2013-04-18 Thread Jun Rao
Can you use the java client on Windows? Thanks, Jun On Thu, Apr 18, 2013 at 12:11 AM, Oleg Ruchovets wrote: > Hi . > I am working on project: > Project has producer which runs on Windows OS (C#) . Consumer is java on > Linux. > > Question: >What is the way to write to Kafka from Window

Re: producer creating multiple log files after message sending

2013-04-18 Thread Jason Huang
Jay, This makes perfect sense and it is very helpful. thanks, Jason On Thu, Apr 18, 2013 at 12:20 PM, Jay Kreps wrote: > I think there is a misunderstanding here. The log file division is > completely transparent to the consumer. The consumer consumes by > offset (0,1,2,3,...) and these offs

Re: producer creating multiple log files after message sending

2013-04-18 Thread Jay Kreps
I think there is a misunderstanding here. The log file division is completely transparent to the consumer. The consumer consumes by offset (0,1,2,3,...) and these offsets span log files (i.e. there is a single, monotonically increasing set of offsets over all log files in a partition). In other wor

Re: producer creating multiple log files after message sending

2013-04-18 Thread Jason Huang
Number of open file handles is one thing. But I am thinking more of a use case where you may want to fetch information from two separate log files of same topic and same partition. Could we use one high level consumer API call to achieve that? One naive idea is that if we can merge those two log fi

Re: producer creating multiple log files after message sending

2013-04-18 Thread Neha Narkhede
There is no way to merge those log files in 0.8. But why would you want to do that ? To reduce the number of open file handles ? Thanks, Neha On Thursday, April 18, 2013, Jason Huang wrote: > Just curious - given the situation above, if you have two sets of .log and > .index files under the same

Re: producer creating multiple log files after message sending

2013-04-18 Thread Jason Huang
Just curious - given the situation above, if you have two sets of .log and .index files under the same topic and same partition, is there any way to merge these two sets of log files and index files? On Thu, Apr 18, 2013 at 5:42 AM, Swapnil Ghike wrote: > Otherwise if you are using the high lev

Re: producer creating multiple log files after message sending

2013-04-18 Thread Swapnil Ghike
Otherwise if you are using the high level consumer (ZookeeperConsumerConnector) in code, you will have to start a new consumer with a different group id. Thanks, Swapnil On 4/18/13 2:09 AM, "Swapnil Ghike" wrote: >Snehalata, > >Are you using the console consumer? You can pass a --from-beginning

Re: producer creating multiple log files after message sending

2013-04-18 Thread Swapnil Ghike
Snehalata, Are you using the console consumer? You can pass a --from-beginning flag to the console consumer to get all the messages posted to that topic. Thanks, Swapnil On 4/18/13 1:32 AM, "Snehalata Nagaje" wrote: >Hi Neha, > >I was able to figure this out, this is happening due to time base

RE: producer creating multiple log files after message sending

2013-04-18 Thread Snehalata Nagaje
Hi Neha, I was able to figure this out, this is happening due to time based rolling. But now problem, when I fetch the messages for particular topic, server is reading only one log file, returns those messages only. Is there any way I can combine all log files which are already produced into one

What is the way to produce events from windows OS

2013-04-18 Thread Oleg Ruchovets
Hi . I am working on project: Project has producer which runs on Windows OS (C#) . Consumer is java on Linux. Question: What is the way to write to Kafka from Windows OS? As I understand C# client is legacy. So what is the way to write to Kafka from Windows? Can I use C/C++ client from Wind