Re: Unable to send and consume compressed events.

2013-08-30 Thread Jun Rao
Not sure if this is a bug. I think the issue is that the log4j property is not set properly so that no log shows up. Thanks, Jun On Fri, Aug 30, 2013 at 9:32 AM, Jay Kreps wrote: > This seems like more of a bug then a FAQ, no? We are swallowing the > exception... > > -Jay > > > On Thu, Aug 29

Re: Unable to send and consume compressed events.

2013-08-30 Thread Jay Kreps
This seems like more of a bug then a FAQ, no? We are swallowing the exception... -Jay On Thu, Aug 29, 2013 at 11:30 PM, Lu Xuechao wrote: > Hi Jun, > > Thanks for you help. Finally, I found the reason by enabling producer side > DEBUG info output. The snappy jar is not included in the classpat

Re: Unable to send and consume compressed events.

2013-08-30 Thread Joe Stein
yeah the error should have showed up , will create JIRA On Fri, Aug 30, 2013 at 12:32 PM, Jay Kreps wrote: > This seems like more of a bug then a FAQ, no? We are swallowing the > exception... > > -Jay > > > On Thu, Aug 29, 2013 at 11:30 PM, Lu Xuechao wrote: > > > Hi Jun, > > > > Thanks for you

Re: Unable to send and consume compressed events.

2013-08-30 Thread Joe Stein
I think it is still good to have this one in the FAQ even issue sometimes folks need to know where to work around and how things until there fixed here is the JIRA for the defect https://issues.apache.org/jira/browse/KAFKA-1037 <== great place I think for someone looking to jump in and start cont

Re: Unable to send and consume compressed events.

2013-08-30 Thread Jun Rao
Xuechao, Thanks for updating the wiki. Not setting up log4j properly seems to be a general problem, not just limited to this particular problem. Could you reword it a bit to make it more general? Jun On Fri, Aug 30, 2013 at 2:02 AM, Lu Xuechao wrote: > Hi, Joe. wiki updated. Hope it helps. >

Re: Unable to send and consume compressed events.

2013-08-30 Thread Lu Xuechao
Hi, Joe. wiki updated. Hope it helps. On Fri, Aug 30, 2013 at 3:22 PM, Joe Stein wrote: > I feel like this is maybe a usual case as we have heard it before now a few > bits > > Lu Xuechao would you mind updating the FAQ > https://cwiki.apache.org/confluence/display/KAFKA/FAQ with what the > pro

Re: Unable to send and consume compressed events.

2013-08-30 Thread Joe Stein
I feel like this is maybe a usual case as we have heard it before now a few bits Lu Xuechao would you mind updating the FAQ https://cwiki.apache.org/confluence/display/KAFKA/FAQ with what the problem was and your solution just to capture this thread in the wiki please, thanx! /***

Re: Unable to send and consume compressed events.

2013-08-29 Thread Lu Xuechao
Hi Jun, Thanks for you help. Finally, I found the reason by enabling producer side DEBUG info output. The snappy jar is not included in the classpath. Added it and it worked. Thanks again. On Fri, Aug 30, 2013 at 12:53 PM, Lu Xuechao wrote: > No. > > > On Fri, Aug 30, 2013 at 11:57 AM, Jun

Re: Unable to send and consume compressed events.

2013-08-29 Thread Lu Xuechao
No. On Fri, Aug 30, 2013 at 11:57 AM, Jun Rao wrote: > These are the metadata requests. Do you see Producer requests from your > client? > > Thanks, > > Jun > > > On Thu, Aug 29, 2013 at 5:40 PM, Lu Xuechao wrote: > > > After I sent 1,000 compressed events, I saw these messages in broker's > l

Re: Unable to send and consume compressed events.

2013-08-29 Thread Jun Rao
These are the metadata requests. Do you see Producer requests from your client? Thanks, Jun On Thu, Aug 29, 2013 at 5:40 PM, Lu Xuechao wrote: > After I sent 1,000 compressed events, I saw these messages in broker's log > files: > > in kafka-request.log > > [2013-08-30 08:38:18,713] TRACE Pro

Re: Unable to send and consume compressed events.

2013-08-29 Thread Lu Xuechao
Update: Sending compressed events with console producer works: kafka-console-producer.bat --broker-list localhost:9092 --sync --topic topic1 --compress I am working on Windows 7. On Fri, Aug 30, 2013 at 8:40 AM, Lu Xuechao wrote: > After I sent 1,000 compressed events, I saw these messages in

Re: Unable to send and consume compressed events.

2013-08-29 Thread Lu Xuechao
After I sent 1,000 compressed events, I saw these messages in broker's log files: in kafka-request.log [2013-08-30 08:38:18,713] TRACE Processor 6 received request : Name: TopicMetadataRequest; Version: 0; CorrelationId: 0; ClientId: ; Topics: topic1 (kafka.network.RequestChannel$) [2013-08-30 08

Re: Unable to send and consume compressed events.

2013-08-29 Thread Jun Rao
Did you see any error in the producer log? Did the broker receive the produce request (you can look at the request log in the broker)? Thanks, Jun On Thu, Aug 29, 2013 at 6:29 AM, Lu Xuechao wrote: > Let me post my test code here. I could see producer.send(data); returned > with no error. > >

Re: Unable to send and consume compressed events.

2013-08-29 Thread Lu Xuechao
Let me post my test code here. I could see producer.send(data); returned with no error. public class TestProducer extends Thread { private final Producer producer; private final int m_events; private final int m_threadNumber; private static String msg = StringUtils.rightPad("", 1

Re: Unable to send and consume compressed events.

2013-08-29 Thread Lu Xuechao
Thanks Paul. Yes, I am using 0.8 beta1. I followed your suggestion to set request.required.acks=1 and got the same result. No error message seen in broker logs, the size of the partition files were after sending 1,000,000 events, the size of each event was 1KB : .index 10240

Re: Unable to send and consume compressed events.

2013-08-29 Thread Paul Mackles
I assume this is kafka 0.8, right? Are there any corresponding errors in the broker logs? With the configuration below, I don't think any errors will be reported back to the producer. You could also try setting erquest.required.acks=1 to see if errors are reported back to the client. On 8/29/13 4

Unable to send and consume compressed events.

2013-08-29 Thread Lu Xuechao
Hi , I am trying to enable gzip compression for my events. But after I switched compression.codec to "1" I found the produced events were even not be persisted to disk log file. Of course, the consumer could not receive any compressed events. I sent 10,000 or more events but the broker's log file