Unable to send and consume compressed events.

Hi , I am trying to enable gzip compression for my events. But after I switched compression.codec to "1" I found the produced events were even not be persisted to disk log file. Of course, the consumer could not receive any compressed events. I sent 10,000 or more events but the broker's log file

Re: Unable to send and consume compressed events.

I assume this is kafka 0.8, right? Are there any corresponding errors in the broker logs? With the configuration below, I don't think any errors will be reported back to the producer. You could also try setting erquest.required.acks=1 to see if errors are reported back to the client. On 8/29/13 4

Re: Unable to send and consume compressed events.

Thanks Paul. Yes, I am using 0.8 beta1. I followed your suggestion to set request.required.acks=1 and got the same result. No error message seen in broker logs, the size of the partition files were after sending 1,000,000 events, the size of each event was 1KB : .index 10240

Re: Unable to send and consume compressed events.

Let me post my test code here. I could see producer.send(data); returned with no error. public class TestProducer extends Thread { private final Producer producer; private final int m_events; private final int m_threadNumber; private static String msg = StringUtils.rightPad("", 1

hello

hi, My network does not connect to Internet. I want to install "kafka-0.7.2-incubating-src.tgz (asc, md5)", so I can not exec shell " ./sbt update ./sbt package". My quesion is in this case how I can install the software? mzl

Re: Loadbalancing producers

I think a typical setup is that you have a load balancer in front of a bunch of frontend services, each of which has an embedded producer. Thanks, Jun On Wed, Aug 28, 2013 at 9:01 PM, Mark wrote: > Is it common/necessary to load balance the connections your producers use? > I'm looking at the

RE: zookeeper session time out

Thanks for your answer, Neha. Currently we didn't save the GC log. I will add that option and keep monitoring the issue. Regards, Libo -Original Message- From: Neha Narkhede [mailto:neha.narkh...@gmail.com] Sent: Wednesday, August 28, 2013 4:25 PM To: users@kafka.apache.org Subject: Re

Re: Unable to send and consume compressed events.

Did you see any error in the producer log? Did the broker receive the produce request (you can look at the request log in the broker)? Thanks, Jun On Thu, Aug 29, 2013 at 6:29 AM, Lu Xuechao wrote: > Let me post my test code here. I could see producer.send(data); returned > with no error. > >

Re: hello

We don't have a binary release for 0.7.2. So you have to find a computer with network to build the binary. For 0.8 beta1 release, we do have a binary version that you can download. Thanks, Jun On Thu, Aug 29, 2013 at 12:07 AM, 马占雷 <2005120...@163.com> wrote: > hi, > My network does not con

Num of streams for consumers using TopicFilter.

So my understanding is num of active streams that a consumer can utilize is number of partitions in topic. This is fine if we consumer from specific topic. But if we consumer from TopicFilter, I thought consumer should able to utilize (number of topics that match filter * number of partitions in to

Re: Loadbalancing producers

We have a few dozen front-end web apps running rails. Each one of these rails instances has an embedded producer which in turn connects to a 3 brokers who are behind a load-balancer. Does that sound about right? Is there anything special to account for since these TCP connections are long lived

Re: Loadbalancing producers

The LB in front of the brokers doesn't make sense. On Aug 29, 2013 8:42 AM, "Mark" wrote: > We have a few dozen front-end web apps running rails. Each one of these > rails instances has an embedded producer which in turn connects to a 3 > brokers who are behind a load-balancer. Does that sound ab

Re: Loadbalancing producers

Can you explain why not? On Aug 29, 2013, at 8:43 AM, Benjamin Black wrote: > The LB in front of the brokers doesn't make sense. > On Aug 29, 2013 8:42 AM, "Mark" wrote: > >> We have a few dozen front-end web apps running rails. Each one of these >> rails instances has an embedded producer whi

Question on # of partitions

If I have 3 brokers with 3 partitions does that mean: 1) I have 3 partitions per broker so I can have up to 9 consumers or 2) There is only 1 partition per brokers which means I can have only 3 consumers Thanks

Re: Question on # of partitions

It means the first. Philip On Thu, Aug 29, 2013 at 8:55 AM, Mark wrote: > If I have 3 brokers with 3 partitions does that mean: > > 1) I have 3 partitions per broker so I can have up to 9 consumers > > or > > 2) There is only 1 partition per brokers which means I can have only 3 > consumers >

Re: Getting LeaderNotAvailableException in console producer after increasing partitions from 4 to 16.

Created JIRA and attached patch to it. Please review. On Wed, Aug 28, 2013 at 1:11 PM, Guozhang Wang wrote: > I think this patch can be made in trunk. You can mark it as 0.8.1 > > Guozhang > > On Wednesday, August 28, 2013, Rajasekar Elango >

Re: Loadbalancing producers

As Benjamin indicated, you wouldn't get much out of an LB in front of brokers, especially if the producers are configured in async mode (which you would want to in order to increase throughput). On Thu, Aug 29, 2013 at 8:54 AM, Mark wrote: > Can you explain why not? > > On Aug 29, 2013, at 8:4

Re: Loadbalancing producers

Producers discover the broker that owns each partition and send messages directly. There isn't anything to load balance at layer 4. On Aug 29, 2013 8:54 AM, "Mark" wrote: > Can you explain why not? > > On Aug 29, 2013, at 8:43 AM, Benjamin Black wrote: > > > The LB in front of the brokers doesn'

Re: Num of streams for consumers using TopicFilter.

>> 2) When I started mirrormaker with num.streams=16, looks like 16 consumer threads were created, but only 8 are showing up as active as owner in consumer offset tracker and all topics/partitions are distributed between 8 consumer threads. This is because currently the consumer rebalancing proces

Securing kafka

Is there a way to stop a malicious user to connect directly to a kafka broker and send any messages? Could we have the brokers to accept a message to a list of know IPs?

Re: Securing kafka

IP filters on the hosts. On Aug 29, 2013 10:03 AM, "Calvin Lei" wrote: > Is there a way to stop a malicious user to connect directly to a kafka > broker and send any messages? Could we have the brokers to accept a message > to a list of know IPs? >

0.7 question on topics

How can one configure the number of partitions per topic in 0.7? I understand 0.8 has a command line tool to help with this. Is my only option setting a global value for numPartition in server.properties? Thanks

Re: 0.7 question on topics

Sorry forgot to point to the 07 configuration page - http://kafka.apache.org/07/configuration.html Thanks, Neha On Thu, Aug 29, 2013 at 10:26 AM, Neha Narkhede wrote: > The 07 configuration is explained here > > The property you are looking for is topic.partition.count.map. > > Thanks, > Neha >

Re: 0.7 question on topics

The 07 configuration is explained here The property you are looking for is topic.partition.count.map. Thanks, Neha On Thu, Aug 29, 2013 at 10:23 AM, Mark wrote: > How can one configure the number of partitions per topic in 0.7? I > understand 0.8 has a command line tool to help with this. Is

is it possible to commit offsets on a per stream basis?

Hi team, This is our current use case: Assume there is a topic with multiple partitions. 1 Create a connector first and create multiple streams from the connector for a topic. 2 Create multiple threads, one for each stream. You can assume the thread's job is to save the message into the database

Re: is it possible to commit offsets on a per stream basis?

1 We can create multiple connectors. From each connector create only one stream. 2 Use a single thread for a stream. In this case, the connector in each thread can commit freely without any dependence on the other threads. Is this the right way to go? Will it introduce any dead lock when multiple

Producer/Consumer questions 0.7

1) Should a producer be aware of which broker to write to or is this somehow managed by Kafka itself. For example, If I have 2 brokers with a configured partition size of 1 will my messages be written in a round-robin type of fashion to each broker or is this completed dependent on which broker

Re: Producer/Consumer questions 0.7

Also, are the consumer offsets store in Kafka or Zookeeper? On Aug 29, 2013, at 11:09 AM, Mark wrote: > 1) Should a producer be aware of which broker to write to or is this somehow > managed by Kafka itself. For example, If I have 2 brokers with a configured > partition size of 1 will my messa

RE: Securing kafka

Another possible solution is to use stunnel to authenticate clients with a certificate. It's a bit harder to spoof a certificate than an IP address. -Original Message- From: Benjamin Black [mailto:b...@b3k.us] Sent: Thursday, August 29, 2013 1:10 PM To: users@kafka.apache.org Subject: R

Re: zookeeper session time out

I have not run into this issue with Kafka but have definitely run into issues with ZK expiring sessions and needing to diagnose why. Looking at GC is obviously very important for this. When you turn on gc logging make sure that you include a timestamp in the gc.log filename in your start script. By

RE: is it possible to commit offsets on a per stream basis?

Thanks, Neha. That is a great answer. Regards, Libo -Original Message- From: Neha Narkhede [mailto:neha.narkh...@gmail.com] Sent: Thursday, August 29, 2013 1:55 PM To: users@kafka.apache.org Subject: Re: is it possible to commit offsets on a per stream basis? 1 We can create multiple

Looking for a Kafka 0.8 consultant

Hello Kafka community, We are in the process of enabling Kafka 0.8 in production. We need to use it, rather than 0.7 because we need the ACK functionality and replication. We are running into some show-stopper issues (i.e. Kafka servers freezing) and need someone to come in for a day or so to help

Re: Num of streams for consumers using TopicFilter.

So what is best way to load balance multiple consumers consuming from topic filter. Let's say we have 4 topics with 8 partitions and 2 consumers. Option 1) To load balance consumers, we can set num.streams=4 so that both consumers split 8 partitions. but can only use half of consumer streams. Op

Re: Producer/Consumer questions 0.7

On Thu, Aug 29, 2013 at 11:09 AM, Mark wrote: > 1) Should a producer be aware of which broker to write to or is this > somehow managed by Kafka itself. For example, If I have 2 brokers with a > configured partition size of 1 will my messages be written in a round-robin > type of fashion to each b

Re: Producer/Consumer questions 0.7

On Thu, Aug 29, 2013 at 11:11 AM, Mark wrote: > Also, are the consumer offsets store in Kafka or Zookeeper? > Zookeeper. > > On Aug 29, 2013, at 11:09 AM, Mark wrote: > > > 1) Should a producer be aware of which broker to write to or is this > somehow managed by Kafka itself. For example, If

Re: Producer/Consumer questions 0.7

I'm assuming for this auto-detection of partitions is only with the ZookeeperConsumerConnector and not the ConsumerConnector? On Aug 29, 2013, at 1:04 PM, Philip O'Toole wrote: > On Thu, Aug 29, 2013 at 11:09 AM, Mark wrote: > >> 1) Should a producer be aware of which broker to write to or is

What are my options? (Ruby/Rails environment)

We are thinking about using Kafka to collect events from our Rails application and I was hoping to get some input from the Kafka community. Currently the only gems available are: https://github.com/acrosa/kafka-rb https://github.com/bpot/poseidon (Can't use since we are only running 1.8.7) Now

Re: Producer/Consumer questions 0.7

On Thu, Aug 29, 2013 at 1:28 PM, Mark wrote: > I'm assuming for this auto-detection of partitions is only with the > ZookeeperConsumerConnector and not the ConsumerConnector? > Plain ol' ConsumerConnector does it for us, in our Java-based high-level consumers. > > On Aug 29, 2013, at 1:04 PM,

JMX

Can you view Kafka metrics via JConsole? I've tried connecting to port with no such luck?

Re: JMX

Maybe kafka is on a different port? I do 'ps -ef | grep kafka' and look at the command line to see the jmx port usually. -Suren On Thu, Aug 29, 2013 at 6:41 PM, Mark wrote: > Can you view Kafka metrics via JConsole? I've tried connecting to port > with no such luck? --

Re: JMX

I tried changing the ports and still no luck. Does it work with JConsole and/or do I need anything in my class path? On Aug 29, 2013, at 3:44 PM, Surendranauth Hiraman wrote: > Maybe kafka is on a different port? > > I do 'ps -ef | grep kafka' and look at the command line to see the jmx port

Re: JMX

What options is Kafka running with? On Thu, Aug 29, 2013 at 3:59 PM, Mark wrote: > I tried changing the ports and still no luck. Does it work with JConsole > and/or do I need anything in my class path? > > > On Aug 29, 2013, at 3:44 PM, Surendranauth Hiraman < > suren.hira...@sociocast.com> wro

Re: JMX

I should note this is Kafka 0.7 On Aug 29, 2013, at 3:59 PM, Mark wrote: > I tried changing the ports and still no luck. Does it work with JConsole > and/or do I need anything in my class path? > > > On Aug 29, 2013, at 3:44 PM, Surendranauth Hiraman > wrote: > >> Maybe kafka is on a diffe

Re: JMX

I can't remember my success with JConsole but I know it works with jmxterm - http://wiki.cyclopsgroup.org/jmxterm/tutorial On Thu, Aug 29, 2013 at 7:02 PM, Mark wrote: > I should note this is Kafka 0.7 > > On Aug 29, 2013, at 3:59 PM, Mark wrote: > > > I tried changing the ports and still no l

Re: JMX

I use JConsole, works fine. On Aug 29, 2013, at 7:03 PM, Surendranauth Hiraman wrote: > I can't remember my success with JConsole but I know it works with jmxterm > - http://wiki.cyclopsgroup.org/jmxterm/tutorial > > > On Thu, Aug 29, 2013 at 7:02 PM, Mark wrote: > >> I should note this is

Re: JMX

Strange.. looks like that works. Not sure if because I am using that locally whereas jconsole and visualvm are remote On Aug 29, 2013, at 4:03 PM, Surendranauth Hiraman wrote: > I can't remember my success with JConsole but I know it works with jmxterm > - http://wiki.cyclopsgroup.org/jmxterm/

Re: JMX

I needed to add the hostname to get it working.. -Djava.rmi.server.hostname=${HOSTNAME} On Aug 29, 2013, at 4:47 PM, Mark wrote: > Strange.. looks like that works. Not sure if because I am using that locally > whereas jconsole and visualvm are remote > > On Aug 29, 2013, at 4:03 PM, Surendr

Mirrormaker 0.7

I'm having a lot of trouble getting throughput of mirrormaker to be consistent. On ec2, I have ~10 topics with pretty different throughputs, mirroring from 4 regions into 1. Total of about 20MBps. Sometimes it seems to be going fine, and then it seems like one of the consumers stops altogether, a

Re: Securing kafka

+1 We don't have any application-level security at this time so the answer is whatever you can do at the network/system level. -Jay On Thu, Aug 29, 2013 at 10:09 AM, Benjamin Black wrote: > IP filters on the hosts. > On Aug 29, 2013 10:03 AM, "Calvin Lei" wrote: > > > Is there a way to stop

Re: Unable to send and consume compressed events.

After I sent 1,000 compressed events, I saw these messages in broker's log files: in kafka-request.log [2013-08-30 08:38:18,713] TRACE Processor 6 received request : Name: TopicMetadataRequest; Version: 0; CorrelationId: 0; ClientId: ; Topics: topic1 (kafka.network.RequestChannel$) [2013-08-30 08

Re: Unable to send and consume compressed events.

Update: Sending compressed events with console producer works: kafka-console-producer.bat --broker-list localhost:9092 --sync --topic topic1 --compress I am working on Windows 7. On Fri, Aug 30, 2013 at 8:40 AM, Lu Xuechao wrote: > After I sent 1,000 compressed events, I saw these messages in

Re: Securing kafka

One use case I have been discussing recently with a few clients is verifying the digital signature of a message as part of the acceptance criteria of it being committed to the log and/or when it is consumed. I would be very interested in discussing different scenarios such as Kafka as a service, p

Re: Securing kafka

We have made changes to kafka code to support certificate based mutual SSL authentication. So the clients and broker will exchange trusted certificates for successful communication. This provides both authentication and ssl encryption. Planning to contribute that code back to kafka soon. Thanks, R

Re: Question on # of partitions

In 0.7, total # partitions = # brokers * #partitions per broker. In 0.8, total # partitions is determined at topic creation time and it doesn't change with the addition of brokers (changing # partitions requires using a separate admin tool). Thanks, Jun On Thu, Aug 29, 2013 at 8:55 AM, Mark w

Re: Securing kafka

are the certificate stores by topic? very interesting!!! looking forward to trying it out and review it /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop

Re: Securing kafka

No certificates are not per topic. It is for entire broker. Thanks, Raja. On Thu, Aug 29, 2013 at 11:33 PM, Joe Stein wrote: > are the certificate stores by topic? very interesting!!! looking forward to > trying it out and review it > > /*** > Joe Stein

Re: Num of streams for consumers using TopicFilter.

You can always use more partitions to get more parallelism in the consumers. Thanks, Jun On Thu, Aug 29, 2013 at 12:44 PM, Rajasekar Elango wrote: > So what is best way to load balance multiple consumers consuming from topic > filter. > > Let's say we have 4 topics with 8 partitions and 2 cons

Re: What are my options? (Ruby/Rails environment)

I assume this for Kafka 0.7. One option is to use a VIP in front of the brokers for load balancing. Thanks, Jun On Thu, Aug 29, 2013 at 1:39 PM, Mark wrote: > We are thinking about using Kafka to collect events from our Rails > application and I was hoping to get some input from the Kafka com

Re: Num of streams for consumers using TopicFilter.

Hi Jun, If you read my previous posts, based on current re balancing logic, if we consumer from topic filter, consumer actively use all streams. Can you provide your recommendation of option 1 vs option 2 in my previous post? Thanks, Raja. On Thu, Aug 29, 2013 at 11:42 PM, Jun Rao wrote: > Yo

Re: Mirrormaker 0.7

The following may help: https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Howtoimprovethethroughputofaremoteconsumer%3F Thanks, Jun On Thu, Aug 29, 2013 at 4:47 PM, Walter King wrote: > I'm having a lot of trouble getting throughput of mirrormaker to be > consistent. On ec2, I have

Re: Unable to send and consume compressed events.

These are the metadata requests. Do you see Producer requests from your client? Thanks, Jun On Thu, Aug 29, 2013 at 5:40 PM, Lu Xuechao wrote: > After I sent 1,000 compressed events, I saw these messages in broker's log > files: > > in kafka-request.log > > [2013-08-30 08:38:18,713] TRACE Pro

Re: Num of streams for consumers using TopicFilter.

The drawback of 2), as you said is no auto failover. I was suggesting that you use 16 partitions. Then you can use option 1) with 8 streams in each consumer. Thanks, Jun On Thu, Aug 29, 2013 at 8:51 PM, Rajasekar Elango wrote: > Hi Jun, > > If you read my previous posts, based on current re ba

Re: Num of streams for consumers using TopicFilter.

With option 1) I can't really use 8 streams in each consumer, If I do only one consumer seem to be doing all work. So I had to actually use total 8 streams with 4 for each consumer. On Fri, Aug 30, 2013 at 12:01 AM, Jun Rao wrote: > The drawback of 2), as you said is no auto failover. I was su

Re: Unable to send and consume compressed events.

No. On Fri, Aug 30, 2013 at 11:57 AM, Jun Rao wrote: > These are the metadata requests. Do you see Producer requests from your > client? > > Thanks, > > Jun > > > On Thu, Aug 29, 2013 at 5:40 PM, Lu Xuechao wrote: > > > After I sent 1,000 compressed events, I saw these messages in broker's > l

Re: Unable to send and consume compressed events.

Hi Jun, Thanks for you help. Finally, I found the reason by enabling producer side DEBUG info output. The snappy jar is not included in the classpath. Added it and it worked. Thanks again. On Fri, Aug 30, 2013 at 12:53 PM, Lu Xuechao wrote: > No. > > > On Fri, Aug 30, 2013 at 11:57 AM, Jun