Re: Receive Facebook real time updates into Kafka

2015-01-21 Thread Shannon Lloyd
FB will require specific endpoints to POST data to (and it looks like it also requires a GET endpoint that it can query to validate your service), and the data won't be in the format expected by the Dropwizard Kafka API. The amount of work required in implementing a simple REST API to do this yours

Re: latency - how to reduce?

2015-01-19 Thread Shannon Lloyd
This is a good point, even though you mentioned that you also have latency issues locally. I just migrated a 3-node test cluster from m3.large instances to c4.xlarge instances (3-node ZK migrated from m3.medium to c4.large) in an EC2 placement group (better network IO and more consistent latencies)

Configuring location for server (log4j) logs

2015-01-14 Thread Shannon Lloyd
By default Kafka writes its server logs into a "logs" directory underneath the installation root. I'm trying to override this to get it to write logs to an external location so that I can separate all the read/write logs/data from the read-only binaries. But I can't get it to work correctly. I've

Javadoc errors in MetricName when building with Java 8

2015-01-13 Thread Shannon Lloyd
Is Java 8 supported for building Kafka? Or do you only support Java 7? I just noticed that the latest code on the 0.8.2 branch fails on the javadoc in the new MetricName class due to Java 8's javadoc tool being much stricter when checking comments (especially around things like HTML tags). For exa

Version numbers

2015-01-04 Thread Shannon Lloyd
Is there a reason why the 0.8.2 branch (which I assume is essentially a release branch for the pending 0.8.2 release) has the version number statically set to 0.8.2-beta rather than a -SNAPSHOT version (e.g. 0.8.2-SNAPSHOT or similar)? 0.8.2-beta should technically refer to the released beta, yes?

Re: [DISCUSSION] adding the serializer api back to the new java producer

2014-12-17 Thread Shannon Lloyd
Are you guys planning another beta for everyone to try out the changes before you cut 0.8.2 final? Cheers, Shannon On 18 December 2014 at 11:24, Rajiv Kurian wrote: > > Has the mvn repo been updated too? > > On Wed, Dec 17, 2014 at 4:31 PM, Jun Rao wrote: > > > > Thanks everyone for the feedbac

Re: How to cleanly shut down ConsumerConnector

2014-12-12 Thread Shannon Lloyd
Maybe I've just got something misconfigured. Cheers, Shannon On 13 December 2014 at 02:29, Jun Rao wrote: > Hmm, that's what we do in our unit tests and it seems that we can cleanly > shutdown the consumer. > > Thanks, > > Jun > > On Mon, Dec 8, 2014 at 9:17 PM

Re: How to cleanly shut down ConsumerConnector

2014-12-08 Thread Shannon Lloyd
rk.Processor is on the broker. Are you killing the brokers as > well? > > Thanks, > > Jun > > On Thu, Dec 4, 2014 at 5:33 PM, Shannon Lloyd wrote: > > > Hi, > > > > I am using the high-level consumer on 0.8.2-beta. I'm attempting to > close a

How to cleanly shut down ConsumerConnector

2014-12-04 Thread Shannon Lloyd
Hi, I am using the high-level consumer on 0.8.2-beta. I'm attempting to close a ConsumerConnector (actually a handful of connectors), but am not having much luck actually getting it to close cleanly. When I call shutdown on the connector, I see an error in my application's log (these are always IO