Re: Kafka High Level Consumer

2014-09-12 Thread Joe Stein
/*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop> / On Fri, Sep 12, 2014 at 5:49 AM, Rahul Mittal wrote: > Hi , &

Re: Dynamic partitioning

2014-09-12 Thread Joe Stein
) /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop> / On Fri, Sep 12, 2014 at 11:33 AM, István wrote: > Hi al

Re: Getting replicas back in sync

2014-09-12 Thread Joe Stein
. /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop> / On Fri, Sep 12, 2014 at 9:04 PM, Stephen S

Re: Why is Scala 2.9.2 version of kafka recommended?

2014-09-16 Thread Joe Stein
That is for the broker. It is what was most used in production (by LinkedIn) prior to the official release. The client is binary and language independent. You can run 2.10 (for the broker) too, no probs. /*** Joe Stein Founder, Principal Consultant Big

Re: Log 4j Conversion Pattern not working kafka.producer.KafkaLog4jAppender in Kafka.0.8.1.1

2014-09-22 Thread Joe Stein
This should be fixed in 0.8.2 https://issues.apache.org/jira/browse/KAFKA-847 you can apply the v2 patch https://issues.apache.org/jira/secure/attachment/12664646/KAFKA-847-v2.patch backwards if you want too. /*** Joe Stein Founder, Principal Consultant

Apache Kafka Meetup => Strata Hadoop World => NYC

2014-09-22 Thread Joe Stein
even want to talk for a few minutes about how you are using Kafka too =8^) /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop> /

Re: BadVersion state in Kafka Logs

2014-09-30 Thread Joe Stein
src release issues (and the jars in the repo)). I will send that on dev/user in a little bit (please comment +1 community support please on that thread for the release). /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http

Re: kafka & docker

2014-09-30 Thread Joe Stein
-broker.sh e.g. https://github.com/stealthly/metrics-kafka/blob/master/bootstrap.sh#L13 /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop <http://www.twitter.com/allthingshad

[DISCUSS] 0.8.1.2 Release

2014-09-30 Thread Joe Stein
hat happen). Once that is all done, I will build 0.8.1.2 release artifacts and call a VOTE for RC1. /******* Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop> /

Re: Still Stale TopicMetadata

2014-09-30 Thread Joe Stein
I believe this is the ticket https://issues.apache.org/jira/browse/KAFKA-972 /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop <http://www.twitter.com/allthingshad

Re: kafka & docker

2014-09-30 Thread Joe Stein
e though for persisting if need be. /*********** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop> / On Tue, Sep 30, 2014 at 3

Re: BadVersion state in Kafka Logs

2014-09-30 Thread Joe Stein
tates) something is going on that may not just be fixed by a single patch. /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop <http://www.twitter.com/allthin

Re: BadVersion state in Kafka Logs

2014-09-30 Thread Joe Stein
herd affect up/down stream occuring. You need to figure out what is causing the long session timeout and resolve that, IMHO. Zookeeper health is the first place to look. Next would be the network. /*** Joe Stein Founder, Principal Consultant Big Data Open

Re: BadVersion state in Kafka Logs

2014-09-30 Thread Joe Stein
ausing the timeouts. On Tue, Sep 30, 2014 at 6:33 PM, Joe Stein wrote: > It sounds like you have a much deeper rooted problem. Is zookeeper > swapping? Something has to be causing this. After you fix this symptom > you will probably start to see constant leader elections and the isr >

Re: Created topic by 2 partitions, only can use the one partition

2014-10-01 Thread Joe Stein
and see if it works trying to explicitly write to both partitions, if that works then see above and if it doesn't work then you should get an error. /******* Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly

Re: Reassigning Partition Failing

2014-10-02 Thread Joe Stein
What version of zookeeper are you running? First check to see if there is a znode for the "/admin/reassign_partitions" in zookeeper. If so, you could try a graceful shutdown of the controller broker. Once the new controller leader elects on another broker look at zk the znode "/admin/reassign_pa

Re: kafka & docker

2014-10-02 Thread Joe Stein
apter? > > Best Regards, > Mingtao > > Best Regards, > Mingtao > > On Tue, Sep 30, 2014 at 3:31 PM, Joe Stein wrote: > > > << Is there a 'Kafka->HDFS with Camus' docker as well one can play > around > > with? > > > > Not that I

Re: map reduce Outputformat KafkaOutputFormat

2014-10-04 Thread Joe Stein
atest/CDH5-Installation-Guide/cdh5ig_flume_package_install.html for Kafka <=> Hadoop. You can also use Spark read from kafka create RDD save in HDFS or the latest Storm HDFS connector as a bolt from the Kafka Spout too. Sorta depends on your use case. /*****

Re: programmatically get number of items in topic/partition

2014-10-05 Thread Joe Stein
https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-OffsetAPI /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop

Re: Reassigning Partition Failing

2014-10-06 Thread Joe Stein
Agreed, I think it is also a replacement scenario since we may want to bring in a new broker for the dead broker. We should support explicit remove also. We could add to the ReassignPartitionsCommand for an explicit "make new broker just like the old broker". This is important very much in cloud

Re: kafka-web-console

2014-10-13 Thread Joe Stein
rm -f /root/.ivy2/.sbt.ivy.lock On Mon, Oct 13, 2014 at 8:39 PM, Sa Li wrote: > All, > > Again, I am still unable to install, seems to stuck on ivy.lock, any ideas > to continue? > > thanks > > Alec > > On Oct 12, 2014, at 7:38 PM, Sa Li wrote: > > > Hi > >

Re: [DISCUSS] Release 0.8.2-beta before 0.8.2?

2014-10-16 Thread Joe Stein
+1 for a 0.8.2-beta next week and 0.8.2.0 final 4-5 weeks later. I agree to the tickets you brought up to have in 0.8.2-beta and also https://issues.apache.org/jira/browse/KAFKA-1493 for lz4 compression. /*** Joe Stein Founder, Principal Consultant Big

Re: Kafka producer iOS and android

2014-10-20 Thread Joe Stein
production services to pick how to write that producer. That producer and interface should change much and you have to think a little about configurations but most all of that is server side depending on the message. /*** Joe Stein Founder, Principal

Re: Clean Kafka Queue

2014-10-21 Thread Joe Stein
can start fresh easier. That should work in the mean time. 0.8.2-beta should be out this week :) /******* Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop <http://www.twitter.com/al

Re: [DISCUSS] Release 0.8.2-beta before 0.8.2?

2014-10-21 Thread Joe Stein
nal I think yes. I just sent a vote on the dev thread for 0.8.2-beta feel free to comment/vote on that thread if folks feel different about having KAFKA-1647 in the beta to make sure we get the most out of it then we can roll another RC once it is in. /*** Joe

Re: Changing partitions dynamically from client end

2014-10-24 Thread Joe Stein
ticket (or such) for the admin request/response changes too. /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop <http://www.twitter.com/allthingshad

Re: Changing partitions dynamically from client end

2014-10-24 Thread Joe Stein
the response from it /******* Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop> / On Fri, Oct 24, 2014 at 4:04 PM, David Ch

[ANNOUNCEMENT] Apache Kafka 0.8.2-beta Released

2014-10-27 Thread Joe Stein
The Apache Kafka community is pleased to announce the beta release for Apache Kafka 0.8.2. The 0.8.2-beta release introduces many new features, improvements and fixes including: - A new Java producer for ease of implementation and enhanced performance. - Delete topic support. - Per topic conf

Re: [ANNOUNCEMENT] Apache Kafka 0.8.2-beta Released

2014-10-30 Thread Joe Stein
Hey, yeah! For the new producer https://archive.apache.org/dist/kafka/0.8.2-beta/java-doc/ The java consumer is slated in 0.9 more on that here https://cwiki.apache.org/confluence/display/KAFKA/Kafka+0.9+Consumer+Rewrite+Design /*** Joe Stein Founder

Re: Dynamically adding Kafka brokers

2014-11-03 Thread Joe Stein
Most folks strip the IP and use that as the broker.id. KAFKA-1070 does not yet accommodate for that very widely used method. I think it would be bad if KAFKA-1070 only worked for new installations because that is how people use Kafka today (per https://issues.apache.org/jira/browse/KAFKA-1070?focus

No longer supporting Java 6, if? when?

2014-11-05 Thread Joe Stein
tly. /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop> /

Re: powered by kafka

2014-11-10 Thread Joe Stein
I added both Cityzen Data and Exoscale to the Wiki. Please feel free to edit and expand on how more. If you need edit permission let me know. /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter

Re: powered by kafka

2014-11-10 Thread Joe Stein
nd growing) > > Yann. > ________ > From: Joe Stein > Sent: Monday, November 10, 2014 2:03 PM > To: users@kafka.apache.org > Subject: Re: powered by kafka > > I added both Cityzen Data and Exoscale to the Wiki. Please feel free to > edit and expand on

Re: powered by kafka

2014-11-10 Thread Joe Stein
ng > kafka for over a year for stream processing and log transfer (over 2M > messages/s and growing) > > > > Yann. > > > > From: Joe Stein > > Sent: Monday, November 10, 2014 2:03 PM > > To: users@kafka.apache.org &

Re: zookeeper snapshot files eat up disk space

2014-11-11 Thread Joe Stein
http://zookeeper.apache.org/doc/r3.4.6/zookeeperAdmin.html#Ongoing+Data+Directory+Cleanup /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop <http://www.twitter.

Re: Security in 0.8.2 beta

2014-11-11 Thread Joe Stein
a different secrete key encrypted with your public key) The other ones I can't talk about =8^) but this one I mention is interesting solution to this problem with Kafka I really like. /******* Joe Stein Founder, Principal Consultant Big Data Open

Re: One of two consumers is always Idle though I have 2 partitions

2014-11-13 Thread Joe Stein
. /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop / On Nov 13, 2014 8:34 PM, "Chia-Chun Shih" wrote: > Hi Palur, > > When producing messages, did you sp

Re: powered by kafka

2014-11-15 Thread Joe Stein
Hey, I added you to the power by page and also ekaf to the Client Library page too https://cwiki.apache.org/confluence/display/KAFKA/Clients#Clients-Erlang /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http

Re: Programmatic Kafka version detection/extraction?

2014-11-15 Thread Joe Stein
. Something like that, yeah. I will create the JIRA when I get back to computer desk later today. /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop

Re: selecting java producer (0.8.2 or 0.8.1.1?)

2014-11-18 Thread Joe Stein
0.8.2-beta java doc https://archive.apache.org/dist/kafka/0.8.2-beta/java-doc/ /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop / On

Re: Partition Key Cannot be Send Out by Producer

2014-11-20 Thread Joe Stein
iI would helpful to see the full stack trace. Also, how have you instantiated your Producer class? Did you set a value for "serializer.class" in the property? /******* Joe Stein Founder, Principal Consultant Big Data Open Source Security

Re: Partition Key Cannot be Send Out by Producer

2014-11-20 Thread Joe Stein
, hopefully the former =8^) /******* Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop> /

Re: Elastsic Scaling

2014-11-20 Thread Joe Stein
sking currently (as per the wire protocol https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol ) it is all about how the consumer is designed and developed. /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC

Re: Elastsic Scaling

2014-11-20 Thread Joe Stein
more later for sure. /******* Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop> / On Thu, Nov 20, 2014 at 11:57 PM, Daniel Compton wr

Re: Elastsic Scaling

2014-11-20 Thread Joe Stein
Meant to say "burst 1,000,000 messages per second on those X partitions for 10 minutes" On Fri, Nov 21, 2014 at 12:12 AM, Joe Stein wrote: > You need to be thoughtful about adding more partitions. This is paramount > if you are doing semantic partitioning in which case adding

Re: Is there a plan to build a ubiquitous web service API to manage the kafka cluster

2014-11-24 Thread Joe Stein
however they want. Work for that has already gotten underway parent ticket https://issues.apache.org/jira/browse/KAFKA-1694 /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop

Re: [DISCUSSION] adding the serializer api back to the new java producer

2014-11-25 Thread Joe Stein
in easier with a build/binaries to download and use from maven also. +1 on the change now prior to the 0.8.2 release. - Joe Stein On Mon, Nov 24, 2014 at 11:43 PM, Sriram Subramanian < srsubraman...@linkedin.com.invalid> wrote: > Looked at the patch. +1 from me. > > On 11/24/14

Re: rule to set number of brokers for each server

2014-11-28 Thread Joe Stein
). /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop> / On Fri, Nov 28, 2014 at 12:24 PM, Sa Li wrote: >

Re: Kafka 0.8.2 release - before Santa Claus?

2014-12-01 Thread Joe Stein
If we can have a build ready by Dec 26th I think that is feasible. I could prepare and post that if we think we have the votes and a stable version. /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter

Re: Which consumer to use with Kafka 0.8.2-beta

2014-12-05 Thread Joe Stein
The new consumer client is slated for 0.9.0 https://cwiki.apache.org/confluence/display/KAFKA/Future+release+plan /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop <h

Re: Trying to start Kafka

2014-12-07 Thread Joe Stein
That is for a previous version. This is for the 0.8.2-beta quick start https://kafka.apache.org/082/documentation.html#quickstart /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter

Re: unable to create topic

2014-12-08 Thread Joe Stein
Cloudera Manager utilizes the chroot znode structures so you should connect with --zookeeper localhost:2181/kafka Or whatever value CM has set for the chroot path of your installation. /*** Joe Stein Founder, Principal Consultant Big Data Open Source

Re: Kafka Issue [Corrupted broker]

2014-12-09 Thread Joe Stein
. /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop> / On Tue, Dec 9, 2014 at 2:54 AM, ashendra bansal wrote: > Hi, > >

Re: how to achieve availability with no data loss when replica = 1?

2014-12-10 Thread Joe Stein
no messages will be lost as long as at least one in sync replica remains. /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop <http://www.twitter.com/allthingshad

New Go Kafka Client

2014-12-10 Thread Joe Stein
Hi, we open sourced a new Go Kafka Client http://github.com/stealthly/go_kafka_client some more info on blog post http://allthingshadoop.com/2014/12/10/making-big-data-go/ for those working with Go or looking to get into Go and Apache Kafka. /*** Joe Stein

Re: how to achieve availability with no data loss when replica = 1?

2014-12-10 Thread Joe Stein
r down, and the topic replica is set to 1, the > whole system is still available and the data would go to other partitions > without loss. > > > THANKS again. > > On Thu, Dec 11, 2014 at 12:37 AM, Joe Stein wrote: > > > If you want no data loss then you need to se

Re: csv files to kafka

2014-12-14 Thread Joe Stein
program, or you can use as reference and rewrite it in Java. /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop <http://www.twitter.com/allthingshad

Re: spark kafka batch integration

2014-12-14 Thread Joe Stein
? /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop> / On Sun, Dec 14, 2014 at 8:22 PM, Koert Kuipers wrote: > > hello

Re: Kafka getting some love from Logstash and Elasticsearch

2014-12-15 Thread Joe Stein
Nice! On Mon, Dec 15, 2014 at 11:41 AM, Roger Hoover wrote: > > Joseph, > > That's great! Thank you for writing that plugin. > > Cheers, > > Roger > > On Mon, Dec 15, 2014 at 7:24 AM, Joseph Lawson > wrote: > > > > ?Kafka made some headlines with Logstash announcing their latest version > > bet

Re: Better exception handling in kafka.producer.async.DefaultEventHandler

2014-12-18 Thread Joe Stein
I would suggest to use the new client java producer in 0.8.2-beta http://kafka.apache.org/082/javadoc/org/apache/kafka/clients/producer/KafkaProducer.html it handles the case you brought up (among lots of other goodies). /*** Joe Stein Founder, Principal

Re: Max. storage for Kafka and impact

2014-12-19 Thread Joe Stein
see some comments inline On Fri, Dec 19, 2014 at 11:30 AM, Achanta Vamsi Subhash < achanta.va...@flipkart.com> wrote: > > We require: > - many topics > - ordering of messages for every topic > Ordering is only on a per partition basis so you might have to pick a partition key that makes sense for

Re: Max. storage for Kafka and impact

2014-12-19 Thread Joe Stein
commit your offset in batch post processing. Then you only need more partitions to fetch batches to process in parallel. Regards, Joestein On Fri, Dec 19, 2014 at 12:01 PM, Joe Stein wrote: > > see some comments inline > > On Fri, Dec 19, 2014 at 11:30 AM, Achanta Vamsi Subhash &l

Re: Kafka in C#

2014-12-22 Thread Joe Stein
Another option is a HTTP wrapper around the actual producer and doing a HTTP POST from C# to a REST server e.g. https://github.com/stealthly/dropwizard-kafka-http which I know folks have done successfully. /*** Joe Stein Founder, Principal Consultant Big

Re: Kafka in C#

2014-12-22 Thread Joe Stein
Thunder, can you add that to https://cwiki.apache.org/confluence/display/KAFKA/Clients#Clients-.net didn't know it existed but cool that it uses Rx On Mon, Dec 22, 2014 at 12:52 PM, Thunder Stumpges wrote: > Hi there, > > We looked at both of these a while back and ended up writing our own ( > h

Re: csv files to kafka

2014-12-28 Thread Joe Stein
works right from the repo better without having to-do much local for others so they don't bump into this and then work around it locally by installing protobuf. /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC

Re: under replicated topics

2014-12-30 Thread Joe Stein
as such... bursty traffic (especially with batched messages) can upset this, yup. /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop <http://www.twitter.com/allthingshad

Re: Kafka getMetadata api

2015-01-02 Thread Joe Stein
sumer/ConsoleConsumer.scala#L196 /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop <http://www.twitter.com/allthin

Re: kafka logs gone after reboot the server

2015-01-02 Thread Joe Stein
That is because your logs are in /tmp which you can change by setting log.dirs to something else. /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop <http://www.twitter.

Re: kafka logs gone after reboot the server

2015-01-02 Thread Joe Stein
you should probably have it be consistent on each node. /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop <http://www.twitter.com/allthingshad

Re: kafka-web-console error

2015-01-02 Thread Joe Stein
which is the default likely for your OS. /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop> **

Re: Kafka getMetadata api

2015-01-02 Thread Joe Stein
t in these scenarios. /******* Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop> / On Fri, Jan 2, 2015 at 8:25 PM, Mukes

Re: Consumer and offset management support in 0.8.2 and 0.9

2015-01-05 Thread Joe Stein
think we should treat the version field not just to stop from breaking the wire protocol calls but also as a "feature flag" preserving upgrades and multiple pathways. I updated the JIRA for the feature flag needs for OffsetFetch and OffsetCommit too. /******

Re: no space left error

2015-01-06 Thread Joe Stein
for shared memory file: /tmp/hsperfdata_root/19721 I don't think this is Kafka related though so what we are talking about partition and retention is not applicable, even though important. /******* Joe Stein Founder, Principal Consultant Big

Re: messages lost

2015-01-06 Thread Joe Stein
counting the message from each partition using https://github.com/edenhill/kafkacat and pipe to wc -l it makes for a nice simple sanity check to where the problem might be. /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http

Re: Consumer and offset management support in 0.8.2 and 0.9

2015-01-07 Thread Joe Stein
and nothing breaks =8^) /******* Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop> / On Wed, Jan 7, 2015 at 1:04 PM,

Re: kafka monitoring

2015-01-08 Thread Joe Stein
You need to export the JMX_PORT for kafka to use on startup before starting up /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop <http://www.twitter.com/allthingshad

Re: metric-kafka problems

2015-01-09 Thread Joe Stein
this help, thanks! /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop> / On Fri, Jan 9

Re: kafka cluster on aws

2015-01-14 Thread Joe Stein
/infrastructure/aws/bastion) and supervisor (https://github.com/stealthly/minotaur/tree/master/supervisor) and after that it is really nice and easy. /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter

Re: Best Go Client Library

2015-01-14 Thread Joe Stein
consuming with Kafka through Sarama. /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop> /

Re: [VOTE] 0.8.2.0 Candidate 1

2015-01-15 Thread Joe Stein
I think that is a change of behavior that organizations may get burned on. Right now there is no delete data feature. If an operations teams upgrades to 0.8.2 and someone decides to delete a topic then there will be data loss. The organization may not have wanted that to happen. I would argue to no

Syslog Producer to Kafka

2015-01-16 Thread Joe Stein
try out and get feedback so that as we continue to stabilize the existing deployments we can build in more usage scenarios. /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop <h

Re: dumping JMX data

2015-01-16 Thread Joe Stein
-statsd-metrics2 and sending to https://github.com/kamon-io/docker-grafana-graphite you can quickly chart and see everything going on. There are also software as a service options too. /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security

Re: Consumer questions

2015-01-17 Thread Joe Stein
either way. The above is less lint in zk long term. It is all just 1s and 0s and just a matter of how many you put together yourself vs take out of the box that are given too you =8^) /*********** Joe Stein Founder, Principal Consultant Big Data Open Source Secur

Re: [VOTE] 0.8.2.0 Candidate 1

2015-01-18 Thread Joe Stein
It works ok in gradle but fails if your using maven. taking a look at the patch you uploaded now. /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop <http://www.twitter.

Re: Issue size message

2015-01-19 Thread Joe Stein
*fetch.message.max.bytes* in your consumers properties (whoever that might be configured for your specific consumer being used) so that they can read that data otherwise you won't see messages downstream. /*** Joe Stein Founder, Principal Consultant

Re: Can I run Kafka Producer in BG and then run a program that outputs to the console?

2015-01-20 Thread Joe Stein
This is a stdin/stdout producer/consumer that works great for (what I think) you are trying to-do https://github.com/edenhill/kafkacat /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter

Re: Isr difference between Metadata Response vs /kafka-topics.sh --describe

2015-01-21 Thread Joe Stein
Sounds like you are bumping into this https://issues.apache.org/jira/browse/KAFKA-1367 /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop <http://www.twitter.

Re: Kafka Producers and Virtual Kafka Endpoint

2015-01-21 Thread Joe Stein
he list of kafka brokers (through a metadatarequest) and their current ownership. More on that here https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-Partitioningandbootstrapping if you are interested. /******

Re: Help: Kafka LeaderNotAvailableException

2015-01-22 Thread Joe Stein
Vishal, Does this error happen every time you are sending? Or just the first time? Joe Stein On Thu, Jan 22, 2015 at 4:33 AM, wrote: > Hi, > Let me overview on the issue that I am facing on producing message in > Kafka: > I have horthonworks HDP-2.1 installed, along that we h

Re: Regarding Kafka release 0.8.2-beta

2015-01-23 Thread Joe Stein
The new consumer is scheduled for 0.9.0. Currently Kafka release candidate 2 for 0.8.2.0 is being voted on. There is an in progress patch to the new consumer that you can try out https://issues.apache.org/jira/browse/KAFKA-1760 /*** Joe Stein Founder

Re: [kafka-clients] Re: [VOTE] 0.8.2.0 Candidate 2 (with the correct links)

2015-01-23 Thread Joe Stein
Should value.serializer in the new java producer be defaulted to Array[Byte] ? I was working on testing some upgrade paths and got this ! return exception in callback when buffer cannot accept message ConfigException: Missing required configuration "value.serializer" which has no defau

Re: [kafka-clients] Re: [VOTE] 0.8.2.0 Candidate 2 (with the correct links)

2015-01-24 Thread Joe Stein
hat exception will make them ask that > question right? > > -Jay > > On Fri, Jan 23, 2015 at 9:06 PM, Joe Stein wrote: > >> Should value.serializer in the new java producer be defaulted to >> Array[Byte] ? >> >> I was working on testing some upgrade paths

Re: [kafka-clients] Re: [VOTE] 0.8.2.0 Candidate 2 (with the correct links)

2015-01-24 Thread Joe Stein
th the old clients and the > conclusion is that serialization is something you basically have to think > about to use Kafka and trying to guess just makes things worse. > > -Jay > > On Sat, Jan 24, 2015 at 2:51 PM, Joe Stein wrote: > >> Maybe. I think the StringSerialzer

Re: [kafka-clients] Re: [VOTE] 0.8.2.0 Candidate 2 (with the correct links)

2015-01-26 Thread Joe Stein
topic deletion, > compaction and random testing > > On Mon, Jan 26, 2015 at 6:14 AM, Neha Narkhede wrote: > >> +1 (binding) >> Verified keys, quick start, unit tests. >> >> On Sat, Jan 24, 2015 at 4:26 PM, Joe Stein wrote: >> >> > That makes sense, tha

Re: Regarding Kafka release 0.8.2-beta

2015-01-26 Thread Joe Stein
Matve wr should add to the documentation experimental so folks that don't know understand. /******* Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingsh

Re: Poll: Producer/Consumer impl/language you use?

2015-01-28 Thread Joe Stein
rs is taking a look at that now unless someone is already working on that? We are using the partition static assignment in the new consumer and just using Mesos for handling re-balance for us. When he gets further along and if it makes sense we will shoot a KIP around people can chat about it on dev.

Re: [kafka-clients] Re: [VOTE] 0.8.2.0 Candidate 3

2015-01-30 Thread Joe Stein
+1 (binding) verified signatures, ran quick start, tests all passed. - Joe Stein On Fri, Jan 30, 2015 at 12:04 PM, Jun Rao wrote: > This is a reminder that the vote will close tomorrow night. Please test > RC3 out and vote before the deadline. > > Thanks, > > Jun > >

Re: [kafka-clients] Re: [VOTE] 0.8.2.0 Candidate 3

2015-02-02 Thread Joe Stein
Huzzah! Thanks Jun for preparing the release candidates and getting this out to the community. - Joe Stein On Mon, Feb 2, 2015 at 2:27 PM, Jun Rao wrote: > The following are the results of the votes. > > +1 binding = 3 votes > +1 non-binding = 1 votes > -1 = 0 votes > 0 = 0

Re: How to configure "asynchronous replication"

2015-02-04 Thread Joe Stein
too (same line --partitions). /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop> /

Re: New Producer - ONLY sync mode?

2015-02-04 Thread Joe Stein
not as so much sync vs async but my crystal ball just back from the shop so well see =8^) /******* Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop <http://www.twitter.com/al

  1   2   3   4   >