Re: [Announcement] Hermes - pub / sub broker built on top of Kafka

Interesting. Thanks for sharing and working on this! Can you comment on how this compares to Confluent's REST proxy? On Sat, May 16, 2015 at 12:01 AM, Marcin Kuthan wrote: > Hi Everyone > > Hermes is an asynchronous message broker built on top of Kafka. It > provides reliable, fault tolerant RE

Re: [Announcement] Hermes - pub / sub broker built on top of Kafka

Hi Stevo Thank's for your comment. On 16 May 2015 at 17:43, Stevo Slavić wrote: > Nice, thanks for sharing! > > Is 30k msgs/sec publishing or push throughput? Will check, hopefully > performance tests are included in sources. 30k msgs/sec is our regular traffic handled by hermes on 4-nodes clu

Re: [Announcement] Hermes - pub / sub broker built on top of Kafka

Hi Warren With Hermes, the publisher is never blocked. Even if message has not been sent to Kafka cluster, or if message has been sent but not acknowledged. It is especially useful if your system needs to have strict SLA guarantees. >From the consumers perspective there is retrying policy if the

SPARKTA: a real-time aggregation engine

Hi there, We have released our real-time aggregation engine based on Spark Streaming. SPARKTA is fully open source (Apache2) and our goal is that no code would be required to work with it. The most interesting input is Apache Kafka and we are already working to improve the agreggation policies

Why is there no ZookeeperProducerConnector ?

Hi all, I'm looking for a simple way to create *a producer using zookeeper has source for metadata.broker.list.* I found the consumer equivalent : ZookeeperConsumerConnector but no sign of ZookeeperProducerConnector. Thanks for your help. -- *Thomas Pocreau* Recherche & Développement | *Colisw

Re: Partitions reassignment failed due to Partition reassignment currently in progress for Map()

Perfect. Thank you very much! Chris On Sun, May 17, 2015 at 10:41 PM, Clark Haskins wrote: > Yeah you can start using the reassignment tool again. > > It's likely some issue with the controller that put you in this state. > Without gathering the logs it will be hard to tell. > > -Clark > > Sen

Re: Experiences testing new producer performance across multiple threads/producer counts

Garry, Do you mind to share the source code that you did for the profiling? On Sun, May 17, 2015 at 4:59 PM, Garry Turkington < g.turking...@improvedigital.com> wrote: > Hi Guozhang/Jay/Becket, > > Thanks for the responses. > > Regarding my point on performance dropping when the number of partit

Re: [Announcement] Hermes - pub / sub broker built on top of Kafka

Hello everyone, I'm technical team lead of Hermes project. I will try to answer already posted questions, but feel free to ask me anything. 1) Can you comment on how this compares to Confluent's REST proxy? We do not perceive Hermes as mere proxy. While Confluent product wants to help services w

Delete topic pending

If a broker doesn't have the topic, and I run delete topic, that topic will be in a "pending delete" state forever. What am I doing wrong here? Also, what if I have data loss and I just want to delete the dang topic form zookeeper directly with non of this pending stuff. Thanks

Re: Delete topic pending

May be thats a bug. We should probably alert the user saying the topic doesn’t exist rather than go and add it to the zookeeper for deletion trigger. Also to delete any topic you need to set “delete.topic.enable” to true --  Harsha On May 18, 2015 at 10:16:46 AM, Dillian Murphey (crackshotm...

Re: Delete topic pending

Hmm, which Kafka version are you running? From the code it looks we should have already ignored the deletion of a topic if it does not exist. Jiangjie (Becket) Qin On 5/18/15, 10:15 AM, "Dillian Murphey" wrote: >If a broker doesn't have the topic, and I run delete topic, that topic >will >be in

kafkacat

Hi All, Does anyone know of an implementation of kafkacat that reads from socket? Or for that matter any kafka producer client that can read from socket and publish to kafka? thanks Clay

Which Kafka Consumer should I use?

I want to rebuild my message queue from based on Redis to Kafka (because Redis persistence is terrible). Now I am using the new KafkaProducer, and I am wondering which consumer should I use. Currently there are 3 Kafka Consumers (until 0.8.2.1): 1. High Level Consumer 2. Simple Consumer 3. Kaf

Re: Which Kafka Consumer should I use?

I suppose you can either use the High Level Consumer or the simple consumer as per your use case. Simple Consumer does not do offset management and you will have to do it yourself but provides more control. High level consumer on the other hand does the offset management for you. The new Kafka Cons

Re: kafkacat

Flume with netcat source and Kafka Channel or Kafka Sink will do that. A bit more complex than a Kafkacat equivalent, but will get the job done. On Tue, May 19, 2015 at 3:02 AM, clay teahouse wrote: > Hi All, > > Does anyone know of an implementation of kafkacat that reads from socket? > Or for