Re: using Kafka Streams in connectors?

2016-06-27 Thread Guozhang Wang
Hello Alex, Currently Kafka Connect has some simple "T" function on a per-message-basis since 0.10 release, but that may not be sufficient for your use case. We are planning to have some Kafka Streams / Connect integration in the near future, so that users can specify non-Kafka sources / sinks in

Re: Kafka Streams reducebykey and tumbling window - need final windowed KTable values

2016-06-27 Thread Eno Thereska
Hi Clive, We are working on exposing the state store behind a KTable as part of allowing for queries to the structures currently hidden behind the language (DSL). The KIP should be out today or tomorrow for you to have a look. You can probably do what you need using the low-level processor API

More OS packages, please!

2016-06-27 Thread Andrew Pennebaker
Could Kafka please publish more OS packages, so it's easier for people to install? * Gentoo/emerge * RedHat/rpm * Debian/apt -- Cheers, Andrew

Re: Kafka Streams backend for Apache Beam?

2016-06-27 Thread Jay Kreps
Even though I'm not aware of anyone working on it, I think we'd definitely be open to it if someone wants to take a swing at it. -Jay On Sun, Jun 26, 2016 at 9:13 AM, Alex Glikson wrote: > Hi all, > > I wonder whether there are plans to implement Apache Beam backend based on > Kafka Streams? >

Re: delay of producer and consumer in kafka 0.9 is too big to be accepted

2016-06-27 Thread Kafka
can someone please explain why latency is so big for me?thanks > 在 2016年6月25日,下午11:16,Jay Kreps 写道: > > Can you sanity check this with the end-to-end latency test that ships with > Kafka in the tools package? > > https://apache.googlesource.com/kafka/+/1769642bb779921267bd57d3d338591dbdf33842/c

Using Consumer Groups for Archive Processing

2016-06-27 Thread Daniel Schierbeck
Since it is apparently ill-advised to use Kafka for long-term storage I'm investigating ways to archive messages in a standardized way that adds the least possible complexity overhead to consumer applications that may need to process all data ever written to a topic before continuing to stream mess

Re: partition running out of disk space

2016-06-27 Thread Tom Crayford
The broker shuts itself down. Once you've fixed the disk space issue (eg. by attaching more disks) then it'll start back up just fine. Thanks Tom Crayford, Heroku Kafka On Mon, Jun 27, 2016 at 2:44 PM, Nomar Morado wrote: > Hi > > What is kaka's behavior if the partition it is running runs out

partition running out of disk space

2016-06-27 Thread Nomar Morado
Hi What is kaka's behavior if the partition it is running runs out of disk space? Do producers get error? Does the entire kafka stop running? etc... Appreciate your help. Thanks.

Re: Producer Properties

2016-06-27 Thread Chris Barlock
Anybody? Chris From: Chris Barlock/Raleigh/IBM@IBMUS To: users@kafka.apache.org Date: 06/24/2016 04:56 PM Subject:Producer Properties I started porting our code from Kafka 0.8.2.1 to 0.10.0.0 and found my producer code blowing up because of some changes to the config. For