Re: Reactive Kafka performance

2016-08-02 Thread Michael Noll
David, you wrote: > Each task would effectively block on DB-io for every history-set retrieval; > obviously we would use a TTL cache (KTable could be useful here, but it > wouldn’t be able to hold “all” of the history for every user) Can you elaborate a bit why you think a KTable wouldn't be abl

Re: Reactive Kafka performance

2016-07-30 Thread David Garcia
Hey Guozhang, our basic road block is asynchronous processing (this is actually related to my previous post about asynchronous processing). Here is the simplest use-case: The streaming job receives messages. Each message is a user-event and needs to randomly look up that user’s history (for 1

Re: Reactive Kafka performance

2016-07-30 Thread Guozhang Wang
Hello David, I'd love to learn details about the "flexibility" of Reactive Kafka compared with KStreams in order to see if KStreams can improve on that end. Would you like to elaborate a bit more on that end? Guozhang On Thu, Jul 28, 2016 at 12:16 PM, David Garcia wrote: > Our team is evaluat

Reactive Kafka performance

2016-07-28 Thread David Garcia
Our team is evaluating KStreams and Reactive Kafka (version 0.11-M4) on a confluent 3.0 cluster. Our testing is very simple (pulling from one topic, doing a simple transform) and then writing out to another topic. The performance for the two systems is night and day. Both applications were ru