Re: kafka streaming: passing config string to Processor

2016-04-18 Thread rss rss
Hello, thanks, I plan to carry on some tests. And I will support sources in actual state - https://github.com/rssdev10/streaming-benchmarks/tree/master/kafka-benchmarks. As the Kafka 10 is absent in apache mirrors and in maven central it is impossible to use default yahoo's start script. Workaro

Re: kafka streaming: passing config string to Processor

2016-04-17 Thread Guozhang Wang
Hello, I think Kafka Streams is better treated as one approach in streaming processing systems for a variety of customers. For example, say if you already have a YARN cluster, and you have a dedicated team operating it and many teams wants to use this for their various streaming jobs, then submitt

Re: kafka streaming: passing config string to Processor

2016-04-17 Thread rss rss
Thanks for the answer. But is it correct in this case to use yahoo streaming benchmark to compare Kafka, Flink and Storm? Or Kafka streaming processor is for other category of customers? Best regards 2016-04-17 16:48 GMT+02:00 Matthias J. Sax : > KafkaStreams works quite different than other sys

Re: kafka streaming: passing config string to Processor

2016-04-17 Thread Matthias J. Sax
KafkaStreams works quite different than other systems like Flink/Storm. It is not a system but a library. If you start a KafkaStreams application, it runs locally. Scaling/Parallelism comes into place if you start the same application on multiple nodes. For this, Kafka's parallelization model is u

Re: kafka streaming: passing config string to Processor

2016-04-17 Thread rss rss
Ok... Is it really perform distribution of the field to several instances of a cluster? I expected to see some way via ProcessorContext... But may be this is result of the Flink's experience... Thanks 2016-04-17 12:38 GMT+02:00 Matthias J. Sax : > I guess the simplest way would be to use a const

Re: kafka streaming: passing config string to Processor

2016-04-17 Thread Matthias J. Sax
I guess the simplest way would be to use a constructor parameter: > public static class CampaignProcessor implements > ProcessorSupplier, List> > { > private final String jedis_server; > > public CampaignProcessor(String jedisServer) { > this.jedis_server = jedisServ

kafka streaming: passing config string to Processor

2016-04-16 Thread rss rss
Hello, I'm implementing yahoo streaming benchmark with Kafka streaming processing. How to pass a string into Processor or ValueTransformer instance? Accordingly to yahoo benchmark logic I need to transfer an address of Redis server. https://github.com/rssdev10/streaming-benchmarks/blob/master/ka