Kafka and Flink integration

2017-10-12 Thread Gabriele Di Bernardo
Hi everyone, I have a question that is not really related to Flink but maybe someone of you can help me understanding what I am doing wrong. I have a Flink job that processes events generated by a Java application. The output of the Flink job is emitted on Kafka; the Java application runs a Kaf

Kafka 0.11

2017-08-18 Thread Gabriele Di Bernardo
Hi guys, Is the Flink Connector Kafka 0.10 fully compatible with Kafka 0.11? Thank you in advance. Best, Gabriele

Operations dependencies between values with different key in a ConnectedStreams

2017-07-28 Thread Gabriele Di Bernardo
Hi guys, I have a question for you. I have an application with two keyed data streams: one for control and the other one for the data. Each control message represents an operation to be performed on the data values marked with a certain identifier. I connected the two streams and I process the

Type erasure exception

2017-07-24 Thread Gabriele Di Bernardo
Hi guys, When I run my Flink topology (locally) I get this error: The return type of function 'main(Job.java:69)' could not be determined automatically, due to type erasure. You can give type information hints by using the returns(...) method on the result of the transformation call, or by let

Find the running median from a data stream

2017-07-23 Thread Gabriele Di Bernardo
Hi guys, I want to keep track of the running median of a keyed data stream. I was considering to apply a RichMapFunction to the stream and store in a ValueState object two heaps (PriorityQueue) in order to find the running median. However, I am not really sure if this is the best approach perfo

Re: Kafka control source in addition to Kafka data source

2017-07-20 Thread Gabriele Di Bernardo
is makes sense to you? If you share a little bit about the use > case. In particular, it would be relevant if both streams share a common > key, on which they can be partitioned. > > Cheers, > > Konstantin > > On 18.07.2017 21:14, Gabriele Di Bernardo wrote: >> Hell

Kafka control source in addition to Kafka data source

2017-07-18 Thread Gabriele Di Bernardo
Hello everyone, I am a Flink newcomer and I would like to implement a Flink application with two Kafka sources: one for the data stream to be processed and the other one for control purposes. The application should be able to read from the control stream and then apply the control operation to

Latency on Flink

2017-06-05 Thread Gabriele Di Bernardo
Hello everyone, I am a completely newcomer of streaming engines and big data platforms but I am considering using Flink for my master thesis and before starting using it I am trying to do some kind of evaluation of the system. In particular I am interested in observing how the system reacts in