How to use keyBy on ConnectedStream?

2018-05-10 Thread Ishwara Varnasi
sageType2 However, I get following exception: Caused by: org.apache.flink.api.common.InvalidProgramException: This type (GenericType) cannot be used as key. How to use keyBy with ConnectedStream and ensure that grouped messages are handled by same instance of CoFlatMap? thanks Ishwara Varnasi

Maximizing resource allocation in EMR (EC2)

2018-02-06 Thread Ishwara Varnasi
ocate just two third of available memory to yarn. Thanks for any suggestions. thanks Ishwara Varnasi

Flink on AWS EMR - how to use flink-log4j configuration?

2018-02-01 Thread Ishwara Varnasi
I didn't find an example of flink-log4j configuration while creating EMR cluster for running Flink. What should be passed to "flink-log4j" config? Actual log4j config or path to file? Also, how to see application logs in EMR? thanks Ishwara Varnasi

Scheduled/timed source/sink

2018-01-29 Thread Ishwara Varnasi
What is the best way to add timer to a source/sink? (Apologies if I’m asking the question that’s already being answered elsewhere), also is it possible to use process function as source or sink? Thanks Ishwara

Re: Data exchange between tasks (operators/sources) at streaming api runtime

2018-01-25 Thread Ishwara Varnasi
ing the main Flink application. > Probably would be simpler and easier to maintain. > > Piotrek > >> On 25 Jan 2018, at 13:47, Ishwara Varnasi wrote: >> >> The FLIP-17 is promising. Until it’s available I’m planning to do this: >> extend Kafka consumer and add lo

Re: Data exchange between tasks (operators/sources) at streaming api runtime

2018-01-25 Thread Ishwara Varnasi
the pointers. Ishwara Varnasi Sent from my iPhone > On Jan 25, 2018, at 4:03 AM, Piotr Nowojski wrote: > > Hi, > > As far as I know there is currently no simple way to do this: > Join stream with static data in > https://cwiki.apache.org/confluence/display/FLINK/FL

Data exchange between tasks (operators/sources) at streaming api runtime

2018-01-24 Thread Ishwara Varnasi
start consuming messages. How can I achieve this? thanks Ishwara Varnasi