Re: Flink RMQSource Consumer: How I get the RabbitMQ UserId

2018-09-09 Thread vino yang
Hi Marke, As soon as I didn't really implement this code, but I think you can replace this line of code: *OUT result = schema.deserialize(delivery.getBody()); //RMQSource#run* instead of defining an abstract method in RMQSource, such as: normalize/deserialize, the input parameter is Delivery, an

Question about akka configuration for FLIP-6

2018-09-09 Thread Tony Wei
Hi, I'm going to migrate my flink cluster from 1.4.0 to 1.5.3, and I have been trying to map config file to the latest version. I used to use these three configuration. Are they still needed in FLIP-6 mode? Moreover, is any akka config still needed in FLIP-6 mode? Since I had a impression that FLI

Re: Facing Issues while migrating to S3A

2018-09-09 Thread yinhua.dai
Hi, I am still have the same problem, googled many ways but still failed. I have downloaded and added hadoop.dll and winutils.exe to class path. To verify that is working, I called "System.loadLibrary("haddop")" at the beginning of my java program and it succeed. BTW: I run my program in windows

Aggregator State in Keyed Windowed Stream

2018-09-09 Thread Ning Shi
Since the aggregate() function on a keyed window stream does not allow using rich functions, I can only use an AggregateFunction. Is the accumulator state of the AggregateFunction backed by RocksDB and persisted in checkpoints if I use the RocksDB backend. My job looks like the following, sourceSt

Re: Question about akka configuration for FLIP-6

2018-09-09 Thread Gary Yao
Hi Tony, You are right that with FLIP-6 Akka is abstracted away. If you want custom heartbeat settings, you can configure the options below [1]: heatbeat.interval heartbeat.timeout The config option taskmanager.exit-on-fatal-akka-error is also not relevant anymore. I closest I can think

Re: Question about akka configuration for FLIP-6

2018-09-09 Thread Gary Yao
I should add that in FLIP-6 mode we are not relying on Akka's DeathWatch but because Flink's RPC framework uses Akka, you are still able to configure the other Akka config options [1]. [1] https://ci.apache.org/projects/flink/flink-docs-release-1.5/ops/config.html#distributed-coordination-via-akka

Re: Question about akka configuration for FLIP-6

2018-09-09 Thread 陈梓立
Hi Gray, Thanks for your useful information! Here I wonder if the following configs still valid on FLIP-6 mode. 1. akka.transport.heartbeat.interval 2. akka.transport.heartbeat.pause It seems they are different from HeartbeatServices and possibly still valid. Best, tison. Gary Yao 于2018年9月10

[ANNOUNCE] Weekly community update #37

2018-09-09 Thread Till Rohrmann
Dear community, this is the weekly community update thread #37. Please post any news and updates you want to share with the community to this thread. # Flink Forward Berlin 2018 over Last week the community met in Berlin for the 4th edition of Flink Forward 2018 [1]. It was great to see people c

Re: Question about akka configuration for FLIP-6

2018-09-09 Thread Gary Yao
Hi Tison, These can be still be set but judging from the documentation [1], they have never been very relevant in Flink: Heartbeat interval for Akka's transport failure detector. Since Flink uses TCP, the detector is not necessary. Therefore, the detector is disabled by setting the in