Re: Application log on Yarn FlinkCluster

Hi Max, The runBlocking(..) problem was due to a Netty dependency issue on my project, it works fine now :) To pinpoint the logging issue, I just ran a single flink job on yarn as per the documentation "./bin/flink run -m yarn-cluster -yn 2 ./examples/streaming/Iteration.jar" and I have the

Migrating from one state backend to another

I'm trying to decide whether to set-up RocksDB now or later. The state for my Flink jobs right now will be able to fit into memory, so I can use the filesystem state backend. In a few months time it is likely the state will no longer fit into memory, so I will want to use the RocksDB backend. I w

Lazy Evaluation

Hi Mailing List, I probably have a problem with the Lazy Evaluation. Depending of the “return” Datatype of my last Transformation (GroupReduce), the integrated Flink Mini Clusters does not start. I have done the following: // Configuration Configuration parameters = new Configuration(); paramet

Send to all in gelly scatter

Hi Is it possible to send a message to all vertices in gelly scatter? How the Ids of all vertices can be found out? Thanks in advance

Re: Custom Barrier?

... and those events are in order On 14 Jun 2016 14:04, "Paul Wilson" wrote: > Hi, > > No these super-structure events only serve the purpose of defining the > boundaries of a join, and do not relate to the keys of the sub-events. > > Thanks, > Paul > > On 14 June 2016 at 10:32, Aljoscha Krettek

Re: HBase reads and back pressure

I would need to restart it to be sure (and when it starts to be stuck, the web interface doesn't give the backpressure anymore), but it seems so. I put a text file as the output and it took 5h to complete: aggregates.writeAsText("hdfs:///user/christophe/flinkHBase"); What is weird is that I have a

Re: Checkpoint takes long with FlinkKafkaConsumer

Hello Hironori, The logs just show that you get stuck in the Kafka consumer polling loop, which does not allow the consumer lock to be released. Thus the Flink part of the consumer is never actually called. To my understanding this does not seem to be a Flink issue. Or at least this is not shown

Re: Custom Barrier?

Hi, No these super-structure events only serve the purpose of defining the boundaries of a join, and do not relate to the keys of the sub-events. Thanks, Paul On 14 June 2016 at 10:32, Aljoscha Krettek wrote: > Hi, > would these super-structure events occur per key? If yes, then I think you >

Re: Checkpoint takes long with FlinkKafkaConsumer

Kostas, I have attached a log file from one of the taskManager. (The same host I executed jstack) I noticed that there are lots of "Marking the coordinator 2147482645 dead" message in the log. MyContinuousProcessingTimeTriggerGlobal in the log is my custom trigger which is based on ContinuousProce

Re: Checkpoint takes long with FlinkKafkaConsumer

Hi Hironori, Could you also provide the logs of the taskManager? As you described, it seems that the consumer is stuck in the polling loop, although Flink polls with a timeout. This would normally mean that periodically it should release the lock for the checkpoints to go through. The logs of

Re: Checkpoint takes long with FlinkKafkaConsumer

Kostas, Thank you for your response. Yes, I am using latest Flink, which is 1.0.3. Thanks, Hironori 2016-06-14 19:02 GMT+09:00 Kostas Kloudas : > Hello Hironori, > > Are you using the latest Flink version? > There were some changes in the FlinkConsumer in the latest releases. > > Thanks, > Kosta

what is flink hardware require????

the official document no

Gelly Scatter/Gather - Vertex update

Hi everybodyIn Gelly scatter/gather when no message is sent to a vertex in one iteration, it will not enter the scatter function in next iteration. Why? I need all vertices enter the scatter function in all iterations, but some of them receive a message and will be updated. thanks in advance

Re: Checkpoint takes long with FlinkKafkaConsumer

Hello Hironori, Are you using the latest Flink version? There were some changes in the FlinkConsumer in the latest releases. Thanks, Kostas > On Jun 14, 2016, at 11:52 AM, Hironori Ogibayashi > wrote: > > Hello, > > I am running Flink job which reads topics from Kafka and write results > to

Checkpoint takes long with FlinkKafkaConsumer

Hello, I am running Flink job which reads topics from Kafka and write results to Redis. I use FsStatebackend with HDFS. I noticed that taking checkpoint takes serveral minutes and sometimes expires. --- 2016-06-14 17:25:40,734 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - C

Re: How to maintain the state of a variable in a map transformation.

Hi Maximilian, Thank you for the response. Yeah its possible to break up global state but its very tricky to merge two local state variables and also I have to refactor my algorithm logic. Is there way where I can create a new object every time in reduce function so that I can assign the compute

Re: Custom Barrier?

Hi, would these super-structure events occur per key? If yes, then I think you can process this using the currently available windowing mechanism by writing a custom WindowAssigner and Trigger. This, of course, assumes that the events arrive in-order, i.e. if A-End arrives before A-Start or if elem