Re: Missing MapState when Timer fires after restored state

Sorry for the incorrect information, that's not the case. Best, Sihua On 05/19/2018 07:58，sihua zhou wrote： Hi Juho & Stefan, just a quick summarize, good news is that I think I found the bug, bad news is that we are currently at the RC4... The bug is here. try (RocksIteratorWrapper iter

Re: Missing MapState when Timer fires after restored state

Hi Juho & Stefan, just a quick summarize, good news is that I think I found the bug, bad news is that we are currently at the RC4... The bug is here. try (RocksIteratorWrapper iterator = getRocksIterator(restoreDb, columnFamilyHandle)) { int startKeyGroup = stateBackend.getKeyGroupRange().getS

RE: How to Flink can solve this example

Hi Esa :) I don't think that there are people here that want to make your homework. Think about, read flink's documentation and ask concrete questions. Anyway - I took a look and think that it's possible. Regards, Georgi From: Esa Heikkinen [mailto:esa.heikki...@student.tut.fi] Sent: Thursday, M

Re: entrypoint for executing job in task manager

Well this thread was super helpful. I'm also looking for ways to integrate DI In particular something like the JVM init hooks for TM and JM would be nice. On Wed, Mar 21, 2018 at 5:08 PM, Steven Wu wrote: > Thanks, let me clarify the requirement. Sorry that it wasn't clear in the > original em

Flink Forward Berlin 2018 - Call for Presentations open until 4th June

Hi all, Flink Forward returns to Berlin on September 3-5, 2018. We are happy to announce the Call for Presentations is now open! Please submit a proposal if you'd like to present your Apache Flink experience, best practices or use case in front of an international audience of highly skilled and e

Stream to Stream Join Memory Management

Hello! I'm trying to do a simple DataStream to DataStream join. Have two kafka topics that has common field. I'm trying to join by via keyBy-join-where-equalTo-TumblingWindow API in Flink 1.4.1. My tumbling window size is 1 day. There will be more data than machine has memory. I know that Flink u

Re: chained operator with different parallelism question

Functions with different parallelism cannot be chained. Chaining means that Functions are fused into a single operator and that records are passed by method calls (instead of serializing them into an in-memory or network channel). Hence, chaining does not work if you have different parallelism. Be

chained operator with different parallelism question

someStream.filter(...).map(...).map(...); there operators are supposed to chained. but what if there are set different parallelism like below: someStream.filter(...).setParallelism(X).map(...).setParallelism(Y).map(...).setParallelism(Z); X != Y != Z what will happen? -- Sent from: http://apache

Re: Fwd: Decrease initial source read speed

Hi, Right now it is a Kafka source, but I had the same issue when reading data from local FS. It looks like a common problem for many (all?) sources. When incoming data is very small (paths to large archives) but each entry requires a significant time to process (unpack, parse, etc.) Flink detec

Re: Fwd: Decrease initial source read speed

Andrei Shumanski: which source are u using? -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Missing MapState when Timer fires after restored state

Hi Juho, thanks for trying this out. I'm running out of myself now... Let's do bref summarize. - have we reached a consensus that the map state losts when restoring?(because you can restore successfully without rescaling) - the timer state is correctly restore, because for timer, when restoring

Re: Missing MapState when Timer fires after restored state

Tested with http://people.apache.org/~trohrmann/flink-1.5.0-rc4/flink-1.5.0-bin-hadoop27-scala_2.11.tgz . It hits the same problem. Btw, why is this error logged on INFO level? 2018-05-18 09:03:52,595 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph- PlatformIDsProcessFunction

Re: Missing MapState when Timer fires after restored state

Thanks Sihua, I'll give that RC a try. On Fri, May 18, 2018 at 10:58 AM, sihua zhou wrote: > Hi Juho, > would you like to try out the latest RC(http://people.apache.org/~ > trohrmann/flink-1.5.0-rc4/) to rescaling the job from the "problematic" > checkpoint? The latest RC includes a fix for the

Re: Missing MapState when Timer fires after restored state

> If you say that you can reproduce the problem, does that mean reproduce from the single existing checkpoint Yes. I haven't tried creating another checkpoint and rescaling from it. I can try that. > We are including rescaling in some end-to-end tests now and then let’s see what happens. If I u

Re: Missing MapState when Timer fires after restored state

Hi Juho, would you like to try out the latest RC(http://people.apache.org/~trohrmann/flink-1.5.0-rc4/) to rescaling the job from the "problematic" checkpoint? The latest RC includes a fix for the potential silently data lost. If it's the reason, you will see a different exception when you tryin

Re: Missing MapState when Timer fires after restored state

Hi, I had a look at the logs from the restoring job and couldn’t find anything suspicious in them. Everything looks as expected and the state files are properly found and transferred from S3. We are including rescaling in some end-to-end tests now and then let’s see what happens. If you say th

Re: are there any ways to test the performance of rocksdb state backend?

Hi, Sihua is right, of course we would like to update our RocksDB version but are currently blocked on a performance regression. Here is our issue in the RocksDB tracker for this: https://github.com/facebook/rocksdb/issues/3865 . Best, Stefan

Re: Missing MapState when Timer fires after restored state

I see. I appreciate keeping this option available even if it's "beta". The current situation could be documented better, though. As long as rescaling from checkpoint is not officially supported, I would put it behind a flag similar to --allowNonRestoredState. The flag could be called --allowRescal