Re: If you are an expert in flink sql, then I really need your help...

2018-12-04 Thread Timo Walther
Unfortunately, setting the parallelism per SQL operator is not supported right now. We are currently thinking about a way of having fine-grained control about properties of SQL operators but this is in an early design phase and might take a while Am 04.12.18 um 13:05 schrieb clay: hi

Re: If you are an expert in flink sql, then I really need your help...

2018-12-04 Thread clay4444
hi Timo: first very thank u, I have solve the ploblems, Regarding the problem of too large state, I set the global parallelism to 7 for the program, which solved my problem very well, checkpoint is very fast, but I would like to ask if there is a way to set parallelism for each operator(translat

Re: If you are an expert in flink sql, then I really need your help...

2018-12-04 Thread Timo Walther
Hi, yes this was a unintended behavior that got fixed in Flink 1.7. See https://issues.apache.org/jira/browse/FLINK-10474 Regards, Timo Am 04.12.18 um 05:21 schrieb clay: I have found out that checkpoint is not triggered. Regarding the in operation in flink sql, this sql will trigger che

Re: If you are an expert in flink sql, then I really need your help...

2018-12-03 Thread clay4444
I have found out that checkpoint is not triggered. Regarding the in operation in flink sql, this sql will trigger checkpoint normally. select name,age from user where id in (5102,597816,597830,597817,597818,597819,597805,27,597820,597821,597822,597823,597825,597826,597827,597828,597839,597831,59

Re: If you are an expert in flink sql, then I really need your help...

2018-12-03 Thread clay4444
hi Timo: The LAST_VALUE function simply groups by id and then takes the latest row of data for each primary key. I was inspired by this answer: https://stackoverflow.com/questions/48554999/apache-flink-how-to-enable-upsert-mode-for-dynamic-tables Its implementation is also very simple: class Mid

Re: If you are an expert in flink sql, then I really need your help...

2018-12-03 Thread Timo Walther
Hi, it is very difficult to spot the problem with the little information you gave us. Maybe you can show us a simplified SQL query and the implementation of the `LAST_VALUE` function? An initial guess would be that you are running out of memory such that YARN kills your task manager. If yo

If you are an expert in flink sql, then I really need your help...

2018-12-03 Thread clay4444
I am using flink sql to do some complicated calculations. I have encountered some very difficult problems in this process, so I would like to ask everyone for your help. My goal is to build a data stream with a very accurate result, which is also in line with the Streaming System. The core id