Hi Tison, Jark Wu:
Thanks for your reply !!!
What's the statebackend are you using? Is it Heap statebackend?
rocksdb backend uses incremental checkpoint.
Could you share the stack traces?
I looked at the flame chart myself and found that it was stuck at the end
of the window where the calculation took place.
I found that checkpoint failed because of watermark. For the same operator,
the difference between max watermark and min watermark is 30 minutes (my
checkpoint interval is 10 minutes). This may be caused by the slow calculation
of windows.
Best forideal
At 2020-04-18 21:51:13, "Jark Wu" <imj...@gmail.com> wrote:
Hi,
What's the statebackend are you using? Is it Heap statebackend?
Best,
Jark
On Sat, 18 Apr 2020 at 07:06, tison <wander4...@gmail.com> wrote:
Hi,
Could you share the stack traces?
Best,
tison.
forideal <fszw...@163.com> 于2020年4月18日周六 上午12:33写道：
Hello friend
I have two SQL, checkpoint fails all the time. One task is to open a sliding
window for an hour, and then another task consumes the output data of the
previous task. There will be no problem with the two tasks submitted separately.
-- first Calculation
-- second Write the calculation to redis
-- first
insertintodw_access_logselecttime_key,query_nor,query_nor_counter,'1'asgroup_keyfrom(selectHOP_START(event_time_fake,interval'1'MINUTE,interval'60'MINUTE)astime_key,query_nor,count(1)asquery_nor_counterfrom(selectRED_JSON_VALUE(request,'$.query_nor')asquery_nor,RED_JSON_VALUE(request,'$.target')astarget,event_time_fakefrom(selectred_pb_parser(body,'request')asrequest,event_time_fakefromaccess_log_source))groupbyquery_nor,HOP(--
sliding window size one hour, step one minute
event_time_fake,interval'1'MINUTE,interval'60'MINUTE))wherequery_nor_counter>100;--
second
insertintodw_sink_access_logselect'fix_key'as`key`,get_json_value(query_nor,query_nor_counter)as`value`--
agg_func
fromdw_access_loggroupbytumble(time_key_fake,interval'1'MINUTE),group_key
Article Link：https://zhuanlan.zhihu.com/p/132764573
Picture Link：
https://pic4.zhimg.com/80/v2-d3b1105b1419fef3ea6b9176085a5597_1440w.jpg
https://pic3.zhimg.com/80/v2-b6ea7b4a8368c4bae03afb94c723bcca_1440w.jpg
Best, forideal
