Re: Gradually increasing checkpoint size

2021-03-12 Thread Dan Hill
ckpoint UI ? For example, >> the >> attached operator has a `alreadyOutputed` value state, which seems to keep >> increasing if there are always new keys ? >> >> Best, >> Yun >> >> >> --Original Mail -- >> *Sender:*Dan Hill >&

Re: Gradually increasing checkpoint size

2021-03-11 Thread Dawid Wysakowicz
ted` value state, which seems > to keep > increasing if there are always new keys ? > > Best, > Yun > > > --Original Mail -- > *Sender:*Dan Hill <mailto:quietgol...@gmail.com>> > *Send Date:*Tue Ma

Re: Re: Gradually increasing checkpoint size

2021-03-08 Thread Dan Hill
ttached operator has a `alreadyOutputed` value state, which seems to keep > increasing if there are always new keys ? > > Best, > Yun > > > --Original Mail -- > *Sender:*Dan Hill > *Send Date:*Tue Mar 9 00:59:24 2021 > *Recipient

Re: Re: Gradually increasing checkpoint size

2021-03-08 Thread Yun Gao
, Yun --Original Mail -- Sender:Dan Hill Send Date:Tue Mar 9 00:59:24 2021 Recipients:Yun Gao CC:user Subject:Re: Gradually increasing checkpoint size Hi Yun! Thanks for the quick reply. One of the lowerBounds is large but the table being joined with is ~500 rows

Re: Gradually increasing checkpoint size

2021-03-08 Thread Dan Hill
k. > > Best, > Yun > > > --Original Mail -- > *Sender:*Dan Hill > *Send Date:*Mon Mar 8 14:59:48 2021 > *Recipients:*user > *Subject:*Gradually increasing checkpoint size > >> Hi! >> >> I'm running a backfill Flink

Re: Gradually increasing checkpoint size

2021-03-08 Thread Yun Gao
Hi Dan, Have you use a too large upperBound or lowerBound? If not, could you also check the watermark strategy ? The interval join operator depends on the event-time timer for cleanup, and the event-time timer would be triggered via watermark. Best, Yun --Original Mail -

Gradually increasing checkpoint size

2021-03-07 Thread Dan Hill
Hi! I'm running a backfill Flink stream job over older data. It has multiple interval joins. I noticed my checkpoint is regularly gaining in size. I'd expect my checkpoints to stabilize and not grow. Is there a setting to prune useless data from the checkpoint? My top guess is that my checkpo