I am pulling Jark and Godfrey who are more familiar with the planner internals.
Best, Dawid On 22/01/2021 20:11, Rex Fenley wrote: > Hello, > > Does anyone have any more information here? > > Thanks! > > On Wed, Jan 20, 2021 at 9:13 PM Rex Fenley <r...@remind101.com > <mailto:r...@remind101.com>> wrote: > > Hi, > > Our job was experiencing high write amplification on aggregates so > we decided to give mini-batch a go. There's a few things I've > noticed that are different from our previous job and I would like > some clarification. > > 1) Our operators now say they have Watermarks. We never explicitly > added watermarks, and our state is essentially unbounded across > all time since it consumes from Debezium and reshapes our database > data into another store. Why does it say we have Watermarks then? > > 2) In our sources I see MiniBatchAssigner(interval=[1000ms], > mode=[ProcTime], what does that do? > > 3) I don't really see anything else different yet in the shape of > our plan even though we've turned on > configuration.setString( > "table.optimizer.agg-phase-strategy", > "TWO_PHASE" > ) > is there a way to check that this optimization is on? We use user > defined aggregate functions, does it work for UDAF? > > Thanks! > > -- > > Rex Fenley | Software Engineer - Mobile and Backend > > > Remind.com <https://www.remind.com/>| BLOG > <http://blog.remind.com/> | FOLLOW US > <https://twitter.com/remindhq> | LIKE US > <https://www.facebook.com/remindhq> > > > > -- > > Rex Fenley | Software Engineer - Mobile and Backend > > > Remind.com <https://www.remind.com/>| BLOG > <http://blog.remind.com/> | FOLLOW US > <https://twitter.com/remindhq> | LIKE US > <https://www.facebook.com/remindhq> >
signature.asc
Description: OpenPGP digital signature
