I set sql.exec.runtime-filter.wait to true. HiveTableSource take much longer time but get same result. I think the reason is not commit preAggregateAccumulator. But I dont know why it happens?
JingsongLee <lzljs3620...@aliyun.com> 于 2020年3月2日周一 下午3:22写道： > Hi, > > Does runtime filter probe side wait for building runtime filter? > Can you check the start time of build side and probe side? > > Best, > Jingsong Lee > > ------------------------------------------------------------------ > From:faaron zheng <faaronzh...@gmail.com> > Send Time:2020年3月2日(星期一) 14:55 > To:user <user@flink.apache.org> > Subject:Question about runtime filter > > Hi, everyone > > These days, I am trying to implement runtime filter in flink1.10 with > flink-sql-benchmark according to blink. I mainly change three part of > flink code: add runtime filter rule; modify the code gen and bloomfilter; > add some aggregatedaccumulator methods according to accumulator. Now, It > seems runtime filter works in execution graph as follows: > Source: HiveTableSource(i_item_sk, i_item_id, i_rec_start_date, > i_rec_end_date, i_item_desc, i_current_price, i_wholesale_cost, i_brand_id, > i_brand, i_class_id, i_class, i_category_id, i_category, i_manufact_id, > i_manufact, i_size, i_formulation, i_color, i_units, i_container, > i_manager_id, i_product_name) TablePath: tpcds_bin_orc_2.item, > PartitionPruned: false, PartitionNums: null, ProjectedFields: [0, 10, 12] > -> Calc(select=[i_item_sk], where=[((i_category = > _UTF-16LE'Jewelry':VARCHAR(2147483647) CHARACTER SET "UTF-16LE") AND > (i_class = _UTF-16LE'consignment':VARCHAR(2147483647) CHARACTER SET > "UTF-16LE") AND RUNTIME_FILTER_BUILDER_0(i_item_sk))]) > > and > > Source: HiveTableSource(d_date_sk, d_date_id, d_date, d_month_seq, > d_week_seq, d_quarter_seq, d_year, d_dow, d_moy, d_dom, d_qoy, d_fy_year, > d_fy_quarter_seq, d_fy_week_seq, d_day_name, d_quarter_name, d_holiday, > d_weekend, d_following_holiday, d_first_dom, d_last_dom, d_same_day_ly, > d_same_day_lq, d_current_day, d_current_week, d_current_month, > d_current_quarter, d_current_year) TablePath: tpcds_bin_orc_2.date_dim, > PartitionPruned: false, PartitionNums: null, ProjectedFields: [0, 3] -> > Calc(select=[d_date_sk, d_month_seq], where=[RUNTIME_FILTER_2(d_date_sk)]) > > > However,the number of records sent is the same as normal. Anyone who can > give me some advices? > > > > Thanks > >
