I finally got through the runtimefilter in 1.10, the reason why it didn't call commit method is in OperatorCodeGenerator. It should call endInput() method correctly in generateOneInputStreamOperator. A complete process of runtimefilter is: 1.Add insert and remove rule in batch rules. 2.Build side constructs bloomfilter and commit. 3. Jobmanager merge bloomfilter as a global one. 4. Probe side get global bloomfilter and filter data. Although runtimefilter is already achieved in blink, it doesn't have a independent commit. So it's a little hard to merge whole code once. I hope it helps if anyone try to do same thing.
faaron zheng <faaronzh...@gmail.com> 于 2020年3月2日周一 下午7:52写道： > I set sql.exec.runtime-filter.wait to true. HiveTableSource take much > longer time but get same result. I think the reason is not commit > preAggregateAccumulator. But I dont know why it happens? > > JingsongLee <lzljs3620...@aliyun.com> 于 2020年3月2日周一 下午3:22写道： > >> Hi, >> >> Does runtime filter probe side wait for building runtime filter? >> Can you check the start time of build side and probe side? >> >> Best, >> Jingsong Lee >> >> ------------------------------------------------------------------ >> From:faaron zheng <faaronzh...@gmail.com> >> Send Time:2020年3月2日(星期一) 14:55 >> To:user <user@flink.apache.org> >> Subject:Question about runtime filter >> >> Hi, everyone >> >> These days, I am trying to implement runtime filter in flink1.10 with >> flink-sql-benchmark according to blink. I mainly change three part of >> flink code: add runtime filter rule; modify the code gen and bloomfilter; >> add some aggregatedaccumulator methods according to accumulator. Now, It >> seems runtime filter works in execution graph as follows: >> Source: HiveTableSource(i_item_sk, i_item_id, i_rec_start_date, >> i_rec_end_date, i_item_desc, i_current_price, i_wholesale_cost, i_brand_id, >> i_brand, i_class_id, i_class, i_category_id, i_category, i_manufact_id, >> i_manufact, i_size, i_formulation, i_color, i_units, i_container, >> i_manager_id, i_product_name) TablePath: tpcds_bin_orc_2.item, >> PartitionPruned: false, PartitionNums: null, ProjectedFields: [0, 10, 12] >> -> Calc(select=[i_item_sk], where=[((i_category = >> _UTF-16LE'Jewelry':VARCHAR(2147483647) CHARACTER SET "UTF-16LE") AND >> (i_class = _UTF-16LE'consignment':VARCHAR(2147483647) CHARACTER SET >> "UTF-16LE") AND RUNTIME_FILTER_BUILDER_0(i_item_sk))]) >> >> and >> >> Source: HiveTableSource(d_date_sk, d_date_id, d_date, d_month_seq, >> d_week_seq, d_quarter_seq, d_year, d_dow, d_moy, d_dom, d_qoy, d_fy_year, >> d_fy_quarter_seq, d_fy_week_seq, d_day_name, d_quarter_name, d_holiday, >> d_weekend, d_following_holiday, d_first_dom, d_last_dom, d_same_day_ly, >> d_same_day_lq, d_current_day, d_current_week, d_current_month, >> d_current_quarter, d_current_year) TablePath: tpcds_bin_orc_2.date_dim, >> PartitionPruned: false, PartitionNums: null, ProjectedFields: [0, 3] -> >> Calc(select=[d_date_sk, d_month_seq], where=[RUNTIME_FILTER_2(d_date_sk)]) >> >> >> However,the number of records sent is the same as normal. Anyone who can >> give me some advices? >> >> >> >> Thanks >> >>
