Re: NOT IN subquery optimization

2020-03-26 Thread Li, Zheng
>BTW, so far as I can see, the only reason you're bothering with the whole thing is to compare the size of the subquery output with work_mem, because that's all that subplan_is_hashable does. I wonder whether that consideration is even still necessary in the wake of 1f39bce02. If

Re: NOT IN subquery optimization

2020-03-24 Thread Li, Zheng
Hi Tom, Thanks for the feedback. * I find it entirely unacceptable to stick some planner temporary fields into struct SubLink. If you need that storage you'll have to find some other place to put it. But in point of fact I don't think you need it; it doesn't look to me to be

Re: NOT IN subquery optimization

2020-01-06 Thread Li, Zheng
e of the queries significantly. On your test cases (from the comments of the patch) the subquery hashing has the same execution time with queries No.13-17. At the queries No.1-12 it is not so slow as without hashing, but works more slowly (up to 3 orders) than NOT IN optimizati

Re: Converting NOT IN to anti-joins during planning

2019-06-14 Thread Li, Zheng
- The big "IF" here is if we can calculate the size of the subplan to know if it'll be hashed or not at the point in planning where this conversion is done. I personally can't quite see how that'll work reliably without actually planning the subquery, which I really doubt is

Re: NOT IN subquery optimization

2019-03-01 Thread Li, Zheng
wrote: On Sat, 2 Mar 2019 at 12:13, Tom Lane wrote: > > "Li, Zheng" writes: > > Although adding "or var is NULL" to the anti join condition forces the planner to choose nested loop anti join, it is always faster compared to the original plan. >

Re: NOT IN subquery optimization

2019-03-01 Thread Li, Zheng
Thanks all for the feedbacks! I'm working on a refined patch. Although adding "or var is NULL" to the anti join condition forces the planner to choose nested loop anti join, it is always faster compared to the original plan. In order to enable the transformation from NOT IN to anti join when the

Re: NOT IN subquery optimization

2019-02-26 Thread Li, Zheng
I'm totally fine with setting the target to PG13. -- I'm interested to know how this works without testing for inner nullability. If any of the inner side's join exprs are NULL then no records can match. What do you propose to work around that? -- We still check for inner side's nullability, whe

Re: NOT IN subquery optimization

2019-02-26 Thread Li, Zheng
I agree we will need some runtime smarts (such as a new anti join type as pointed out by Richard) to "ultimately" account for all the cases of NOT IN queries. However, given that the March CommitFest is imminent and the runtime smarts patent concerns David had pointed out (which I was not aware