>BTW, so far as I can see, the only reason you're bothering with the whole thing is to compare the size of the subquery output with work_mem, because that's all that subplan_is_hashable does. I wonder whether that consideration is even still necessary in the wake of 1f39bce02. If it is, I wonder whether there isn't a cheaper way to figure it out. (Note similar comment in make_subplan.)
The comment in make_subplan says there is no cheaper way to figure out: /* At present, however, we can only check hashability after * we've made the subplan :-(. (Determining whether it'll fit in work_mem * is the really hard part.) */ I don't see why commit 1f39bce02 is related to this problem. Can you expand on this? >But can't you detect that case directly? It seems like you'd need to figure out the NULL situation anyway to know whether the transformation to antijoin is valid in the first place. Yes, we do need to figure out the NULL situation, and there is always valid transformation to antijoin, it's just in the NULL case we need to stuff additional clause to the anti join condition, and in these cases the transformation actually outperforms Subplan (non-hashed), but underperforms the hashed Subplan. The unmodified anti hash join has similar performance compared to hashed Subplan.