[Note: this question has been moved from the Conversation in [SPARK-4226][SQL]Add subquery (not) in/exists support #9055
to the dev mailing list.] We've added our own In/Exists - plus Subquery in Select - support to a partial fork of Spark SQL Catalyst (which we use in transformations from our own query language to SQL for relational databases). But since In, Exists and Select projections are Expressions which will then contain LogicalPlans (Subquery/Select with nested LogicalPlans with potential nested Expressions) this makes whole-tree transformations really cumbersome since we've got to deal with 'pivot points' for these 2 types of TreeNodes, where a recursive transformation can only be done on 1 specific type of children, and then has to be dealt with again within the same PartialFunction for the other type in which the matching case(s) can be nested. Why was the choice made in Catalyst to make LogicalPlan/QueryPlan and Expression separate subclasses of TreeNode, instead of e.g. also make QueryPlan inherit from Expression? The code also contains duplicate functionality, like LeafNode/LeafExpression, UnaryNode/UnaryExpression and BinaryNode/BinaryExpression. Much obliged, Roland Reumerman mendix.com
