Hi everyone, In the DSv2 sync this week, we discussed adding a new SQL module, sql-api, that would contain the interfaces for authors to plug in external sources. The rationale for adding this package is that the common logical plans and rules to validate those plans should live in Catalyst, but no classes in catalyst are currently public for plugin authors to extend. Catalyst would depend on the sql-api module to pull in the interfaces that plugin authors implement.
I was just working on moving the proposed TableCatalog interface into a new sql-api module, but I ran into a problem: the new APIs still need to reference classes in Catalyst, like DataType/StructType, AnalysisException, and Statistics. I don’t think it makes sense to move all of the referenced classes into sql-api as well, but I could be convinced otherwise. If we decide not to move them, then that leaves us back where we started: we can either expose the v2 API from the catalyst package, or we can keep the v2 API, logical plans, and rules in core instead of catalyst. Anyone want to weigh in with a preference for how to move forward? rb -- Ryan Blue Software Engineer Netflix
