Hello here, At Pycon US earlier this year I had a number of interesting conversations and one of the - very interesting - conversations I had was with the Ibis team and I thought maybe we should consider releasing "common.dataframe" provider for Airflow - following up after "common.sql" and "common.io".
Ibis is gaining a lot of popularity recently and it might be at more-or-less the same "place" as fsspec when Bolke added "common.io". Plus if airflow adds it as a community provider, it might also bring Ibis' popularity up. In short - Ibis is a "Portable Python dataframe library". It becomes more and more popular and it not only serves 20+ dataframe backends with the same, portable API, but also allows to mix SQL with dataframes and few more things. Some time ago there were some ideas that we could add "SQLAlchemy" as an additional "common" interface in "common.sql" - but actually it seems that Ibis provides a much better abstraction that unifies SQL and Dataframe approach nicely - way better suited for the "data science" world of Airflow. You can see very nice overview "why Ibis" here: https://ibis-project.org/why - and I think it would be pretty natural thing to add on top of "common.sql" and "common.io" - following "Airflow As a Platform" mantra. WDYT? J.
