Vikram - no core changes needed. Provider will be enough Daniel: yes I was also considering that. Main reason why I think common.dataframe will be better is that it will follow the same pattern we have with common providers - SQL and Io. Namely that we use connections (and Hooks) coning feon various other providers and use them to build the authentication information. I.e. I imagine the implementatio where we define a DataframeHook and let DuckDb, Bigquery and other hooks implement it - returning the right url that ibis would recognize. It looks like those are slightly different than any other URLs we have - because there is no common url schema that would be universal we need to find a place where airflow connection with authentication information is mapped to Ibis-recognised url. And similarly like in case of common.sql, the right hook will be instantiated based on the connection id, and the 'dataframe' url will have to be returned by the hook.
That means that a number of our providers will depend on this 'ibis' provider because the hooks wil have to implement that DataframeHook. It could be Ibis provider and Ibis Hook as well, but with 20+ supported integrations - out of which we have already providers and Hooks implementing Airflow's connection and authentication schema, 'common.daraframe' seems like a better name. It could be just 'ibis' provider or 'common.ibis' - but also 'common.dataframe' has - I think - better discoverability. J wt., 25 cze 2024, 23:59 użytkownik Daniel Standish <daniel.stand...@astronomer.io.invalid> napisał: > Why would it be common.dataframe instead of providers.ibis? >
