I do think the URI format for airflow connections also serves the purpose of conveying the intent, similar to how we URI in a browser to connect to a website. We use Ariflow Connection URI to connect to one of the services supported by Airflow. Due to this reason, I lean toward Jarek's proposal of adding "airflowconnection+" or just "airflow+" prefix to the schema, which should also solve its misinterpretation as a common URI. Also, I share the concerns regarding the breaking changes.
So I would like to vote -1. Thanks, Utkarsh Sharma On Sun, Nov 19, 2023 at 3:21 AM Bolke de Bruin <bdbr...@gmail.com> wrote: > So -1 on deprecating. I think we should do the opposite and go all in. The > suggestion by Jarek makes sense in this case, as it remains and is fully > standards compliant. The JSON representation isn't: it is our own crafted > format. > > My 2 cents, > > Bolke > > On Sat, 18 Nov 2023 at 22:15, Jarek Potiuk <ja...@potiuk.com> wrote: > > > I agree with Ash that it's much easier to define a lot of simpler > > connections this way. But it is also very confusing the way now how you > > can mix the "real" url with "airflow connection" URL. > > And Daniel is very right about the magnitude of breaking change. > > > > But possibly there is a way to eat cake and have it too. > > > > Answering Andrey's thinking: 1) I propose to deprecate it (raise warning) > > and even remove it from documentation > > But following Daniel's "pain": 2) possibly we should not even schedule > it > > for removal for Airflow 3 > > And answering Ash's concern: 3) we should likely figure a non-confusing > way > > how to define connection in "airlfow way". > > > > We could be following SQLlAlchemy approach - and define connection with > > "airflowconnection+" prefix in the schema > > > > AIRFLOW_CONN_MY_DB=airflowconnection+postgresql://user:pass@host > > > > I think this would remove all the confusion, keep an easy way of defining > > connection but also avoid "heavily" breaking change. > > > > J. > > > > > > > > On Sat, Nov 18, 2023 at 5:41 PM Daniel Standish > > <daniel.stand...@astronomer.io.invalid> wrote: > > > > > The thing that makes *me* hesitant to deprecate is the sheer magnitude > of > > > breaking it would bring (even though we're only talking about a > > > hypothetical 3.0 release), balanced against the actual pain it causes. > > > I.e. it's confusing to use, and takes up space in docs (when, if > removed, > > > we could just show one good way of doing things), but, it doesn't > create > > > that much of a maintenance burden anymore, and has certainly become > more > > > reliable and easy to use over time (e.g. since we added get_uri). > > > > > > > > > On Sat, Nov 18, 2023 at 3:30 AM Ash Berlin-Taylor <a...@apache.org> > > wrote: > > > > > > > -1 — for the simple case the URI format is perfect: > > > > > > > > AIRFLOW_CONN_MY_DB=postgresql://user:pass@host > > > > > > > > We can push JSON format for more complex cases, and/or limit URI to > > only > > > > the simple cases as long as we don’t remove it entirely. > > > > > > > > -ash > > > > > > > > > On 17 Nov 2023, at 22:44, Andrey Anshin <andrey.ans...@taragol.is> > > > > wrote: > > > > > > > > > > Greetings! > > > > > > > > > > I want to propose to deprecate Airflow Connection URI > representation > > > and > > > > > remove it in Airflow 3 in favor of the already existing > replacement - > > > > JSON > > > > > representation. > > > > > > > > > > In the past URI representation helped to add one of the awesome > > > features > > > > - > > > > > Alternative Secrets Backends: Environment Variables, Files or > > Provider > > > > > specific Backends. > > > > > > > > > > However I have a feeling that nowadays it have some sort of > > limitations > > > > due > > > > > to the parser, see: > > > > > - https://github.com/apache/airflow/issues/33442 > > > > > > > > > > Or miss interpretation that Airflow connection it is something more > > > than > > > > > just representation of Airflow Connection and it is a the > replacement > > > to > > > > > any other type of URIs such as: HTTP, Postgres URIs, SQLAlchemy > URI, > > > > Spark > > > > > URI and etc > > > > > > > > > > And attempt to fix it > > > > > - https://github.com/apache/airflow/issues/10913 > > > > > - https://github.com/apache/airflow/pull/27796 > > > > > - https://github.com/apache/airflow/pull/35712 > > > > > > > > > > And attempt to use it in DbApiHooks, for detail see: > > > > > - https://lists.apache.org/thread/8rhmz3qh30hvkondct4sfmgk4vd07mn5 > > > > > > > > > > In addition, complicated Connection in the URI format is not human > > > > > readable, requires additional decoding, and it is hard to create it > > > > > manually. > > > > > > > > > > The only one beneficial part of URI representation is ability to > > create > > > > it > > > > > programatically > > > > > > > > > > > > > > > https://airflow.apache.org/docs/apache-airflow/stable/howto/connection.html#generating-a-connection-uri > > > > > , there is no such of method for generate JSON, but I don't see the > > > > problem > > > > > to create such of method in Connection object which return string > > > > > representation of JSON connection. > > > > > > > > > > So my suggestion would be pretty simple: > > > > > 1. Deprecate accept URI as parameter to the Connection object > > > > > 2. Deprecate uri property > > > > > 3. Replace get_uri by get_json > > > > > 4. Replace all URI examples in Providers Connections by JSON > > > alternatives > > > > > > > > > > > > > > > WDYT? Maybe I miss something and we should keep Airflow Connection > as > > > URI > > > > > in the future Airflow's major versions and support both ways to > > provide > > > > > connections as JSON and URI. > > > > > > > > > > ---- > > > > > Best Wishes > > > > > *Andrey Anshin* > > > > > > > > > > > > > > > > -- > > -- > Bolke de Bruin > bdbr...@gmail.com >
