I also agree with others and aside from the survey, MSSQL was a headache. I think so many pain points would delay Airflow 3 development if we reconsider MSSQL. Maybe any reconsideration should be after Airflow 3?
On Thu, 30 May 2024 at 23:48, Jarek Potiuk <ja...@potiuk.com> wrote: > Agree with all comments above. Also I think bringing MySQL back is going to > make it way more complex to implement some of the improvements we thought > about - mostly async DB operations (only recently - November 2023 async > support has been added to MSSQL and we know from the history that MSSQL > gave us a lot of headache while developing it and there is no reason to > believe it will be different. And "helping in CI" is not going to cut it - > we need every maintainer who wants to implement a new DB change to become > expert on what is different in MSSQL. > > Honestly - if I'd lose 5% of users because their internal rules say > MSSQL-only (and no Postgres, which as mentioned above is widely supported > and popular including Azure) at the expense of better performance, less > resource usage (as we expect with asyncio) delivered faster to remaining > 95% users, then I know what my decision is. > > BTW. That's not really a criteria we use for such decisions about > technology, but unlike Amazon and Google, Microsoft Azure Data Factory > Airflow team is generally absent from any of those discussions we have > here. Despite us reaching out in various ways they have never "Shown" here, > never contributed anything (or at least we have no knowledge about it) - > including contributions, improvements, system tests nor any other > activities in the community. They are simply not giving back to the > community., > > If they did and officially said (and had proven as the Amazon and Google > team did multiple times for their integrations) that they are willing to > support and maintain MSSQL DB, maybe we would reconsider - mostly because > we could have counted on having them step in when needed (again - as it > happened multiple times with Amazon and Google - when we reach out and need > their help we know we can count on it). I don't see a particular reason why > we should support their proprietary technology. > > J. > > > On Fri, May 31, 2024 at 12:16 AM Damian Shaw <ds...@striketechnologies.com > > > wrote: > > > I would say that MSSQL was often marked as "experimental" ( > > > https://airflow.apache.org/docs/apache-airflow/2.6.0/howto/set-up-database.html > ), > > so IMO I don't think the evidence of it only being used by 5% is > > particularly convincing that it wouldn't eventually be popular. Users who > > might want to primarily use MSSQL because of internal corporate > > restrictions might have a large overlap with users who have restrictions > on > > anything that says "experimental". > > > > I think the more important fact is it was a real burden on development, > > and there was no MSSQL champion in the Airflow maintainers. > > > > -----Original Message----- > > From: Andrey Anshin <andrey.ans...@taragol.is> > > Sent: Thursday, May 30, 2024 2:39 PM > > To: dev@airflow.apache.org > > Cc: james.du...@improving.com.invalid > > Subject: Re: [DISCUSS] Restore the SQL server backend > > > > There was a proposal to keep it in the past [1] with a short explanation > > why the maintainers did not want to keep it. > > > > > many Microsoft customers who are using Airflow > > > > Microsoft also supports and participates in the development of > PostgreSQL, > > there is one Core Team member and couple of Major Contributors working in > > Microsoft [2] and in addition a couple years ago Microsoft acquired one > of > > the PostgreSQL vendors [3]. So I would like to believe that Microsoft > also > > could offer different services around PostgreSQL for their customers. > > > > > > [1] Keep Mssql support: > > https://lists.apache.org/thread/ot58ms069z4pyhj786j1m0dqds6lhjks > > [2] PostgreSQL: Contributors Profiles: > > https://www.postgresql.org/community/contributors/ > > [3] Microsoft Acquires Citus Data: > > https://www.citusdata.com/blog/2019/01/24/microsoft-acquires-citus-data/ > > > > On Thu, 30 May 2024 at 21:18, Pierre Jeambrun <pierrejb...@gmail.com> > > wrote: > > > > > I share Jed feeling. The effort required to maintain those compare to > > > the value it actually brings combined with the usage from the survey, > > > it doesn’t seem worth it to me. > > > > > > On Thu 30 May 2024 at 19:16, Jed Cunningham <jedcunning...@apache.org> > > > wrote: > > > > > > > Just for context, here were (roughly) the results from the 2023 > > > > Airflow > > > > survey: > > > > > > > > PostgreSQL: 75% > > > > MySQL: 15% > > > > MSSQL: 5% > > > > > > > > Also, there are already discussions about potentially dropping MySQL > > > > support in Airflow 3. Given all that and the points from the past > > > > vote, I don't think it makes much sense to bring MSSQL back. > > > > > > > > > ________________________________ > > Strike Technologies, LLC (“Strike”) is part of the GTS family of > > companies. Strike is a technology solutions provider, and is not a broker > > or dealer and does not transact any securities related business directly > > whatsoever. This communication is the property of Strike and its > > affiliates, and does not constitute an offer to sell or the solicitation > of > > an offer to buy any security in any jurisdiction. It is intended only for > > the person to whom it is addressed and may contain information that is > > privileged, confidential, or otherwise protected from disclosure. > > Distribution or copying of this communication, or the information > contained > > herein, by anyone other than the intended recipient is prohibited. If you > > have received this communication in error, please immediately notify > Strike > > at i...@striketechnologies.com, and delete and destroy any copies > hereof. > > ________________________________ > > > > CONFIDENTIALITY / PRIVILEGE NOTICE: This transmission and any attachments > > are intended solely for the addressee. This transmission is covered by > the > > Electronic Communications Privacy Act, 18 U.S.C ''2510-2521. The > > information contained in this transmission is confidential in nature and > > protected from further use or disclosure under U.S. Pub. L. 106-102, 113 > > U.S. Stat. 1338 (1999), and may be subject to attorney-client or other > > legal privilege. Your use or disclosure of this information for any > purpose > > other than that intended by its transmittal is strictly prohibited, and > may > > subject you to fines and/or penalties under federal and state law. If you > > are not the intended recipient of this transmission, please DESTROY ALL > > COPIES RECEIVED and confirm destruction to the sender via return > > transmittal. > > >
