+1 - and following the above comments - I also think it might make sense to bring all operators/triggers/sensors from airlfow package to a single provider package (rather than having separate "time", "python", "branch" (or whatever we choose for all the other operators).
This is mainly because there are indeed some inter-dependencies and common classes already used across mutliple operators (for example BaseBranchOperator is used in a number of unrelated branch operators - like BranchSQLOperator in common.sql). Separating them all together in a single package has the drawback that they cannot be upgraded separately from each other - so they are coupled, but on the other hand it makes it easier to release and manage them and avoid cross-dependency issues. In this context - I like the idea of the "common.operators" provider - common has several meanings as Vincent mentioned and some parts of those operators will be actually used by other operators (like the BaseBranch one). Possibly we will find few other "common" things that we will need to pull to that provider as well. J. On Wed, Aug 14, 2024 at 4:11 PM Vincent Beck <vincb...@apache.org> wrote: > I like that idea too! +1. The name does not bother me either, "common" can > refer to "common use cases" or "common usage" which makes sense (at least > to me). > > On 2024/08/14 13:23:26 Elad Kalif wrote: > > Thanks for owning it Rom! > > +1 from me > > > > The common is needed because we have convention where the providers are > > packed under entity > > https://github.com/apache/airflow/tree/main/airflow/providers > > > > Common isnt new. We already have common.compat, common.io, common.sql > > > > Personally I don't mind about the name as long as it is extracted from > core. > > > > I am also +1 for Ash suggestion about having no operators/sensors in > Core. > > I think all of them should be migratrd to providers. > > > > בתאריך יום ד׳, 14 באוג׳ 2024, 16:16, מאת Ash Berlin-Taylor < > a...@apache.org > > >: > > > > > I like the idea, and you beat me to proposing this since in my > prototyping > > > of AIP-72 I realised it would be much much better if 100% of operators > et > > > al were providers. > > > > > > One thing in this case though: we shouldn't have `common` in the name > as > > > that should be a convention of saying "this provider is designed for > use by > > > other providers" but in this case it's for direct use in dags. > > > > > > -ash > > > > > > On 14 August 2024 13:51:42 BST, rom sharon <r...@apache.org> wrote: > > > >Hi, > > > > > > > >I would like to propose a new provider. > > > > > > > >*Background* > > > > > > > >As noted in the comment on this PR > > > ><https://github.com/apache/airflow/pull/41441>, several operators and > > > >sensors related to time are currently located in the core of Airflow: > > > >BranchDayOfWeekOperator, BranchDateTimeOperator, DateTimeSensor, > > > >TimeDeltaSensor, TimeSensor, and DayOfWeekSensor. > > > > > > > >Because these components reside in the core, any modifications to them > > > >require changes to the Airflow core rather than the providers. > > > > > > > >*Suggestion* > > > > > > > >I propose creating a new provider, Common.time, to collect all these > > > >time-related operators and sensors. This would help decouple them > from the > > > >core. > > > > > > > >I'm happy to lead this effort and raise a PR. > > > > > > > >WDYT? > > > > > > > >Rom Sharon > > > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > For additional commands, e-mail: dev-h...@airflow.apache.org > >
