I just wondering, what happens if user set _PIP_ADDITIONAL_REQUIREMENTS in our sample https://airflow.apache.org/docs/apache-airflow/2.7.0/docker-compose.yaml? We should also remove it from the sample, it doesn't help in some cases, e.g. users use docker-compose which was provided long time ago, optimize for their requirements and do not check changes in samples, like this one: https://github.com/apache/airflow/issues/33688#issuecomment-1694920053 . In each case we would have situation that we break someone pipeline, we have a choice between: bad, worst and awful, where awful it keep as it is :D
In addition we have `airflow standalone`, maybe we should make work DEBUG on top of it? No idea how to resolve this into docker endpoint :-( ---- Best Wishes *Andrey Anshin* On Thu, 31 Aug 2023 at 09:38, Jarek Potiuk <ja...@potiuk.com> wrote: > Yep. I am also for the idea described by Pierre. In short - setting the > requirement automatically turn on a DEBUG mode with all possible DEBUG > features turned on. > > On Thu, Aug 31, 2023 at 6:03 AM Amogh Desai <amoghdesai....@gmail.com> > wrote: > > > Going through the discussions above, I was leaning towards the idea of > > removing it completely initially but came across Pierre's idea after. I > > like that! > > > > Thanks & Regards, > > Amogh Desai > > > > On Wed, Aug 30, 2023 at 1:58 PM Jarek Potiuk <ja...@potiuk.com> wrote: > > > > > Whoa.. I am glad I started it... I see some really good ideas here :). > > > > > > On Wed, Aug 30, 2023 at 9:20 AM Pierre Jeambrun <pierrejb...@gmail.com > > > > > wrote: > > > > > > > Maybe we can introduce a global “DEBUG” config option/env variable. > > This > > > > could control some more verbose logging but most importantly only > when > > > this > > > > is turned on, we could use the sequential executor, debug executor, > > > > _PIP_ADDITIONAL_REQUIREMENTS > > > > and any other “debug/development” purpose options. > > > > > > > > Like for other frameworks this would be documented and user would > have > > a > > > > warning when using it. > > > > > > > > Then there is nothing more we can do if the user puts in production a > > > > cluster in debug mode, this would be deliberate. (Like set the debug > > > option > > > > to True and then use the sequential executor). > > > > > > > > This could also automatically start the web server in debug mode and > > > other > > > > components in a similar way when detected to true. > > > > > > > > We could even limit the number of workers to 1 or other things that > > would > > > > make it unsuitable in a production environment. > > > > > > > > Just an idea. > > > > > > > > On Wed 30 Aug 2023 at 03:55, Pankaj Koti <pankaj.k...@astronomer.io > > > > .invalid> > > > > wrote: > > > > > > > > > I agree with Maciej's rationale here and inclined towards removing > it > > > > > straight away. > > > > > > > > > > Also, is it the case that users have discovered this flag > themselves > > > > > without us documenting it anywhere as a feature to use? > > > > > > > > > > *We have a leading underscore for the variable and it hints that > it's > > > for > > > > > internal use.* > > > > > > > > > > I would be up for failing the image without any deprecation/warning > > if > > > we > > > > > have not documented it as a feature to use. > > > > > > > > > > On Wed, 30 Aug 2023, 03:16 Oliveira, Niko, > > <oniko...@amazon.com.invalid > > > > > > > > > wrote: > > > > > > > > > > > I'd vote for a period of time with warnings (either in the logs > > > and/or > > > > in > > > > > > the Airflow UI), as a deprecation warning of sorts. Followed by > > > > removing > > > > > > the feature later on, unless we find that the warnings are enough > > to > > > > > lower > > > > > > the operational load this causes us, but I think that's unlikely. > > > > > > > > > > > > Cheers, > > > > > > Niko > > > > > > > > > > > > ________________________________ > > > > > > From: Jed Cunningham <jedcunning...@apache.org> > > > > > > Sent: Tuesday, August 29, 2023 10:05:01 AM > > > > > > To: dev@airflow.apache.org > > > > > > Subject: RE: [EXTERNAL] [DISCUSS] Preventing users from misusing > > > > > > _PIP_ADDITIONAL_REQUIREMENTS ? > > > > > > > > > > > > CAUTION: This email originated from outside of the organization. > Do > > > not > > > > > > click links or open attachments unless you can confirm the sender > > and > > > > > know > > > > > > the content is safe. > > > > > > > > > > > > > > > > > > > > > > > > I also don't like the 10 minute thing. I'd rather we remove it, > or > > > > > display > > > > > > a message like we do sequential executor (we can only do so much, > > > this > > > > is > > > > > > as visible as we can make it really), I think in that order? > > > > > > > > > > > > > > > > > > > > >
