Do we all agree on the conclusion here? I think in a moment a cat will be out of the bag when we release 2.7.0 and it will be rather difficult to change it....
J. On Mon, Jul 24, 2023 at 6:09 AM Wei Lee <weilee...@gmail.com> wrote: > > > That would be a breaking change compared to what has been done so far. I > > think all operators have been implemented with something like > > if deferrable: > > defer() > > elif wait: > > poll() > > It would only be breaking for operators that have wait=False by default > > though, but it might be counter intuitive to have to set 2 booleans to have > > the expected behavior. > > > > Most operators follow this logic, but some may not be affected by it or may > wait (in different ways) regardless of the "deferrable" value. > > 1. When wait_for_termination=True and deferrable=False, we submit and > > "poll" on the worker > > 2. When wait_for_termination=True and deferrable=True, we submit and > > "defer" using Triggerer > > 3. When wait_for_termination=False and deferrable=False, we only submit > > 4. When wait_for_termination=False and deferrable=True, we only submit > > and no deferrable takes place <--- this is where we should warn saying > > "Deferrable=True" does not have any effect as wait_for_termination=False > > > > It appears to me that Phani's suggestion may be a more reasonable solution. > However, in our current implementations, we handle these two variables > separately. Additionally, not all of them are using > "wait_for_termination." Some may be named "wait_for_completion" or even > "asynchronize". This could be a problem as well. > > I think a better way to see it is not a combination of 2 booleans > > (wait/defer) like you present it, but of 3 : wait param, defer param and > > defer config. > > > > In my opinion, it would be better to establish a clear definition of how > the "deferrable" and "wait_.*" parameters work together. This would remove > the necessity of implementing an additional level of logic to handle three > arguments. > > Vandon, Raphael <vand...@amazon.com.invalid> 於 2023年7月22日 週六 上午12:12寫道： > > > > > > Do we know how many of those affected operators have "wait=False" by > > > > > default? I think that should help us to determine the action. > > > > It's not only about those that are False by default. If the user is > > specifying False manually, we are in the same situation. > > And we have no idea how often users override the "wait" value with False. > > > > > > > I would like to propose the following order of precedence: > > > > > > 1. If wait_for_completion=False, it should ignore what the > > default_deferrable is set to > > > and should not wait for completion whether synchronously or > > asynchronously(deferral) > > > > That would be a breaking change compared to what has been done so far. I > > think all operators have been implemented with something like > > if deferrable: > > defer() > > elif wait: > > poll() > > It would only be breaking for operators that have wait=False by default > > though, but it might be counter intuitive to have to set 2 booleans to have > > the expected behavior. > > > > I think a better way to see it is not a combination of 2 booleans > > (wait/defer) like you present it, but of 3 : wait param, defer param and > > defer config. > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > > For additional commands, e-mail: dev-h...@airflow.apache.org > > --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org For additional commands, e-mail: dev-h...@airflow.apache.org
