> That would be a breaking change compared to what has been done so far. I > think all operators have been implemented with something like > if deferrable: > defer() > elif wait: > poll() > It would only be breaking for operators that have wait=False by default > though, but it might be counter intuitive to have to set 2 booleans to have > the expected behavior. >
Most operators follow this logic, but some may not be affected by it or may wait (in different ways) regardless of the "deferrable" value. 1. When wait_for_termination=True and deferrable=False, we submit and > "poll" on the worker > 2. When wait_for_termination=True and deferrable=True, we submit and > "defer" using Triggerer > 3. When wait_for_termination=False and deferrable=False, we only submit > 4. When wait_for_termination=False and deferrable=True, we only submit > and no deferrable takes place <--- this is where we should warn saying > "Deferrable=True" does not have any effect as wait_for_termination=False > It appears to me that Phani's suggestion may be a more reasonable solution. However, in our current implementations, we handle these two variables separately. Additionally, not all of them are using "wait_for_termination." Some may be named "wait_for_completion" or even "asynchronize". This could be a problem as well. I think a better way to see it is not a combination of 2 booleans > (wait/defer) like you present it, but of 3 : wait param, defer param and > defer config. > In my opinion, it would be better to establish a clear definition of how the "deferrable" and "wait_.*" parameters work together. This would remove the necessity of implementing an additional level of logic to handle three arguments. Vandon, Raphael <vand...@amazon.com.invalid> 於 2023年7月22日 週六 上午12:12寫道： > > > > Do we know how many of those affected operators have "wait=False" by > > > > default? I think that should help us to determine the action. > > It's not only about those that are False by default. If the user is > specifying False manually, we are in the same situation. > And we have no idea how often users override the "wait" value with False. > > > > I would like to propose the following order of precedence: > > > > 1. If wait_for_completion=False, it should ignore what the > default_deferrable is set to > > and should not wait for completion whether synchronously or > asynchronously(deferral) > > That would be a breaking change compared to what has been done so far. I > think all operators have been implemented with something like > if deferrable: > defer() > elif wait: > poll() > It would only be breaking for operators that have wait=False by default > though, but it might be counter intuitive to have to set 2 booleans to have > the expected behavior. > > I think a better way to see it is not a combination of 2 booleans > (wait/defer) like you present it, but of 3 : wait param, defer param and > defer config. > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > For additional commands, e-mail: dev-h...@airflow.apache.org >
