uranusjr commented on PR #63848: URL: https://github.com/apache/airflow/pull/63848#issuecomment-4116370278
I have a use case question. Say I have a downstream workflow that wants to react to partitioned upstreams, but doesn’t really care about the particular partitions (maybe to do some global calculations or cache busting etc). Is there a way to do that, or is the downstream forced to have some kind of partition even if its tasks (or even further downstream workflows that depend on _its_ asset events) just ignore it? Or maybe the design question is more fundamental. An asset event means the asset has changed in some way, and there are really two ways you may want to be notified when change happens: Only on a “full” change (e.g. file replacement, deletion), or on _any_ change. The former kind shouldn’t trigger on a partitioned update, but the latter would want to. Is there a way to distinguish the two? When should this distinguishing logic happen? On the upstream (when an asset event is emitted), or the downstream (when listening to an asset)? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
