phanikumv commented on code in PR #63419: URL: https://github.com/apache/airflow/pull/63419#discussion_r2974024313
########## airflow-core/newsfragments/63419.significant.rst: ########## @@ -0,0 +1,34 @@ +AIP-76: Asset Partitioning + +Airflow 3.2 introduces asset partitioning (time-based), allowing DAGs to produce and consume +partitioned asset events identified by a ``partition_key`` (e.g. ``2026-03-10T09:00:00`` for +an hourly partition, or ``us|2026-03-10T09:00:00`` for a composite key). + +**New timetables** (``airflow.timetables.partitioned``): + +- ``CronPartitionTimetable`` — schedules a producer DAG on a cron expression and emits a + partitioned asset event on each run. +- ``PartitionedAssetTimetable`` — schedules a consumer DAG when the expected partitioned + asset events have arrived. + +**Partition mappers** transform upstream partition keys before passing them to downstream DAGs: + +- ``IdentityMapper`` — passes the key through unchanged (default). +- ``HourlyMapper``, ``DailyMapper``, ``WeeklyMapper``, ``MonthlyMapper``, ``QuarterlyMapper``, + ``YearlyMapper`` — normalize a datetime partition key to a coarser time granularity + (e.g. ``DailyMapper`` maps ``2026-03-10T09:30:00`` → ``2026-03-10``). +- ``ProductMapper`` — applies a separate mapper to each ``|``-delimited segment of a + composite key (e.g. ``ProductMapper(IdentityMapper(), DailyMapper())`` maps + ``"us|2026-03-10T09:30:00"`` → ``"us|2026-03-10"``). +- ``AllowedKeyMapper`` — validates the key is in a fixed allowlist and passes it through + unchanged, raising ``ValueError`` otherwise (e.g. + ``AllowedKeyMapper(["us", "eu", "ap"])`` accepts ``"us"`` but rejects ``"cn"``). + +Mappers can be set globally on a ``PartitionedAssetTimetable`` or overridden per upstream asset via ``partition_mapper_config``. + +The ``partition_key`` is also exposed on ``dag_run.partition_key`` inside task context, and can be supplied when triggering a DAG run manually via the REST API Review Comment: Done -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
