Lee-W commented on code in PR #63419:
URL: https://github.com/apache/airflow/pull/63419#discussion_r2964866066
##########
airflow-core/newsfragments/63419.significant.rst:
##########
@@ -0,0 +1,34 @@
+AIP-76: Asset Partitioning
+
+Airflow 3.2 introduces asset partitioning (time-based), allowing DAGs to
produce and consume
Review Comment:
```suggestion
Airflow 3.2 introduces asset partitioning (time-based), allowing Dags to
produce and consume
```
##########
airflow-core/newsfragments/63419.significant.rst:
##########
@@ -0,0 +1,34 @@
+AIP-76: Asset Partitioning
+
+Airflow 3.2 introduces asset partitioning (time-based), allowing DAGs to
produce and consume
+partitioned asset events identified by a ``partition_key`` (e.g.
``2026-03-10T09:00:00`` for
+an hourly partition, or ``us|2026-03-10T09:00:00`` for a composite key).
+
+**New timetables** (``airflow.timetables.partitioned``):
+
+- ``CronPartitionTimetable`` — schedules a producer DAG on a cron expression
and emits a
Review Comment:
```suggestion
- ``CronPartitionTimetable`` — schedules a producer Dag on a cron expression
and emits a
```
##########
airflow-core/newsfragments/63419.significant.rst:
##########
@@ -0,0 +1,34 @@
+AIP-76: Asset Partitioning
+
+Airflow 3.2 introduces asset partitioning (time-based), allowing DAGs to
produce and consume
+partitioned asset events identified by a ``partition_key`` (e.g.
``2026-03-10T09:00:00`` for
+an hourly partition, or ``us|2026-03-10T09:00:00`` for a composite key).
+
+**New timetables** (``airflow.timetables.partitioned``):
+
+- ``CronPartitionTimetable`` — schedules a producer DAG on a cron expression
and emits a
+ partitioned asset event on each run.
+- ``PartitionedAssetTimetable`` — schedules a consumer DAG when the expected
partitioned
Review Comment:
```suggestion
- ``PartitionedAssetTimetable`` — schedules a consumer Dag when the expected
partitioned
```
##########
airflow-core/newsfragments/63419.significant.rst:
##########
@@ -0,0 +1,34 @@
+AIP-76: Asset Partitioning
+
+Airflow 3.2 introduces asset partitioning (time-based), allowing DAGs to
produce and consume
+partitioned asset events identified by a ``partition_key`` (e.g.
``2026-03-10T09:00:00`` for
+an hourly partition, or ``us|2026-03-10T09:00:00`` for a composite key).
+
+**New timetables** (``airflow.timetables.partitioned``):
+
+- ``CronPartitionTimetable`` — schedules a producer DAG on a cron expression
and emits a
+ partitioned asset event on each run.
+- ``PartitionedAssetTimetable`` — schedules a consumer DAG when the expected
partitioned
+ asset events have arrived.
+
+**Partition mappers** transform upstream partition keys before passing them to
downstream DAGs:
+
+- ``IdentityMapper`` — passes the key through unchanged (default).
+- ``HourlyMapper``, ``DailyMapper``, ``WeeklyMapper``, ``MonthlyMapper``,
``QuarterlyMapper``,
+ ``YearlyMapper`` — normalize a datetime partition key to a coarser time
granularity
Review Comment:
```suggestion
``ToYearlyMapper`` — normalize a datetime partition key to a coarser time
granularity
```
##########
airflow-core/newsfragments/63419.significant.rst:
##########
@@ -0,0 +1,34 @@
+AIP-76: Asset Partitioning
+
+Airflow 3.2 introduces asset partitioning (time-based), allowing DAGs to
produce and consume
+partitioned asset events identified by a ``partition_key`` (e.g.
``2026-03-10T09:00:00`` for
+an hourly partition, or ``us|2026-03-10T09:00:00`` for a composite key).
+
+**New timetables** (``airflow.timetables.partitioned``):
+
+- ``CronPartitionTimetable`` — schedules a producer DAG on a cron expression
and emits a
+ partitioned asset event on each run.
+- ``PartitionedAssetTimetable`` — schedules a consumer DAG when the expected
partitioned
+ asset events have arrived.
+
+**Partition mappers** transform upstream partition keys before passing them to
downstream DAGs:
+
+- ``IdentityMapper`` — passes the key through unchanged (default).
+- ``HourlyMapper``, ``DailyMapper``, ``WeeklyMapper``, ``MonthlyMapper``,
``QuarterlyMapper``,
+ ``YearlyMapper`` — normalize a datetime partition key to a coarser time
granularity
+ (e.g. ``DailyMapper`` maps ``2026-03-10T09:30:00`` → ``2026-03-10``).
+- ``ProductMapper`` — applies a separate mapper to each ``|``-delimited
segment of a
+ composite key (e.g. ``ProductMapper(IdentityMapper(), DailyMapper())`` maps
+ ``"us|2026-03-10T09:30:00"`` → ``"us|2026-03-10"``).
+- ``AllowedKeyMapper`` — validates the key is in a fixed allowlist and passes
it through
+ unchanged, raising ``ValueError`` otherwise (e.g.
+ ``AllowedKeyMapper(["us", "eu", "ap"])`` accepts ``"us"`` but rejects
``"cn"``).
+
+Mappers can be set globally on a ``PartitionedAssetTimetable`` or overridden
per upstream asset via ``partition_mapper_config``.
+
+The ``partition_key`` is also exposed on ``dag_run.partition_key`` inside task
context, and can be supplied when triggering a DAG run manually via the REST API
+(``POST /dags/{dag_id}/dagRuns``).
+
+* Migration rules needed
+
+ * None — asset partitioning is a new feature. Existing DAGs and assets are
unaffected
Review Comment:
```suggestion
* None — asset partitioning is a new feature. Existing Dags and assets are
unaffected
```
##########
airflow-core/newsfragments/63419.significant.rst:
##########
@@ -0,0 +1,34 @@
+AIP-76: Asset Partitioning
+
+Airflow 3.2 introduces asset partitioning (time-based), allowing DAGs to
produce and consume
+partitioned asset events identified by a ``partition_key`` (e.g.
``2026-03-10T09:00:00`` for
+an hourly partition, or ``us|2026-03-10T09:00:00`` for a composite key).
Review Comment:
```suggestion
an hourly partition).
```
We decided not to mentioned it here for less confussion
##########
airflow-core/newsfragments/63419.significant.rst:
##########
@@ -0,0 +1,34 @@
+AIP-76: Asset Partitioning
+
+Airflow 3.2 introduces asset partitioning (time-based), allowing DAGs to
produce and consume
+partitioned asset events identified by a ``partition_key`` (e.g.
``2026-03-10T09:00:00`` for
+an hourly partition, or ``us|2026-03-10T09:00:00`` for a composite key).
+
+**New timetables** (``airflow.timetables.partitioned``):
+
+- ``CronPartitionTimetable`` — schedules a producer DAG on a cron expression
and emits a
+ partitioned asset event on each run.
+- ``PartitionedAssetTimetable`` — schedules a consumer DAG when the expected
partitioned
+ asset events have arrived.
+
+**Partition mappers** transform upstream partition keys before passing them to
downstream DAGs:
Review Comment:
```suggestion
**Partition mappers** transform upstream partition keys before passing them
to downstream Dags:
```
##########
airflow-core/newsfragments/63419.significant.rst:
##########
@@ -0,0 +1,34 @@
+AIP-76: Asset Partitioning
+
+Airflow 3.2 introduces asset partitioning (time-based), allowing DAGs to
produce and consume
+partitioned asset events identified by a ``partition_key`` (e.g.
``2026-03-10T09:00:00`` for
+an hourly partition, or ``us|2026-03-10T09:00:00`` for a composite key).
+
+**New timetables** (``airflow.timetables.partitioned``):
+
+- ``CronPartitionTimetable`` — schedules a producer DAG on a cron expression
and emits a
+ partitioned asset event on each run.
+- ``PartitionedAssetTimetable`` — schedules a consumer DAG when the expected
partitioned
+ asset events have arrived.
+
+**Partition mappers** transform upstream partition keys before passing them to
downstream DAGs:
+
+- ``IdentityMapper`` — passes the key through unchanged (default).
+- ``HourlyMapper``, ``DailyMapper``, ``WeeklyMapper``, ``MonthlyMapper``,
``QuarterlyMapper``,
+ ``YearlyMapper`` — normalize a datetime partition key to a coarser time
granularity
+ (e.g. ``DailyMapper`` maps ``2026-03-10T09:30:00`` → ``2026-03-10``).
Review Comment:
```suggestion
(e.g. ``ToDailyMapper`` maps ``2026-03-10T09:30:00`` → ``2026-03-10``).
```
##########
airflow-core/newsfragments/63419.significant.rst:
##########
@@ -0,0 +1,34 @@
+AIP-76: Asset Partitioning
+
+Airflow 3.2 introduces asset partitioning (time-based), allowing DAGs to
produce and consume
+partitioned asset events identified by a ``partition_key`` (e.g.
``2026-03-10T09:00:00`` for
+an hourly partition, or ``us|2026-03-10T09:00:00`` for a composite key).
+
+**New timetables** (``airflow.timetables.partitioned``):
+
+- ``CronPartitionTimetable`` — schedules a producer DAG on a cron expression
and emits a
+ partitioned asset event on each run.
+- ``PartitionedAssetTimetable`` — schedules a consumer DAG when the expected
partitioned
+ asset events have arrived.
+
+**Partition mappers** transform upstream partition keys before passing them to
downstream DAGs:
+
+- ``IdentityMapper`` — passes the key through unchanged (default).
+- ``HourlyMapper``, ``DailyMapper``, ``WeeklyMapper``, ``MonthlyMapper``,
``QuarterlyMapper``,
Review Comment:
```suggestion
- ``ToHourlyMapper``, ``ToDailyMapper``, ``ToWeeklyMapper``,
``ToMonthlyMapper``, ``ToQuarterlyMapper``,
```
##########
airflow-core/newsfragments/63419.significant.rst:
##########
@@ -0,0 +1,34 @@
+AIP-76: Asset Partitioning
+
+Airflow 3.2 introduces asset partitioning (time-based), allowing DAGs to
produce and consume
+partitioned asset events identified by a ``partition_key`` (e.g.
``2026-03-10T09:00:00`` for
+an hourly partition, or ``us|2026-03-10T09:00:00`` for a composite key).
+
+**New timetables** (``airflow.timetables.partitioned``):
+
+- ``CronPartitionTimetable`` — schedules a producer DAG on a cron expression
and emits a
+ partitioned asset event on each run.
+- ``PartitionedAssetTimetable`` — schedules a consumer DAG when the expected
partitioned
+ asset events have arrived.
+
+**Partition mappers** transform upstream partition keys before passing them to
downstream DAGs:
+
+- ``IdentityMapper`` — passes the key through unchanged (default).
+- ``HourlyMapper``, ``DailyMapper``, ``WeeklyMapper``, ``MonthlyMapper``,
``QuarterlyMapper``,
+ ``YearlyMapper`` — normalize a datetime partition key to a coarser time
granularity
+ (e.g. ``DailyMapper`` maps ``2026-03-10T09:30:00`` → ``2026-03-10``).
+- ``ProductMapper`` — applies a separate mapper to each ``|``-delimited
segment of a
+ composite key (e.g. ``ProductMapper(IdentityMapper(), DailyMapper())`` maps
+ ``"us|2026-03-10T09:30:00"`` → ``"us|2026-03-10"``).
+- ``AllowedKeyMapper`` — validates the key is in a fixed allowlist and passes
it through
+ unchanged, raising ``ValueError`` otherwise (e.g.
+ ``AllowedKeyMapper(["us", "eu", "ap"])`` accepts ``"us"`` but rejects
``"cn"``).
+
+Mappers can be set globally on a ``PartitionedAssetTimetable`` or overridden
per upstream asset via ``partition_mapper_config``.
+
+The ``partition_key`` is also exposed on ``dag_run.partition_key`` inside task
context, and can be supplied when triggering a DAG run manually via the REST API
Review Comment:
```suggestion
The ``partition_key`` is also exposed on ``dag_run.partition_key`` inside
task context, and can be supplied when triggering a Dag run manually via the
REST API
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
