+1, this is very cool. Thanks for adding this, XD!

One use case that comes to mind is Iceberg table maintenance. If you run
Iceberg tables in production, you need to do things like compact data files
and expire old snapshots. Native support in Airflow and Polaris would make
it easier to schedule those jobs as part of your existing Airflow workflows.

A bit about me for those who don’t know me. I’m a PMC member of both Apache
Polaris and Apache Iceberg. I also wrote the policy store and table
maintenance features in Polaris.
Yufei


On Tue, Nov 17, 2026 at 2:16 PM Jarek Potiuk <[email protected]> wrote:

> +1
>
> On Wed, Nov 18, 2026 at 9:01 AM constance.astronomer.io via dev <
> [email protected]> wrote:
>
>> +1 love it
>>
>> > On Nov 17, 2026, at 12:03 PM, Vikram Koka via dev <
>> [email protected]> wrote:
>> >
>> > +1
>> > Seems like a great addition
>> >
>> > Vikram
>> >
>> >
>> >> On Tue, Nov 17, 2026 at 4:18 AM Kaxil Naik <[email protected]>
>> wrote:
>> >>
>> >> +1 Would love to have polaris support.
>> >>
>> >>
>> >> On Tue, 17 Nov 2026 at 04:18, Xiaodong Deng via dev <
>> >> [email protected]>
>> >> wrote:
>> >>
>> >>> Hi Airflow community,
>> >>>
>> >>> We'd like to propose adding a new community provider for Apache
>> Polaris:
>> >>> apache-airflow-providers-apache-polaris.
>> >>>
>> >>> Integration overview
>> >>> ---------------------------
>> >>> Apache Polaris (https://polaris.apache.org/) is an open-source
>> catalog
>> >> for
>> >>> Apache Iceberg. It implements the Iceberg REST catalog API and adds
>> >>> centralized access control, credential vending and catalog-level
>> >> policies.
>> >>> Many teams running Iceberg already orchestrate their data platform
>> with
>> >>> Airflow. Today, managing Polaris from a Dag means hand-written REST
>> calls
>> >>> or shelling out to the Polaris CLI.
>> >>>
>> >>> The proposed provider would add:
>> >>>
>> >>> - A "polaris" connection type. It supports Polaris's internal OAuth
>> >> tokens,
>> >>> and it masks every token it fetches.
>> >>> - Hooks for the Polaris Management API and the catalog APIs. They can
>> >> also
>> >>> hand ready-made catalog properties to pyiceberg or to Spark and other
>> >>> engines.
>> >>> - Operators for catalogs, namespaces, tables, principals, principal
>> and
>> >>> catalog roles, grants, and policies. Typical uses include
>> provisioning a
>> >>> team's catalog with its roles and grants, onboarding a pipeline
>> >> principal,
>> >>> and managing table lifecycle.
>> >>> - A helper that turns Polaris table-maintenance policies (compaction,
>> >>> snapshot expiry, orphan-file removal) into Iceberg Spark procedure
>> calls,
>> >>> so a Dag can execute the maintenance a policy defines.
>> >>>
>> >>> It is built on the official Apache Polaris Python SDK
>> >>> (apache-polaris>=1.9.0). It needs Airflow 3.1.1+ and Python 3.11+, and
>> >>> pyiceberg is an optional extra. It has been tested against Polaris
>> server
>> >>> 1.8 and 1.9, and installs cleanly against Airflow's constraints for
>> every
>> >>> 3.1.1–3.4.0 release.
>> >>>
>> >>> System tests
>> >>> -----------------
>> >>> Polaris is open source and runs as a single container, so no external
>> >> paid
>> >>> service is involved. The PR adds:
>> >>>
>> >>> - a Breeze integration (breeze --integration polaris) that starts
>> Polaris
>> >>> 1.9.0 in CI, with integration tests against it;
>> >>> - system-test example Dags for provisioning, tables and policies,
>> which
>> >> run
>> >>> against the same server.
>> >>>
>> >>> All tests can run in Airflow's own CI, so no external dashboard is
>> >> needed.
>> >>>
>> >>> Proposed stewards
>> >>> -------------------------
>> >>> - Yufei Gu, "flyrain", [email protected] (Apache Polaris
>> PMC/Committer)
>> >>> - Yong Zheng, "MonkeyCanCode", [email protected] (Apache Polaris
>> >>> PMC/Committer)
>> >>> - Xiaodong Deng, "XD-DENG", [email protected] (Apache Airflow
>> >>> PMC/Committer)
>> >>>
>> >>> Sponsoring committer
>> >>> -----------------------------
>> >>> - Xiaodong Deng, "XD-DENG", [email protected]
>> >>>
>> >>> Working implementation
>> >>> --------------------------------
>> >>> Draft PR: https://github.com/apache/airflow/pull/77188. It includes
>> >> unit,
>> >>> integration and system tests, and how-to docs for every operator.
>> >>>
>> >>> Incubation commitment
>> >>> -------------------------------
>> >>> We commit to:
>> >>> - maintaining the provider and responding to issues within a
>> reasonable
>> >>> time;
>> >>> - meeting the incubation health metrics within 6 months;
>> >>> - participating in quarterly governance updates.
>> >>>
>> >>>
>> >>> Regards,
>> >>>
>> >>> XD
>> >>>
>> >>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [email protected]
>> For additional commands, e-mail: [email protected]
>>
>>

Reply via email to