+1 Seems like a great addition Vikram
On Tue, Nov 17, 2026 at 4:18 AM Kaxil Naik <[email protected]> wrote: > +1 Would love to have polaris support. > > > On Tue, 17 Nov 2026 at 04:18, Xiaodong Deng via dev < > [email protected]> > wrote: > > > Hi Airflow community, > > > > We'd like to propose adding a new community provider for Apache Polaris: > > apache-airflow-providers-apache-polaris. > > > > Integration overview > > --------------------------- > > Apache Polaris (https://polaris.apache.org/) is an open-source catalog > for > > Apache Iceberg. It implements the Iceberg REST catalog API and adds > > centralized access control, credential vending and catalog-level > policies. > > Many teams running Iceberg already orchestrate their data platform with > > Airflow. Today, managing Polaris from a Dag means hand-written REST calls > > or shelling out to the Polaris CLI. > > > > The proposed provider would add: > > > > - A "polaris" connection type. It supports Polaris's internal OAuth > tokens, > > and it masks every token it fetches. > > - Hooks for the Polaris Management API and the catalog APIs. They can > also > > hand ready-made catalog properties to pyiceberg or to Spark and other > > engines. > > - Operators for catalogs, namespaces, tables, principals, principal and > > catalog roles, grants, and policies. Typical uses include provisioning a > > team's catalog with its roles and grants, onboarding a pipeline > principal, > > and managing table lifecycle. > > - A helper that turns Polaris table-maintenance policies (compaction, > > snapshot expiry, orphan-file removal) into Iceberg Spark procedure calls, > > so a Dag can execute the maintenance a policy defines. > > > > It is built on the official Apache Polaris Python SDK > > (apache-polaris>=1.9.0). It needs Airflow 3.1.1+ and Python 3.11+, and > > pyiceberg is an optional extra. It has been tested against Polaris server > > 1.8 and 1.9, and installs cleanly against Airflow's constraints for every > > 3.1.1–3.4.0 release. > > > > System tests > > ----------------- > > Polaris is open source and runs as a single container, so no external > paid > > service is involved. The PR adds: > > > > - a Breeze integration (breeze --integration polaris) that starts Polaris > > 1.9.0 in CI, with integration tests against it; > > - system-test example Dags for provisioning, tables and policies, which > run > > against the same server. > > > > All tests can run in Airflow's own CI, so no external dashboard is > needed. > > > > Proposed stewards > > ------------------------- > > - Yufei Gu, "flyrain", [email protected] (Apache Polaris PMC/Committer) > > - Yong Zheng, "MonkeyCanCode", [email protected] (Apache Polaris > > PMC/Committer) > > - Xiaodong Deng, "XD-DENG", [email protected] (Apache Airflow > > PMC/Committer) > > > > Sponsoring committer > > ----------------------------- > > - Xiaodong Deng, "XD-DENG", [email protected] > > > > Working implementation > > -------------------------------- > > Draft PR: https://github.com/apache/airflow/pull/77188. It includes > unit, > > integration and system tests, and how-to docs for every operator. > > > > Incubation commitment > > ------------------------------- > > We commit to: > > - maintaining the provider and responding to issues within a reasonable > > time; > > - meeting the incubation health metrics within 6 months; > > - participating in quarterly governance updates. > > > > > > Regards, > > > > XD > > >
