+1 On Wed, Nov 18, 2026 at 9:01 AM constance.astronomer.io via dev < [email protected]> wrote:
> +1 love it > > > On Nov 17, 2026, at 12:03 PM, Vikram Koka via dev < > [email protected]> wrote: > > > > +1 > > Seems like a great addition > > > > Vikram > > > > > >> On Tue, Nov 17, 2026 at 4:18 AM Kaxil Naik <[email protected]> wrote: > >> > >> +1 Would love to have polaris support. > >> > >> > >> On Tue, 17 Nov 2026 at 04:18, Xiaodong Deng via dev < > >> [email protected]> > >> wrote: > >> > >>> Hi Airflow community, > >>> > >>> We'd like to propose adding a new community provider for Apache > Polaris: > >>> apache-airflow-providers-apache-polaris. > >>> > >>> Integration overview > >>> --------------------------- > >>> Apache Polaris (https://polaris.apache.org/) is an open-source catalog > >> for > >>> Apache Iceberg. It implements the Iceberg REST catalog API and adds > >>> centralized access control, credential vending and catalog-level > >> policies. > >>> Many teams running Iceberg already orchestrate their data platform with > >>> Airflow. Today, managing Polaris from a Dag means hand-written REST > calls > >>> or shelling out to the Polaris CLI. > >>> > >>> The proposed provider would add: > >>> > >>> - A "polaris" connection type. It supports Polaris's internal OAuth > >> tokens, > >>> and it masks every token it fetches. > >>> - Hooks for the Polaris Management API and the catalog APIs. They can > >> also > >>> hand ready-made catalog properties to pyiceberg or to Spark and other > >>> engines. > >>> - Operators for catalogs, namespaces, tables, principals, principal and > >>> catalog roles, grants, and policies. Typical uses include provisioning > a > >>> team's catalog with its roles and grants, onboarding a pipeline > >> principal, > >>> and managing table lifecycle. > >>> - A helper that turns Polaris table-maintenance policies (compaction, > >>> snapshot expiry, orphan-file removal) into Iceberg Spark procedure > calls, > >>> so a Dag can execute the maintenance a policy defines. > >>> > >>> It is built on the official Apache Polaris Python SDK > >>> (apache-polaris>=1.9.0). It needs Airflow 3.1.1+ and Python 3.11+, and > >>> pyiceberg is an optional extra. It has been tested against Polaris > server > >>> 1.8 and 1.9, and installs cleanly against Airflow's constraints for > every > >>> 3.1.1–3.4.0 release. > >>> > >>> System tests > >>> ----------------- > >>> Polaris is open source and runs as a single container, so no external > >> paid > >>> service is involved. The PR adds: > >>> > >>> - a Breeze integration (breeze --integration polaris) that starts > Polaris > >>> 1.9.0 in CI, with integration tests against it; > >>> - system-test example Dags for provisioning, tables and policies, which > >> run > >>> against the same server. > >>> > >>> All tests can run in Airflow's own CI, so no external dashboard is > >> needed. > >>> > >>> Proposed stewards > >>> ------------------------- > >>> - Yufei Gu, "flyrain", [email protected] (Apache Polaris PMC/Committer) > >>> - Yong Zheng, "MonkeyCanCode", [email protected] (Apache Polaris > >>> PMC/Committer) > >>> - Xiaodong Deng, "XD-DENG", [email protected] (Apache Airflow > >>> PMC/Committer) > >>> > >>> Sponsoring committer > >>> ----------------------------- > >>> - Xiaodong Deng, "XD-DENG", [email protected] > >>> > >>> Working implementation > >>> -------------------------------- > >>> Draft PR: https://github.com/apache/airflow/pull/77188. It includes > >> unit, > >>> integration and system tests, and how-to docs for every operator. > >>> > >>> Incubation commitment > >>> ------------------------------- > >>> We commit to: > >>> - maintaining the provider and responding to issues within a reasonable > >>> time; > >>> - meeting the incubation health metrics within 6 months; > >>> - participating in quarterly governance updates. > >>> > >>> > >>> Regards, > >>> > >>> XD > >>> > >> > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > >
