+1
Seems like a great addition

Vikram


On Tue, Nov 17, 2026 at 4:18 AM Kaxil Naik <[email protected]> wrote:

> +1 Would love to have polaris support.
>
>
> On Tue, 17 Nov 2026 at 04:18, Xiaodong Deng via dev <
> [email protected]>
> wrote:
>
> > Hi Airflow community,
> >
> > We'd like to propose adding a new community provider for Apache Polaris:
> > apache-airflow-providers-apache-polaris.
> >
> > Integration overview
> > ---------------------------
> > Apache Polaris (https://polaris.apache.org/) is an open-source catalog
> for
> > Apache Iceberg. It implements the Iceberg REST catalog API and adds
> > centralized access control, credential vending and catalog-level
> policies.
> > Many teams running Iceberg already orchestrate their data platform with
> > Airflow. Today, managing Polaris from a Dag means hand-written REST calls
> > or shelling out to the Polaris CLI.
> >
> > The proposed provider would add:
> >
> > - A "polaris" connection type. It supports Polaris's internal OAuth
> tokens,
> > and it masks every token it fetches.
> > - Hooks for the Polaris Management API and the catalog APIs. They can
> also
> > hand ready-made catalog properties to pyiceberg or to Spark and other
> > engines.
> > - Operators for catalogs, namespaces, tables, principals, principal and
> > catalog roles, grants, and policies. Typical uses include provisioning a
> > team's catalog with its roles and grants, onboarding a pipeline
> principal,
> > and managing table lifecycle.
> > - A helper that turns Polaris table-maintenance policies (compaction,
> > snapshot expiry, orphan-file removal) into Iceberg Spark procedure calls,
> > so a Dag can execute the maintenance a policy defines.
> >
> > It is built on the official Apache Polaris Python SDK
> > (apache-polaris>=1.9.0). It needs Airflow 3.1.1+ and Python 3.11+, and
> > pyiceberg is an optional extra. It has been tested against Polaris server
> > 1.8 and 1.9, and installs cleanly against Airflow's constraints for every
> > 3.1.1–3.4.0 release.
> >
> > System tests
> > -----------------
> > Polaris is open source and runs as a single container, so no external
> paid
> > service is involved. The PR adds:
> >
> > - a Breeze integration (breeze --integration polaris) that starts Polaris
> > 1.9.0 in CI, with integration tests against it;
> > - system-test example Dags for provisioning, tables and policies, which
> run
> > against the same server.
> >
> > All tests can run in Airflow's own CI, so no external dashboard is
> needed.
> >
> > Proposed stewards
> > -------------------------
> > - Yufei Gu, "flyrain", [email protected] (Apache Polaris PMC/Committer)
> > - Yong Zheng, "MonkeyCanCode", [email protected] (Apache Polaris
> > PMC/Committer)
> > - Xiaodong Deng, "XD-DENG", [email protected] (Apache Airflow
> > PMC/Committer)
> >
> > Sponsoring committer
> > -----------------------------
> > - Xiaodong Deng, "XD-DENG", [email protected]
> >
> > Working implementation
> > --------------------------------
> > Draft PR: https://github.com/apache/airflow/pull/77188. It includes
> unit,
> > integration and system tests, and how-to docs for every operator.
> >
> > Incubation commitment
> > -------------------------------
> > We commit to:
> > - maintaining the provider and responding to issues within a reasonable
> > time;
> > - meeting the incubation health metrics within 6 months;
> > - participating in quarterly governance updates.
> >
> >
> > Regards,
> >
> > XD
> >
>

Reply via email to