+1 On Tue, Nov 17, 2026 at 7:41 PM Yufei Gu <[email protected]> wrote:
> +1, this is very cool. Thanks for adding this, XD! > > One use case that comes to mind is Iceberg table maintenance. If you run > Iceberg tables in production, you need to do things like compact data files > and expire old snapshots. Native support in Airflow and Polaris would make > it easier to schedule those jobs as part of your existing Airflow > workflows. > > A bit about me for those who don’t know me. I’m a PMC member of both Apache > Polaris and Apache Iceberg. I also wrote the policy store and table > maintenance features in Polaris. > Yufei > > > On Tue, Nov 17, 2026 at 2:16 PM Jarek Potiuk <[email protected]> wrote: > > > +1 > > > > On Wed, Nov 18, 2026 at 9:01 AM constance.astronomer.io via dev < > > [email protected]> wrote: > > > >> +1 love it > >> > >> > On Nov 17, 2026, at 12:03 PM, Vikram Koka via dev < > >> [email protected]> wrote: > >> > > >> > +1 > >> > Seems like a great addition > >> > > >> > Vikram > >> > > >> > > >> >> On Tue, Nov 17, 2026 at 4:18 AM Kaxil Naik <[email protected]> > >> wrote: > >> >> > >> >> +1 Would love to have polaris support. > >> >> > >> >> > >> >> On Tue, 17 Nov 2026 at 04:18, Xiaodong Deng via dev < > >> >> [email protected]> > >> >> wrote: > >> >> > >> >>> Hi Airflow community, > >> >>> > >> >>> We'd like to propose adding a new community provider for Apache > >> Polaris: > >> >>> apache-airflow-providers-apache-polaris. > >> >>> > >> >>> Integration overview > >> >>> --------------------------- > >> >>> Apache Polaris (https://polaris.apache.org/) is an open-source > >> catalog > >> >> for > >> >>> Apache Iceberg. It implements the Iceberg REST catalog API and adds > >> >>> centralized access control, credential vending and catalog-level > >> >> policies. > >> >>> Many teams running Iceberg already orchestrate their data platform > >> with > >> >>> Airflow. Today, managing Polaris from a Dag means hand-written REST > >> calls > >> >>> or shelling out to the Polaris CLI. > >> >>> > >> >>> The proposed provider would add: > >> >>> > >> >>> - A "polaris" connection type. It supports Polaris's internal OAuth > >> >> tokens, > >> >>> and it masks every token it fetches. > >> >>> - Hooks for the Polaris Management API and the catalog APIs. They > can > >> >> also > >> >>> hand ready-made catalog properties to pyiceberg or to Spark and > other > >> >>> engines. > >> >>> - Operators for catalogs, namespaces, tables, principals, principal > >> and > >> >>> catalog roles, grants, and policies. Typical uses include > >> provisioning a > >> >>> team's catalog with its roles and grants, onboarding a pipeline > >> >> principal, > >> >>> and managing table lifecycle. > >> >>> - A helper that turns Polaris table-maintenance policies > (compaction, > >> >>> snapshot expiry, orphan-file removal) into Iceberg Spark procedure > >> calls, > >> >>> so a Dag can execute the maintenance a policy defines. > >> >>> > >> >>> It is built on the official Apache Polaris Python SDK > >> >>> (apache-polaris>=1.9.0). It needs Airflow 3.1.1+ and Python 3.11+, > and > >> >>> pyiceberg is an optional extra. It has been tested against Polaris > >> server > >> >>> 1.8 and 1.9, and installs cleanly against Airflow's constraints for > >> every > >> >>> 3.1.1–3.4.0 release. > >> >>> > >> >>> System tests > >> >>> ----------------- > >> >>> Polaris is open source and runs as a single container, so no > external > >> >> paid > >> >>> service is involved. The PR adds: > >> >>> > >> >>> - a Breeze integration (breeze --integration polaris) that starts > >> Polaris > >> >>> 1.9.0 in CI, with integration tests against it; > >> >>> - system-test example Dags for provisioning, tables and policies, > >> which > >> >> run > >> >>> against the same server. > >> >>> > >> >>> All tests can run in Airflow's own CI, so no external dashboard is > >> >> needed. > >> >>> > >> >>> Proposed stewards > >> >>> ------------------------- > >> >>> - Yufei Gu, "flyrain", [email protected] (Apache Polaris > >> PMC/Committer) > >> >>> - Yong Zheng, "MonkeyCanCode", [email protected] (Apache Polaris > >> >>> PMC/Committer) > >> >>> - Xiaodong Deng, "XD-DENG", [email protected] (Apache Airflow > >> >>> PMC/Committer) > >> >>> > >> >>> Sponsoring committer > >> >>> ----------------------------- > >> >>> - Xiaodong Deng, "XD-DENG", [email protected] > >> >>> > >> >>> Working implementation > >> >>> -------------------------------- > >> >>> Draft PR: https://github.com/apache/airflow/pull/77188. It includes > >> >> unit, > >> >>> integration and system tests, and how-to docs for every operator. > >> >>> > >> >>> Incubation commitment > >> >>> ------------------------------- > >> >>> We commit to: > >> >>> - maintaining the provider and responding to issues within a > >> reasonable > >> >>> time; > >> >>> - meeting the incubation health metrics within 6 months; > >> >>> - participating in quarterly governance updates. > >> >>> > >> >>> > >> >>> Regards, > >> >>> > >> >>> XD > >> >>> > >> >> > >> > >> --------------------------------------------------------------------- > >> To unsubscribe, e-mail: [email protected] > >> For additional commands, e-mail: [email protected] > >> > >> >
