1. Visibility is one of the motivations, but not the only one. Our main goal is to ensure a better experience for users who are already using Couchbase with Airflow, and to support future users with a provider that is more easily discoverable, better integrated with Airflow’s ecosystem, and aligned with its standards and lifecycle.
1. We fully understand that becoming an official provider comes with reduced flexibility, such as adhering to Airflow’s version requirements—but we believe this is a reasonable trade-off, as it simplifies compatibility and aligns us more closely with the Airflow ecosystem. The provider currently only exposes cluster, scope, and collection methods, along with basic CRUD operations. Changes to these interfaces are very rare, so we expect the provider to remain stable with minimal maintenance overhead. 1. Being part of the official Airflow repo would ensure the Couchbase provider is automatically tested with new Airflow releases, helping us catch issues early and stay compatible. Regards, Shyam Venkat From: Vikram Koka <vik...@astronomer.io.INVALID> Date: Friday, 11 April 2025 at 11:49 PM To: dev@airflow.apache.org <dev@airflow.apache.org> Subject: Re: [DISCUSSION] Proposal to Add Couchbase Provider to Apache Airflow Shyam, First off, I'm really glad to hear that Couchbase is interested in a deeper integration with Airflow and that Couchbase users are already finding value in it. Just to follow up with a couple of specific questions around your request: Could you share more about what's missing today with the integration being maintained in the Couchbase repo, versus having it packaged within the Airflow repo? You mentioned benefits like "better visibility, easier adoption, and tighter integration with Airflow's architecture and release processes." If you could expand on those - especially in terms of what you see changing before vs. after, and ideally any metrics or examples - that context would be very helpful. Best regards, Vikram On Fri, Apr 11, 2025 at 3:02 AM Jarek Potiuk <ja...@potiuk.com> wrote: > I am not convinced. > > First of all what do you mean by visibility ? I see no problem with the > visibility of your current provider - couchbase seems to have well > established ecosystem with multiple plugins and extensions - > https://github.com/Couchbase-Ecosystem and I would say it's quite likely > Couchbase users will be looking for Airflow integration among those. Also > you will have (as you have now) freedom to release whatever changes and > improvements needed as Couchbase evolves and if you already commit to > maintain and keep the couchbase integration by Couchbase team (you wrote it > above) - you can equally well maintain it in and release it in your repo. > > You already seem to be a good steward of that integration and by keeping > your ecosystem healthy you do a good job, I do not see a particular reason > why you'd want to contribute it to Airflow. We have no knowledge of > Couchbase itself here, you are the experts and you can decide what is the > best integration level you provide, the set of operators and hooks are best > for you - and I think it's better if you listen to the Couchbase community > feedback and get your community to contribute there - Airflow provides a > nice set of interfaces for you to adhere, and that's basically all that > (from Airflow Community point of view) you need to know and follow - and > you already do it apparently. > > Couchbase is "source-available" database > https://www.couchbase.com/blog/couchbase-adopts-bsl-license/ - which is > not > an OSI compliant open-source produict, which makes it even more suitable > for your company to keep maintaining the integration. Of course eventually > it's a community decision whether to accept such a provider or not, but I > would be rather against it - just for that reason. > > J. > > > > On Fri, Apr 11, 2025 at 11:17 AM Shyam Rajamannar > <shyam.ven...@couchbase.com.invalid> wrote: > > > Couchbase has customers already using Apache Airflow, and having an > > officially supported provider within the ecosystem would simplify > > integration, improve reliability, and encourage adoption. > > > > We previously released airflow-providers-couchbase< > > https://github.com/Couchbase-Ecosystem/airflow-providers-couchbase><https://github.com/Couchbase-Ecosystem/airflow-providers-couchbase%3e> > > through an external repository also listed under airflow ecosystem. > > However, it has been less visible to our customers and the broader > Airflow > > community. Contributing this provider directly to the official Airflow > > ecosystem will ensure better visibility, easier adoption, and tighter > > integration with Airflow’s architecture and release processes. > > > > As part of the initial implementation, we have created a Couchbase hook > > that exposes the Couchbase client, scope, and collection, allowing users > to > > easily interact with Couchbase within Airflow tasks. The provider also > > includes functionality to customize connection field behaviour, adhering > to > > Airflow’s connection management standards. > > > > Our initial scope is deliberately minimal—focused on foundational > > operations. While we may introduce CRUD functionality later, we expect > the > > provider to remain stable with minimal and infrequent changes. > > > > We are committed to maintaining the provider, ensuring compatibility with > > future Couchbase API changes, and responding to community > feedback—similar > > to the stewardship shown by other officially supported database > providers. > > > > If maintenance is a concern, we are fully willing to act as active > > maintainers, ensuring the provider remains well-supported, reliable, and > > aligned with Airflow’s evolution. > > > > Regards, > > Shyam Venkat > > > > > > >
