> 1. Visibility is one of the motivations, but not the only one. Our main goal is to ensure a better experience for users who are already using Couchbase with Airflow, and to support future users with a provider that is more easily discoverable, better integrated with Airflow’s ecosystem, and aligned with its standards and lifecycle.
Having providers outside of our codebase is perfectly fine with our standards and alignments. That's a valid way to release your provider. Many other parties are doing it. 1. We fully understand that becoming an official provider comes with > reduced flexibility, such as adhering to Airflow’s version requirements—but > we believe this is a reasonable trade-off, as it simplifies compatibility > and aligns us more closely with the Airflow ecosystem. The provider > currently only exposes cluster, scope, and collection methods, along with > basic CRUD operations. Changes to these interfaces are very rare, so we > expect the provider to remain stable with minimal maintenance overhead. > Then you should not need to upgrade it as well. Just released provider from your repo should work in this case and the only time change is needed, you could release it then. I do not see what being part of regular releases would bring other than more effort on the community side and less effort on your side. > > 1. Being part of the official Airflow repo would ensure the Couchbase > provider is automatically tested with new Airflow releases, helping us > catch issues early and stay compatible. > Not really. We are not testing it with the real "couchbase" system when we release it. We only run unit tests in Airflow and you can do the same every time we release a new airflow version . The tests with couchbase service (which to be honest is far more likely to change than internals of airflow providers) is something only you can do having access to couchbase account and service. We have none and we do not really want to bother with testing the provider when anything changes on the couchbase side - such as when new library is released for couchbase, when Couchbase changes the APIs or modifies anything, this is is only done for some providers by system tests https://airflow.apache.org/ecosystem/#airflow-provider-system-test-dashboards run by some of our stakeholders who agreed to build and run such dashboards and tests. And we are not running those tests. This will not change if the code is moved to the Airflow codebase. I think it would really be great if you answer the specific questions from Vikram - about metrics and examples of your visibility. > > Regards, > Shyam Venkat > > From: Vikram Koka <vik...@astronomer.io.INVALID> > Date: Friday, 11 April 2025 at 11:49 PM > To: dev@airflow.apache.org <dev@airflow.apache.org> > Subject: Re: [DISCUSSION] Proposal to Add Couchbase Provider to Apache > Airflow > Shyam, > > First off, I'm really glad to hear that Couchbase is interested in a deeper > integration with Airflow and that Couchbase users are already finding value > in it. > > Just to follow up with a couple of specific questions around your request: > Could you share more about what's missing today with the integration being > maintained in the Couchbase repo, versus having it packaged within the > Airflow repo? > > You mentioned benefits like "better visibility, easier adoption, and > tighter integration with Airflow's architecture and release processes." > If you could expand on those - especially in terms of what you see changing > before vs. after, and ideally any metrics or examples - that context would > be very helpful. > > Best regards, > Vikram > > > On Fri, Apr 11, 2025 at 3:02 AM Jarek Potiuk <ja...@potiuk.com> wrote: > > > I am not convinced. > > > > First of all what do you mean by visibility ? I see no problem with the > > visibility of your current provider - couchbase seems to have well > > established ecosystem with multiple plugins and extensions - > > https://github.com/Couchbase-Ecosystem and I would say it's quite likely > > Couchbase users will be looking for Airflow integration among those. Also > > you will have (as you have now) freedom to release whatever changes and > > improvements needed as Couchbase evolves and if you already commit to > > maintain and keep the couchbase integration by Couchbase team (you wrote > it > > above) - you can equally well maintain it in and release it in your repo. > > > > You already seem to be a good steward of that integration and by keeping > > your ecosystem healthy you do a good job, I do not see a particular > reason > > why you'd want to contribute it to Airflow. We have no knowledge of > > Couchbase itself here, you are the experts and you can decide what is the > > best integration level you provide, the set of operators and hooks are > best > > for you - and I think it's better if you listen to the Couchbase > community > > feedback and get your community to contribute there - Airflow provides a > > nice set of interfaces for you to adhere, and that's basically all that > > (from Airflow Community point of view) you need to know and follow - and > > you already do it apparently. > > > > Couchbase is "source-available" database > > https://www.couchbase.com/blog/couchbase-adopts-bsl-license/ - which is > > not > > an OSI compliant open-source produict, which makes it even more suitable > > for your company to keep maintaining the integration. Of course > eventually > > it's a community decision whether to accept such a provider or not, but I > > would be rather against it - just for that reason. > > > > J. > > > > > > > > On Fri, Apr 11, 2025 at 11:17 AM Shyam Rajamannar > > <shyam.ven...@couchbase.com.invalid> wrote: > > > > > Couchbase has customers already using Apache Airflow, and having an > > > officially supported provider within the ecosystem would simplify > > > integration, improve reliability, and encourage adoption. > > > > > > We previously released airflow-providers-couchbase< > > > https://github.com/Couchbase-Ecosystem/airflow-providers-couchbase>< > https://github.com/Couchbase-Ecosystem/airflow-providers-couchbase%3e> > > > through an external repository also listed under airflow ecosystem. > > > However, it has been less visible to our customers and the broader > > Airflow > > > community. Contributing this provider directly to the official Airflow > > > ecosystem will ensure better visibility, easier adoption, and tighter > > > integration with Airflow’s architecture and release processes. > > > > > > As part of the initial implementation, we have created a Couchbase hook > > > that exposes the Couchbase client, scope, and collection, allowing > users > > to > > > easily interact with Couchbase within Airflow tasks. The provider also > > > includes functionality to customize connection field behaviour, > adhering > > to > > > Airflow’s connection management standards. > > > > > > Our initial scope is deliberately minimal—focused on foundational > > > operations. While we may introduce CRUD functionality later, we expect > > the > > > provider to remain stable with minimal and infrequent changes. > > > > > > We are committed to maintaining the provider, ensuring compatibility > with > > > future Couchbase API changes, and responding to community > > feedback—similar > > > to the stewardship shown by other officially supported database > > providers. > > > > > > If maintenance is a concern, we are fully willing to act as active > > > maintainers, ensuring the provider remains well-supported, reliable, > and > > > aligned with Airflow’s evolution. > > > > > > Regards, > > > Shyam Venkat > > > > > > > > > > > >
