Hi Elad, Yes, sorry for the vagueness. I am unsure if it is a bug and there is no linked documentation with the PR suggesting a change needed in the configuration. I am waiting for a response from the author. At the moment, it is a regression for me as my previous remote logging setup using Elasticsearch is no longer working and the webserver reports errors when it tries to fetch logs from the Elasticsearch server whereas I see that logs are getting shipped correctly to Elasticsearch.
I would like to cast a *+0 (non-binding) *because a series of other RCs mentioned below work fine, it's just Elasticsearch RC waiting for a response. Below RCs work fine: - apache-airflow-providers-amazon - apache-airflow-providers-apache-hive - apache-airflow-providers-apache-livy - apache-airflow-providers-cncf-kubernetes - apache-airflow-providers-databricks - apache-airflow-providers-dbt-cloud - apache-airflow-providers-google - apache-airflow-providers-microsoft-azure - apache-airflow-providers-snowflake Regards, Pankaj Koti *Senior Software Engineer, *OSS Engineering Team. Location: Pune, India Timezone: Indian Standard Time (IST) Email: pankaj.k...@astronomer.io Mobile: +91 9730079985 On Fri, Jul 7, 2023 at 1:45 PM Elad Kalif <elad...@apache.org> wrote: > Hi Pankaj, > > Troubles is a very vague word :) > Does troubles mean regression/blocker bug or maybe this just requires > further doc clarification? > From the PR comments it's not clear to me if this is just a > mis-configuration concern or an actual bug. > > Please cast a non-binding with explanation of the impact of this RC release > (you can change your vote later if new information comes to light) > > > On Fri, Jul 7, 2023 at 10:33 AM Pankaj Koti > <pankaj.k...@astronomer.io.invalid> wrote: > > > I am facing trouble with using the ElasticSearch RC. > > The remote logging setup that used to work for me previously is no longer > > working with this change. > > I have reached out for help on the PR with comment > > https://github.com/apache/airflow/pull/31920#issuecomment-1623629476 > > > > Regards, > > > > > > > > Pankaj Koti > > > > *Senior Software Engineer, *OSS Engineering Team. > > Location: Pune, India > > > > Timezone: Indian Standard Time (IST) > > > > Email: pankaj.k...@astronomer.io > > > > Mobile: +91 9730079985 > > > > > > On Fri, Jul 7, 2023 at 3:56 AM Jarek Potiuk <ja...@potiuk.com> wrote: > > > > > +1 (binding): checked signatures, checksums, licences. verified that > the > > > sources from the provider"tag" are used for all new providers. tested > my > > > change in hive. All looks good > > > > > > On Thu, Jul 6, 2023 at 9:43 PM Hussain, Syed > <syeda...@amazon.com.invalid > > > > > > wrote: > > > > > > > Thanks Jarek for clarifying 😊 > > > > > > > > > > > > > Did you test with the last version of cncf.kubernetes provider? > > > > > > > > > > > > No I did not. By default, the system test script uses the most recent > > > > release, which is 7.1.0. I'll try running the tests again with the > > newer > > > > version. As I mentioned, I don't except there to be any problems. > It's > > > very > > > > clear that the issue came from the missing code 😃 > > > > > > > > ________________________________ > > > > From: Jarek Potiuk <ja...@potiuk.com> > > > > Sent: Thursday, July 6, 2023 12:31:59 PM > > > > To: dev@airflow.apache.org > > > > Subject: RE: [EXTERNAL][VOTE] Airflow Providers prepared on July 06, > > 2023 > > > > > > > > CAUTION: This email originated from outside of the organization. Do > not > > > > click links or open attachments unless you can confirm the sender and > > > know > > > > the content is safe. > > > > > > > > > > > > > > > > Just to clarify the state here - in case further discussions are > > needed. > > > > > > > > Amazon Provider has this optional extra dependency: > > > > > > > > - name: cncf.kubernetes > > > > dependencies: > > > > - apache-airflow-providers-cncf-kubernetes>=7.2.0 > > > > > > > > We cannot enforce it when you install just a single "amazon" > provider. > > > but > > > > the presence of the "[cncf.kubernetes]" extra with > > > > apache-airflow-providers-cncf-kubernetes>=7.2.0 is a clear indication > > > that > > > > the new 7.2.0 provider is needed to run "kubernetes" optional > features > > in > > > > the provider. Once both providers are released, either constraints > for > > a > > > > given version or even "pip install > > > apache-airflow[amazon,cncf.kubernetes]" > > > > will pull the latest versions, so this will work fine. > > > > > > > > The way `pip` and Python dependency management work - you cannot > > > provide a > > > > limit to a dependency that is optional (other than providing extra > like > > > we > > > > do). If "some" features of the `amazon` provider depend on > > > > `cncf.kubernetes`, then other than extra (as we did) there is no way > to > > > > inform `pip` or any other tool that "when you install those two > > packages > > > > the other should be at least this version". Such feature does not > exist > > > in > > > > `pip` or any other resolver. > > > > > > > > There are two ways to resolve it: > > > > > > > > 1) provide optional extra that informs about the limit but unless you > > do > > > > `pip install apache-airflow-providers-amazon[cncf.kubernetes]` there > is > > > no > > > > enforcement - for example there is no way to "upgrade" > cncf.kubernetes > > > > automatically when you upgrade `amazon` one. This is what we have. > > > > 2) make cncf.kubernetes a REQUIRED dependency. We could add just > > regular > > > > "apache-airflow-provider-cncf-kubernetes>=7.2.0" as a requirement of > > the > > > > amazon provider. We COULD do it if we decide that cncf.kubernetes is > > > always > > > > going to be installed with the amazon one - regardless if someone > uses > > > EKS > > > > or not. > > > > > > > > The case 2) has the unintended side effect in this case, that if > > someone > > > > uses amazon (but not EKS) and kubernetes, they will have no way to > > > > downgrade the cncf.kubernetes provider to a lower version - thus > > > > introducing strong coupling between these two providers. > > > > > > > > J. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Jul 6, 2023 at 9:11 PM Hussein Awala <huss...@awala.fr> > wrote: > > > > > > > > > Hello Hussain, > > > > > > > > > > Did you test with the last version of cncf.kubernetes provider? ( > > > > > > > > > > > > > > > https://pypi.org/project/apache-airflow-providers-cncf-kubernetes/7.2.0rc2 > > > > > ) > > > > > > > > > > On Thu, Jul 6, 2023 at 8:59 PM Hussain, Syed > > > <syeda...@amazon.com.invalid > > > > > > > > > > wrote: > > > > > > > > > > > Hello, > > > > > > > > > > > > I ran the system tests for the Amazon Provider Package for > this > > > > > > release, using Airflow version 2.6.2. Here is the link to the > > > dashboard > > > > > for > > > > > > the results: > > > > > > > > > > > > > > > > > > > > > https://aws-mwaa.github.io/open-source/system-tests/version/2.6.2_8.3.0rc2.html > > > > > > > > > > > > > > > > > > All system tests related to EKS failed, due to being unable to > > import > > > > > > "OnFinishAction" ( > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/airflow/blob/main/airflow/providers/cncf/kubernetes/utils/pod_manager.py#L718 > > > > > ). > > > > > > This was added 6 days ago, and is not included in the current > > > release. > > > > > > > > > > > > ________________________________ > > > > > > From: Elad Kalif <elad...@apache.org> > > > > > > Sent: Wednesday, July 5, 2023 10:46:58 PM > > > > > > To: dev@airflow.apache.org > > > > > > Subject: [EXTERNAL] [VOTE] Airflow Providers prepared on July 06, > > > 2023 > > > > > > > > > > > > CAUTION: This email originated from outside of the organization. > Do > > > not > > > > > > click links or open attachments unless you can confirm the sender > > and > > > > > know > > > > > > the content is safe. > > > > > > > > > > > > > > > > > > > > > > > > Hey all,I have just cut RC2 wave Airflow Providers packages. This > > > > > > email is calling a vote on the release,which will last for 72 > > hours - > > > > > > which means that it will end on July 09, 2023 05:50 AM > UTCConsider > > > > > > this my (binding) +1. > > > > > > Airflow Providers are available > > > > > > at: > > > > > > > > > > > > > > > > > > > > > https://dist.apache.org/repos/dist/dev/airflow/providers/*apache-airflow-providers- > > > > > > <PROVIDER>-*.tar.gz* > > > > > > are the binary Python "sdist" release - they are also official > > > > > > "sources" for the provider > > > > > > packages.*apache_airflow_providers_<PROVIDER>-*.whl are the > binary > > > > > > Python "wheel" release.The test procedure for PMC members who > would > > > > > > like to test the RC candidates are described > > > > > > inhttps:// > > > > > > > > > > > > > > > > > > > > > github.com/apache/airflow/blob/main/dev/README_RELEASE_PROVIDER_PACKAGES.md#verify-the-release-by-pmc-membersand > > > > > > for Contributors: > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/airflow/blob/main/dev/README_RELEASE_PROVIDER_PACKAGES.md#verify-by-contributorsPublic > > > > > > keys are available > > > > > > at:https://dist.apache.org/repos/dist/release/airflow/KEYSPlease > > > vote > > > > > > accordingly:[ ] +1 approve[ ] +0 no opinion[ ] -1 disapprove with > > the > > > > > > reasonOnly votes from PMC members are binding, but members of the > > > > > > community are encouraged to test the release and vote with > > > > > > "(non-binding)".Please note that the version number excludes the > > > 'rcX' > > > > > > string.This will allow us to rename the artifact without > > modifyingthe > > > > > > artifact checksums when we actually release.The status of testing > > the > > > > > > providers by the community is kept > > > > > > here:https://github.com/apache/airflow/issues/32389 > > > > > > You can find packages as well as detailed changelog following the > > > below > > > > > > links: > > > > > > > > > > > > > > https://pypi.org/project/apache-airflow-providers-alibaba/2.5.0rc2/ > > > > > > > https://pypi.org/project/apache-airflow-providers-amazon/8.3.0rc2/ > > > > > > > > > > > > https://pypi.org/project/apache-airflow-providers-apache-hive/6.1.2rc2/ > > > > > > > > > > > > > > > > > > > > > https://pypi.org/project/apache-airflow-providers-apache-impala/1.1.2rc2/ > > > > > > > > > > > > https://pypi.org/project/apache-airflow-providers-apache-kafka/1.1.2rc2/ > > > > > > > > > > > > https://pypi.org/project/apache-airflow-providers-apache-livy/3.5.2rc2/ > > > > > > > > https://pypi.org/project/apache-airflow-providers-apprise/1.0.0rc2/ > > > > > > > > > > > > > > > > > > > > > https://pypi.org/project/apache-airflow-providers-cncf-kubernetes/7.2.0rc2/ > > > > > > > > > https://pypi.org/project/apache-airflow-providers-common-sql/1.6.0rc2/ > > > > > > > > > https://pypi.org/project/apache-airflow-providers-databricks/4.3.1rc2/ > > > > > > > > > https://pypi.org/project/apache-airflow-providers-dbt-cloud/3.2.2rc2/ > > > > > > > > > > > > > > > > > > > > > https://pypi.org/project/apache-airflow-providers-elasticsearch/5.0.0rc2/ > > > > > > > > https://pypi.org/project/apache-airflow-providers-google/10.3.0rc2/ > > > > > > > > > https://pypi.org/project/apache-airflow-providers-hashicorp/3.4.2rc2/ > > > > > > > > > > > > > > > > > > > > > https://pypi.org/project/apache-airflow-providers-microsoft-azure/6.2.0rc2/ > > > > > > > > https://pypi.org/project/apache-airflow-providers-postgres/5.5.2rc2/ > > > > > > > > > https://pypi.org/project/apache-airflow-providers-snowflake/4.3.0rc2/ > > > > > > > https://pypi.org/project/apache-airflow-providers-trino/5.2.0rc2/ > > > > > > > > https://pypi.org/project/apache-airflow-providers-vertica/3.5.0rc2/ > > > > > > > > > > > > Cheers,Elad Kalif > > > > > > > > > > > > > > > > > > > > >
