Re: [DISCUSS] Dependency management in Apache Beam Python SDK

2023-05-02 Thread Valentyn Tymofieiev via dev
Hi All, just wanted to give a quick update on the effort discussed here: The action items from the retrospective are tracked in https://github.com/apache/beam/issues/25652. Many outdated dependencies were updated in https://github.com/apache/beam/pull/24599 by +Anand Inguva and remaining older

Re: [DISCUSS] Dependency management in Apache Beam Python SDK

2022-08-26 Thread Jarek Potiuk
Happy to help and I hope we can work together with Valentyn and others to get the "google clients" approach improved :) J. On Fri, Aug 26, 2022 at 3:40 PM Kerry Donny-Clark via dev < dev@beam.apache.org> wrote: > Jarek, I really appreciate you sharing your experience and expertise here. > I thi

Re: [DISCUSS] Dependency management in Apache Beam Python SDK

2022-08-26 Thread Kerry Donny-Clark via dev
Jarek, I really appreciate you sharing your experience and expertise here. I think Beam would benefit from adopting some of these practices. Kerry On Fri, Aug 26, 2022, 7:35 AM Jarek Potiuk wrote: > >> I'm curious Jarek, does Airflow take any dependencies on popular >> libraries like pandas, num

Re: [DISCUSS] Dependency management in Apache Beam Python SDK

2022-08-26 Thread Jarek Potiuk
> > I'm curious Jarek, does Airflow take any dependencies on popular libraries > like pandas, numpy, pyarrow, scipy, etc... which users are likely to have > their own dependency on? I think these dependencies are challenging in a > different way than the client libraries - ideally we would support

Re: [DISCUSS] Dependency management in Apache Beam Python SDK

2022-08-25 Thread Brian Hulette via dev
Thanks for writing this up Valentyn! I'm curious Jarek, does Airflow take any dependencies on popular libraries like pandas, numpy, pyarrow, scipy, etc... which users are likely to have their own dependency on? I think these dependencies are challenging in a different way than the client libraries

Re: [DISCUSS] Dependency management in Apache Beam Python SDK

2022-08-25 Thread Valentyn Tymofieiev via dev
Hi Jarek, Thanks a lot for detailed feedback and sharing the Airflow story, this is exactly what I was hoping to hear in response from the mailing list! 600+ dependencies is very impressive, so I'd be happy to chat more and learn from your experience. On Wed, Aug 24, 2022 at 5:50 AM Jarek Potiuk

Re: [DISCUSS] Dependency management in Apache Beam Python SDK

2022-08-24 Thread Jarek Potiuk
Comment (from a bit outsider) Fantastic document Valentyn. Very, very insightful and interesting. We feel a lot of the same pain in Apache Airflow (actually even more because we have not 20 but 620+ dependencies) but we are also a bit more advanced in the way how we are managing the dependencies