Beam High Priority Issue Report (69)

This is your daily summary of Beam's current high priority issues that may need attention. See https://beam.apache.org/contribute/issue-priorities for the meaning and expectations around issue priorities. Unassigned P1 Issues: https://github.com/apache/beam/issues/22779 [Bug]: SpannerIO.re

Keynote Speaker for a Stream processing workshop at IEEE Big data 2022 Osaka, JP

Dear all, I am not sure, I am on the right mailing list for this request, if not, don't hesitate to inform me what would be the best course of action. I am the co-chair of the 7th workshop on real-time stream analytics workshop, colocated with the IEE

Re: [DISCUSS] Dependency management in Apache Beam Python SDK

> > I'm curious Jarek, does Airflow take any dependencies on popular libraries > like pandas, numpy, pyarrow, scipy, etc... which users are likely to have > their own dependency on? I think these dependencies are challenging in a > different way than the client libraries - ideally we would support

Re: [DISCUSS] Dependency management in Apache Beam Python SDK

Jarek, I really appreciate you sharing your experience and expertise here. I think Beam would benefit from adopting some of these practices. Kerry On Fri, Aug 26, 2022, 7:35 AM Jarek Potiuk wrote: > >> I'm curious Jarek, does Airflow take any dependencies on popular >> libraries like pandas, num

Re: [DISCUSS] Dependency management in Apache Beam Python SDK

Happy to help and I hope we can work together with Valentyn and others to get the "google clients" approach improved :) J. On Fri, Aug 26, 2022 at 3:40 PM Kerry Donny-Clark via dev < dev@beam.apache.org> wrote: > Jarek, I really appreciate you sharing your experience and expertise here. > I thi

Re: [ANNOUNCE] Apache Beam 2.41.0 Released

Hi Team, Looking forward to trying and testing the new version, It's always fascinating. On Fri, 26 Aug 2022 at 03:45, Pablo Estrada via user wrote: > Thank you Kiley! > > On Thu, Aug 25, 2022 at 10:55 AM Kiley Sok wrote: > >> The Apache Beam team is pleased to announce the release of version

Re: [ANNOUNCE] Apache Beam 2.41.0 Released

Thank you Kiley! On Fri, Aug 26, 2022 at 6:38 AM P Singh wrote: > Hi Team, > > Looking forward to trying and testing the new version, It's always > fascinating. > > On Fri, 26 Aug 2022 at 03:45, Pablo Estrada via user > wrote: > >> Thank you Kiley! >> >> On Thu, Aug 25, 2022 at 10:55 AM Kiley S

Re: [ANNOUNCE] Apache Beam 2.41.0 Released

Thanks Kiley! On Fri, Aug 26, 2022 at 9:37 AM Ahmet Altay via dev wrote: > Thank you Kiley! > > On Fri, Aug 26, 2022 at 6:38 AM P Singh > wrote: > >> Hi Team, >> >> Looking forward to trying and testing the new version, It's always >> fascinating. >> >> On Fri, 26 Aug 2022 at 03:45, Pablo Estra

RE: Java object serialization error, java.io.InvalidClassException: org.apache.spark.deploy.ApplicationDescription; local class incompatible

So, it turns out this was a Spark version mismatch between the Beam JobServer and the Spark platform. I'm running both Beam and Spark on Docker; the Spark image [0] provided version 3.3.0 and Scala version 2.12, but I used apache/beam_spark_job_server:2.41.0 [1] which provides Spark 2.4.x librarie

Re: Java object serialization error, java.io.InvalidClassException: org.apache.spark.deploy.ApplicationDescription; local class incompatible

Woohoo! Glad that was figured out. On Fri, Aug 26, 2022, 4:40 PM Elliot Metsger wrote: > > So, it turns out this was a Spark version mismatch between the Beam > JobServer and the Spark platform. > > I'm running both Beam and Spark on Docker; the Spark image [0] provided > version 3.3.0 and Scala

Re: Java object serialization error, java.io.InvalidClassException: org.apache.spark.deploy.ApplicationDescription; local class incompatible

Yep! Thanks Robert for engaging on Slack! Just had to dig in a bit - I ended up building the 3.3.0 jobserver from scratch (the docker build env is very slick, despite encountering a couple of hiccups), before I realized the image was available on docker hub :facepalm: ... On Fri, Aug 26, 2022 at