Hi Dmytro, Unfortunately, SparkStructuredStreamingRunner still doesn’t support streaming job (please, take a look on “Note" section here [1]) We have a Jira issue for tracking this [2] - in two words, it’s blocked because of issues on Spark with multiple aggregations for Structured Streaming component [3]
So far, it’s recommended to use a “Classical” RDD or Portable Spark runners to run Beam pipelines on Spark. Regards, Alexey [1] https://beam.apache.org/documentation/runners/spark/#three-flavors-of-the-spark-runner [2] https://issues.apache.org/jira/browse/BEAM-9933 [3] https://issues.apache.org/jira/browse/SPARK-26655 > On 20 Jan 2021, at 10:21, Dmytro Dragan <dd...@softserveinc.com> wrote: > > Hi guys, > > I have tried SparkStructuredStreamingRunner for streaming job with Spark 2.4 > and it does not work. > I faced with the same error which is described on stackoverflow: > https://stackoverflow.com/questions/64361372/apache-bean-spark-runner-does-not-work-on-streaming-mode-java-lang-illegalacce > > <https://stackoverflow.com/questions/64361372/apache-bean-spark-runner-does-not-work-on-streaming-mode-java-lang-illegalacce> > > Exception in thread "main" java.lang.IllegalAccessException: Class > org.apache.spark.sql.streaming.DataStreamReader can not access a member of > class > org.apache.beam.runners.spark.structuredstreaming.translation.streaming.DatasetSourceStreaming > with modifiers "" > > It gives the same result for 2.25, 2.26, 2.27 versions. > > Is there any Jira issue ticket to track progress regarding it? > > > Best regards, > Dmytro Dragan | dd...@softserveinc.com <mailto:dd...@softserveinc.com> | Lead > Big Data Engineer| SoftServe <http://www.softserveinc.com/>
