Re: [DISCUSS] Drop support for Flink 1.10

2021-05-31 Thread Jan Lukavský
Hi, +1 to remove the support for 1.10.  Jan On 5/28/21 10:00 PM, Ismaël Mejía wrote: Hello, With Beam support for Flink 1.13 just merged it is the time to discuss the end of support for Flink 1.10 following the agreed policy on supporting only the latest three Flink releases [1]. I would

No filesystem found for scheme hdfs

2021-05-31 Thread Gershi, Noam
Hi I am using Spark-runner, and when I am using Apache Beam TextIO to read a file from HDFS: .apply(TextIO.read().from("hdfs://path-to-file") I get: Caused by: java.lang.IllegalArgumentException: No filesystem found for scheme hdfs at org.apache.beam.sdk.io.FileSystems.getFileSystemInte

Re: No filesystem found for scheme hdfs

2021-05-31 Thread Ismaël Mejía
You probably need to include the beam-sdks-java-io-hadoop-file-system module. On Mon, May 31, 2021 at 11:41 AM Gershi, Noam wrote: > Hi > > > > I am using Spark-runner, and when I am using Apache Beam TextIO to read a > file from HDFS: > > > > .apply(TextIO.read().from(“hdfs://path-to-file”) >

Re: [DISCUSS] Drop support for Flink 1.10

2021-05-31 Thread David Morávek
Hi, +1 as we've agreed to keep support for three latest major releases in the past D. On Mon, May 31, 2021 at 9:54 AM Jan Lukavský wrote: > Hi, > > +1 to remove the support for 1.10. > > Jan > On 5/28/21 10:00 PM, Ismaël Mejía wrote: > > Hello, > > With Beam support for Flink 1.13 just merged

RenameFields behaves differently in DirectRunner

2021-05-31 Thread Matthew Ouyang
I’m trying to use the RenameFields transform prior to inserting into BigQuery on nested fields. Insertion into BigQuery is successful with DirectRunner, but DataflowRunner has an issue with renamed nested fields The error message I’m receiving, : Error while reading data, error message: JSON pars