You probably need to include the beam-sdks-java-io-hadoop-file-system
module.
On Mon, May 31, 2021 at 11:41 AM Gershi, Noam wrote:
> Hi
>
>
>
> I am using Spark-runner, and when I am using Apache Beam TextIO to read a
> file from HDFS:
>
>
>
> .apply(TextIO.read().from(“hdfs://path-to-file”)
>
Thanks for sharing that
Tim,
Sent from my iPhone
> On 26 Oct 2018, at 17:50, Juan Carlos Garcia wrote:
>
> Just for everyone to know we figure it out, it was an environment problem.
>
> In our case we have our cluster in a network that is not accessible directly,
> so to deploy we need to use
Just for everyone to know we figure it out, it was an environment problem.
In our case we have our cluster in a network that is not accessible
directly, so to deploy we need to uses Jenkins with some slaves that have
access to that network.
During deployment in the *main* method of the class we
Hi Tim,
I am using FileIO directly with the AvroIO.sink(...), however having
experienced BEAM-2277 with the SparkRunner few months ago, i got the
feeling this is something different (maybe some dependency
mismatch/missing).
Thanks
On Fri, Oct 26, 2018 at 1:33 PM Tim Robertson
wrote:
> Hi Juan
Hi Juan
This sounds reminiscent of https://issues.apache.org/jira/browse/BEAM-2277
which we believed fixed in 2.7.0.
What IO are you using to write your files and can you paste a snippet of
your code please?
On BEAM-2277 I posted a workaround for AvroIO (it might help you find a
workaround too):
