I think Isilon also supports HDFS. Years ago, I used to work with Hadoop
and Spark with data that was stored on Isilon appliances, that were
normally accessed using NFS, and we used HDFS to access that data (and to
write data back to Isilon). I did not participate myself in the
configuration of Isi
I would guess that you have some existing code that expects random IO
access to the files via the Java IO or NIO interface (the common blocking
IO in a DoFn pattern), so using a Beam IO which is what we recommend and
are discussing here would be a significant rewrite?
I worked on Isilon from 6.5 -
I would also suggest looking at NFS client implementations in Java that
would allow you to talk to the NFS server without needing to mount it
within the OS. A quick search yielded https://github.com/raisercostin/yanfs
or https://github.com/EMCECS/nfs-client-java
On Tue, Jan 31, 2023 at 3:31 PM Cha
Thanks for the info. We are going to test this further and we'll let you
know how it goes.
-chad
On Mon, Jan 30, 2023 at 2:14 PM Valentyn Tymofieiev
wrote:
> It applies to custom containers as well. You can find the container
> manifest in the GCE VM metadata, and it should have an entry for
It applies to custom containers as well. You can find the container
manifest in the GCE VM metadata, and it should have an entry for privileged
mode. The reason for this was to enable GPU accelerator support, but agree
with Robert that it is not part of any contracts, so in theory this could
change
I'm also not sure it's part of the contract that the containerization
technology we use will always have these capabilities.
On Mon, Jan 30, 2023 at 10:53 AM Chad Dombrova wrote:
>
> Hi Valentyn,
>
>>
>> Beam SDK docker containers on Dataflow VMs are currently launched in
>> privileged mode.
>
>
Hi Valentyn,
> Beam SDK docker containers on Dataflow VMs are currently launched in
> privileged mode.
>
Does this only apply to stock sdk containers? I'm asking because we use a
custom sdk container that we build. We've tried various ways of running
mount from within our custom beam container
Beam SDK docker containers on Dataflow VMs are currently launched in
privileged mode.
On Mon, Jan 30, 2023 at 9:52 AM Robert Bradshaw via user <
user@beam.apache.org> wrote:
> Different idea: is it possible to serve this data via another protocol
> (e.g. sftp) rather than requiring a mount?
>
> O
Different idea: is it possible to serve this data via another protocol
(e.g. sftp) rather than requiring a mount?
On Mon, Jan 30, 2023 at 9:26 AM Chad Dombrova wrote:
>
> Hi Robert,
> I know very little about the FileSystem classes, but I don’t think it’s
> possible for a process running in dock
Hi Robert,
I know very little about the FileSystem classes, but I don’t think it’s
possible for a process running in docker to create an NFS mount without
running in privileged [1] mode, which cannot be done with Dataflow. The
other ways of gaining access to a mount are:
A. the node running docker
If it's your input/output data, presumably you could implement a
https://beam.apache.org/releases/javadoc/2.3.0/org/apache/beam/sdk/io/FileSystem.html
for nfs. (I don't know what all that would entail...)
On Mon, Jan 30, 2023 at 9:04 AM Chad Dombrova wrote:
>
> Hi Israel,
> Thanks for responding.
Hi Israel,
Thanks for responding.
And could not the dataset be accessed from Cloud Storage? Does it need to
> be specifically NFS?
>
No unfortunately it can't be accessed from Cloud Storage. Our data
resides on high performance Isilon [1] servers using a posix filesystem,
and NFS is the tried a
And could not the dataset be accessed from Cloud Storage? Does it need to
be specifically NFS?
On Thu, 26 Jan 2023 at 18:16, Chad Dombrova wrote:
> Hi all,
> We have large data sets which we would like to mount over NFS within
> Dataflow. As far as I know, this is not possible. Has anything ch
13 matches
Mail list logo
