Aw: Re: Best way transferring large binary data to nodes

[Kramer]

It will be thousands of chunks of each 30MB in very short time. Still fine to use IgniteCompute simplified like this to pass the byte data over to nodes?

 

 

        try (Ignite ignite = Ignition.start())
        {
            final byte[] buffer = new byte[30 * 1024 * 1024];

            int length = ignite.compute().call(new IgniteCallable<Integer>()
            {
                @Override
                public Integer call() throws Exception
                {
                    // do something with buffer
                    // ...
                    // ...

                    return buffer.length;
                }
            });
            
            System.out.println("Length: " + length);
        }

 

 

 

Gesendet: Mittwoch, 17. November 2021 um 15:27 Uhr
Von: "Pavel Tupitsyn" <ptupit...@apache.org>
An: "user" <user@ignite.apache.org>
Betreff: Re: Best way transferring large binary data to nodes

If you only need to process the data, but not store it, I would suggest using IgniteCompute.

Yes, sending byte[] is efficient. 30MB is not that much and should be fine.

 

On Wed, Nov 17, 2021 at 12:34 PM Thomas Kramer <don.tequ...@gmx.de> wrote:

I'll need to transfer large amounts of binary data (in ~30MB chunks)
from the compute job sender to the nodes that run the compute jobs. The
data will be needed in the compute jobs but each chunk is only needed on
one node, while another chunk is needed on another node that computes
the same job. I'm wondering what is the most performant way doing this?

a) Technically I could use byte[] objects on the sender and use this in
IgniteCallable functions, is this efficiently transferring the data to
the nodes?

b) I could also first put the data into a cache on the job sender and
access the data on each node within the job. Ideally using co-location
features.

c) Is there a difference to B if I load the data into the cache using a
DataStreamer? Would it be more efficient?

d) Of course I could also use something outside of Ignite, i.e. JeroMQ.
Is this the most efficient way transferring data?

Appreciate any help on this.