If you only need to process the data, but not store it, I would suggest using IgniteCompute. Yes, sending byte[] is efficient. 30MB is not that much and should be fine.
On Wed, Nov 17, 2021 at 12:34 PM Thomas Kramer <don.tequ...@gmx.de> wrote: > I'll need to transfer large amounts of binary data (in ~30MB chunks) > from the compute job sender to the nodes that run the compute jobs. The > data will be needed in the compute jobs but each chunk is only needed on > one node, while another chunk is needed on another node that computes > the same job. I'm wondering what is the most performant way doing this? > > a) Technically I could use byte[] objects on the sender and use this in > IgniteCallable functions, is this efficiently transferring the data to > the nodes? > > b) I could also first put the data into a cache on the job sender and > access the data on each node within the job. Ideally using co-location > features. > > c) Is there a difference to B if I load the data into the cache using a > DataStreamer? Would it be more efficient? > > d) Of course I could also use something outside of Ignite, i.e. JeroMQ. > Is this the most efficient way transferring data? > > Appreciate any help on this. > >
