I'll need to transfer large amounts of binary data (in ~30MB chunks) from the compute job sender to the nodes that run the compute jobs. The data will be needed in the compute jobs but each chunk is only needed on one node, while another chunk is needed on another node that computes the same job. I'm wondering what is the most performant way doing this?
a) Technically I could use byte[] objects on the sender and use this in IgniteCallable functions, is this efficiently transferring the data to the nodes? b) I could also first put the data into a cache on the job sender and access the data on each node within the job. Ideally using co-location features. c) Is there a difference to B if I load the data into the cache using a DataStreamer? Would it be more efficient? d) Of course I could also use something outside of Ignite, i.e. JeroMQ. Is this the most efficient way transferring data? Appreciate any help on this.
