Re: Low throughput when trying to send data with Sockets

2018-01-26 Thread Aljoscha Krettek
Taking a step back: why do you want to manually implement communication via sockets in the first place? With this you will not get any fault-tolerance guarantees and I would guess that maintaining a custom solution is more difficult than using, say, Kafka. Best, Aljoscha > On 16. Jan 2018, at

Re: Low throughput when trying to send data with Sockets

2018-01-16 Thread Nico Kruber
(back to the ml again) If you implement the ParallelSourceFunction interface instead, Flink will run as many source instances as the configured parallelism. Each instance will run the same code and you'll thus have multiple sockets to connect to, if that is what you wanted. One more thing regard

Re: Low throughput when trying to send data with Sockets

2018-01-16 Thread Nico Kruber
Hi George, I suspect issuing a read operation for every 68 bytes incurs too much overhead to perform as you would like it to. Instead, create a bigger buffer (64k?) and extract single events from sub-regions of this buffer instead. Please note, however, that then the first buffer will only be proce