Hey Nicholas,
I am not sure if the problem of Flink dropping TM is connected to ethernet
because it happens in our Prod cluster too where we have standard blade
servers with Gigabyte network (din spent time yet on doing the RCA).
Nonetheless, its really cool what you are doing there with rPis :))).
Cheers, Ankit
On Sun, Apr 29, 2018 at 4:45 PM, Nicholas Walton <nwal...@me.com> wrote:
> Thanks for the reply, but I tracked the problem down to a missing M in
> task m=
> anager.sh
>
> # export JVM_ARGS=3D"${JVM_ARGS} -Xms${TM_HEAP_SIZE}M -Xmx${TM_HEAP_SIZE}M
> -=
> XX:MaxDirectMemorySize=3D${TM_MAX_OFFHEAP_SIZE}"
> export JVM_ARGS=3D"${JVM_ARGS} -Xms${TM_HEAP_SIZE}M -Xmx${TM_HEAP_SIZE}M"
>
> I had to remove {TM_MAX_OFFHEAP_SIZE} which caused the Pi Java to barf,
> and h=
> ad accidentally deleted the final M.
>
> After that I could run with a heap set to 512m with minimal problem.
>
> As for the Pi as a Flink client, it may not be powerful in conventional
> sens=
> e but I can directly attach a cluster to a sensor array and process the
> data=
> in real time from multiple streams, which is neat. I processed 3 million
> re=
> adings in the prototype in just over an hour with a modest amount of
> process=
> ing required, all on 5pis with a sixth acting as the job manager. My only
> co=
> mplaint is that Flink has a habit of dropping task managers even though
> they=
> 're still running. I suspect that's because of slow Ethernet on the Pi3
> whic=
> h has been addressed on the Pi3+ just released. Realistically sized rack
> mou=
> nted clusters are now possible from http://www.bitscope.com/product/blade/
> a=
> fter Los Alamos National Lab went looking for a cheap means to build
> prototy=
> pe clusters, and settled on the Pi to build a 750+ node cluster.
>
