Re: Strange DataSet behavior when using custom FileInputFormat

2019-08-09 Thread Zhu Zhu
Hi Hynek, In execution, matrices.first(6).print() is different from matrices.print(). It is adding a reducer operator to the job which only collects the first 6000 records from the source. So if your InputFormat can generate more than 6 (which can be unexpected though), and the trailing da

Strange DataSet behavior when using custom FileInputFormat

2019-08-09 Thread Hynek Noll
Hi, I'm trying to implement a custom FileInputFormat (to read the MNIST Dataset). The creation of Flink DataSet (DataSet matrices) seems to be OK, but when I try to print it using either matrices.print(); or matrices.collect(); It finishes with exit code -17. (Before, I compiled using Java 11 and