Re: Spark 1.6.1 : SPARK-12089 : java.lang.NegativeArraySizeException

2016-03-13 Thread Ted Yu
Here is related code: final int length = totalSize() + neededSize; if (buffer.length < length) { // This will not happen frequently, because the buffer is re-used. final byte[] tmp = new byte[length * 2]; Looks like length was positive (since it was bigger than buffer.length)

Spark 1.6.1 : SPARK-12089 : java.lang.NegativeArraySizeException

2016-03-13 Thread Ravindra Rawat
Greetings, I am getting following exception on joining a few parquet files. SPARK-12089 description has details of the overflow condition which is marked as fixed in 1.6.1. I recall seeing another issue related to csv files creating same exception. Any pointers on how to debug this or possible wo