Hi!
I get a strange error when executing a complex SQL-query involving 4
tables that are left-outer-joined:
Caused by: org.codehaus.commons.compiler.CompileException: File
'generated.java', Line 37, Column 18: failed to compile:
org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 37,
Column 18: No applicable constructor/method found for actual parameters "int";
candidates are:
"org.apache.spark.sql.execution.ExternalAppendOnlyUnsafeRowArray(org.apache.spark.memory.TaskMemoryManager,
org.apache.spark.storage.BlockManager,
org.apache.spark.serializer.SerializerManager, org.apache.spark.TaskContext,
int, long, int, int)",
"org.apache.spark.sql.execution.ExternalAppendOnlyUnsafeRowArray(int, int)"
...
/* 037 */ smj_matches = new
org.apache.spark.sql.execution.ExternalAppendOnlyUnsafeRowArray(2147483647);
The same query works with Spark 2.2.0.
I checked the Spark source code and saw that in
ExternalAppendOnlyUnsafeRowArray a second int was introduced into the
constructor in 2.2.1
But looking at the codegeneration part of SortMergeJoinExec:
// A list to hold all matched rows from right side. val matches =
ctx.freshName("matches")
val clsName = classOf[ExternalAppendOnlyUnsafeRowArray].getName
val spillThreshold = getSpillThreshold
val inMemoryThreshold = getInMemoryThreshold
ctx.addMutableState(clsName, matches, s"$matches= new
$clsName($inMemoryThreshold, $spillThreshold);")
it should get 2 parameters, not just one.
May be anyone has an idea?
Best,
Rico.
