Thank you for reporting a problem.
Would it be possible to create a JIRA entry with a small program that can
reproduce this problem?
Best Regards,
Kazuaki Ishizaki
From: Rico Bergmann <i...@ricobergmann.de>
To: "user@spark.apache.org" <user@spark.apache.org>
Date: 2018/06/05 19:58
Subject: Strange codegen error for SortMergeJoin in Spark 2.2.1
Hi!
I get a strange error when executing a complex SQL-query involving 4
tables that are left-outer-joined:
Caused by: org.codehaus.commons.compiler.CompileException: File
'generated.java', Line 37, Column 18: failed to compile:
org.codehaus.commons.compiler.CompileException: File 'generated.java',
Line 37, Column 18: No applicable constructor/method found for actual
parameters "int"; candidates are:
"org.apache.spark.sql.execution.ExternalAppendOnlyUnsafeRowArray(org.apache.spark.memory.TaskMemoryManager,
org.apache.spark.storage.BlockManager,
org.apache.spark.serializer.SerializerManager,
org.apache.spark.TaskContext, int, long, int, int)",
"org.apache.spark.sql.execution.ExternalAppendOnlyUnsafeRowArray(int,
int)"
...
/* 037 */ smj_matches = new
org.apache.spark.sql.execution.ExternalAppendOnlyUnsafeRowArray(2147483647);
The same query works with Spark 2.2.0.
I checked the Spark source code and saw that in
ExternalAppendOnlyUnsafeRowArray a second int was introduced into the
constructor in 2.2.1
But looking at the codegeneration part of SortMergeJoinExec:
// A list to hold all matched rows from right side.
val matches = ctx.freshName("matches")
val clsName = classOf[ExternalAppendOnlyUnsafeRowArray].getName
val spillThreshold = getSpillThreshold
val inMemoryThreshold = getInMemoryThreshold
ctx.addMutableState(clsName, matches,
s"$matches = new $clsName($inMemoryThreshold, $spillThreshold);")
it should get 2 parameters, not just one.
May be anyone has an idea?
Best,
Rico.
