Based on the latest spark code(commit
608353c8e8e50461fafff91a2c885dca8af3aaa8) and used the same Spark SQL query
to test two group of combined configuration and seemed that currently it
don't work fine in "tungsten-sort" shuffle manager from below results:
*Test 1# (PASSED)*
spark.shuffle.manager
Thank you for your reply!
Do you mean that currently if i want to use this Tungsten feature, we had to
set sort shuffle manager(spark.shuffle.manager=sort) ,right ? However, I
saw a slide "Deep Dive into Project Tungsten: Bringing Spark Closer to Bare
Metal" published in Spark Summit 2015 and it s
It would also be great to test this with codegen and unsafe enabled but
while continuing to use sort shuffle manager instead of the new
tungsten-sort one.
On Fri, Jul 31, 2015 at 1:39 AM, Reynold Xin wrote:
> Is this deterministically reproducible? Can you try this on the latest
> master branch?
Is this deterministically reproducible? Can you try this on the latest
master branch?
Would be great to turn debug logging and and dump the generated code. Also
would be great to dump the array size at your line 314 in UnsafeRow (and
whatever master branch's appropriate line is).
On Fri, Jul 31,
Another error：
15/07/31 16:15:28 INFO spark.MapOutputTrackerMasterEndpoint: Asked to send
map output locations for shuffle 3 to bignode1:40443
15/07/31 16:15:28 INFO spark.MapOutputTrackerMaster: Size of output statuses
for shuffle 3 is 583 bytes
15/07/31 16:15:28 INFO spark.MapOutputTrackerMasterE
