?
Thanks.
Yang
--
View this message in context:
http://apache-spark-developers-list.1001551.n3.nabble.com/TakeOrderedAndProject-operator-may-causes-an-OOM-tp16208.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com
Hi,
Currently the TakeOrderedAndProject operator in spark sql uses RDD’s
takeOrdered method. When we pass a large limit to operator, however, it will
return partitionNum*limit number of records to the driver which may cause an
OOM.
Are there any plans to deal with the problem in the community?
