Re: Catalyst optimizer cpu/Io cost
Hi,
There no way to retrieve that information in spark.
In fact, the current optimizer only consider the byte size of outputs in
LogicalPlan.
Related code can be found in
https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql
: Friday, June 10, 2016 3:27 PM
To: Srinivasan Hariharan02
Cc: Takeshi Yamamuro ; user@spark.apache.org
Subject: Re: Catalyst optimizer cpu/Io cost
in an SMP system such as Oracle or Sybase the CBO will take into account LIO,
PIO and CPU costing or use some empirical costing.
In a distributed system
gt; *Regards,*
>
> *Srinivasan Hariharan*
>
> *+91-9940395830 <%2B91-9940395830>*
>
>
>
> *From:* Takeshi Yamamuro [mailto:linguin@gmail.com]
> *Sent:* Friday, June 10, 2016 2:38 PM
> *To:* Srinivasan Hariharan02
> *Cc:* user@spark.apache.org
> *Subject:
[mailto:linguin@gmail.com]
Sent: Friday, June 10, 2016 2:38 PM
To: Srinivasan Hariharan02
Cc: user@spark.apache.org
Subject: Re: Catalyst optimizer cpu/Io cost
Hi,
There no way to retrieve that information in spark.
In fact, the current optimizer only consider the byte size of outputs in
Hi,
There no way to retrieve that information in spark.
In fact, the current optimizer only consider the byte size of outputs in
LogicalPlan.
Related code can be found in
https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/LogicalPla
Hi,,
How can I get spark sql query cpu and Io cost after optimizing for the best
logical plan. Is there any api to retrieve this information?. If anyone point
me to the code where actually cpu and Io cost computed in catalyst module.
Regards,
Srinivasan Hariharan
+91-9940395830
