Thanks for your response. What do you mean when you said "immediately return"?
On Wed, Mar 28, 2018, 10:33 PM Jörn Franke <jornfra...@gmail.com> wrote: > I don’t think select * is a good benchmark. You should do a more complex > operation, otherwise optimizes might see that you don’t do anything in the > query and immediately return (similarly count might immediately return by > using some statistics). > > On 29. Mar 2018, at 02:03, Tin Vu <tvu...@ucr.edu> wrote: > > Hi, > > I am executing a benchmark to compare performance of SparkSQL, Apache > Drill and Presto. My experimental setup: > > - TPCDS dataset with scale factor 100 (size 100GB). > - Spark, Drill, Presto have a same number of workers: 12. > - Each worked has same allocated amount of memory: 4GB. > - Data is stored by Hive with ORC format. > > I executed a very simple SQL query: "SELECT * from table_name" > The issue is that for some small size tables (even table with few dozen of > records), SparkSQL still required about 7-8 seconds to finish, while Drill > and Presto only needed less than 1 second. > For other large tables with billions records, SparkSQL performance was > reasonable when it required 20-30 seconds to scan the whole table. > Do you have any idea or reasonable explanation for this issue? > > Thanks, > >
