Re: Why so slow

2015-05-12 Thread Jianshi Huang
| 169771| 0.40812641191854626 | >> | 6 | 542447| 0.5238256418341465| >> | 7 | 160324| 0.29442847034840386 | >> | 8 | 2099 | -0.9165701665162977 | >> | 9 | 3104 | 0.3845685004598235| >> +-+--

Re: Why so slow

2015-05-12 Thread Olivier Girardot
030563 | > | 5 | 169771| 0.40812641191854626 | > | 6 | 542447| 0.5238256418341465| > | 7 | 160324| 0.29442847034840386 | > | 8 | 2099 | -0.9165701665162977 | > | 9 | 3104 | 0.3845685004598235| > +-----+---+---

Why so slow

2015-05-12 Thread Jianshi Huang
162977 | | 9 | 3104 | 0.3845685004598235| +-+---+---+ 10 rows selected (130.5 seconds) The total number of rows is less than 20M. Why so slow? I'm running on Spark 1.4.0-SNAPSHOT with 100 executors each having 4GB ram and 2 CPU core. Look