Hello Sir/Madam,
I am writing one application using spark sql.
i made the vary big table using the following command
*val dfCustomers1 =
sc.textFile("/root/Desktop/database.txt").map(_.split(",")).map(p =>
Customer1(p(0),p(1).trim.toInt, p(2).trim.toInt, p(3)))toDF*
Now i want to search the address(many address) fields in the table and
then extends the new table as per the searching.
*var k = dfCustomers1.filter(dfCustomers1("Address").equalTo(lines(0)))*
*for( a <-1 until 1500) {*
* | var temp=
dfCustomers1.filter(dfCustomers1("Address").equalTo(lines(a)))*
* | k = temp.unionAll(k)*
*}*
*k.show*
But this is taking so long time. So can you suggest me the any optimized
way, so i can reduce the execution time.
My cluster has 3 slaves and 1 master.
Thanks.
