For your jsons, can you tell us what is your benchmark when running on a single machine using just plain Java (without Spark and Spark sql)?
Regards Sab On 28-Aug-2015 7:29 am, "Gavin Yue" <[email protected]> wrote: > Hey > > I am using the Json4s-Jackson parser coming with spark and parsing roughly > 80m records with totally size 900mb. > > But the speed is slow. It took my 50 nodes(16cores cpu,100gb mem) roughly > 30mins to parse Json to use spark sql. > > Jackson has the benchmark saying parsing should be ms level. > > Any way to increase speed? > > I am using spark 1.4 on Hadoop 2.7 with Java 8. > > Thanks a lot ! > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > >
