Can you give an example with my extract? Mélanie Gallois
2015-07-29 16:55 GMT+02:00 Young, Matthew T <[email protected]>: > The built-in Spark JSON functionality cannot read normal JSON arrays. The > format it expects is a bunch of individual JSON objects without any outer > array syntax, with one complete JSON object per line of the input file. > > AFAIK your options are to read the JSON in the driver and parallelize it > out to the workers or to fix your input file to match the spec. > > For one-off conversions I usually use a combination of jq and > regex-replaces to get the source file in the right format. > > ________________________________________ > From: SparknewUser [[email protected]] > Sent: Wednesday, July 29, 2015 6:37 AM > To: [email protected] > Subject: How to read a Json file with a specific format? > > I'm trying to read a Json file which is like : > [ > > {"IFAM":"EQR","KTM":1430006400000,"COL":21,"DATA":[{"MLrate":"30","Nrout":"0","up":null,"Crate":"2"} > ,{"MLrate":"30","Nrout":"0","up":null,"Crate":"2"} > ,{"MLrate":"30","Nrout":"0","up":null,"Crate":"2"} > ,{"MLrate":"30","Nrout":"0","up":null,"Crate":"2"} > ,{"MLrate":"30","Nrout":"0","up":null,"Crate":"2"} > ,{"MLrate":"30","Nrout":"0","up":null,"Crate":"2"} > ]} > > ,{"IFAM":"EQR","KTM":1430006400000,"COL":22,"DATA":[{"MLrate":"30","Nrout":"0","up":null,"Crate":"2"} > ,{"MLrate":"30","Nrout":"0","up":null,"Crate":"2"} > ,{"MLrate":"30","Nrout":"0","up":null,"Crate":"2"} > ,{"MLrate":"30","Nrout":"0","up":null,"Crate":"2"} > ,{"MLrate":"30","Nrout":"0","up":null,"Crate":"2"} > ,{"MLrate":"30","Nrout":"0","up":null,"Crate":"2"} > ]} > ] > > I've tried the command: > val df = sqlContext.read.json("namefile") > df.show() > > > But this does not work : my columns are not recognized... > > > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/How-to-read-a-Json-file-with-a-specific-format-tp24061.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > > -- *Mélanie*
