In PySpark, given a DataFrame, I am attempting to save it as JSON
Lines/ndjson. I run this code:
json_lines = on_time_dataframe.map(lambda x: json.dumps(x))
json_lines.saveAsTextFile('../data/On_Time_On_Time_Performance_2015.jsonl')
This results in simple arrays of fields, instead of JSON objects:
[2015, 1, 1, 1, 4, "2015-01-01", "AA", 19805, "AA", "N787AA", 1, 12478,
1247802, 31703, "JFK", "New York, NY", "NY", 36, "New York", 22, 12892,
1289203, 32575, "LAX", "Los Angeles, CA", "CA", 6, "California", 91, 900,
855, -5.0, 0.0, 0.0, -1, "0900-0959", 17.0, 912, 1230, 7.0, 1230, 1237,
7.0, 7.0, 0.0, 0, "1200-1259", 0.0, "", 0.0, 390.0, 402.0, 378.0, 1.0,
2475.0, 10, null, null, null, null, null, null, null, null, 0, null, null,
null, null, "", null, null, null, null, null, null, "", "", null, null,
null, null, null, null, "", "", null, null, null, null, null, "", "", "",
"", "", "", "", "", "", "", "", "", "", "", "", "", "", "", ""]
What I actually want is JSON objects, with a field name for each field:
{"year": "2015", "month": 1, ...}
How can I achieve this in PySpark?
Thanks!
--
Russell Jurney twitter.com/rjurney russell.jur...@gmail.com relato.io
