Underneath the covers, jsonFile uses TextInputFormat, which will split files correctly based on new lines. Thus, there is no fixed maximum size for a json object (other than the fact that it must fit into memory on the executors).
On Mon, Dec 15, 2014 at 7:22 AM, Madabhattula Rajesh Kumar < mrajaf...@gmail.com> wrote: > > Hi Peter, > > Thank you for the clarification. > > Now we need to store each JSON object into one line. Is there any > limitation of length of JSON object? So, JSON object will not go to the > next line. > > What will happen if JSON object is a big/huge one? Will it store in a > single line in HDFS? > > What will happen, if JSON object contains BLOB/CLOB value? Is this entire > JSON object stores in single line of HDFS? > > What will happen, if JSON object exceeding the HDFS block size. For > example, single JSON object split into two different worker nodes. In this > case, How Spark will read this JSON object? > > Could you please clarify above questions > > Regards, > Rajesh > > > On Mon, Dec 15, 2014 at 6:52 PM, Peter Vandenabeele < > pe...@vandenabeele.com> wrote: >> >> >> >> On Sat, Dec 13, 2014 at 5:43 PM, Helena Edelson < >> helena.edel...@datastax.com> wrote: >> >>> One solution can be found here: >>> https://spark.apache.org/docs/1.1.0/sql-programming-guide.html#json-datasets >>> >>> >> As far as I understand, the people.json file is not really a proper json >> file, but a file documented as: >> >> "... JSON files where each line of the files is a JSON object.". >> >> This means that is a file with multiple lines, but each line needs to >> have a fully self-contained JSON object >> (initially confusing, this will not parse a standard multi-line JSON >> file). We are working to clarify this in >> https://github.com/apache/spark/pull/3517 >> >> HTH, >> >> Peter >> >> >> >> >>> - Helena >>> @helenaedelson >>> >>> On Dec 13, 2014, at 11:18 AM, Madabhattula Rajesh Kumar < >>> mrajaf...@gmail.com> wrote: >>> >>> Hi Team, >>> >>> I have a large JSON file in Hadoop. Could you please let me know >>> >>> 1. How to read the JSON file >>> 2. How to parse the JSON file >>> >>> Please share any example program based on Scala >>> >>> Regards, >>> Rajesh >>> >>> >>> >> >> >> -- >> Peter Vandenabeele >> http://www.allthingsdata.io >> http://www.linkedin.com/in/petervandenabeele >> https://twitter.com/peter_v >> gsm: +32-478-27.40.69 >> e-mail: pe...@vandenabeele.com >> skype: peter_v_be >> >
