On 14 Nov 2017, at 15:32, Alec Swan <alecs...@gmail.com<mailto:alecs...@gmail.com>> wrote:
But I wonder if there is a way to stream/batch the content of JSON file in order to convert it to ORC piecemeal and avoid reading the whole JSON file in memory in the first place? That is what you'll need to do; you'd hit similar problems if you had the same files, same allocated JVM space and the same # of threads trying to read in the files. Jackson has a streaming API: http://www.baeldung.com/jackson-streaming-api
