Hey Jeff, in addition to what Sandy said, there are two more reasons that this might not be as bad as it seems; I may be incorrect in my understanding though.
First, the "additional step" you're referring to is not likely to be adding any overhead; the "extra map" is really just materializing the data once (as opposed to zero times), which is what you want (assuming your access pattern couldn't be reformulated in the way Sandy described, i.e. where all the objects in a partition don't need to be in memory at the same time). Secondly, even if this was an "extra" map step, it wouldn't add any extra stages to a given pipeline, being a "narrow" dependency, so it would likely be low-cost anyway. Let me know if any of the above seems incorrect, thanks! On Thu, Nov 19, 2015 at 12:41 AM Sandy Ryza <sandy.r...@cloudera.com> wrote: > Hi Jeff, > > Many access patterns simply take the result of hadoopFile and use it to > create some other object, and thus have no need for each input record to > refer to a different object. In those cases, the current API is more > performant than an alternative that would create an object for each record, > because it avoids the unnecessary overhead of creating Java objects. As > you've pointed out, this is at the expense of making the code more verbose > when caching. > > -Sandy > > On Fri, Nov 13, 2015 at 10:29 AM, jeff saremi <jeffsar...@hotmail.com> > wrote: > >> So we tried reading a sequencefile in Spark and realized that all our >> records have ended up becoming the same. >> THen one of us found this: >> >> Note: Because Hadoop's RecordReader class re-uses the same Writable >> object for each record, directly caching the returned RDD or directly >> passing it to an aggregation or shuffle operation will create many >> references to the same object. If you plan to directly cache, sort, or >> aggregate Hadoop writable objects, you should first copy them using a map >> function. >> >> Is there anyone that can shed some light on this bizzare behavior and the >> decisions behind it? >> And I also would like to know if anyone's able to read a binary file and >> not to incur the additional map() as suggested by the above? What format >> did you use? >> >> thanks >> Jeff >> > >
