Re: Using Encoding to reduce GraphX's static graph memory consumption

Actually another improvement would be to use something like compressed sparse row encoding which can be used to store A and A^T relatively efficiently (I think using 5 arrays instead of 6). There is an option to also be more cache aware using something like a block compressed sparse row encodin

Re: Using Encoding to reduce GraphX's static graph memory consumption

Hi I have created a jira for this issue here. As for the pull request, my implementation is based on removing localSrcIds and storing an array of offsets into localDs

Re: Using Encoding to reduce GraphX's static graph memory consumption

Yes, sounds good. I can submit the pull request. On 22 Feb 2016 00:35, "Reynold Xin" wrote: > + Joey > > We think this is worth doing. Are you interested in submitting a pull > request? > > > On Sat, Feb 20, 2016 at 8:05 PM ahaider3 wrote: > >> Hi, >> I have been looking through the GraphX sourc

Re: Using Encoding to reduce GraphX's static graph memory consumption

+ Joey We think this is worth doing. Are you interested in submitting a pull request? On Sat, Feb 20, 2016 at 8:05 PM ahaider3 wrote: > Hi, > I have been looking through the GraphX source code, dissecting the reason > for its high memory consumption compared to the on-disk size of the graph. >

Using Encoding to reduce GraphX's static graph memory consumption

Hi, I have been looking through the GraphX source code, dissecting the reason for its high memory consumption compared to the on-disk size of the graph. I have found that there may be room to reduce the memory footprint of the graph structures. I think the biggest savings can come from the localSrc