Hi Uli, Thanks for the thoughtful reply. I do think I could probably reduce the data load of my graph by wider use of primitives, but it would require a fair bit of re-engineering, and there's not a lot of time to do that… (Have to finish my PhD someday!) Also, since the system should, at least in theory, be able to learn from absurdly huge corpora, I think I'd still likely hit the memory wall at some point. Which is simply to say that freeing myself from memory limitations is probably a good idea anyway. Having the data structures on disk will, of course, slow things down considerably, but I think will be worth the trade-off. I'll keep poking around with Core Data until I figure out a reasonable solution.
That said, I'll also put further thought into how I might reduce the memory footprint of my model, as is. I'm sure it's possible. Thanks, J. On 2013-03-03, at 1:57 AM, Uli Kusterer <witness.of.teacht...@gmx.net> wrote: > Just a question: Do you really *need* NSDictionaries? I.e. is this something > where arbitrary string keys need to be mapped to objects? Because otherwise > this might be one of the cases where NSDictionary is too generic, and you may > just want to create your own class. > > E.g. when I started out, I did a game engine with all NSDictionaries, > NSNumbers etc. Rewriting the dictionaries to be simple objects with NSPoint > iVars etc. yielded a huge speed-up (because I was obviously kinda abusing > NSDictionary). > > Objects are preferable, but they do have a trade-off: For each of them, you > allocate a new block on the heap, which on most platform has an overhead of > at least 8 bytes (32-bit) or 16 bytes (64-bit) to keep a pointer to the class > (isa) and the block's length. Often it uses more to allow the system to keep > track of empty spots on the heap. > > If your payload is a 4-byte int, that means each NSNumber "wastes" about 2x > to 4x the memory it actually needs (and that's a conservative estimate). So > if it makes sense to group several ints or floats in ivars of a single > object, that should quickly reduce your memory use. Similarly, if you have a > list of primitive types, not objects, you can use CFArray with custom > callbacks to store ints directly instead of boxing each one in an NSNumber. > > Or just grab one of the numerous IntArray classes out there that mallocs its > own buffer, if all you need is really a straight, linear, indexed array w/o > all the optimizations that NSArray uses to speed up search for a particular > value etc., but you want the convenience of an ObjC API that CFArray doesn't > quite offer. > > Cheers, > -- Uli Kusterer > "The Witnesses of TeachText are everywhere..." > > > On 03.03.2013, at 05:23, James Maxwell <jbmaxw...@rubato-music.com> wrote: >> Hello All, >> >> I have an app that creates a large and complex graph structure. It's working >> nicely now, after a fairly lengthy development process, but I'm having >> scaling issues -- the data structure gets too big to keep in memory. So, I'm >> thinking about Core Data. The thing is, the process of building the object >> graph is very specific (it's a complex undirected graph structure, built via >> machine learning) and since I've got it working properly I really don't want >> to port the whole thing over to Core Data, since that would just be begging >> for a huge headache, I'm sure. >> >> All of the data is held in NSMutableDictionaries (basically working as >> adjacency lists), owned by one particular class. What I'm wondering is >> whether there's some way to use Core Data to perform the work of the >> dictionaries, without necessarily rebuilding all of the object relationships >> etc. using Core Data entities? Looking around a bit it looks like >> "Transformable" attributes might offer a solution. What I'd like to do is to >> basically swap out the NSMutableDictionaries for Core Data entities that >> store my custom objects as transformable attributes. Is that at all >> feasible? Worth trying? >> >> Thanks in advance. >> >> J. >> _______________________________________________ >> >> Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) >> >> Please do not post admin requests or moderator comments to the list. >> Contact the moderators at cocoa-dev-admins(at)lists.apple.com >> >> Help/Unsubscribe/Update your Subscription: >> https://lists.apple.com/mailman/options/cocoa-dev/witness.of.teachtext%40gmx.net >> >> This email sent to witness.of.teacht...@gmx.net > _______________________________________________ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: https://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
