On Tue, Feb 03, 2015 at 06:06:20PM +0100, Mark Janssen (isync) wrote: > Added anothe valgrind log which core dumps. It seems the maildir > driver is never releasing memory, so if you have a large number of > messages (or very large messages) the maildir_fetch_msg will consume > all memory. > that's because *_fetch_msg() loads the entire message into ram, and only *_store_msg() releases it after completion. when the opposite side is asynchronous (as is the case with imap), all the messages get queued up in form of imap commands. this is usually no problem, as generally only a few messages are transferred at a time. but of course it blows up when you do the initial sync of a huge mailbox to an imap server.
to address this problem, there would have to be a limit on simultaneously buffered messages. this means making the core algorithm asynchronous, which sounds like an interesting exercise ... the other problem are huge messages. to address it, one would have to stream the data between source and sink in small chunks. i'm not sure this is worth the effort, given that even the largest messages (which any server would accept) are orders of magnitude smaller than what fits into a reasonable computer's memory. ------------------------------------------------------------------------------ Dive into the World of Parallel Programming. The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net/ _______________________________________________ isync-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/isync-devel
