Re: Reading Huge UnixMailbox Files

2011-04-27 Thread Nobody
On Tue, 26 Apr 2011 14:02:23 -0700, Dan Stromberg wrote: > For the archive: This assumes traditional mbox. A SysV-ish sendmail, > for example, may not like it. sendmail itself doesn't deal with mailboxes or spool files; that task is left to the local delivery agent (e.g. mail.local or procmail).

Re: Reading Huge UnixMailbox Files

2011-04-26 Thread Dan Stromberg
On Tue, Apr 26, 2011 at 1:23 PM, Nobody wrote: > E.g. the following script reads a mailbox on stdin and writes a separate > file for each message: > >        #!/usr/bin/awk -f >        BEGIN { >                num = 0; >                ofile = ""; >        } > >        /^From / { >                

Re: Reading Huge UnixMailbox Files

2011-04-26 Thread Nobody
On Tue, 26 Apr 2011 15:39:37 -0400, Brandon McGinty wrote: > I'm trying to import hundreds of thousands of e-mail messages into a > database with Python. > However, some of these mailboxes are so large that they are giving > errors when being read with the standard mailbox module. > I created a bu

Re: Reading Huge UnixMailbox Files

2011-04-26 Thread Dan Stromberg
On Tue, Apr 26, 2011 at 12:39 PM, Brandon McGinty wrote: > List, > I'm trying to import hundreds of thousands of e-mail messages into a > database with Python. > However, some of these mailboxes are so large that they are giving > errors when being read with the standard mailbox module. > I create

Reading Huge UnixMailbox Files

2011-04-26 Thread Brandon McGinty
List, I'm trying to import hundreds of thousands of e-mail messages into a database with Python. However, some of these mailboxes are so large that they are giving errors when being read with the standard mailbox module. I created a buffered reader, that reads chunks of the mailbox, splits them usi