On Feb 27, 1:18 am, Stefan Behnel <stefan...@behnel.de> wrote: > bearophileh...@lycos.com wrote: > > odeits: > >> How big of a list are we talking about? If the list is so big that the > >> entire list cannot fit in memory at the same time this approach wont > >> work e.g. removing duplicate lines from a very large file. > > > If the data are lines of a file, and keeping the original order isn't > > important, then the first to try may be to use the unix (or cygwin on > > Windows) commands sort and uniq. > > or preferably "sort -u", in case that's supported. > > Stefan
Although this is true, that is more of an answer to the question "How do i remove duplicates from a huge list in Unix?". -- http://mail.python.org/mailman/listinfo/python-list
