Re: Iterating over files of a huge directory

2012-12-17 Thread Terry Reedy
On 12/17/2012 10:28 AM, Gilles Lenfant wrote: Hi, I have googled but did not find an efficient solution to my problem. My customer provides a directory with a hge list of files (flat, potentially 10+) and I cannot reasonably use os.listdir(this_path) unless creating a big memory footprin

Re: Iterating over files of a huge directory

2012-12-17 Thread Chris Angelico
On Tue, Dec 18, 2012 at 5:29 AM, MRAB wrote: > > Years ago I had to deal with an in-house application that was written > using a certain database package. The package stored each predefined > query in a separate file in the same directory. > > I found that if I packed all the predefined queries i

Re: Iterating over files of a huge directory

2012-12-17 Thread Evan Driscoll
On 12/17/2012 01:50 PM, Oscar Benjamin wrote: > On 17 December 2012 18:40, Evan Driscoll wrote: >> On 12/17/2012 09:52 AM, Oscar Benjamin wrote: >>> https://github.com/benhoyt/betterwalk >> >> This is very useful to know about; thanks. >> >> I actually wrote something very similar on my own (I wan

Re: Re: Iterating over files of a huge directory

2012-12-17 Thread Oscar Benjamin
On 17 December 2012 18:40, Evan Driscoll wrote: > On 12/17/2012 09:52 AM, Oscar Benjamin wrote: >> https://github.com/benhoyt/betterwalk > > This is very useful to know about; thanks. > > I actually wrote something very similar on my own (I wanted to get > information about whether each directory

Re: Re: Iterating over files of a huge directory

2012-12-17 Thread Evan Driscoll
On 12/17/2012 09:52 AM, Oscar Benjamin wrote: > In the last couple of months there has been a lot of discussion (on > python-list or python-dev - not sure) about creating a library to more > efficiently iterate over the files in a directory. The result so far > is this library on github: > https://

Re: Iterating over files of a huge directory

2012-12-17 Thread MRAB
On 2012-12-17 17:27, Paul Rudin wrote: Chris Angelico writes: On Tue, Dec 18, 2012 at 2:28 AM, Gilles Lenfant wrote: Hi, I have googled but did not find an efficient solution to my problem. My customer provides a directory with a hge list of files (flat, potentially 10+) and I canno

Re: Iterating over files of a huge directory

2012-12-17 Thread Paul Rudin
Chris Angelico writes: > On Tue, Dec 18, 2012 at 2:28 AM, Gilles Lenfant > wrote: >> Hi, >> >> I have googled but did not find an efficient solution to my >> problem. My customer provides a directory with a hge list of >> files (flat, potentially 10+) and I cannot reasonably use >> os.li

Re: Iterating over files of a huge directory

2012-12-17 Thread Gilles Lenfant
Le lundi 17 décembre 2012 16:52:19 UTC+1, Oscar Benjamin a écrit : > On 17 December 2012 15:28, Gilles Lenfant <...> wrote: > > > In the last couple of months there has been a lot of discussion (on > > python-list or python-dev - not sure) about creating a library to more > > efficiently iterat

Re: Iterating over files of a huge directory

2012-12-17 Thread marduk
On Mon, Dec 17, 2012, at 10:28 AM, Gilles Lenfant wrote: > Hi, > > I have googled but did not find an efficient solution to my problem. My > customer provides a directory with a hge list of files (flat, > potentially 10+) and I cannot reasonably use os.listdir(this_path) > unless creatin

Re: Iterating over files of a huge directory

2012-12-17 Thread Oscar Benjamin
On 17 December 2012 15:28, Gilles Lenfant wrote: > I have googled but did not find an efficient solution to my problem. My > customer provides a directory with a hge list of files (flat, potentially > 10+) and I cannot reasonably use os.listdir(this_path) unless creating a > big memory

Re: Iterating over files of a huge directory

2012-12-17 Thread Tim Golden
On 17/12/2012 15:41, Chris Angelico wrote: > On Tue, Dec 18, 2012 at 2:28 AM, Gilles Lenfant > wrote: >> Hi, >> >> I have googled but did not find an efficient solution to my >> problem. My customer provides a directory with a hge list of >> files (flat, potentially 10+) and I cannot rea

Re: Iterating over files of a huge directory

2012-12-17 Thread Chris Angelico
On Tue, Dec 18, 2012 at 2:28 AM, Gilles Lenfant wrote: > Hi, > > I have googled but did not find an efficient solution to my problem. My > customer provides a directory with a hge list of files (flat, potentially > 10+) and I cannot reasonably use os.listdir(this_path) unless creating a

Iterating over files of a huge directory

2012-12-17 Thread Gilles Lenfant
Hi, I have googled but did not find an efficient solution to my problem. My customer provides a directory with a hge list of files (flat, potentially 10+) and I cannot reasonably use os.listdir(this_path) unless creating a big memory footprint. So I'm looking for an iterator that yield