rkl wrote:
I might be a little late with my comment here.
David Beazley in his PyCon'2008 presentation "Generator Tricks
For Systems Programmers" had this very elegant example of handling an
unlimited numbers of files:
David Beazley's generator stuff is definitely worth recommending
on. I thi
On Jun 15, 2:35 am, tom wrote:
> i can traverse adirectoryusing os.listdir() or os.walk(). but if
> adirectoryhas a very large number of files, these methods produce very
> large objects talking a lot of memory.
>
> in other languages one can avoid generating such an object by walking
> adirector
On Jun 15, 6:35 am, Andre Engels wrote:
> What kind of directories are those that just a list of files would
> result in a "very large" object? I don't think I have ever seen
> directories with more than a few thousand files...
(a...@lucrezia:~/pit/lsa/act:5)$ ls -1 | wc -l
142607
There, you'
Nick Craig-Wood wrote:
> Jean-Paul Calderone wrote:
> > On Mon, 15 Jun 2009 09:29:33 -0500, Nick Craig-Wood
> > wrote:
> > >Hrvoje Niksic wrote:
> > >> Nick Craig-Wood writes:
> > >>
> > >> > Here is a ctypes generator listdir for unix-like OSes.
> > >>
> > >> ctypes code scares me with i
On Jun 15, 6:56 am, Steven D'Aprano
wrote:
> On Sun, 14 Jun 2009 22:35:50 +0200, Andre Engels wrote:
> > On Sun, Jun 14, 2009 at 6:35 PM, tom wrote:
> >> i can traverse a directory using os.listdir() or os.walk(). but if a
> >> directory has a very large number of files, these methods produce very
Nick Craig-Wood writes:
> It can be done properly with gccxml though which converts structures
> into ctypes definitions.
That sounds interesting.
> That said the dirent struct is specified by POSIX so if you get the
> correct types for all the individual members then it should be
> correct eve
On Mon, 15 Jun 2009 15:35:04 -0400
Terry Reedy wrote:
> Christian Heimes wrote:
> > Terry Reedy wrote:
> >> You did not specify version. In Python3, os.walk has become a generater
> >> function. So, to answer your question, use 3.1.
> >
> > I'm sorry to inform you that Python 3.x still returns
Jean-Paul Calderone wrote:
> On Mon, 15 Jun 2009 09:29:33 -0500, Nick Craig-Wood
> wrote:
> >Hrvoje Niksic wrote:
> >> Nick Craig-Wood writes:
> >>
> >> > Here is a ctypes generator listdir for unix-like OSes.
> >>
> >> ctypes code scares me with its duplication of the contents of system
>
Christian Heimes wrote:
Terry Reedy wrote:
You did not specify version. In Python3, os.walk has become a generater
function. So, to answer your question, use 3.1.
I'm sorry to inform you that Python 3.x still returns a list, not a
generator.
>>> type(os.walk('.'))
However, it is a genera
On Mon, 15 Jun 2009 09:29:33 -0500, Nick Craig-Wood wrote:
Hrvoje Niksic wrote:
Nick Craig-Wood writes:
> Here is a ctypes generator listdir for unix-like OSes.
ctypes code scares me with its duplication of the contents of system
headers. I understand its use as a proof of concept, or f
Hrvoje Niksic wrote:
> Nick Craig-Wood writes:
>
> > Here is a ctypes generator listdir for unix-like OSes.
>
> ctypes code scares me with its duplication of the contents of system
> headers. I understand its use as a proof of concept, or for hacks one
> needs right now, but can anyone ser
tom wrote:
> i can traverse a directory using os.listdir() or os.walk(). but if a
> directory has a very large number of files, these methods produce very
> large objects talking a lot of memory.
if we assume the number of files to be a million (which certainly qualifies
as one of the larger dire
Nick Craig-Wood writes:
> Here is a ctypes generator listdir for unix-like OSes.
ctypes code scares me with its duplication of the contents of system
headers. I understand its use as a proof of concept, or for hacks one
needs right now, but can anyone seriously propose using this kind of
code i
Terry Reedy writes:
> You did not specify version. In Python3, os.walk has become a
> generater function. So, to answer your question, use 3.1.
os.walk has been a generator function all along, but that doesn't help
OP because it still uses os.listdir internally. This means that it
both create
tom wrote:
> On Jun 14, 1:35 pm, Tim Golden wrote:
> >
> > If you're on Windows, you can use the win32file.FindFilesIterator
> > function from the pywin32 package. (Which wraps the Win32 API
> > FindFirstFile / FindNextFile pattern).
>
> thanks, tim.
>
> however, i'm not using windows. freeb
On Sun, 14 Jun 2009 22:35:50 +0200, Andre Engels wrote:
> On Sun, Jun 14, 2009 at 6:35 PM, tom wrote:
>> i can traverse a directory using os.listdir() or os.walk(). but if a
>> directory has a very large number of files, these methods produce very
>> large objects talking a lot of memory.
>>
>> in
i can traverse a directory using os.listdir() or os.walk(). but if a
directory has a very large number of files, these methods produce very
large objects talking a lot of memory.
in other languages one can avoid generating such an object by walking
a directory as a liked list. for example, in c,
Terry Reedy wrote:
> You did not specify version. In Python3, os.walk has become a generater
> function. So, to answer your question, use 3.1.
I'm sorry to inform you that Python 3.x still returns a list, not a
generator.
ython 3.1rc1+ (py3k:73396, Jun 12 2009, 22:45:18)
[GCC 4.3.3] on linux2
Christian Heimes wrote:
tom schrieb:
i can traverse a directory using os.listdir() or os.walk(). but if a
directory has a very large number of files, these methods produce very
large objects talking a lot of memory.
in other languages one can avoid generating such an object by walking
a directo
Andre Engels wrote:
> What kind of directories are those that just a list of files would
> result in a "very large" object? I don't think I have ever seen
> directories with more than a few thousand files...
I've seen directories with several hundreds of thousand files. Depending
on the file syste
tom wrote:
i can traverse a directory using os.listdir() or os.walk(). but if a
directory has a very large number of files, these methods produce very
large objects talking a lot of memory.
in other languages one can avoid generating such an object by walking
a directory as a liked list. for exa
tom schrieb:
> i can traverse a directory using os.listdir() or os.walk(). but if a
> directory has a very large number of files, these methods produce very
> large objects talking a lot of memory.
>
> in other languages one can avoid generating such an object by walking
> a directory as a liked l
On Sun, Jun 14, 2009 at 6:35 PM, tom wrote:
> i can traverse a directory using os.listdir() or os.walk(). but if a
> directory has a very large number of files, these methods produce very
> large objects talking a lot of memory.
>
> in other languages one can avoid generating such an object by walk
tom wrote:
On Jun 14, 1:35 pm, Tim Golden wrote:
If you're on Windows, you can use the win32file.FindFilesIterator
function from the pywin32 package. (Which wraps the Win32 API
FindFirstFile / FindNextFile pattern).
thanks, tim.
however, i'm not using windows. freebsd and os x.
Presumably,
On Jun 14, 1:35 pm, Tim Golden wrote:
>
> If you're on Windows, you can use the win32file.FindFilesIterator
> function from the pywin32 package. (Which wraps the Win32 API
> FindFirstFile / FindNextFile pattern).
thanks, tim.
however, i'm not using windows. freebsd and os x.
--
http://mail.pyth
tom wrote:
i can traverse a directory using os.listdir() or os.walk(). but if a
directory has a very large number of files, these methods produce very
large objects talking a lot of memory.
in other languages one can avoid generating such an object by walking
a directory as a liked list. for exa
26 matches
Mail list logo
