On Tue, Jun 22, 2004 at 03:52:03PM -0400, Madison Kelly wrote:
> What is happening now is that the program does an 'ls' (system call)
> to get a list of the files and directories starting at the root of a
> mounted partition. These are read into an array which perl then
> processes one at a time. the 'ls' value is searched for in the database
> and if it doesn't exist, the values are inserted. If they do exist, they
> are updated (at 1/10th the speed). If the file is in fact a directory
> perl jumps into it and again reads in it's contents into another array
> and processes the one at a time. It will do this until all files or
> directories on the partition have been processed.
So you read the entire filesystem again and again? Sounds like a
horrible idea to me. Have you tried using the mtimes, etc?
> My previous question was performance based, now I just need to get
> the darn thing working again. Like I said, after ~300 seconds perl dies.
Out of memory? If you save your whole filesystem in a Perl array you
are going to consume a lot of memory. This is, of course, not Postgres
related, so I'm not sure why you are asking here.
--
Alvaro Herrera (<alvherre[a]dcc.uchile.cl>)
Jajaja! Solo hablaba en serio!
---------------------------(end of broadcast)---------------------------
TIP 9: the planner will ignore your desire to choose an index scan if your
joining column's datatypes do not match
