Re: Yet another INDEX builder.

2006-10-02 Thread Vasil Dimov
On Mon, Oct 02, 2006 at 04:21:32PM +0200, Michel Talon wrote: [...] > > Thanks a lot for this very interesting measurement. I note that using > parallel makes takes 18 minutes at best and 20 minutes at worst, while > build_index takes 26 minutes. I suppose parallel make runs faster because > more

Re: Yet another INDEX builder.

2006-10-02 Thread Vasil Dimov
On Sat, Sep 30, 2006 at 02:46:36PM -0400, Kris Kennaway wrote: [...] > > The only situation where you might reasonably expect to see something > approaching a factor of 2 speed improvement is if your ports tree is > on a striped or mirrored device on separate controllers, so 2 reads > can be compl

Re: Yet another INDEX builder.

2006-09-30 Thread Kris Kennaway
On Sat, Sep 30, 2006 at 03:49:51PM +0200, Michel Talon wrote: > Kris Kennaway said: > > > FYI, I'm not sure if the python version is parallelizable, but you do > > get a small benefit from using parallelized 'make index' builds (via > > INDEX_JOBS) on a typical SMP machine. > > My script runs pa

Re: Yet another INDEX builder.

2006-09-30 Thread Thierry Thomas
Le Sam 30 sep 06 à 15:39:13 +0200, Michel Talon <[EMAIL PROTECTED]> écrivait : > Also perhaps my disk is faster > > niobe% diskinfo -t /dev/ad0 > ... > Transfer rates: > outside: 102400 kbytes in 2.092096 sec =48946 kbytes/sec > middle:102400 kbytes in 2.4290

Re: Yet another INDEX builder.

2006-09-30 Thread Michel Talon
Kris Kennaway said: > FYI, I'm not sure if the python version is parallelizable, but you do > get a small benefit from using parallelized 'make index' builds (via > INDEX_JOBS) on a typical SMP machine. My script runs parallel makes, of the order of ten simultaneous makes. You can see them, runn

Re: Yet another INDEX builder.

2006-09-30 Thread Michel Talon
> It took longer on my P4 3.6GHz: > > The tree (15689ports) takes 2789.84287214 seconds to explore. > Now computing the recursively extended dependencies. > Took 3.02815794945 seconds. > Now converting to packages and sorting. > Last phase takes 4.7659368515 seconds. > Total time spent: 2800.9

Re: Yet another INDEX builder.

2006-09-30 Thread Matthew Seaman
Michel Talon wrote: > Hello, > > based on the documentation in > http://www.lpthe.jussieu.fr/~talon/freebsdports.html Why, thank you *so* much for not even mentioning my name and giving me any credit at all for writing FreeBSD::Portindex, a program that you mention, but state that cannot even be

Re: Yet another INDEX builder.

2006-09-29 Thread Kris Kennaway
On Sat, Sep 30, 2006 at 12:25:19AM +0200, Thierry Thomas wrote: > Le Ven 29 sep 06 ? 19:21:22 +0200, Michel Talon <[EMAIL PROTECTED]> > ?crivait?: > > Hello, > > Hi, > > > i have written yet another index builder in python, which can be found here: >

Re: Yet another INDEX builder.

2006-09-29 Thread Thierry Thomas
Le Ven 29 sep 06 à 19:21:22 +0200, Michel Talon <[EMAIL PROTECTED]> écrivait : > Hello, Hi, > i have written yet another index builder in python, which can be found here: > http://www.lpthe.jussieu.fr/~talon/build_index.py > I think it works relatively well and fast, and ca

Re: Yet another INDEX builder.

2006-09-29 Thread Michel Talon
On Sat, Sep 30, 2006 at 06:14:26AM +1000, Peter Jeremy wrote: > On Fri, 2006-Sep-29 19:21:22 +0200, Michel Talon wrote: > >timings (*) one gets. On my machine, a P4 3Ghz, oldish, it takes 23 minutes > >to > >complete building the INDEX, but the python script doesn't run more than 1 > >minute. Ever

Re: Yet another INDEX builder.

2006-09-29 Thread Eric
Peter Jeremy wrote: > On Fri, 2006-Sep-29 19:21:22 +0200, Michel Talon wrote: >> timings (*) one gets. On my machine, a P4 3Ghz, oldish, it takes 23 minutes >> to >> complete building the INDEX, but the python script doesn't run more than 1 >> minute. Everything else is IO, i think. So one cannot

Re: Yet another INDEX builder.

2006-09-29 Thread Peter Jeremy
On Fri, 2006-Sep-29 19:21:22 +0200, Michel Talon wrote: >timings (*) one gets. On my machine, a P4 3Ghz, oldish, it takes 23 minutes to >complete building the INDEX, but the python script doesn't run more than 1 >minute. Everything else is IO, i think. So one cannot expect any performance >improvem

Yet another INDEX builder.

2006-09-29 Thread Michel Talon
Hello, based on the documentation in http://www.lpthe.jussieu.fr/~talon/freebsdports.html i have written yet another index builder in python, which can be found here: http://www.lpthe.jussieu.fr/~talon/build_index.py I think it works relatively well and fast, and can provide building blocks for