Hi, I am the developer of the gdal_retile.py
I think the problem here is based on the following issue.
To make a pyramid with tiles in the Dimension width * hight based and
already tiled image with the same tile width and height, I have to
load the base tile and all 8 neighbors touching the tile. As you
stated, your tiles are on a remote file server. This must result in
heavy network traffic.
Open gdal_retily.py with an editor and search for "class
DataSetCache". You see a line
self.cacheSize=8
which means the the cache is holding 8 tiles. Try to increase this
number. A good number would be
(pixel width of your first pyramid / tile width ) * 3 + 8
I hope you get no memory problems
Quoting acangi <[email protected]>:
Hello,
It looks like gdal_retile.py slows down when processing a new level of the
pyramid.
Source files : 8065 geotiffs orthophotos of 4000 x 4000 pixels, in total 400
GB.
Command : gdal_retile.py -targetDir pyramid/ -levels 12 -v -s_srs
/mnt/webgisdata/geoserver/config/3812.prj -ps 512 512 -r bilinear -co
"COMPRESS=JPEG" -co "INTERLEAVE=PIXEL" -tileIndex orthocol -tileIndexField
location /opt/geodata/ortholinks/*.tif
Level 0 was processed in 6 days (500000 files, 1 file is processed in 1
second, total 64 GB).
Level 1 starts processing : 1 file is processed in 5 minutes. As it'd take
me a year to finish the level, I stopped it.
Restarted with the same command and -pyramidOnly parameter
Level 1 was processed in 6 days (125000 files, 1 file is processed in 4
seconds, total 17 GB).
Level 2 starts processing : 1 file is processed in 3 minutes. It would take
2 monthes to finish this level.
I'll wait a little more to see the evolution and then stop the process
again, and restart it based on the level 1 files.
Any idea why it slows down like this ?
The source files and the output are on a CIFS fileserver, the computer
processing gdal_retile is a 4 CPU, 2.5 GB RAM. top says :
top - 10:12:57 up 7 days, 23:37, 1 user, load average: 1.00, 1.06, 1.09
Tasks: 1 total, 0 running, 1 sleeping, 0 stopped, 0 zombie
Cpu(s): 1.2%us, 0.0%sy, 0.0%ni, 98.2%id, 0.0%wa, 0.0%hi, 0.0%si,
0.7%st
Mem: 2560220k total, 2506852k used, 53368k free, 21316k buffers
Swap: 2097144k total, 5496k used, 2091648k free, 949224k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4913 ctiadmin 20 0 243m 163m 8064 D 6 6.5 283:36.06 python
Any idea appreciated,
Alain
--
View this message in context:
http://osgeo-org.1803224.n2.nabble.com/GDAL-retile-slowdown-at-next-level-tp5034870p5034870.html
Sent from the GDAL - Dev mailing list archive at Nabble.com.
_______________________________________________
gdal-dev mailing list
[email protected]
http://lists.osgeo.org/mailman/listinfo/gdal-dev
----------------------------------------------------------------
This message was sent using IMP, the Internet Messaging Program.
_______________________________________________
gdal-dev mailing list
[email protected]
http://lists.osgeo.org/mailman/listinfo/gdal-dev
