Dear GDAL Community,
I am encountering a performance issue when using a VRT consisting of a
large number of source rasters and built-in C++ pixel function ("max"). I
would appreciate any guidance on whether some GDAL config option can
improve this, or I am doing something wrong, or this is a potential
optimization opportunity.
I have A VRT file referencing ~750 individual rasters (Byte data type, avg
size ~1000x1000 pixels, untiled, and same CRS for all source rasters). The
VRT uses the built-in “max” pixel function.
Running gdal_translate to convert the VRT to GTiff takes ~1.5 hours and
consumes ~4GB RAM.
gdal_translate -of GTiff merged.vrt OUTPUT.tif
When tripling the number of source rasters (to ~2250) by duplicating
entries in the VRT, processing time increases to ~4.5 hours, with RAM usage
rising to ~11.5GB.
gdal_translate -of GTiff merged_3x.vrt OUTPUT.tif
Extracting a small subset of the VRT via -projwin does not improve
performance (still slow, ~14GB RAM used).
gdal_translate -projwin -146955.241 797044.4497 -138766.1444 789656.0648
-of GTiff merged_3x.vrt OUTPUT.tif
*Removing the pixel function makes processing instantaneous, even with 2250
rasters.*
Performance is unaffected by --config GDAL_CACHEMAX or -co
NUM_THREADS=ALL_CPUS.
The issue persists across other datasets that I have. In fact, it gets
really worse when source rasters are relatively larger or of Float32 data
type.
Gdalinfo on one of the source rasters: https://pastebin.com/gjHDA2Wd
*Data:*
https://drive.google.com/file/d/1LGlzGGZvkPyXvKKgkPVzQGbBw55p5dRd/view?
System: Windows, 8 CPUs, 32GB RAM (GDAL 3.9.2 via OSGeo4W).
Thank you for your time and insights. Please reply if you are aware of how
performance can be improved.
Regards,
*Abdul Siddiqui, PE*
*abdul.siddi...@ertcorp.com <abdul.siddi...@ertcorp.com>*
*ERT | *Earth Resources Technology, Inc.
14401 Sweitzer Ln. Ste 300
Laurel, MD 20707
https://www.ertcorp.com
_______________________________________________
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev
