Hi Abdul, Can you bump https://gdal.org/en/stable/user/configoptions.html#config-GDAL_MAX_DATASET_POOL_SIZE and see if it helps? And do all inputs have the same block size of 1048x7? Ideally, your inputs and output will all have the same block size or a multiple of it, so you could also add -co BLOCKXSIZE=1048 -co BLOCKYSIZE=7.
Note that these should apply just as well when you don't have a pixel function, so it's probably not it. https://github.com/OSGeo/gdal/pull/11850 might also help in the future, but you'll need to change your workflow for it. Laurentiu On Mon, Apr 14, 2025, at 07:52, Abdul Raheem Siddiqui via gdal-dev wrote: > Dear GDAL Community, > > I am encountering a performance issue when using a VRT consisting of a large > number of source rasters and built-in C++ pixel function ("max"). I would > appreciate any guidance on whether some GDAL config option can improve this, > or I am doing something wrong, or this is a potential optimization > opportunity. > > I have A VRT file referencing ~750 individual rasters (Byte data type, avg > size ~1000x1000 pixels, untiled, and same CRS for all source rasters). The > VRT uses the built-in “max” pixel function. > > Running gdal_translate to convert the VRT to GTiff takes ~1.5 hours and > consumes ~4GB RAM. > gdal_translate -of GTiff merged.vrt OUTPUT.tif > > When tripling the number of source rasters (to ~2250) by duplicating entries > in the VRT, processing time increases to ~4.5 hours, with RAM usage rising to > ~11.5GB. > gdal_translate -of GTiff merged_3x.vrt OUTPUT.tif > > Extracting a small subset of the VRT via -projwin does not improve > performance (still slow, ~14GB RAM used). > gdal_translate -projwin -146955.241 797044.4497 -138766.1444 789656.0648 -of > GTiff merged_3x.vrt OUTPUT.tif > > *Removing the pixel function makes processing instantaneous, even with 2250 > rasters.* > > Performance is unaffected by --config GDAL_CACHEMAX or -co > NUM_THREADS=ALL_CPUS. > > The issue persists across other datasets that I have. In fact, it gets really > worse when source rasters are relatively larger or of Float32 data type. > > Gdalinfo on one of the source rasters: https://pastebin.com/gjHDA2Wd > *Data:* > https://drive.google.com/file/d/1LGlzGGZvkPyXvKKgkPVzQGbBw55p5dRd/view? > > System: Windows, 8 CPUs, 32GB RAM (GDAL 3.9.2 via OSGeo4W). > > Thank you for your time and insights. Please reply if you are aware of how > performance can be improved. > > Regards, > *Abdul Siddiqui, PE* > _abdul.siddiqui@ertcorp.com_ > > *ERT | *Earth Resources Technology, Inc. > 14401 Sweitzer Ln. Ste 300 > Laurel, MD 20707 > https://www.ertcorp.com > > > _______________________________________________ > gdal-dev mailing list > gdal-dev@lists.osgeo.org > https://lists.osgeo.org/mailman/listinfo/gdal-dev >
_______________________________________________ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
