Chris, ok I reproduce your issue.
The GRIB driver actually caches all the raster data from a band the first type you access it, and never releases it. This is to speed-up successive RasterIO operations on a band, which is a nice feature generally. But if you iterate over all the bands, it means that GDAL will end up allocating (number_of_bands * x_size * y_size * sizeof(double) ) bytes. In your case : 1129 * 720 * 360 * 8 = 2.3 GB indeed. I'm going to try to find a fix where GDAL wouldn't cache more than XXX bytes from a dataset to avoid this situation. In the meantime, you can perhaps try reworking your algorithm to iterate on a limited number of bands (let's say 100) at a time. "At a time" means that between each iteration you close and re-open the dataset. (GDAL will recover nicely the memory it has cached at dataset closing). Or more simply close and re-open each time you process a band (opening time on your dataset doesn't seem to be so slow). Best regards, Even Le jeudi 24 mai 2012 21:55:15, Chris Barker a écrit : > Hi folks, > > I"m finding what appears to be a memory leak, using the GRIB reader, > with the python bindings. > > What I'm trying to do is read the data one band at a time, then throw > it away and read the next band -- there are 1129 bands in the file at > hand, and I can't hold it all in memory (32 bit still...) > > However, when I do this, memory use just keeps climbing. > > Should the memory be freed? I would expect so. > > I'm using RasterBand.ReadAsArray() > > Is this a leak? or is supposed to keep it around in memory? > > Either way, is there a way to force it to release that memory (I"m > already doing and exlicite del and gc.collect call, so I dont think > it's a python reference counting issue) > > I've enclosed a simpel test script -- watch the memory climb. > > The data file is to big (186MB) to enclose here, you can get it here: > > http://nomads.ncep.noaa.gov/pub/data/nccf/com/cfs/prod/cfs/cfs.20120522/18/ > time_grib_01/ocnu5.01.2012052218.daily.grb2 > > If you want to give this a try. > > > (note -- Grib giving some pretty good compression -- this climbs to > 2.3GB when I read it) > > I could actually live with the 2.3GB -- but in my real use case, I'm > reading two of these at the same time, so I max out what I can do with > 32 bit python... > > GDAL 1.8.1 > Python 2.7 (32 bit Intel) > OS-X 10.6 > I think it's the Kyng Chaos build of GDAL. > > Thanks, > -Chris _______________________________________________ gdal-dev mailing list [email protected] http://lists.osgeo.org/mailman/listinfo/gdal-dev
