On Thu, May 24, 2012 at 2:40 PM, Even Rouault <[email protected]> wrote: > Ok, see http://trac.osgeo.org/gdal/ticket/4682 for a fix. Basically, the > current caching strategy is maintained (cache all bands that have been > accessed), until a threshold is reached (arbitrarly set to 100 MB by default).
seems reasonable -- is there an API to change that threshold at run time? Not I think it's needed... > When the threshold is reached, then we only cache one band at a time. That > could be made smarter, but I think this is good enough for now. it sounds good to me. meanwhile, closing the dataset every once in a while (in my case, every 28 bands read) works great -- topping out at around 180MB memory use. Thanks for the hint and the fix. >> maybe -- but what is GDAL policy usually? It doesn't read the data >> until you ask for it, and I would have expected to keep copy myself if >> want to use it again. > > I'm not a specialist of the GRIB API, but from what I see, it only returns the > data for a full band, and not for partial reads. So for example, if you > accessed one line at a time, and that GDAL didn't do any caching, it would > mean that GDAL would have to decode the whole band each time. Pretty slow ! IIUC, GRIB compresses, so yes, it does make sense to read a whole band at once. > The previous strategy didn't cache all the bands, but each band once you have > you read it once. Obviously, if you read all the bands, then at the end, it > has cached all the bands ;-) The rationale behind this was that it was the > best strategy to minimize the number of lines of codes in the GRIB driver ;-) understandable, but sub-optimum! Thanks, -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception [email protected] _______________________________________________ gdal-dev mailing list [email protected] http://lists.osgeo.org/mailman/listinfo/gdal-dev
