Dear all
I ran into a problem when trying to create overviews for large
LZW-compressed GeoTIFF images using gdaladdo. The TIFF's are internally
tiled, the file type is Byte. GDAL versions tested are 2.1.1 and 1.11,
running on 64bit Linux.
The first try was using an input file of size 1038336 x 565248 pixels.
Setting CPL_DEBUG=ON the processing goes up to the message
...10...20...30...40...50...60...70..GDAL: Potential thrashing on band 1
of tileindex_nodsm_otsu_2
and then it remains at this "70.." % forever, with no further log
output. This happened for all files of similar size, on multiple
machines, on both GDAL versions. Always hangs at this "70.." state, the
process remains alive but nothing is written any more to the output
file. I used the command like
gdaladdo -ro --config COMPRESS_OVERVIEW LZW INPUT_FILE 2 4 8 16 32 64
Trying to specify BIGTIFF or not for the overview did not make any
difference.
Using a smaller file or converting the large input file to 50% size and
running the gdaladdo on this smaller file finished successfully, but
also here I got some warnings of a similar kind, and the process got
stuck for a while after the ".70..GDAL: Potential thrashing..." warning,
but then continued to the end. The full log is attached.
An additional observation I made was that the overview file stopping at
70% was already bigger than the input file.
I was thinking to create single lower resolution files and merging them
to an overview file, but there's no built-in functionality for this, so
did not follow it up further.
Could there be a sort of limitation of maximum pixels x*y, or with the
LZW compression? Any ideas how this could be solved?
Thanks in advance for any hint
Armin
GDAL: GDALOpen(/scratch/mosaic_20161219/tileindex_esm_otsu_2.tif, this=0xc36800) succeeds as GTiff.
GTiff: File open for read-only accessing, creating overviews externally.
GDAL: GDALDefaultOverviews::OverviewScan()
GDAL: GDALOpen(/scratch/mosaic_20161219/tileindex_esm_otsu_2.tif.ovr, this=0xc485f0) succeeds as GTiff.
GTiff: ScanDirectories()
GTiff: Opened 55296x38400 overview.
GTiff: Opened 27648x19200 overview.
GTiff: Opened 13824x9600 overview.
GTiff: Opened 6912x4800 overview.
GTiff: Opened 3456x2400 overview.
0GDAL: GDAL_CACHEMAX = 116073 MB
...10...20...30...40...50...60...70..GDAL: Potential thrashing on band 1 of /scratch/mosaic_20161219/tileindex_esm_otsu_2
.tif.ovr.
.80...90.GDAL: Potential thrashing on band 1 of .
..GDAL: Potential thrashing on band 1 of .
GDAL: Potential thrashing on band 1 of .
GDAL: Potential thrashing on band 1 of .
GDAL: 259200 block reads on 129600 block band 1 of .
GDAL: 64800 block reads on 32400 block band 1 of .
GDAL: 16200 block reads on 8100 block band 1 of .
GDAL: 4104 block reads on 2052 block band 1 of .
GDAL: GDALClose(/scratch/mosaic_20161219/tileindex_esm_otsu_2.tif.ovr, this=0xc485f0)
GDAL: 1036800 block reads on 518400 block band 1 of /scratch/mosaic_20161219/tileindex_esm_otsu_2.tif.ovr.
GDAL: GDALOpen(/scratch/mosaic_20161219/tileindex_esm_otsu_2.tif.ovr, this=0xc48470) succeeds as GTiff.
GTiff: ScanDirectories()
GTiff: Opened 55296x38400 overview.
GTiff: Opened 27648x19200 overview.
GTiff: Opened 13824x9600 overview.
GTiff: Opened 6912x4800 overview.
GTiff: Opened 3456x2400 overview.
GDAL: GDALClose(/scratch/mosaic_20161219/tileindex_esm_otsu_2.tif, this=0xc36800)
GDAL: GDALClose(/scratch/mosaic_20161219/tileindex_esm_otsu_2.tif.ovr, this=0xc48470)
GDAL: In GDALDestroy - unloading GDAL shared library.
100 - done. _______________________________________________
gdal-dev mailing list
[email protected]
http://lists.osgeo.org/mailman/listinfo/gdal-dev
