Tim,
wil be solved (in 3.11 only) per https://github.com/OSGeo/gdal/pull/11915
In the meantime, check for gdal.Warp() return value: if None, an error
occurred
Even
Le 01/03/2025 à 01:00, Tim Harris via gdal-dev a écrit :
Hi, I'm having some trouble warping a VRT to a TIF where the VRT and
its constituent rasters are in S3. It stems from a random read error,
but the real problem is that the gdal.Warp call didn't throw an
exception when it was supposed to.
I saw there's this pretty old GitHub issue that may be related, but
I'm not sure:
https://github.com/OSGeo/gdal/issues/3372
In my situation what really happened was a random /vsicurl read
failed... sort of like an issue I reported in December that was fixed
in GDAL 3.10.1:
https://github.com/OSGeo/gdal/issues/11552
Here's a section of my log file (filename sanitized):
ERROR 11: CURL error: Empty reply from server
0...10...20...30...40...50...60...70...80...90...100 - done.
ERROR 4: `/vsis3/path/to/raster.tif' does not exist in the file
system, and is not recognized as a supported dataset name.
Notice the CURL error but the progress still goes to 100%. The warp
completed successfully despite the read error.
I have a hopefully simple reproduce case here. The first few shell
commands will make two TIFs, put them in S3, then make a VRT of those
TIFs and also put that in S3. Then to simulate the CURL error above,
it just deletes one of the TIFs so that the VRT references a
non-existent file:
gdal_create -of GTiff -outsize 1000 1000 -bands 1 -a_srs EPSG:4326
-a_ullr 0 1 1 0 input1.tif
gdal_create -of GTiff -outsize 1000 1000 -bands 1 -a_srs EPSG:4326
-a_ullr 1 1 2 0 input2.tif
aws s3 cp input1.tif s3://your-bucket/test/input1.tif
aws s3 cp input2.tif s3://your-bucket/test/input2.tif
gdalbuildvrt input.vrt /vsis3/your-bucket/test/input1.tif
/vsis3/your-bucket/test/input2.tif
aws s3 cp input.vrt s3://your-bucket/test/input.vrt
aws s3 rm s3://your-bucket/test/input2.tif
Then in Python:
from osgeo import gdal
gdal.UseExceptions()
gdal.Warp("output.tif", "/vsis3/tharris-bucket/test/input.vrt",
callback=gdal.TermProgress)
This throws a RuntimeError, as expected. But delete the partial file
and try with multithread=True:
rm output.tif
gdal.Warp("output.tif", "/vsis3/tharris-bucket/test/input.vrt",
callback=gdal.TermProgress, multithread=True)
This logs the error, but doesn't throw an exception. The end result is
that I have a TIF that is missing the contents of the file that
suffered from that CURL read error.
For what it's worth this also happens with the command line utility,
it's just not as obvious. Without -multi, it only prints the "0" for
the progress before the error halts execution:
gdalwarp /vsis3/your-bucket/test/input.vrt output.tif
Processing /vsis3/your-bucket/test/input.vrt [1/1] : 0ERROR 4:
`/vsis3/your-bucket/test/input2.tif' does not exist in the file
system, and is not recognized as a supported dataset name.
But with -multi, notice the progress goes to 100%:
gdalwarp /vsis3/your-bucket/test/input.vrt output.tif -multi
Processing /vsis3/your-bucket/test/input.vrt [1/1] : 0ERROR 4:
`/vsis3/your-bucket/test/input2.tif' does not exist in the file
system, and is not recognized as a supported dataset name.
...10...20...30...40...50...60...70...80...90...100 - done.
I guess my workaround is to just not use multithread=True. But is this
something that could be fixed?
Thanks
_______________________________________________
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev
--
http://www.spatialys.com
My software is free, but my time generally not.
_______________________________________________
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev
