Thanks Even! On Tue, Mar 4, 2025 at 7:51 AM Even Rouault <even.roua...@spatialys.com> wrote:
> Tim, > > wil be solved (in 3.11 only) per https://github.com/OSGeo/gdal/pull/11915 > > In the meantime, check for gdal.Warp() return value: if None, an error > occurred > > Even > Le 01/03/2025 à 01:00, Tim Harris via gdal-dev a écrit : > > Hi, I'm having some trouble warping a VRT to a TIF where the VRT and its > constituent rasters are in S3. It stems from a random read error, but the > real problem is that the gdal.Warp call didn't throw an exception when it > was supposed to. > > I saw there's this pretty old GitHub issue that may be related, but I'm > not sure: > https://github.com/OSGeo/gdal/issues/3372 > > In my situation what really happened was a random /vsicurl read failed... > sort of like an issue I reported in December that was fixed in GDAL 3.10.1: > https://github.com/OSGeo/gdal/issues/11552 > > Here's a section of my log file (filename sanitized): > ERROR 11: CURL error: Empty reply from server > 0...10...20...30...40...50...60...70...80...90...100 - done. > ERROR 4: `/vsis3/path/to/raster.tif' does not exist in the file system, > and is not recognized as a supported dataset name. > > Notice the CURL error but the progress still goes to 100%. The warp > completed successfully despite the read error. > > I have a hopefully simple reproduce case here. The first few shell > commands will make two TIFs, put them in S3, then make a VRT of those TIFs > and also put that in S3. Then to simulate the CURL error above, it just > deletes one of the TIFs so that the VRT references a non-existent file: > gdal_create -of GTiff -outsize 1000 1000 -bands 1 -a_srs EPSG:4326 -a_ullr > 0 1 1 0 input1.tif > gdal_create -of GTiff -outsize 1000 1000 -bands 1 -a_srs EPSG:4326 -a_ullr > 1 1 2 0 input2.tif > aws s3 cp input1.tif s3://your-bucket/test/input1.tif > aws s3 cp input2.tif s3://your-bucket/test/input2.tif > gdalbuildvrt input.vrt /vsis3/your-bucket/test/input1.tif > /vsis3/your-bucket/test/input2.tif > aws s3 cp input.vrt s3://your-bucket/test/input.vrt > aws s3 rm s3://your-bucket/test/input2.tif > > Then in Python: > from osgeo import gdal > gdal.UseExceptions() > gdal.Warp("output.tif", "/vsis3/tharris-bucket/test/input.vrt", > callback=gdal.TermProgress) > > This throws a RuntimeError, as expected. But delete the partial file and > try with multithread=True: > rm output.tif > gdal.Warp("output.tif", "/vsis3/tharris-bucket/test/input.vrt", > callback=gdal.TermProgress, multithread=True) > > This logs the error, but doesn't throw an exception. The end result is > that I have a TIF that is missing the contents of the file that suffered > from that CURL read error. > > For what it's worth this also happens with the command line utility, it's > just not as obvious. Without -multi, it only prints the "0" for the > progress before the error halts execution: > gdalwarp /vsis3/your-bucket/test/input.vrt output.tif > Processing /vsis3/your-bucket/test/input.vrt [1/1] : 0ERROR 4: > `/vsis3/your-bucket/test/input2.tif' does not exist in the file system, and > is not recognized as a supported dataset name. > > But with -multi, notice the progress goes to 100%: > gdalwarp /vsis3/your-bucket/test/input.vrt output.tif -multi > Processing /vsis3/your-bucket/test/input.vrt [1/1] : 0ERROR 4: > `/vsis3/your-bucket/test/input2.tif' does not exist in the file system, and > is not recognized as a supported dataset name. > ...10...20...30...40...50...60...70...80...90...100 - done. > > I guess my workaround is to just not use multithread=True. But is this > something that could be fixed? > > Thanks > > _______________________________________________ > gdal-dev mailing > listgdal-dev@lists.osgeo.orghttps://lists.osgeo.org/mailman/listinfo/gdal-dev > > -- http://www.spatialys.com > My software is free, but my time generally not. > >
_______________________________________________ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
