Aha, yes, it does appear to happen just with local files. In my attempts to
create a reproduce case yesterday, I overlooked that part. I did try it
early on but just with a single TIF in the VRT, in which case it does the
right thing and raises an exception. But with two TIFs and one missing, it
behaves as I explained with /vsis3.
So create two TIFs and a VRT, then remove one TIF:
gdal_create -of GTiff -outsize 1000 1000 -bands 1 -a_srs EPSG:4326 -a_ullr
0 1 1 0 input1.tif
gdal_create -of GTiff -outsize 1000 1000 -bands 1 -a_srs EPSG:4326 -a_ullr
1 1 2 0 input2.tif
gdalbuildvrt input.vrt input1.tif input2.tif
rm input2.tif
Then in Python:
from osgeo import gdal
gdal.UseExceptions()
This raises an exception
gdal.Warp("output.tif", "input.vrt", callback=gdal.TermProgress)
But this does not:
gdal.Warp("output.tif", "input.vrt", callback=gdal.TermProgress,
multithread=True)
On Sat, Mar 1, 2025 at 7:58 AM Javier Jimenez Shaw <j...@jimenezshaw.com>
wrote:
> Does it also happen without S3, just with local files?
> It would simplify the analysis
>
> On Sat, 1 Mar 2025, 01:01 Tim Harris via gdal-dev, <
> gdal-dev@lists.osgeo.org> wrote:
>
>> Hi, I'm having some trouble warping a VRT to a TIF where the VRT and its
>> constituent rasters are in S3. It stems from a random read error, but the
>> real problem is that the gdal.Warp call didn't throw an exception when it
>> was supposed to.
>>
>> I saw there's this pretty old GitHub issue that may be related, but I'm
>> not sure:
>> https://github.com/OSGeo/gdal/issues/3372
>>
>> In my situation what really happened was a random /vsicurl read failed...
>> sort of like an issue I reported in December that was fixed in GDAL 3.10.1:
>> https://github.com/OSGeo/gdal/issues/11552
>>
>> Here's a section of my log file (filename sanitized):
>> ERROR 11: CURL error: Empty reply from server
>> 0...10...20...30...40...50...60...70...80...90...100 - done.
>> ERROR 4: `/vsis3/path/to/raster.tif' does not exist in the file system,
>> and is not recognized as a supported dataset name.
>>
>> Notice the CURL error but the progress still goes to 100%. The warp
>> completed successfully despite the read error.
>>
>> I have a hopefully simple reproduce case here. The first few shell
>> commands will make two TIFs, put them in S3, then make a VRT of those TIFs
>> and also put that in S3. Then to simulate the CURL error above, it just
>> deletes one of the TIFs so that the VRT references a non-existent file:
>> gdal_create -of GTiff -outsize 1000 1000 -bands 1 -a_srs EPSG:4326
>> -a_ullr 0 1 1 0 input1.tif
>> gdal_create -of GTiff -outsize 1000 1000 -bands 1 -a_srs EPSG:4326
>> -a_ullr 1 1 2 0 input2.tif
>> aws s3 cp input1.tif s3://your-bucket/test/input1.tif
>> aws s3 cp input2.tif s3://your-bucket/test/input2.tif
>> gdalbuildvrt input.vrt /vsis3/your-bucket/test/input1.tif
>> /vsis3/your-bucket/test/input2.tif
>> aws s3 cp input.vrt s3://your-bucket/test/input.vrt
>> aws s3 rm s3://your-bucket/test/input2.tif
>>
>> Then in Python:
>> from osgeo import gdal
>> gdal.UseExceptions()
>> gdal.Warp("output.tif", "/vsis3/tharris-bucket/test/input.vrt",
>> callback=gdal.TermProgress)
>>
>> This throws a RuntimeError, as expected. But delete the partial file and
>> try with multithread=True:
>> rm output.tif
>> gdal.Warp("output.tif", "/vsis3/tharris-bucket/test/input.vrt",
>> callback=gdal.TermProgress, multithread=True)
>>
>> This logs the error, but doesn't throw an exception. The end result is
>> that I have a TIF that is missing the contents of the file that suffered
>> from that CURL read error.
>>
>> For what it's worth this also happens with the command line utility, it's
>> just not as obvious. Without -multi, it only prints the "0" for the
>> progress before the error halts execution:
>> gdalwarp /vsis3/your-bucket/test/input.vrt output.tif
>> Processing /vsis3/your-bucket/test/input.vrt [1/1] : 0ERROR 4:
>> `/vsis3/your-bucket/test/input2.tif' does not exist in the file system, and
>> is not recognized as a supported dataset name.
>>
>> But with -multi, notice the progress goes to 100%:
>> gdalwarp /vsis3/your-bucket/test/input.vrt output.tif -multi
>> Processing /vsis3/your-bucket/test/input.vrt [1/1] : 0ERROR 4:
>> `/vsis3/your-bucket/test/input2.tif' does not exist in the file system, and
>> is not recognized as a supported dataset name.
>> ...10...20...30...40...50...60...70...80...90...100 - done.
>>
>> I guess my workaround is to just not use multithread=True. But is this
>> something that could be fixed?
>>
>> Thanks
>> _______________________________________________
>> gdal-dev mailing list
>> gdal-dev@lists.osgeo.org
>> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>>
>
_______________________________________________
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev
