Re: [gdal-dev] Missing exception when using multithread=True for gdal.Warp

[Tim Harris via gdal-dev]

Thanks Even!

On Tue, Mar 4, 2025 at 7:51 AM Even Rouault 
wrote:

> Tim,
>
> wil be solved (in 3.11 only) per https://github.com/OSGeo/gdal/pull/11915
>
> In the meantime, check for gdal.Warp() return value: if None, an error
> occurred
>
> Even
> Le 01/03/2025 à 01:00, Tim Harris via gdal-dev a écrit :
>
> Hi, I'm having some trouble warping a VRT to a TIF where the VRT and its
> constituent rasters are in S3. It stems from a random read error, but the
> real problem is that the gdal.Warp call didn't throw an exception when it
> was supposed to.
>
> I saw there's this pretty old GitHub issue that may be related, but I'm
> not sure:
> https://github.com/OSGeo/gdal/issues/3372
>
> In my situation what really happened was a random /vsicurl read failed...
> sort of like an issue I reported in December that was fixed in GDAL 3.10.1:
> https://github.com/OSGeo/gdal/issues/11552
>
> Here's a section of my log file (filename sanitized):
> ERROR 11: CURL error: Empty reply from server
> 0...10...20...30...40...50...60...70...80...90...100 - done.
> ERROR 4: `/vsis3/path/to/raster.tif' does not exist in the file system,
> and is not recognized as a supported dataset name.
>
> Notice the CURL error but the progress still goes to 100%. The warp
> completed successfully despite the read error.
>
> I have a hopefully simple reproduce case here. The first few shell
> commands will make two TIFs, put them in S3, then make a VRT of those TIFs
> and also put that in S3. Then to simulate the CURL error above, it just
> deletes one of the TIFs so that the VRT references a non-existent file:
> gdal_create -of GTiff -outsize 1000 1000 -bands 1 -a_srs EPSG:4326 -a_ullr
> 0 1 1 0 input1.tif
> gdal_create -of GTiff -outsize 1000 1000 -bands 1 -a_srs EPSG:4326 -a_ullr
> 1 1 2 0 input2.tif
> aws s3 cp input1.tif s3://your-bucket/test/input1.tif
> aws s3 cp input2.tif s3://your-bucket/test/input2.tif
> gdalbuildvrt input.vrt /vsis3/your-bucket/test/input1.tif
> /vsis3/your-bucket/test/input2.tif
> aws s3 cp input.vrt s3://your-bucket/test/input.vrt
> aws s3 rm s3://your-bucket/test/input2.tif
>
> Then in Python:
> from osgeo import gdal
> gdal.UseExceptions()
> gdal.Warp("output.tif", "/vsis3/tharris-bucket/test/input.vrt",
> callback=gdal.TermProgress)
>
> This throws a RuntimeError, as expected. But delete the partial file and
> try with multithread=True:
> rm output.tif
> gdal.Warp("output.tif", "/vsis3/tharris-bucket/test/input.vrt",
> callback=gdal.TermProgress, multithread=True)
>
> This logs the error, but doesn't throw an exception. The end result is
> that I have a TIF that is missing the contents of the file that suffered
> from that CURL read error.
>
> For what it's worth this also happens with the command line utility, it's
> just not as obvious. Without -multi, it only prints the "0" for the
> progress before the error halts execution:
> gdalwarp /vsis3/your-bucket/test/input.vrt output.tif
> Processing /vsis3/your-bucket/test/input.vrt [1/1] : 0ERROR 4:
> `/vsis3/your-bucket/test/input2.tif' does not exist in the file system, and
> is not recognized as a supported dataset name.
>
> But with -multi, notice the progress goes to 100%:
> gdalwarp /vsis3/your-bucket/test/input.vrt output.tif -multi
> Processing /vsis3/your-bucket/test/input.vrt [1/1] : 0ERROR 4:
> `/vsis3/your-bucket/test/input2.tif' does not exist in the file system, and
> is not recognized as a supported dataset name.
> ...10...20...30...40...50...60...70...80...90...100 - done.
>
> I guess my workaround is to just not use multithread=True. But is this
> something that could be fixed?
>
> Thanks
>
> ___
> gdal-dev mailing 
> listgdal-dev@lists.osgeo.orghttps://lists.osgeo.org/mailman/listinfo/gdal-dev
>
> -- http://www.spatialys.com
> My software is free, but my time generally not.
>
>
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Motion: Add Michael Smith to GDAL PSC

[Daniel Baston via gdal-dev]

+1 Dan

On Mon, Mar 3, 2025 at 11:30 AM Daniel Morissette via gdal-dev <
gdal-dev@lists.osgeo.org> wrote:

> Hi PSC members,
>
> I would like to nominate Michael Smith to become a member of the GDAL
> Project Steering Committee.
>
>https://www.osgeo.org/member/michael-smith/
>
> Mike is a long time GDAL power user and contributor/supporter.  He is
> also very active in several other OSGeo projects including as member of
> the MapServer and PDAL project PSC's and was actually one of the
> instigators behind the PDAL project, the GDAL of Point Clouds.
>
> Mike has been active on the OSGeo Board since 2014 as treasurer. Given
> his years of experience with the management of open source projects (and
> non-coding activities), a side benefit of having him on the GDAL PSC is
> that he could also help us with managing the sponsorship program and the
> NumFocus relationship and take some of this operational work off
> Howard's shoulders.
>
> I start with my +1
>
> Daniel
>
> --
> Daniel Morissette
> Mapgears Inc
> T: +1 418-696-5056 #201
>
> ___
> gdal-dev mailing list
> gdal-dev@lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Missing exception when using multithread=True for gdal.Warp

[Even Rouault via gdal-dev]

Tim,

wil be solved (in 3.11 only) per https://github.com/OSGeo/gdal/pull/11915

In the meantime, check for gdal.Warp() return value: if None, an error 
occurred


Even

Le 01/03/2025 à 01:00, Tim Harris via gdal-dev a écrit :
Hi, I'm having some trouble warping a VRT to a TIF where the VRT and 
its constituent rasters are in S3. It stems from a random read error, 
but the real problem is that the gdal.Warp call didn't throw an 
exception when it was supposed to.


I saw there's this pretty old GitHub issue that may be related, but 
I'm not sure:

https://github.com/OSGeo/gdal/issues/3372

In my situation what really happened was a random /vsicurl read 
failed... sort of like an issue I reported in December that was fixed 
in GDAL 3.10.1:

https://github.com/OSGeo/gdal/issues/11552

Here's a section of my log file (filename sanitized):
ERROR 11: CURL error: Empty reply from server
0...10...20...30...40...50...60...70...80...90...100 - done.
ERROR 4: `/vsis3/path/to/raster.tif' does not exist in the file 
system, and is not recognized as a supported dataset name.


Notice the CURL error but the progress still goes to 100%. The warp 
completed successfully despite the read error.


I have a hopefully simple reproduce case here. The first few shell 
commands will make two TIFs, put them in S3, then make a VRT of those 
TIFs and also put that in S3. Then to simulate the CURL error above, 
it just deletes one of the TIFs so that the VRT references a 
non-existent file:
gdal_create -of GTiff -outsize 1000 1000 -bands 1 -a_srs EPSG:4326 
-a_ullr 0 1 1 0 input1.tif
gdal_create -of GTiff -outsize 1000 1000 -bands 1 -a_srs EPSG:4326 
-a_ullr 1 1 2 0 input2.tif

aws s3 cp input1.tif s3://your-bucket/test/input1.tif
aws s3 cp input2.tif s3://your-bucket/test/input2.tif
gdalbuildvrt input.vrt /vsis3/your-bucket/test/input1.tif 
/vsis3/your-bucket/test/input2.tif

aws s3 cp input.vrt s3://your-bucket/test/input.vrt
aws s3 rm s3://your-bucket/test/input2.tif

Then in Python:
from osgeo import gdal
gdal.UseExceptions()
gdal.Warp("output.tif", "/vsis3/tharris-bucket/test/input.vrt", 
callback=gdal.TermProgress)


This throws a RuntimeError, as expected. But delete the partial file 
and try with multithread=True:

rm output.tif
gdal.Warp("output.tif", "/vsis3/tharris-bucket/test/input.vrt", 
callback=gdal.TermProgress, multithread=True)


This logs the error, but doesn't throw an exception. The end result is 
that I have a TIF that is missing the contents of the file that 
suffered from that CURL read error.


For what it's worth this also happens with the command line utility, 
it's just not as obvious. Without -multi, it only prints the "0" for 
the progress before the error halts execution:

gdalwarp /vsis3/your-bucket/test/input.vrt output.tif
Processing /vsis3/your-bucket/test/input.vrt [1/1] : 0ERROR 4: 
`/vsis3/your-bucket/test/input2.tif' does not exist in the file 
system, and is not recognized as a supported dataset name.


But with -multi, notice the progress goes to 100%:
gdalwarp /vsis3/your-bucket/test/input.vrt output.tif -multi
Processing /vsis3/your-bucket/test/input.vrt [1/1] : 0ERROR 4: 
`/vsis3/your-bucket/test/input2.tif' does not exist in the file 
system, and is not recognized as a supported dataset name.

...10...20...30...40...50...60...70...80...90...100 - done.

I guess my workaround is to just not use multithread=True. But is this 
something that could be fixed?


Thanks

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev


--
http://www.spatialys.com
My software is free, but my time generally not.
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Motion: Add Michael Smith to GDAL PSC

[Sean Gillies via gdal-dev]

+1

On Mon, Mar 3, 2025 at 9:30 AM Daniel Morissette via gdal-dev <
gdal-dev@lists.osgeo.org> wrote:

> Hi PSC members,
>
> I would like to nominate Michael Smith to become a member of the GDAL
> Project Steering Committee.
>
>https://www.osgeo.org/member/michael-smith/
>
> Mike is a long time GDAL power user and contributor/supporter.  He is
> also very active in several other OSGeo projects including as member of
> the MapServer and PDAL project PSC's and was actually one of the
> instigators behind the PDAL project, the GDAL of Point Clouds.
>
> Mike has been active on the OSGeo Board since 2014 as treasurer. Given
> his years of experience with the management of open source projects (and
> non-coding activities), a side benefit of having him on the GDAL PSC is
> that he could also help us with managing the sponsorship program and the
> NumFocus relationship and take some of this operational work off
> Howard's shoulders.
>
> I start with my +1
>
> Daniel
>
>
-- 
Sean Gillies
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Mute HDF5-DIAG diagnostic output

[Laurențiu Nicola via gdal-dev]

I guess that's because you're enabling exceptions, but even without them I get 
no error logging at all, at least on the file I tried with.

Laurentiu

On Wed, Mar 5, 2025, at 08:55, Christian Beilschmidt via gdal-dev wrote:
> I tried it with the Python bindings, but there I don't get the `HDF5-DIAG 
> `messages at all`.`
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Mute HDF5-DIAG diagnostic output

[Christian Beilschmidt via gdal-dev]

I tried it with the Python bindings, but there I don't get the
HDF5-DIAG messages
at all.

# Ubuntu 24 LTS version
pip install gdal[numpy]=="3.8.4.*"

**
from osgeo import gdal
gdal.UseExceptions()
dataset = gdal.OpenEx(
"path/to/file.nc",
nOpenFlags=gdal.OF_MULTIDIM_RASTER | gdal.OF_READONLY,
)
print(dataset.GetDriver().ShortName)

rootGroup = dataset.GetRootGroup()
print(rootGroup.GetGroupNames())

rootGroup.OpenGroup('scenario_6')
**

**
netCDF
['scenario_1', 'scenario_2', 'scenario_3', 'scenario_4', 'scenario_5']
Traceback (most recent call last):
  File "/examples/hdf5.py", line 14, in 
rootGroup.OpenGroup('scenario_6')
  File "/examples/env/lib/python3.12/site-packages/osgeo/gdal.py", line
4088, in OpenGroup
return _gdal.Group_OpenGroup(self, *args)
   ^^
RuntimeError: Group scenario_6 does not exist
**

This is what I would hope for, but it is strange indeed.

Best regards
 Christian

Am Di., 4. März 2025 um 09:22 Uhr schrieb Laurențiu Nicola via gdal-dev <
gdal-dev@lists.osgeo.org>:

> PS: the only reference to HDF5's error handling I found while skimming the
> GDAL code was
> https://github.com/OSGeo/gdal/blob/0caa252/frmts/hdf5/hdf5vfl.h#L236-L241.
>
> On Tue, Mar 4, 2025, at 10:09, Laurențiu Nicola via gdal-dev wrote:
>
> Hi Christian,
>
> On Tue, Mar 4, 2025, at 10:05, Christian Beilschmidt via gdal-dev wrote:
>
> I tried sending
> GDAL's error logging to /dev/null but this does not change anything.
>
>
> I take it
> https://docs.rs/gdal/latest/gdal/config/fn.set_error_handler.html doesn't
> work? Can you try e.g. gdal.PushErrorHandler('CPLQuietErrorHandler') in
> Python to make sure?
>
> Laurentiu
> ___
> gdal-dev mailing list
> gdal-dev@lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>
>
> ___
> gdal-dev mailing list
> gdal-dev@lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Mute HDF5-DIAG diagnostic output

[Laurențiu Nicola via gdal-dev]

PS: the only reference to HDF5's error handling I found while skimming the GDAL 
code was 
https://github.com/OSGeo/gdal/blob/0caa252/frmts/hdf5/hdf5vfl.h#L236-L241.

On Tue, Mar 4, 2025, at 10:09, Laurențiu Nicola via gdal-dev wrote:
> Hi Christian,
> 
> On Tue, Mar 4, 2025, at 10:05, Christian Beilschmidt via gdal-dev wrote:
 I tried sending GDAL's 
error logging to /dev/null but this does not change anything.
> 
> I take it https://docs.rs/gdal/latest/gdal/config/fn.set_error_handler.html 
> doesn't work? Can you try e.g. gdal.PushErrorHandler('CPLQuietErrorHandler') 
> in Python to make sure?
> 
> Laurentiu
> ___
> gdal-dev mailing list
> gdal-dev@lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev
> 
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Mute HDF5-DIAG diagnostic output

[Laurențiu Nicola via gdal-dev]

Hi Christian,

On Tue, Mar 4, 2025, at 10:05, Christian Beilschmidt via gdal-dev wrote:
 I tried sending GDAL's 
error logging to /dev/null but this does not change anything.

I take it https://docs.rs/gdal/latest/gdal/config/fn.set_error_handler.html 
doesn't work? Can you try e.g. gdal.PushErrorHandler('CPLQuietErrorHandler') in 
Python to make sure?

Laurentiu___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

[gdal-dev] Mute HDF5-DIAG diagnostic output

[Christian Beilschmidt via gdal-dev]

 

Dear GDAL devs,

I am using GDAL 3.8.4 and have some tests in a Rust codebase that check
behavior on trying to open non-existing NetCDF files, non-existing groups
within the NetCDF, etc.

I am working with the dataset by opening it as a multidimensional GDAL
dataset and then use GDALGroupOpenGroup, etc.

The testing works fine, but I am getting loads of diagnostic messages from
the underlying NetCDF library.

HDF5-DIAG: Error detected in HDF5 (1.10.10) thread 3:
  #000: ../../../src/H5A.c line 484 in H5Aopen_by_name(): can't open attribute
major: Attribute
minor: Can't open object
  #001: ../../../src/H5Aint.c line 542 in H5A__open_by_name(): unable
to load attribute info from object header
major: Attribute
minor: Unable to initialize object
  #002: ../../../src/H5Oattribute.c line 496 in
H5O__attr_open_by_name(): can't locate attribute:
'_QuantizeGranularBitRoundNumberOfSignificantDigits'
major: Attribute
minor: Object not found


How can I mute these HDF5-DIAG messages?

I tried sending GDAL's error logging to /dev/null but this does not change
anything.


Best regards

 Christian
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] pansharpening on images with RPCs post GDAL 3.6

[Joe McGlinchy via gdal-dev]

Hello,

I figured a way around this, and to be completely honest this issue was coming 
up from using the DIMAP files (for Pleiades NEO and HR) for the panchromatic 
and multispectral images and so were loaded with the DIMAP driver.

1. My images for input to pansharpening had no geotransform, as they were 
generated through several math processing steps using the DIMAP files (for 
Pleiades NEO and HR) for the panchromatic and multispectral images and so were 
loaded with the DIMAP driver. The Geotransform was of the form (0.0, 1.0, 0.0, 
0.0, 0.0, 1.0), and this makes gdal_pansharpen unhappy.
2. I set the geotransform to that of one of the tif files (formatted with RxCx, 
using the first one) that contains a geotransform that makes sense for the 
origin of the image and RPCs, i.e., R1C1 i.e., the upper leftmost image in the 
delivery. This geotransform matches well enough to make pansharpening happy 
across the full dataset extent.
3. On the pansharpened result, remove the geotransform with gdal_edit.py 
-unsetgt  and copy in the RPCs from the original DIMAP file.

This produces a file that can be used downstream for orthorectification or any 
other operations utilizing the RPCs.

Thanks for the input!

Cheers,
Joe


From: Joe McGlinchy 
Sent: Tuesday, February 25, 2025 14:53
To: Rahkonen Jukka ; Howard Butler 
; gdal-dev@lists.osgeo.org 
Subject: Re: [gdal-dev] pansharpening on images with RPCs post GDAL 3.6

Hi Jukka,

That is almost exactly along the lines of what I was thinking: fake the 
geotransform  and CRS data to get past the non-existent CRS data within this 
level of imagery, and then copy back in the appropriate metadata. Thanks for 
your response! I'll give it a try.

Cheers,
Joe

From: Rahkonen Jukka 
Sent: Tuesday, February 25, 2025 14:51
To: Joe McGlinchy ; Howard Butler ; 
gdal-dev@lists.osgeo.org 
Subject: Re: [gdal-dev] pansharpening on images with RPCs post GDAL 3.6


EXTERNAL SENDER. DO NOT click links, or open attachments, if the sender is 
unknown, or the message seems suspicious in any way. DO NOT provide your user 
ID or Password.





Hi,



I have never done anything with RPCs, but maybe you could remove RPCs with 
something like “gdal_translate -co profile=baseline input.tif output.tif” 
(delete the aux.xml or use that magic config option that I do not remember out 
of my head), define some fake geotransformation by defining same ullr 
coordinates and crs for both PAN and MS images, then pansharpen, and finally 
copy the RPCs back like you have done before. I apologize if this does not make 
any sense.



-Jukka Rahkonen-



Lähettäjä: gdal-dev  Puolesta Joe McGlinchy 
via gdal-dev
Lähetetty: tiistai 25. helmikuuta 2025 23.38
Vastaanottaja: Howard Butler ; gdal-dev@lists.osgeo.org
Aihe: Re: [gdal-dev] pansharpening on images with RPCs post GDAL 3.6



My apologies for the repost, was not aware that is against the rules of the 
list-serv. noted!



The workflow I described, pansharpening on data in image space, is a common 
practice. This being deprecated was surprising to see as a user, but I 
understand the reasoning in the migration notes. If I end up determining a 
workflow that allows similar functionality with gdal > 3.6, I'll be sure to 
respond accordingly for the benefit of the community.



As for the rest of the reply, from one working professional to another, I'll 
look past it. thanks!



-Joe



From: Howard Butler mailto:how...@hobu.co>>
Sent: Tuesday, February 25, 2025 14:25
To: Joe McGlinchy mailto:j...@aidash.com>>
Cc: gdal-dev@lists.osgeo.org 
mailto:gdal-dev@lists.osgeo.org>>
Subject: Re: [gdal-dev] pansharpening on images with RPCs post GDAL 3.6



EXTERNAL SENDER. DO NOT click links, or open attachments, if the sender is 
unknown, or the message seems suspicious in any way. DO NOT provide your user 
ID or Password.





Joe,



Ask the AI from AiDASH to tell you how to do it? Oh, right, the GDAL list is 
your training corpus, not the inference data.



Sorry for being snotty, but please refrain from reposting questions to the list 
like this. If no one answered your question the first time, the LLM 
hallucinates and pukes out-of-order GDAL arguments at you, and you cannot dig 
through the source code and documentation to figure it out yourself, you should 
hire a consultant. https://gdal.org/en/stable/community/service_providers.html 
is a list of a few, and inspecting the commit history of the repository should 
give you some inspiration for others to privately contact.



Howard





On Feb 25, 2025, at 2:51 PM, Joe McGlinchy via gdal-dev 
mailto:gdal-dev@lists.osgeo.org>> wrote:



Any advice on how to pansharpen image space data with RPCs in gdal >= 3.7?? 
Repeating below.



Hello,



I saw in the migration notes from GDAL 3.6 -> 3.7:

- Pansharpening now requires that panchromatic and multispectral bands have 
valid geotrans