Oh, its just the second open that works In [1]: from osgeo import gdal In [2]: pszFilename = "/vsis3/grid-dev-publiclidar/estonia/dtm/estonia_dtm_5m.tif" In [3]: hDataset = gdal.OpenEx(pszFilename, gdal.GA_ReadOnly) In [4]: hDataset In [5]: hDataset = gdal.Open(pszFilename, gdal.GA_ReadOnly) In [6]: hDataset Out[6]: <osgeo.gdal.Dataset; proxy of <Swig Object of type 'GDALDatasetShadow *' at 0x7fef5b3fe1c0> > In [7]: hDataset.GetProjectionRef() Out[7]: 'PROJCS["Estonian Coordinate System of 1997",GEOGCS["EST97",DATUM["Estonia_1997",SPHEROID["GRS 1980",6378137,298.257222101,AUTHORITY["EPSG","7019"]],AUTHORITY["EPSG","6180"]],PRIMEM["Greenwich",0,AUTHORITY["EPSG","8901"]],UNIT["degree",0.0174532925199433,AUTHORITY["EPSG","9122"]],AUTHORITY["EPSG","4180"]],PROJECTION["Lambert_Conformal_Conic_2SP"],PARAMETER["latitude_of_origin",57.5175539305556],PARAMETER["central_meridian",24],PARAMETER["standard_parallel_1",59.3333333333333],PARAMETER["standard_parallel_2",58],PARAMETER["false_easting",500000],PARAMETER["false_northing",6375000],UNIT["metre",1,AUTHORITY["EPSG","9001"]],AXIS["Northing",NORTH],AXIS["Easting",EAST],AUTHORITY["EPSG","3301"]]'
On Sun, Nov 20, 2022 at 8:47 AM Michael Smith <michael.smith.e...@gmail.com> wrote: > Is there a reason why OpenEx would work but Open wouldn’t? > > > > In [1]: from osgeo import gdal > > In [2]: pszFilename = > "/vsis3/grid-dev-publiclidar/estonia/dtm/estonia_dtm_5m.tif" > > In [3]: hDataset = gdal.Open(pszFilename, gdal.GA_ReadOnly) > > In [4]: hDataset > > In [5]: hDataset = gdal.OpenEx(pszFilename, gdal.GA_ReadOnly) > > In [6]: hDataset > > Out[6]: <osgeo.gdal.Dataset; proxy of <Swig Object of type > 'GDALDatasetShadow *' at 0x7f827217c450> > > > In [7]: hDataset.GetGeoTransform() > > Out[7]: (365000.0, 5.0, 0.0, 6635000.0, 0.0, -5.0) > > > > Mike > > > > > > *From: *Even Rouault <even.roua...@spatialys.com> > *Date: *Saturday, November 19, 2022 at 10:08 AM > *To: *<michael.smith.e...@gmail.com> > *Cc: *gdal-dev <gdal-dev@lists.osgeo.org> > *Subject: *Re: [gdal-dev] errors using IAM instance profile auth in s3 > > > > > > Le 19/11/2022 à 16:00, michael.smith.e...@gmail.com a écrit : > > Correct, not a public bucket, which is why the IAM credentials are needed. > If I set them manually, it all works fine. > > That's super weird if the result of a range request changes depending on > how credentials have been set... Perhaps enable CPL_CURL_VERBOSE=ON env > variable and diff the logs ? > > You could also try the gdal_cp.py sample script at > https://github.com/OSGeo/gdal/blob/master/swig/python/gdal-utils/osgeo_utils/samples/gdal_cp.py > , which is a cp-like utility working with GDAL virtual file systems, with > the 2 authentication methods > > python gdal_cp.py > /vsis3/grid-dev-publiclidar/estonia/dtm/estonia_dtm_5m.tif out.tif > > (you can interrupt it with ctrl-c after a few seconds. that will be enough > to get the first bytes) > > you might need to run an hexadecimal editor to inspect a bit the content. > > > > [ u02]$ export AWS_ACCESS_KEY_ID=xxxxx > > Yes, a 206 response code means success here as we are requesting only > bytes 0-16383. So maybe the file is not a valid TIFF ? > > ( "grid-dev-publiclidar" must not be so public I guess, because when > trying with my credentials, I get a Access Denied) > > Le 19/11/2022 à 15:40, michael.smith.e...@gmail.com a écrit : > > I’m seeing that it’s getting a 206 response code, so wouldn’t that > indicate auth is working? > > > > gdalinfo /vsis3/grid-dev-publiclidar/estonia/dtm/estonia_dtm_5m.tif > > HTTP: Fetch(http://169.254.169.254/latest/api/token) > > HTTP: libcurl/7.86.0 OpenSSL/3.0.7 zlib/1.2.13 libssh2/1.10.0 > nghttp2/1.47.0 > > HTTP: These HTTP headers were set: X-aws-ec2-metadata-token-ttl-seconds: 10 > > HTTP: Fetch( > http://169.254.169.254/latest/meta-data/iam/security-credentials/) > > HTTP: Fetch( > http://169.254.169.254/latest/meta-data/iam/security-credentials/iam-grid-s3 > ) > > AWS: Storing AIM credentials until 2022-11-19T20:42:58Z > > S3: Downloading 0-16383 ( > https://grid-dev-publiclidar.s3.amazonaws.com/estonia/dtm/estonia_dtm_5m.tif > )... > > S3: Got response_code=206 > > gdalinfo failed - unable to open > '/vsis3/grid-dev-publiclidar/estonia/dtm/estonia_dtm_5m.tif'. > > > > > > Mike > > > > > > > > On Nov 19, 2022, at 9:26 AM, Even Rouault <even.roua...@spatialys.com> > <even.roua...@spatialys.com> wrote: > > Hi Mike, > > could you send the output of > > curl > http://169.254.169.254/latest/meta-data/iam/security-credentials/iam-grid-s3 > > Slightly redacted of course, but with the exact formatting. This part of > thee code currently uses a "simple JSON parser" ( > https://github.com/OSGeo/gdal/blob/c61d116a469821b769630a112dee7f1a61fed885/port/cpl_aws.cpp#L554), > which is actually just a non JSON-aware string tokenizer, and I suspect it > could be defeated by a new formatting of S3 or something specific to your > credentials. > > It could also be that something unhandled by that parser appears inside > quoted strings, like an escaped double quote or some other JSON escaped > character (like an escaped forward slash \/ ) > > If that was the case we should likely switch to proper JSON > deserialization (that part of the code must predate libjson-c being a build > requirement of GDAL). > > Even > > > -- > http://www.spatialys.com > My software is free, but my time generally not. > > -- > > http://www.spatialys.com > > My software is free, but my time generally not. > > -- > > http://www.spatialys.com > > My software is free, but my time generally not. > >
_______________________________________________ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
