Hi Michael,
This error comes very deep inside libparquet, actually in libthrift. The
more natural assumption would be that it would be due to a corrupted
Parquet file. If you disabled multithreading (GDAL_NUM_THREADS=1) and
enabled --debug on, perhaps this would happen on the same file ? I was
wondering also if that couldn't come from RAM exhaustion, but that
doesn't seem likely (did you monitor the RAM consumption?). It would be
interesting to see if that would also occur after fetching locally the
files under overturemaps-us-west-2/release/2024-07-22.0/theme=buildings
and running the conversion from the local files
Even
Le 30/07/2024 à 20:37, Michael Smith via gdal-dev a écrit :
Hi all,
I was converting the overture maps parquet data to geopackage and got
this error using gdal master (via conda).
GDAL_NUM_THREADS=ALL_CPUS CPL_TMPDIR=/data2 ogr2ogr -f gpkg
/data/overture_buildings.gpkg
/vsis3/overturemaps-us-west-2/release/2024-07-22.0/theme=buildings
"theme=buildings" -progress -nlt PROMOTE_TO_MULTI -nln buildings
-skipfailures
ERROR 1: ReadNext() failed: Couldn't deserialize thrift:
TProtocolException: Exceeded size limit
Not sure what size limit it exceeded? I’ve had this happen several
times but at very different points in the process.
Ideas?
Mike
--
Michael Smith
US Army Corps of Engineers
Remote Sensing/GIS Center
_______________________________________________
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev
--
http://www.spatialys.com
My software is free, but my time generally not.
_______________________________________________
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev
