Le 28/09/2023 à 18:47, Scott via gdal-dev a écrit :
I should have been more specific.
One particular machine has 8GB of memory. When I try to do the most
simple ogr2ogr command on large files, the host runs out of memory
(vmstat shows this) and ogr2ogr terminates with 'Killed', nothing more.
The data formats I have experienced this with are .fgb, .parquet and
.gpkg. The data files are 10's of GB.
As input ? as output? Which operating system ? Which GDAL version ? The
output of "ogrinfo -al -so the_input" might also be helpful. An exact
ogr2ogr command line invocation that triggers the issue would certainly
be useful. In general, most GDAL drivers and ogr2ogr itself operate in
streaming mode with low RAM requirements, but there might be exceptions
(some configurations of GeoJSON file may require full ingestion on
reading for example). I'm also aware of issues with RAM fragmentation
due to how some memory allocators work, but they seem to be restricted
to multithreaded uses
(https://gdal.org/user/multithreading.html#ram-fragmentation-and-multi-threading),
which current ogr2ogr shouldn't trigger
Even
Thanks for the responses!
_______________________________________________
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev
--
http://www.spatialys.com
My software is free, but my time generally not.
_______________________________________________
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev
