Jérémy,
good point. Adressed per https://github.com/uclouvain/openjpeg/pull/1560
and https://github.com/OSGeo/gdal/pull/11270
Even
Le 13/11/2024 à 15:27, Jérémy Anger via gdal-dev a écrit :
Hello,
Trying to optimize the reading speed of JP2 files from object
storages, I've stumbled upon this PR
https://github.com/uclouvain/openjpeg/pull/514.
It introduced a behaviour particularly harmful for network cropping
JP2 files, as it leads to many excessive requests (time, bandwidth and
cost wasted).
As described in the PR, it is possible to disable the fix through the
compilation flag -DOPJ_DISABLE_TPSOT_FIX=ON.
Here is a small experiment to compare with and without the fix:
Setup:
$ gdalinfo --version
GDAL 3.9.2, released 2024/08/13
$ git clone https://github.com/uclouvain/openjpeg --branch v2.5.2
--depth 1
$ cd openjpeg
# build with the fix (default)
$ mkdir build-enabled-fix && cd build-enabled-fix && cmake ..
-DCMAKE_BUILD_TYPE=Release -DOPJ_DISABLE_TPSOT_FIX=OFF && make -j && cd ..
# build without the fix
$ mkdir build-disabled-fix && cd build-disabled-fix && cmake ..
-DCMAKE_BUILD_TYPE=Release -DOPJ_DISABLE_TPSOT_FIX=ON && make -j && cd ..
Benchmark:
export GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR
export CPL_CURL_VERBOSE=ON
# it is possible to tweak CPL_VSIL_CURL_CHUNK_SIZE to reduce the
number of requests, but this is not studied here
# this file is composed of 121 tiles of size 1024x1024
url=/vsicurl/https://storage.googleapis.com/gcp-public-data-sentinel-2/tiles/31/T/CJ/S2B_MSIL1C_20230209T105109_N0509_R051_T31TCJ_20230209T124951.SAFE/GRANULE/L1C_T31TCJ_A030965_20230209T105200/IMG_DATA/T31TCJ_20230209T105109_B02.jp2
# tile at the beginning
$ LD_LIBRARY_PATH=$PWD/openjpeg/build-enabled-fix/bin
gdal_translate -srcwin 0 0 500 500 "$url" o.tif 2>&1 | grep ^Range: |
wc -l
122
$ LD_LIBRARY_PATH=$PWD/openjpeg/build-disabled-fix/bin
gdal_translate -srcwin 0 0 500 500 "$url" o.tif 2>&1 | grep ^Range: |
wc -l
2
# tile in the middle
$ LD_LIBRARY_PATH=$PWD/openjpeg/build-enabled-fix/bin
gdal_translate -srcwin 5120 5120 500 500 "$url" o.tif 2>&1 | grep
^Range: | wc -l
122
$ LD_LIBRARY_PATH=$PWD/openjpeg/build-disabled-fix/bin
gdal_translate -srcwin 5120 5120 500 500 "$url" o.tif 2>&1 | grep
^Range: | wc -l
62
#.tile at the end
$ LD_LIBRARY_PATH=$PWD/openjpeg/build-enabled-fix/bin
gdal_translate -srcwin 10240 10240 500 500 "$url" o.tif 2>&1 | grep
^Range: | wc -l
122
$ LD_LIBRARY_PATH=$PWD/openjpeg/build-disabled-fix/bin
gdal_translate -srcwin 10240 10240 500 500 "$url" o.tif 2>&1 | grep
^Range: | wc -l
122
Note that disabling the fix is particularly important when TLM markers
are present (on the master branch of openjpeg), but this is not shown
here. The tests above are from openjpeg v2.5.2 and the Sentinel-2
rasters do not have TLM markers.
It might be worth converting the compilation flag into a runtime flag
on openjpeg's side, and expose it through a GDAL option at least (or
disabling the fix from GDAL by default).
What do you think?
Best regards,
Jérémy
_______________________________________________
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev
--
http://www.spatialys.com
My software is free, but my time generally not.
Butcher of all kinds of standards, open or closed formats. At the end, this is
just about bytes.
_______________________________________________
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev
