I've come up with a quicky&dirty script
https://github.com/rouault/symbol_clash_detector/blob/master/symbol_clash_detector.py
that works only for Linux and attempts to find C symbol clashes in the
.so files loaded in a process (C++ is trickier because due to template &
inline code, functions from the standard library are duplicated in C++
libraries)
When run on Conda gdal-3.6.0, it outputs:
$ PYTHONPATH=/home/even/symbol_clash_detector python -c "from osgeo
import gdal; import symbol_clash_detector"
WARNING: libVersionPoint present in
/home/even/miniconda3/envs/test/lib/libplc4.so is also present in
/home/even/miniconda3/envs/test/lib/libnspr4.so
WARNING: locale_charset present in
/home/even/miniconda3/envs/test/lib/libiconv.so.2.6.1 is also present in
/home/even/miniconda3/envs/test/lib/libcharset.so.1.0.0
WARNING: libVersionPoint present in
/home/even/miniconda3/envs/test/lib/libplds4.so is also present in
/home/even/miniconda3/envs/test/lib/libnspr4.so
I'm not knowledgeable enough to assess if those 3 instances are
problematic (symbols with same name and content are OK), hopefully not,
as there are more or less system libraries.
When run on my dev GDAL build with drivers using proprietary SDKs, it
does find real symbol clashes that I was aware of, due to those SDKs
embedding symbol libraries (like libxml2 in FileGDB SDK, lcms in ECW
SDK, ...)
Le 21/11/2022 à 13:37, Even Rouault a écrit :
Paul,
Looking at
https://stackoverflow.com/questions/63161532/python-segmentation-fault-when-using-zipfile-while-gdal-package-loaded
shows an interesting thing in a gdb backtrace
|#0 0x00007ffff7e3aa50 in free () from /usr/lib/libc.so.6 #1
0x00007ffff485ae0f in inflateReset2 () from /usr/lib/libcfitsio.so.9
#2 0x00007ffff39ec1a4 in inflateInit2_ () from /usr/lib/libz.so.1 |
||
One can see that libz inflateInit2_() ends up calling libcfitsio's
inflateReset2(), but libcfitsio is not a dependency of libz ! That's
the reverse. Which suggests that some builds of libcfitsio use an
internal outdated copy of libz, without symbol renaming. Looking at
conda-forge cfitsio recipe I see
https://github.com/conda-forge/cfitsio-feedstock/blob/main/recipe/unvendor_zlib.patch
(added per https://github.com/conda-forge/cfitsio-feedstock/pull/22)
which changes cfitsio to use conda-forge zlib, to avoid such issue.
With latest conda-forge gdal, requests, openpyxl, I can't trigger
crashes with the reproducers of the various mentioned tickets.
I'd suggest you run your code under gdb or Valgrind and look if there
are not funny symbol calls like the above.
Also if use pip module, avoid using their binary wheels. For example
to make sure that pyproj's embedded PROJ doesn't conflict with the
PROJ used by GDAL, to avoid similar issues where you'd have 2 versions
of the same lib in the same process.
Even
Le 21/11/2022 à 12:40, Paul Harwood a écrit :
This may not be a GDAL bug and as such I have not raised it as an
issue but I wanted to put it here for anyone else who comes across
the same problem.
There does appear to be some interference between, at least the conda
distribution of, GDAL and something else within Python causing a
segmentation fault.
----------------------------------------
SYMPTOMS
I have a Python application that uses GDAL (and PDAL Python and MDAL
Python and PyProj) wrapped in Kivy - so it is relatively complicated.
I was just clearing tech debt / updating etc. GDAL to 3.6 but also
the other apps to latest versions (including kivy).
BEFORE - if I tried to open an unrecognised file in GDAL I got an
Exception (using UseExceptions() ). This is related to Python Duck Typing
AFTER - I get a seg fault and no debug info.
This happened in exactly the same way on two machines - one Mac and
one Windows - so I don't think it is a bad config. Attempts to
create a minimum example do not work - suggesting that it is a
complicated interaction between the various components
I am using conda as the environment manager.
------------------------------------------
ULTIMATE(-ish) SOLUTION
After getting some diagnostic data and some experimentation, it
turned out that the fault is coming in the error handling. I managed
to avoid the segmentation fault by creating and registering a
dedicated error handler for GDAL.
--------------------------------------------
DEEPER DISCUSSION
I found some discussion from a couple of years ago about a problem
that sounds similar (see
https://github.com/conda-forge/gdal-feedstock/issues/365) - it was
not solved but was not very reproducible so was closed.
This suggests an interference between requests and gdal - depending
on the order in which they are imported. Certainly - kivy (or more
precisely kivyMD) is using requests but I was not able to solve the
problem by changing import orders (kivy has a lot of asynch and event
based things happening and the app is very modular so it is not that
simple).
However - if the interference is in the error handler - then that
would explain why it was difficult to reproduce (need to hit the
right exception in the right way).
Not sure if we can fix this one - but if anyone else does come across
a similar problem I would suggest to try and create a custom error
handler to see if the problem is there.
_______________________________________________
gdal-dev mailing list
[email protected]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
--
http://www.spatialys.com
My software is free, but my time generally not.
_______________________________________________
gdal-dev mailing list
[email protected]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
--
http://www.spatialys.com
My software is free, but my time generally not.
_______________________________________________
gdal-dev mailing list
[email protected]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
