Re: [gdal-dev] There's something strange about two tests that are very similar

Even Rouault via gdal-dev Wed, 13 Mar 2024 03:47:22 -0700

Hi,

I've checkout'ed your branch locally and I can reproduce the error whenrunning the Python tests. Some of the CI checks give an interesting hintabout this being related to a UTF8 issue, and I wondered why

But running "ogrinfoautotest/ogr/data/miramon/Polygons/SimplePolygons/SimplePolFile.pol"outputs:

"""

ERROR 4: Error pMMPolygonLayer.MMArc.pF: Cannot open fileautotest/ogr/data/miramon/Polygons/SimplePolygons/SimplePolFile.arc


[QUALITY:LINEAGE:PROCESS1]
nOrganismes=1
history=C:\MiraMon\MM64.exe
date=20230628 16204988+0200

[QUALITY:LINEAGE:PROCESS1:ORGANISME_1]
IndividualName=Abel Pau
PositionName=T�cnic SIG
OrganisationName=Students and educational institutions

[...]

""""

So there are several problems:

1) On failure, you shouldn't output the content of the file in the errormessage, to keep it brief the UTF-8 error comes from the fact that thefile content is not UTF-8 encoded)


2) There is an error opening the file.

3) Interestingly something in your code *deletes* fileautotest/ogr/data/miramon/Polygons/SimplePolygons/SimplePolFile.pol !

4) You should really invest in running Valgrind. After restoring thedeleted file, I can see:

$ valgrind ogrinfoautotest/ogr/data/miramon/Polygons/SimplePolygons/SimplePolFile.pol

==1737168== Memcheck, a memory error detector
==1737168== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.

==1737168== Using Valgrind-3.15.0 and LibVEX; rerun with -h forcopyright info==1737168== Command: ogrinfoautotest/ogr/data/miramon/Polygons/SimplePolygons/SimplePolFile.pol

==1737168==
==1737168== Conditional jump or move depends on uninitialised value(s)

==1737168== at 0x4843C6F: strstr (in/usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)==1737168== by 0x635FC44: MMReturnValueFromSectionINIFile(mm_wrlayr.c:5019)

==1737168==    by 0x636219A: MMCheck_REL_FILE (mm_wrlayr.c:5785)
==1737168==    by 0x63586B0: MMInitPolygonLayer (mm_wrlayr.c:1665)
==1737168==    by 0x6358BC6: MMInitLayerByType (mm_wrlayr.c:1761)
==1737168==    by 0x636B818: MMInitLayerToRead (mm_rdlayr.c:142)

==1737168== by 0x634D6A7: OGRMiraMonLayer::OGRMiraMonLayer(charconst*, VSIVirtualHandle*, OGRSpatialReference const*, int, char**,MiraMonVectMapInfo*) (ogrmiramonlayer.cpp:219)==1737168== by 0x634C03E: OGRMiraMonDataSource::Open(char const*,VSIVirtualHandle*, OGRSpatialReference const*, int, char**)(ogrmiramondatasource.cpp:71)==1737168== by 0x634C997: OGRMiraMonDriverOpen(GDALOpenInfo*)(ogrmiramondriver.cpp:87)==1737168== by 0x67DD68C: GDALDriver::Open(GDALOpenInfo*, bool)(gdaldriver.cpp:116)

==1737168==    by 0x67F61F4: GDALOpenEx (gdaldataset.cpp:3745)

==1737168== by 0x10AC95: GDALDataset::Open(char const*, unsigned int,char const* const*, char const* const*, char const* const*)(gdal_priv.h:835)

==1737168==
==1737168== Conditional jump or move depends on uninitialised value(s)

==1737168== at 0x483EF58: strlen (in/usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)

==1737168==    by 0x5262132: CPLStrdup (cpl_conv.cpp:316)

==1737168== by 0x635FC6E: MMReturnValueFromSectionINIFile(mm_wrlayr.c:5023)

==1737168==    by 0x636219A: MMCheck_REL_FILE (mm_wrlayr.c:5785)
==1737168==    by 0x63586B0: MMInitPolygonLayer (mm_wrlayr.c:1665)
==1737168==    by 0x6358BC6: MMInitLayerByType (mm_wrlayr.c:1761)
==1737168==    by 0x636B818: MMInitLayerToRead (mm_rdlayr.c:142)

==1737168==    by 0x67F61F4: GDALOpenEx (gdaldataset.cpp:3745)
==1737168==
[...]


Le 12/03/2024 à 20:43, Abel Pau a écrit :

>>> Are sure all needed files have been committed and pushed for thefailing test?
Yes, sure. 9 files are needed for oine polygon layer. Absolutely sure.
>>> Perhaps an issue Windows CR-LF vs Unix LF end of line characters ? Thedefault settings of git on Windows may mess with binary files. cfhttps://docs.github.com/en/get-started/getting-started-with-git/configuring-git-to-handle-line-endings
I’ll take a look but in binary files the format are bytes ofinformation or blanks (there is no line endings or something similar).The drivers Writes and reads bytes and it goes to different featuresusing offsets of 4 or 8 bytes depending on the version. 4 in thiscase. Here is the format description for more information.https://www.miramon.cat/new_note/eng/notes/MiraMon_structured_vectors_file_format.pdf
Taking a look to the log I can see an advice about GDALOpenEx (See below).
Thinking about differences between extension POL and the other two(PNT and ARC) is that is the main.
poDriver->SetMetadataItem(GDAL_DMD_EXTENSION, "pol");

poDriver->SetMetadataItem(GDAL_DMD_EXTENSIONS, "pol arc pnt");
I look into multiple examples where ds = gdal.OpenEx() is called withone, two or three arguments.
So, could it be related with this issue?

Thanks!!!

THE LOG:
/home/runner/work/gdal/gdal/build-ubuntu_22.04/autotest/ogr/ogr_miramon_vector.py:175:
17281<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17282>__ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ __ _ _ _
17282<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17283>
17283<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17284>args= ('data/miramon/Polygons/SimplePolygons/SimplePolFile.pol',), kwargs= {}
17284<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17285>
17285<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17286>def OpenEx(*args, **kwargs) -> "GDALDatasetShadow *":
17286<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17287>r"""
17287<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17288>OpenEx(char const * utf8_path, unsigned int nOpenFlags=0, char **allowed_drivers=None, char ** open_options=None, char **sibling_files=None) -> Dataset
17288<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17289>
17289<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17290>
17290<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17291>Open a raster or vector file as a :py:class:`Dataset`.
17291<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17292>See :cpp:func:`GDALOpenEx`.
17292<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17293>
17293<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17294>Parameters
17294<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17295>----------
17295<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17296>utf8_path : str
17296<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17297>name of the file to open
17297<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17298>flags : int
17298<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17299>Flags controlling how the Dataset is opened. Multiple ``gdal.OF_XXX``flags
17299<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17300>may be combined using the ``|`` operator. See :cpp:func:`GDALOpenEx`.
17300<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17301>allowed_drivers : list, optional
17301<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17302>A list of the names of drivers that may attempt to open the dataset.
17302<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17303>open_options : dict/list, optional
17303<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17304>A dict or list of name=value driver-specific opening options.
17304<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17305>sibling_files: list, optional
17305<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17306>A list of filenames that are auxiliary to the main filename
17306<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17307>
17307<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17308>Returns
17308<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17309>-------
17309<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17310>Dataset, or ``None`` on failure.
17310<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17311>
17311<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17312>See Also
17312<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17313>--------
17313<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17314>:py:func:`Open`
17314<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17315>:py:func:`OpenShared`
Le 12/03/2024 à 17:22, Abel Pau via gdal-dev a écrit :

    Hi,

    I have in a single test python file with 3 tests inside:
    gdal/autotest/ogr/ogr_miramon_vector.py at master · AbelPau/gdal
    (github.com)
    
<https://github.com/AbelPau/gdal/blob/master/autotest/ogr/ogr_miramon_vector.py#L134>

    In some actions the first and second are Ok but the third is not.
    Example of failure:

    Some improvements for empty layers · AbelPau/gdal@90a289c
    (github.com)
    
<https://github.com/AbelPau/gdal/actions/runs/8251854865/job/22569933640#step:12:4863>

    Here you have the log (return _gdal.OpenEx(*args, **kwargs)

    4912
    
<https://github.com/AbelPau/gdal/actions/runs/8251854865/job/22569933640#step:12:4913>23:
    E RuntimeError: Cannot open file %s.) <--- This happen in the
    driver when open fails.

    The first two are OK:

    gdal/autotest/ogr/ogr_miramon_vector.py at master · AbelPau/gdal
    (github.com)
    
<https://github.com/AbelPau/gdal/blob/master/autotest/ogr/ogr_miramon_vector.py#L48>

    gdal/autotest/ogr/ogr_miramon_vector.py at master · AbelPau/gdal
    (github.com)
    
<https://github.com/AbelPau/gdal/blob/master/autotest/ogr/ogr_miramon_vector.py#L99>

    But this is not OK:

    gdal/autotest/ogr/ogr_miramon_vector.py at master · AbelPau/gdal
    (github.com)
    
<https://github.com/AbelPau/gdal/blob/master/autotest/ogr/ogr_miramon_vector.py#L134>

    And the file are all on the correct place (I am sure about that):

    For the second one, for example:

    gdal/autotest/ogr/data/miramon/Arcs/SimpleArcs at master ·
    AbelPau/gdal (github.com)
    
<https://github.com/AbelPau/gdal/tree/master/autotest/ogr/data/miramon/Arcs/SimpleArcs>

    And for the third one:

    gdal/autotest/ogr/data/miramon/Polygons/SimplePolygons at master ·
    AbelPau/gdal (github.com)
    
<https://github.com/AbelPau/gdal/tree/master/autotest/ogr/data/miramon/Polygons/SimplePolygons>

    On local tests are perfectly OK (see below). So, after some days
    of trying to figure out WHY and what is de difference between 2
    and 3th test I have to ask you because it must be obvious but not
    to me.

    Any idea? Some permissions stuff?? THANKS!

    root@f5b8c1fb00ae:/gdal/build/autotest# python3 -m pytest -vv -p
    no:sugar --color=no ogr/ogr_miramon_vector.py

    ================================================= test session
    starts ==================================================

    platform linux -- Python 3.10.12, pytest-8.1.1, pluggy-1.4.0 --
    /usr/bin/python3

    cachedir: .pytest_cache

    benchmark: 4.0.0 (defaults: timer=time.perf_counter
    disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0
    calibration_precision=10 warmup=False warmup_iterations=100000)

    GDAL Build Info:

      PAM_ENABLED: YES

      OGR_ENABLED: YES

      CURL_ENABLED: YES

      CURL_VERSION: 7.81.0

      GEOS_ENABLED: YES

      GEOS_VERSION: 3.10.2-CAPI-1.16.0

      PROJ_BUILD_VERSION: 8.2.1

    PROJ_RUNTIME_VERSION: 8.2.1

      COMPILER: GCC 11.4.0

    GDAL_DOWNLOAD_TEST_DATA: undefined (tests relying on downloaded
    data may be skipped)

    GDAL_RUN_SLOW_TESTS: undefined (tests marked as "slow" will be
    skipped)

    rootdir: /gdal/build/autotest

    configfile: pytest.ini

    plugins: env-1.1.3, benchmark-4.0.0

    collected 3 items

    ogr/ogr_miramon_vector.py::test_ogr_miramon_simple_point
    PASSED                                                  [ 33%]

    ogr/ogr_miramon_vector.py::test_ogr_miramon_simple_arc PASSED [ 66%]

    ogr/ogr_miramon_vector.py::test_ogr_miramon_simple_polygon PASSED
    [100%]

    ================================================== 3 passed in
    0.11s ===================================================

    *Abel Pau Garcia*

    *GIS developer*

        

    https://www.creaf.cat/sites/default/files/creaf-signature.png

    *a....@creaf.uab.cat* <mailto:a....@creaf.uab.cat>

    *Let's chat on Teams!*
    <https://teams.microsoft.com/l/chat/0/0?users=a....@creaf.uab.cat>

    *Tel. +34 934814277*

        

    https://www.creaf.cat/sites/default/files/so-en-signature.png

    https://www.creaf.cat/sites/default/files/twitter-icon-signature.png
    
<https://twitter.com/CREAF_ecologia>https://www.creaf.cat/sites/default/files/linkedin-icon-signature.png
    
<https://www.linkedin.com/company/1363052?trk=tyah&trkInfo=clickedVertical:company,clickedEntityId:1363052,idx:2-1-2,tarId:1465807877789,tas:creaf>https://www.creaf.cat/sites/default/files/youtube-icon-signature.png
    
<https://www.youtube.com/c/creafecologia>https://www.creaf.cat/sites/default/files/instagram-icon-signature.png
    <https://www.instagram.com/CREAF_ecologia/>

    *www.creaf.cat* <http://www.creaf.cat>*| **http://blog.creaf.cat*
    <http://blog.creaf.cat>

        

    https://www.creaf.cat/sites/default/files/uab_logo_signatura.png

    CREAF. Campus UAB. Edifici C. 08193 Bellaterra (Barcelona)


    Before printing this electronic message, think about the environment.

    http://www.creaf.uab.cat/_signatura/line.gif



    _______________________________________________

    gdal-dev mailing list

    gdal-dev@lists.osgeo.org

    https://lists.osgeo.org/mailman/listinfo/gdal-dev

--
http://www.spatialys.com
My software is free, but my time generally not.


--
http://www.spatialys.com
My software is free, but my time generally not.

_______________________________________________
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] There's something strange about two tests that are very similar

Reply via email to