[fpc-pascal] DLL calling Firebird 2: crashes at the end
After hopefully getting parameter passing fixed (I hope, thanks Mark), I'm still facing this problem: dll using Firebird crashes on Windows and Linux Using a dummy db unit that does not connect to Firebird: everything ok. Compiler options on a Windows compile: -MObjFPC -Scaghi -WR -Cirot -Os -gw2 -gl -vewnhibq -Filib\i386-win32 -Fu3rdparty\csvdocument -Fu3rdparty\fpspreadsheet -Fu. -FUlib\i386-win32 -O-1 -Cg -dFPCONLY -dDPDLLDEBUG Linux bt shows (the location of the error seems weird - perhaps memory messed up due to Firebird .so problems?) (gdb) run Starting program: /home/pascaldev/postcode/Source/dlldemo [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". [New Thread 0x7472d700 (LWP 7731)] Starting dll demo program. Looking up house number 68a, postcode 2514GL: Program received signal SIGSEGV, Segmentation fault. 0x7785b8e5 in fpc_ansistr_decr_ref () from /usr/lib/libdutchpostcode.so (gdb) bt #0 0x7785b8e5 in fpc_ansistr_decr_ref () from /usr/lib/libdutchpostcode.so #1 0x77fe6b00 in ?? () #2 0x7785b975 in fpc_ansistr_assign () from /usr/lib/libdutchpostcode.so #3 0x0044 in ?? () #4 0x00400770 in SetCity@plt () #5 0x0044 in ?? () #6 0x778c33db in SETCITY (this=0x77fe6ac0, AVALUE=0x77fdf618 '''s-Gravenhage') at dutchaddress.pas:262 #7 0x778bc22e in POSTCODE2ADDRESS (this=0x77fce960, ADDRESS=0x77fe6ac0) at businesslogic.pas:390 #8 0x77851682 in POSTCODE2ADDRESS (NUMBER=0x474058 '68a', POSTCODE=0x474050 '2514GL') at dutchpostcode.lpr:153 #9 0x00400d01 in main () at dlldemo.lpr:115 If I add cmem first in the .so uses clause, I get this on running my dlldemo application: Inconsistency detected by ld.so: dl-minimal.c: 136: realloc: Assertion `ptr == alloc_last_block' failed! Should I enable other options? Longer term, I think this is a good reason to start PostgreSQL support and check if that works. Pointers/advice/requests for further info welcome, thanks. ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] DLL calling Firebird 2: crashes at the end
Reinier Olislagers wrote: After hopefully getting parameter passing fixed (I hope, thanks Mark), I'm still facing this problem: dll using Firebird crashes on Windows and Linux Using a dummy db unit that does not connect to Firebird: everything ok. Can you link the lookup DLL/so statically for test purposes? If I add cmem first in the .so uses clause, I get this on running my dlldemo application: Inconsistency detected by ld.so: dl-minimal.c: 136: realloc: Assertion `ptr == alloc_last_block' failed! If adding cmem to one you might need it in both to be effective. Should I enable other options? Longer term, I think this is a good reason to start PostgreSQL support and check if that works. Pointers/advice/requests for further info welcome, thanks. The only thing I can add at this point is that there are a couple of cases where I've written code to interface to multiple databases (Firebird, PostgreSQL) but while the underlying database library (presumably written in C) might be linked dynamically I've not tried interposing a custom demand-loaded library... I presume that in your case you're leaving the OS to do the load when the program starts, but I think that the same points about memory managers etc. apply. I have used demand-loaded libraries to interface to e.g. custom processing for MIDI or to optionally load e.g. libpcap, and once working it's reliable. -- Mark Morgan Lloyd markMLl .AT. telemetry.co .DOT. uk [Opinions above are the author's, not those of his employers or colleagues] ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] DLL calling Firebird 2: crashes at the end
On Sun, 28 Sep 2014, Reinier Olislagers wrote: After hopefully getting parameter passing fixed (I hope, thanks Mark), I'm still facing this problem: dll using Firebird crashes on Windows and Linux Using a dummy db unit that does not connect to Firebird: everything ok. Perhaps a stupid question, but: why are you not simply using sqldb ? I do a lot of work with firebird embedded, and I've never even come close to the problems you're experiencing ? Michael. ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] DLL calling Firebird 2: crashes at the end
On 28/09/2014 11:32, Michael Van Canneyt wrote: > > > On Sun, 28 Sep 2014, Reinier Olislagers wrote: > >> After hopefully getting parameter passing fixed (I hope, thanks Mark), >> I'm still facing this problem: >> dll using Firebird crashes on Windows and Linux >> Using a dummy db unit that does not connect to Firebird: everything ok. > > Perhaps a stupid question, but: why are you not simply using sqldb ? Why do you think I don't? My library does use sqldb. I just want to provide postcode lookup/address validation services to other programmers in a .dll/.so. > I do a lot of work with firebird embedded, and I've never even come > close to the problems you're experiencing ? It's not limited to Firebird embedded; I ran tests on Windows & Linux with Firebird client/server; same thing ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] DLL calling Firebird 2: crashes at the end
On 28/09/2014 11:30, Mark Morgan Lloyd wrote: > Reinier Olislagers wrote: > Can you link the lookup DLL/so statically for test purposes? I'll have a look. >> If I add cmem first in the .so uses clause, I get this on running my >> dlldemo application: >> Inconsistency detected by ld.so: dl-minimal.c: 136: realloc: Assertion >> `ptr == alloc_last_block' failed! > > If adding cmem to one you might need it in both to be effective. Ah yes. That does fix that ld error. Backtrace changes #0 0x7785be75 in fpc_ansistr_decr_ref () from /usr/lib/libdutchpostcode.so #1 0x006c0ca0 in ?? () #2 0x7785bf05 in fpc_ansistr_assign () from /usr/lib/libdutchpostcode.so #3 0x006add18 in U_$SYSTEM_$$_OUTPUT () #4 0x00400ac0 in SetCity@plt () #5 0x006add18 in U_$SYSTEM_$$_OUTPUT () #6 0x778c3c8b in SETSTREET (this=0x6c0c78, AVALUE=0x0) at dutchaddress.pas:272 #7 0x7785178f in CLEARADDRESS () at dutchpostcode.lpr:91 #8 0x00401118 in main () at dlldemo.lpr:116 > I presume that in your > case you're leaving the OS to do the load when the program starts, Yes, I am. > but I > think that the same points about memory managers etc. apply. I have used > demand-loaded libraries to interface to e.g. custom processing for MIDI > or to optionally load e.g. libpcap, and once working it's reliable. May be me, but I have trouble understanding why: 1. A regular program loading Firebird .so/.dlls works without any specified cmem etc. Here application<>Firebird dll 2. An FPC library, coded similarly to 1., fails. Here application<>dll<>Firebir dll The difference between the 2 seems to me application coupled directly to FB dll versus coupling to FPC dll. Why should that make a difference? Why should I worry about memory managers etc in the second case but not the first? (Assuming I did not create problems with application<>dll parameter passing) ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] DLL calling Firebird 2: crashes at the end
Reinier Olislagers wrote: May be me, but I have trouble understanding why: 1. A regular program loading Firebird .so/.dlls works without any specified cmem etc. Here application<>Firebird dll 2. An FPC library, coded similarly to 1., fails. Here application<>dll<>Firebir dll The difference between the 2 seems to me application coupled directly to FB dll versus coupling to FPC dll. Why should that make a difference? Why should I worry about memory managers etc in the second case but not the first? (Assuming I did not create problems with application<>dll parameter passing) Everything below is my understanding. I'm happy to be corrected. I've not used the Firebird interface libraries directly to any great extent (only the PostgreSQL ones), but the thing that occurs to me is that when the database library allocates memory for e.g. a resultset of indeterminate size it is subsequently responsible for freeing it. This is akin to OS (as distinct from application) programming, where the caller of an OS function probably has limited access rights to the memory containing structures that the OS returns, and definitely shouldn't assume that it's got arbitrary allocate/free rights to it. So in your case (2) there are three separate binaries which potentially allocate and free memory: (i) the database library (ii) the custom DLL/so and (iii) the application program, and in the general case each one should free precisely those areas of memory that it allocated: not a block more and preferably not a block less. Now if we assume that the database library has its own competent memory manager, and that its callers correctly tell it to free memory as soon as they're done with it (e.g. once a resultset has been parsed); that leaves potential problems in the custom library and application code. If the custom library never made any attempt to access Pascal-type (long) strings, dynamic arrays, or anything else on the heap which might be reallocated or extended when referenced (i.e. not just at creation), then you'd not need to do anything special. But as soon as the custom library tries to e.g. change the length of a string or dynamic array it makes a call to the memory manager, and if it's not sharing the same memory manager as the main app used to allocate the string/array that's a recipe for disaster. Now it might, potentially, be possible to tell a DLL/so as it is being loaded to link itself to the heap manager built into the main app. But the current preferred way is for the library and the app to use the same external memory manager, which is where importing cmem comes in. So if you're getting crashes and your parameters are right, the first thing to check is that your custom library and the main app are both importing and using cmem before anything else such as HeapTrc. My experience is that that works, and if it isn't working then time you put into fixing it will probably be a useful investment. Here endeth the lesson. Go in peace. -- Mark Morgan Lloyd markMLl .AT. telemetry.co .DOT. uk [Opinions above are the author's, not those of his employers or colleagues] ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
[fpc-pascal] Is it possible to use top directory of subtree with -Fu -Fi -Fl options?
Hi all, (I just registered in this list, so I don’t know if this question has come up before, and the list archives do not seem seem to a search capability.) I have a long list of subdirectories for ‘include’ and ‘units’ but they all are organized under a single subdirectory. For example, \progs\pascal\units\a \progs\pascal\units\b \progs\pascal\units\c \progs\pascal\units\d \progs\pascal\units\e Is it possible to somehow tell FPC to look into every subdirectory of \progs\pascal\units rather than specify the full path of each subdirectory separately? For example, could something equivalent to \progs\pascal\units\* be used to? (This I tried, it didn’t work.) TIA ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] DLL calling Firebird 2: crashes at the end
On 28/09/2014 13:02, Mark Morgan Lloyd wrote: > So in your case (2) there are three separate binaries which potentially > allocate and free memory: (i) the database library (ii) the custom > DLL/so and (iii) the application program, and in the general case each > one should free precisely those areas of memory that it allocated: not a > block more and preferably not a block less. Yes, that sounds like how it should work/what I understand as well. > Now if we assume that the database library has its own competent memory > manager, and that its callers correctly tell it to free memory as soon > as they're done with it (e.g. once a resultset has been parsed); that > leaves potential problems in the custom library and application code. Agreed. > If the custom library never made any attempt to access Pascal-type > (long) strings, dynamic arrays, or anything else on the heap which might > be reallocated or extended when referenced (i.e. not just at creation), > then you'd not need to do anything special. But as soon as the custom > library tries to e.g. change the length of a string or dynamic array it > makes a call to the memory manager, and if it's not sharing the same > memory manager as the main app used to allocate the string/array that's > a recipe for disaster. Yes. 3 things: 1. I hopefully fixed memory allocation issues between application<>my dll in the previous thread. Of course if you mean my dll modifying strings etc in the firebird .so/.dll yes of course but why don't these problems then occur with the simpler application<>Firebird .dll/.so? 2. Corruption does not occur when I compile my dll with database layer code that does not call the Firebird sqldb layer i.e. the Firebird .so/.dll 3. I've tried both with and without cmem in both application and my dll so they are definitely sharing the same memory manager. What the fb .so uses... well... who knows, probably libc?!?! > Now it might, potentially, be possible to tell a DLL/so as it is being > loaded to link itself to the heap manager built into the main app. But > the current preferred way is for the library and the app to use the same > external memory manager, which is where importing cmem comes in. > > So if you're getting crashes and your parameters are right, the first > thing to check is that your custom library and the main app are both > importing and using cmem before anything else such as HeapTrc. My > experience is that that works, and if it isn't working then time you put > into fixing it will probably be a useful investment. Yes, see above. > Here endeth the lesson. Go in peace. Et lectori salutem & the same to you :) ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] DLL calling Firebird 2: crashes at the end
Reinier Olislagers wrote: If the custom library never made any attempt to access Pascal-type (long) strings, dynamic arrays, or anything else on the heap which might be reallocated or extended when referenced (i.e. not just at creation), then you'd not need to do anything special. But as soon as the custom library tries to e.g. change the length of a string or dynamic array it makes a call to the memory manager, and if it's not sharing the same memory manager as the main app used to allocate the string/array that's a recipe for disaster. Yes. 3 things: 1. I hopefully fixed memory allocation issues between application<>my dll in the previous thread. Of course if you mean my dll modifying strings etc in the firebird .so/.dll yes of course but why don't these problems then occur with the simpler application<>Firebird .dll/.so? But when your code modifies strings in the context of the database library they're preallocated C-type strings, i.e. can't be extended or shrunk so don't make unexpected reference to any memory manager. 2. Corruption does not occur when I compile my dll with database layer code that does not call the Firebird sqldb layer i.e. the Firebird .so/.dll Interesting. So if I understand you correctly it's only when you've got two levels of dynamically-loaded library, i.e. one DLL is responsible for loading another. Now would probably be a good time to try Postgres as an alternative. 3. I've tried both with and without cmem in both application and my dll so they are definitely sharing the same memory manager. What the fb .so uses... well... who knows, probably libc?!?! Probably libc, but not really relevant since all the allocate/free operations are done in the same context: the lib giveth, and the lib taketh away :-) -- Mark Morgan Lloyd markMLl .AT. telemetry.co .DOT. uk [Opinions above are the author's, not those of his employers or colleagues] ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] DLL calling Firebird 2: crashes at the end
On 28/09/2014 13:54, Mark Morgan Lloyd wrote: > Reinier Olislagers wrote: > But when your code modifies strings in the context of the database > library they're preallocated C-type strings, i.e. can't be extended or > shrunk so don't make unexpected reference to any memory manager. Yes, I agree. However, this problem should also appear in regular FPC applications + Firebird library, right? Why don't I see any problems there? As mentioned, I'm using regular sqldb to talk to Firebird, nothing special. > Interesting. So if I understand you correctly it's only when you've got > two levels of dynamically-loaded library, i.e. one DLL is responsible > for loading another. Yes. When/if I add Postgresql support we may now more ;) > Now would probably be a good time to try Postgres as an alternative. Ah, a mind reader ;) ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] DLL calling Firebird 2: crashes at the end
Reinier Olislagers wrote: On 28/09/2014 13:54, Mark Morgan Lloyd wrote: Reinier Olislagers wrote: But when your code modifies strings in the context of the database library they're preallocated C-type strings, i.e. can't be extended or shrunk so don't make unexpected reference to any memory manager. Yes, I agree. However, this problem should also appear in regular FPC applications + Firebird library, right? Why don't I see any problems there? As mentioned, I'm using regular sqldb to talk to Firebird, nothing special. Interesting. So if I understand you correctly it's only when you've got two levels of dynamically-loaded library, i.e. one DLL is responsible for loading another. Yes. When/if I add Postgresql support we may now more ;) Now would probably be a good time to try Postgres as an alternative. Ah, a mind reader ;) I think the first thing I'd try is a trivial program+DLL checking that the DLL can initialise a database library, in case there's something wrong with multistage initialisation. If I understand things correctly, your app is linking to the custom library as part of the program-load operation handled by the OS, but the custom library is opening the database library dynamically. It might be worth checking this latter part with utter paranoia, my recollection is that the FPC loader library now saves any error message returned by the OS and I've found this useful in the past. -- Mark Morgan Lloyd markMLl .AT. telemetry.co .DOT. uk [Opinions above are the author's, not those of his employers or colleagues] ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] DLL calling Firebird 2: crashes at the end
On 28/09/2014 14:52, Mark Morgan Lloyd wrote: > Reinier Olislagers wrote: >> On 28/09/2014 13:54, Mark Morgan Lloyd wrote: > I think the first thing I'd try is a trivial program+DLL checking that > the DLL can initialise a database library, in case there's something > wrong with multistage initialisation. Yes, thanks, good idea. > custom library is opening the database library dynamically. It might be > worth checking this latter part with utter paranoia, my recollection is > that the FPC loader library now saves any error message returned by the > OS and I've found this useful in the past. Good pointer, thanks. ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Is it possible to use top directory of subtree with -Fu -Fi -Fl options?
> For example, could something equivalent to \progs\pascal\units\* be used to? (This I tried, it didn’t work.) It should. Try compiling with -vt, you'll see how the compiler looks for files. My fpc.cfg only has single line of -Fu, yet the unit directories are organized exactly the way your layout looks like. I even works for all my cross targets as well. -- View this message in context: http://free-pascal-general.1045716.n5.nabble.com/Is-it-possible-to-use-top-directory-of-subtree-with-Fu-Fi-Fl-options-tp5720176p5720182.html Sent from the Free Pascal - General mailing list archive at Nabble.com. ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] DLL calling Firebird 2: crashes at the end
On Sun, 28 Sep 2014, Reinier Olislagers wrote: On 28/09/2014 11:32, Michael Van Canneyt wrote: On Sun, 28 Sep 2014, Reinier Olislagers wrote: After hopefully getting parameter passing fixed (I hope, thanks Mark), I'm still facing this problem: dll using Firebird crashes on Windows and Linux Using a dummy db unit that does not connect to Firebird: everything ok. Perhaps a stupid question, but: why are you not simply using sqldb ? Why do you think I don't? Because you are speaking about cmem, low-level pchars and whatnot. My library does use sqldb. I just want to provide postcode lookup/address validation services to other programmers in a .dll/.so. Ok. I do a lot of work with firebird embedded, and I've never even come close to the problems you're experiencing ? It's not limited to Firebird embedded; I ran tests on Windows & Linux with Firebird client/server; same thing Could be. From your story, I doubt your problems are Firebird related. Firebird (or any other DB client lib) just uses the buffers you provide. Michael. ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
