On 02/05/2013, at 11:37 PM, Xavier Lashmar <xlash...@uottawa.ca> wrote:
> Ah, finally got it. Can you go to frame 3 (up <ret> up <ret> up <ret>) and run print input print input+100 print input+200 ...etc... until you reach the end of the string? Then I'll be able to reproduce (and fix) locally. > > Core was generated by `/usr/libexec/pacemaker/crmd'. > Program terminated with signal 6, Aborted. > #0 0x00007f81896ac8a5 in raise (sig=6) at > ../nptl/sysdeps/unix/sysv/linux/raise.c:64 > 64 return INLINE_SYSCALL (tgkill, 3, pid, selftid, sig); > Missing separate debuginfos, use: debuginfo-install > libtool-ltdl-2.2.6-15.5.el6.x86_64 > (gdb) bt > #0 0x00007f81896ac8a5 in raise (sig=6) at > ../nptl/sysdeps/unix/sysv/linux/raise.c:64 > #1 0x00007f81896ae085 in abort () at abort.c:92 > #2 0x00007f818bb8a56b in crm_abort (file=0x7f818bba9d58 "xml.c", > function=0x7f818bbab6b4 "string2xml", line=650, > assert_condition=0x7f818bbaa01a "String parsing error", do_core=<value > optimized out>, do_fork=<value optimized out>) at utils.c:1073 > #3 0x00007f818bb933af in string2xml ( > input=0x1e745f8 "<lrmd_notify lrmd_origin=\"send_cmd_complete_notify\" > lrmd_timeout=\"30000\" lrmd_rsc_interval=\"15000\" > lrmd_rsc_start_delay=\"15000\" lrmd_exec_rc=\"0\" lrmd_exec_op_status=\"1\" > lrmd_callid=\"2747\" lrmd_rsc_del"...) at xml.c:650 > #4 0x00007f818b76a2fc in lrmd_ipc_dispatch (buffer=<value optimized out>, > length=<value optimized out>, userdata=0x1e72910) at lrmd_client.c:310 > #5 0x00007f818bba2e90 in mainloop_gio_callback (gio=<value optimized out>, > condition=G_IO_IN, data=0x1e73be0) at mainloop.c:585 > #6 0x00007f8188fbbf0e in g_main_dispatch (context=0x1d4f120) at gmain.c:1960 > #7 IA__g_main_context_dispatch (context=0x1d4f120) at gmain.c:2513 > #8 0x00007f8188fbf938 in g_main_context_iterate (context=0x1d4f120, block=1, > dispatch=1, self=<value optimized out>) at gmain.c:2591 > #9 0x00007f8188fbfd55 in IA__g_main_loop_run (loop=0x1e734a0) at gmain.c:2799 > #10 0x00000000004052ce in crmd_init () at main.c:154 > #11 0x00000000004055cc in main (argc=1, argv=0x7fffe77a4f88) at main.c:120 > > > Xavier Lashmar > Analyste de Systèmes | Systems Analyst > Service étudiants, service de l'informatique et des communications/Student > services, computing and communications services. > 1 Nicholas Street (810) > Ottawa ON K1N 7B7 > Tél. | Tel. 613-562-5800 (2120) > > > > > -----Original Message----- > From: Andrew Beekhof [mailto:and...@beekhof.net] > Sent: Wednesday, May 1, 2013 7:07 PM > To: The Pacemaker cluster resource manager > Subject: Re: [Pacemaker] Pacemaker core dumps > > > On 01/05/2013, at 11:36 PM, Xavier Lashmar <xlash...@uottawa.ca> wrote: > >> I'm not sure if anyone has run into this issue but I can't seem to >> find a debuginfo package for one of the libraries for CentOS 6.3 with >> Kernel 2.6.32-279.9.1el6.x86_64 : libtool-ltdl >> >> Here's what I get so far from the core dump, but I think it's incomplete: >> >> ... >> ... >> ... >> Reading symbols from /lib64/libfreebl3.so... >> warning: the debug information found in >> "/usr/lib/debug//lib64/libfreebl3.so.debug" does not match >> "/lib64/libfreebl3.so" (CRC mismatch). >> >> warning: the debug information found in >> "/usr/lib/debug/lib64/libfreebl3.so.debug" does not match >> "/lib64/libfreebl3.so" (CRC mismatch). >> >> Missing separate debuginfo for /lib64/libfreebl3.so >> Try: yum --disablerepo='*' --enablerepo='*-debug*' install >> /usr/lib/debug/.build-id/68/195872ecfb188389d29aaf01031a976fd18168.deb >> ug >> (no debugging symbols found)...done. >> Loaded symbols for /lib64/libfreebl3.so Reading symbols from >> /lib64/libnss_files-2.12.so...Reading symbols from >> /usr/lib/debug/lib64/libnss_files-2.12.so.debug...done. >> done. >> Loaded symbols for /lib64/libnss_files-2.12.so Core was generated by >> `/usr/libexec/pacemaker/crmd'. >> Program terminated with signal 6, Aborted. >> #0 0x00007f81896ac8a5 in raise (sig=6) at >> ../nptl/sysdeps/unix/sysv/linux/raise.c:64 >> 64 return INLINE_SYSCALL (tgkill, 3, pid, selftid, sig); >> Missing separate debuginfos, use: debuginfo-install >> libtool-ltdl-2.2.6-15.5.el6.x86_64 >> >> Any info about either finding the right debuginfo files, or about the error >> itself would be greatly appreciated. > > The libtool parts aren't so interesting. > Were there no other frames? (lines starting with # and a number) > >> >> Xavier Lashmar >> Analyste de Systèmes | Systems Analyst Service étudiants, service de >> l'informatique et des communications/Student services, computing and >> communications services. >> 1 Nicholas Street (810) >> Ottawa ON K1N 7B7 >> Tél. | Tel. 613-562-5800 (2120) >> >> >> >> >> -----Original Message----- >> From: Andrew Beekhof [mailto:and...@beekhof.net] >> Sent: Monday, April 29, 2013 11:00 PM >> To: The Pacemaker cluster resource manager >> Subject: Re: [Pacemaker] Pacemaker core dumps >> >> >> On 30/04/2013, at 1:32 AM, Xavier Lashmar <xlash...@uottawa.ca> wrote: >> >>> Hello Andrew, >>> >>> Thanks for your help. We've upgrade to pacemaker 1.1.9 and still have the >>> same issue. >> >> Thats a disappointing but useful data point. >> >>> >>> We are trying to get the core information but we are missing some debuginfo >>> files which we are trying to get our hands on. I'll try to forward this >>> information soon. >> >> Great >> >>> >>> Is there something we need to do to the CIB when we upgrade? >> >> No, anything that needs to happen will be done under the hood. >> >>> >>> >>> Xavier Lashmar >>> Analyste de Systèmes | Systems Analyst Service étudiants, service de >>> l'informatique et des communications/Student services, computing and >>> communications services. >>> 1 Nicholas Street (810) >>> Ottawa ON K1N 7B7 >>> Tél. | Tel. 613-562-5800 (2120) >>> >>> >>> >>> -----Original Message----- >>> From: Andrew Beekhof [mailto:and...@beekhof.net] >>> Sent: Thursday, April 25, 2013 8:15 PM >>> To: The Pacemaker cluster resource manager >>> Subject: Re: [Pacemaker] Pacemaker core dumps >>> >>> >>> On 26/04/2013, at 10:06 AM, Andrew Beekhof <and...@beekhof.net> wrote: >>> >>>> >>>> On 25/04/2013, at 11:59 PM, Xavier Lashmar <xlash...@uottawa.ca> wrote: >>>> >>>>> Following further investigation, we were able to determine that upgrading >>>>> both nodes (in a two node cluster) from Pacemaker 1.1.7-6 to Pacemaker >>>>> 1.1.8-7 (CentOS 6.3 or Centos 6.4) caused these errors to begin happening: >>>> >>>> Would you be able to try the 1.1.9 packages from >>>> http://www.clusterlabs.org/rpm-next to see if they are also affected? >>>> >>>>> >>>>> We were able to replicate the initiation of the errors by upgrading >>>>> another cluster in the same manner. This other cluster is now >>>>> experiencing the same core-dumping and errors as the previous cluster: >>>>> >>>>> Apr 25 09:46:22 xxxx crmd[1764]: error: crm_xml_err: XML Error: >>>>> Entity: line 1: parser error : invalid character in attribute value >>>>> Apr 25 09:46:22 xxxx crmd[1764]: error: crm_xml_err: XML Error: >>>>> a-72fc-47e1-81b4-51b500c967f9" lrmd_rsc_output="tomcat6 (pid 3282) is >>>>> running... >>>>> Apr 25 09:46:22 xxxx crmd[1764]: error: crm_xml_err: XML Error: >>>>> ^ >>>>> Apr 25 09:46:22 xxxx crmd[1764]: error: crm_xml_err: XML Error: >>>>> Entity: line 1: parser error : attributes construct error >>>>> Apr 25 09:46:22 xxxx crmd[1764]: error: crm_xml_err: XML Error: >>>>> a-72fc-47e1-81b4-51b500c967f9" lrmd_rsc_output="tomcat6 (pid 3282) is >>>>> running... >>>>> Apr 25 09:46:22 xxxx crmd[1764]: error: crm_xml_err: XML Error: >>>>> ^ >>>>> Apr 25 09:46:22 xxxx crmd[1764]: error: crm_xml_err: XML Error: >>>>> Entity: line 1: parser error : Couldn't find end of Start Tag lrmd_notify >>>>> line 1 >>>>> Apr 25 09:46:22 xxxx crmd[1764]: error: crm_xml_err: XML Error: >>>>> a-72fc-47e1-81b4-51b500c967f9" lrmd_rsc_output="tomcat6 (pid 3282) is >>>>> running... >>>>> Apr 25 09:46:22 xxxx crmd[1764]: error: crm_xml_err: XML Error: >>>>> ^ >>>>> Apr 25 09:46:22 xxxx crmd[1764]: error: crm_xml_err: XML Error: >>>>> Entity: line 1: parser error : Extra content at the end of the document >>>>> Apr 25 09:46:22 xxxx crmd[1764]: error: crm_xml_err: XML Error: >>>>> a-72fc-47e1-81b4-51b500c967f9" lrmd_rsc_output="tomcat6 (pid 3282) is >>>>> running... >>>>> Apr 25 09:46:22 xxxx crmd[1764]: error: crm_xml_err: XML Error: >>>>> ^ >>>>> Apr 25 09:46:22 xxxx crmd[1764]: warning: string2xml: Parsing >>>>> failed (domain=1, level=3, code=5): Extra content at the end of the >>>>> document Apr 25 09:46:22 xxxx crmd[1764]: warning: string2xml: String >>>>> start: >>>>> <lrmd_notify lrmd_origin="send_cmd_complete_notify >>>>> Apr 25 09:46:22 xxxx crmd[1764]: warning: string2xml: String start+688: >>>>> 0000" CRM_meta_start_delay="15000"/></lrmd_notify> >>>>> Apr 25 09:46:22 xxxx crmd[1764]: error: crm_abort: string2xml: Forked >>>>> child 4182 to record non-fatal assert at xml.c:605 : String parsing error >>> >>> Also, it would be very useful if you could open up the core file for >>> 4182 and print the contents of the input passed to string2xml() >>> _______________________________________________ >>> Pacemaker mailing list: Pacemaker@oss.clusterlabs.org >>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker >>> >>> Project Home: http://www.clusterlabs.org Getting started: >>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf >>> Bugs: http://bugs.clusterlabs.org >>> >>> _______________________________________________ >>> Pacemaker mailing list: Pacemaker@oss.clusterlabs.org >>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker >>> >>> Project Home: http://www.clusterlabs.org Getting started: >>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf >>> Bugs: http://bugs.clusterlabs.org >> >> >> _______________________________________________ >> Pacemaker mailing list: Pacemaker@oss.clusterlabs.org >> http://oss.clusterlabs.org/mailman/listinfo/pacemaker >> >> Project Home: http://www.clusterlabs.org Getting started: >> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf >> Bugs: http://bugs.clusterlabs.org >> >> _______________________________________________ >> Pacemaker mailing list: Pacemaker@oss.clusterlabs.org >> http://oss.clusterlabs.org/mailman/listinfo/pacemaker >> >> Project Home: http://www.clusterlabs.org Getting started: >> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf >> Bugs: http://bugs.clusterlabs.org > > > _______________________________________________ > Pacemaker mailing list: Pacemaker@oss.clusterlabs.org > http://oss.clusterlabs.org/mailman/listinfo/pacemaker > > Project Home: http://www.clusterlabs.org Getting started: > http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf > Bugs: http://bugs.clusterlabs.org > > _______________________________________________ > Pacemaker mailing list: Pacemaker@oss.clusterlabs.org > http://oss.clusterlabs.org/mailman/listinfo/pacemaker > > Project Home: http://www.clusterlabs.org > Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf > Bugs: http://bugs.clusterlabs.org _______________________________________________ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
