Hmmm…I’m not seeing a failure. Let me try on another system.
Modifying libevent is not a viable solution :-(
> On Jan 15, 2015, at 10:26 AM, Leonid <lchis...@pathscale.com> wrote:
>
> Hi Ralph.
>
> Of course that may indicate an issue with custom compiler, but given that it
> fails with gcc and inserted delay I still think it is a OMPI bug, since such
> a delay could be caused by operating system at that exact point.
>
> For me simply commenting out "base->event_gotterm = base->event_break = 0;"
> seems to do the trick, but I am not completely sure if that won't cause any
> other troubles.
>
> I've tried to update my master branch to the latest version (including your
> fix) but now it just crashes for me on *all* benchmarks that I am trying
> (both with gcc and our compiler).
>
> On 15.01.2015 18:57, Ralph Castain wrote:
>> Thought about this some more and realized that the orte progress engine
>> wasn’t using the opal_progress_thread support functions, which include a
>> “break” event to kick us out of just such problems. So I changed it on the
>> master. From your citing of libevent 2.0.22, I believe that must be where
>> you are working, yes?
>>
>> If so, give the changed version a try and see if your problem is resolved.
>>
>>
>>> On Jan 15, 2015, at 12:55 AM, Ralph Castain <r...@open-mpi.org> wrote:
>>>
>>> Given that you could only reproduce it with either your custom compiler or
>>> by forcibly introducing a delay, is this indicating an issue with the
>>> custom compiler? It does seem strange that we don't see this anywhere else,
>>> given the number of times that code gets run.
>>>
>>> Only alternative solution I can think of would be to push the finalize
>>> request into the event loop, and thus execute the loopbreak from within an
>>> event. You might try and see if that solves the problem.
>>>
>>>
>>>> On Jan 14, 2015, at 11:54 PM, Leonid <lchis...@pathscale.com> wrote:
>>>>
>>>> Hi all.
>>>>
>>>> I believe there is a bug in event_base_loop() function from file event.c
>>>> (opal/mca/event/libevent2022/libevent/).
>>>>
>>>> Consider the case when application is going to be finalized and both
>>>> event_base_loop() and event_base_loopbreak() are called in the same time
>>>> in parallel threads.
>>>>
>>>> Then if event_base_loopbreak() happens to acquire lock first, it will set
>>>> "event_base->event_break = 1", but won't send any signal to event loop,
>>>> because it did not started yet.
>>>>
>>>> After that, event_base_loop() will acquire the lock and will clear
>>>> event_break flag with the following statement: "base->event_gotterm =
>>>> base->event_break = 0;". Then it will go into polling with timeout = -1
>>>> and thus block forever.
>>>>
>>>> This issue was reproduced on a custom compiler (using Lulesh benchmark and
>>>> x86 4-core PC), but it can be also reproduced for me with GCC compiler (on
>>>> almost any benchmark and in same HW settings) by putting some delay to
>>>> orte_progress_thread_engine() function:
>>>>
>>>> static void* orte_progress_thread_engine(opal_object_t *obj)
>>>> {
>>>> while (orte_event_base_active) {
>>>> usleep(1000); // add sleep to allow orte_ess_base_app_finalize() set
>>>> orte_event_base_active flag to false
>>>> opal_event_loop(orte_event_base, OPAL_EVLOOP_ONCE);
>>>> }
>>>> return OPAL_THREAD_CANCELLED;
>>>> }
>>>>
>>>> I am not completely sure what should be the best fix for described problem.
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> users mailing list
>>>> us...@open-mpi.org
>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>> Link to this post:
>>>> http://www.open-mpi.org/community/lists/users/2015/01/26181.php
>> _______________________________________________
>> users mailing list
>> us...@open-mpi.org
>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>> Link to this post:
>> http://www.open-mpi.org/community/lists/users/2015/01/26185.php
>
> _______________________________________________
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post:
> http://www.open-mpi.org/community/lists/users/2015/01/26188.php
