Hi,
I keep getting a vapi error when running any NAS benchmark on Infiniband
with mpi_leave_pinned option set to 1. This problem does not occur with
mpi_leave_pinned set to 0. Please find attached the error trace.
Thanks,
Aniruddha
----------------------------------------------------------------------------------
Aniruddha Shet | Project webpage: http://forge-fre.ornl.gov/molar/index.html
Graduate Research Associate | Project webpage: www.cs.unm.edu/~fastos
Dept. of Comp. Sci. & Engg | Personal webpage: www.cse.ohio-state.edu/~shet
The Ohio State University | Office: DL 474
2015 Neil Avenue | Phone: +1 (614) 292 7036
Columbus OH 43210-1277 | Cell: +1 (614) 446 1630
----------------------------------------------------------------------------------
[0,1,2][btl_mvapi_component.c:666:mca_btl_mvapi_component_progress] Got error :
VAPI_WR_FLUSH_ERR, Vendor code : 249 Frag :
0x4118903c[0,1,2][btl_mvapi_component.c:666:mca_btl_mvapi_component_progress]
Got error : VAPI_WR_FLUSH_ERR, Vendor code : 0 Frag :
0x41199110[0,1,2][btl_mvapi_component.c:666:mca_btl_mvapi_component_progress]
Got error : VAPI_WR_FLUSH_ERR, Vendor code : 0 Frag :
0x411a91e4[0,1,2][btl_mvapi_component.c:666:mca_btl_mvapi_component_progress]
Got error : VAPI_WR_FLUSH_ERR, Vendor code : 0 Frag :
0x411b92b8[0,1,2][btl_mvapi_component.c:666:mca_btl_mvapi_component_progress]
Got error : VAPI_WR_FLUSH_ERR, Vendor code : 0 Frag :
0x411c938c[0,1,2][btl_mvapi_component.c:666:mca_btl_mvapi_component_progress]
Got error : VAPI_WR_FLUSH_ERR, Vendor code : 0 Frag :
0x411d9460[0,1,2][btl_mvapi_component.c:666:mca_btl_mvapi_component_progress]
Got error : VAPI_WR_FLUSH_ERR, Vendor code : 0 Frag :
0x411e9534[0,1,1][btl_mvapi_component.c:666:mca_btl_mvapi_component_progress]
[0,1,1][btl_mvapi.c:735:async_event_handler] Got an asynchronous event:
VAPI_LOCAL_WQ_ACCESS_VIOL_ERROR (VAPI_EV_SYNDROME_NONE)Got error :
VAPI_REM_ACCESS_ERR, Vendor code : 136 Frag :
0x9f893e0[0,1,1][btl_mvapi_component.c:666:mca_btl_mvapi_component_progress]
Got error : VAPI_WR_FLUSH_ERR, Vendor code : 0 Frag :
0x9f89308[0,1,1][btl_mvapi_component.c:666:mca_btl_mvapi_component_progress]
[0,1,2][btl_mvapi_component.c:666:mca_btl_mvapi_component_progress] Got error :
VAPI_WR_FLUSH_ERR, Vendor code : 0 Frag :
0x411f9608[0,1,2][btl_mvapi_component.c:666:mca_btl_mvapi_component_progress]
Got error : VAPI_WR_FLUSH_ERR, Vendor code : 0 Frag :
0x412096dc[0,1,2][btl_mvapi_component.c:666:mca_btl_mvapi_component_progress]
Got error : VAPI_WR_FLUSH_ERR, Vendor code : 0 Frag :
0x412197b0[0,1,2][btl_mvapi_component.c:666:mca_btl_mvapi_component_progress]
Got error : VAPI_WR_FLUSH_ERR, Vendor code : 0 Frag :
0x41229884[0,1,2][btl_mvapi_component.c:666:mca_btl_mvapi_component_progress]
Got error : VAPI_WR_FLUSH_ERR, Vendor code : 0 Frag :
0x41239958[0,1,2][btl_mvapi_component.c:666:mca_btl_mvapi_component_progress]
Got error : VAPI_WR_FLUSH_ERR, Vendor code : 0 Frag : 0x41249a2cGot error :
VAPI_WR_FLUSH_ERR, Vendor code : 249 Frag :
0x4118903c[0,1,1][btl_mvapi_component.c:666:mca_btl_mvapi_component_progress]
Got error : VAPI_WR_FLUSH_ERR, Vendor code : 0 Frag :
0x41199110[0,1,1][btl_mvapi_component.c:666:mca_btl_mvapi_component_progress]
Got error : VAPI_WR_FLUSH_ERR, Vendor code : 0 Frag :
0x411a91e4[0,1,1][btl_mvapi_component.c:666:mca_btl_mvapi_component_progress]
Got error : VAPI_WR_FLUSH_ERR, Vendor code : 0 Frag :
0x411b92b8[0,1,1][btl_mvapi_component.c:666:mca_btl_mvapi_component_progress]
Got error : VAPI_WR_FLUSH_ERR, Vendor code : 0 Frag :
0x411c938c[0,1,1][btl_mvapi_component.c:666:mca_btl_mvapi_component_progress]
Got error : VAPI_WR_FLUSH_ERR, Vendor code : 0 Frag :
0x411d9460[0,1,1][btl_mvapi_component.c:666:mca_btl_mvapi_component_progress]
[0,1,2][btl_mvapi_component.c:666:mca_btl_mvapi_component_progress] Got error :
VAPI_WR_FLUSH_ERR, Vendor code : 0 Frag :
0x41259b00[0,1,2][btl_mvapi_component.c:666:mca_btl_mvapi_component_progress]
Got error : VAPI_WR_FLUSH_ERR, Vendor code : 0 Frag :
0x41269bd4[0,1,2][btl_mvapi_component.c:666:mca_btl_mvapi_component_progress]
Got error : VAPI_WR_FLUSH_ERR, Vendor code : 0 Frag : 0x41279ca8Got error :
VAPI_WR_FLUSH_ERR, Vendor code : 0 Frag :
0x411e9534[0,1,1][btl_mvapi_component.c:666:mca_btl_mvapi_component_progress]
Got error : VAPI_WR_FLUSH_ERR, Vendor code : 0 Frag :
0x411f9608[0,1,1][btl_mvapi_component.c:666:mca_btl_mvapi_component_progress]
Got error : VAPI_WR_FLUSH_ERR, Vendor code : 0 Frag :
0x412096dc[0,1,2][btl_mvapi_component.c:666:mca_btl_mvapi_component_progress]
Got error : VAPI_WR_FLUSH_ERR, Vendor code : 0 Frag :
0x41289d7c[0,1,2][btl_mvapi_component.c:666:mca_btl_mvapi_component_progress]
Got error : VAPI_WR_FLUSH_ERR, Vendor code : 0 Frag :
0x41299e50[0,1,2][btl_mvapi_component.c:666:mca_btl_mvapi_component_progress]
Got error : VAPI_WR_FLUSH_ERR, Vendor code : 0 Frag :
0x412a9f24[0,1,1][btl_mvapi_component.c:666:mca_btl_mvapi_component_progress]
Got error : VAPI_WR_FLUSH_ERR, Vendor code : 0 Frag :
0x412197b0[0,1,1][btl_mvapi_component.c:666:mca_btl_mvapi_component_progress]
Got error : VAPI_WR_FLUSH_ERR, Vendor code : 0 Frag :
0x41229884[0,1,1][btl_mvapi_component.c:666:mca_btl_mvapi_component_progress]
Got error : VAPI_WR_FLUSH_ERR, Vendor code : 0 Frag :
0x41239958[0,1,1][btl_mvapi_component.c:666:mca_btl_mvapi_component_progress]
Got error : VAPI_WR_FLUSH_ERR, Vendor code : 0 Frag :
0x41249a2c[0,1,1][btl_mvapi_component.c:666:mca_btl_mvapi_component_progress]
Got error : VAPI_WR_FLUSH_ERR, Vendor code : 0 Frag :
0x41259b00[0,1,1][btl_mvapi_component.c:666:mca_btl_mvapi_component_progress]
Got error : VAPI_WR_FLUSH_ERR, Vendor code : 0 Frag :
0x41269bd4[0,1,1][btl_mvapi_component.c:666:mca_btl_mvapi_component_progress]
Got error : VAPI_WR_FLUSH_ERR, Vendor code : 0 Frag :
0x41279ca8[0,1,1][btl_mvapi_component.c:666:mca_btl_mvapi_component_progress]
Got error : VAPI_WR_FLUSH_ERR, Vendor code : 0 Frag :
0x41289d7c[0,1,1][btl_mvapi_component.c:666:mca_btl_mvapi_component_progress]
Got error : VAPI_WR_FLUSH_ERR, Vendor code : 0 Frag :
0x41299e50[0,1,2][btl_mvapi_component.c:666:mca_btl_mvapi_component_progress]
Got error : VAPI_WR_FLUSH_ERR, Vendor code : 244 Frag :
0x9f89590[0,1,2][btl_mvapi_component.c:666:mca_btl_mvapi_component_progress]
Got error : VAPI_WR_FLUSH_ERR, Vendor code : 0 Frag :
0x9f89158[0,1,1][btl_mvapi_component.c:666:mca_btl_mvapi_component_progress]
Got error : VAPI_WR_FLUSH_ERR, Vendor code : 0 Frag :
0x412a9f24[0,1,2][btl_mvapi.c:735:async_event_handler] Got an asynchronous
event: VAPI_LOCAL_WQ_ACCESS_VIOL_ERROR
(VAPI_EV_SYNDROME_NONE)[0,1,0][btl_mvapi.c:735:async_event_handler] Got an
asynchronous event: VAPI_LOCAL_WQ_ACCESS_VIOL_ERROR
(VAPI_EV_SYNDROME_NONE)[0,1,0][btl_mvapi_component.c:666:mca_btl_mvapi_component_progress]
Got error : VAPI_WR_FLUSH_ERR, Vendor code : 249 Frag :
0x4118903c[0,1,3][btl_mvapi_component.c:666:mca_btl_mvapi_component_progress]
Got error : VAPI_REM_ACCESS_ERR, Vendor code : 136 Frag :
0x9fa1080[0,1,3][btl_mvapi_component.c:666:mca_btl_mvapi_component_progress]
Got error : VAPI_WR_FLUSH_ERR, Vendor code : 0 Frag :
0x9fa1158[0,1,3][btl_mvapi_component.c:666:mca_btl_mvapi_component_progress]
Got error : VAPI_WR_FLUSH_ERR, Vendor code : 244 Frag :
0x41058080[0,1,3][btl_mvapi_component.c:666:mca_btl_mvapi_component_progress]
Got error : VAPI_WR_FLUSH_ERR, Vendor code : 0 Frag :
0x41068154[0,1,3][btl_mvapi_component.c:666:mca_btl_mvapi_component_progress]
Got error : VAPI_WR_FLUSH_ERR, Vendor code : 0 Frag :
0x41078228[0,1,3][btl_mvapi_component.c:666:mca_btl_mvapi_component_progress]
Got error : VAPI_WR_FLUSH_ERR, Vendor code : 0 Frag :
0x410882fc[0,1,3][btl_mvapi_component.c:666:mca_btl_mvapi_component_progress]
Got error : VAPI_WR_FLUSH_ERR, Vendor code : 0 Frag :
0x410983d0[0,1,3][btl_mvapi_component.c:666:mca_btl_mvapi_component_progress]
Got error : VAPI_WR_FLUSH_ERR, Vendor code : 0 Frag :
0x410a84a4[0,1,3][btl_mvapi_component.c:666:mca_btl_mvapi_component_progress]
Got error : VAPI_WR_FLUSH_ERR, Vendor code : 0 Frag :
0x410b8578[0,1,3][btl_mvapi_component.c:666:mca_btl_mvapi_component_progress]
Got error : VAPI_WR_FLUSH_ERR, Vendor code : 0 Frag :
0x410c864c[0,1,3][btl_mvapi_component.c:666:mca_btl_mvapi_component_progress]
Got error : VAPI_WR_FLUSH_ERR, Vendor code : 0 Frag :
0x410d8720[0,1,3][btl_mvapi_component.c:666:mca_btl_mvapi_component_progress]
Got error : VAPI_WR_FLUSH_ERR, Vendor code : 0 Frag :
0x410e87f4[0,1,3][btl_mvapi_component.c:666:mca_btl_mvapi_component_progress]
Got error : VAPI_WR_FLUSH_ERR, Vendor code : 0 Frag :
0x410f88c8[0,1,3][btl_mvapi_component.c:666:mca_btl_mvapi_component_progress]
Got error : VAPI_WR_FLUSH_ERR, Vendor code : 0 Frag :
0x4110899c[0,1,3][btl_mvapi_component.c:666:mca_btl_mvapi_component_progress]
Got error : VAPI_WR_FLUSH_ERR, Vendor code : 0 Frag :
0x41118a70[0,1,3][btl_mvapi_component.c:666:mca_btl_mvapi_component_progress]
Got error : VAPI_WR_FLUSH_ERR, Vendor code : 0 Frag :
0x41128b44[0,1,0][btl_mvapi_component.c:666:mca_btl_mvapi_component_progress]
Got error : VAPI_WR_FLUSH_ERR, Vendor code : 0 Frag :
0x41199110[0,1,3][btl_mvapi_component.c:666:mca_btl_mvapi_component_progress]
Got error : VAPI_WR_FLUSH_ERR, Vendor code : 0 Frag :
0x41138c18[0,1,3][btl_mvapi_component.c:666:mca_btl_mvapi_component_progress]
Got error : VAPI_WR_FLUSH_ERR, Vendor code : 0 Frag :
0x41148cec[0,1,3][btl_mvapi_component.c:666:mca_btl_mvapi_component_progress]
Got error : VAPI_WR_FLUSH_ERR, Vendor code : 0 Frag :
0x41158dc0[0,1,3][btl_mvapi_component.c:666:mca_btl_mvapi_component_progress]
Got error : VAPI_WR_FLUSH_ERR, Vendor code : 0 Frag :
0x41168e94[0,1,3][btl_mvapi_component.c:666:mca_btl_mvapi_component_progress]
Got error : VAPI_WR_FLUSH_ERR, Vendor code : 0 Frag :
0x41178f68[0,1,0][btl_mvapi_component.c:666:mca_btl_mvapi_component_progress]
Got error : VAPI_WR_FLUSH_ERR, Vendor code : 0 Frag :
0x411a91e4[0,1,0][btl_mvapi_component.c:666:mca_btl_mvapi_component_progress]
Got error : VAPI_WR_FLUSH_ERR, Vendor code : 0 Frag :
0x411b92b8[0,1,0][btl_mvapi_component.c:666:mca_btl_mvapi_component_progress]
Got error : VAPI_WR_FLUSH_ERR, Vendor code : 0 Frag :
0x411c938c[0,1,0][btl_mvapi_component.c:666:mca_btl_mvapi_component_progress]
Got error : VAPI_WR_FLUSH_ERR, Vendor code : 0 Frag :
0x411d9460[0,1,0][btl_mvapi_component.c:666:mca_btl_mvapi_component_progress]
Got error : VAPI_WR_FLUSH_ERR, Vendor code : 0 Frag :
0x411e9534[0,1,0][btl_mvapi_component.c:666:mca_btl_mvapi_component_progress]
Got error : VAPI_WR_FLUSH_ERR, Vendor code : 0 Frag :
0x411f9608[0,1,0][btl_mvapi_component.c:666:mca_btl_mvapi_component_progress]
Got error : VAPI_WR_FLUSH_ERR, Vendor code : 0 Frag :
0x412096dc[0,1,0][btl_mvapi_component.c:666:mca_btl_mvapi_component_progress]
Got error : VAPI_WR_FLUSH_ERR, Vendor code : 0 Frag :
0x412197b0[0,1,0][btl_mvapi_component.c:666:mca_btl_mvapi_component_progress]
Got error : VAPI_WR_FLUSH_ERR, Vendor code : 0 Frag :
0x41229884[0,1,0][btl_mvapi_component.c:666:mca_btl_mvapi_component_progress]
Got error : VAPI_WR_FLUSH_ERR, Vendor code : 0 Frag :
0x41239958[0,1,0][btl_mvapi_component.c:666:mca_btl_mvapi_component_progress]
Got error : VAPI_WR_FLUSH_ERR, Vendor code : 0 Frag :
0x41249a2c[0,1,0][btl_mvapi_component.c:666:mca_btl_mvapi_component_progress]
Got error : VAPI_WR_FLUSH_ERR, Vendor code : 0 Frag :
0x41259b00[0,1,0][btl_mvapi_component.c:666:mca_btl_mvapi_component_progress]
Got error : VAPI_WR_FLUSH_ERR, Vendor code : 0 Frag :
0x41269bd4[0,1,0][btl_mvapi_component.c:666:mca_btl_mvapi_component_progress]
Got error : VAPI_WR_FLUSH_ERR, Vendor code : 0 Frag :
0x41279ca8[0,1,0][btl_mvapi_component.c:666:mca_btl_mvapi_component_progress]
Got error : VAPI_WR_FLUSH_ERR, Vendor code : 0 Frag :
0x41289d7c[0,1,0][btl_mvapi_component.c:666:mca_btl_mvapi_component_progress]
Got error : VAPI_WR_FLUSH_ERR, Vendor code : 0 Frag :
0x41299e50[0,1,0][btl_mvapi_component.c:666:mca_btl_mvapi_component_progress]
Got error : VAPI_WR_FLUSH_ERR, Vendor code : 0 Frag :
0x412a9f24[0,1,0][btl_mvapi_component.c:666:mca_btl_mvapi_component_progress]
Got error : VAPI_WR_FLUSH_ERR, Vendor code : 249 Frag :
0x9f89308[0,1,0][btl_mvapi_component.c:666:mca_btl_mvapi_component_progress]
Got error : VAPI_WR_FLUSH_ERR, Vendor code : 249 Frag : 0x9f89590
