> > Switching to SRQ and some guess of queue values selected appears to let the > code run. > S,4096,128:S,12288,128:S,65536,12 > > Two questions, > > This is a ConnectX fabric, should I switch them to XRC queues? And should I > use the same queue size/count? That a safe assumption? > X,4096,128:X,12288,128:X,65536,12
Yeah, I would use the same values as a starting point. > > > When should I use one queue type over the other? Generally speaking XRC transport has much better scalability that RC. > > Is there a way to get stat feedback on the use of your shared queues (SRQ or > XRC) ? > > Example, using code 'not from here' and would like to know, "hey you are > always running out of your queue of size X" Or " your queue of size Y is > never used" > > We are kinda blind for a lot of our applications :-) Right now we don't have such hooks in openib BTL. It is not very difficult to add some code that will report stat for QP utilization. In you other email you mentioned MXM. I would recommend to try both XRC and MXM and see which one performance better. On relatively small system I would guess XRC will perform better, on large system MXM should demonstrate better performance. But again, it all depends on your application. - Pasha
