So I've had a chance to re-visit this since Bécholey Alexandre was kind enough to let me know how to compile Ceph with the RDMACM library (thankyou again!).
At this stage it compiles and runs but there appears to be a problem with calling rshutdown in Pipe as it seems to just wait forever for the pipe to close which causes commands like 'ceph osd tree' to hang indefinitely after they work successfully. Debug MS is here - http://pastebin.com/WzMJNKZY I also tried RADOS bench but it appears to be doing something similar. Debug MS is here - http://pastebin.com/3aXbjzqS It seems like it's very close to working... I must be missing something small that's causing some grief. You can see the OSD coming up in the ceph monitor and the PG's all become active+clean. When shutting down the monitor I get the below which show's it waiting for the pipes to close - 2013-08-09 15:08:31.339394 7f4643cfd700 20 accepter.accepter closing 2013-08-09 15:08:31.382075 7f4643cfd700 10 accepter.accepter stopping 2013-08-09 15:08:31.382115 7f464bd397c0 20 -- 172.16.0.1:6789/0 wait: stopped accepter thread 2013-08-09 15:08:31.382127 7f464bd397c0 20 -- 172.16.0.1:6789/0 wait: stopping reaper thread 2013-08-09 15:08:31.382146 7f4645500700 10 -- 172.16.0.1:6789/0reaper_entry done 2013-08-09 15:08:31.382182 7f464bd397c0 20 -- 172.16.0.1:6789/0 wait: stopped reaper thread 2013-08-09 15:08:31.382194 7f464bd397c0 10 -- 172.16.0.1:6789/0 wait: closing pipes 2013-08-09 15:08:31.382200 7f464bd397c0 10 -- 172.16.0.1:6789/0 reaper 2013-08-09 15:08:31.382205 7f464bd397c0 10 -- 172.16.0.1:6789/0 reaper done 2013-08-09 15:08:31.382210 7f464bd397c0 10 -- 172.16.0.1:6789/0 wait: waiting for pipes 0x3014c80,0x3015180,0x3015400 to close The git repo has been updated if anyone has a few spare minutes to take a look - https://github.com/funkBuild/ceph-rsockets Thanks again -Matt On Thu, Jun 20, 2013 at 5:09 PM, Matthew Anderson <manderson8...@gmail.com>wrote: > Hi All, > > I've had a few conversations on IRC about getting RDMA support into Ceph > and thought I would give it a quick attempt to hopefully spur some > interest. What I would like to accomplish is an RSockets only > implementation so I'm able to use Ceph, RBD and QEMU at full speed over an > Infiniband fabric. > > What I've tried to do is port Pipe.cc and Acceptor.cc to rsockets by > replacing the regular socket calls with the rsocket equivalent. > Unfortunately it doesn't compile and I get an error of - > > CXXLD ceph-osd > ./.libs/libglobal.a(libcommon_la-Accepter.o): In function > `Accepter::stop()': > /home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Accepter.cc:243: undefined > reference to `rshutdown' > /home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Accepter.cc:251: undefined > reference to `rclose' > ./.libs/libglobal.a(libcommon_la-Accepter.o): In function > `Accepter::entry()': > /home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Accepter.cc:213: undefined > reference to `raccept' > /home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Accepter.cc:230: undefined > reference to `rclose' > ./.libs/libglobal.a(libcommon_la-Accepter.o): In function > `Accepter::bind(entity_addr_t const&, int, int)': > /home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Accepter.cc:61: undefined > reference to `rsocket' > /home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Accepter.cc:80: undefined > reference to `rsetsockopt' > /home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Accepter.cc:87: undefined > reference to `rbind' > /home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Accepter.cc:118: undefined > reference to `rgetsockname' > /home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Accepter.cc:128: undefined > reference to `rlisten' > /home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Accepter.cc:100: undefined > reference to `rbind' > /home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Accepter.cc:87: undefined > reference to `rbind' > ./.libs/libglobal.a(libcommon_la-Pipe.o): In function > `Pipe::tcp_write(char const*, int)': > /home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Pipe.cc:2175: undefined > reference to `rsend' > /home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Pipe.cc:2162: undefined > reference to `rshutdown' > ./.libs/libglobal.a(libcommon_la-Pipe.o): In function > `Pipe::do_sendmsg(msghdr*, int, bool)': > /home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Pipe.cc:1867: undefined > reference to `rsendmsg' > ./.libs/libglobal.a(libcommon_la-Pipe.o): In function > `Pipe::tcp_read_nonblocking(char*, int)': > /home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Pipe.cc:2129: undefined > reference to `rrecv' > ./.libs/libglobal.a(libcommon_la-Pipe.o): In function > `Pipe::tcp_read(char*, int)': > /home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Pipe.cc:2079: undefined > reference to `rshutdown' > ./.libs/libglobal.a(libcommon_la-Pipe.o): In function `Pipe::connect()': > /home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Pipe.cc:768: undefined > reference to `rclose' > /home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Pipe.cc:773: undefined > reference to `rsocket' > /home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Pipe.cc:781: undefined > reference to `rconnect' > ./.libs/libglobal.a(libcommon_la-Pipe.o): In function `Pipe::writer()': > /home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Pipe.cc:1471: undefined > reference to `rwrite' > collect2: error: ld returned 1 exit status > make[3]: *** [ceph-mon] Error 1 > > > > From the looks of it I need to include the 'rdma/rsocket.h' library > somewhere else or add librdmacm but I'm not sure where. > > Full disclaimer, I am terrible at C++. If anyone has a few spare minutes > to have a look into the error messages and can point out where I've gone > wrong it would be greatly appreciated. > > I've put the code up at - https://github.com/funkBuild/ceph-rsockets > > Thanks again > -Matt > > >
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
