Hello collective wisdom,
ceph version 13.2.2 (02899bfda814146b021136e9d8e80eba494e1126) mimic (stable)
here.
I have a working cluster here consisting of 3 monitor hosts, 64 OSD processes
across 4 osd hosts, plus 2 MDSs, plus 2 MGRs. All of that is consumed by 10
client nodes.
Every host in the cluster, including clients is
RHEL 7.5
Mellanox OFED 4.4-2.0.7.0
RoCE NICs are either MCX416A-CCAT or MCX414A-CCAT @ 50Gbit/sec
The NICs are all mlx5_0 port 1
ring and ib_send_bw work fine both ways on any two nodes in the cluster.
Full configuration of the cluster is pasted below, but RDMA related parameters
are configured as following:
ms_public_type = async+rdma
ms_cluster = async+rdma
# Exclude clients for now
ms_type = async+posix
ms_async_rdma_device_name = mlx5_0
ms_async_rdma_polling_us = 0
ms_async_rdma_port_num=1
When I try to start MON, it immediately fails as below. Anybody has seen this
or could give any pointers what to/where to look next?
------ceph-mon.rio.log--begin------
2018-12-12 22:35:30.011 7f515dc39140 0 set uid:gid to 167:167 (ceph:ceph)
2018-12-12 22:35:30.011 7f515dc39140 0 ceph version 13.2.2
(02899bfda814146b021136e9d8e80eba494e1126) mimic (stable), process ceph-mon,
pid 2129843
2018-12-12 22:35:30.011 7f515dc39140 0 pidfile_write: ignore empty --pid-file
2018-12-12 22:35:30.036 7f515dc39140 0 load: jerasure load: lrc load: isa
2018-12-12 22:35:30.036 7f515dc39140 0 set rocksdb option compression =
kNoCompression
2018-12-12 22:35:30.036 7f515dc39140 0 set rocksdb option
level_compaction_dynamic_level_bytes = true
2018-12-12 22:35:30.036 7f515dc39140 0 set rocksdb option write_buffer_size =
33554432
2018-12-12 22:35:30.036 7f515dc39140 0 set rocksdb option compression =
kNoCompression
2018-12-12 22:35:30.036 7f515dc39140 0 set rocksdb option
level_compaction_dynamic_level_bytes = true
2018-12-12 22:35:30.036 7f515dc39140 0 set rocksdb option write_buffer_size =
33554432
2018-12-12 22:35:30.147 7f51442ed700 2 Event(0x55d927e95700 nevent=5000
time_id=1).set_owner idx=1 owner=139987012998912
2018-12-12 22:35:30.147 7f51442ed700 10 stack operator() starting
2018-12-12 22:35:30.147 7f5143aec700 2 Event(0x55d927e95200 nevent=5000
time_id=1).set_owner idx=0 owner=139987004606208
2018-12-12 22:35:30.147 7f5144aee700 2 Event(0x55d927e95c00 nevent=5000
time_id=1).set_owner idx=2 owner=139987021391616
2018-12-12 22:35:30.147 7f5143aec700 10 stack operator() starting
2018-12-12 22:35:30.147 7f5144aee700 10 stack operator() starting
2018-12-12 22:35:30.147 7f515dc39140 0 starting mon.rio rank 0 at public addr
192.168.1.58:6789/0 at bind addr 192.168.1.58:6789/0 mon_data
/var/lib/ceph/mon/ceph-rio fsid 376540c8-a362-41cc-9a58-9c8ceca0e4ee
2018-12-12 22:35:30.147 7f515dc39140 10 -- - bind bind 192.168.1.58:6789/0
2018-12-12 22:35:30.147 7f515dc39140 10 -- - bind Network Stack is not ready
for bind yet - postponed
2018-12-12 22:35:30.147 7f515dc39140 0 starting mon.rio rank 0 at
192.168.1.58:6789/0 mon_data /var/lib/ceph/mon/ceph-rio fsid
376540c8-a362-41cc-9a58-9c8ceca0e4ee
2018-12-12 22:35:30.148 7f515dc39140 0 mon.rio@-1(probing).mds e84 new map
2018-12-12 22:35:30.148 7f515dc39140 0 mon.rio@-1(probing).mds e84 print_map
e84
enable_multiple, ever_enabled_multiple: 0,0
compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable
ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses
versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout
v2,10=snaprealm v2}
legacy client fscid: -1
No filesystems configured
Standby daemons:
5906437: 192.168.1.152:6800/1077205146 'prince' mds.-1.0 up:standby seq 2
6284118: 192.168.1.59:6800/1266235911 'salvador' mds.-1.0 up:standby seq
2
2018-12-12 22:35:30.148 7f515dc39140 0 mon.rio@-1(probing).osd e25894 crush
map has features 288514051259236352, adjusting msgr requires
2018-12-12 22:35:30.148 7f515dc39140 0 mon.rio@-1(probing).osd e25894 crush
map has features 288514051259236352, adjusting msgr requires
2018-12-12 22:35:30.148 7f515dc39140 0 mon.rio@-1(probing).osd e25894 crush
map has features 1009089991638532096, adjusting msgr requires
2018-12-12 22:35:30.148 7f515dc39140 0 mon.rio@-1(probing).osd e25894 crush
map has features 288514051259236352, adjusting msgr requires
2018-12-12 22:35:30.149 7f515dc39140 10 -- - create_connect
192.168.1.88:6800/1638, creating connection and registering
2018-12-12 22:35:30.149 7f515dc39140 10 -- - >> 192.168.1.88:6800/1638
conn(0x55d9281fbe00 :-1 s=STATE_NONE pgs=0 cs=0 l=0)._connect csq=0
2018-12-12 22:35:30.149 7f515dc39140 10 -- - get_connection mgr.5894115
192.168.1.88:6800/1638 new 0x55d9281fbe00
2018-12-12 22:35:30.150 7f515dc39140 1 -- - --> 192.168.1.88:6800/1638 --
mgropen(unknown.rio) v3 -- 0x55d92844e000 con 0
2018-12-12 22:35:30.151 7f515dc39140 1 -- - start start
2018-12-12 22:35:30.151 7f515dc39140 1 -- - start start
2018-12-12 22:35:30.151 7f515dc39140 10 -- - ready -
2018-12-12 22:35:30.151 7f515dc39140 10 -- - bind bind 192.168.1.58:6789/0
2018-12-12 22:35:30.151 7f515dc39140 10 Processor -- bind
2018-12-12 22:35:30.154 7f5144aee700 1 Infiniband binding_port found active
port 1
2018-12-12 22:35:30.154 7f5144aee700 1 Infiniband init receive queue length is
4096 receive buffers
2018-12-12 22:35:30.154 7f5144aee700 1 Infiniband init assigning: 1024 send
buffers
2018-12-12 22:35:30.154 7f5144aee700 1 Infiniband init device allow 4194303
completion entries
2018-12-12 22:35:30.509 7f515dc39140 10 Processor -- bind bound to
192.168.1.58:6789/0
2018-12-12 22:35:30.509 7f515dc39140 1 -- 192.168.1.58:6789/0 learned_addr
learned my addr 192.168.1.58:6789/0
2018-12-12 22:35:30.509 7f515dc39140 1 -- 192.168.1.58:6789/0 _finish_bind
bind my_inst.addr is 192.168.1.58:6789/0
2018-12-12 22:35:30.510 7f515dc39140 10 -- - ready -
2018-12-12 22:35:30.510 7f515dc39140 1 Processor -- start
2018-12-12 22:35:30.510 7f515dc39140 0 mon.rio@-1(probing) e5 my rank is now
0 (was -1)
2018-12-12 22:35:30.510 7f515dc39140 1 -- 192.168.1.58:6789/0
shutdown_connections
2018-12-12 22:35:30.510 7f515dc39140 1 -- 192.168.1.58:6789/0 _send_message-->
mon.1 192.168.1.59:6789/0 -- mon_probe(probe
376540c8-a362-41cc-9a58-9c8ceca0e4ee name rio) v6 -- ?+0 0x55d928525680
2018-12-12 22:35:30.510 7f515dc39140 10 -- 192.168.1.58:6789/0 create_connect
192.168.1.59:6789/0, creating connection and registering
2018-12-12 22:35:30.510 7f515dc39140 10 -- 192.168.1.58:6789/0 >>
192.168.1.59:6789/0 conn(0x55d9281fc400 :-1 s=STATE_NONE pgs=0 cs=0
l=0)._connect csq=0
2018-12-12 22:35:30.510 7f515dc39140 1 -- 192.168.1.58:6789/0 -->
192.168.1.59:6789/0 -- mon_probe(probe 376540c8-a362-41cc-9a58-9c8ceca0e4ee
name rio) v6 -- 0x55d928525680 con 0
2018-12-12 22:35:30.510 7f515dc39140 1 -- 192.168.1.58:6789/0 _send_message-->
mon.2 192.168.1.65:6789/0 -- mon_probe(probe
376540c8-a362-41cc-9a58-9c8ceca0e4ee name rio) v6 -- ?+0 0x55d928525900
2018-12-12 22:35:30.510 7f515dc39140 10 -- 192.168.1.58:6789/0 create_connect
192.168.1.65:6789/0, creating connection and registering
2018-12-12 22:35:30.510 7f515dc39140 10 -- 192.168.1.58:6789/0 >>
192.168.1.65:6789/0 conn(0x55d9281fca00 :-1 s=STATE_NONE pgs=0 cs=0
l=0)._connect csq=0
2018-12-12 22:35:30.510 7f515dc39140 1 -- 192.168.1.58:6789/0 -->
192.168.1.65:6789/0 -- mon_probe(probe 376540c8-a362-41cc-9a58-9c8ceca0e4ee
name rio) v6 -- 0x55d928525900 con 0
2018-12-12 22:35:30.513 7f5143aec700 10 NetHandler generic_connect connect:
(111) Connection refused
2018-12-12 22:35:30.513 7f51442ed700 10 NetHandler generic_connect connect:
(111) Connection refused
2018-12-12 22:35:30.513 7f5144aee700 10 Infiniband send_msg sending: 0, 12894,
0, 0, fe800000000000007efe90fffe1e2524
2018-12-12 22:35:30.513 7f5143aec700 1 RDMAStack connect try connecting failed.
2018-12-12 22:35:30.513 7f51442ed700 1 RDMAStack connect try connecting failed.
2018-12-12 22:35:30.513 7f5144aee700 10 -- - >> 192.168.1.88:6800/1638
conn(0x55d9281fbe00 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0
l=0)._process_connection nonblock connect inprogress
2018-12-12 22:35:30.513 7f5143aec700 10 -- 192.168.1.58:6789/0 >>
192.168.1.59:6789/0 conn(0x55d9281fc400 :-1 s=STATE_CONNECTING pgs=0 cs=0
l=0).fault waiting 0.200000
2018-12-12 22:35:30.513 7f51442ed700 10 -- 192.168.1.58:6789/0 >>
192.168.1.65:6789/0 conn(0x55d9281fca00 :-1 s=STATE_CONNECTING pgs=0 cs=0
l=0).fault waiting 0.200000
2018-12-12 22:35:30.513 7f5144aee700 10 -- - >> 192.168.1.88:6800/1638
conn(0x55d9281fbe00 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=0).handle_write
2018-12-12 22:35:30.513 7f5143aec700 10 -- 192.168.1.58:6789/0 >>
192.168.1.59:6789/0 conn(0x55d9281fc400 :-1 s=STATE_CONNECTING pgs=0 cs=0
l=0).handle_write
2018-12-12 22:35:30.513 7f51442ed700 10 -- 192.168.1.58:6789/0 >>
192.168.1.65:6789/0 conn(0x55d9281fca00 :-1 s=STATE_CONNECTING pgs=0 cs=0
l=0).handle_write
2018-12-12 22:35:30.513 7f5144aee700 5 Infiniband recv_msg recevd: 105, 0,
262144, 0,
2018-12-12 22:35:30.513 7f5144aee700 -1 RDMAConnectedSocketImpl activate
failed to transition to RTR state: (113) No route to host
2018-12-12 22:35:30.515 7f5144aee700 -1
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.2/rpm/el7/BUILD/ceph-13.2.2/src/msg/async/rdma/RDMAConnectedSocketImpl.cc:
In function 'void RDMAConnectedSocketImpl::handle_connection()' thread
7f5144aee700 time 2018-12-12 22:35:30.514762
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.2/rpm/el7/BUILD/ceph-13.2.2/src/msg/async/rdma/RDMAConnectedSocketImpl.cc:
224: FAILED assert(!r)
ceph version 13.2.2 (02899bfda814146b021136e9d8e80eba494e1126) mimic (stable)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0xff)
[0x7f515506e6bf]
2: (()+0x285887) [0x7f515506e887]
3: (RDMAConnectedSocketImpl::handle_connection()+0x6f5) [0x7f51551b9655]
4: (EventCenter::process_events(unsigned int, std::chrono::duration<unsigned
long, std::ratio<1l, 1000000000l> >*)+0x695) [0x7f51551a76f5]
5: (()+0x3c15cc) [0x7f51551aa5cc]
6: (()+0x6afaef) [0x7f5155498aef]
7: (()+0x7e25) [0x7f5154396e25]
8: (clone()+0x6d) [0x7f5150cb7bad]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to
interpret this.
------ceph-mon.rio.log--end------
-----ceph.conf---begin-----
[client]
rbd_cache = False
rbd_cache_writethrough_until_flush = False
#admin socket = /var/run/ceph/$cluster-$type.$id.$pid.$cctid.asok
#log file = /var/log/ceph/
[global]
ms_type = async+posix
# set RDMA messaging just for the public or cluster network
ms_public_type = async+rdma
ms_cluster = async+rdma
#
# set a device name according to IB or ROCE device used, e.g.
ms_async_rdma_device_name = mlx5_0
#
# for better performance if using LUMINOUS 12.2.x release
ms_async_rdma_polling_us = 0
ms_async_rdma_port_num=1
rgw_override_bucket_index_max_shards=1
auth client required = none
auth cluster required = none
auth service required = none
auth supported = none
cephx require signatures = False
cephx sign messages = False
cluster network = 192.168.1.0/24
debug asok = 0/0
debug auth = 0/0
debug bluefs = 0/0
debug bluestore = 0/0
debug buffer = 0/0
debug client = 0/0
debug context = 0/0
debug crush = 0/0
debug filer = 0/0
debug filestore = 0/0
debug finisher = 0/0
#debug hadoop = 0/0 -- doesn't work in E8 setup
debug heartbeatmap = 0/0
debug journal = 0/0
debug journaler = 0/0
debug lockdep = 0/0
debug log = 0
debug mds = 0/0
debug mds_balancer = 0/0
debug mds_locker = 0/0
debug mds_log = 0/0
debug mds_log_expire = 0/0
debug mds_migrator = 0/0
debug mon = 0/0
debug monc = 0/0
debug ms = 10/10
#debug ms = 0/0
debug objclass = 0/0
debug objectcacher = 0/0
debug objecter = 0/0
debug optracker = 0/0
debug osd = 0/0
debug paxos = 0/0
debug perfcounter = 0/0
debug rados = 0/0
debug rbd = 0/0
debug rgw = 0/0
debug rocksdb = 0/0
debug throttle = 0/0
debug timer = 0/0
debug tp = 0/0
#debug zs = 0/0 -- doesn't work in E8 setup
fsid = 376540c8-a362-41cc-9a58-9c8ceca0e4ee
mon_host = 192.168.1.58,192.168.1.65,192.168.1.59
mon pg warn max per osd = 800
mon_allow_pool_delete = True
mon_max_pg_per_osd = 800
ms type = async
ms_crc_data = False
#ms_crc_header = False -- broken!
osd objectstore = bluestore
osd_pool_default_size = 2
perf = True
public network = 192.168.1.0/24
rocksdb_perf = True
# Parameter not present in Micron's config, but introduced by ceph-deploy
mon_initial_members = rio
# The following param claims to reduce CPU usage; found it at
# http://docs.ceph.com/docs/mimic/rados/configuration/ms-ref/
ms nocrc = true
#rbd_op_threads=4
[mon]
mon_max_pool_pg_num = 166496
mon_osd_max_split_count = 10000
[osd]
osd_min_pg_log_entries = 10
osd_max_pg_log_entries = 10
osd_pg_log_dups_tracked = 10
osd_pg_log_trim_min = 10
bluestore_cache_kv_max = 96G
bluestore_cache_kv_ratio = 0.2
bluestore_cache_meta_ratio = 0.8
bluestore_cache_size_ssd = 7G
bluestore_csum_type = none
bluestore_extent_map_shard_max_size = 200
bluestore_extent_map_shard_min_size = 50
bluestore_extent_map_shard_target_size = 100
bluestore_rocksdb_options =
compression=kNoCompression,max_write_buffer_number=64,min_write_buffer_number_to_merge=32,recycle_log_file_num=64,compaction_style=kCompactionStyleLevel,write_buffer_size=4MB,target_file_size_base=4MB,max_background_compactions=64,level0_file_num_compaction_trigger=64,level0_slowdown_writes_trigger=128,level0_stop_writes_trigger=256,max_bytes_for_level_base=6GB,compaction_threads=32,flusher_threads=8,compaction_readahead_size=2MB
[client.rgw.sm26]
rgw_frontends = "civetweb port=7480"
#[osd.0]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01ae
#
#[osd.1]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01ae
#
#[osd.2]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01ae
#
#[osd.3]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01ae
#
#[osd.4]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01ae
#
#[osd.5]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01ae
#
#[osd.6]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01ae
#
#[osd.7]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01ae
#
#[osd.8]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01ae
#
#[osd.9]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01ae
#
#[osd.10]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01ae
#
#[osd.11]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01ae
#
#[osd.12]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01ae
#
#[osd.13]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01ae
#
#[osd.14]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01ae
#
#[osd.15]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01ae
#
## BONJOVI1
#
#[osd.16]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01af
#
#[osd.17]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01af
#
#[osd.18]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01af
#
#[osd.19]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01af
#
#[osd.20]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01af
#
#[osd.21]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01af
#
#[osd.22]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01af
#
#[osd.23]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01af
#
#[osd.24]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01af
#
#[osd.25]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01af
#
#[osd.26]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01af
#
#[osd.27]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01af
#
#[osd.28]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01af
#
#[osd.29]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01af
#
#[osd.30]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01af
#
#[osd.31]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01af
#
## PRINCE
#
#[osd.32]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:0198
#
#[osd.33]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:0198
#[osd.34]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:0198
#
#[osd.35]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:0198
#
#[osd.36]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:0198
#
#[osd.37]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:0198
#
#[osd.38]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:0198
#
#[osd.39]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:0198
#
#[osd.40]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:0198
#
#[osd.41]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:0198
#
#[osd.42]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:0198
#
#[osd.43]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:0198
#
#[osd.44]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:0198
#
#[osd.45]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:0198
#
#[osd.46]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:0198
#
#[osd.47]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:0198
#
#
## RINGO
#
#[osd.48]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:018e
#
#[osd.49]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:018e
#
#[osd.50]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:018e
#
#[osd.51]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:018e
#
#[osd.52]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:018e
#
#[osd.53]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:018e
#
#[osd.54]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:018e
#
#[osd.55]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:018e
#
#[osd.56]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:018e
#
#[osd.57]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:018e
#
#[osd.58]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:018e
#
#[osd.59]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:018e
#
#[osd.60]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:018e
#
#[osd.61]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:018e
#
#[osd.62]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:018e
#
#[osd.63]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:018e
#
#[mon.rio]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:013a
#
#[mon.salvador]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:013b
#
#[mon.medellin]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:0141
-----ceph.conf---end-----
--
Michael Green
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
