It was a firewall issue on the controller nodes.After allowing ceph-mgr port in iptables everything is displaying correctly.Thanks to people on IRC.
Thanks alot, Kevin On Thu, Dec 21, 2017 at 5:24 PM, kevin parrikar <kevin.parker...@gmail.com> wrote: > accidently removed mailing list email > > ++ceph-users > > Thanks a lot JC for looking into this issue. I am really out of ideas. > > > ceph.conf on mgr node which is also monitor node. > > [global] > fsid = 06c5c906-fc43-499f-8a6f-6c8e21807acf > mon_initial_members = node-16 node-30 node-31 > mon_host = 172.16.1.9 172.16.1.3 172.16.1.11 > auth_cluster_required = cephx > auth_service_required = cephx > auth_client_required = cephx > filestore_xattr_use_omap = true > log_to_syslog_level = info > log_to_syslog = True > osd_pool_default_size = 2 > osd_pool_default_min_size = 1 > osd_pool_default_pg_num = 64 > public_network = 172.16.1.0/24 > log_to_syslog_facility = LOG_LOCAL0 > osd_journal_size = 2048 > auth_supported = cephx > osd_pool_default_pgp_num = 64 > osd_mkfs_type = xfs > cluster_network = 172.16.1.0/24 > osd_recovery_max_active = 1 > osd_max_backfills = 1 > mon allow pool delete = true > > [client] > rbd_cache_writethrough_until_flush = True > rbd_cache = True > > [client.radosgw.gateway] > rgw_keystone_accepted_roles = _member_, Member, admin, swiftoperator > keyring = /etc/ceph/keyring.radosgw.gateway > rgw_frontends = fastcgi socket_port=9000 socket_host=127.0.0.1 > rgw_socket_path = /tmp/radosgw.sock > rgw_keystone_revocation_interval = 1000000 > rgw_keystone_url = http://192.168.1.3:35357 > rgw_keystone_admin_token = jaJSmlTNxgsFp1ttq5SuAT1R > rgw_init_timeout = 360000 > host = controller3 > rgw_dns_name = *.sapiennetworks.com > rgw_print_continue = True > rgw_keystone_token_cache_size = 10 > rgw_data = /var/lib/ceph/radosgw > user = www-data > > > > > ceph auth list > > > osd.100 > key: AQAtZjpaVZOFBxAAwl0yFLdUOidLzPFjv+HnjA== > caps: [mgr] allow profile osd > caps: [mon] allow profile osd > caps: [osd] allow * > osd.101 > key: AQA4ZjpaS4wwGBAABwgoXQRc1J8sav4MUkWceQ== > caps: [mgr] allow profile osd > caps: [mon] allow profile osd > caps: [osd] allow * > osd.102 > key: AQBDZjpaBS2tEBAAtFiPKBzh8JGi8Nh3PtAGCg== > caps: [mgr] allow profile osd > caps: [mon] allow profile osd > caps: [osd] allow * > > client.admin > key: AQD0yXFYflnYFxAAEz/2XLHO/6RiRXQ5HXRAnw== > caps: [mds] allow * > caps: [mgr] allow * > caps: [mon] allow * > caps: [osd] allow * > client.backups > key: AQC0y3FY4YQNNhAAs5fludq0yvtp/JJt7RT4HA== > caps: [mgr] allow r > caps: [mon] allow r > caps: [osd] allow class-read object_prefix rbd_children, allow rwx > pool=backups, allow rwx pool=volumes > client.bootstrap-mds > key: AQD5yXFYyIxiFxAAyoqLPnxxqWmUr+zz7S+qVQ== > caps: [mgr] allow r > caps: [mon] allow profile bootstrap-mds > client.bootstrap-mgr > key: AQBmOTpaXqHQDhAAyDXoxlPmG9QovfmmUd8gIg== > caps: [mon] allow profile bootstrap-mgr > client.bootstrap-osd > key: AQD0yXFYuGkSIhAAelSb3TCPuXRFoFJTBh7Vdg== > caps: [mgr] allow r > caps: [mon] allow profile bootstrap-osd > client.bootstrap-rbd > key: AQBnOTpafDS/IRAAnKzuI9AYEF81/6mDVv0QgQ== > caps: [mon] allow profile bootstrap-rbd > > client.bootstrap-rgw > key: AQD3yXFYxt1mLRAArxOgRvWmmzT9pmsqTLpXKw== > caps: [mgr] allow r > caps: [mon] allow profile bootstrap-rgw > client.compute > key: AQCbynFYRcNWOBAAPzdAKfP21GvGz1VoHBimGQ== > caps: [mgr] allow r > caps: [mon] allow r > caps: [osd] allow class-read object_prefix rbd_children, allow rwx > pool=volumes, allow rx pool=images, allow rwx pool=compute > client.images > key: AQCyy3FYSMtlJRAAbJ8/U/R82NXvWBC5LmkPGw== > caps: [mgr] allow r > caps: [mon] allow r > caps: [osd] allow class-read object_prefix rbd_children, allow rwx > pool=images > client.radosgw.gateway > key: AQA3ynFYAYMSAxAApvfe/booa9KhigpKpLpUOA== > caps: [mgr] allow r > caps: [mon] allow rw > caps: [osd] allow rwx > client.volumes > key: AQCzy3FYa3paKBAA9BlYpQ1PTeR770ghVv1jKQ== > caps: [mgr] allow r > caps: [mon] allow r > caps: [osd] allow class-read object_prefix rbd_children, allow rwx > pool=volumes, allow rx pool=images > mgr.controller2 > key: AQAmVTpaA+9vBhAApD3rMs//Qri+SawjUF4U4Q== > caps: [mds] allow * > caps: [mgr] allow * > caps: [mon] allow * > caps: [osd] allow * > mgr.controller3 > key: AQByfDparprIEBAAj7Pxdr/87/v0kmJV49aKpQ== > caps: [mds] allow * > caps: [mgr] allow * > caps: [mon] allow * > caps: [osd] allow * > > Regards, > Kevin > > On Thu, Dec 21, 2017 at 8:10 AM, kevin parrikar <kevin.parker...@gmail.com > > wrote: > >> Thanks JC, >> I tried >> ceph auth caps client.admin osd 'allow *' mds 'allow *' mon 'allow *' mgr >> 'allow *' >> >> but still status is same,also mgr.log is being flooded with below errors. >> >> 2017-12-21 02:39:10.622834 7fb40a22b700 0 Cannot get stat of OSD 140 >> 2017-12-21 02:39:10.622835 7fb40a22b700 0 Cannot get stat of OSD 141 >> Not sure whats wrong in my setup >> >> Regards, >> Kevin >> >> >> On Thu, Dec 21, 2017 at 2:37 AM, Jean-Charles Lopez <jelo...@redhat.com> >> wrote: >> >>> Hi, >>> >>> make sure client.admin user has an MGR cap using ceph auth list. At some >>> point there was a glitch with the update process that was not adding the >>> MGR cap to the client.admin user. >>> >>> JC >>> >>> >>> On Dec 20, 2017, at 10:02, kevin parrikar <kevin.parker...@gmail.com> >>> wrote: >>> >>> hi All, >>> I have upgraded the cluster from Hammer to Jewel and to Luminous . >>> >>> i am able to upload/download glance images but ceph -s shows 0kb used >>> and Available and probably because of that cinder create is failing. >>> >>> >>> ceph -s >>> cluster: >>> id: 06c5c906-fc43-499f-8a6f-6c8e21807acf >>> health: HEALTH_WARN >>> Reduced data availability: 6176 pgs inactive >>> Degraded data redundancy: 6176 pgs unclean >>> >>> services: >>> mon: 3 daemons, quorum controller3,controller2,controller1 >>> mgr: controller3(active) >>> osd: 71 osds: 71 up, 71 in >>> rgw: 1 daemon active >>> >>> data: >>> pools: 4 pools, 6176 pgs >>> objects: 0 objects, 0 bytes >>> usage: 0 kB used, 0 kB / 0 kB avail >>> pgs: 100.000% pgs unknown >>> 6176 unknown >>> >>> >>> i deployed ceph-mgr using ceph-deploy gather-keys && ceph-deploy mgr >>> create ,it was successfull but for some reason ceph -s is not showing >>> correct values. >>> Can some one help me here please >>> >>> Regards, >>> Kevin >>> _______________________________________________ >>> ceph-users mailing list >>> ceph-users@lists.ceph.com >>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>> >>> >>> >> >
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
