Re: [ceph-users] typo in news for PG auto-scaler

[Junk]

also 
$ ceph osd pool set foo pg_autoscaler_mode on

pg_autoscaleR_mode should be 

$ ceph osd pool set foo pg_autoscale_mode on

On Fri, 2019-04-05 at 08:05 +0200, Lars Täuber wrote:
> Hi everybody!
> There is a small mistake in the news about the PG autoscaler
> https://ceph.com/rados/new-in-nautilus-pg-merging-and-autotuning/
> 
> The command $ ceph osd pool set foo target_ratio .8should actually be
> $ ceph osd pool set foo target_size_ratio .8
> 
> Thanks for this great improvement!
> Cheers,Lars___ceph-users
> mailing listceph-us...@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] bluefs-bdev-expand experience

[Yury Shevchuk]

Hello all!

We have a toy 3-node Ceph cluster running Luminous 12.2.11 with one
bluestore osd per node.  We started with pretty small OSDs and would
like to be able to expand OSDs whenever needed.  We had two issues
with the expansion: one turned out user-serviceable while the other
probably needs developers' look.  I will describe both shortly.

Round 1
~~~
Trying to expand osd.2 by 1TB:

  # lvextend -L+1T /dev/vg0/osd2
Size of logical volume vg0/osd2 changed from 232.88 GiB (59618 extents) to 
1.23 TiB (321762 extents).
Logical volume vg0/osd2 successfully resized.

  # ceph-bluestore-tool bluefs-bdev-expand --path /var/lib/ceph/osd/ceph-2
  inferring bluefs devices from bluestore path
   slot 1 /var/lib/ceph/osd/ceph-2//block
  1 : size 0x13a3880 : own 0x[1bf220~25430]
  Expanding...
  1 : can't be expanded. Bypassing...
  #

It didn't work.  The explaination can be found in
ceph/src/os/bluestore/BlueFS.cc at line 310:

  // returns true if specified device is under full bluefs control
  // and hence can be expanded
  bool BlueFS::is_device_expandable(unsigned id)
  {
if (id >= MAX_BDEV || bdev[id] == nullptr) {
  return false;
}
switch(id) {
case BDEV_WAL:
  return true;

case BDEV_DB:
  // true if DB volume is non-shared
  return bdev[BDEV_SLOW] != nullptr;
}
return false;
  }

So we have to use separate block.db and block.wal for OSD to be
expandable.  Indeed, our OSDs were created without separate block.db
and block.wal, like this:

  ceph-volume lvm create --bluestore --data /dev/vg0/osd2

Recreating osd.2 with separate block.db and block.wal:

  # ceph-volume lvm zap --destroy --osd-id 2
  # lvcreate -L1G -n osd2wal vg0
Logical volume "osd2wal" created.
  # lvcreate -L40G -n osd2db vg0
Logical volume "osd2db" created.
  # lvcreate -L400G -n osd2 vg0
Logical volume "osd2" created.
  # ceph-volume lvm create --osd-id 2 --bluestore --data vg0/osd2 --block.db 
vg0/osd2db --block.wal vg0/osd2wal

Resync takes some time, and then we have expandable osd.2.


Round 2
~~~
Trying to expand osd.2 from 400G to 700G:

  # lvextend -L+300G /dev/vg0/osd2
Size of logical volume vg0/osd2 changed from 400.00 GiB (102400 extents) to 
700.00 GiB (179200 extents).
Logical volume vg0/osd2 successfully resized.

  # ceph-bluestore-tool bluefs-bdev-expand --path /var/lib/ceph/osd/ceph-2/
  inferring bluefs devices from bluestore path
   slot 0 /var/lib/ceph/osd/ceph-2//block.wal
   slot 1 /var/lib/ceph/osd/ceph-2//block.db
   slot 2 /var/lib/ceph/osd/ceph-2//block
  0 : size 0x4000 : own 0x[1000~3000]
  1 : size 0xa : own 0x[2000~9e000]
  2 : size 0xaf : own 0x[30~4]
  Expanding...
  #


This time expansion appears to work: 0xaf = 700GiB.

However, the size in the block device label have not changed:

  # ceph-bluestore-tool show-label --dev /dev/vg0/osd2
  {
  "/dev/vg0/osd2": {
  "osd_uuid": "a18ff7f7-0de1-4791-ba4b-f3b6d2221f44",
  "size": 429496729600,

429496729600 = 400GiB

Worse, ceph osd df shows the added space as used, not available:

# ceph osd df
ID CLASS WEIGHT  REWEIGHT SIZEUSE AVAIL  %USE  VAR  PGS
 0   hdd 0.22739  1.0  233GiB 8.06GiB 225GiB  3.46 0.13 128
 1   hdd 0.22739  1.0  233GiB 8.06GiB 225GiB  3.46 0.13 128
 2   hdd 0.22739  1.0  700GiB  301GiB 399GiB 43.02 1.58  64
TOTAL 1.14TiB  317GiB 849GiB 27.21
MIN/MAX VAR: 0.13/1.58  STDDEV: 21.43

If I expand osd.2 by another 100G, the space also goes to
"USE" column:

pier42:~# ceph osd df
ID CLASS WEIGHT  REWEIGHT SIZEUSE AVAIL  %USE  VAR  PGS
 0   hdd 0.22739  1.0  233GiB 8.05GiB 225GiB  3.46 0.10 128
 1   hdd 0.22739  1.0  233GiB 8.05GiB 225GiB  3.46 0.10 128
 3   hdd 0.227390  0B  0B 0B 00   0
 2   hdd 0.22739  1.0  800GiB  408GiB 392GiB 51.01 1.52 128
TOTAL 1.24TiB  424GiB 842GiB 33.51
MIN/MAX VAR: 0.10/1.52  STDDEV: 26.54

So OSD expansion almost works, but not quite.  If you had better luck
with bluefs-bdev-expand, could you please share your story?


-- Yury
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] bluefs-bdev-expand experience

[Igor Fedotov]

Hi Yuri,

wrt Round 1 - an ability to expand block(main) device has been added to 
Nautilus,


see: https://github.com/ceph/ceph/pull/25308


wrt Round 2:

- not setting 'size' label looks like a bug although I recall I fixed 
it... Will double check.


- wrong stats output is probably related to the lack of monitor restart 
- could you please try that and report back if it helps? Or even restart 
the whole cluster.. (well I understand that's a bad approach for 
production but just to verify my hypothesis)



Thanks,

Igor


On 4/5/2019 2:06 PM, Yury Shevchuk wrote:

Hello all!

We have a toy 3-node Ceph cluster running Luminous 12.2.11 with one
bluestore osd per node.  We started with pretty small OSDs and would
like to be able to expand OSDs whenever needed.  We had two issues
with the expansion: one turned out user-serviceable while the other
probably needs developers' look.  I will describe both shortly.

Round 1
~~~
Trying to expand osd.2 by 1TB:

   # lvextend -L+1T /dev/vg0/osd2
 Size of logical volume vg0/osd2 changed from 232.88 GiB (59618 extents) to 
1.23 TiB (321762 extents).
 Logical volume vg0/osd2 successfully resized.

   # ceph-bluestore-tool bluefs-bdev-expand --path /var/lib/ceph/osd/ceph-2
   inferring bluefs devices from bluestore path
slot 1 /var/lib/ceph/osd/ceph-2//block
   1 : size 0x13a3880 : own 0x[1bf220~25430]
   Expanding...
   1 : can't be expanded. Bypassing...
   #

It didn't work.  The explaination can be found in
ceph/src/os/bluestore/BlueFS.cc at line 310:

   // returns true if specified device is under full bluefs control
   // and hence can be expanded
   bool BlueFS::is_device_expandable(unsigned id)
   {
 if (id >= MAX_BDEV || bdev[id] == nullptr) {
   return false;
 }
 switch(id) {
 case BDEV_WAL:
   return true;

 case BDEV_DB:
   // true if DB volume is non-shared
   return bdev[BDEV_SLOW] != nullptr;
 }
 return false;
   }

So we have to use separate block.db and block.wal for OSD to be
expandable.  Indeed, our OSDs were created without separate block.db
and block.wal, like this:

   ceph-volume lvm create --bluestore --data /dev/vg0/osd2

Recreating osd.2 with separate block.db and block.wal:

   # ceph-volume lvm zap --destroy --osd-id 2
   # lvcreate -L1G -n osd2wal vg0
 Logical volume "osd2wal" created.
   # lvcreate -L40G -n osd2db vg0
 Logical volume "osd2db" created.
   # lvcreate -L400G -n osd2 vg0
 Logical volume "osd2" created.
   # ceph-volume lvm create --osd-id 2 --bluestore --data vg0/osd2 --block.db 
vg0/osd2db --block.wal vg0/osd2wal

Resync takes some time, and then we have expandable osd.2.


Round 2
~~~
Trying to expand osd.2 from 400G to 700G:

   # lvextend -L+300G /dev/vg0/osd2
 Size of logical volume vg0/osd2 changed from 400.00 GiB (102400 extents) 
to 700.00 GiB (179200 extents).
 Logical volume vg0/osd2 successfully resized.

   # ceph-bluestore-tool bluefs-bdev-expand --path /var/lib/ceph/osd/ceph-2/
   inferring bluefs devices from bluestore path
slot 0 /var/lib/ceph/osd/ceph-2//block.wal
slot 1 /var/lib/ceph/osd/ceph-2//block.db
slot 2 /var/lib/ceph/osd/ceph-2//block
   0 : size 0x4000 : own 0x[1000~3000]
   1 : size 0xa : own 0x[2000~9e000]
   2 : size 0xaf : own 0x[30~4]
   Expanding...
   #


This time expansion appears to work: 0xaf = 700GiB.

However, the size in the block device label have not changed:

   # ceph-bluestore-tool show-label --dev /dev/vg0/osd2
   {
   "/dev/vg0/osd2": {
   "osd_uuid": "a18ff7f7-0de1-4791-ba4b-f3b6d2221f44",
   "size": 429496729600,

429496729600 = 400GiB

Worse, ceph osd df shows the added space as used, not available:

# ceph osd df
ID CLASS WEIGHT  REWEIGHT SIZEUSE AVAIL  %USE  VAR  PGS
  0   hdd 0.22739  1.0  233GiB 8.06GiB 225GiB  3.46 0.13 128
  1   hdd 0.22739  1.0  233GiB 8.06GiB 225GiB  3.46 0.13 128
  2   hdd 0.22739  1.0  700GiB  301GiB 399GiB 43.02 1.58  64
 TOTAL 1.14TiB  317GiB 849GiB 27.21
MIN/MAX VAR: 0.13/1.58  STDDEV: 21.43

If I expand osd.2 by another 100G, the space also goes to
"USE" column:

pier42:~# ceph osd df
ID CLASS WEIGHT  REWEIGHT SIZEUSE AVAIL  %USE  VAR  PGS
  0   hdd 0.22739  1.0  233GiB 8.05GiB 225GiB  3.46 0.10 128
  1   hdd 0.22739  1.0  233GiB 8.05GiB 225GiB  3.46 0.10 128
  3   hdd 0.227390  0B  0B 0B 00   0
  2   hdd 0.22739  1.0  800GiB  408GiB 392GiB 51.01 1.52 128
 TOTAL 1.24TiB  424GiB 842GiB 33.51
MIN/MAX VAR: 0.10/1.52  STDDEV: 26.54

So OSD expansion almost works, but not quite.  If you had better luck
with bluefs-bdev-expand, could you please share your story?


-- Yury
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

__

Re: [ceph-users] bluefs-bdev-expand experience

[Yury Shevchuk]

On Fri, Apr 05, 2019 at 02:42:53PM +0300, Igor Fedotov wrote:
> wrt Round 1 - an ability to expand block(main) device has been added to
> Nautilus,
>
> see: https://github.com/ceph/ceph/pull/25308

Oh, that's good.  But still separate wal&db may be good for studying
load on each volume (blktrace) or moving db/wal to another physical
disk by means of LVM transparently to ceph.

> wrt Round 2:
>
> - not setting 'size' label looks like a bug although I recall I fixed it...
> Will double check.
>
> - wrong stats output is probably related to the lack of monitor restart -
> could you please try that and report back if it helps? Or even restart the
> whole cluster.. (well I understand that's a bad approach for production but
> just to verify my hypothesis)

Mon restart didn't help:

node0:~# systemctl restart ceph-mon@0
node1:~# systemctl restart ceph-mon@1
node2:~# systemctl restart ceph-mon@2
node2:~# ceph osd df
ID CLASS WEIGHT  REWEIGHT SIZEUSE AVAIL  %USE  VAR  PGS
 0   hdd 0.22739  1.0  233GiB 9.44GiB 223GiB  4.06 0.12 128
 1   hdd 0.22739  1.0  233GiB 9.44GiB 223GiB  4.06 0.12 128
 3   hdd 0.227390  0B  0B 0B 00   0
 2   hdd 0.22739  1.0  800GiB  409GiB 391GiB 51.18 1.51 128
TOTAL 1.24TiB  428GiB 837GiB 33.84
MIN/MAX VAR: 0.12/1.51  STDDEV: 26.30

Restarting mgrs and then all ceph daemons on all three nodes didn't
help either:

node2:~# ceph osd df
ID CLASS WEIGHT  REWEIGHT SIZEUSE AVAIL  %USE  VAR  PGS
 0   hdd 0.22739  1.0  233GiB 9.43GiB 223GiB  4.05 0.12 128
 1   hdd 0.22739  1.0  233GiB 9.43GiB 223GiB  4.05 0.12 128
 3   hdd 0.227390  0B  0B 0B 00   0
 2   hdd 0.22739  1.0  800GiB  409GiB 391GiB 51.18 1.51 128
TOTAL 1.24TiB  428GiB 837GiB 33.84
MIN/MAX VAR: 0.12/1.51  STDDEV: 26.30

Maybe we should upgrade to v14.2.0 Nautilus instead of studying old
bugs... after all, this is a toy cluster for now.

Thank you for responding,


-- Yury

> On 4/5/2019 2:06 PM, Yury Shevchuk wrote:
> > Hello all!
> >
> > We have a toy 3-node Ceph cluster running Luminous 12.2.11 with one
> > bluestore osd per node.  We started with pretty small OSDs and would
> > like to be able to expand OSDs whenever needed.  We had two issues
> > with the expansion: one turned out user-serviceable while the other
> > probably needs developers' look.  I will describe both shortly.
> >
> > Round 1
> > ~~~
> > Trying to expand osd.2 by 1TB:
> >
> ># lvextend -L+1T /dev/vg0/osd2
> >  Size of logical volume vg0/osd2 changed from 232.88 GiB (59618 
> > extents) to 1.23 TiB (321762 extents).
> >  Logical volume vg0/osd2 successfully resized.
> >
> ># ceph-bluestore-tool bluefs-bdev-expand --path /var/lib/ceph/osd/ceph-2
> >inferring bluefs devices from bluestore path
> > slot 1 /var/lib/ceph/osd/ceph-2//block
> >1 : size 0x13a3880 : own 0x[1bf220~25430]
> >Expanding...
> >1 : can't be expanded. Bypassing...
> >#
> >
> > It didn't work.  The explaination can be found in
> > ceph/src/os/bluestore/BlueFS.cc at line 310:
> >
> >// returns true if specified device is under full bluefs control
> >// and hence can be expanded
> >bool BlueFS::is_device_expandable(unsigned id)
> >{
> >  if (id >= MAX_BDEV || bdev[id] == nullptr) {
> >return false;
> >  }
> >  switch(id) {
> >  case BDEV_WAL:
> >return true;
> >
> >  case BDEV_DB:
> >// true if DB volume is non-shared
> >return bdev[BDEV_SLOW] != nullptr;
> >  }
> >  return false;
> >}
> >
> > So we have to use separate block.db and block.wal for OSD to be
> > expandable.  Indeed, our OSDs were created without separate block.db
> > and block.wal, like this:
> >
> >ceph-volume lvm create --bluestore --data /dev/vg0/osd2
> >
> > Recreating osd.2 with separate block.db and block.wal:
> >
> ># ceph-volume lvm zap --destroy --osd-id 2
> ># lvcreate -L1G -n osd2wal vg0
> >  Logical volume "osd2wal" created.
> ># lvcreate -L40G -n osd2db vg0
> >  Logical volume "osd2db" created.
> ># lvcreate -L400G -n osd2 vg0
> >  Logical volume "osd2" created.
> ># ceph-volume lvm create --osd-id 2 --bluestore --data vg0/osd2 
> > --block.db vg0/osd2db --block.wal vg0/osd2wal
> >
> > Resync takes some time, and then we have expandable osd.2.
> >
> >
> > Round 2
> > ~~~
> > Trying to expand osd.2 from 400G to 700G:
> >
> ># lvextend -L+300G /dev/vg0/osd2
> >  Size of logical volume vg0/osd2 changed from 400.00 GiB (102400 
> > extents) to 700.00 GiB (179200 extents).
> >  Logical volume vg0/osd2 successfully resized.
> >
> ># ceph-bluestore-tool bluefs-bdev-expand --path /var/lib/ceph/osd/ceph-2/
> >inferring bluefs devices from bluestore path
> > slot 0 /var/lib/ceph/osd/ceph-2//block.wal
> > slot 1 /var/lib/ceph/osd/ceph-2//block.db
> > slot 2 /var/lib/ceph/osd/ceph-2//block
> >

[ceph-users] unable to turn on pg_autoscale

[Daniele Riccucci]

Hello,
I'm running a (very) small cluster and I'd like to turn on pg_autoscale.
In the documentation here > 
http://docs.ceph.com/docs/nautilus/rados/operations/placement-groups/ it 
tells me that running


ceph config set global osd_pool_default_autoscale_mode 

should enable this by default, however running the below:

$ ceph config help osd_pool_default_autoscale_mode
Error ENOENT:

seems to tell me that the setting isn't correct.
By grepping around I found osd_pool_default_pg_autoscale_mode (with _pg 
after default) to be the correct setting, however it seems this can't be 
changed at runtime as docs suggest:


$ ceph config help osd_pool_default_pg_autoscale_mode
osd_pool_default_pg_autoscale_mode - Default PG autoscaling behavior for 
new pools

  (str, advanced)
  Default: warn
  Possible values:  off warn on
  Can update at runtime: false

So after enabling this via ceph-ansible by overriding the conf I tried 
to set it also on existing pools.

This is the output before my attempt:

$ ceph osd pool autoscale-status -f json-pretty | jq '[ .[] | 
{'pool_name': .pool_name, 'pg_autoscale_mode': .pg_autoscale_mode }]'


[
  {
"pool_name": "cephfs_metadata",
"pg_autoscale_mode": "warn"
  },
  {
"pool_name": "default.rgw.meta",
"pg_autoscale_mode": "warn"
  },
  {
"pool_name": "cephfs_data",
"pg_autoscale_mode": "warn"
  },
  {
"pool_name": "default.rgw.control",
"pg_autoscale_mode": "warn"
  },
  {
"pool_name": ".rgw.root",
"pg_autoscale_mode": "warn"
  },
  {
"pool_name": "default.rgw.log",
"pg_autoscale_mode": "warn"
  }
]

Then I tried executing the following:

$ docker exec -it ceph-mon-sagittarius ceph osd pool set cephfs_metadata
pg_autoscale_mode on
set pool 2 pg_autoscale_mode to on

However, while it seems the operation was successful, the following 
command returns:


$ docker exec -it ceph-mon-sagittarius ceph osd pool get cephfs_metadata 
pg_autoscale_mode

pg_autoscale_mode: warn


And since pg_autoscaler module is turned on, this is the output of ceph 
status:


ceph status
  cluster:
id: d5c50302-0d8e-47cb-ab86-c15842372900
health: HEALTH_WARN
5 pools have too many placement groups

  services:
mon: 1 daemons, quorum sagittarius (age 5m)
mgr: sagittarius(active, since 2m)
mds: cephfs:1 {0=sagittarius=up:active}
osd: 4 osds: 4 up, 4 in
rgw: 1 daemon active (sagittarius.rgw0)

  data:
pools:   6 pools, 184 pgs
objects: 242.08k objects, 862 GiB
usage:   1.7 TiB used, 27 TiB / 29 TiB avail
pgs: 184 active+clean

What am I doing wrong?
Thank you.

Daniele
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] RGW: Reshard index of non-master zones in multi-site

[Casey Bodley]

Hi Iain,

Resharding is not supported in multisite. The issue is that the master zone
needs to be authoritative for all metadata. If bucket reshard commands run
on the secondary zone, they create new bucket instance metadata that the
master zone never sees, so replication can't reconcile those changes.

The 'stale-instances rm' command is not safe to run in multisite because it
can misidentify as 'stale' some bucket instances that were deleted on the
master zone, where data sync on the secondary zone hasn't yet finished
deleting all of the objects it contained. Deleting these bucket instances
and their associated bucket index objects would leave any remaining objects
behind as orphans and leak storage capacity.

On Thu, Apr 4, 2019 at 3:28 PM Iain Buclaw  wrote:

> On Wed, 3 Apr 2019 at 09:41, Iain Buclaw  wrote:
> >
> > On Tue, 19 Feb 2019 at 10:11, Iain Buclaw  wrote:
> > >
> > >
> > > # ./radosgw-gc-bucket-indexes.sh master.rgw.buckets.index | wc -l
> > > 7511
> > >
> > > # ./radosgw-gc-bucket-indexes.sh secondary1.rgw.buckets.index | wc -l
> > > 3509
> > >
> > > # ./radosgw-gc-bucket-indexes.sh secondary2.rgw.buckets.index | wc -l
> > > 3801
> > >
> >
> > Documentation is a horrid mess around the subject on multi-site
> resharding
> >
> >
> http://docs.ceph.com/docs/mimic/radosgw/dynamicresharding/#manual-bucket-resharding
> >
> >
> https://www.suse.com/documentation/suse-enterprise-storage-5/book_storage_admin/data/ogw_bucket_sharding.html
> > (Manual Resharding)
> >
> >
> https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/3/html-single/object_gateway_guide_for_red_hat_enterprise_linux/index#manually-resharding-buckets-with-multisite-rgw
> >
> > All disagree with each other over the correct process to reshard
> > indexes in multi-site.  Worse, none of them seem to work correctly
> > anyway.
> >
> > Changelog of 13.2.5 looked promising up until the sentence: "These
> > commands should not be used on a multisite setup as the stale
> > instances may be unlikely to be from a reshard and can have
> > consequences".
> >
> > http://docs.ceph.com/docs/master/releases/mimic/#v13-2-5-mimic
> >
>
> The stale-instances feature only correctly identifies one stale shard.
>
> # radosgw-admin reshard stale-instances list
> [
> "mybucket:0ef1a91a-4aee-427e-bdf8-30589abb2d3e.97248676.1"
> ]
>
> I can confirm this is one of the orphaned index objects.
>
> # rados -p .rgw.buckets.index ls | grep
> 0ef1a91a-4aee-427e-bdf8-30589abb2d3e.97248676.1
> .dir.0ef1a91a-4aee-427e-bdf8-30589abb2d3e.97248676.1.0
> .dir.0ef1a91a-4aee-427e-bdf8-30589abb2d3e.97248676.1.3
> .dir.0ef1a91a-4aee-427e-bdf8-30589abb2d3e.97248676.1.9
> .dir.0ef1a91a-4aee-427e-bdf8-30589abb2d3e.97248676.1.5
> .dir.0ef1a91a-4aee-427e-bdf8-30589abb2d3e.97248676.1.2
> .dir.0ef1a91a-4aee-427e-bdf8-30589abb2d3e.97248676.1.7
> .dir.0ef1a91a-4aee-427e-bdf8-30589abb2d3e.97248676.1.1
> .dir.0ef1a91a-4aee-427e-bdf8-30589abb2d3e.97248676.1.10
> .dir.0ef1a91a-4aee-427e-bdf8-30589abb2d3e.97248676.1.4
> .dir.0ef1a91a-4aee-427e-bdf8-30589abb2d3e.97248676.1.6
> .dir.0ef1a91a-4aee-427e-bdf8-30589abb2d3e.97248676.1.11
> .dir.0ef1a91a-4aee-427e-bdf8-30589abb2d3e.97248676.1.8
>
> I would assume then that unlike what documentation says, it's safe to
> run 'reshard stale-instances rm' on a multi-site setup.
>
> However it is quite telling if the author of this feature doesn't
> trust what they have written to work correctly.
>
> There are still thousands of stale index objects that 'stale-instances
> list' didn't pick up though.  But it appears that radosgw-admin only
> looks at 'metadata list bucket' data, and not what is physically
> inside the pool.
>
> --
> Iain Buclaw
>
> *(p < e ? p++ : p) = (c & 0x0f) + '0';
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Ceph Replication not working

[Vikas Rana]

Hi there,

We are trying to setup a rbd-mirror replication and after the setup,
everything looks good but images are not replicating.

 

Can some please please help?

 

Thanks,

-Vikas

 

root@remote:/var/log/ceph# rbd --cluster cephdr mirror pool info nfs

Mode: pool

Peers:

  UUID NAME CLIENT

  bcd54bc5-cd08-435f-a79a-357bce55011d ceph client.mirrorprod

 

root@local:/etc/ceph# rbd  mirror pool info nfs

Mode: pool

Peers:

  UUID NAME   CLIENT

  612151cf-f70d-49d0-94e2-a7b850a53e4f cephdr client.mirrordr

 

 

root@local:/etc/ceph# rbd info nfs/test01

rbd image 'test01':

size 102400 kB in 25 objects

order 22 (4096 kB objects)

block_name_prefix: rbd_data.11cd3c238e1f29

format: 2

features: layering, exclusive-lock, object-map, fast-diff,
deep-flatten, journaling

flags:

journal: 11cd3c238e1f29

mirroring state: enabled

mirroring global id: 06fbfe68-b7e4-4d3a-93b2-cd18c569f7f7

mirroring primary: true

 

 

root@remote:/var/log/ceph# rbd --cluster cephdr mirror pool status nfs
--verbose

health: OK

images: 0 total

 

root@remote:/var/log/ceph# rbd info nfs/test01

rbd: error opening image test01: (2) No such file or directory

 

 

root@remote:/var/log/ceph# ceph -s --cluster cephdr

  cluster:

id: ade49174-1f84-4c3c-a93c-b293c3655c93

health: HEALTH_WARN

noout,nodeep-scrub flag(s) set

 

  services:

mon:3 daemons, quorum nidcdvtier1a,nidcdvtier2a,nidcdvtier3a

mgr:nidcdvtier1a(active), standbys: nidcdvtier2a

osd:12 osds: 12 up, 12 in

flags noout,nodeep-scrub

rbd-mirror: 1 daemon active

 

  data:

pools:   5 pools, 640 pgs

objects: 1.32M objects, 5.03TiB

usage:   10.1TiB used, 262TiB / 272TiB avail

pgs: 640 active+clean

 

  io:

client:   170B/s rd, 0B/s wr, 0op/s rd, 0op/s wr

 

 

2019-04-05 12:07:29.720742 7f0fa5e284c0  0 ceph version 12.2.11
(26dc3775efc7bb286a1d6d66faee0ba30ea23eee) luminous (stable), process
rbd-mirror, pid 3921391

2019-04-05 12:07:29.721752 7f0fa5e284c0  0 pidfile_write: ignore empty
--pid-file

2019-04-05 12:07:29.726580 7f0fa5e284c0 20 rbd::mirror::ServiceDaemon:
0x560200d29bb0 ServiceDaemon: 

2019-04-05 12:07:29.732654 7f0fa5e284c0 20 rbd::mirror::ServiceDaemon:
0x560200d29bb0 init: 

2019-04-05 12:07:29.734920 7f0fa5e284c0  1 mgrc service_daemon_register
rbd-mirror.admin metadata {arch=x86_64,ceph_version=ceph version 12.2.11
(26dc3775efc7bb286a1d6d66faee0ba30ea23eee) luminous (stable),cpu=Intel(R)
Xeon(R) CPU E5-2690 v2 @ 3.00GHz,distro=ubuntu,distro_description=Ubuntu
14.04.5
LTS,distro_version=14.04,hostname=nidcdvtier3a,instance_id=464360,kernel_des
cription=#93 SMP Sat Jun 17 04:01:23 EDT
2017,kernel_version=3.19.0-85-vtier,mem_swap_kb=67105788,mem_total_kb=131999
112,os=Linux}

2019-04-05 12:07:29.735779 7f0fa5e284c0 20 rbd::mirror::Mirror:
0x560200d27f90 run: enter

2019-04-05 12:07:29.735793 7f0fa5e284c0 20
rbd::mirror::ClusterWatcher:0x560200dcd930 refresh_pools: enter

2019-04-05 12:07:29.735809 7f0f77fff700 20 rbd::mirror::ImageDeleter:
0x560200dcd9c0 run: enter

2019-04-05 12:07:29.735819 7f0f77fff700 20 rbd::mirror::ImageDeleter:
0x560200dcd9c0 run: waiting for delete requests

2019-04-05 12:07:29.739019 7f0fa5e284c0 10
rbd::mirror::ClusterWatcher:0x560200dcd930 read_pool_peers: mirroring is
disabled for pool docnfs

2019-04-05 12:07:29.741090 7f0fa5e284c0 10
rbd::mirror::ClusterWatcher:0x560200dcd930 read_pool_peers: mirroring is
disabled for pool doccifs

2019-04-05 12:07:29.742620 7f0fa5e284c0 10
rbd::mirror::ClusterWatcher:0x560200dcd930 read_pool_peers: mirroring is
disabled for pool fcp-dr

2019-04-05 12:07:29.76 7f0fa5e284c0 10
rbd::mirror::ClusterWatcher:0x560200dcd930 read_pool_peers: mirroring is
disabled for pool cifs

2019-04-05 12:07:29.746958 7f0fa5e284c0 20 rbd::mirror::ServiceDaemon:
0x560200d29bb0 add_pool: pool_id=8, pool_name=nfs

2019-04-05 12:07:29.748181 7f0fa5e284c0 20 rbd::mirror::Mirror:
0x560200d27f90 update_pool_replayers: enter

2019-04-05 12:07:29.748212 7f0fa5e284c0 20 rbd::mirror::Mirror:
0x560200d27f90 update_pool_replayers: starting pool replayer for uuid:
bcd54bc5-cd08-435f-a79a-357bce55011d cluster: ceph client: client.mirrorprod

2019-04-05 12:07:29.748249 7f0fa5e284c0 20 rbd::mirror::PoolReplayer:
0x560200de8f30 init: replaying for uuid:
bcd54bc5-cd08-435f-a79a-357bce55011d cluster: ceph client: client.mirrorprod

2019-04-05 12:07:29.853633 7f0fa5e284c0 20 rbd::mirror::PoolReplayer:
0x560200de8f30 init: connected to uuid: bcd54bc5-cd08-435f-a79a-357bce55011d
cluster: ceph client: client.mirrorprod

2019-04-05 12:07:29.853660 7f0fa5e284c0 20 rbd::mirror::InstanceReplayer:
0x560200ff9350 init: 

2019-04-05 12:07:29.853747 7f0f97626700 20 rbd::mirror::InstanceReplayer:
0x560200ff9350 schedule_image_state_check_task: scheduling image state check
afte

Re: [ceph-users] Ceph Replication not working

[Jason Dillaman]

What is the version of rbd-mirror daemon and your OSDs? It looks it
found two replicated images and got stuck on the "wait_for_deletion"
step. Since I suspect those images haven't been deleted, it should
have immediately proceeded to the next step of the image replay state
machine. Are there any additional log messages after 2019-04-05
12:07:29.981203?

On Fri, Apr 5, 2019 at 1:56 PM Vikas Rana  wrote:
>
> Hi there,
>
> We are trying to setup a rbd-mirror replication and after the setup, 
> everything looks good but images are not replicating.
>
>
>
> Can some please please help?
>
>
>
> Thanks,
>
> -Vikas
>
>
>
> root@remote:/var/log/ceph# rbd --cluster cephdr mirror pool info nfs
>
> Mode: pool
>
> Peers:
>
>   UUID NAME CLIENT
>
>   bcd54bc5-cd08-435f-a79a-357bce55011d ceph client.mirrorprod
>
>
>
> root@local:/etc/ceph# rbd  mirror pool info nfs
>
> Mode: pool
>
> Peers:
>
>   UUID NAME   CLIENT
>
>   612151cf-f70d-49d0-94e2-a7b850a53e4f cephdr client.mirrordr
>
>
>
>
>
> root@local:/etc/ceph# rbd info nfs/test01
>
> rbd image 'test01':
>
> size 102400 kB in 25 objects
>
> order 22 (4096 kB objects)
>
> block_name_prefix: rbd_data.11cd3c238e1f29
>
> format: 2
>
> features: layering, exclusive-lock, object-map, fast-diff, 
> deep-flatten, journaling
>
> flags:
>
> journal: 11cd3c238e1f29
>
> mirroring state: enabled
>
> mirroring global id: 06fbfe68-b7e4-4d3a-93b2-cd18c569f7f7
>
> mirroring primary: true
>
>
>
>
>
> root@remote:/var/log/ceph# rbd --cluster cephdr mirror pool status nfs 
> --verbose
>
> health: OK
>
> images: 0 total
>
>
>
> root@remote:/var/log/ceph# rbd info nfs/test01
>
> rbd: error opening image test01: (2) No such file or directory
>
>
>
>
>
> root@remote:/var/log/ceph# ceph -s --cluster cephdr
>
>   cluster:
>
> id: ade49174-1f84-4c3c-a93c-b293c3655c93
>
> health: HEALTH_WARN
>
> noout,nodeep-scrub flag(s) set
>
>
>
>   services:
>
> mon:3 daemons, quorum nidcdvtier1a,nidcdvtier2a,nidcdvtier3a
>
> mgr:nidcdvtier1a(active), standbys: nidcdvtier2a
>
> osd:12 osds: 12 up, 12 in
>
> flags noout,nodeep-scrub
>
> rbd-mirror: 1 daemon active
>
>
>
>   data:
>
> pools:   5 pools, 640 pgs
>
> objects: 1.32M objects, 5.03TiB
>
> usage:   10.1TiB used, 262TiB / 272TiB avail
>
> pgs: 640 active+clean
>
>
>
>   io:
>
> client:   170B/s rd, 0B/s wr, 0op/s rd, 0op/s wr
>
>
>
>
>
> 2019-04-05 12:07:29.720742 7f0fa5e284c0  0 ceph version 12.2.11 
> (26dc3775efc7bb286a1d6d66faee0ba30ea23eee) luminous (stable), process 
> rbd-mirror, pid 3921391
>
> 2019-04-05 12:07:29.721752 7f0fa5e284c0  0 pidfile_write: ignore empty 
> --pid-file
>
> 2019-04-05 12:07:29.726580 7f0fa5e284c0 20 rbd::mirror::ServiceDaemon: 
> 0x560200d29bb0 ServiceDaemon:
>
> 2019-04-05 12:07:29.732654 7f0fa5e284c0 20 rbd::mirror::ServiceDaemon: 
> 0x560200d29bb0 init:
>
> 2019-04-05 12:07:29.734920 7f0fa5e284c0  1 mgrc service_daemon_register 
> rbd-mirror.admin metadata {arch=x86_64,ceph_version=ceph version 12.2.11 
> (26dc3775efc7bb286a1d6d66faee0ba30ea23eee) luminous (stable),cpu=Intel(R) 
> Xeon(R) CPU E5-2690 v2 @ 3.00GHz,distro=ubuntu,distro_description=Ubuntu 
> 14.04.5 
> LTS,distro_version=14.04,hostname=nidcdvtier3a,instance_id=464360,kernel_description=#93
>  SMP Sat Jun 17 04:01:23 EDT 
> 2017,kernel_version=3.19.0-85-vtier,mem_swap_kb=67105788,mem_total_kb=131999112,os=Linux}
>
> 2019-04-05 12:07:29.735779 7f0fa5e284c0 20 rbd::mirror::Mirror: 
> 0x560200d27f90 run: enter
>
> 2019-04-05 12:07:29.735793 7f0fa5e284c0 20 
> rbd::mirror::ClusterWatcher:0x560200dcd930 refresh_pools: enter
>
> 2019-04-05 12:07:29.735809 7f0f77fff700 20 rbd::mirror::ImageDeleter: 
> 0x560200dcd9c0 run: enter
>
> 2019-04-05 12:07:29.735819 7f0f77fff700 20 rbd::mirror::ImageDeleter: 
> 0x560200dcd9c0 run: waiting for delete requests
>
> 2019-04-05 12:07:29.739019 7f0fa5e284c0 10 
> rbd::mirror::ClusterWatcher:0x560200dcd930 read_pool_peers: mirroring is 
> disabled for pool docnfs
>
> 2019-04-05 12:07:29.741090 7f0fa5e284c0 10 
> rbd::mirror::ClusterWatcher:0x560200dcd930 read_pool_peers: mirroring is 
> disabled for pool doccifs
>
> 2019-04-05 12:07:29.742620 7f0fa5e284c0 10 
> rbd::mirror::ClusterWatcher:0x560200dcd930 read_pool_peers: mirroring is 
> disabled for pool fcp-dr
>
> 2019-04-05 12:07:29.76 7f0fa5e284c0 10 
> rbd::mirror::ClusterWatcher:0x560200dcd930 read_pool_peers: mirroring is 
> disabled for pool cifs
>
> 2019-04-05 12:07:29.746958 7f0fa5e284c0 20 rbd::mirror::ServiceDaemon: 
> 0x560200d29bb0 add_pool: pool_id=8, pool_name=nfs
>
> 2019-04-05 12:07:29.748181 7f0fa5e284c0 20 rbd::mirror::Mirror: 
> 0x560200d27f90 update_pool_replayers: enter
>
> 2019-04-05 12:07:29.748212 7f0fa5e284c0 20 rbd::mirror::Mirror: 
> 0x560200d27f90 update_pool_replayers: starting pool replayer for uuid: 

[ceph-users] Cephalocon Barcelona, May 19-20

[Sage Weil]

Hi everyone,

This is a reminder that Cephalocon Barcelona is coming up next month (May 
19-20), and it's going to be great!  We have two days of Ceph content over 
four tracks, including:

 - A Rook tutorial for deploy Ceph over SSD instances
 - Several other Rook and Kubernetes related talks, including 
   self-service provisioning of object storage via kubernetes claim-like 
   APIs
 - Two sessions on project Crimson, which is a reimplementation
   of the Ceph OSD using Seastar, DPDK, and SPDK, targeting high 
   performance hardware and storage devices
 - Talks from Ceph power users with large-scale deployments from CERN, 
   China Mobile, OVH, MeerKAT (part of the Square Kilometer Array), UWisc, 
   and more.
 - How to build fast NVMe-based Ceph clusters on a reasonable budget
 - Using Ceph to provide NFS, CIFS, iSCSI service
 - The latest status of the Ceph management dashboard
 - Best practices for multi-cluster capabilities in RGW and RBD
 - S3 API compatibility in RGW, now and over the long term
 - and lots more!

You can see the full schedule here (minus a few keynote slots that haven't 
been finalized yet):


https://ceph.com/cephalocon/barcelona-2019/cephalocon-2019-barcelona-schedule/

There are going to be a lot of Ceph developers in Barcelona for the 
conference, both for Cephalocon (Sunday and Monday) and for KubeCon 
(Tuesday through Thursday).

For more information (including registration), see

https://ceph.com/cephalocon/barcelona-2019/

For those interested in Kubernetes and the intersection of storage with 
the leading container orchestration platform, KubeCon + CloudNativeCon 
runs directly after Cephalocon for the rest of the week.

We're looking forward to seeing many of you there!

sage
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] VM management setup

[jesper]

Hi. Knowing this is a bit off-topic but seeking recommendations
and advise anyway.

We're seeking a "management" solution for VM's - currently in the 40-50
VM - but would like to have better access in managing them and potintially
migrate them across multiple hosts, setup block devices, etc, etc.

This is only to be used internally in a department where a bunch of
engineering people will manage it, no costumers and that kind of thing.

Up until now we have been using virt-manager with kvm - and have been
quite satisfied when we were in the "few vms", but it seems like the
time to move on.

Thus we're looking for something "simple" that can help manage a ceph+kvm
based setup -  the simpler and more to the point the better.

Any recommendations?

.. found a lot of names allready ..
OpenStack
CloudStack
Proxmox
..

But recommendations are truely welcome.

Thanks.

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] VM management setup

[Kenneth Van Alstyne]

This is purely anecdotal (obviously), but I have found that OpenNebula is not 
only easy to setup, is relatively lightweight, and has very good Ceph support.  
5.8.0 was recently released, but has a few bugs related to live migrations with 
Ceph as the backend datastore.  You may want to look at 5.6.1 or wait for 5.8.1 
to be released since the issues have already been fixed upstream.

Thanks,

--
Kenneth Van Alstyne
Systems Architect
Knight Point Systems, LLC
Service-Disabled Veteran-Owned Business
1775 Wiehle Avenue Suite 101 | Reston, VA 20190
c: 228-547-8045 f: 571-266-3106
www.knightpoint.com
DHS EAGLE II Prime Contractor: FC1 SDVOSB Track
GSA Schedule 70 SDVOSB: GS-35F-0646S
GSA MOBIS Schedule: GS-10F-0404Y
ISO 9001 / ISO 2 / ISO 27001 / CMMI Level 3

Notice: This e-mail message, including any attachments, is for the sole use of 
the intended recipient(s) and may contain confidential and privileged 
information. Any unauthorized review, copy, use, disclosure, or distribution is 
STRICTLY prohibited. If you are not the intended recipient, please contact the 
sender by reply e-mail and destroy all copies of the original message.

On Apr 5, 2019, at 2:34 PM, jes...@krogh.cc wrote:

Hi. Knowing this is a bit off-topic but seeking recommendations
and advise anyway.

We're seeking a "management" solution for VM's - currently in the 40-50
VM - but would like to have better access in managing them and potintially
migrate them across multiple hosts, setup block devices, etc, etc.

This is only to be used internally in a department where a bunch of
engineering people will manage it, no costumers and that kind of thing.

Up until now we have been using virt-manager with kvm - and have been
quite satisfied when we were in the "few vms", but it seems like the
time to move on.

Thus we're looking for something "simple" that can help manage a ceph+kvm
based setup -  the simpler and more to the point the better.

Any recommendations?

.. found a lot of names allready ..
OpenStack
CloudStack
Proxmox
..

But recommendations are truely welcome.

Thanks.

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] VM management setup

[Ronny Aasen]

Proxmox VE is a simple solution.
https://www.proxmox.com/en/proxmox-ve

based on debian. can administer an internal ceph cluster or connect to 
an external connected . easy and almost self explanatory web interface.


good luck in your search !

Ronny



On 05.04.2019 21:34, jes...@krogh.cc wrote:

Hi. Knowing this is a bit off-topic but seeking recommendations
and advise anyway.

We're seeking a "management" solution for VM's - currently in the 40-50
VM - but would like to have better access in managing them and potintially
migrate them across multiple hosts, setup block devices, etc, etc.

This is only to be used internally in a department where a bunch of
engineering people will manage it, no costumers and that kind of thing.

Up until now we have been using virt-manager with kvm - and have been
quite satisfied when we were in the "few vms", but it seems like the
time to move on.

Thus we're looking for something "simple" that can help manage a ceph+kvm
based setup -  the simpler and more to the point the better.

Any recommendations?

.. found a lot of names allready ..
OpenStack
CloudStack
Proxmox
..

But recommendations are truely welcome.

Thanks.

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] VM management setup

[Brad Hubbard]

If you want to do containers at the same time, or transition some/all
to containers at some point in future maybe something based on
kubevirt [1] would be more futureproof?

[1] http://kubevirt.io/

CNV is an example,
https://www.redhat.com/en/resources/container-native-virtualization

On Sat, Apr 6, 2019 at 7:37 AM Ronny Aasen  wrote:
>
>
> Proxmox VE is a simple solution.
> https://www.proxmox.com/en/proxmox-ve
>
> based on debian. can administer an internal ceph cluster or connect to
> an external connected . easy and almost self explanatory web interface.
>
> good luck in your search !
>
> Ronny
>
>
>
> On 05.04.2019 21:34, jes...@krogh.cc wrote:
> > Hi. Knowing this is a bit off-topic but seeking recommendations
> > and advise anyway.
> >
> > We're seeking a "management" solution for VM's - currently in the 40-50
> > VM - but would like to have better access in managing them and potintially
> > migrate them across multiple hosts, setup block devices, etc, etc.
> >
> > This is only to be used internally in a department where a bunch of
> > engineering people will manage it, no costumers and that kind of thing.
> >
> > Up until now we have been using virt-manager with kvm - and have been
> > quite satisfied when we were in the "few vms", but it seems like the
> > time to move on.
> >
> > Thus we're looking for something "simple" that can help manage a ceph+kvm
> > based setup -  the simpler and more to the point the better.
> >
> > Any recommendations?
> >
> > .. found a lot of names allready ..
> > OpenStack
> > CloudStack
> > Proxmox
> > ..
> >
> > But recommendations are truely welcome.
> >
> > Thanks.
> >
> > ___
> > ceph-users mailing list
> > ceph-users@lists.ceph.com
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



-- 
Cheers,
Brad
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com