[ceph-users] What is the best way to "move" rgw.buckets.data pool to another cluster?

2019-06-28 Thread Fulvio Galeazzi 

Hallo!
  Due to severe maintenance which is going to cause a prolonged 
shutdown, I need to move my RGW pools to a different cluster (and 
geographical site): my problem is with default.rgw.buckets.data pool, 
which is now 100 TB.


Moreover, I'd also like to take advantage of the move to convert from 
replicated to erasure-coded.
Initially I though about rbd-mirror, but then realized it requires 
setting the journaling flag and I have 33M objects... (and also realized 
it's called RBD-mirror whereas I have an rgw pool).
"rados cppool" is going to be removed, if I understand it correctly? 
(apart from not being the right tool for my use-case)



What is the best strategy to copy (or rsync/mirror) an object-store pool 
to a different cluster?


  Thanks for your help!

Fulvio




smime.p7s
Description: S/MIME Cryptographic Signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] MGR Logs after Failure Testing

2019-06-28 Thread Eugen Block 
You may want to configure your standby-mds's to be "standby-replay" so  
the mds that's taking over from the failed one takes less time to take  
over. To manage this you add to your ceph.conf something like this:


---snip---
[mds.server1]
mds_standby_replay = true
mds_standby_for_rank = 0

[mds.server2]
mds_standby_replay = true
mds_standby_for_rank = 0

[mds.server3]
mds_standby_replay = true
mds_standby_for_rank = 0
---snip---

For your setup this would mean you have one active mds, one as  
standby-replay (that takes over immediately, depending on the load a  
very short interruption could happen) and one as standby ("cold  
standby" if you will). Currently both your standby mds servers are  
"cold".



Zitat von dhils...@performair.com:


Eugen;

All services are running, yes, though they didn't all start when I  
brought the host up (configured not to start because the last thing  
I had done is physically relocate the entire cluster).


All services are running, and happy.

# ceph status
  cluster:
id: 1a8a1693-fa54-4cb3-89d2-7951d4cee6a3
health: HEALTH_OK

  services:
mon: 3 daemons, quorum S700028,S700029,S700030 (age 20h)
mgr: S700028(active, since 17h), standbys: S700029, S700030
mds: cifs:1 {0=S700029=up:active} 2 up:standby
osd: 6 osds: 6 up (since 21h), 6 in (since 21h)

  data:
pools:   16 pools, 192 pgs
objects: 449 objects, 761 MiB
usage:   724 GiB used, 65 TiB / 66 TiB avail
pgs: 192 active+clean

# ceph osd tree
ID CLASS WEIGHT   TYPE NAMESTATUS REWEIGHT PRI-AFF
-1   66.17697 root default
-5   22.05899 host S700029
 2   hdd 11.02950 osd.2up  1.0 1.0
 3   hdd 11.02950 osd.3up  1.0 1.0
-7   22.05899 host S700030
 4   hdd 11.02950 osd.4up  1.0 1.0
 5   hdd 11.02950 osd.5up  1.0 1.0
-3   22.05899 host s700028
 0   hdd 11.02950 osd.0up  1.0 1.0
 1   hdd 11.02950 osd.1up  1.0 1.0

The question about configuring the MDS as failover struck me as a  
potential, since I don't remember doing that, however it look like  
S700029 (10.0.200.111) took over from S700028 (10.0.200.110) as the  
active MDS.


Thank you,

Dominic L. Hilsbos, MBA
Director - Information Technology
Perform Air International Inc.
dhils...@performair.com
www.PerformAir.com



-Original Message-
From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On  
Behalf Of Eugen Block

Sent: Thursday, June 27, 2019 8:23 AM
To: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] MGR Logs after Failure Testing

Hi,

some more information about the cluster status would be helpful, such as

ceph -s
ceph osd tree

service status of all MONs, MDSs, MGRs.
Are all services up? Did you configure the spare MDS as standby for
rank 0 so that a failover can happen?

Regards,
Eugen


Zitat von dhils...@performair.com:


All;

I built a demonstration and testing cluster, just 3 hosts
(10.0.200.110, 111, 112).  Each host runs mon, mgr, osd, mds.

During the demonstration yesterday, I pulled the power on one of the hosts.

After bringing the host back up, I'm getting several error messages
every second or so:
2019-06-26 16:01:56.424 7fcbe0af9700  0 ms_deliver_dispatch:
unhandled message 0x55e80a728f00 mgrreport(mds.S700030 +0-0 packed
6) v7 from mds.? v2:10.0.200.112:6808/980053124
2019-06-26 16:01:56.425 7fcbf4cd1700  1 mgr finish mon failed to
return metadata for mds.S700030: (2) No such file or directory
2019-06-26 16:01:56.429 7fcbe0af9700  0 ms_deliver_dispatch:
unhandled message 0x55e809f8e600 mgrreport(mds.S700029 +110-0 packed
1366) v7 from mds.0 v2:10.0.200.111:6808/2726495738
2019-06-26 16:01:56.430 7fcbf4cd1700  1 mgr finish mon failed to
return metadata for mds.S700029: (2) No such file or directory

Thoughts?

Thank you,

Dominic L. Hilsbos, MBA
Director - Information Technology
Perform Air International Inc.
dhils...@performair.com
www.PerformAir.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] What is the best way to "move" rgw.buckets.data pool to another cluster?

2019-06-28 Thread Fulvio Galeazzi 
Hallo again, to reply to my own message... I guess the easiest will be 
to setup multisite replication.
So now I will fight a bit with this and get back to the list in case of 
troubles.


  Sorry for the noise...

Fulvio

On 06/28/2019 10:36 AM, Fulvio Galeazzi wrote:

Hallo!
   Due to severe maintenance which is going to cause a prolonged 
shutdown, I need to move my RGW pools to a different cluster (and 
geographical site): my problem is with default.rgw.buckets.data pool, 
which is now 100 TB.


Moreover, I'd also like to take advantage of the move to convert from 
replicated to erasure-coded.
Initially I though about rbd-mirror, but then realized it requires 
setting the journaling flag and I have 33M objects... (and also realized 
it's called RBD-mirror whereas I have an rgw pool).
"rados cppool" is going to be removed, if I understand it correctly? 
(apart from not being the right tool for my use-case)



What is the best strategy to copy (or rsync/mirror) an object-store pool 
to a different cluster?


   Thanks for your help!

     Fulvio




___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com





smime.p7s
Description: S/MIME Cryptographic Signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] osd-mon failed with "failed to write to db"

2019-06-28 Thread Paul Emmerich 
Did you run the cluster with only a single monitor?


Paul

-- 
Paul Emmerich

Looking for help with your Ceph cluster? Contact us at https://croit.io

croit GmbH
Freseniusstr. 31h
81247 München
www.croit.io
Tel: +49 89 1896585 90


On Thu, Jun 27, 2019 at 4:32 PM Anton Aleksandrov 
wrote:

> Hello community,
>
> we have developed a cluster on latest mimic release. We are on quite old
> hardware, but using Centos7. Monitor, manager and all the same host.
> Cluster has been running for some week without actual workload. There might
> have been some sort of power failure (not proved), but at some point
> monitor node died and won't start anymore. Below is a log from
> /var/log/messages. What can be done here? Can this be recovered somehow or
> did we loose everything? All the OSDs seems to be running fine, just that
> the cluster is not working.
>
> The log is not full, but I think that those line are quite critical..
>
> Jun 27 17:14:06 mds1 ceph-mon: -311> 2019-06-27 17:14:06.169 7f086aa22700
> -1 *rocksdb: submit_common error: Corruption: block checksum mismatch*:
> expected 3317957558, got 2609532897  in
> /var/lib/ceph/mon/ceph-mds1/store.db/022334.sst offset 12775887 size 21652
> code = 2 Rocksdb transaction:
> Jun 27 17:14:06 mds1 ceph-mon: Put( Prefix = p key =
> 'xos'0x006c6173't_committed' Value size = 8)
> Jun 27 17:14:06 mds1 ceph-mon: Put( Prefix = m key =
> 'nitor_store'0x006c6173't_metadata' Value size = 612)
> Jun 27 17:14:06 mds1 ceph-mon: Put( Prefix = l key =
> 'gm'0x0066756c'l_155850' Value size = 31307)
> Jun 27 17:14:06 mds1 ceph-mon: Put( Prefix = l key =
> 'gm'0x0066756c'l_latest' Value size = 8)
> Jun 27 17:14:06 mds1 ceph-mon: Put( Prefix = l key = 'gm'0x00313535'851'
> Value size = 672)
> Jun 27 17:14:06 mds1 ceph-mon: Put( Prefix = l key =
> 'gm'0x006c6173't_committed' Value size = 8)
> Jun 27 17:14:06 mds1 ceph-mon: -311> 2019-06-27 17:14:06.172 7f086aa22700
> -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE
> _ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.6/rpm/el7/BUILD/ceph-13.2.6/src/mon/MonitorDBStore.h:
> In function
>  'int MonitorDBStore::apply_transaction(MonitorDBStore::TransactionRef)'
> thread 7f086aa22700 time 2019-06-27 17:14:06.171474
> Jun 27 17:14:06 mds1 ceph-mon:
> /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/cento
> s7/MACHINE_SIZE/huge/release/13.2.6/rpm/el7/BUILD/ceph-13.2.6/src/mon/MonitorDBStore.h:
> 311: FAILED assert(0 ==* "failed to write to db"*)
> Jun 27 17:14:06 mds1 ceph-mon: ceph version 13.2.6
> (7b695f835b03642f85998b2ae7b6dd093d9fbce4) mimic (stable)
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph-volume ignores cluster name from ceph.conf

2019-06-28 Thread Stolte, Felix 


smime.p7m
Description: S/MIME encrypted message
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] troubleshooting space usage

2019-06-28 Thread Andrei Mikhailovsky 
Hi 

Could someone please explain / show how to troubleshoot the space usage in Ceph 
and how to reclaim the unused space? 

I have a small cluster with 40 osds, replica of 2, mainly used as a backend for 
cloud stack as well as the S3 gateway. The used space doesn't make any sense to 
me, especially the rgw pool, so I am seeking help. 

Here is what I found from the client: 

Ceph -s shows the 

usage: 89 TiB used, 24 TiB / 113 TiB avail 

Ceph df shows: 

Primary-ubuntu-1 5 27 TiB 90.11 3.0 TiB 7201098 
Primary-ubuntu-1-ssd 57 1.2 TiB 89.62 143 GiB 359260 
.rgw.buckets 19 15 TiB 83.73 3.0 TiB 874 

the usage of the Primary-ubuntu-1 and Primary-ubuntu-1-ssd is in line with my 
expectations. However, the .rgw.buckets pool seems to be using way too much. 
The usage of all rgw buckets shows 6.5TB usage (looking at the size_kb values 
from the "radosgw-admin bucket stats"). I am trying to figure out why 
.rgw.buckets is using 15TB of space instead of the 6.5TB as shown from the 
bucket usage. 

Thanks 

Andrei 
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph-volume ignores cluster name from ceph.conf

2019-06-28 Thread Alfredo Deza 
On Fri, Jun 28, 2019 at 7:53 AM Stolte, Felix  wrote:
>
> Thanks for the update Alfredo. What steps need to be done to rename my 
> cluster back to "ceph"?

That is a tough one, the ramifications of a custom cluster name are
wild - it touches everything. I am not sure there is a step-by-step
guide on how to do this, I would personally recommend re-doing the
cluster (knowing well this might not be possible in certain cases)
>
> The clustername is in several folder- and filenames etc
>
> Regards
> Felix
> -
> -
> Forschungszentrum Juelich GmbH
> 52425 Juelich
> Sitz der Gesellschaft: Juelich
> Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
> Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher
> Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender),
> Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
> Prof. Dr. Sebastian M. Schmidt
> -
> -
>
>
> Am 27.06.19, 15:09 schrieb "Alfredo Deza" :
>
> Although ceph-volume does a best-effort to support custom cluster
> names, the Ceph project does not support custom cluster names anymore
> even though you can still see settings/options that will allow you to
> set it.
>
> For reference see: https://bugzilla.redhat.com/show_bug.cgi?id=1459861
>
> On Thu, Jun 27, 2019 at 7:59 AM Stolte, Felix  
> wrote:
> >
> > Hi folks,
> >
> > I have a nautilus 14.2.1 cluster with a non-default cluster name 
> (ceph_stag instead of ceph). I set “cluster = ceph_stag” in 
> /etc/ceph/ceph_stag.conf.
> >
> > ceph-volume is using the correct config file but does not use the 
> specified clustername. Did I hit a bug or do I need to define the clustername 
> elsewere?
> >
> > Regards
> > Felix
> > IT-Services
> > Telefon 02461 61-9243
> > E-Mail: f.sto...@fz-juelich.de
> > 
> -
> > 
> -
> > Forschungszentrum Juelich GmbH
> > 52425 Juelich
> > Sitz der Gesellschaft: Juelich
> > Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
> > Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher
> > Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender),
> > Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
> > Prof. Dr. Sebastian M. Schmidt
> > 
> -
> > 
> -
> >
> >
> > ___
> > ceph-users mailing list
> > ceph-users@lists.ceph.com
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph-volume ignores cluster name from ceph.conf

2019-06-28 Thread Erik McCormick 
On Fri, Jun 28, 2019, 10:05 AM Alfredo Deza  wrote:

> On Fri, Jun 28, 2019 at 7:53 AM Stolte, Felix 
> wrote:
> >
> > Thanks for the update Alfredo. What steps need to be done to rename my
> cluster back to "ceph"?
>
> That is a tough one, the ramifications of a custom cluster name are
> wild - it touches everything. I am not sure there is a step-by-step
> guide on how to do this, I would personally recommend re-doing the
> cluster (knowing well this might not be possible in certain cases)
> >
> > The clustername is in several folder- and filenames etc
> >
> > Regards
> > Felix
> >
>
Actually renaming is not really complicated at all. I did it manually
because paranoid, but you could easily enough script or ansiblize it.

Sage suggested a process to keep here:

http://lists.ceph.com/pipermail/ceph-users-ceph.com/2017-June/018521.html

And I finally reported back here:

http://lists.ceph.com/pipermail/ceph-users-ceph.com/2017-November/022202.html

Cheers,
Erik

> Forschungszentrum Juelich GmbH
> > 52425 Juelich
> > Sitz der Gesellschaft: Juelich
> > Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
> > Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher
> > Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender),
> > Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
> > Prof. Dr. Sebastian M. Schmidt
> >
> -
> >
> -
> >
> >
> > Am 27.06.19, 15:09 schrieb "Alfredo Deza" :
> >
> > Although ceph-volume does a best-effort to support custom cluster
> > names, the Ceph project does not support custom cluster names anymore
> > even though you can still see settings/options that will allow you to
> > set it.
> >
> > For reference see:
> https://bugzilla.redhat.com/show_bug.cgi?id=1459861
> >
> > On Thu, Jun 27, 2019 at 7:59 AM Stolte, Felix <
> f.sto...@fz-juelich.de> wrote:
> > >
> > > Hi folks,
> > >
> > > I have a nautilus 14.2.1 cluster with a non-default cluster name
> (ceph_stag instead of ceph). I set “cluster = ceph_stag” in
> /etc/ceph/ceph_stag.conf.
> > >
> > > ceph-volume is using the correct config file but does not use the
> specified clustername. Did I hit a bug or do I need to define the
> clustername elsewere?
> > >
> > > Regards
> > > Felix
> > > IT-Services
> > > Telefon 02461 61-9243
> > > E-Mail: f.sto...@fz-juelich.de
> > >
> -
> > >
> -
> > > Forschungszentrum Juelich GmbH
> > > 52425 Juelich
> > > Sitz der Gesellschaft: Juelich
> > > Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B
> 3498
> > > Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher
> > > Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt
> (Vorsitzender),
> > > Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
> > > Prof. Dr. Sebastian M. Schmidt
> > >
> -
> > >
> -
> > >
> > >
> > > ___
> > > ceph-users mailing list
> > > ceph-users@lists.ceph.com
> > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Migrating a cephfs data pool

2019-06-28 Thread Jorge Garcia 
This seems to be an issue that gets brought up repeatedly, but I haven't 
seen a definitive answer yet. So, at the risk of repeating a question 
that has already been asked:


How do you migrate a cephfs data pool to a new data pool? The obvious 
case would be somebody that has set up a replicated pool for their 
cephfs data and then wants to convert it to an erasure code pool. Is 
there a simple way to do this, other than creating a whole new ceph 
cluster and copying the data using rsync?


Thanks for any clues

Jorge

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Migrating a cephfs data pool

2019-06-28 Thread Marc Roos 
 
What about adding the new data pool, mounting it and then moving the 
files? (read copy because move between data pools does not what you 
expect it do)


-Original Message-
From: Jorge Garcia [mailto:jgar...@soe.ucsc.edu] 
Sent: vrijdag 28 juni 2019 17:26
To: ceph-users
Subject: *SPAM* [ceph-users] Migrating a cephfs data pool

This seems to be an issue that gets brought up repeatedly, but I haven't 
seen a definitive answer yet. So, at the risk of repeating a question 
that has already been asked:

How do you migrate a cephfs data pool to a new data pool? The obvious 
case would be somebody that has set up a replicated pool for their 
cephfs data and then wants to convert it to an erasure code pool. Is 
there a simple way to do this, other than creating a whole new ceph 
cluster and copying the data using rsync?

Thanks for any clues

Jorge

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Migrating a cephfs data pool

2019-06-28 Thread Jorge Garcia 
Are you talking about adding the new data pool to the current 
filesystem? Like:


  $ ceph fs add_data_pool my_ceph_fs new_ec_pool

I have done that, and now the filesystem shows up as having two data pools:

  $ ceph fs ls
  name: my_ceph_fs, metadata pool: cephfs_meta, data pools: 
[cephfs_data new_ec_pool ]


but then I run into two issues:

1. How do I actually copy/move/migrate the data from the old pool to the 
new pool?
2. When I'm done moving the data, how do I get rid of the old data pool? 
I know there's a rm_data_pool option, but I have read on the mailing 
list that you can't remove the original data pool from a cephfs filesystem.


The other option is to create a whole new cephfs with a new metadata 
pool and the new data pool, but creating multiple filesystems is still 
experimental and not allowed by default...


On 6/28/19 8:28 AM, Marc Roos wrote:
  
What about adding the new data pool, mounting it and then moving the

files? (read copy because move between data pools does not what you
expect it do)


-Original Message-
From: Jorge Garcia [mailto:jgar...@soe.ucsc.edu]
Sent: vrijdag 28 juni 2019 17:26
To: ceph-users
Subject: *SPAM* [ceph-users] Migrating a cephfs data pool

This seems to be an issue that gets brought up repeatedly, but I haven't
seen a definitive answer yet. So, at the risk of repeating a question
that has already been asked:

How do you migrate a cephfs data pool to a new data pool? The obvious
case would be somebody that has set up a replicated pool for their
cephfs data and then wants to convert it to an erasure code pool. Is
there a simple way to do this, other than creating a whole new ceph
cluster and copying the data using rsync?

Thanks for any clues

Jorge

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Migrating a cephfs data pool

2019-06-28 Thread Marc Roos 
 

1.
change data pool for a folder on the file system:
setfattr -n ceph.dir.layout.pool -v fs_data.ec21 foldername

2. 
cp /oldlocation /foldername
Remember that you preferably want to use mv, but this leaves (meta) data 
on the old pool, that is not what you want when you want to delete that 
pool.

3. When everything is copied-removed, you should end up with an empty 
datapool with zero objects. 

4. Verify here with others, if you can just remove this one.

I think this is a reliable technique to switch, because you use the 
basic cephfs functionality that supposed to work. I prefer that the ceph 
guys implement a mv that does what you expect from it. Now it acts more 
or less like a linking.




-Original Message-
From: Jorge Garcia [mailto:jgar...@soe.ucsc.edu] 
Sent: vrijdag 28 juni 2019 17:52
To: Marc Roos; ceph-users
Subject: Re: [ceph-users] Migrating a cephfs data pool

Are you talking about adding the new data pool to the current 
filesystem? Like:

   $ ceph fs add_data_pool my_ceph_fs new_ec_pool

I have done that, and now the filesystem shows up as having two data 
pools:

   $ ceph fs ls
   name: my_ceph_fs, metadata pool: cephfs_meta, data pools: 
[cephfs_data new_ec_pool ]

but then I run into two issues:

1. How do I actually copy/move/migrate the data from the old pool to the 
new pool?
2. When I'm done moving the data, how do I get rid of the old data pool? 

I know there's a rm_data_pool option, but I have read on the mailing 
list that you can't remove the original data pool from a cephfs 
filesystem.

The other option is to create a whole new cephfs with a new metadata 
pool and the new data pool, but creating multiple filesystems is still 
experimental and not allowed by default...

On 6/28/19 8:28 AM, Marc Roos wrote:
>   
> What about adding the new data pool, mounting it and then moving the 
> files? (read copy because move between data pools does not what you 
> expect it do)
>
>
> -Original Message-
> From: Jorge Garcia [mailto:jgar...@soe.ucsc.edu]
> Sent: vrijdag 28 juni 2019 17:26
> To: ceph-users
> Subject: *SPAM* [ceph-users] Migrating a cephfs data pool
>
> This seems to be an issue that gets brought up repeatedly, but I 
> haven't seen a definitive answer yet. So, at the risk of repeating a 
> question that has already been asked:
>
> How do you migrate a cephfs data pool to a new data pool? The obvious 
> case would be somebody that has set up a replicated pool for their 
> cephfs data and then wants to convert it to an erasure code pool. Is 
> there a simple way to do this, other than creating a whole new ceph 
> cluster and copying the data using rsync?
>
> Thanks for any clues
>
> Jorge
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] How does monitor know OSD is dead?

2019-06-28 Thread Robert LeBlanc 
I'm not sure why the monitor did not mark it down after 600 seconds
(default). The reason it is so long is that you don't want to move data
around unnecessarily if the osd is just being rebooted/restarted. Usually,
you will still have min_size OSDs available for all PGs that will allow IO
to continue. Then when the down timeout expires it will start backfilling
and recovering the PGs that were affected. Double check that size !=
min_size for your pools.

Robert LeBlanc
PGP Fingerprint 79A2 9CA4 6CC4 45DD A904  C70E E654 3BB2 FA62 B9F1


On Thu, Jun 27, 2019 at 5:26 PM Bryan Henderson 
wrote:

> What does it take for a monitor to consider an OSD down which has been
> dead as
> a doornail since the cluster started?
>
> A couple of times, I have seen 'ceph status' report an OSD was up, when it
> was
> quite dead.  Recently, a couple of OSDs were on machines that failed to
> boot
> up after a power failure.  The rest of the Ceph cluster came up, though,
> and
> reported all OSDs up and in.  I/Os stalled, probably because they were
> waiting
> for the dead OSDs to come back.
>
> I waited 15 minutes, because the manual says if the monitor doesn't hear a
> heartbeat from an OSD in that long (default value of
> mon_osd_report_timeout),
> it marks it down.  But it didn't.  I did "osd down" commands for the dead
> OSDs
> and the status changed to down and I/O started working.
>
> And wouldn't even 15 minutes of grace be unacceptable if it means I/Os
> have to
> wait that long before falling back to a redundant OSD?
>
> --
> Bryan Henderson   San Jose, California
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Migrating a cephfs data pool

2019-06-28 Thread Robert LeBlanc 
Given that the MDS knows everything, it seems trivial to add a ceph 'mv'
command to do this. I looked at using tiering to try and do the move, but I
don't know how to tell cephfs that the data is now on the new pool instead
of the old pool name. Since we can't take a long enough downtime to move
hundreds of Terabytes, we need something that can be done online, and if it
has a minute or two of downtime would be okay.

Robert LeBlanc
PGP Fingerprint 79A2 9CA4 6CC4 45DD A904  C70E E654 3BB2 FA62 B9F1


On Fri, Jun 28, 2019 at 9:02 AM Marc Roos  wrote:

>
>
> 1.
> change data pool for a folder on the file system:
> setfattr -n ceph.dir.layout.pool -v fs_data.ec21 foldername
>
> 2.
> cp /oldlocation /foldername
> Remember that you preferably want to use mv, but this leaves (meta) data
> on the old pool, that is not what you want when you want to delete that
> pool.
>
> 3. When everything is copied-removed, you should end up with an empty
> datapool with zero objects.
>
> 4. Verify here with others, if you can just remove this one.
>
> I think this is a reliable technique to switch, because you use the
> basic cephfs functionality that supposed to work. I prefer that the ceph
> guys implement a mv that does what you expect from it. Now it acts more
> or less like a linking.
>
>
>
>
> -Original Message-
> From: Jorge Garcia [mailto:jgar...@soe.ucsc.edu]
> Sent: vrijdag 28 juni 2019 17:52
> To: Marc Roos; ceph-users
> Subject: Re: [ceph-users] Migrating a cephfs data pool
>
> Are you talking about adding the new data pool to the current
> filesystem? Like:
>
>$ ceph fs add_data_pool my_ceph_fs new_ec_pool
>
> I have done that, and now the filesystem shows up as having two data
> pools:
>
>$ ceph fs ls
>name: my_ceph_fs, metadata pool: cephfs_meta, data pools:
> [cephfs_data new_ec_pool ]
>
> but then I run into two issues:
>
> 1. How do I actually copy/move/migrate the data from the old pool to the
> new pool?
> 2. When I'm done moving the data, how do I get rid of the old data pool?
>
> I know there's a rm_data_pool option, but I have read on the mailing
> list that you can't remove the original data pool from a cephfs
> filesystem.
>
> The other option is to create a whole new cephfs with a new metadata
> pool and the new data pool, but creating multiple filesystems is still
> experimental and not allowed by default...
>
> On 6/28/19 8:28 AM, Marc Roos wrote:
> >
> > What about adding the new data pool, mounting it and then moving the
> > files? (read copy because move between data pools does not what you
> > expect it do)
> >
> >
> > -Original Message-
> > From: Jorge Garcia [mailto:jgar...@soe.ucsc.edu]
> > Sent: vrijdag 28 juni 2019 17:26
> > To: ceph-users
> > Subject: *SPAM* [ceph-users] Migrating a cephfs data pool
> >
> > This seems to be an issue that gets brought up repeatedly, but I
> > haven't seen a definitive answer yet. So, at the risk of repeating a
> > question that has already been asked:
> >
> > How do you migrate a cephfs data pool to a new data pool? The obvious
> > case would be somebody that has set up a replicated pool for their
> > cephfs data and then wants to convert it to an erasure code pool. Is
> > there a simple way to do this, other than creating a whole new ceph
> > cluster and copying the data using rsync?
> >
> > Thanks for any clues
> >
> > Jorge
> >
> > ___
> > ceph-users mailing list
> > ceph-users@lists.ceph.com
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >
> >
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Migrating a cephfs data pool

2019-06-28 Thread Marc Roos 
 
Afaik is the mv now fast because it is not moving any real data, just 
some meta data. Thus a real mv will be slow (only in the case between 
different pools) because it copies the data to the new pool and when 
successful deletes the old one. This will of course take a lot more 
time, but you at least are able to access the cephfs on both locations 
during this time and can fix things in your client access.

My problem with mv now is that if you accidentally use it between data 
pools, it does not really move data. 



-Original Message-
From: Robert LeBlanc [mailto:rob...@leblancnet.us] 
Sent: vrijdag 28 juni 2019 18:30
To: Marc Roos
Cc: ceph-users; jgarcia
Subject: Re: [ceph-users] Migrating a cephfs data pool

Given that the MDS knows everything, it seems trivial to add a ceph 'mv' 
command to do this. I looked at using tiering to try and do the move, 
but I don't know how to tell cephfs that the data is now on the new pool 
instead of the old pool name. Since we can't take a long enough downtime 
to move hundreds of Terabytes, we need something that can be done 
online, and if it has a minute or two of downtime would be okay.


Robert LeBlanc
PGP Fingerprint 79A2 9CA4 6CC4 45DD A904  C70E E654 3BB2 FA62 B9F1


On Fri, Jun 28, 2019 at 9:02 AM Marc Roos  
wrote:


 

1.
change data pool for a folder on the file system:
setfattr -n ceph.dir.layout.pool -v fs_data.ec21 foldername

2. 
cp /oldlocation /foldername
Remember that you preferably want to use mv, but this leaves (meta) 
data 
on the old pool, that is not what you want when you want to delete 
that 
pool.

3. When everything is copied-removed, you should end up with an 
empty 
datapool with zero objects. 

4. Verify here with others, if you can just remove this one.

I think this is a reliable technique to switch, because you use the 

basic cephfs functionality that supposed to work. I prefer that the 
ceph 
guys implement a mv that does what you expect from it. Now it acts 
more 
or less like a linking.




-Original Message-
From: Jorge Garcia [mailto:jgar...@soe.ucsc.edu] 
Sent: vrijdag 28 juni 2019 17:52
To: Marc Roos; ceph-users
Subject: Re: [ceph-users] Migrating a cephfs data pool

Are you talking about adding the new data pool to the current 
filesystem? Like:

   $ ceph fs add_data_pool my_ceph_fs new_ec_pool

I have done that, and now the filesystem shows up as having two 
data 
pools:

   $ ceph fs ls
   name: my_ceph_fs, metadata pool: cephfs_meta, data pools: 
[cephfs_data new_ec_pool ]

but then I run into two issues:

1. How do I actually copy/move/migrate the data from the old pool 
to the 
new pool?
2. When I'm done moving the data, how do I get rid of the old data 
pool? 

I know there's a rm_data_pool option, but I have read on the 
mailing 
list that you can't remove the original data pool from a cephfs 
filesystem.

The other option is to create a whole new cephfs with a new 
metadata 
pool and the new data pool, but creating multiple filesystems is 
still 
experimental and not allowed by default...

On 6/28/19 8:28 AM, Marc Roos wrote:
>   
> What about adding the new data pool, mounting it and then moving 
the 
> files? (read copy because move between data pools does not what 
you 
> expect it do)
>
>
> -Original Message-
> From: Jorge Garcia [mailto:jgar...@soe.ucsc.edu]
> Sent: vrijdag 28 juni 2019 17:26
> To: ceph-users
> Subject: *SPAM* [ceph-users] Migrating a cephfs data pool
>
> This seems to be an issue that gets brought up repeatedly, but I 
> haven't seen a definitive answer yet. So, at the risk of 
repeating a 
> question that has already been asked:
>
> How do you migrate a cephfs data pool to a new data pool? The 
obvious 
> case would be somebody that has set up a replicated pool for 
their 
> cephfs data and then wants to convert it to an erasure code pool. 
Is 
> there a simple way to do this, other than creating a whole new 
ceph 
> cluster and copying the data using rsync?
>
> Thanks for any clues
>
> Jorge
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>


___

[ceph-users] RADOSGW S3 - Continuation Token Ignored?

2019-06-28 Thread DHilsbos 
All;

I've got a RADOSGW instance setup, backed by my demonstration Ceph cluster.  
I'm using Amazon's S3 SDK, and I've run into an annoying little snag.

My code looks like this:
amazonS3 = builder.build();

ListObjectsV2Request req = new 
ListObjectsV2Request().withBucketName("WorkOrder").withMaxKeys(MAX_KEYS);
ListObjectsV2Result result;

do
{
result = amazonS3.listObjectsV2(req);

for (S3ObjectSummary objectSummary : result.getObjectSummaries())
{
summaries.add(objectSummary);
}

String token = result.getNextContinuationToken();
req.setContinuationToken(token);
}
while (result.isTruncated());

The problem is, the ContinuationToken seems to be ignored, i.e. every call to 
amazonS3.listObjectsV2(req) returns the same set, and the loop never ends 
(until the summaries LinkedList overflows).

Thoughts?

Thank you,

Dominic L. Hilsbos, MBA 
Director - Information Technology 
Perform Air International Inc.
dhils...@performair.com 
www.PerformAir.com



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] RADOSGW S3 - Continuation Token Ignored?

2019-06-28 Thread Matt Benjamin 
Hi Dominic,

The reason is likely that RGW doesn't yet support ListObjectsV2.

Support is nearly here though:  https://github.com/ceph/ceph/pull/28102

Matt


On Fri, Jun 28, 2019 at 12:43 PM  wrote:
>
> All;
>
> I've got a RADOSGW instance setup, backed by my demonstration Ceph cluster.  
> I'm using Amazon's S3 SDK, and I've run into an annoying little snag.
>
> My code looks like this:
> amazonS3 = builder.build();
>
> ListObjectsV2Request req = new 
> ListObjectsV2Request().withBucketName("WorkOrder").withMaxKeys(MAX_KEYS);
> ListObjectsV2Result result;
>
> do
> {
> result = amazonS3.listObjectsV2(req);
>
> for (S3ObjectSummary objectSummary : result.getObjectSummaries())
> {
> summaries.add(objectSummary);
> }
>
> String token = result.getNextContinuationToken();
> req.setContinuationToken(token);
> }
> while (result.isTruncated());
>
> The problem is, the ContinuationToken seems to be ignored, i.e. every call to 
> amazonS3.listObjectsV2(req) returns the same set, and the loop never ends 
> (until the summaries LinkedList overflows).
>
> Thoughts?
>
> Thank you,
>
> Dominic L. Hilsbos, MBA
> Director - Information Technology
> Perform Air International Inc.
> dhils...@performair.com
> www.PerformAir.com
>
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>


--

Matt Benjamin
Red Hat, Inc.
315 West Huron Street, Suite 140A
Ann Arbor, Michigan 48103

http://www.redhat.com/en/technologies/storage

tel.  734-821-5101
fax.  734-769-8938
cel.  734-216-5309
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Migrating a cephfs data pool

2019-06-28 Thread Robert LeBlanc 
Yes, 'mv' on the client is just a metadata operation and not what I'm
talking about. The idea is to bring the old pool in as a cache layer, then
bring the new pool in as a lower layer, then flush/evict the data from the
cache and Ceph will move the data to the new pool, but still be able to
access it by the old pool name. You then add an overlay so that the new
pool name acts the same, then the idea is that you can remove the old pool
from the cache and remove the overlay. The only problem is updating cephfs
to look at the new pool name for data that it knows is at the old pool name.

The other option is to add a data mover to cephfs so you can do something
like `ceph fs mv old_pool new_pool` and it would move all the objects and
update the metadata as it performs the data moving. The question is how to
do the data movement since the MDS is not in the data path.

Since both pool names act the same with the overlay, the best option sounds
like; configure the tiering, add the overlay, then do a `ceph fs migrate
old_pool new_pool` which causes the MDS to scan through all the metadata
and update all references of 'old_pool' to 'new_pool'. Once that is done
and the eviction is done, then you can remove the pool from cephfs and the
overlay. That way the OSDs are the one doing the data movement.

I don't know that part of the code, so I can't quickly propose any patches.

Robert LeBlanc
PGP Fingerprint 79A2 9CA4 6CC4 45DD A904  C70E E654 3BB2 FA62 B9F1


On Fri, Jun 28, 2019 at 9:37 AM Marc Roos  wrote:

>
> Afaik is the mv now fast because it is not moving any real data, just
> some meta data. Thus a real mv will be slow (only in the case between
> different pools) because it copies the data to the new pool and when
> successful deletes the old one. This will of course take a lot more
> time, but you at least are able to access the cephfs on both locations
> during this time and can fix things in your client access.
>
> My problem with mv now is that if you accidentally use it between data
> pools, it does not really move data.
>
>
>
> -Original Message-
> From: Robert LeBlanc [mailto:rob...@leblancnet.us]
> Sent: vrijdag 28 juni 2019 18:30
> To: Marc Roos
> Cc: ceph-users; jgarcia
> Subject: Re: [ceph-users] Migrating a cephfs data pool
>
> Given that the MDS knows everything, it seems trivial to add a ceph 'mv'
> command to do this. I looked at using tiering to try and do the move,
> but I don't know how to tell cephfs that the data is now on the new pool
> instead of the old pool name. Since we can't take a long enough downtime
> to move hundreds of Terabytes, we need something that can be done
> online, and if it has a minute or two of downtime would be okay.
>
> 
> Robert LeBlanc
> PGP Fingerprint 79A2 9CA4 6CC4 45DD A904  C70E E654 3BB2 FA62 B9F1
>
>
> On Fri, Jun 28, 2019 at 9:02 AM Marc Roos 
> wrote:
>
>
>
>
> 1.
> change data pool for a folder on the file system:
> setfattr -n ceph.dir.layout.pool -v fs_data.ec21 foldername
>
> 2.
> cp /oldlocation /foldername
> Remember that you preferably want to use mv, but this leaves
> (meta)
> data
> on the old pool, that is not what you want when you want to delete
> that
> pool.
>
> 3. When everything is copied-removed, you should end up with an
> empty
> datapool with zero objects.
>
> 4. Verify here with others, if you can just remove this one.
>
> I think this is a reliable technique to switch, because you use
> the
>
> basic cephfs functionality that supposed to work. I prefer that
> the
> ceph
> guys implement a mv that does what you expect from it. Now it acts
> more
> or less like a linking.
>
>
>
>
> -Original Message-
> From: Jorge Garcia [mailto:jgar...@soe.ucsc.edu]
> Sent: vrijdag 28 juni 2019 17:52
> To: Marc Roos; ceph-users
> Subject: Re: [ceph-users] Migrating a cephfs data pool
>
> Are you talking about adding the new data pool to the current
> filesystem? Like:
>
>$ ceph fs add_data_pool my_ceph_fs new_ec_pool
>
> I have done that, and now the filesystem shows up as having two
> data
> pools:
>
>$ ceph fs ls
>name: my_ceph_fs, metadata pool: cephfs_meta, data pools:
> [cephfs_data new_ec_pool ]
>
> but then I run into two issues:
>
> 1. How do I actually copy/move/migrate the data from the old pool
> to the
> new pool?
> 2. When I'm done moving the data, how do I get rid of the old data
> pool?
>
> I know there's a rm_data_pool option, but I have read on the
> mailing
> list that you can't remove the original data pool from a cephfs
> filesystem.
>
> The other option is to create a whole new cephfs with a new
> metadata
> pool and the new data pool, but creating multiple filesystems is
> still

Re: [ceph-users] RADOSGW S3 - Continuation Token Ignored?

2019-06-28 Thread DHilsbos 
Matt;

Yep, that would certainly explain it.

My apologies, I almost searched for that information before sending the email.

Thank you,

Dominic L. Hilsbos, MBA 
Director – Information Technology 
Perform Air International Inc.
dhils...@performair.com 
www.PerformAir.com



-Original Message-
From: Matt Benjamin [mailto:mbenj...@redhat.com] 
Sent: Friday, June 28, 2019 9:48 AM
To: Dominic Hilsbos
Cc: ceph-users
Subject: Re: [ceph-users] RADOSGW S3 - Continuation Token Ignored?

Hi Dominic,

The reason is likely that RGW doesn't yet support ListObjectsV2.

Support is nearly here though:  https://github.com/ceph/ceph/pull/28102

Matt


On Fri, Jun 28, 2019 at 12:43 PM  wrote:
>
> All;
>
> I've got a RADOSGW instance setup, backed by my demonstration Ceph cluster.  
> I'm using Amazon's S3 SDK, and I've run into an annoying little snag.
>
> My code looks like this:
> amazonS3 = builder.build();
>
> ListObjectsV2Request req = new 
> ListObjectsV2Request().withBucketName("WorkOrder").withMaxKeys(MAX_KEYS);
> ListObjectsV2Result result;
>
> do
> {
> result = amazonS3.listObjectsV2(req);
>
> for (S3ObjectSummary objectSummary : result.getObjectSummaries())
> {
> summaries.add(objectSummary);
> }
>
> String token = result.getNextContinuationToken();
> req.setContinuationToken(token);
> }
> while (result.isTruncated());
>
> The problem is, the ContinuationToken seems to be ignored, i.e. every call to 
> amazonS3.listObjectsV2(req) returns the same set, and the loop never ends 
> (until the summaries LinkedList overflows).
>
> Thoughts?
>
> Thank you,
>
> Dominic L. Hilsbos, MBA
> Director - Information Technology
> Perform Air International Inc.
> dhils...@performair.com
> www.PerformAir.com
>
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>


--

Matt Benjamin
Red Hat, Inc.
315 West Huron Street, Suite 140A
Ann Arbor, Michigan 48103

http://www.redhat.com/en/technologies/storage

tel.  734-821-5101
fax.  734-769-8938
cel.  734-216-5309
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] RADOSGW S3 - Continuation Token Ignored?

2019-06-28 Thread Matt Benjamin 
FYI, this PR just merged.  I would expect to see backports at least as
far as N, and others would be possible.

regards,

Matt

On Fri, Jun 28, 2019 at 3:43 PM  wrote:
>
> Matt;
>
> Yep, that would certainly explain it.
>
> My apologies, I almost searched for that information before sending the email.
>
> Thank you,
>
> Dominic L. Hilsbos, MBA
> Director – Information Technology
> Perform Air International Inc.
> dhils...@performair.com
> www.PerformAir.com
>
>
>
> -Original Message-
> From: Matt Benjamin [mailto:mbenj...@redhat.com]
> Sent: Friday, June 28, 2019 9:48 AM
> To: Dominic Hilsbos
> Cc: ceph-users
> Subject: Re: [ceph-users] RADOSGW S3 - Continuation Token Ignored?
>
> Hi Dominic,
>
> The reason is likely that RGW doesn't yet support ListObjectsV2.
>
> Support is nearly here though:  https://github.com/ceph/ceph/pull/28102
>
> Matt
>
>
> On Fri, Jun 28, 2019 at 12:43 PM  wrote:
> >
> > All;
> >
> > I've got a RADOSGW instance setup, backed by my demonstration Ceph cluster. 
> >  I'm using Amazon's S3 SDK, and I've run into an annoying little snag.
> >
> > My code looks like this:
> > amazonS3 = builder.build();
> >
> > ListObjectsV2Request req = new 
> > ListObjectsV2Request().withBucketName("WorkOrder").withMaxKeys(MAX_KEYS);
> > ListObjectsV2Result result;
> >
> > do
> > {
> > result = amazonS3.listObjectsV2(req);
> >
> > for (S3ObjectSummary objectSummary : result.getObjectSummaries())
> > {
> > summaries.add(objectSummary);
> > }
> >
> > String token = result.getNextContinuationToken();
> > req.setContinuationToken(token);
> > }
> > while (result.isTruncated());
> >
> > The problem is, the ContinuationToken seems to be ignored, i.e. every call 
> > to amazonS3.listObjectsV2(req) returns the same set, and the loop never 
> > ends (until the summaries LinkedList overflows).
> >
> > Thoughts?
> >
> > Thank you,
> >
> > Dominic L. Hilsbos, MBA
> > Director - Information Technology
> > Perform Air International Inc.
> > dhils...@performair.com
> > www.PerformAir.com
> >
> >
> >
> > ___
> > ceph-users mailing list
> > ceph-users@lists.ceph.com
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >
> >
>
>
> --
>
> Matt Benjamin
> Red Hat, Inc.
> 315 West Huron Street, Suite 140A
> Ann Arbor, Michigan 48103
>
> http://www.redhat.com/en/technologies/storage
>
> tel.  734-821-5101
> fax.  734-769-8938
> cel.  734-216-5309



-- 

Matt Benjamin
Red Hat, Inc.
315 West Huron Street, Suite 140A
Ann Arbor, Michigan 48103

http://www.redhat.com/en/technologies/storage

tel.  734-821-5101
fax.  734-769-8938
cel.  734-216-5309
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Migrating a cephfs data pool

2019-06-28 Thread Jorge Garcia 


On 6/28/19 9:02 AM, Marc Roos wrote:

3. When everything is copied-removed, you should end up with an empty
datapool with zero objects.


I copied the data to a new directory and then removed the data from the 
old directory, but df still reports some objects in the old pool (not 
zero). Is there a way to track down what's still in the old pool, and 
how to delete it?


Before delete:

# ceph df
GLOBAL:
    SIZE    AVAIL   RAW USED %RAW USED
    392 TiB 389 TiB  3.3 TiB  0.83
POOLS:
    NAME    ID USED    %USED MAX AVAIL OBJECTS
    cephfs-meta  6   17 MiB 0   123 TiB 27
    cephfs-data   7  763 GiB  0.60   123 TiB 195233
    new-ec-pool  8  641 GiB  0.25   245 TiB 163991

After delete:

# ceph df
GLOBAL:
    SIZE    AVAIL   RAW USED %RAW USED
    392 TiB 391 TiB  1.2 TiB  0.32
POOLS:
    NAME    ID USED    %USED MAX AVAIL OBJECTS
    cephfs-meta  6   26 MiB 0   124 TiB 29
    cephfs-data   7   83 GiB  0.07   124 TiB 21175
    new-ec-pool  8  641 GiB  0.25   247 TiB 163991

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] How does monitor know OSD is dead?

2019-06-28 Thread solarflow99 
The thing i've seen a lot is where an OSD would get marked down because of
a failed drive, then then it would add itself right back again


On Fri, Jun 28, 2019 at 9:12 AM Robert LeBlanc  wrote:

> I'm not sure why the monitor did not mark it down after 600 seconds
> (default). The reason it is so long is that you don't want to move data
> around unnecessarily if the osd is just being rebooted/restarted. Usually,
> you will still have min_size OSDs available for all PGs that will allow IO
> to continue. Then when the down timeout expires it will start backfilling
> and recovering the PGs that were affected. Double check that size !=
> min_size for your pools.
> 
> Robert LeBlanc
> PGP Fingerprint 79A2 9CA4 6CC4 45DD A904  C70E E654 3BB2 FA62 B9F1
>
>
> On Thu, Jun 27, 2019 at 5:26 PM Bryan Henderson 
> wrote:
>
>> What does it take for a monitor to consider an OSD down which has been
>> dead as
>> a doornail since the cluster started?
>>
>> A couple of times, I have seen 'ceph status' report an OSD was up, when
>> it was
>> quite dead.  Recently, a couple of OSDs were on machines that failed to
>> boot
>> up after a power failure.  The rest of the Ceph cluster came up, though,
>> and
>> reported all OSDs up and in.  I/Os stalled, probably because they were
>> waiting
>> for the dead OSDs to come back.
>>
>> I waited 15 minutes, because the manual says if the monitor doesn't hear a
>> heartbeat from an OSD in that long (default value of
>> mon_osd_report_timeout),
>> it marks it down.  But it didn't.  I did "osd down" commands for the dead
>> OSDs
>> and the status changed to down and I/O started working.
>>
>> And wouldn't even 15 minutes of grace be unacceptable if it means I/Os
>> have to
>> wait that long before falling back to a redundant OSD?
>>
>> --
>> Bryan Henderson   San Jose, California
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Migrating a cephfs data pool

2019-06-28 Thread Patrick Hein 
Afaik MDS doesn't delete the objects immediately but defer it for later. If
you check that again now, how many objects does it report?

Jorge Garcia  schrieb am Fr., 28. Juni 2019, 23:16:

>
> On 6/28/19 9:02 AM, Marc Roos wrote:
> > 3. When everything is copied-removed, you should end up with an empty
> > datapool with zero objects.
>
> I copied the data to a new directory and then removed the data from the
> old directory, but df still reports some objects in the old pool (not
> zero). Is there a way to track down what's still in the old pool, and
> how to delete it?
>
> Before delete:
>
> # ceph df
> GLOBAL:
>  SIZEAVAIL   RAW USED %RAW USED
>  392 TiB 389 TiB  3.3 TiB  0.83
> POOLS:
>  NAMEID USED%USED MAX AVAIL OBJECTS
>  cephfs-meta  6   17 MiB 0   123 TiB 27
>  cephfs-data   7  763 GiB  0.60   123 TiB 195233
>  new-ec-pool  8  641 GiB  0.25   245 TiB 163991
>
> After delete:
>
> # ceph df
> GLOBAL:
>  SIZEAVAIL   RAW USED %RAW USED
>  392 TiB 391 TiB  1.2 TiB  0.32
> POOLS:
>  NAMEID USED%USED MAX AVAIL OBJECTS
>  cephfs-meta  6   26 MiB 0   124 TiB 29
>  cephfs-data   7   83 GiB  0.07   124 TiB 21175
>  new-ec-pool  8  641 GiB  0.25   247 TiB 163991
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Migrating a cephfs data pool

2019-06-28 Thread Jorge Garcia 
This was after a while (I did notice that the number of objects went 
higher before it went lower). It is actually reporting more objects now. 
I'm not sure if some co-worker or program is writing to the 
filesystem... It got to these numbers and hasn't changed for the past 
couple hours.


# ceph df
GLOBAL:
    SIZE    AVAIL   RAW USED %RAW USED
    392 TiB 391 TiB  1.3 TiB  0.33
POOLS:
    NAME    ID USED    %USED MAX AVAIL OBJECTS
    cephfs-meta  6   27 MiB 0   124 TiB 29
    cephfs-data   7  100 GiB  0.08   124 TiB 25600
    new-ec-pool  8  641 GiB  0.25   247 TiB 163991

On 6/28/19 4:04 PM, Patrick Hein wrote:
Afaik MDS doesn't delete the objects immediately but defer it for 
later. If you check that again now, how many objects does it report?


Jorge Garcia mailto:jgar...@soe.ucsc.edu>> 
schrieb am Fr., 28. Juni 2019, 23:16:



On 6/28/19 9:02 AM, Marc Roos wrote:
> 3. When everything is copied-removed, you should end up with an
empty
> datapool with zero objects.

I copied the data to a new directory and then removed the data
from the
old directory, but df still reports some objects in the old pool (not
zero). Is there a way to track down what's still in the old pool, and
how to delete it?

Before delete:

# ceph df
GLOBAL:
 SIZE    AVAIL   RAW USED %RAW USED
 392 TiB 389 TiB  3.3 TiB  0.83
POOLS:
 NAME    ID USED    %USED MAX AVAIL OBJECTS
 cephfs-meta  6   17 MiB 0   123 TiB 27
 cephfs-data   7  763 GiB  0.60   123 TiB 195233
 new-ec-pool  8  641 GiB  0.25   245 TiB 163991

After delete:

# ceph df
GLOBAL:
 SIZE    AVAIL   RAW USED %RAW USED
 392 TiB 391 TiB  1.2 TiB  0.32
POOLS:
 NAME    ID USED    %USED MAX AVAIL OBJECTS
 cephfs-meta  6   26 MiB 0   124 TiB 29
 cephfs-data   7   83 GiB  0.07   124 TiB 21175
 new-ec-pool  8  641 GiB  0.25   247 TiB 163991

___
ceph-users mailing list
ceph-users@lists.ceph.com 
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Migrating a cephfs data pool

2019-06-28 Thread Jorge Garcia 
Ok, actually, the problem was somebody writing to the filesystem. So I 
moved their files and got to 0 objects. But then I tried to remove the 
original data pool and got an error:


  # ceph fs rm_data_pool cephfs cephfs-data
  Error EINVAL: cannot remove default data pool

So it seems I will never be able to remove the original data pool. I 
could leave it there as a ghost pool, which is not optimal, but I guess 
there's currently not a better option.


On 6/28/19 4:04 PM, Patrick Hein wrote:
Afaik MDS doesn't delete the objects immediately but defer it for 
later. If you check that again now, how many objects does it report?


Jorge Garcia mailto:jgar...@soe.ucsc.edu>> 
schrieb am Fr., 28. Juni 2019, 23:16:



On 6/28/19 9:02 AM, Marc Roos wrote:
> 3. When everything is copied-removed, you should end up with an
empty
> datapool with zero objects.

I copied the data to a new directory and then removed the data
from the
old directory, but df still reports some objects in the old pool (not
zero). Is there a way to track down what's still in the old pool, and
how to delete it?

Before delete:

# ceph df
GLOBAL:
 SIZE    AVAIL   RAW USED %RAW USED
 392 TiB 389 TiB  3.3 TiB  0.83
POOLS:
 NAME    ID USED    %USED MAX AVAIL OBJECTS
 cephfs-meta  6   17 MiB 0   123 TiB 27
 cephfs-data   7  763 GiB  0.60   123 TiB 195233
 new-ec-pool  8  641 GiB  0.25   245 TiB 163991

After delete:

# ceph df
GLOBAL:
 SIZE    AVAIL   RAW USED %RAW USED
 392 TiB 391 TiB  1.2 TiB  0.32
POOLS:
 NAME    ID USED    %USED MAX AVAIL OBJECTS
 cephfs-meta  6   26 MiB 0   124 TiB 29
 cephfs-data   7   83 GiB  0.07   124 TiB 21175
 new-ec-pool  8  641 GiB  0.25   247 TiB 163991

___
ceph-users mailing list
ceph-users@lists.ceph.com 
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] could not find secret_id--auth to unkown host

2019-06-28 Thread lin zhou 
Hi,cephers

recently I found auth error logs in most of my osds but not all,
escape some nodes I rebooted after the installation.
it looks like this osd verify to 10.108.87.250:0  first and then the
correct mon.a.

10.108.87.250 now is my radosgw node, maybe I use it as mon in my
first installation, but I rebuild all service when the next
installation.
now so how to fresh this info.

2019-06-29 09:50:49.838046 7f64c208f700 10 cephx: verify_authorizer
decrypted service osd secret_id=1018
2019-06-29 09:50:49.838069 7f64c208f700  0 auth: could not find secret_id=1018
2019-06-29 09:50:49.838074 7f64c208f700 10 auth: dump_rotating:
2019-06-29 09:50:49.838077 7f64c208f700 10 auth:  id 3642
AQA6nRZdFMiRLRAAaT2bOUegYD9CsevenMdH/Q== expires 2019-06-29
09:05:30.764503
2019-06-29 09:50:49.838131 7f64c208f700 10 auth:  id 3643
AQBMqxZdteDsHBAAE/rcaMaH+Bi2ofItBo/Kkg== expires 2019-06-29
10:05:32.485261
2019-06-29 09:50:49.838151 7f64c208f700 10 auth:  id 3644
AQBeuRZdHacnCRAAZhz3MBI46Qfhxly/NWccjg== expires 2019-06-29
11:05:34.153571
2019-06-29 09:50:49.838168 7f64c208f700  0 cephx: verify_authorizer
could not get service secret for service osd secret_id=1018
2019-06-29 09:50:49.838176 7f64c208f700  0 -- 10.108.87.52:6844/18613
>> 10.108.87.250:0/2177746984 pipe(0x564181eca000 sd=320 :6844 s=0
pgs=0 cs=0 l=1 c=0x5641842e9900).accept: got bad authorizer
2019-06-29 09:50:52.571038 7f64a1b8c700 10 _calc_signature seq 1070776
front_crc_ = 2561703746 middle_crc = 0 data_crc = 380810349 sig =
18040079941791182855
2019-06-29 09:50:52.572726 7f64a2697700 10 _calc_signature seq 1070663
front_crc_ = 706854132 middle_crc = 0 data_crc = 0 sig =
13706262240826385922
2019-06-29 09:50:52.572745 7f64a2697700 20 Putting signature in client
message(seq # 1070663): sig = 13706262240826385922
2019-06-29 09:50:52.600227 7f64a1b8c700 10 _calc_signature seq 1070777
front_crc_ = 2099810396 middle_crc = 0 data_crc = 419609417 sig =
7933649248258496080
2019-06-29 09:50:52.601286 7f64a2697700 10 _calc_signature seq 1070664
front_crc_ = 3085931208 middle_crc = 0 data_crc = 0 sig =
17236156192840324023
2019-06-29 09:50:52.601306 7f64a2697700 20 Putting signature in client
message(seq # 1070664): sig = 17236156192840324023
2019-06-29 09:50:53.964899 7f64e49e3700 10 monclient:
_send_mon_message to mon.a at 10.108.87.246:6789/0
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com