[ceph-users] Luminous radosgw hangs after a few hours

[Martin Emrich]

Since upgrading to 12.1, our Object Gateways hang after a few hours, I only see 
these messages in the log file:

2017-06-29 07:52:20.877587 7fa8e01e5700  0 ERROR: keystone revocation 
processing returned error r=-22
2017-06-29 08:07:20.877761 7fa8e01e5700  0 ERROR: keystone revocation 
processing returned error r=-22
2017-06-29 08:07:29.994979 7fa8e11e7700  0 process_single_logshard: Error in 
get_bucket_info: (2) No such file or directory
2017-06-29 08:22:20.877911 7fa8e01e5700  0 ERROR: keystone revocation 
processing returned error r=-22
2017-06-29 08:27:30.086119 7fa8e11e7700  0 process_single_logshard: Error in 
get_bucket_info: (2) No such file or directory
2017-06-29 08:37:20.878108 7fa8e01e5700  0 ERROR: keystone revocation 
processing returned error r=-22
2017-06-29 08:37:30.187696 7fa8e11e7700  0 process_single_logshard: Error in 
get_bucket_info: (2) No such file or directory
2017-06-29 08:52:20.878283 7fa8e01e5700  0 ERROR: keystone revocation 
processing returned error r=-22
2017-06-29 08:57:30.280881 7fa8e11e7700  0 process_single_logshard: Error in 
get_bucket_info: (2) No such file or directory
2017-06-29 09:07:20.878451 7fa8e01e5700  0 ERROR: keystone revocation 
processing returned error r=-22

FYI: we do not use Keystone or Openstack.

This started after upgrading from jewel (via kraken) to luminous.

What could I do to fix this?
Is there some "fsck" like consistency check + repair for the radosgw buckets?

Thanks,

Martin

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Very HIGH Disk I/O latency on instances

[Peter Maloney]

On 06/28/17 21:57, Gregory Farnum wrote:
>
>
> On Wed, Jun 28, 2017 at 9:17 AM Peter Maloney
>  > wrote:
>
> On 06/28/17 16:52, keynes_...@wistron.com
>  wrote:
>> [...]backup VMs is create a snapshot by Ceph commands (rbd
>> snapshot) then download (rbd export) it.
>>
>>  
>>
>> We found a very high Disk Read / Write latency during creating /
>> deleting snapshots, it will higher than 1 ms.
>>
>>  
>>
>> Even not during backup jobs, we often see a more than 4000 ms
>> latency occurred.
>>
>>  
>>
>> Users start to complain.
>>
>> Could you please help us to how to start the troubleshooting?
>>
>>  
>>
> For creating snaps and keeping them, this was marked wontfix
> http://tracker.ceph.com/issues/10823
>
> For deleting, see the recent "Snapshot removed, cluster thrashed"
> thread for some config to try.
>
>
> Given he says he's seeing 4 second IOs even without snapshot
> involvement, I think Keynes must be seeing something else in his cluster.

If you have few enough OSDs and slow enough journals that seem ok
without snaps, with snaps can be much worse than 4s IOs if you have any
sync heavy clients, like ganglia.

Before I figured out that it was exclusive-lock causing VMs to hang, I
tested many things and spent months on it and found that out. Also some
people in freenode irc ##proxmox channel with cheap home setups with
ceph complain about such things often.

>
>
> 
> https://storageswiss.com/2016/04/01/snapshot-101-copy-on-write-vs-redirect-on-write/
>> Consider a *copy-on-write* system, which /copies/ any blocks
>> before they are overwritten with new information (i.e. it copies
>> on writes). In other words, if a block in a protected entity is
>> to be modified, the system will copy that block to a separate
>> snapshot area before it is overwritten with the new information.
>> This approach requires three I/O operations for each write: one
>> read and two writes. [...] This decision process for each block
>> also comes with some computational overhead.
>
>> A *redirect-on-write* system uses pointers to represent all
>> protected entities. If a block needs modification, the storage
>> system merely /redirects/ the pointer for that block to another
>> block and writes the data there. [...] There is zero
>> computational overhead of reading a snapshot in a
>> redirect-on-write system.
>
>> The redirect-on-write system uses 1/3 the number of I/O
>> operations when modifying a protected block, and it uses no extra
>> computational overhead reading a snapshot. Copy-on-write systems
>> can therefore have a big impact on the performance of the
>> protected entity. The more snapshots are created and the longer
>> they are stored, the greater the impact to performance on the
>> protected entity.
>
>
> I wouldn't consider that a very realistic depiction of the tradeoffs
> involved in different snapshotting strategies[1], but BlueStore uses
> "redirect-on-write" under the formulation presented in those quotes.
> RBD clones of protected images will remain copy-on-write forever, I
> imagine.
> -Greg
It was simply the first link I found which I could quote, but I didn't
find it too bad... just it describes it like all implementations are the
same.
>
> [1]: There's no reason to expect a copy-on-write system will first
> copy the original data and then overwrite it with the new data when it
> can simply inject the new data along the way. *Some* systems will copy
> the "old" block into a new location and then overwrite in the existing
> location (it helps prevent fragmentation), but many don't. And a
> "redirect-on-write" system needs to persist all those block metadata
> pointers, which may be much cheaper or much, much more expensive than
> just duplicating the blocks.

After a snap is unprotected, will the clones be redirect-on-write? Or
after the image is flattened (like dd if=/dev/zero to the whole disk)?

Are there other cases where you get a copy-on-write behavior?

Glad to hear bluestore has something better. Is that avaliable and
default behavior on kraken (which I tested but where it didn't seem to
be fixed, although all storage backends were less block prone on kraken)?

If it was a true redirect-on-write system, I would expect that when you
make a snap, there is just the overhead of organizing some metadata, and
then after that, any writes just write as normal, to a new place, not
requiring the old data to be copied, ideally not any of it, even
partially written objects. And I don't think I saw that behavior on my
kraken tests, although the performance was better (due to no blocked
requests, but the iops at peak was basically the same; and I didn't
measure total IO or something that would be more reliable...just looked
at performance effects and blocking).

[ceph-users] Ceph New OSD cannot be started

[Luescher Claude]

Hello,

I have a cluster of 3 debian ceph machines running version:

ceph version 0.80.1 (a38fe1169b6d2ac98b427334c12d7cf81f809b74)

A new disk was added to one node but it does not want to start it. I 
have tried everything like removing and readding the disk any times.


The current ceph osd tree:

# idweight  type name   up/down reweight
-1  29  root default
-2  9.06host store1
0   1.81osd.0   up  1
5   1.81osd.5   up  1
6   1.81osd.6   up  1
9   3.63osd.9   up  1
-3  9.06host store3
1   1.81osd.1   up  1
2   1.81osd.2   up  1
8   1.81osd.8   up  1
11  3.63osd.11  up  1
-4  10.88   host store2
3   1.81osd.3   up  1
7   1.81osd.7   up  1
10  3.63osd.10  up  1
4   3.63osd.4   down0   < problem is with this 
disk

All the disks are 4TB.

/etc/init.d/ceph start osd.4
=== osd.4 ===
create-or-move updated item name 'osd.4' weight 3.63 at location 
{host=store2,root=default} to crush map

Starting Ceph osd.4 on store2...
starting osd.4 at :/0 osd_data /var/lib/ceph/osd/ceph-4 
/var/lib/ceph/osd/ceph-4/journal


If I look into the disk, it did create the basic ceph directory but it 
does not use it:


/dev/sdc13.7T  5.1G  3.7T   1% /var/lib/ceph/osd/ceph-4


The ceph health status:

cluster 
 health HEALTH_WARN 4 pgs degraded; 113 pgs stuck unclean; recovery 
45/5084377 objects degraded (0.001%); 1 near full osd(s)
 monmap e3: 3 mons at 
{cephmon1=IP1:6789/0,store2=IP2:6789/0,store3=IP3:6789/0}, election 
epoch 6980, quorum 0,1,2 store2,store3,cephmon1

 osdmap e5709: 12 osds: 11 up, 11 in
  pgmap v79160817: 1216 pgs, 5 pools, 9763 GB data, 2481 kobjects
19644 GB used, 6370 GB / 26014 GB avail
45/5084377 objects degraded (0.001%)
1103 active+clean
   4 active+degraded
 109 active+remapped
  client io 21341 B/s rd, 477 kB/s wr, 118 op/s


Any idea how to fix this? If it possible without upgrade. I don't want 
to upgrade this cluster to other version ever. It does it's job as it 
should.



Thank you,
Claude
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph New OSD cannot be started

[Eugen Block]

Hi,

what does systemctl status -l ceph-osd@4.service say? Is anything  
suspicious in the syslog?


Regards,
Eugen


Zitat von Luescher Claude :


Hello,

I have a cluster of 3 debian ceph machines running version:

ceph version 0.80.1 (a38fe1169b6d2ac98b427334c12d7cf81f809b74)

A new disk was added to one node but it does not want to start it. I  
have tried everything like removing and readding the disk any times.


The current ceph osd tree:

# idweight  type name   up/down reweight
-1  29  root default
-2  9.06host store1
0   1.81osd.0   up  1
5   1.81osd.5   up  1
6   1.81osd.6   up  1
9   3.63osd.9   up  1
-3  9.06host store3
1   1.81osd.1   up  1
2   1.81osd.2   up  1
8   1.81osd.8   up  1
11  3.63osd.11  up  1
-4  10.88   host store2
3   1.81osd.3   up  1
7   1.81osd.7   up  1
10  3.63osd.10  up  1
4   3.63osd.4   down0   < problem is with this 
disk

All the disks are 4TB.

/etc/init.d/ceph start osd.4
=== osd.4 ===
create-or-move updated item name 'osd.4' weight 3.63 at location  
{host=store2,root=default} to crush map

Starting Ceph osd.4 on store2...
starting osd.4 at :/0 osd_data /var/lib/ceph/osd/ceph-4  
/var/lib/ceph/osd/ceph-4/journal


If I look into the disk, it did create the basic ceph directory but  
it does not use it:


/dev/sdc13.7T  5.1G  3.7T   1% /var/lib/ceph/osd/ceph-4


The ceph health status:

cluster 
 health HEALTH_WARN 4 pgs degraded; 113 pgs stuck unclean;  
recovery 45/5084377 objects degraded (0.001%); 1 near full osd(s)
 monmap e3: 3 mons at  
{cephmon1=IP1:6789/0,store2=IP2:6789/0,store3=IP3:6789/0}, election  
epoch 6980, quorum 0,1,2 store2,store3,cephmon1

 osdmap e5709: 12 osds: 11 up, 11 in
  pgmap v79160817: 1216 pgs, 5 pools, 9763 GB data, 2481 kobjects
19644 GB used, 6370 GB / 26014 GB avail
45/5084377 objects degraded (0.001%)
1103 active+clean
   4 active+degraded
 109 active+remapped
  client io 21341 B/s rd, 477 kB/s wr, 118 op/s


Any idea how to fix this? If it possible without upgrade. I don't  
want to upgrade this cluster to other version ever. It does it's job  
as it should.



Thank you,
Claude
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




--
Eugen Block voice   : +49-40-559 51 75
NDE Netzdesign und -entwicklung AG  fax : +49-40-559 51 77
Postfach 61 03 15
D-22423 Hamburg e-mail  : ebl...@nde.ag

Vorsitzende des Aufsichtsrates: Angelika Mozdzen
  Sitz und Registergericht: Hamburg, HRB 90934
  Vorstand: Jens-U. Mozdzen
   USt-IdNr. DE 814 013 983

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph New OSD cannot be started

[Christian Balzer]

Hello,

On Thu, 29 Jun 2017 08:53:25 + Eugen Block wrote:

> Hi,
> 
> what does systemctl status -l ceph-osd@4.service say? Is anything  
> suspicious in the syslog?
>
I'm pretty sure (the OP didn't specify that or other obvious things) that
his is a Debian version that predates the horror that is systemd, given
the Ceph version alone.

To the OP, more info, like the Debian version and what Eugen said, logs,
logs, logs.

More below.

> Regards,
> Eugen
> 
> 
> Zitat von Luescher Claude :
> 
> > Hello,
> >
> > I have a cluster of 3 debian ceph machines running version:
> >
> > ceph version 0.80.1 (a38fe1169b6d2ac98b427334c12d7cf81f809b74)
> >
> > A new disk was added to one node but it does not want to start it. I  
> > have tried everything like removing and readding the disk any times.
> >
How did you do that, manually or with ceph-deploy?

> > The current ceph osd tree:
> >
> > # idweight  type name   up/down reweight
> > -1  29  root default
> > -2  9.06host store1
> > 0   1.81osd.0   up  1
> > 5   1.81osd.5   up  1
> > 6   1.81osd.6   up  1
> > 9   3.63osd.9   up  1
> > -3  9.06host store3
> > 1   1.81osd.1   up  1
> > 2   1.81osd.2   up  1
> > 8   1.81osd.8   up  1
> > 11  3.63osd.11  up  1
> > -4  10.88   host store2
> > 3   1.81osd.3   up  1
> > 7   1.81osd.7   up  1
> > 10  3.63osd.10  up  1
> > 4   3.63osd.4   down0   < problem is with this 
> > disk
> >
> > All the disks are 4TB.
> >
Why the different weights then?

> > /etc/init.d/ceph start osd.4
> > === osd.4 ===
> > create-or-move updated item name 'osd.4' weight 3.63 at location  
> > {host=store2,root=default} to crush map
> > Starting Ceph osd.4 on store2...
> > starting osd.4 at :/0 osd_data /var/lib/ceph/osd/ceph-4  
> > /var/lib/ceph/osd/ceph-4/journal
> >
> > If I look into the disk, it did create the basic ceph directory but  
> > it does not use it:
> >
> > /dev/sdc13.7T  5.1G  3.7T   1% /var/lib/ceph/osd/ceph-4
> >
So the disk got mounted, are all the usual suspects in that directory?

Christian
> >
> > The ceph health status:
> >
> > cluster 
> >  health HEALTH_WARN 4 pgs degraded; 113 pgs stuck unclean;  
> > recovery 45/5084377 objects degraded (0.001%); 1 near full osd(s)
> >  monmap e3: 3 mons at  
> > {cephmon1=IP1:6789/0,store2=IP2:6789/0,store3=IP3:6789/0}, election  
> > epoch 6980, quorum 0,1,2 store2,store3,cephmon1
> >  osdmap e5709: 12 osds: 11 up, 11 in
> >   pgmap v79160817: 1216 pgs, 5 pools, 9763 GB data, 2481 kobjects
> > 19644 GB used, 6370 GB / 26014 GB avail
> > 45/5084377 objects degraded (0.001%)
> > 1103 active+clean
> >4 active+degraded
> >  109 active+remapped
> >   client io 21341 B/s rd, 477 kB/s wr, 118 op/s
> >
> >
> > Any idea how to fix this? If it possible without upgrade. I don't  
> > want to upgrade this cluster to other version ever. It does it's job  
> > as it should.
> >
> >
> > Thank you,
> > Claude
> > ___
> > ceph-users mailing list
> > ceph-users@lists.ceph.com
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com  
> 
> 
> 


-- 
Christian BalzerNetwork/Systems Engineer
ch...@gol.com   Rakuten Communications
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph New OSD cannot be started

[Eugen Block]

I'm pretty sure (the OP didn't specify that or other obvious things) that
his is a Debian version that predates the horror that is systemd, given
the Ceph version alone.


I thought about that, but was not really sure since I dived into ceph  
only a year ago ;-) In this case I mean the systemv equivalent, of  
course ;-)



Zitat von Christian Balzer :


Hello,

On Thu, 29 Jun 2017 08:53:25 + Eugen Block wrote:


Hi,

what does systemctl status -l ceph-osd@4.service say? Is anything
suspicious in the syslog?


I'm pretty sure (the OP didn't specify that or other obvious things) that
his is a Debian version that predates the horror that is systemd, given
the Ceph version alone.

To the OP, more info, like the Debian version and what Eugen said, logs,
logs, logs.

More below.


Regards,
Eugen


Zitat von Luescher Claude :

> Hello,
>
> I have a cluster of 3 debian ceph machines running version:
>
> ceph version 0.80.1 (a38fe1169b6d2ac98b427334c12d7cf81f809b74)
>
> A new disk was added to one node but it does not want to start it. I
> have tried everything like removing and readding the disk any times.
>

How did you do that, manually or with ceph-deploy?


> The current ceph osd tree:
>
> # id   weight  type name   up/down reweight
> -1 29  root default
> -2 9.06host store1
> 0  1.81osd.0   up  1
> 5  1.81osd.5   up  1
> 6  1.81osd.6   up  1
> 9  3.63osd.9   up  1
> -3 9.06host store3
> 1  1.81osd.1   up  1
> 2  1.81osd.2   up  1
> 8  1.81osd.8   up  1
> 11 3.63osd.11  up  1
> -4 10.88   host store2
> 3  1.81osd.3   up  1
> 7  1.81osd.7   up  1
> 10 3.63osd.10  up  1
> 4  3.63osd.4   down0   < problem is with this disk
>
> All the disks are 4TB.
>

Why the different weights then?


> /etc/init.d/ceph start osd.4
> === osd.4 ===
> create-or-move updated item name 'osd.4' weight 3.63 at location
> {host=store2,root=default} to crush map
> Starting Ceph osd.4 on store2...
> starting osd.4 at :/0 osd_data /var/lib/ceph/osd/ceph-4
> /var/lib/ceph/osd/ceph-4/journal
>
> If I look into the disk, it did create the basic ceph directory but
> it does not use it:
>
> /dev/sdc13.7T  5.1G  3.7T   1% /var/lib/ceph/osd/ceph-4
>

So the disk got mounted, are all the usual suspects in that directory?

Christian

>
> The ceph health status:
>
> cluster 
>  health HEALTH_WARN 4 pgs degraded; 113 pgs stuck unclean;
> recovery 45/5084377 objects degraded (0.001%); 1 near full osd(s)
>  monmap e3: 3 mons at
> {cephmon1=IP1:6789/0,store2=IP2:6789/0,store3=IP3:6789/0}, election
> epoch 6980, quorum 0,1,2 store2,store3,cephmon1
>  osdmap e5709: 12 osds: 11 up, 11 in
>   pgmap v79160817: 1216 pgs, 5 pools, 9763 GB data, 2481 kobjects
> 19644 GB used, 6370 GB / 26014 GB avail
> 45/5084377 objects degraded (0.001%)
> 1103 active+clean
>4 active+degraded
>  109 active+remapped
>   client io 21341 B/s rd, 477 kB/s wr, 118 op/s
>
>
> Any idea how to fix this? If it possible without upgrade. I don't
> want to upgrade this cluster to other version ever. It does it's job
> as it should.
>
>
> Thank you,
> Claude
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com






--
Christian BalzerNetwork/Systems Engineer
ch...@gol.com   Rakuten Communications




--
Eugen Block voice   : +49-40-559 51 75
NDE Netzdesign und -entwicklung AG  fax : +49-40-559 51 77
Postfach 61 03 15
D-22423 Hamburg e-mail  : ebl...@nde.ag

Vorsitzende des Aufsichtsrates: Angelika Mozdzen
  Sitz und Registergericht: Hamburg, HRB 90934
  Vorstand: Jens-U. Mozdzen
   USt-IdNr. DE 814 013 983

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] What caps are necessary for FUSE-mounts of the FS?

[Riccardo Murri]

Hello!

The documentation at  states:

"""
Before mounting a Ceph File System in User Space (FUSE), ensure that
the client host has a copy of the Ceph configuration file and a
keyring with CAPS for the Ceph metadata server.
"""

Now, I have two questions:

1. What capabilities should be given to the FS user in the keyring?
Would this be creating a keyring file with the minimally-required caps
to mount the FS read+write?

ceph-authtool --create-keyring ceph.fs.keyring --gen-key
--caps mds 'allow rwx'

2. Is the full config file needed on the client?  Or is it that just
the MON addresses are needed? In the latter case, is it enough to pass
the MON addr:port through the `-m` option and *not* have a `ceph.conf`
file?

Thanks for any help!

Riccardo
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Cannot mount Ceph FS

[Riccardo Murri]

Hello!

I tried to create and mount a filesystem using the instructions at
 and
 but I am getting
errors:

$ sudo ceph fs new cephfs cephfs_metadata cephfs_data
new fs with metadata pool 1 and data pool 2
$ sudo ceph mds stat
e6: 0/0/1 up
$ sudo mount -t ceph mds001:/ /mnt -o
name=admin,secretfile=/etc/ceph/client.admin.secret
mount error 110 = Connection timed out

I found this `mds cluster_up` command and thought I need to bring the
MDS cluster up before using FS functions but I get errors there as
well:

$ sudo ceph mds cluster_up
unmarked fsmap DOWN

Still, the cluster does not show any health issue:

$ sudo ceph -s
cluster 00baac7a-0ad4-4ab7-9d5e-fdaf7d122aee
 health HEALTH_OK
 monmap e1: 1 mons at {mon001=172.23.140.181:6789/0}
election epoch 3, quorum 0 mon001
  fsmap e7: 0/0/1 up
 osdmap e19: 3 osds: 3 up, 3 in
flags sortbitwise,require_jewel_osds
  pgmap v1278: 192 pgs, 3 pools, 0 bytes data, 0 objects
9728 MB used, 281 GB / 290 GB avail
 192 active+clean

Any hints?  What I am doing wrong?

Thanks,
Riccardo
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Cannot mount Ceph FS

[Burkhard Linke]

Hi,


On 06/29/2017 01:26 PM, Riccardo Murri wrote:

Hello!

I tried to create and mount a filesystem using the instructions at
 and
 but I am getting
errors:

$ sudo ceph fs new cephfs cephfs_metadata cephfs_data
new fs with metadata pool 1 and data pool 2
$ sudo ceph mds stat
e6: 0/0/1 up
$ sudo mount -t ceph mds001:/ /mnt -o
name=admin,secretfile=/etc/ceph/client.admin.secret
mount error 110 = Connection timed out

I found this `mds cluster_up` command and thought I need to bring the
MDS cluster up before using FS functions but I get errors there as
well:

$ sudo ceph mds cluster_up
unmarked fsmap DOWN

Still, the cluster does not show any health issue:

$ sudo ceph -s
 cluster 00baac7a-0ad4-4ab7-9d5e-fdaf7d122aee
  health HEALTH_OK
  monmap e1: 1 mons at {mon001=172.23.140.181:6789/0}
 election epoch 3, quorum 0 mon001
   fsmap e7: 0/0/1 up
  osdmap e19: 3 osds: 3 up, 3 in
 flags sortbitwise,require_jewel_osds
   pgmap v1278: 192 pgs, 3 pools, 0 bytes data, 0 objects
 9728 MB used, 281 GB / 290 GB avail
  192 active+clean

Any hints?  What I am doing wrong?


You need a running MDS daemon for CephFS.

Regards,
Burkhard Linke
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Cannot mount Ceph FS

[Jaime Ibar]

Hi,

I'd say there is no mds running

$ ceph mds stat
e47262: 1/1/1 up {0=ceph01=up:active}, 2 up:standby

$ ceph -s

[...]

  fsmap e47262: 1/1/1 up {0=ceph01=up:active}, 2 up:standby

[...]

Is mds up and running?

Jaime


On 29/06/17 12:26, Riccardo Murri wrote:

Hello!

I tried to create and mount a filesystem using the instructions at
 and
 but I am getting
errors:

$ sudo ceph fs new cephfs cephfs_metadata cephfs_data
new fs with metadata pool 1 and data pool 2
$ sudo ceph mds stat
e6: 0/0/1 up
$ sudo mount -t ceph mds001:/ /mnt -o
name=admin,secretfile=/etc/ceph/client.admin.secret
mount error 110 = Connection timed out

I found this `mds cluster_up` command and thought I need to bring the
MDS cluster up before using FS functions but I get errors there as
well:

$ sudo ceph mds cluster_up
unmarked fsmap DOWN

Still, the cluster does not show any health issue:

$ sudo ceph -s
 cluster 00baac7a-0ad4-4ab7-9d5e-fdaf7d122aee
  health HEALTH_OK
  monmap e1: 1 mons at {mon001=172.23.140.181:6789/0}
 election epoch 3, quorum 0 mon001
   fsmap e7: 0/0/1 up
  osdmap e19: 3 osds: 3 up, 3 in
 flags sortbitwise,require_jewel_osds
   pgmap v1278: 192 pgs, 3 pools, 0 bytes data, 0 objects
 9728 MB used, 281 GB / 290 GB avail
  192 active+clean

Any hints?  What I am doing wrong?

Thanks,
Riccardo
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



--

Jaime Ibar
High Performance & Research Computing, IS Services
Lloyd Building, Trinity College Dublin, Dublin 2, Ireland.
http://www.tchpc.tcd.ie/ | ja...@tchpc.tcd.ie
Tel: +353-1-896-3725

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] What caps are necessary for FUSE-mounts of the FS?

[John Spray]

On Thu, Jun 29, 2017 at 11:42 AM, Riccardo Murri
 wrote:
> Hello!
>
> The documentation at  states:
>
> """
> Before mounting a Ceph File System in User Space (FUSE), ensure that
> the client host has a copy of the Ceph configuration file and a
> keyring with CAPS for the Ceph metadata server.
> """
>
> Now, I have two questions:
>
> 1. What capabilities should be given to the FS user in the keyring?
> Would this be creating a keyring file with the minimally-required caps
> to mount the FS read+write?
>
> ceph-authtool --create-keyring ceph.fs.keyring --gen-key
> --caps mds 'allow rwx'

Docs are here:
http://docs.ceph.com/docs/master/cephfs/client-auth/

> 2. Is the full config file needed on the client?  Or is it that just
> the MON addresses are needed? In the latter case, is it enough to pass
> the MON addr:port through the `-m` option and *not* have a `ceph.conf`
> file?

Yes.  You can use -m and --keyring.  In practice, it is usually useful
to have a ceph.conf anyway so that you have a place to add e.g. any
debug log settings you need.

John

>
> Thanks for any help!
>
> Riccardo
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Cannot mount Ceph FS

[John Spray]

On Thu, Jun 29, 2017 at 12:26 PM, Riccardo Murri
 wrote:
> Hello!
>
> I tried to create and mount a filesystem using the instructions at
>  and
>  but I am getting
> errors:
>
> $ sudo ceph fs new cephfs cephfs_metadata cephfs_data
> new fs with metadata pool 1 and data pool 2
> $ sudo ceph mds stat
> e6: 0/0/1 up
> $ sudo mount -t ceph mds001:/ /mnt -o
> name=admin,secretfile=/etc/ceph/client.admin.secret
> mount error 110 = Connection timed out

Unless mds001 happens to also be a mon node, that's the problem -- the
addresses in the mount command are supposed to be the mons.

John

>
> I found this `mds cluster_up` command and thought I need to bring the
> MDS cluster up before using FS functions but I get errors there as
> well:
>
> $ sudo ceph mds cluster_up
> unmarked fsmap DOWN
>
> Still, the cluster does not show any health issue:
>
> $ sudo ceph -s
> cluster 00baac7a-0ad4-4ab7-9d5e-fdaf7d122aee
>  health HEALTH_OK
>  monmap e1: 1 mons at {mon001=172.23.140.181:6789/0}
> election epoch 3, quorum 0 mon001
>   fsmap e7: 0/0/1 up
>  osdmap e19: 3 osds: 3 up, 3 in
> flags sortbitwise,require_jewel_osds
>   pgmap v1278: 192 pgs, 3 pools, 0 bytes data, 0 objects
> 9728 MB used, 281 GB / 290 GB avail
>  192 active+clean
>
> Any hints?  What I am doing wrong?
>
> Thanks,
> Riccardo
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Cannot mount Ceph FS

[John Spray]

On Thu, Jun 29, 2017 at 12:53 PM, Riccardo Murri
 wrote:
> On 29 June 2017 at 13:48, John Spray  wrote:
>> On Thu, Jun 29, 2017 at 12:26 PM, Riccardo Murri
>>  wrote:
>>> Hello!
>>>
>>> I tried to create and mount a filesystem using the instructions at
>>>  and
>>>  but I am getting
>>> errors:
>>>
>>> $ sudo ceph fs new cephfs cephfs_metadata cephfs_data
>>> new fs with metadata pool 1 and data pool 2
>>> $ sudo ceph mds stat
>>> e6: 0/0/1 up
>>> $ sudo mount -t ceph mds001:/ /mnt -o
>>> name=admin,secretfile=/etc/ceph/client.admin.secret
>>> mount error 110 = Connection timed out
>>
>> Unless mds001 happens to also be a mon node, that's the problem -- the
>> addresses in the mount command are supposed to be the mons.
>
> Sorry, I copy+paste'd the wrong line.  Yes, I tried with `mon001` as well.
> It fails eventually (takes much longer) with error 5.

In that case you're probably using an old kernel.  Sufficiently old
kernels will not work with new ceph clusters.  Check if it works with
ceph-fuse.

John

>
> Thanks,
> Riccardo
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Cannot mount Ceph FS

[Riccardo Murri]

On 29 June 2017 at 13:48, John Spray  wrote:
> On Thu, Jun 29, 2017 at 12:26 PM, Riccardo Murri
>  wrote:
>> Hello!
>>
>> I tried to create and mount a filesystem using the instructions at
>>  and
>>  but I am getting
>> errors:
>>
>> $ sudo ceph fs new cephfs cephfs_metadata cephfs_data
>> new fs with metadata pool 1 and data pool 2
>> $ sudo ceph mds stat
>> e6: 0/0/1 up
>> $ sudo mount -t ceph mds001:/ /mnt -o
>> name=admin,secretfile=/etc/ceph/client.admin.secret
>> mount error 110 = Connection timed out
>
> Unless mds001 happens to also be a mon node, that's the problem -- the
> addresses in the mount command are supposed to be the mons.

Sorry, I copy+paste'd the wrong line.  Yes, I tried with `mon001` as well.
It fails eventually (takes much longer) with error 5.

Thanks,
Riccardo
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph and IPv4 -> IPv6

[Brenno Augusto Falavinha Martinez]

What about moving just gateways to ipv6? is that possible?
Brenno


Em 29/06/2017 03:13:26, Wido den Hollander escreveu:
> > Op 28 juni 2017 om 22:12 schreef Gregory Farnum > :
> 
> 
> On Wed, Jun 28, 2017 at 6:33 AM >  wrote:
> 
> > > I don't think either. I don't think there is another way then just
> > 'hacky' changing the MONMaps. There have been talks of being able to make
> > Ceph dual-stack, but I don't think there is any code in the source right
> > now.
> >
> > Yeah, that's what I'd like to know. What do the Ceph team think of
> > providing ways for switching?
> > We're not in any need to do so now but, it'd be good to know if the team
> > dual-stack support or at least to test and document a way to do it as
> > opposed to a "this should work but you're on your own" kind of deal.
> >
> 
> My understanding is that sepia's current infrastructure hosting can't give
> us real IPv6, so we're not in any position to really make any promises
> about interoperability. Sorry. :(
> 
> That said, I think Wido's list is the way to go — but I don't remember the
> problem with running mixed. Is it just that the messenger implementations
> don't handle it live, I think? Certainly shutting down, updating all the
> monmaps/configs (if you are setting monitor IP addresses instead of hosts),
> and then turning it all back on should work but you're on your own... ;)
> -Greg
> 

The problem with mixed is this:
- The OSDMap/MONMap can only contain ONE IP, eg, IPv4 or IPv6
- Messengers can't talk dual-stacked

So you have to shut down the whole cluster and update the MONMap. Afterwards 
the OSDs will boot and 'up' themselves in the cluster with their IPv6 address 
and that will be recorded in the OSDMap.

Wido

> 
> >
> >
> > > 
> > > From: Wido den Hollander [w...@42on.com]
> > > Sent: 27 June 2017 19:19
> > > To: Vasilakakos, George (STFC,RAL,SC); ceph-users@lists.ceph.com
> > > Subject: Re: [ceph-users] Ceph and IPv4 -> IPv6
> > >
> > > > Op 27 juni 2017 om 19:00 schreef george.vasilaka...@stfc.ac.uk:
> > > >
> > > >
> > > > Hey Ceph folks,
> > > >
> > > > I was wondering what the current status/roadmap/intentions etc. are on
> > the possibility of providing a way of transitioning a cluster from IPv4 to
> > IPv6 in the future.
> > > >
> > > > My current understanding is that this not possible at the moment and
> > that one should deploy initially with the version they want long term.
> > > >
> > > > However, given the general lack of widespread readiness, I think lots
> > of us have deployed with IPv4 and were hoping to go to IPv6 when the rest
> > of our environments enabled it.
> > > >
> > > > Is adding such a capability to a future version of Ceph being
> > considered?
> > > >
> > >
> > > I think you can, but not without downtime.
> > >
> > > The main problem is the monmap which contains IPv4 addresses and you
> > want to change that to IPv6.
> > >
> > > I haven't tried this, but I think you should be able to:
> > > - Extract MONMap
> > > - Update the IPv4 addresses to IPv6 using monmaptool
> > > - Set noout flag
> > > - Stop all OSDs
> > > - Inject new monmap
> > > - Stop MONs
> > > - Make sure IPv6 is fixed on MONs
> > > - Start MONs
> > > - Start OSDs
> > >
> > > Again, this is from the top of my head, haven't tried it, but something
> > like that should probably work.
> > >
> > > Wido
> > >
> > >
> > > >
> > > > Best regards,
> > > >
> > > > George V.
> > > > ___
> > > > ceph-users mailing list
> > > > ceph-users@lists.ceph.com
> > > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com> 
> > ___
> > ceph-users mailing list
> > ceph-users@lists.ceph.com
> > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com> 
> >
___
ceph-users mailing list
ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com> 
> @stfc.ac.uk>> @redhat.com>


-


"Esta mensagem do SERVIÇO FEDERAL DE PROCESSAMENTO DE DADOS (SERPRO), empresa 
pública federal regida pelo disposto na Lei Federal nº 5.615, é enviada 
exclusivamente a seu destinatário e pode conter informações confidenciais, 
protegidas por sigilo profissional. Sua utilização desautorizada é ilegal e 
sujeita o infrator às penas da lei. Se você a recebeu indevidamente, queira, 
por gentileza, reenviá-la ao emitente, esclarecendo o equívoco."

"This message from SERVIÇO FEDERAL DE PROCESSAMENTO DE DADOS (SERPRO) -- a 
government company established under Brazilian law (5.615/70) -- is directed 
exclusively to its addressee and may contain confidential data, protected under 
professional secrecy rules. Its unauthorized use is illegal and may subject the 
transgressor to the law's penalties. If you're not the addressee, please send 
it back, elucidating the failure."
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ce

Re: [ceph-users] Ceph and IPv4 -> IPv6

[Wido den Hollander]

> Op 29 juni 2017 om 14:50 schreef Brenno Augusto Falavinha Martinez 
> :
> 
> 
> What about moving just gateways to ipv6? is that possible?

I assume you mean the RGW? Yes, no problem!

I have it running the other way around. The RGW has IPv4 and IPv6, but the Ceph 
cluster is IPv6-only.

RGW/librados talks to Ceph ovre IPv6 and handles client traffic on both 
protocols.

No problem to run the RGW dual-stacked.

Wido

> Brenno
> 
> 
> Em 29/06/2017 03:13:26, Wido den Hollander escreveu:
> > > Op 28 juni 2017 om 22:12 schreef Gregory Farnum > :
> > 
> > 
> > On Wed, Jun 28, 2017 at 6:33 AM >  wrote:
> > 
> > > > I don't think either. I don't think there is another way then just
> > > 'hacky' changing the MONMaps. There have been talks of being able to make
> > > Ceph dual-stack, but I don't think there is any code in the source right
> > > now.
> > >
> > > Yeah, that's what I'd like to know. What do the Ceph team think of
> > > providing ways for switching?
> > > We're not in any need to do so now but, it'd be good to know if the team
> > > dual-stack support or at least to test and document a way to do it as
> > > opposed to a "this should work but you're on your own" kind of deal.
> > >
> > 
> > My understanding is that sepia's current infrastructure hosting can't give
> > us real IPv6, so we're not in any position to really make any promises
> > about interoperability. Sorry. :(
> > 
> > That said, I think Wido's list is the way to go — but I don't remember the
> > problem with running mixed. Is it just that the messenger implementations
> > don't handle it live, I think? Certainly shutting down, updating all the
> > monmaps/configs (if you are setting monitor IP addresses instead of hosts),
> > and then turning it all back on should work but you're on your own... ;)
> > -Greg
> > 
> 
> The problem with mixed is this:
> - The OSDMap/MONMap can only contain ONE IP, eg, IPv4 or IPv6
> - Messengers can't talk dual-stacked
> 
> So you have to shut down the whole cluster and update the MONMap. Afterwards 
> the OSDs will boot and 'up' themselves in the cluster with their IPv6 address 
> and that will be recorded in the OSDMap.
> 
> Wido
> 
> > 
> > >
> > >
> > > > 
> > > > From: Wido den Hollander [w...@42on.com]
> > > > Sent: 27 June 2017 19:19
> > > > To: Vasilakakos, George (STFC,RAL,SC); ceph-users@lists.ceph.com
> > > > Subject: Re: [ceph-users] Ceph and IPv4 -> IPv6
> > > >
> > > > > Op 27 juni 2017 om 19:00 schreef george.vasilaka...@stfc.ac.uk:
> > > > >
> > > > >
> > > > > Hey Ceph folks,
> > > > >
> > > > > I was wondering what the current status/roadmap/intentions etc. are on
> > > the possibility of providing a way of transitioning a cluster from IPv4 to
> > > IPv6 in the future.
> > > > >
> > > > > My current understanding is that this not possible at the moment and
> > > that one should deploy initially with the version they want long term.
> > > > >
> > > > > However, given the general lack of widespread readiness, I think lots
> > > of us have deployed with IPv4 and were hoping to go to IPv6 when the rest
> > > of our environments enabled it.
> > > > >
> > > > > Is adding such a capability to a future version of Ceph being
> > > considered?
> > > > >
> > > >
> > > > I think you can, but not without downtime.
> > > >
> > > > The main problem is the monmap which contains IPv4 addresses and you
> > > want to change that to IPv6.
> > > >
> > > > I haven't tried this, but I think you should be able to:
> > > > - Extract MONMap
> > > > - Update the IPv4 addresses to IPv6 using monmaptool
> > > > - Set noout flag
> > > > - Stop all OSDs
> > > > - Inject new monmap
> > > > - Stop MONs
> > > > - Make sure IPv6 is fixed on MONs
> > > > - Start MONs
> > > > - Start OSDs
> > > >
> > > > Again, this is from the top of my head, haven't tried it, but something
> > > like that should probably work.
> > > >
> > > > Wido
> > > >
> > > >
> > > > >
> > > > > Best regards,
> > > > >
> > > > > George V.
> > > > > ___
> > > > > ceph-users mailing list
> > > > > ceph-users@lists.ceph.com
> > > > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com> 
> > > ___
> > > ceph-users mailing list
> > > ceph-users@lists.ceph.com
> > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com> 
> > >
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com> 
> > @stfc.ac.uk>> @redhat.com>
> 
> 
> -
> 
> 
> "Esta mensagem do SERVIÇO FEDERAL DE PROCESSAMENTO DE DADOS (SERPRO), empresa 
> pública federal regida pelo disposto na Lei Federal nº 5.615, é enviada 
> exclusivamente a seu destinatário e pode conter informações confidenciais, 
> protegidas por sigilo profissional. Sua utilização desautorizada é ilegal e 
> sujeita o infrator às penas da lei. Se você a recebeu indevidamente, queira, 

Re: [ceph-users] Cannot mount Ceph FS

[Riccardo Murri]

On 29 June 2017 at 13:58, Jaime Ibar  wrote:
>
> I'd say there is no mds running

This was indeed the issue: the configuration file named the MDS as `mds.0`
which is not allowed (error message was "MDS names may not start with
a numeric digit.")
so the MDS daemon kept "committing suicide" (unnoticed at the start,
since the daemon seems
to start correctly and dies later on).

Thank you all for the help!

Riccardo
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] risk mitigation in 2 replica clusters

[Lars Marowsky-Bree]

On 2017-06-22T00:51:38, Blair Bethwaite  wrote:

> I'm doing some work to evaluate the risks involved in running 2r storage
> pools. On the face of it my naive disk failure calculations give me 4-5
> nines for a 2r pool of 100 OSDs (no copyset awareness, i.e., secondary disk
> failure based purely on chance of any 1 of the remaining 99 OSDs failing
> within recovery time). 5 nines is just fine for our purposes, but of course
> multiple disk failures are only part of the story.

You are confounding availability with data durability, too.

"Traditional" multi-node replicated storage solutions can get away with
only two nodes to mirrot the data inbetween because they typically have
an additional RAID5/6 at the local node level. (Which also helps with
recovery impact of a single device failure.) Ceph typically doesn't.

(That's why rbd-mirror between two Ceph clusters can be OK too.)

A disk failing while a node is down, or being rebooted, ...


> thereof, e.g., something that would enable the functional equivalent of:
> "this OSD/node is going to go offline so please create a 3rd replica in
> every PG it is participating in before we shutdown that/those OSD/s"...?

You can evacuate the node by setting it's weight to 0. It's a very
expensive operation, just like your proposal would be.


Regards,
Lars

-- 
Architect SDS
SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 
(AG Nürnberg)
"Experience is the name everyone gives to their mistakes." -- Oscar Wilde

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] v12.1.0 Luminous RC released

[Lars Marowsky-Bree]

On 2017-06-26T11:28:36, Ashley Merrick  wrote:

> With the EC Overwite support, if currently running behind a cache tier in 
> Jewel will the overwrite still be of benefit through the cache tier and 
> remove the need to promote the full block to make any edits?
> 
> Or we better totally removing the cache tier once fully upgraded?

I think cache tiering still promotes the whole object.

You will no longer *need* the cache tier from a functional perspective,
but it might still provide a significant performance advantage. 

Is that worth the additional capacity needs and complexity? (Plus the
promote/demote traffic, etc.)

This all will depend on your workload.


Regards,
Lars

-- 
Architect SDS
SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 
(AG Nürnberg)
"Experience is the name everyone gives to their mistakes." -- Oscar Wilde

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Kernel mounted RBD's hanging

[Nick Fisk]

Hi All,

Putting out a call for help to see if anyone can shed some light on this.

Configuration:
Ceph cluster presenting RBD's->XFS->NFS->ESXi
Running 10.2.7 on the OSD's and 4.11 kernel on the NFS gateways in a
pacemaker cluster
Both OSD's and clients are go into a pair of switches, single L2 domain (no
sign from pacemaker that there is network connectivity issues)

Symptoms:
- All RBD's on a single client randomly hang for 30s to several minutes,
confirmed by pacemaker and ESXi hosts complaining
- Cluster load is minimal when this happens most times
- All other clients with RBD's are not affected (Same RADOS pool), so its
seems more of a client issue than cluster issue
- It looks like pacemaker tries to also stop RBD+FS resource, but this also
hangs
- Eventually pacemaker succeeds in stopping resources and immediately
restarts them, IO returns to normal
- No errors, slow requests, or any other non normal Ceph status is reported
on the cluster or ceph.log
- Client logs show nothing apart from pacemaker

Things I've tried:
- Different kernels (potentially happened less with older kernels, but can't
be 100% sure)
- Disabling scrubbing and anything else that could be causing high load
- Enabling Kernel RBD debugging (Problem maybe happens a couple of times a
day, debug logging was not practical as I can't reproduce on demand)

Anyone have any ideas?

Thanks,
Nick

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] free space calculation

[Papp Rudolf Péter]

Looks like I found the answer. The preparation was not the proper way. I 
found valuable information in ceph-disk prepare --help page; the cluster 
oprating better way:


NAMEMAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
sda   8:00 931,5G  0 disk
├─sda18:10   476M  0 part
├─sda28:20  46,6G  0 part
│ └─md0   9:00  46,5G  0 raid1 /
├─sda38:30   100M  0 part /var/lib/ceph/osd/ceph-0
└─sda48:40 884,4G  0 part
sdb   8:16   0 931,5G  0 disk
├─sdb18:17   0   476M  0 part  /boot/efi
├─sdb28:18   0  46,6G  0 part
│ └─md0   9:00  46,5G  0 raid1 /
├─sdb38:19   0   100M  0 part /var/lib/ceph/osd/ceph-5
└─sdb48:20   0 884,4G  0 part
sdc   8:32   0 232,9G  0 disk
├─sdc18:33   020G  0 part
├─sdc28:34   0   576M  0 part
├─sdc38:35   020G  0 part
└─sdc48:36   0   576M  0 part
udev3,9G 0  3,9G   0% /dev
tmpfs   790M   59M  732M   8% /run
/dev/md0 46G  2,5G   41G   6% /
tmpfs   3,9G 0  3,9G   0% /dev/shm
tmpfs   5,0M 0  5,0M   0% /run/lock
tmpfs   3,9G 0  3,9G   0% /sys/fs/cgroup
/dev/sdb1   476M  3,4M  472M   1% /boot/efi
/dev/sda394M  5,4M   89M   6% /var/lib/ceph/osd/ceph-0
/dev/sdb394M  5,4M   89M   6% /var/lib/ceph/osd/ceph-5
tmpfs   790M 0  790M   0% /run/user/1001
/dev/sda :
 /dev/sda1 other, vfat
 /dev/sda2 other, linux_raid_member
 /dev/sda3 ceph data, active, cluster ceph, osd.0, block /dev/sda4, 
block.db /dev/sdc1, block.wal /dev/sdc2

 /dev/sda4 ceph block, for /dev/sda3
/dev/sdb :
 /dev/sdb1 other, vfat, mounted on /boot/efi
 /dev/sdb2 other, linux_raid_member
 /dev/sdb3 ceph data, active, cluster ceph, osd.5, block /dev/sdb4, 
block.db /dev/sdc3, block.wal /dev/sdc4

 /dev/sdb4 ceph block, for /dev/sdb3
/dev/sdc :
 /dev/sdc1 ceph block.db, for /dev/sda3
 /dev/sdc2 ceph block.wal, for /dev/sda3
 /dev/sdc3 ceph block.db, for /dev/sdb3
 /dev/sdc4 ceph block.wal, for /dev/sdb3
ID WEIGHT  REWEIGHT SIZE  USEAVAIL %USE VAR  PGS TYPE NAME
-1 4.38956- 4494G   111M 4494G 0.00 1.00   0 root default
-2 0.85678-  877G 38008k  877G 0.00 1.71   0 host cl1
 3 0.42839  1.0  438G 19004k  438G 0.00 1.71 117 osd.3
 4 0.42839  1.0  438G 19004k  438G 0.00 1.71 139 osd.4
-3 1.76639- 1808G 38008k 1808G 0.00 0.83   0 host cl2
 0 0.88319  1.0  904G 19004k  904G 0.00 0.83 119 osd.0
 5 0.88319  1.0  904G 19004k  904G 0.00 0.83 137 osd.5
-4 1.76639- 1808G 38008k 1808G 0.00 0.83   0 host cl3
 1 0.88319  1.0  904G 19004k  904G 0.00 0.83 133 osd.1
 2 0.88319  1.0  904G 19004k  904G 0.00 0.83 123 osd.2
  TOTAL 4494G   111M 4494G 0.00
MIN/MAX VAR: 0.83/1.71  STDDEV: 0.00

David, thanks for your attention!


2017-06-27 06:06 keltezéssel, Papp Rudolf Péter írta:


sudo df -h:
udev3,9G 0  3,9G   0% /dev
tmpfs   790M   19M  771M   3% /run
/dev/md0 46G  2,5G   41G   6% /
tmpfs   3,9G 0  3,9G   0% /dev/shm
tmpfs   5,0M 0  5,0M   0% /run/lock
tmpfs   3,9G 0  3,9G   0% /sys/fs/cgroup
/dev/sdb1   476M  3,4M  472M   1% /boot/efi
/dev/sda3   885G  1,4G  883G   1% /var/lib/ceph/osd/ceph-3
/dev/sdb3   885G  1,6G  883G   1% /var/lib/ceph/osd/ceph-0
tmpfs   790M 0  790M   0% /run/user/1001

ceph df:
GLOBAL:
SIZE AVAIL RAW USED %RAW USED
179G  179G 116M  0.06
POOLS:
NAME ID USED %USED MAX AVAIL OBJECTS
dev  6 0 061401M   0


2017-06-26 22:55 keltezéssel, David Turner írta:
And the `sudo df -h`?  Also a `ceph df` might be helpful to see 
what's going on.


On Mon, Jun 26, 2017 at 4:41 PM Papp Rudolf Péter > wrote:


Hi David!

lsblk:

NAMEMAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
sda   8:00 931,5G  0 disk
├─sda18:10   476M  0 part
├─sda28:20  46,6G  0 part
│ └─md0   9:00  46,5G  0 raid1 /
└─sda38:30 884,5G  0 part /var/lib/ceph/osd/ceph-3
sdb   8:16   0 931,5G  0 disk
├─sdb18:17   0   476M  0 part  /boot/efi
├─sdb28:18   0  46,6G  0 part
│ └─md0   9:00  46,5G  0 raid1 /
└─sdb38:19   0 884,5G  0 part /var/lib/ceph/osd/ceph-0
sdc   8:32   0 232,9G  0 disk
├─sdc18:33   020G  0 part
├─sdc28:34   0   576M  0 part
├─sdc38:35   020G  0 part
└─sdc48:36   0   576M  0 part


2017-06-26 22:37 keltezéssel, David Turner írta:

What is the output of `lsblk`?

On Mon, Jun 26, 2017 at 4:32 PM Papp Rudolf Péter mailto:p...@peer.hu>> wrote:

Dear cephers,

Could someone show me an url where can I found how ceph
calculate the
available space?

I've installed a small ceph (Kraken) environment with
bluestore OSDs.
The servers contains 2 disks

[ceph-users] slow cluster perfomance during snapshot restore

[Stanislav Kopp]

Hi,

we're testing ceph cluster as storage backend for our virtualization
(proxmox), we're using RBD for raw VM images. If I'm trying to restore
some snapshot with  "rbd snap rollback", the whole cluster becomes
really slow, the "apply_latency" goes to 4000-6000ms from normally
0-10ms, I see load on OSDs with many read/writes and many blocked
processes in "vmstat", after restore is  finished everything is fine
again.
My question is, is it possibly to set some "priority" for snapshot
restore, like "nice"  so it doesn't stresses OSDs so much?

BTW, I'm using ceph 11.2 on Ubuntu 16.04, 4 nodes, with 16 OSDs (8TB
each) + Intel 3710 SSD per 4 OSDs for journals.

Best,
Stan
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Zabbix plugin for ceph-mgr

[Wido den Hollander]

Just opened a PR: https://github.com/ceph/ceph/pull/16019

Reviews and comments are welcome!

Wido

> Op 27 juni 2017 om 16:57 schreef Wido den Hollander :
> 
> 
> 
> > Op 27 juni 2017 om 16:13 schreef David Turner :
> > 
> > 
> > Back to just Zabbix, this thread could go on forever if it devolves into
> > monitoring system comparisons.
> > 
> > As a home project I was curious what I could do with Zabbix for monitoring
> > Ceph.  I added a lot of Items for all of the PG states, osd counts (up, in,
> > etc), rados space (used, percent, total), IO stats (recovery read/write,
> > client read/write, IOPS), etc.  These Items aren't really used for alerts,
> > I have more specific alerting like if I have too many failure domains with
> > OSDs down and the ceph health message.  What all of these triggers are used
> > for is the graphs in Zabbix.  For my home setup, I don't need/want a full
> > carbon/graphite system running... yet.  Zabbix graphs based on trigger-less
> > items is perfect for that.
> > 
> > All of that said, I think that parsing ceph cli commands to get this data
> > is clunky and a mgr daemon would make a lot of sense to have an option to
> > push to zabbix.  Disable it by default, but put it in the ceph.conf as an
> > option for mgr_monitoring_zabbix = true.  That sort of syntax would work
> > well with other monitoring solutions as well that can accept push stats.
> > The problem is if this makes it so that Ceph needs to keep feature parity
> > with Zabbix releases if they change how they want the data to come in with
> > different releases.  mgr_monitoring_zabbix_v2, mgr_monitoring_zabbix_v3,
> > etc would be a nuisance for the Ceph team to deal with.
> 
> I would be something like that
> 
> [mgr]
> mgr modules = zabbix
> 
> Or if you want multiple modules
> 
> [mgr]
> mgr modules = dashboard zabbix
> 
> You then configure the config-keys and then data will be send to Zabbix.
> 
> I wrote the first pieces of code and it can be found here: 
> https://github.com/wido/ceph/commits/mgr-plugin-zabbix
> 
> Doesn't work yet, I'll have to do this between the other work I'm going.
> 
> Wido
> 
> > 
> > I found this github for my project to get all of the items into Zabbix
> > quickly.  There's another one based on python.  It got me 90% of the way to
> > where I wanted Zabbix to be for monitoring Ceph.
> > https://github.com/thelan/ceph-zabbix
> > 
> > On Tue, Jun 27, 2017 at 5:59 AM Wido den Hollander  wrote:
> > 
> > >
> > > > Op 27 juni 2017 om 11:24 schreef Marc Roos :
> > > >
> > > >
> > > >
> > > > FYI, 5 or even more years ago I was trying zabbix and when I noticed
> > > > that when the monitored hosts increased, the load on the mysql server
> > > > was increasing. Without being able to recall exactly what was wrong (I
> > > > think every sample they did, was one insert statement), I do remember
> > > > that I got quite an 'amateur' feeling of these guys. And when they apply
> > > > 'strange logics' in one situation, they are likely to apply this more
> > > > often elsewhere in their code. Then I moved to nagios.
> > > >
> > >
> > > I see Zabbix envs running with thousands of hosts and 10ks of items in
> > > there without any issues.
> > >
> > > It's ofcourse a personal preference. Working at a location now who are
> > > eager to go to Luminous and would like to see such a module for ceph-mgr.
> > >
> > > Wido
> > >
> > > >
> > > >
> > > > -Original Message-
> > > > From: Wido den Hollander [mailto:w...@42on.com]
> > > > Sent: dinsdag 27 juni 2017 11:09
> > > > To: ceph-us...@ceph.com
> > > > Subject: [ceph-users] Zabbix plugin for ceph-mgr
> > > >
> > > > Hi,
> > > >
> > > > After looking at the documentation [0] on how to write a plugin for
> > > > ceph-mgr I've been playing with the idea to create a Zabbix [1] plugin
> > > > for ceph-mgr.
> > > >
> > > > Before I start writing one I'd like to check if I'm thinking in the
> > > > right direction.
> > > >
> > > > Zabbix supports Items [2] and Triggers. Triggers are based on Items's
> > > > values. A Item could be from the type 'Trapper' where a application can
> > > > simply send key=value pairs, for example:
> > > >
> > > > my.host.name ceph.health HEALTH_OK
> > > > my.host.name ceph.osd.up 499
> > > > my.host.name ceph.osd.in 498
> > > >
> > > > A simple ceph-mgr module could do:
> > > >
> > > > def serve(self):
> > > >   while True:
> > > > send_data_to_zabbix()
> > > > time.sleep(60)
> > > >
> > > > If for example the key ceph.health is != OK for >1h Zabbix could fire a
> > > > trigger and send our an alert to an admin.
> > > >
> > > > Now, would that be a sane plugin for ceph-mgr or is this something you
> > > > shouldn't put in the mgr? To me it seems like a good place since it
> > > > already has all the data present. This way data is pushed to Zabbix
> > > > instead of the need for polling the data and parsing JSON output of
> > > > 'ceph -s'
> > > >
> > > > Wido
> > > >
> > > > [0]: http://docs.ceph.com/docs/master/mgr

Re: [ceph-users] RGW lifecycle not expiring objects

[Daniel Gryniewicz]

On 06/28/2017 02:30 PM, Graham Allan wrote:

That seems to be it! I couldn't see a way to specify the auth version
with aws cli (is there a way?). However it did work with s3cmd and v2 auth:

% s3cmd --signature-v2 setlifecycle lifecycle.xml s3://testgta
s3://testgta/: Lifecycle Policy updated


Good stuff.



(I believe that with Kraken, this threw an error and failed to set the
policy, but I'm not certain at this point... besides which radosgw
didn't then have access to the default.rgw.lc pool, which may have
caused further issues)

No way to read the lifecycle policy back with s3cmd, so:


I submitted a patch a while ago to add getlifecycle to s3cmd, and it was 
accepted, but I don't know about releases or distro packaging.  It will 
be there eventually.




% aws --endpoint-url https://xxx.xxx.xxx.xxx s3api \
get-bucket-lifecycle-configuration --bucket=testgta
{
"Rules": [
{
"Status": "Enabled",
"Prefix": "",
"Expiration": {
"Days": 1
},
"ID": "test"
}
]
}

and looks encouraging at the server side:

#  radosgw-admin lc list
[
{
"bucket": ":gta:default.6985397.1",
"status": "UNINITIAL"
},
{
"bucket": ":testgta:default.6790451.1",
"status": "UNINITIAL"
}
]

then:
#  radosgw-admin lc process

and all the (very old) objects disappeared from the test bucket.


Good to know.

Daniel
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Kernel mounted RBD's hanging

[Ilya Dryomov]

On Thu, Jun 29, 2017 at 4:30 PM, Nick Fisk  wrote:
> Hi All,
>
> Putting out a call for help to see if anyone can shed some light on this.
>
> Configuration:
> Ceph cluster presenting RBD's->XFS->NFS->ESXi
> Running 10.2.7 on the OSD's and 4.11 kernel on the NFS gateways in a
> pacemaker cluster
> Both OSD's and clients are go into a pair of switches, single L2 domain (no
> sign from pacemaker that there is network connectivity issues)
>
> Symptoms:
> - All RBD's on a single client randomly hang for 30s to several minutes,
> confirmed by pacemaker and ESXi hosts complaining

Hi Nick,

What is a "single client" here?

> - Cluster load is minimal when this happens most times

Can you post gateway syslog and point at when this happened?
Corresponding pacemaker excerpts won't hurt either.

> - All other clients with RBD's are not affected (Same RADOS pool), so its
> seems more of a client issue than cluster issue
> - It looks like pacemaker tries to also stop RBD+FS resource, but this also
> hangs
> - Eventually pacemaker succeeds in stopping resources and immediately
> restarts them, IO returns to normal
> - No errors, slow requests, or any other non normal Ceph status is reported
> on the cluster or ceph.log
> - Client logs show nothing apart from pacemaker
>
> Things I've tried:
> - Different kernels (potentially happened less with older kernels, but can't
> be 100% sure)

But still happened?  Do you have a list of all the kernels you've tried?

> - Disabling scrubbing and anything else that could be causing high load
> - Enabling Kernel RBD debugging (Problem maybe happens a couple of times a
> day, debug logging was not practical as I can't reproduce on demand)

When did it start occuring?  Can you think of any configuration changes
that might have been the trigger or is this a new setup?

Thanks,

Ilya
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] slow cluster perfomance during snapshot restore

[Ashley Merrick]

Many others I’m sure will comment on the snapshot specifics.

However running a cluster with some 8TB drives I have noticed huge differences 
between 4TB and 8TB drives and their peak latency’s when busy. So along with 
the known snapshot performance you may find the higher seek time and higher 
TB/disk ratio not helping much.

,Ashley

Sent from my iPhone
29 Jun 2017, at 10:43 PM, Stanislav Kopp 
mailto:stask...@gmail.com>> wrote:

Hi,

we're testing ceph cluster as storage backend for our virtualization
(proxmox), we're using RBD for raw VM images. If I'm trying to restore
some snapshot with  "rbd snap rollback", the whole cluster becomes
really slow, the "apply_latency" goes to 4000-6000ms from normally
0-10ms, I see load on OSDs with many read/writes and many blocked
processes in "vmstat", after restore is  finished everything is fine
again.
My question is, is it possibly to set some "priority" for snapshot
restore, like "nice"  so it doesn't stresses OSDs so much?

BTW, I'm using ceph 11.2 on Ubuntu 16.04, 4 nodes, with 16 OSDs (8TB
each) + Intel 3710 SSD per 4 OSDs for journals.

Best,
Stan
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Kernel mounted RBD's hanging

[Nick Fisk]

> -Original Message-
> From: Ilya Dryomov [mailto:idryo...@gmail.com]
> Sent: 29 June 2017 16:58
> To: Nick Fisk 
> Cc: Ceph Users 
> Subject: Re: [ceph-users] Kernel mounted RBD's hanging
> 
> On Thu, Jun 29, 2017 at 4:30 PM, Nick Fisk  wrote:
> > Hi All,
> >
> > Putting out a call for help to see if anyone can shed some light on this.
> >
> > Configuration:
> > Ceph cluster presenting RBD's->XFS->NFS->ESXi Running 10.2.7 on the
> > OSD's and 4.11 kernel on the NFS gateways in a pacemaker cluster Both
> > OSD's and clients are go into a pair of switches, single L2 domain (no
> > sign from pacemaker that there is network connectivity issues)
> >
> > Symptoms:
> > - All RBD's on a single client randomly hang for 30s to several
> > minutes, confirmed by pacemaker and ESXi hosts complaining
> 
> Hi Nick,
> 
> What is a "single client" here?

I mean a node of the pacemaker cluster. So all RBD's on the same pacemaker node 
hang.

> 
> > - Cluster load is minimal when this happens most times
> 
> Can you post gateway syslog and point at when this happened?
> Corresponding pacemaker excerpts won't hurt either.

Jun 28 16:35:38 MS-CEPH-Proxy1 lrmd[2026]:  warning: 
p_export_ceph-ds1_monitor_6 process (PID 17754) timed out
Jun 28 16:35:43 MS-CEPH-Proxy1 lrmd[2026]: crit: 
p_export_ceph-ds1_monitor_6 process (PID 17754) will not die!
Jun 28 16:43:51 MS-CEPH-Proxy1 lrmd[2026]:  warning: 
p_export_ceph-ds1_monitor_6:17754 - timed out after 3ms
Jun 28 16:43:52 MS-CEPH-Proxy1 IPaddr(p_vip_ceph-ds1)[28482]: INFO: ifconfig 
ens224:0 down
Jun 28 16:43:52 MS-CEPH-Proxy1 lrmd[2026]:   notice: 
p_vip_ceph-ds1_stop_0:28482:stderr [ SIOCDELRT: No such process ]
Jun 28 16:43:52 MS-CEPH-Proxy1 crmd[2029]:   notice: Operation 
p_vip_ceph-ds1_stop_0: ok (node=MS-CEPH-Proxy1, call=471, rc=0, cib-update=318, 
confirmed=true)
Jun 28 16:43:52 MS-CEPH-Proxy1 exportfs(p_export_ceph-ds1)[28499]: INFO: 
Un-exporting file system ...
Jun 28 16:43:52 MS-CEPH-Proxy1 exportfs(p_export_ceph-ds1)[28499]: INFO: 
unexporting 10.3.20.0/24:/mnt/Ceph-DS1
Jun 28 16:43:52 MS-CEPH-Proxy1 exportfs(p_export_ceph-ds1)[28499]: INFO: 
Unlocked NFS export /mnt/Ceph-DS1
Jun 28 16:43:52 MS-CEPH-Proxy1 exportfs(p_export_ceph-ds1)[28499]: INFO: 
Un-exported file system(s)
Jun 28 16:43:52 MS-CEPH-Proxy1 crmd[2029]:   notice: Operation 
p_export_ceph-ds1_stop_0: ok (node=MS-CEPH-Proxy1, call=473, rc=0, 
cib-update=319, confirmed=true)
Jun 28 16:43:52 MS-CEPH-Proxy1 exportfs(p_export_ceph-ds1)[28549]: INFO: 
Exporting file system(s) ...
Jun 28 16:43:52 MS-CEPH-Proxy1 exportfs(p_export_ceph-ds1)[28549]: INFO: 
exporting 10.3.20.0/24:/mnt/Ceph-DS1
Jun 28 16:43:52 MS-CEPH-Proxy1 exportfs(p_export_ceph-ds1)[28549]: INFO: 
directory /mnt/Ceph-DS1 exported
Jun 28 16:43:52 MS-CEPH-Proxy1 crmd[2029]:   notice: Operation 
p_export_ceph-ds1_start_0: ok (node=MS-CEPH-Proxy1, call=474, rc=0, 
cib-update=320, confirmed=true)

If I enable the read/write checks for the FS resource, they also timeout at the 
same time.

> 
> > - All other clients with RBD's are not affected (Same RADOS pool), so
> > its seems more of a client issue than cluster issue
> > - It looks like pacemaker tries to also stop RBD+FS resource, but this
> > also hangs
> > - Eventually pacemaker succeeds in stopping resources and immediately
> > restarts them, IO returns to normal
> > - No errors, slow requests, or any other non normal Ceph status is
> > reported on the cluster or ceph.log
> > - Client logs show nothing apart from pacemaker
> >
> > Things I've tried:
> > - Different kernels (potentially happened less with older kernels, but
> > can't be 100% sure)
> 
> But still happened?  Do you have a list of all the kernels you've tried?

4.5 and 4.11. 

> 
> > - Disabling scrubbing and anything else that could be causing high
> > load
> > - Enabling Kernel RBD debugging (Problem maybe happens a couple of
> > times a day, debug logging was not practical as I can't reproduce on
> > demand)
> 
> When did it start occuring?  Can you think of any configuration changes that
> might have been the trigger or is this a new setup?

It has always done this from what I can tell. The majority of the time IO 
resumed before ESXi went All Paths Down, so it wasn't on my list of priorities 
to fix. But recently the hangs are lasting a lot longer. I need to go back to 
the 4.5 kernel, as I don't remember it happening as often or being as 
disruptive since upgrading to 4.11.

> 
> Thanks,
> 
> Ilya

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ideas on the UI/UX improvement of ceph-mgr: Cluster Status Dashboard

[Reed Dier]

I’d like to see per pool iops/usage, et al.

Being able to see rados vs rbd vs whatever else performance, or pools with 
different backing mediums and see which workloads result in what performance.

Most of this I pretty well cobble together with collectd, but it would still be 
nice to have out of the box going forward.

Reed

> On Jun 25, 2017, at 11:49 PM, saumay agrawal  wrote:
> 
> Hi everyone!
> 
> I am working on the improvement of the web-based dashboard for Ceph.
> My intention is to add some UI elements to visualise some performance
> counters of a Ceph cluster. This gives a better overview to the users
> of the dashboard about how the Ceph cluster is performing and, if
> necessary, where they can make necessary optimisations to get even
> better performance from the cluster.
> 
> Here is my suggestion on the two perf counters, commit latency and
> apply latency. They are visualised using line graphs. I have prepared
> UI mockups for the same.
> 1. OSD apply latency
> [https://drive.google.com/open?id=0ByXy5gIBzlhYNS1MbTJJRDhtSG8]
> 2. OSD commit latency
> [https://drive.google.com/open?id=0ByXy5gIBzlhYNElyVU00TGtHeVU]
> 
> These mockups show the latency values (y-axis) against the instant of
> time (x-axis). The latency values for different OSDs are highlighted
> using different colours. The average latency value of all OSDs is
> shown specifically in red. This representation allows the dashboard
> user to compare the performances of an OSD with other OSDs, as well as
> with the average performance of the cluster.
> 
> The line width in these graphs is specially kept less, so as to give a
> crisp and clear representation for more number of OSDs. However, this
> approach may clutter the graph and make it incomprehensible for a
> cluster having significantly higher number of OSDs. For such
> situations, we can retain only the average latency indications from
> both the graphs to make things more simple for the dashboard user.
> 
> Also, higher latency values suggest bad performance. We can come up
> with some specific values for both the counters, above which we can
> say that the cluster is performing very bad. If the value of any of
> the OSDs exceeds this value, we can highlight entire graph in a light
> red shade to draw the attention of user towards it.
> 
> I am planning to use AJAX based templates and plugins (like
> Flotcharts) for these graphs. This would allow real-time update of the
> graphs without having any need to reload the entire dashboard page.
> 
> Another feature I propose to add is the representation of the version
> distribution of all the clients in a cluster. This can be categorised
> into distribution
> 1. on the basis of ceph version
> [https://drive.google.com/open?id=0ByXy5gIBzlhYYmw5cXF2bkdTWWM] and,
> 2. on the basis of kernel version
> [https://drive.google.com/open?id=0ByXy5gIBzlhYczFuRTBTRDcwcnc]
> 
> I have used doughnut charts instead of regular pie charts, as they
> have some whitespace at their centre. This whitespace makes the chart
> appear less cluttered, while properly indicating the appropriate
> fraction of the total value. Also, we can later add some data to
> display at this centre space when we hover over a particular slice of
> the chart.
> 
> The main purpose of this visualisation is to identify any number of
> clients left behind while updating the clients of the cluster. Suppose
> a cluster has 50 clients running ceph jewel. In the process of
> updating this cluster, 40 clients get updated to ceph luminous, while
> the other 10 clients remain behind on ceph jewel. This may occur due
> to some bug or any interruption in the update process. In such
> scenarios, the user can find which clients have not been updated and
> update them according to his needs.  It may also give a clear picture
> for troubleshooting, during any package dependency issues due to the
> kernel. The clients are represented in both, absolutes numbers as well
> as the percentage of the entire cluster, for a better overview.
> 
> An interesting approach could be highlighting the older version(s)
> specifically to grab the attention of the user. For example, a user
> running ceph jewel may not need to update as necessarily compared to
> the user running ceph hammer.
> 
> As of now, I am looking for plugins in AdminLTE to implement these two
> elements in the dashboard. I would like to have feedbacks and
> suggestions on these two from the ceph community, on how can I make
> them more informative about the cluster.
> 
> Also a request to the various ceph users and developers. It would be
> great if you could share the various metrics you are using as a
> performance indicator for your cluster, and how you are using them.
> Any metrics being used to identify the issues in a cluster can also be
> shared.
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Question about rbd-mirror

[Jason Dillaman]

On Wed, Jun 28, 2017 at 11:42 PM, YuShengzuo  wrote:
> Hi Jason Dillaman,
>
>
>
> I am using rbd-mirror now (release Jewel).
>
>
>
> 1.
>
> And in many webs or other information introduced rbd-mirror notices that two
> ceph cluster should be the ‘same fsid’.
>
> But It’s nothing bad or wrong when I deploy rbd-mirror between any two cephs
>
> and choose one to be my Openstack cinder backend and the other one to be
> replication ceph.
>
> In some it’s OK in simple test cases, such as doing failover.
>
> So what it is the reason to recommend using the ‘same fsid’

There is no requirement to have the same "fsid" between clusters. Did
you see that in any of our documentation? The only similar requirement
is that the pools be named the same between the two clusters.

>
> 2.
>
> Last week, the Luminous released where I read in ceph.com.
>
>
>
> I notice it is supported something HA wrote in that blog.
>
> Is there any more information about it in detail?  It is multi-rbd-mirror?
>

Yes, for Luminous, you can now run multiple rbd-mirror daemons
concurrently for built-in HA. The daemons will automatically elect one
alive process as the leader and will recover from the failure of the
leader automatically. Previously, bad things would happen if you
attempted to run multiple rbd-mirror daemon processes concurrently.


>
> Hope to get your return, thanks.
>
>
>
>
>
> --
>
>
> Yu Shengzuo
>
> e-mail: yu.sheng...@99cloud.net
>
>


-- 
Jason
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] slow cluster perfomance during snapshot restore

[Gregory Farnum]

On Thu, Jun 29, 2017 at 7:44 AM Stanislav Kopp  wrote:

> Hi,
>
> we're testing ceph cluster as storage backend for our virtualization
> (proxmox), we're using RBD for raw VM images. If I'm trying to restore
> some snapshot with  "rbd snap rollback", the whole cluster becomes
> really slow, the "apply_latency" goes to 4000-6000ms from normally
> 0-10ms, I see load on OSDs with many read/writes and many blocked
> processes in "vmstat", after restore is  finished everything is fine
> again.
> My question is, is it possibly to set some "priority" for snapshot
> restore, like "nice"  so it doesn't stresses OSDs so much?
>

RBD snap rollback is a very expensive operation; it involves going to all
the objects in the RBD volume and doing a full copy from the snapshot into
the active "head" location.

I'm not sure if there are built-in tunable commands available (check the
manpages? Or Jason, do you know?), but if not you can use any generic
tooling which limits how much network traffic the RBD command can run.
-Greg


>
> BTW, I'm using ceph 11.2 on Ubuntu 16.04, 4 nodes, with 16 OSDs (8TB
> each) + Intel 3710 SSD per 4 OSDs for journals.
>
> Best,
> Stan
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Hammer patching on Wheezy?

[Scott Gilbert]

I have experienced a similar problem.  

I found that the ceph repos had previously resided at "ceph.com", but have 
since been moved to "download.ceph.com".  

In my case, simply updating the URL in the repo file resolved the issue. (ie: 
changing "https://ceph.com/..."; to "https://download.ceph.com/...";)



Date: Wed, 28 Jun 2017 09:02:22 +0200
From: Steffen Winther S?rensen 
To: ceph-users 
Subject: [ceph-users] Hammer patching on Wheezy?
Message-ID: 
Content-Type: text/plain; charset="us-ascii"

Ceph users,

Got a Hammer cluster installed on old debian wheezy (7.11) boxes (I know :)

root@node4:~# dpkg -l  | grep -i ceph
ii  ceph 0.94.9-1~bpo70+1 amd64 
   distributed storage and file system
ii  ceph-common  0.94.9-1~bpo70+1 amd64 
   common utilities to mount and interact with a ceph storage cluster
ii  ceph-deploy  1.5.35   all   
   Ceph-deploy is an easy to use configuration tool
ii  ceph-fs-common   0.94.9-1~bpo70+1 amd64 
   common utilities to mount and interact with a ceph file system
ii  ceph-fuse0.94.9-1~bpo70+1 amd64 
   FUSE-based client for the Ceph distributed file system
ii  ceph-mds 0.94.9-1~bpo70+1 amd64 
   metadata server for the ceph distributed file system
ii  libcephfs1   0.94.9-1~bpo70+1 amd64 
   Ceph distributed file system client library
ii  libcurl3-gnutls:amd647.29.0-1~bpo70+1.cephamd64 
   easy-to-use client-side URL transfer library (GnuTLS flavour)
ii  libleveldb1:amd641.12.0-1~bpo70+1.cephamd64 
   fast key-value storage library
ii  python-ceph  0.94.9-1~bpo70+1 amd64 
   Meta-package for python libraries for the Ceph libraries
ii  python-cephfs0.94.9-1~bpo70+1 amd64 
   Python libraries for the Ceph libcephfs library
ii  python-rados 0.94.9-1~bpo70+1 amd64 
   Python libraries for the Ceph librados library
ii  python-rbd   0.94.9-1~bpo70+1 amd64 
   Python libraries for the Ceph librbd library

Currently I fail to patch it to 0.94.10 with apt-get update, I get:

Get:13 http://ceph.com  wheezy Release
   
Err http://ceph.com  wheezy Release

W: A error occurred during the signature verification. The repository is not 
updated and the previous index files will be used. GPG error: http://ceph.com 
 wheezy Release: The following signatures were invalid: 
NODATA 1 NODATA 2

W: Failed to fetch http://ceph.com/debian-hammer/dists/wheezy/Release 



root@node4:~# apt-key list
/etc/apt/trusted.gpg

...
pub   4096R/17ED316D 2012-05-20
uid  Ceph Release Key mailto:s...@newdream.net>>

pub   4096R/F2AE6AB9 2013-04-29
uid  Joe Healy mailto:joehe...@gmail.com>>
sub   4096R/91EE136C 2013-04-29

pub   1024D/03C3951A 2011-02-08
uid  Ceph automated package build (Ceph automated package 
build) mailto:s...@newdream.net>>
sub   4096g/2E457B51 2011-02-08

pub   4096R/460F3994 2015-09-15
uid  Ceph.com  (release key) 
mailto:secur...@ceph.com>>
...

How to fix this?

TIA

/Steffen



-- 
Scott Gilbert
OpenStack Private Cloud Engineer
Rackspace - the #1 managed cloud company

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] slow cluster perfomance during snapshot restore

[Jason Dillaman]

On Thu, Jun 29, 2017 at 1:33 PM, Gregory Farnum  wrote:
> I'm not sure if there are built-in tunable commands available (check the
> manpages? Or Jason, do you know?), but if not you can use any generic
> tooling which limits how much network traffic the RBD command can run.

Long-running RBD actions are controlled by the "rbd concurrent
management ops" configuration value (default 10) which is essentially
how many in-flight IOs it will send to the OSDs in parallel. I am
surprised that a single rollback operation is causing a 100x increase
in latency. Anything special about your environment? cache tiering in
use?

-- 
Jason
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Kernel mounted RBD's hanging

[Ilya Dryomov]

On Thu, Jun 29, 2017 at 6:22 PM, Nick Fisk  wrote:
>> -Original Message-
>> From: Ilya Dryomov [mailto:idryo...@gmail.com]
>> Sent: 29 June 2017 16:58
>> To: Nick Fisk 
>> Cc: Ceph Users 
>> Subject: Re: [ceph-users] Kernel mounted RBD's hanging
>>
>> On Thu, Jun 29, 2017 at 4:30 PM, Nick Fisk  wrote:
>> > Hi All,
>> >
>> > Putting out a call for help to see if anyone can shed some light on this.
>> >
>> > Configuration:
>> > Ceph cluster presenting RBD's->XFS->NFS->ESXi Running 10.2.7 on the
>> > OSD's and 4.11 kernel on the NFS gateways in a pacemaker cluster Both
>> > OSD's and clients are go into a pair of switches, single L2 domain (no
>> > sign from pacemaker that there is network connectivity issues)
>> >
>> > Symptoms:
>> > - All RBD's on a single client randomly hang for 30s to several
>> > minutes, confirmed by pacemaker and ESXi hosts complaining
>>
>> Hi Nick,
>>
>> What is a "single client" here?
>
> I mean a node of the pacemaker cluster. So all RBD's on the same pacemaker 
> node hang.
>
>>
>> > - Cluster load is minimal when this happens most times
>>
>> Can you post gateway syslog and point at when this happened?
>> Corresponding pacemaker excerpts won't hurt either.
>
> Jun 28 16:35:38 MS-CEPH-Proxy1 lrmd[2026]:  warning: 
> p_export_ceph-ds1_monitor_6 process (PID 17754) timed out
> Jun 28 16:35:43 MS-CEPH-Proxy1 lrmd[2026]: crit: 
> p_export_ceph-ds1_monitor_6 process (PID 17754) will not die!
> Jun 28 16:43:51 MS-CEPH-Proxy1 lrmd[2026]:  warning: 
> p_export_ceph-ds1_monitor_6:17754 - timed out after 3ms
> Jun 28 16:43:52 MS-CEPH-Proxy1 IPaddr(p_vip_ceph-ds1)[28482]: INFO: ifconfig 
> ens224:0 down
> Jun 28 16:43:52 MS-CEPH-Proxy1 lrmd[2026]:   notice: 
> p_vip_ceph-ds1_stop_0:28482:stderr [ SIOCDELRT: No such process ]
> Jun 28 16:43:52 MS-CEPH-Proxy1 crmd[2029]:   notice: Operation 
> p_vip_ceph-ds1_stop_0: ok (node=MS-CEPH-Proxy1, call=471, rc=0, 
> cib-update=318, confirmed=true)
> Jun 28 16:43:52 MS-CEPH-Proxy1 exportfs(p_export_ceph-ds1)[28499]: INFO: 
> Un-exporting file system ...
> Jun 28 16:43:52 MS-CEPH-Proxy1 exportfs(p_export_ceph-ds1)[28499]: INFO: 
> unexporting 10.3.20.0/24:/mnt/Ceph-DS1
> Jun 28 16:43:52 MS-CEPH-Proxy1 exportfs(p_export_ceph-ds1)[28499]: INFO: 
> Unlocked NFS export /mnt/Ceph-DS1
> Jun 28 16:43:52 MS-CEPH-Proxy1 exportfs(p_export_ceph-ds1)[28499]: INFO: 
> Un-exported file system(s)
> Jun 28 16:43:52 MS-CEPH-Proxy1 crmd[2029]:   notice: Operation 
> p_export_ceph-ds1_stop_0: ok (node=MS-CEPH-Proxy1, call=473, rc=0, 
> cib-update=319, confirmed=true)
> Jun 28 16:43:52 MS-CEPH-Proxy1 exportfs(p_export_ceph-ds1)[28549]: INFO: 
> Exporting file system(s) ...
> Jun 28 16:43:52 MS-CEPH-Proxy1 exportfs(p_export_ceph-ds1)[28549]: INFO: 
> exporting 10.3.20.0/24:/mnt/Ceph-DS1
> Jun 28 16:43:52 MS-CEPH-Proxy1 exportfs(p_export_ceph-ds1)[28549]: INFO: 
> directory /mnt/Ceph-DS1 exported
> Jun 28 16:43:52 MS-CEPH-Proxy1 crmd[2029]:   notice: Operation 
> p_export_ceph-ds1_start_0: ok (node=MS-CEPH-Proxy1, call=474, rc=0, 
> cib-update=320, confirmed=true)
>
> If I enable the read/write checks for the FS resource, they also timeout at 
> the same time.

What about syslog that the above corresponds to?

Thanks,

Ilya
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Very HIGH Disk I/O latency on instances

[Gregory Farnum]

On Thu, Jun 29, 2017 at 12:16 AM Peter Maloney <
peter.malo...@brockmann-consult.de> wrote:

> On 06/28/17 21:57, Gregory Farnum wrote:
>
> On Wed, Jun 28, 2017 at 9:17 AM Peter Maloney <
> peter.malo...@brockmann-consult.de> wrote:
>
> On 06/28/17 16:52, keynes_...@wistron.com wrote:
>>
> [...]backup VMs is create a snapshot by Ceph commands (rbd snapshot) then
>> download (rbd export) it.
>>
>>
>>
>> We found a very high Disk Read / Write latency during creating / deleting
>> snapshots, it will higher than 1 ms.
>>
>>
>>
>> Even not during backup jobs, we often see a more than 4000 ms latency
>> occurred.
>>
>>
>>
>> Users start to complain.
>>
>> Could you please help us to how to start the troubleshooting?
>>
>>
>>
>> For creating snaps and keeping them, this was marked wontfix
>> http://tracker.ceph.com/issues/10823
>>
>> For deleting, see the recent "Snapshot removed, cluster thrashed" thread
>> for some config to try.
>>
>
> Given he says he's seeing 4 second IOs even without snapshot involvement,
> I think Keynes must be seeing something else in his cluster.
>
>
> If you have few enough OSDs and slow enough journals that seem ok without
> snaps, with snaps can be much worse than 4s IOs if you have any sync heavy
> clients, like ganglia.
>
> Before I figured out that it was exclusive-lock causing VMs to hang, I
> tested many things and spent months on it and found that out. Also some
> people in freenode irc ##proxmox channel with cheap home setups with ceph
> complain about such things often.
>
>
>
>
>>
>> https://storageswiss.com/2016/04/01/snapshot-101-copy-on-write-vs-redirect-on-write/
>>
>> Consider a *copy-on-write* system, which *copies* any blocks before they
>> are overwritten with new information (i.e. it copies on writes). In other
>> words, if a block in a protected entity is to be modified, the system will
>> copy that block to a separate snapshot area before it is overwritten with
>> the new information. This approach requires three I/O operations for each
>> write: one read and two writes. [...] This decision process for each block
>> also comes with some computational overhead.
>>
>>
>> A *redirect-on-write* system uses pointers to represent all protected
>> entities. If a block needs modification, the storage system merely
>> *redirects* the pointer for that block to another block and writes the
>> data there. [...] There is zero computational overhead of reading a
>> snapshot in a redirect-on-write system.
>>
>>
>> The redirect-on-write system uses 1/3 the number of I/O operations when
>> modifying a protected block, and it uses no extra computational overhead
>> reading a snapshot. Copy-on-write systems can therefore have a big impact
>> on the performance of the protected entity. The more snapshots are created
>> and the longer they are stored, the greater the impact to performance on
>> the protected entity.
>>
>>
> I wouldn't consider that a very realistic depiction of the tradeoffs
> involved in different snapshotting strategies[1], but BlueStore uses
> "redirect-on-write" under the formulation presented in those quotes. RBD
> clones of protected images will remain copy-on-write forever, I imagine.
> -Greg
>
> It was simply the first link I found which I could quote, but I didn't
> find it too bad... just it describes it like all implementations are the
> same.
>
>
> [1]: There's no reason to expect a copy-on-write system will first copy
> the original data and then overwrite it with the new data when it can
> simply inject the new data along the way. *Some* systems will copy the
> "old" block into a new location and then overwrite in the existing location
> (it helps prevent fragmentation), but many don't. And a "redirect-on-write"
> system needs to persist all those block metadata pointers, which may be
> much cheaper or much, much more expensive than just duplicating the blocks.
>
>
> After a snap is unprotected, will the clones be redirect-on-write? Or
> after the image is flattened (like dd if=/dev/zero to the whole disk)?
>
> Are there other cases where you get a copy-on-write behavior?
>
> Glad to hear bluestore has something better. Is that avaliable and default
> behavior on kraken (which I tested but where it didn't seem to be fixed,
> although all storage backends were less block prone on kraken)?
>
> If it was a true redirect-on-write system, I would expect that when you
> make a snap, there is just the overhead of organizing some metadata, and
> then after that, any writes just write as normal, to a new place, not
> requiring the old data to be copied, ideally not any of it, even partially
> written objects. And I don't think I saw that behavior on my kraken tests,
> although the performance was better (due to no blocked requests, but the
> iops at peak was basically the same; and I didn't measure total IO or
> something that would be more reliable...just looked at performance effects
> and blocking).
>
>
Bluestore was available for dev/testing in Kraken, but

[ceph-users] RadosGW Swift public links

[Donny Davis]

I have swift working well with keystone authentication, and I can upload
and download files. However when I make a link public, I get nosuchbucket,
and I have no idea what url to find the buckets at.

When I list the buckets with radosgw-admin bucket list i get back some
tenant url+ the bucket.

I am lost on where to start with this. I just want to share files with rgw
and openstack
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] ask about async recovery

[donglifec...@gmail.com]

zhiqiang, Josn

what about the async recovery feature? I didn't see any update on github 
recently, will it be further developed? 

Thanks a lot.



donglifec...@gmail.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Kernel mounted RBD's hanging

[Alex Gorbachev]

On Thu, Jun 29, 2017 at 10:30 AM Nick Fisk  wrote:

> Hi All,
>
> Putting out a call for help to see if anyone can shed some light on this.
>
> Configuration:
> Ceph cluster presenting RBD's->XFS->NFS->ESXi
> Running 10.2.7 on the OSD's and 4.11 kernel on the NFS gateways in a
> pacemaker cluster
> Both OSD's and clients are go into a pair of switches, single L2 domain (no
> sign from pacemaker that there is network connectivity issues)
>
> Symptoms:
> - All RBD's on a single client randomly hang for 30s to several minutes,
> confirmed by pacemaker and ESXi hosts complaining
> - Cluster load is minimal when this happens most times
> - All other clients with RBD's are not affected (Same RADOS pool), so its
> seems more of a client issue than cluster issue
> - It looks like pacemaker tries to also stop RBD+FS resource, but this also
> hangs
> - Eventually pacemaker succeeds in stopping resources and immediately
> restarts them, IO returns to normal
> - No errors, slow requests, or any other non normal Ceph status is reported
> on the cluster or ceph.log
> - Client logs show nothing apart from pacemaker
>
> Things I've tried:
> - Different kernels (potentially happened less with older kernels, but
> can't
> be 100% sure)
> - Disabling scrubbing and anything else that could be causing high load
> - Enabling Kernel RBD debugging (Problem maybe happens a couple of times a
> day, debug logging was not practical as I can't reproduce on demand)
>
> Anyone have any ideas?


Nick, are you using any network aggregation, LACP?  Can you drop to a
simplest possible configuration to make sure there's nothing on the network
switch side?

Do you check the ceph.log for any anomalies?

Any occurrences on OSD nodes, anything in their OSD logs or syslogs?

Aany odd page cache settings on the clients?

Alex


>
> Thanks,
> Nick
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
-- 
--
Alex Gorbachev
Storcium
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Multi Tenancy in Ceph RBD Cluster

[Alex Gorbachev]

On Mon, Jun 26, 2017 at 2:00 AM Mayank Kumar  wrote:

> Hi Ceph Users
> I am relatively new to Ceph and trying to Provision CEPH RBD Volumes using
> Kubernetes.
>
> I would like to know what are the best practices for hosting a multi
> tenant CEPH cluster. Specifically i have the following questions:-
>
> - Is it ok to share a single Ceph Pool amongst multiple tenants ?  If yes,
> how do you guarantee that volumes of one Tenant are not
>  accessible(mountable/mapable/unmappable/deleteable/mutable) to other
> tenants ?
> - Can a single Ceph Pool have multiple admin and user keyrings generated
> for rbd create and rbd map commands ? This way i want to assign different
> keyrings to each tenant
>
> - can a rbd map command be run remotely for any node on which we want to
> mount RBD Volumes or it must be run from the same node on which we want to
> mount ? Is this going to be possible in the future ?
>
> - In terms of ceph fault tolerance and resiliency, is one ceph pool per
> customer a better design or a single pool must be shared with mutiple
> customers
> - In a single pool for all customers, how can we get the ceph statistics
> per customer ? Is it possible to somehow derive this from the RBD volumes ?
>

Is this post helpful?
https://blog-fromsomedude.rhcloud.com/2016/04/26/Allowing-a-RBD-client-to-map-only-one-RBD



> Thanks for your responses
> Mayank
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
-- 
--
Alex Gorbachev
Storcium
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] ask about async recovery

[donglifec...@gmail.com]

zhiqiang, Josn

what about the async recovery feature? I didn't see any update on github 
recently, will it be further developed? 

Thanks a lot.




donglifec...@gmail.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] dropping filestore+btrfs testing for luminous

[Sage Weil]

We're having a series of problems with the valgrind included in xenial[1] 
that have led us to restrict all valgrind tests to centos nodes.  At teh 
same time, we're also seeing spurious ENOSPC errors from btrfs on both 
centos on xenial kernels[2], making trusty the only distro where btrfs 
works reliably.

Teuthology doesn't handle this well when it tries to put together the 
test matrix (we can't test filestore+btrfs+valgrind).

The easiest thing is to

1/ Stop testing filestore+btrfs for luminous onward.  We've recommended 
against btrfs for a long time and are moving toward bluestore anyway.

2/ Leave btrfs in the mix for jewel, and manually tolerate and filter out 
the occasional ENOSPC errors we see.  (They make the test runs noisy but 
are pretty easy to identify.)

If we don't stop testing filestore on btrfs now, I'm not sure when we 
would ever be able to stop, and that's pretty clearly not sustainable.
Does that seem reasonable?  (Pretty please?)

sage


[1] http://tracker.ceph.com/issues/18126 and 
http://tracker.ceph.com/issues/20360
[2] http://tracker.ceph.com/issues/20169
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] RadosGW Swift public links

[David Turner]

Are you accessing the bucket from a URL that is not configured as an
endpoint in your zone?  I bet if you looked at the log you would see that
the bucket that doesn't exist is the URL that you are using to access it.

On Thu, Jun 29, 2017, 9:07 PM Donny Davis  wrote:

> I have swift working well with keystone authentication, and I can upload
> and download files. However when I make a link public, I get nosuchbucket,
> and I have no idea what url to find the buckets at.
>
> When I list the buckets with radosgw-admin bucket list i get back some
> tenant url+ the bucket.
>
> I am lost on where to start with this. I just want to share files with rgw
> and openstack
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] RGW lifecycle not expiring objects

[Félix Barbeira]

I recently check the repo and the new version of s3cmd was released 3 days
ago, including lifecycle commands:

https://github.com/s3tools/s3cmd/releases

These are the lifecycle options:

https://github.com/s3tools/s3cmd/blob/master/s3cmd#L2444-L2448

2017-06-29 17:51 GMT+02:00 Daniel Gryniewicz :

> On 06/28/2017 02:30 PM, Graham Allan wrote:
>
>> That seems to be it! I couldn't see a way to specify the auth version
>> with aws cli (is there a way?). However it did work with s3cmd and v2
>> auth:
>>
>> % s3cmd --signature-v2 setlifecycle lifecycle.xml s3://testgta
>> s3://testgta/: Lifecycle Policy updated
>>
>
> Good stuff.
>
>
>> (I believe that with Kraken, this threw an error and failed to set the
>> policy, but I'm not certain at this point... besides which radosgw
>> didn't then have access to the default.rgw.lc pool, which may have
>> caused further issues)
>>
>> No way to read the lifecycle policy back with s3cmd, so:
>>
>
> I submitted a patch a while ago to add getlifecycle to s3cmd, and it was
> accepted, but I don't know about releases or distro packaging.  It will be
> there eventually.
>
>
>> % aws --endpoint-url https://xxx.xxx.xxx.xxx s3api \
>> get-bucket-lifecycle-configuration --bucket=testgta
>> {
>> "Rules": [
>> {
>> "Status": "Enabled",
>> "Prefix": "",
>> "Expiration": {
>> "Days": 1
>> },
>> "ID": "test"
>> }
>> ]
>> }
>>
>> and looks encouraging at the server side:
>>
>> #  radosgw-admin lc list
>> [
>> {
>> "bucket": ":gta:default.6985397.1",
>> "status": "UNINITIAL"
>> },
>> {
>> "bucket": ":testgta:default.6790451.1",
>> "status": "UNINITIAL"
>> }
>> ]
>>
>> then:
>> #  radosgw-admin lc process
>>
>> and all the (very old) objects disappeared from the test bucket.
>>
>
> Good to know.
>
> Daniel
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>



-- 
Félix Barbeira.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com