[ceph-users] upmap supported in SLES 12SPx

[Thomas]

Hi,
I tried to run this command with failure:
root@ld3955:/mnt/rbd# ceph osd set-require-min-compat-client luminous
Error EPERM: cannot set require_min_compat_client to luminous: 6
connected client(s) look like jewel (missing 0xa20); 19
connected client(s) look like jewel (missing 0x800); 23
connected client(s) look like jewel (missing 0x800); add
--yes-i-really-mean-it to do it anyway

root@ld3955:/mnt/rbd# ceph features
{
    "mon": [
    {
    "features": "0x3ffddff8ffac",
    "release": "luminous",
    "num": 3
    }
    ],
    "mds": [
    {
    "features": "0x3ffddff8ffac",
    "release": "luminous",
    "num": 2
    }
    ],
    "osd": [
    {
    "features": "0x3ffddff8ffac",
    "release": "luminous",
    "num": 368
    }
    ],
    "client": [
    {
    "features": "0x40106b84a842a42",
    "release": "jewel",
    "num": 6
    },
    {
    "features": "0x27018eb84aa42a52",
    "release": "jewel",
    "num": 19
    },
    {
    "features": "0x27018fb86aa42ada",
    "release": "jewel",
    "num": 23
    },
    {
    "features": "0x3ffddff8ffac",
    "release": "luminous",
    "num": 35
    }
    ],
    "mgr": [
    {
    "features": "0x3ffddff8ffac",
    "release": "luminous",
    "num": 3
    }
    ]
}

But I think the number of clients older than Luminous is incorrect.

I'm running 98% of the clients with SLES 12SPx, and therefore the
question is:
Can you confirm in which SLES 12 release the function upmap is supported?
And is it safe to execute ceph osd set-require-min-compat-client
luminous --yes-i-really-mean-it? What happens to the clients then?

Regards
Thomas


___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: upmap supported in SLES 12SPx

[Konstantin Shalygin]

On 9/16/19 3:59 PM, Thomas wrote:

I tried to run this command with failure:
root@ld3955:/mnt/rbd# ceph osd set-require-min-compat-client luminous
Error EPERM: cannot set require_min_compat_client to luminous: 6
connected client(s) look like jewel (missing 0xa20); 19
connected client(s) look like jewel (missing 0x800); 23
connected client(s) look like jewel (missing 0x800); add
--yes-i-really-mean-it to do it anyway

root@ld3955:/mnt/rbd# ceph features
{
     "mon": [
     {
     "features": "0x3ffddff8ffac",
     "release": "luminous",
     "num": 3
     }
     ],
     "mds": [
     {
     "features": "0x3ffddff8ffac",
     "release": "luminous",
     "num": 2
     }
     ],
     "osd": [
     {
     "features": "0x3ffddff8ffac",
     "release": "luminous",
     "num": 368
     }
     ],
     "client": [
     {
     "features": "0x40106b84a842a42",
     "release": "jewel",
     "num": 6
     },
     {
     "features": "0x27018eb84aa42a52",
     "release": "jewel",
     "num": 19
     },
     {
     "features": "0x27018fb86aa42ada",
     "release": "jewel",
     "num": 23
     },
     {
     "features": "0x3ffddff8ffac",
     "release": "luminous",
     "num": 35
     }
     ],
     "mgr": [
     {
     "features": "0x3ffddff8ffac",
     "release": "luminous",
     "num": 3
     }
     ]
}

But I think the number of clients older than Luminous is incorrect.

I'm running 98% of the clients with SLES 12SPx, and therefore the
question is:
Can you confirm in which SLES 12 release the function upmap is supported?
And is it safe to execute ceph osd set-require-min-compat-client
luminous --yes-i-really-mean-it? What happens to the clients then?


Your kernel should be 4.13+ to be upmap compat. Then you should tell 
`--yes-i-really-mean-it` to apply changes.




k
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: upmap supported in SLES 12SPx

[Oliver Freyermuth]

Am 16.09.19 um 11:06 schrieb Konstantin Shalygin:

On 9/16/19 3:59 PM, Thomas wrote:

I tried to run this command with failure:
root@ld3955:/mnt/rbd# ceph osd set-require-min-compat-client luminous
Error EPERM: cannot set require_min_compat_client to luminous: 6
connected client(s) look like jewel (missing 0xa20); 19
connected client(s) look like jewel (missing 0x800); 23
connected client(s) look like jewel (missing 0x800); add
--yes-i-really-mean-it to do it anyway

root@ld3955:/mnt/rbd# ceph features
{
 "mon": [
 {
 "features": "0x3ffddff8ffac",
 "release": "luminous",
 "num": 3
 }
 ],
 "mds": [
 {
 "features": "0x3ffddff8ffac",
 "release": "luminous",
 "num": 2
 }
 ],
 "osd": [
 {
 "features": "0x3ffddff8ffac",
 "release": "luminous",
 "num": 368
 }
 ],
 "client": [
 {
 "features": "0x40106b84a842a42",
 "release": "jewel",
 "num": 6
 },
 {
 "features": "0x27018eb84aa42a52",
 "release": "jewel",
 "num": 19
 },
 {
 "features": "0x27018fb86aa42ada",
 "release": "jewel",
 "num": 23
 },
 {
 "features": "0x3ffddff8ffac",
 "release": "luminous",
 "num": 35
 }
 ],
 "mgr": [
 {
 "features": "0x3ffddff8ffac",
 "release": "luminous",
 "num": 3
 }
 ]
}

But I think the number of clients older than Luminous is incorrect.

I'm running 98% of the clients with SLES 12SPx, and therefore the
question is:
Can you confirm in which SLES 12 release the function upmap is supported?
And is it safe to execute ceph osd set-require-min-compat-client
luminous --yes-i-really-mean-it? What happens to the clients then?


Your kernel should be 4.13+ to be upmap compat. Then you should tell 
`--yes-i-really-mean-it` to apply changes.


SLES 12 is still on 3.12 I think. However, the question is probably rather:
Does SUSE backport things such as pg-upmap to older Kernels, such as RedHat 
does?

Cheers,
Oliver





k
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io





smime.p7s
Description: S/MIME Cryptographic Signature
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Activate Cache Tier on Running Pools

[Eikermann, Robert]

Hi,

I'm using Ceph in combination with Openstack. For the "VMs" Pool I'd like to 
enable writeback caching tier, like described here: 
https://docs.ceph.com/docs/luminous/rados/operations/cache-tiering/ .

Should it be possible to do that on a running pool? I tried to do so and 
immediately all VMs (Linux Ubuntu OS) running on Ceph disks got readonly 
filesystems. No errors were shown in ceph (but also no traffic arrived after 
enabling the cache tier). Removing the cache tier , rebooting the VMs and doing 
a filesystemcheck repaired everything.

Best
Robert

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Activate Cache Tier on Running Pools

[Ashley Merrick]

Have you checked that the user/keys that your VMs are connecting to have access 
rights to the cache pool?





 On Mon, 16 Sep 2019 17:36:38 +0800 Eikermann, Robert 
 wrote 





Hi,

 

I’m using Ceph in combination with Openstack. For the “VMs” Pool I’d like to 
enable writeback caching tier, like described here: 
https://docs.ceph.com/docs/luminous/rados/operations/cache-tiering/ .

 

Should it be possible to do that on a running pool? I tried to do so and 
immediately all VMs (Linux Ubuntu OS) running on Ceph disks got readonly 
filesystems. No errors were shown in ceph (but also no traffic arrived
 after enabling the cache tier). Removing the cache tier , rebooting the VMs 
and doing a filesystemcheck repaired everything.

 

Best

Robert

 



___

ceph-users mailing list -- ceph-users@ceph.io 

To unsubscribe send an email to ceph-users-le...@ceph.io___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Ceph Day London - October 24 (Call for Papers!)

[Wido den Hollander]

Hi,

The CFP is ending today for the Ceph Day London on October 24th.

If you have a talk you would like to submit, please follow the link below!

Wido

On 7/18/19 3:43 PM, Wido den Hollander wrote:
> Hi,
>
> We will be having Ceph Day London October 24th!
>
> https://ceph.com/cephdays/ceph-day-london-2019/
>
> The CFP is now open for you to get your Ceph related content in front
> of the Ceph community ranging from all levels of expertise:
>
> https://forms.zohopublic.com/thingee/form/CephDayLondon2019/formperma/h96jZGAAm1dmEzuk2FxbKSpxSk4Y7u4zLtZJcq2Vijk
>
> If your company is interested in sponsoring the event, we would be
> delighted to have you. Please contact me directly for further information.
>
> The Ceph Day is co-located with the Apache CloudStack project. There
> will be two tracks where people can choose between Ceph and CloudStack.
>
> After the Ceph Day there's going to be beers in the pub nearby to make
> new friends.
>
> Join us in London on October 24th!
>
> Wido
> ___
> ceph-users mailing list
> ceph-us...@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Activate Cache Tier on Running Pools

[Wesley Peng]

Hello

on 2019/9/16 17:36, Eikermann, Robert wrote:
Should it be possible to do that on a running pool? I tried to do so and 
immediately all VMs (Linux Ubuntu OS) running on Ceph disks got readonly 
filesystems. No errors were shown in ceph (but also no traffic arrived 
after enabling the cache tier). Removing the cache tier , rebooting the 
VMs and doing a filesystemcheck repaired everything.


After enable cache tier, did you test to reboot a VM to make it become 
effective?


regards.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Activate Cache Tier on Running Pools

[Wido den Hollander]

On 9/16/19 11:36 AM, Eikermann, Robert wrote:
>
> Hi,
>
>  
>
> I’m using Ceph in combination with Openstack. For the “VMs” Pool I’d
> like to enable writeback caching tier, like described here:
> https://docs.ceph.com/docs/luminous/rados/operations/cache-tiering/.
>
>  
>
Can you explain why? The cache tiering has some serious flaws and can
even decrease performance instead of improve it.

What are you trying to solve?

Wido

> Should it be possible to do that on a running pool? I tried to do so
> and immediately all VMs (Linux Ubuntu OS) running on Ceph disks got
> readonly filesystems. No errors were shown in ceph (but also no
> traffic arrived after enabling the cache tier). Removing the cache
> tier , rebooting the VMs and doing a filesystemcheck repaired everything.
>
>  
>
> Best
>
> Robert
>
>  
>
>
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Activate Cache Tier on Running Pools

[Eikermann, Robert]

We have terrible IO performance when multiple VMs do some file IO. Mainly do 
some java compilation on that servers. If we have 2 parallel jobs everything is 
fine, but having 10 jobs we see the warning "HEALTH_WARN X requests are blocked 
> 32 sec; Y osds have slow requests". I have two enterprise SSDs which gained 
good results in ceph tested with fio. They are too small to have a separate 
"ssd_vms" pool and advertise it in openstack as a separate storage backend

--
-
Robert Eikermann M.Sc.RWTH   | Software Engineering
Lehrstuhl für Software Engineering   | RWTH Aachen University
Ahornstr. 55, 52074 Aachen, Germany  | 
http://www.se-rwth.de
Phone ++49 241 80-21306 / Fax -22218 |

Von: Wido den Hollander [mailto:w...@42on.com]
Gesendet: Montag, 16. September 2019 11:52
An: Eikermann, Robert ; ceph-users@ceph.io
Betreff: Re: [ceph-users] Activate Cache Tier on Running Pools



On 9/16/19 11:36 AM, Eikermann, Robert wrote:
Hi,

I'm using Ceph in combination with Openstack. For the "VMs" Pool I'd like to 
enable writeback caching tier, like described here: 
https://docs.ceph.com/docs/luminous/rados/operations/cache-tiering/ .


Can you explain why? The cache tiering has some serious flaws and can even 
decrease performance instead of improve it.

What are you trying to solve?

Wido
Should it be possible to do that on a running pool? I tried to do so and 
immediately all VMs (Linux Ubuntu OS) running on Ceph disks got readonly 
filesystems. No errors were shown in ceph (but also no traffic arrived after 
enabling the cache tier). Removing the cache tier , rebooting the VMs and doing 
a filesystemcheck repaired everything.

Best
Robert




___

ceph-users mailing list -- ceph-users@ceph.io

To unsubscribe send an email to 
ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Activate Cache Tier on Running Pools

[Ashley Merrick]

I hope the data your running the CEPH server isn't important if your looking to 
run a Cache tier with just 2 SSDS / Replication of 2.

If your cache tier fails, you basically corrupt most data on the pool below.

Also as Wido said, as much as you may get it to work, I don't think it will 
give you the performance you expected, can you not create a separate small SSD 
pool as a scratch disk for the VM's for when they are doing heavy temp I/O?

Leaving your current data and setup untouched.




 On Mon, 16 Sep 2019 18:21:47 +0800 Eikermann, Robert 
 wrote 




We have terrible IO performance when multiple VMs do some file IO. Mainly do 
some java compilation on that servers. If we have 2 parallel jobs everything is 
fine, but having 10 jobs we see the warning
 “HEALTH_WARN X requests are blocked > 32 sec; Y osds have slow requests”. I 
have two enterprise SSDs which gained good results in ceph tested with fio. 
They are too small to have a separate “ssd_vms” pool and advertise it in 
openstack as a separate storage
 backend

 

--
 -
Robert Eikermann M.Sc.RWTH   | Software Engineering

Lehrstuhl für Software Engineering   | RWTH Aachen University

Ahornstr. 55, 52074 Aachen, Germany      | http://www.se-rwth.de/

Phone ++49 241 80-21306 / Fax -22218 |


 

Von: Wido den Hollander [mailto:mailto:w...@42on.com] 
 Gesendet: Montag, 16. September 2019 11:52
 An: Eikermann, Robert ; mailto:ceph-users@ceph.io
 Betreff: Re: [ceph-users] Activate Cache Tier on Running Pools


 

 

On 9/16/19 11:36 AM, Eikermann, Robert wrote:


Hi,

 

I’m using Ceph in combination with Openstack. For the “VMs” Pool I’d like to 
enable writeback caching tier, like described here: 
https://docs.ceph.com/docs/luminous/rados/operations/cache-tiering/ .

 


Can you explain why? The cache tiering has some serious flaws and can even 
decrease performance instead of improve it.

What are you trying to solve?

Wido

Should it be possible to do that on a running pool? I tried to do so and 
immediately all VMs (Linux Ubuntu OS) running on Ceph disks got readonly 
filesystems. No errors were shown in ceph (but also no traffic arrived
 after enabling the cache tier). Removing the cache tier , rebooting the VMs 
and doing a filesystemcheck repaired everything.

 

Best

Robert

 




___

ceph-users mailing list -- mailto:ceph-users@ceph.io

To unsubscribe send an email to mailto:ceph-users-le...@ceph.io




___
ceph-users mailing list -- mailto:ceph-users@ceph.io 
To unsubscribe send an email to mailto:ceph-users-le...@ceph.io___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Activate Cache Tier on Running Pools

[Fyodor Ustinov]

Hi!

Cache tiering is a great solution if the cache size is larger than the hot 
data. Even better if the data can cool quietly in the cache. Otherwise, it’s 
really better not to do this.

- Original Message -
> From: "Wido den Hollander" 
> To: "Eikermann, Robert" , ceph-users@ceph.io
> Sent: Monday, 16 September, 2019 12:51:45
> Subject: [ceph-users] Re: Activate Cache Tier on Running Pools

> On 9/16/19 11:36 AM, Eikermann, Robert wrote:
> 
> 
> 
> 
> 
> Hi,
> 
> 
> 
> I’m using Ceph in combination with Openstack. For the “VMs” Pool I’d like to
> enable writeback caching tier, like described here: [
> https://docs.ceph.com/docs/luminous/rados/operations/cache-tiering/ |
> https://docs.ceph.com/docs/luminous/rados/operations/cache-tiering/ ] .
> 
> 
> 
> 
> Can you explain why? The cache tiering has some serious flaws and can even
> decrease performance instead of improve it.
> 
> What are you trying to solve?
> 
> Wido
> 
> 
> 
> 
> Should it be possible to do that on a running pool? I tried to do so and
> immediately all VMs (Linux Ubuntu OS) running on Ceph disks got readonly
> filesystems. No errors were shown in ceph (but also no traffic arrived after
> enabling the cache tier). Removing the cache tier , rebooting the VMs and 
> doing
> a filesystemcheck repaired everything.
> 
> 
> 
> Best
> 
> Robert
> 
> 
> 
> ___
> ceph-users mailing list -- [ mailto:ceph-users@ceph.io | ceph-users@ceph.io ] 
> To
> unsubscribe send an email to [ mailto:ceph-users-le...@ceph.io |
> ceph-users-le...@ceph.io ]
> 
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Different pools count in ceph -s and ceph osd pool ls

[Fyodor Ustinov]

Hi!

I create bug https://tracker.ceph.com/issues/41832

Maybe someone also encountered such a problem?

WBR,
Fyodor.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Activate Cache Tier on Running Pools

[Eikermann, Robert]

That would be the case for me. I think all data during one day would fit into 
the cache, and we could slowly flush back over night (or even over the 
weekend). But my impression is, I would have to test it. So my initial 
question: Do I have to stop all VMs before activating the cache? And restart 
one by one with the cache enabled?

--
-
Robert Eikermann M.Sc.RWTH   | Software Engineering
Lehrstuhl für Software Engineering   | RWTH Aachen University
Ahornstr. 55, 52074 Aachen, Germany  | http://www.se-rwth.de
Phone ++49 241 80-21306 / Fax -22218 |

-Ursprüngliche Nachricht-
Von: Fyodor Ustinov [mailto:u...@ufm.su] 
Gesendet: Montag, 16. September 2019 13:00
An: Wido den Hollander 
Cc: Eikermann, Robert ; ceph-users@ceph.io
Betreff: Re: [ceph-users] Re: Activate Cache Tier on Running Pools

Hi!

Cache tiering is a great solution if the cache size is larger than the hot 
data. Even better if the data can cool quietly in the cache. Otherwise, it’s 
really better not to do this.

- Original Message -
> From: "Wido den Hollander" 
> To: "Eikermann, Robert" , ceph-users@ceph.io
> Sent: Monday, 16 September, 2019 12:51:45
> Subject: [ceph-users] Re: Activate Cache Tier on Running Pools

> On 9/16/19 11:36 AM, Eikermann, Robert wrote:
> 
> 
> 
> 
> 
> Hi,
> 
> 
> 
> I’m using Ceph in combination with Openstack. For the “VMs” Pool I’d 
> like to enable writeback caching tier, like described here: [ 
> https://docs.ceph.com/docs/luminous/rados/operations/cache-tiering/ | 
> https://docs.ceph.com/docs/luminous/rados/operations/cache-tiering/ ] .
> 
> 
> 
> 
> Can you explain why? The cache tiering has some serious flaws and can 
> even decrease performance instead of improve it.
> 
> What are you trying to solve?
> 
> Wido
> 
> 
> 
> 
> Should it be possible to do that on a running pool? I tried to do so 
> and immediately all VMs (Linux Ubuntu OS) running on Ceph disks got 
> readonly filesystems. No errors were shown in ceph (but also no 
> traffic arrived after enabling the cache tier). Removing the cache 
> tier , rebooting the VMs and doing a filesystemcheck repaired everything.
> 
> 
> 
> Best
> 
> Robert
> 
> 
> 
> ___
> ceph-users mailing list -- [ mailto:ceph-users@ceph.io | 
> ceph-users@ceph.io ] To unsubscribe send an email to [ 
> mailto:ceph-users-le...@ceph.io | ceph-users-le...@ceph.io ]
> 
> ___
> ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an 
> email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Activate Cache Tier on Running Pools

[Kees Meijs]

Hi Robert,

As long as you triple-check permissions on the cache tier (should be the
same as your actual storage pool) you should be fine.

In our setup I applied this a few times. The first time I made the
assumption permissions would be inherited or not applicable but IOPs get
redirected towards the tier and therefore permissions should be sane. It
stopped all IOPs until I removed the overlay again. But alas; some VMs
died already.

Later on I extra checked the permissions and all was well.

K.

On 16-09-2019 13:12, Eikermann, Robert wrote:
> That would be the case for me. I think all data during one day would fit into 
> the cache, and we could slowly flush back over night (or even over the 
> weekend). But my impression is, I would have to test it. So my initial 
> question: Do I have to stop all VMs before activating the cache? And restart 
> one by one with the cache enabled?
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: upmap supported in SLES 12SPx

[Thomas Schneider]

Hello,

the current kernel with SLES 12SP3 is:
ld3195:~ # uname -r
4.4.176-94.88-default


Assuming that this kernel is not supporting upmap, do you recommend to
use balance mode crush-compat then?

Regards
Thomas


Am 16.09.2019 um 11:11 schrieb Oliver Freyermuth:
> Am 16.09.19 um 11:06 schrieb Konstantin Shalygin:
>> On 9/16/19 3:59 PM, Thomas wrote:
>>> I tried to run this command with failure:
>>> root@ld3955:/mnt/rbd# ceph osd set-require-min-compat-client luminous
>>> Error EPERM: cannot set require_min_compat_client to luminous: 6
>>> connected client(s) look like jewel (missing 0xa20); 19
>>> connected client(s) look like jewel (missing 0x800); 23
>>> connected client(s) look like jewel (missing 0x800); add
>>> --yes-i-really-mean-it to do it anyway
>>>
>>> root@ld3955:/mnt/rbd# ceph features
>>> {
>>>  "mon": [
>>>  {
>>>  "features": "0x3ffddff8ffac",
>>>  "release": "luminous",
>>>  "num": 3
>>>  }
>>>  ],
>>>  "mds": [
>>>  {
>>>  "features": "0x3ffddff8ffac",
>>>  "release": "luminous",
>>>  "num": 2
>>>  }
>>>  ],
>>>  "osd": [
>>>  {
>>>  "features": "0x3ffddff8ffac",
>>>  "release": "luminous",
>>>  "num": 368
>>>  }
>>>  ],
>>>  "client": [
>>>  {
>>>  "features": "0x40106b84a842a42",
>>>  "release": "jewel",
>>>  "num": 6
>>>  },
>>>  {
>>>  "features": "0x27018eb84aa42a52",
>>>  "release": "jewel",
>>>  "num": 19
>>>  },
>>>  {
>>>  "features": "0x27018fb86aa42ada",
>>>  "release": "jewel",
>>>  "num": 23
>>>  },
>>>  {
>>>  "features": "0x3ffddff8ffac",
>>>  "release": "luminous",
>>>  "num": 35
>>>  }
>>>  ],
>>>  "mgr": [
>>>  {
>>>  "features": "0x3ffddff8ffac",
>>>  "release": "luminous",
>>>  "num": 3
>>>  }
>>>  ]
>>> }
>>>
>>> But I think the number of clients older than Luminous is incorrect.
>>>
>>> I'm running 98% of the clients with SLES 12SPx, and therefore the
>>> question is:
>>> Can you confirm in which SLES 12 release the function upmap is
>>> supported?
>>> And is it safe to execute ceph osd set-require-min-compat-client
>>> luminous --yes-i-really-mean-it? What happens to the clients then?
>>
>> Your kernel should be 4.13+ to be upmap compat. Then you should tell
>> `--yes-i-really-mean-it` to apply changes.
>
> SLES 12 is still on 3.12 I think. However, the question is probably
> rather:
> Does SUSE backport things such as pg-upmap to older Kernels, such as
> RedHat does?
>
> Cheers,
> Oliver
>
>>
>>
>>
>> k
>> ___
>> ceph-users mailing list -- ceph-users@ceph.io
>> To unsubscribe send an email to ceph-users-le...@ceph.io
>
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] KRBD use Luminous upmap feature.Which version of the kernel should i ues?

[潘东元]

hi,
   my ceph cluster version is Luminous run the kernel version Linux 3.10
   [root@node-1 ~]# ceph features
{
"mon": {
"group": {
"features": "0x3ffddff8eeacfffb",
"release": "luminous",
"num": 3
}
},
"osd": {
"group": {
"features": "0x3ffddff8eeacfffb",
"release": "luminous",
"num": 15
}
},
"client": {
"group": {
"features": "0x40106b84a842a52",
"release": "jewel",
"num": 3
},
"group": {
"features": "0x3ffddff8eeacfffb",
"release": "luminous",
"num": 170
}
}
}
Which version of the kernel should i ues?

Thanks!
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: KRBD use Luminous upmap feature.Which version of the kernel should i ues?

[Wesley Peng]

Hi

on 2019/9/16 20:19, 潘东元 wrote:

my ceph cluster version is Luminous run the kernel version Linux 3.10


Please refer this page:
https://docs.ceph.com/docs/master/start/os-recommendations/

see [LUMINOUS] section.

regards.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: upmap supported in SLES 12SPx

[Ilya Dryomov]

On Mon, Sep 16, 2019 at 2:20 PM Thomas Schneider <74cmo...@gmail.com> wrote:
>
> Hello,
>
> the current kernel with SLES 12SP3 is:
> ld3195:~ # uname -r
> 4.4.176-94.88-default
>
>
> Assuming that this kernel is not supporting upmap, do you recommend to
> use balance mode crush-compat then?

Hi Thomas,

Among the clients listed in your output, only 6 don't support upmap
(those with features 0x40106b84a842a42).  The rest do.

>
> Regards
> Thomas
>
>
> Am 16.09.2019 um 11:11 schrieb Oliver Freyermuth:
> > Am 16.09.19 um 11:06 schrieb Konstantin Shalygin:
> >> On 9/16/19 3:59 PM, Thomas wrote:
> >>> I tried to run this command with failure:
> >>> root@ld3955:/mnt/rbd# ceph osd set-require-min-compat-client luminous
> >>> Error EPERM: cannot set require_min_compat_client to luminous: 6
> >>> connected client(s) look like jewel (missing 0xa20); 19
> >>> connected client(s) look like jewel (missing 0x800); 23
> >>> connected client(s) look like jewel (missing 0x800); add
> >>> --yes-i-really-mean-it to do it anyway
> >>>
> >>> root@ld3955:/mnt/rbd# ceph features
> >>> {
> >>>  "mon": [
> >>>  {
> >>>  "features": "0x3ffddff8ffac",
> >>>  "release": "luminous",
> >>>  "num": 3
> >>>  }
> >>>  ],
> >>>  "mds": [
> >>>  {
> >>>  "features": "0x3ffddff8ffac",
> >>>  "release": "luminous",
> >>>  "num": 2
> >>>  }
> >>>  ],
> >>>  "osd": [
> >>>  {
> >>>  "features": "0x3ffddff8ffac",
> >>>  "release": "luminous",
> >>>  "num": 368
> >>>  }
> >>>  ],
> >>>  "client": [
> >>>  {
> >>>  "features": "0x40106b84a842a42",
> >>>  "release": "jewel",
> >>>  "num": 6
> >>>  },
> >>>  {
> >>>  "features": "0x27018eb84aa42a52",
> >>>  "release": "jewel",
> >>>  "num": 19
> >>>  },
> >>>  {
> >>>  "features": "0x27018fb86aa42ada",
> >>>  "release": "jewel",
> >>>  "num": 23
> >>>  },
> >>>  {
> >>>  "features": "0x3ffddff8ffac",
> >>>  "release": "luminous",
> >>>  "num": 35
> >>>  }
> >>>  ],
> >>>  "mgr": [
> >>>  {
> >>>  "features": "0x3ffddff8ffac",
> >>>  "release": "luminous",
> >>>  "num": 3
> >>>  }
> >>>  ]
> >>> }

Thanks,

Ilya
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: KRBD use Luminous upmap feature.Which version of the kernel should i ues?

[Ilya Dryomov]

On Mon, Sep 16, 2019 at 2:24 PM 潘东元  wrote:
>
> hi,
>my ceph cluster version is Luminous run the kernel version Linux 3.10
>[root@node-1 ~]# ceph features
> {
> "mon": {
> "group": {
> "features": "0x3ffddff8eeacfffb",
> "release": "luminous",
> "num": 3
> }
> },
> "osd": {
> "group": {
> "features": "0x3ffddff8eeacfffb",
> "release": "luminous",
> "num": 15
> }
> },
> "client": {
> "group": {
> "features": "0x40106b84a842a52",
> "release": "jewel",
> "num": 3
> },
> "group": {
> "features": "0x3ffddff8eeacfffb",
> "release": "luminous",
> "num": 170
> }
> }
> }
> Which version of the kernel should i ues?

You need any upstream kernel starting with 4.13 or a downstream kernel
with a backport.  For RHEL/CentOS, this is any kernel starting with 7.5
(3.10.0-862).

Thanks,

Ilya
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: upmap supported in SLES 12SPx

[Paul Emmerich]

Bit 21 in the features bitfield is upmap support

Paul

-- 
Paul Emmerich

Looking for help with your Ceph cluster? Contact us at https://croit.io

croit GmbH
Freseniusstr. 31h
81247 München
www.croit.io
Tel: +49 89 1896585 90

On Mon, Sep 16, 2019 at 3:21 PM Ilya Dryomov  wrote:
>
> On Mon, Sep 16, 2019 at 2:20 PM Thomas Schneider <74cmo...@gmail.com> wrote:
> >
> > Hello,
> >
> > the current kernel with SLES 12SP3 is:
> > ld3195:~ # uname -r
> > 4.4.176-94.88-default
> >
> >
> > Assuming that this kernel is not supporting upmap, do you recommend to
> > use balance mode crush-compat then?
>
> Hi Thomas,
>
> Among the clients listed in your output, only 6 don't support upmap
> (those with features 0x40106b84a842a42).  The rest do.
>
> >
> > Regards
> > Thomas
> >
> >
> > Am 16.09.2019 um 11:11 schrieb Oliver Freyermuth:
> > > Am 16.09.19 um 11:06 schrieb Konstantin Shalygin:
> > >> On 9/16/19 3:59 PM, Thomas wrote:
> > >>> I tried to run this command with failure:
> > >>> root@ld3955:/mnt/rbd# ceph osd set-require-min-compat-client luminous
> > >>> Error EPERM: cannot set require_min_compat_client to luminous: 6
> > >>> connected client(s) look like jewel (missing 0xa20); 19
> > >>> connected client(s) look like jewel (missing 0x800); 23
> > >>> connected client(s) look like jewel (missing 0x800); add
> > >>> --yes-i-really-mean-it to do it anyway
> > >>>
> > >>> root@ld3955:/mnt/rbd# ceph features
> > >>> {
> > >>>  "mon": [
> > >>>  {
> > >>>  "features": "0x3ffddff8ffac",
> > >>>  "release": "luminous",
> > >>>  "num": 3
> > >>>  }
> > >>>  ],
> > >>>  "mds": [
> > >>>  {
> > >>>  "features": "0x3ffddff8ffac",
> > >>>  "release": "luminous",
> > >>>  "num": 2
> > >>>  }
> > >>>  ],
> > >>>  "osd": [
> > >>>  {
> > >>>  "features": "0x3ffddff8ffac",
> > >>>  "release": "luminous",
> > >>>  "num": 368
> > >>>  }
> > >>>  ],
> > >>>  "client": [
> > >>>  {
> > >>>  "features": "0x40106b84a842a42",
> > >>>  "release": "jewel",
> > >>>  "num": 6
> > >>>  },
> > >>>  {
> > >>>  "features": "0x27018eb84aa42a52",
> > >>>  "release": "jewel",
> > >>>  "num": 19
> > >>>  },
> > >>>  {
> > >>>  "features": "0x27018fb86aa42ada",
> > >>>  "release": "jewel",
> > >>>  "num": 23
> > >>>  },
> > >>>  {
> > >>>  "features": "0x3ffddff8ffac",
> > >>>  "release": "luminous",
> > >>>  "num": 35
> > >>>  }
> > >>>  ],
> > >>>  "mgr": [
> > >>>  {
> > >>>  "features": "0x3ffddff8ffac",
> > >>>  "release": "luminous",
> > >>>  "num": 3
> > >>>  }
> > >>>  ]
> > >>> }
>
> Thanks,
>
> Ilya
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: upmap supported in SLES 12SPx

[Thomas Schneider]

Hi,

thanks for your valuable input.

Question:
Can I get more information of the 6 clients (those with features
0x40106b84a842a42), e.g. IP, that allows me to identify it easily?

Regards
Thomas


Am 16.09.2019 um 15:56 schrieb Paul Emmerich:
> Bit 21 in the features bitfield is upmap support
>
> Paul
>


Am 16.09.2019 um 15:23 schrieb Ilya Dryomov:
> On Mon, Sep 16, 2019 at 2:20 PM Thomas Schneider <74cmo...@gmail.com> wrote:
>> Hello,
>>
>> the current kernel with SLES 12SP3 is:
>> ld3195:~ # uname -r
>> 4.4.176-94.88-default
>>
>>
>> Assuming that this kernel is not supporting upmap, do you recommend to
>> use balance mode crush-compat then?
> Hi Thomas,
>
> Among the clients listed in your output, only 6 don't support upmap
> (those with features 0x40106b84a842a42).  The rest do.
>
>> Regards
>> Thomas
>>
>>
>> Am 16.09.2019 um 11:11 schrieb Oliver Freyermuth:
>>> Am 16.09.19 um 11:06 schrieb Konstantin Shalygin:
 On 9/16/19 3:59 PM, Thomas wrote:
> I tried to run this command with failure:
> root@ld3955:/mnt/rbd# ceph osd set-require-min-compat-client luminous
> Error EPERM: cannot set require_min_compat_client to luminous: 6
> connected client(s) look like jewel (missing 0xa20); 19
> connected client(s) look like jewel (missing 0x800); 23
> connected client(s) look like jewel (missing 0x800); add
> --yes-i-really-mean-it to do it anyway
>
> root@ld3955:/mnt/rbd# ceph features
> {
>  "mon": [
>  {
>  "features": "0x3ffddff8ffac",
>  "release": "luminous",
>  "num": 3
>  }
>  ],
>  "mds": [
>  {
>  "features": "0x3ffddff8ffac",
>  "release": "luminous",
>  "num": 2
>  }
>  ],
>  "osd": [
>  {
>  "features": "0x3ffddff8ffac",
>  "release": "luminous",
>  "num": 368
>  }
>  ],
>  "client": [
>  {
>  "features": "0x40106b84a842a42",
>  "release": "jewel",
>  "num": 6
>  },
>  {
>  "features": "0x27018eb84aa42a52",
>  "release": "jewel",
>  "num": 19
>  },
>  {
>  "features": "0x27018fb86aa42ada",
>  "release": "jewel",
>  "num": 23
>  },
>  {
>  "features": "0x3ffddff8ffac",
>  "release": "luminous",
>  "num": 35
>  }
>  ],
>  "mgr": [
>  {
>  "features": "0x3ffddff8ffac",
>  "release": "luminous",
>  "num": 3
>  }
>  ]
> }
> Thanks,
>
> Ilya
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: upmap supported in SLES 12SPx

[Ilya Dryomov]

On Mon, Sep 16, 2019 at 4:40 PM Thomas Schneider <74cmo...@gmail.com> wrote:
>
> Hi,
>
> thanks for your valuable input.
>
> Question:
> Can I get more information of the 6 clients (those with features
> 0x40106b84a842a42), e.g. IP, that allows me to identify it easily?

Yes, although it's not integrated into "ceph features".  Log into
a monitor node and run "ceph daemon mon.a sessions" (mon.a is the name
of the monitor, substitute accordingly).

Thanks,

Ilya
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: upmap supported in SLES 12SPx

[Thomas Schneider]

Wonderbra.

I found some relevant sessions on 2 of 3 monitor nodes.
And I found some others:
root@ld5505:~# ceph daemon mon.ld5505 sessions | grep 0x40106b84a842a42
root@ld5505:~# ceph daemon mon.ld5505 sessions | grep -v luminous
[
    "MonSession(client.32679861 v1:10.97.206.92:0/1183647891 is open
allow *, features 0x27018fb86aa42ada (jewel))",
    "MonSession(client.32692978 v1:10.97.206.91:0/3689092992 is open
allow *, features 0x27018fb86aa42ada (jewel))",
    "MonSession(client.11935413 v1:10.96.6.116:0/3187655474 is open
allow r, features 0x27018eb84aa42a52 (jewel))",
    "MonSession(client.3941901 v1:10.76.179.23:0/2967896845 is open
allow r, features 0x27018fb86aa42ada (jewel))",
    "MonSession(client.28313343 v1:10.76.177.108:0/1303617860 is open
allow r, features 0x27018fb86aa42ada (jewel))",
    "MonSession(client.29311725 v1:10.97.206.94:0/224438037 is open
allow *, features 0x27018fb86aa42ada (jewel))",
    "MonSession(client.4535833 v1:10.76.177.133:0/1269608815 is open
allow r, features 0x27018fb86aa42ada (jewel))",
    "MonSession(client.3919902 v1:10.96.4.243:0/293623521 is open allow
r, features 0x27018eb84aa42a52 (jewel))",
    "MonSession(client.35678944 v1:10.76.179.211:0/4218086982 is open
allow r, features 0x27018eb84aa42a52 (jewel))",
    "MonSession(client.35751316 v1:10.76.179.30:0/1348696702 is open
allow r, features 0x27018eb84aa42a52 (jewel))",
    "MonSession(client.28246527 v1:10.96.4.228:0/1495661381 is open
allow r, features 0x27018fb86aa42ada (jewel))",
    "MonSession(client.3917843 v1:10.76.179.22:0/489863209 is open allow
r, features 0x27018fb86aa42ada (jewel))",
    "MonSession(unknown.0 - is open allow r, features 0x27018eb84aa42a52
(jewel))",
]

Would it make sense to shutdown these clients, too?

What confuses me is that the list includes clients that belong to the
Ceph cluster, namely 10.97.206.0/24.
All nodes of the Ceph cluster are identical in terms of OS, kernel, Ceph.

Regards
Thomas

Am 16.09.2019 um 16:56 schrieb Ilya Dryomov:
> On Mon, Sep 16, 2019 at 4:40 PM Thomas Schneider <74cmo...@gmail.com> wrote:
>> Hi,
>>
>> thanks for your valuable input.
>>
>> Question:
>> Can I get more information of the 6 clients (those with features
>> 0x40106b84a842a42), e.g. IP, that allows me to identify it easily?
> Yes, although it's not integrated into "ceph features".  Log into
> a monitor node and run "ceph daemon mon.a sessions" (mon.a is the name
> of the monitor, substitute accordingly).
>
> Thanks,
>
> Ilya
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: upmap supported in SLES 12SPx

[Ilya Dryomov]

On Mon, Sep 16, 2019 at 5:10 PM Thomas Schneider <74cmo...@gmail.com> wrote:
>
> Wonderbra.
>
> I found some relevant sessions on 2 of 3 monitor nodes.
> And I found some others:
> root@ld5505:~# ceph daemon mon.ld5505 sessions | grep 0x40106b84a842a42
> root@ld5505:~# ceph daemon mon.ld5505 sessions | grep -v luminous
> [
> "MonSession(client.32679861 v1:10.97.206.92:0/1183647891 is open
> allow *, features 0x27018fb86aa42ada (jewel))",
> "MonSession(client.32692978 v1:10.97.206.91:0/3689092992 is open
> allow *, features 0x27018fb86aa42ada (jewel))",
> "MonSession(client.11935413 v1:10.96.6.116:0/3187655474 is open
> allow r, features 0x27018eb84aa42a52 (jewel))",
> "MonSession(client.3941901 v1:10.76.179.23:0/2967896845 is open
> allow r, features 0x27018fb86aa42ada (jewel))",
> "MonSession(client.28313343 v1:10.76.177.108:0/1303617860 is open
> allow r, features 0x27018fb86aa42ada (jewel))",
> "MonSession(client.29311725 v1:10.97.206.94:0/224438037 is open
> allow *, features 0x27018fb86aa42ada (jewel))",
> "MonSession(client.4535833 v1:10.76.177.133:0/1269608815 is open
> allow r, features 0x27018fb86aa42ada (jewel))",
> "MonSession(client.3919902 v1:10.96.4.243:0/293623521 is open allow
> r, features 0x27018eb84aa42a52 (jewel))",
> "MonSession(client.35678944 v1:10.76.179.211:0/4218086982 is open
> allow r, features 0x27018eb84aa42a52 (jewel))",
> "MonSession(client.35751316 v1:10.76.179.30:0/1348696702 is open
> allow r, features 0x27018eb84aa42a52 (jewel))",
> "MonSession(client.28246527 v1:10.96.4.228:0/1495661381 is open
> allow r, features 0x27018fb86aa42ada (jewel))",
> "MonSession(client.3917843 v1:10.76.179.22:0/489863209 is open allow
> r, features 0x27018fb86aa42ada (jewel))",
> "MonSession(unknown.0 - is open allow r, features 0x27018eb84aa42a52
> (jewel))",
> ]
>
> Would it make sense to shutdown these clients, too?
>
> What confuses me is that the list includes clients that belong to the
> Ceph cluster, namely 10.97.206.0/24.
> All nodes of the Ceph cluster are identical in terms of OS, kernel, Ceph.

The above output seems consistent with your "ceph features" output: it
lists clients with features 0x27018eb84aa42a52 and 0x27018fb86aa42ada.
Like I said in my previous email, both of these support upmap.

If you temporarily shut them down, set-require-min-compat-client will
work without --yes-i-really-mean-it.

Thanks,

Ilya
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Using same instance name for rgw

[Eric Choi]

bump. anyone?
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: RGW Passthrough

[Casey Bodley]

Hi Robert,

So far the cloud tiering features are still in the design stages. We're 
working on some initial refactoring work to support this abstraction 
(ie. to either satisfy a request against the local rados cluster, or to 
proxy it somewhere else). With respect to passthrough/tiering to AWS, we 
could use help thinking through the user/credential mapping in 
particular. We have a weekly 'RGW Refactoring' meeting on Wednesdays 
where we discuss design and refactoring progress - it's on the upstream 
community calendar, I'll send you an invite.


On 9/13/19 9:59 PM, Robert LeBlanc wrote:
We are very interested in the RGW Passthrough mentioned for Octupus. 
What's the status and how can we help? We want to connect with AWS S3.


Thank you,

Robert LeBlanc
PGP Fingerprint 79A2 9CA4 6CC4 45DD A904  C70E E654 3BB2 FA62 B9F1

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Ceph FS not releasing space after file deletion

[Guilherme Geronimo]

Thank you, Yan.

It took like 10 minutes to execute the scan_links.
I believe the number of Lost+Found decreased in 60%, but the rest of 
them are still causing the MDS crash.


Any other suggestion?

=D

[]'s
Arthur (aKa Guilherme Geronimo)

On 10/09/2019 23:51, Yan, Zheng wrote:

On Wed, Sep 4, 2019 at 6:39 AM Guilherme  wrote:

Dear CEPHers,
Adding some comments to my colleague's post: we are running Mimic 13.2.6  and 
struggling with 2 issues (that might be related):
1) After a "lack of space" event we've tried to remove a 40TB file. The file is 
not there anymore, but no space was released. No process is using the file either.
2) There are many files in /lost+found (~25TB|~5%). Every time we try to remove 
a file, MDS crashes ([1,2]).

The Dennis Kramer's case [3] led me to believe that I need to recreate the FS.
But I refuse to (dis)believe that CEPH hasn't a  repair tool for it.
I thought   "cephfs-table-tool take_inos" could be the answer for my problem, 
but the message [4] were not clear enough.
Can I run the command without resetting the inodes?

[1] Error at ceph -w - https://pastebin.com/imNqBdmH
[2] Error at mds.log - https://pastebin.com/rznkzLHG

For the mds crash issue. 'cephfs-data-scan scan_link' of nautilus
version (14.2.2) should fix it.
snaptable.  You don't need to upgrade whole cluster. Just install nautilus in a
temp machine or compile ceph from source.


[3] Discussion - 
http://lists.opennebula.org/pipermail/ceph-users-ceph.com/2018-July/027845.html
[4] Discussion - 
http://lists.opennebula.org/pipermail/ceph-users-ceph.com/2018-July/027935.html
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Fwd: ceph-users Digest, Vol 80, Issue 54

[Rom Freiman]

unsubscribe

-- Forwarded message -
From: 
Date: Mon, Sep 16, 2019 at 7:22 PM
Subject: ceph-users Digest, Vol 80, Issue 54
To: 


Send ceph-users mailing list submissions to
ceph-users@ceph.io

To subscribe or unsubscribe via email, send a message with subject or
body 'help' to
ceph-users-requ...@ceph.io

You can reach the person managing the list at
ceph-users-ow...@ceph.io

When replying, please edit your Subject line so it is more specific
than "Re: Contents of ceph-users digest..."

Today's Topics:

   1. Re: upmap supported in SLES 12SPx (Ilya Dryomov)
   2. Re: upmap supported in SLES 12SPx (Thomas Schneider)
   3. Re: upmap supported in SLES 12SPx (Ilya Dryomov)
   4. Re: Using same instance name for rgw (Eric Choi)
   5. Re: RGW Passthrough (Casey Bodley)


--

Date: Mon, 16 Sep 2019 16:56:19 +0200
From: Ilya Dryomov 
Subject: [ceph-users] Re: upmap supported in SLES 12SPx
To: Thomas Schneider <74cmo...@gmail.com>
Cc: Konstantin Shalygin , ceph-users

Message-ID:

Content-Type: text/plain; charset="UTF-8"

On Mon, Sep 16, 2019 at 4:40 PM Thomas Schneider <74cmo...@gmail.com> wrote:
>
> Hi,
>
> thanks for your valuable input.
>
> Question:
> Can I get more information of the 6 clients (those with features
> 0x40106b84a842a42), e.g. IP, that allows me to identify it easily?

Yes, although it's not integrated into "ceph features".  Log into
a monitor node and run "ceph daemon mon.a sessions" (mon.a is the name
of the monitor, substitute accordingly).

Thanks,

Ilya

--

Date: Mon, 16 Sep 2019 17:10:37 +0200
From: Thomas Schneider <74cmo...@gmail.com>
Subject: [ceph-users] Re: upmap supported in SLES 12SPx
To: Ilya Dryomov 
Cc: Konstantin Shalygin , ceph-users

Message-ID: <79b342ac-be00-e3e7-cbec-b6c96d3a0...@gmail.com>
Content-Type: text/plain; charset=utf-8

Wonderbra.

I found some relevant sessions on 2 of 3 monitor nodes.
And I found some others:
root@ld5505:~# ceph daemon mon.ld5505 sessions | grep 0x40106b84a842a42
root@ld5505:~# ceph daemon mon.ld5505 sessions | grep -v luminous
[
"MonSession(client.32679861 v1:10.97.206.92:0/1183647891 is open
allow *, features 0x27018fb86aa42ada (jewel))",
"MonSession(client.32692978 v1:10.97.206.91:0/3689092992 is open
allow *, features 0x27018fb86aa42ada (jewel))",
"MonSession(client.11935413 v1:10.96.6.116:0/3187655474 is open
allow r, features 0x27018eb84aa42a52 (jewel))",
"MonSession(client.3941901 v1:10.76.179.23:0/2967896845 is open
allow r, features 0x27018fb86aa42ada (jewel))",
"MonSession(client.28313343 v1:10.76.177.108:0/1303617860 is open
allow r, features 0x27018fb86aa42ada (jewel))",
"MonSession(client.29311725 v1:10.97.206.94:0/224438037 is open
allow *, features 0x27018fb86aa42ada (jewel))",
"MonSession(client.4535833 v1:10.76.177.133:0/1269608815 is open
allow r, features 0x27018fb86aa42ada (jewel))",
"MonSession(client.3919902 v1:10.96.4.243:0/293623521 is open allow
r, features 0x27018eb84aa42a52 (jewel))",
"MonSession(client.35678944 v1:10.76.179.211:0/4218086982 is open
allow r, features 0x27018eb84aa42a52 (jewel))",
"MonSession(client.35751316 v1:10.76.179.30:0/1348696702 is open
allow r, features 0x27018eb84aa42a52 (jewel))",
"MonSession(client.28246527 v1:10.96.4.228:0/1495661381 is open
allow r, features 0x27018fb86aa42ada (jewel))",
"MonSession(client.3917843 v1:10.76.179.22:0/489863209 is open allow
r, features 0x27018fb86aa42ada (jewel))",
"MonSession(unknown.0 - is open allow r, features 0x27018eb84aa42a52
(jewel))",
]

Would it make sense to shutdown these clients, too?

What confuses me is that the list includes clients that belong to the
Ceph cluster, namely 10.97.206.0/24.
All nodes of the Ceph cluster are identical in terms of OS, kernel, Ceph.

Regards
Thomas

Am 16.09.2019 um 16:56 schrieb Ilya Dryomov:
> On Mon, Sep 16, 2019 at 4:40 PM Thomas Schneider <74cmo...@gmail.com>
wrote:
>> Hi,
>>
>> thanks for your valuable input.
>>
>> Question:
>> Can I get more information of the 6 clients (those with features
>> 0x40106b84a842a42), e.g. IP, that allows me to identify it easily?
> Yes, although it's not integrated into "ceph features".  Log into
> a monitor node and run "ceph daemon mon.a sessions" (mon.a is the name
> of the monitor, substitute accordingly).
>
> Thanks,
>
> Ilya

--

Date: Mon, 16 Sep 2019 17:36:35 +0200
From: Ilya Dryomov 
Subject: [ceph-users] Re: upmap supported in SLES 12SPx
To: Thomas Schneider <74cmo...@gmail.com>
Cc: Konstantin Shalygin , ceph-users

Message-ID:

Content-Type: text/plain; charset="UTF-8"

On Mon, Sep 16, 2019 at 5:10 PM Thomas Schneider <74cmo...@gmail.com> wrote:
>
> Wonderbra.
>
> I found some relevant sessions on 2 of 3 monitor nodes.
> And I found some others:
> root@ld5505:~# ceph daemon mon.ld55

[ceph-users] dashboard not working

[solarflow99]

I have mimic installed and for some reason the dashboard isn't showing up.
I see which mon is listed as active for "mgr", the module is enabled, but
nothing is listening on port 8080:

# ceph mgr module ls
{
"enabled_modules": [
"dashboard",
"iostat",
"status"


tcp0  0 10.8.152.4:6800 0.0.0.0:*   LISTEN
 69049/ceph-mgr
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Intergrate Metadata with ElasticSeach

[tuan dung]

Dear all,
Can you show me steps how to intergrate Metadata of Ceph object with
ElasticSeach to improve searching medata performance?
thank you very much.
-
Br,
Dương Tuấn Dũng
0986153686
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Ceph FS not releasing space after file deletion

[Yan, Zheng]

please send me crash log

On Tue, Sep 17, 2019 at 12:56 AM Guilherme Geronimo
 wrote:
>
> Thank you, Yan.
>
> It took like 10 minutes to execute the scan_links.
> I believe the number of Lost+Found decreased in 60%, but the rest of
> them are still causing the MDS crash.
>
> Any other suggestion?
>
> =D
>
> []'s
> Arthur (aKa Guilherme Geronimo)
>
> On 10/09/2019 23:51, Yan, Zheng wrote:
> > On Wed, Sep 4, 2019 at 6:39 AM Guilherme  
> > wrote:
> >> Dear CEPHers,
> >> Adding some comments to my colleague's post: we are running Mimic 13.2.6  
> >> and struggling with 2 issues (that might be related):
> >> 1) After a "lack of space" event we've tried to remove a 40TB file. The 
> >> file is not there anymore, but no space was released. No process is using 
> >> the file either.
> >> 2) There are many files in /lost+found (~25TB|~5%). Every time we try to 
> >> remove a file, MDS crashes ([1,2]).
> >>
> >> The Dennis Kramer's case [3] led me to believe that I need to recreate the 
> >> FS.
> >> But I refuse to (dis)believe that CEPH hasn't a  repair tool for it.
> >> I thought   "cephfs-table-tool take_inos" could be the answer for my 
> >> problem, but the message [4] were not clear enough.
> >> Can I run the command without resetting the inodes?
> >>
> >> [1] Error at ceph -w - https://pastebin.com/imNqBdmH
> >> [2] Error at mds.log - https://pastebin.com/rznkzLHG
> > For the mds crash issue. 'cephfs-data-scan scan_link' of nautilus
> > version (14.2.2) should fix it.
> > snaptable.  You don't need to upgrade whole cluster. Just install nautilus 
> > in a
> > temp machine or compile ceph from source.
> >
> >> [3] Discussion - 
> >> http://lists.opennebula.org/pipermail/ceph-users-ceph.com/2018-July/027845.html
> >> [4] Discussion - 
> >> http://lists.opennebula.org/pipermail/ceph-users-ceph.com/2018-July/027935.html
> >> ___
> >> ceph-users mailing list -- ceph-users@ceph.io
> >> To unsubscribe send an email to ceph-users-le...@ceph.io
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] 14.2.4 Packages Avaliable

[Ashley Merrick]

Have just noticed their is packages available for 14.2.4..

I know with the whole 14.2.3 release and the notes not going out to a good day 
or so later.. but this is not long after the 14.2.3 release..?

Was this release even meant to have come out? Makes it difficult for people 
installing a new node if they can't reply on the current "stable" packages that 
apt/yum will give them.___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] osds xxx have blocked requests > 1048.58 sec / osd.yyy has stuck requests > 67108.9 sec

[Thomas]

Hi,

I have defined pool hdd which is exclusively used by virtual disks of
multiple KVMs / LXCs.
Yesterday I run these commands
osdmaptool om --upmap out.txt --upmap-pool hdd
source out.txt
and Ceph started rebalancing this pool.

However since then no KVM / LXC is reacting anymore.
If I try to start a new KVM it hangs in boot process.

This is the output of ceph health detail:
root@ld3955:/mnt/rbd# ceph health detail
HEALTH_ERR 28 nearfull osd(s); 1 pool(s) nearfull; Reduced data
availability: 1 pg inactive, 1 pg peering; Degraded data redundancy (low
space): 8 pgs backfill_toofull; 1 subtrees have overcommitted pool
target_size_bytes; 1 subtrees have overcommitted pool target_size_ratio;
2 pools have too many placement groups; 672 slow requests are blocked >
32 sec; 4752 stuck requests are blocked > 4096 sec
OSD_NEARFULL 28 nearfull osd(s)
    osd.42 is near full
    osd.44 is near full
    osd.45 is near full
    osd.77 is near full
    osd.84 is near full
    osd.94 is near full
    osd.101 is near full
    osd.103 is near full
    osd.106 is near full
    osd.109 is near full
    osd.113 is near full
    osd.118 is near full
    osd.120 is near full
    osd.136 is near full
    osd.138 is near full
    osd.142 is near full
    osd.147 is near full
    osd.156 is near full
    osd.159 is near full
    osd.161 is near full
    osd.168 is near full
    osd.192 is near full
    osd.202 is near full
    osd.206 is near full
    osd.208 is near full
    osd.226 is near full
    osd.234 is near full
    osd.247 is near full
POOL_NEARFULL 1 pool(s) nearfull
    pool 'hdb_backup' is nearfull
PG_AVAILABILITY Reduced data availability: 1 pg inactive, 1 pg peering
    pg 30.1b9 is stuck peering for 4722.750977, current state peering,
last acting [183,27,63]
PG_DEGRADED_FULL Degraded data redundancy (low space): 8 pgs
backfill_toofull
    pg 11.465 is active+remapped+backfill_wait+backfill_toofull, acting
[308,351,58]
    pg 11.5c4 is active+remapped+backfill_wait+backfill_toofull, acting
[318,336,54]
    pg 11.afd is active+remapped+backfill_wait+backfill_toofull, acting
[347,220,315]
    pg 11.b82 is active+remapped+backfill_toofull, acting [314,320,254]
    pg 11.1803 is active+remapped+backfill_wait+backfill_toofull, acting
[88,363,302]
    pg 11.1aac is active+remapped+backfill_wait+backfill_toofull, acting
[328,275,95]
    pg 11.1c09 is active+remapped+backfill_wait+backfill_toofull, acting
[55,124,278]
    pg 11.1e36 is active+remapped+backfill_wait+backfill_toofull, acting
[351,92,315]
POOL_TARGET_SIZE_BYTES_OVERCOMMITTED 1 subtrees have overcommitted pool
target_size_bytes
    Pools ['hdb_backup'] overcommit available storage by 1.708x due to
target_size_bytes    0  on pools []
POOL_TARGET_SIZE_RATIO_OVERCOMMITTED 1 subtrees have overcommitted pool
target_size_ratio
    Pools ['hdb_backup'] overcommit available storage by 1.708x due to
target_size_ratio 0.000 on pools []
POOL_TOO_MANY_PGS 2 pools have too many placement groups
    Pool hdd has 512 placement groups, should have 128
    Pool pve_cephfs_metadata has 32 placement groups, should have 4
REQUEST_SLOW 672 slow requests are blocked > 32 sec
    249 ops are blocked > 2097.15 sec
    284 ops are blocked > 1048.58 sec
    108 ops are blocked > 524.288 sec
    9 ops are blocked > 262.144 sec
    22 ops are blocked > 131.072 sec
    osd.9 has blocked requests > 524.288 sec
    osds 0,2,6,68 have blocked requests > 1048.58 sec
    osd.3 has blocked requests > 2097.15 sec
REQUEST_STUCK 4752 stuck requests are blocked > 4096 sec
    1431 ops are blocked > 67108.9 sec
    513 ops are blocked > 33554.4 sec
    909 ops are blocked > 16777.2 sec
    1809 ops are blocked > 8388.61 sec
    90 ops are blocked > 4194.3 sec
    osd.63 has stuck requests > 67108.9 sec


My interpretation is that Ceph
a) is busy with remapping PGs of pool hdb_backup
b) has identified several OSDs with either blocked or stuck requests.

Any of these OSDs belongs to pool hdd, though.
osd.9 belongs to node A, osd.63 and osd.68 belongs to node C (there are
4 nodes serving OSD in the cluster).

I have tried to fix this issue, but it failed with
- ceph osd set noout
- restart of relevant OSD by systemctl restart ceph-osd@
and finally server reboot.

I also tried to migrate the virtual disks to another pool, but this
fails, too.

There are no changes on server side, like network or disks or whatsoever.

How can I resolve this issue?

THX
Thomas
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: 14.2.4 Packages Avaliable

[Ronny Aasen]

On 17.09.2019 06:54, Ashley Merrick wrote:

Have just noticed their is packages available for 14.2.4..

I know with the whole 14.2.3 release and the notes not going out to a 
good day or so later.. but this is not long after the 14.2.3 release..?


Was this release even meant to have come out? Makes it difficult for 
people installing a new node if they can't reply on the current "stable" 
packages that apt/yum will give them.



Never install packages until there is an announcement.

IIRC developers have asked if anyone have experience with running repos 
that could assist in improving the rollout of releases since this have 
been a recurring issue.



If you need to do installs all the time, and can not postpone until the 
repo settle. Consider rsyncing the repo after a release, and use that 
for installs.


kind regards
Ronny

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: KRBD use Luminous upmap feature.Which version of the kernel should i ues?

[潘东元]

Thank you for your reply.
so,i would like to verify this problem. i create a new VM as a
client,it is kernel version:
[root@localhost ~]# uname -a
Linux localhost.localdomain 5.2.9-200.fc30.x86_64 #1 SMP Fri Aug 16
21:37:45 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

First of all,use command:ceph features in my cluster
[root@node-1 ~]# ceph features
{
"mon": {
"group": {
"features": "0x3ffddff8eeacfffb",
"release": "luminous",
"num": 3
}
},
"osd": {
"group": {
"features": "0x3ffddff8eeacfffb",
"release": "luminous",
"num": 12
}
},
"client": {
"group": {
"features": "0x3ffddff8eeacfffb",
"release": "luminous",
"num": 7
}
}
}

now, i have no jewel client.And then,i map a rbd image to new VM
[root@localhost ~]# rbd map test
/dev/rbd0
map successful!
new,use ceph features
[root@node-1 ~]# ceph features
{
"mon": {
"group": {
"features": "0x3ffddff8eeacfffb",
"release": "luminous",
"num": 3
}
},
"osd": {
"group": {
"features": "0x3ffddff8eeacfffb",
"release": "luminous",
"num": 12
}
},
"client": {
"group": {
"features": "0x27018fb86aa42ada",
"release": "jewel",
"num": 1
},
"group": {
"features": "0x3ffddff8eeacfffb",
"release": "luminous",
"num": 7
}
}
}
I have a jewel client.It is not an expectation.
why? is it means i still can not use upmap feature?


Ilya Dryomov  于2019年9月16日周一 下午9:22写道：
>
> On Mon, Sep 16, 2019 at 2:24 PM 潘东元  wrote:
> >
> > hi,
> >my ceph cluster version is Luminous run the kernel version Linux 3.10
> >[root@node-1 ~]# ceph features
> > {
> > "mon": {
> > "group": {
> > "features": "0x3ffddff8eeacfffb",
> > "release": "luminous",
> > "num": 3
> > }
> > },
> > "osd": {
> > "group": {
> > "features": "0x3ffddff8eeacfffb",
> > "release": "luminous",
> > "num": 15
> > }
> > },
> > "client": {
> > "group": {
> > "features": "0x40106b84a842a52",
> > "release": "jewel",
> > "num": 3
> > },
> > "group": {
> > "features": "0x3ffddff8eeacfffb",
> > "release": "luminous",
> > "num": 170
> > }
> > }
> > }
> > Which version of the kernel should i ues?
>
> You need any upstream kernel starting with 4.13 or a downstream kernel
> with a backport.  For RHEL/CentOS, this is any kernel starting with 7.5
> (3.10.0-862).
>
> Thanks,
>
> Ilya
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io