[ceph-users] [RBD] Question about group snapshots conception

[Rudenko Aleksandr]

Hi everyone.
I try to understand conception of RBD group snapshots, but I can’t.
Regular snaps (not a group one) we can use like regular RBD images. We can 
export it or use it directly in qemu-img or we can create new image based on 
snap (clone).
But if we talk about group snaps – we can’t do anything. Only restore original 
image.
We try to implement backup solution for VMs, and in case of multi-volume VM – 
group snaps is good feature which give as consistency between images. But we 
can’t copy this snaps to different storage in safe place. We shouldn’t leave 
this snaps in original RBD storage.
Why are regular snaps and group snaps so different?
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: loosing one node from a 3-node cluster

[Robert Sander]

Hi,

Am 05.04.22 um 02:53 schrieb Felix Joussein:

As the command outputs below show, ceph-iso_metadata consume 19TB 
accordingly to ceph df, how ever, the mounted ceph-iso filesystem is 
only 9.2TB big.


The values nearly add up.

ceph-vm has 2.7 TiB stored and 8.3 TiB used (3x replication).
ceph-iso_data has 6.1 TiB stored and 19 TiB used (3x replication).

Your total capacity is 44 TiB, used is 27 TiB (19 + 8.3), leaving 17 TiB 
capacity. Divided by 3 is ca 6 TiB. Your max avail space for the pools 
is only 3 TiB.


AFAIK the available space computation takes data distribution and 
nearfull ratio (85%) into account.


https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd/#no-free-drive-space

How are the individual OSDs used?
Can you post the output of "ceph osd df tree"?

"df -h" will show an artificial filesystem size which is the sum of the 
used space and the available space of the CephFS data pool. I.e. 6.1 TiB 
+ 3.0 TiB makes a df size of 9.2 TiB.


Regards
--
Robert Sander
Heinlein Consulting GmbH
Schwedter Str. 8/9b, 10119 Berlin

http://www.heinlein-support.de

Tel: 030 / 405051-43
Fax: 030 / 405051-19

Zwangsangaben lt. §35a GmbHG:
HRB 220009 B / Amtsgericht Berlin-Charlottenburg,
Geschäftsführer: Peer Heinlein -- Sitz: Berlin
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Ceph Bluestore tweaks for Bcache

[Richard Bade]

Hi Frank, yes I changed the device class to HDD but there seems to be some
smarts in the background that apply the different settings that are not
based on the class but some other internal mechanism.
However, I did apply the class after creating the osd, rather than during.
If someone knows how to manually specify this, I'd also be interested to
know.

I probably should also have said that I am using Nautilus and it may be
different in newer versions.

Rich


On Tue, 5 Apr 2022, 20:39 Frank Schilder,  wrote:

> Hi Richard,
>
> I'm planning to use dm_cache with bluestore OSDs on LVM. I was also
> wondering how the device will be detected. I guess if I build the OSD
> before assigning dm_cache space it will use the usual HDD defaults. Did you
> try forcing the OSD to be in class HDD on build? I believe the OSD create
> commands have a flag for that.
>
> If any of the OSD gurus looks at this, could you possibly point to a
> reference about what parameters might need attention in such scenarios and
> what the preferred deployment method would be?
>
> Thanks and best regards,
> =
> Frank Schilder
> AIT Risø Campus
> Bygning 109, rum S14
>
> 
> From: Richard Bade 
> Sent: 05 April 2022 00:07:34
> To: Ceph Users
> Subject: [ceph-users] Ceph Bluestore tweaks for Bcache
>
> Hi Everyone,
> I just wanted to share a discovery I made about running bluestore on
> top of Bcache in case anyone else is doing this or considering it.
> We've run Bcache under Filestore for a long time with good results but
> recently rebuilt all the osds on bluestore. This caused some
> degradation in performance that I couldn't quite put my finger on.
> Bluestore osds have some smarts where they detect the disk type.
> Unfortunately in the case of Bcache it detects as SSD, when in fact
> the HDD parameters are better suited.
> I changed the following parameters to match the HDD default values and
> immediately saw my average osd latency during normal workload drop
> from 6ms to 2ms. Peak performance didn't change really, but a test
> machine that I have running a constant iops workload was much more
> stable as was the average latency.
> Performance has returned to Filestore or better levels.
> Here are the parameters.
>
>  ; Make sure that we use values appropriate for HDD not SSD - Bcache
> gets detected as SSD
>  bluestore_prefer_deferred_size = 32768
>  bluestore_compression_max_blob_size = 524288
>  bluestore_deferred_batch_ops = 64
>  bluestore_max_blob_size = 524288
>  bluestore_min_alloc_size = 65536
>  bluestore_throttle_cost_per_io = 67
>
>  ; Try to improve responsiveness when some disks are fully utilised
>  osd_op_queue = wpq
>  osd_op_queue_cut_off = high
>
> Hopefully someone else finds this useful.
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Ceph Bluestore tweaks for Bcache

[Rudenko Aleksandr]

OSD uses sysfs device parameter "rotational" for detecting device type 
(HDD/SSD).

You can see it:

ceph osd metadata {osd_id}

On 05.04.2022, 11:49, "Richard Bade"  wrote:

Hi Frank, yes I changed the device class to HDD but there seems to be some
smarts in the background that apply the different settings that are not
based on the class but some other internal mechanism.
However, I did apply the class after creating the osd, rather than during.
If someone knows how to manually specify this, I'd also be interested to
know.

I probably should also have said that I am using Nautilus and it may be
different in newer versions.

Rich


On Tue, 5 Apr 2022, 20:39 Frank Schilder,  wrote:

> Hi Richard,
>
> I'm planning to use dm_cache with bluestore OSDs on LVM. I was also
> wondering how the device will be detected. I guess if I build the OSD
> before assigning dm_cache space it will use the usual HDD defaults. Did 
you
> try forcing the OSD to be in class HDD on build? I believe the OSD create
> commands have a flag for that.
>
> If any of the OSD gurus looks at this, could you possibly point to a
> reference about what parameters might need attention in such scenarios and
> what the preferred deployment method would be?
>
> Thanks and best regards,
> =
> Frank Schilder
> AIT Risø Campus
> Bygning 109, rum S14
>
> 
> From: Richard Bade 
> Sent: 05 April 2022 00:07:34
> To: Ceph Users
> Subject: [ceph-users] Ceph Bluestore tweaks for Bcache
>
> Hi Everyone,
> I just wanted to share a discovery I made about running bluestore on
> top of Bcache in case anyone else is doing this or considering it.
> We've run Bcache under Filestore for a long time with good results but
> recently rebuilt all the osds on bluestore. This caused some
> degradation in performance that I couldn't quite put my finger on.
> Bluestore osds have some smarts where they detect the disk type.
> Unfortunately in the case of Bcache it detects as SSD, when in fact
> the HDD parameters are better suited.
> I changed the following parameters to match the HDD default values and
> immediately saw my average osd latency during normal workload drop
> from 6ms to 2ms. Peak performance didn't change really, but a test
> machine that I have running a constant iops workload was much more
> stable as was the average latency.
> Performance has returned to Filestore or better levels.
> Here are the parameters.
>
>  ; Make sure that we use values appropriate for HDD not SSD - Bcache
> gets detected as SSD
>  bluestore_prefer_deferred_size = 32768
>  bluestore_compression_max_blob_size = 524288
>  bluestore_deferred_batch_ops = 64
>  bluestore_max_blob_size = 524288
>  bluestore_min_alloc_size = 65536
>  bluestore_throttle_cost_per_io = 67
>
>  ; Try to improve responsiveness when some disks are fully utilised
>  osd_op_queue = wpq
>  osd_op_queue_cut_off = high
>
> Hopefully someone else finds this useful.
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] ceph bluestore

[Ali Akil]

Hallo everybody,

I have two questions regarding bluestore. I am struggling to understand
the documentation :/

I am planning to deploy 3 ceph nodes with 10xHDDs for OSD data, Raid 0
2xSSDs for block.db with replication on host level.

First Question :
Is it possible to deploy block.db on RAID 0 parition ? and Do i need to
backup the SSDs for block.db or the data will be replicated on the other
nodes ?

Second:
Under the BLOCK and BLOCK.DB  section in the documentation is stated the i must
create volume groups and logical volumes, if i want to locate block.db
on another disk. It's not stated though the reason behind that. So why
it's not possible to just assign the block.db to the disk with E.g.
--block.db /dev/sdb without creating a logical volume?

Also what is the role for `ceph-volume lvm prepare` , if one should
create these logical volumes manually?

Thanks a lot,
Ali
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Ceph Bluestore tweaks for Bcache

[Igor Fedotov]

Hi Richard,

just FYI: one can use "bluestore debug enforce settings=hdd" config 
parameter to manually enforce HDD-related  settings for a BlueStore



Thanks,

Igor

On 4/5/2022 1:07 AM, Richard Bade wrote:

Hi Everyone,
I just wanted to share a discovery I made about running bluestore on
top of Bcache in case anyone else is doing this or considering it.
We've run Bcache under Filestore for a long time with good results but
recently rebuilt all the osds on bluestore. This caused some
degradation in performance that I couldn't quite put my finger on.
Bluestore osds have some smarts where they detect the disk type.
Unfortunately in the case of Bcache it detects as SSD, when in fact
the HDD parameters are better suited.
I changed the following parameters to match the HDD default values and
immediately saw my average osd latency during normal workload drop
from 6ms to 2ms. Peak performance didn't change really, but a test
machine that I have running a constant iops workload was much more
stable as was the average latency.
Performance has returned to Filestore or better levels.
Here are the parameters.

  ; Make sure that we use values appropriate for HDD not SSD - Bcache
gets detected as SSD
  bluestore_prefer_deferred_size = 32768
  bluestore_compression_max_blob_size = 524288
  bluestore_deferred_batch_ops = 64
  bluestore_max_blob_size = 524288
  bluestore_min_alloc_size = 65536
  bluestore_throttle_cost_per_io = 67

  ; Try to improve responsiveness when some disks are fully utilised
  osd_op_queue = wpq
  osd_op_queue_cut_off = high

Hopefully someone else finds this useful.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io


--
Igor Fedotov
Ceph Lead Developer

Looking for help with your Ceph cluster? Contact us at https://croit.io

croit GmbH, Freseniusstr. 31h, 81247 Munich
CEO: Martin Verges - VAT-ID: DE310638492
Com. register: Amtsgericht Munich HRB 231263
Web: https://croit.io | YouTube: https://goo.gl/PGE1Bx

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: quincy v17.2.0 QE Validation status

[Ernesto Puerta]

Hi Josh,

I'm stuck with the Grafana (ceph/ceph-grafana) image issue. I'm discussing
this with Dan & David just to see how to move forward:

   - Our Docker hub credentials are no longer working (it seems we don't
   push cephadm images to Docker hub anymore).
   - The Quay.io credentials (Dan's) don't work either (some organization
   account limitation enforced by Quay.io)
   - Dan was suggesting to push the Grafana image to Ceph's Quay instead (
   https://quay.ceph.io/)

I'm ok with either approach (quay.io or Ceph Quay), but I don't find in
Jenkins the required credentials for the Ceph org in quay.io or quay.ceph.io.
Can anyone help here? Thanks!

Once that's sorted out, it'd be just a matter of updating cephadm
references to the Grafana tag and/or registry.

Kind Regards,
Ernesto


On Tue, Apr 5, 2022 at 4:15 AM Venky Shankar  wrote:

> Hey Josh,
>
> On Tue, Apr 5, 2022 at 4:34 AM Josh Durgin  wrote:
> >
> > Hi Venky and Ernesto, how are the mount fix and grafana container build
> looking?
>
> Currently running into various teuthology related issues when testing
> out the mount fix.
>
> We'll want a test run without these failures to be really sure that we
> aren't missing anything.
>
> >
> > Josh
> >
> >
> > On Fri, Apr 1, 2022 at 8:22 AM Venky Shankar 
> wrote:
> >>
> >> On Thu, Mar 31, 2022 at 8:51 PM Venky Shankar 
> wrote:
> >> >
> >> > Hi Yuri,
> >> >
> >> > On Wed, Mar 30, 2022 at 11:24 PM Yuri Weinstein 
> wrote:
> >> > >
> >> > > We merged rgw, cephadm and core PRs, but some work is still pending
> on fs and dashboard components.
> >> > >
> >> > > Seeking approvals for:
> >> > >
> >> > > smoke - Venky
> >> > > fs - Venky
> >> >
> >> > I approved the latest batch for cephfs PRs:
> >> >
> https://trello.com/c/Iq3WtUK5/1494-wip-yuri-testing-2022-03-29-0741-quincy
> >> >
> >> > There is one pending (blocker) PR:
> >> > https://github.com/ceph/ceph/pull/45689 - I'll let you know when the
> >> > backport is available.
> >>
> >> Smoke test passes with the above PR:
> >>
> https://pulpito.ceph.com/vshankar-2022-04-01_12:29:01-smoke-wip-vshankar-testing1-20220401-123425-testing-default-smithi/
> >>
> >> Requested Yuri to run FS suite w/ master (jobs were not getting
> >> scheduled in my run). Thanks, Yuri!
> >>
> >> ___
> >> ceph-users mailing list -- ceph-users@ceph.io
> >> To unsubscribe send an email to ceph-users-le...@ceph.io
> >>
>
>
> --
> Cheers,
> Venky
>
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: ceph bluestore

[Janne Johansson]

Den tis 5 apr. 2022 kl 11:26 skrev Ali Akil :
> Hallo everybody,
> I have two questions regarding bluestore. I am struggling to understand
> the documentation :/
>
> I am planning to deploy 3 ceph nodes with 10xHDDs for OSD data, Raid 0
> 2xSSDs for block.db with replication on host level.
>
> First Question :
> Is it possible to deploy block.db on RAID 0 parition ? and Do i need to
> backup the SSDs for block.db or the data will be replicated on the other
> nodes ?

Raid-0 on the block DB will mean that if one of the SSDs die, *all* of your OSDs
are now lost. I would have each ssd be block.db for half the HDDs, so
that no single
failure causes the whole host to be lost. You would still lose half
the OSDs, but the
other half will still work if one of the SSDs die.

> Second:
> Under the BLOCK and BLOCK.DB  HDDs for OSD, Raid 0 2xSSDs for block.db with replication on host level.
> Do i need to backup the SSDs for block.db or the data will be replicated
> on the other hosts?> section in the documentation is stated the i must
> create volume groups and logical volumes, if i want to locate block.db
> on another disk. It's not stated though the reason behind that. So why
> it's not possible to just assign the block.db to the disk with E.g.
> --block.db /dev/sdb without creating a logical volume?

You can use --block.db /dev/sdb but it will use the whole of sdb for
that one single OSD you are creating.
In order to split a device as block.db for several OSDs, you should
partition it and give each OSD one partition for block.db

> Also what is the role for `ceph-volume lvm prepare` , if one should
> create these logical volumes manually?

The create pass is actually two tasks, prepare and activate. If you
only want to do the first half, "prepare" is there so you can
"activate" them later (i.e. set up systemd autostart and so on).

-- 
May the most significant bit of your life be positive.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: loosing one node from a 3-node cluster

[Felix Joussein]

Hello Robert,

thank you for your reply, so what am I missing?

I thought, that if I have 3-nodes, each 16TB on 4 OSDs, so 16 OSDs having in total 44T, that would leed me at size of 3/2 to:

Either nearly 14TB total pool size knowing, that in case of a lost node, there will be no re-distribution due to no OSD space left, so the cluster-state would be degreaded, which is acceptable for short-term hardware/software maintenance or

66% of 16GB (would be approx. 11TB) so that a re-distribution accross 2 nodes would be possible.

 

The point is,  the advertised pool-sizes of roughly 9 and 6 TB to the operating system which seamed valid sizes to me in terms of "near to 16TB" have never been parametrized from me.

In proxmox, all I did was create new pool and leave all to default.


Also I don't understand, why, if 2 pools, both same num of pgs, same size/min parameters end in beeing one 6 and the other 9 TB.

any clue to that?

regards,

felix

 

Gesendet: Dienstag, 05. April 2022 um 10:44 Uhr
Von: "Robert Sander" 
An: ceph-users@ceph.io
Betreff: [ceph-users] Re: loosing one node from a 3-node cluster

Hi,

Am 05.04.22 um 02:53 schrieb Felix Joussein:

> As the command outputs below show, ceph-iso_metadata consume 19TB
> accordingly to ceph df, how ever, the mounted ceph-iso filesystem is
> only 9.2TB big.

The values nearly add up.

ceph-vm has 2.7 TiB stored and 8.3 TiB used (3x replication).
ceph-iso_data has 6.1 TiB stored and 19 TiB used (3x replication).

Your total capacity is 44 TiB, used is 27 TiB (19 + 8.3), leaving 17 TiB
capacity. Divided by 3 is ca 6 TiB. Your max avail space for the pools
is only 3 TiB.

AFAIK the available space computation takes data distribution and
nearfull ratio (85%) into account.

https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd/#no-free-drive-space

How are the individual OSDs used?
Can you post the output of "ceph osd df tree"?

"df -h" will show an artificial filesystem size which is the sum of the
used space and the available space of the CephFS data pool. I.e. 6.1 TiB
+ 3.0 TiB makes a df size of 9.2 TiB.

Regards
--
Robert Sander
Heinlein Consulting GmbH
Schwedter Str. 8/9b, 10119 Berlin

http://www.heinlein-support.de

Tel: 030 / 405051-43
Fax: 030 / 405051-19

Zwangsangaben lt. §35a GmbHG:
HRB 220009 B / Amtsgericht Berlin-Charlottenburg,
Geschäftsführer: Peer Heinlein -- Sitz: Berlin
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io



___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Recovery or recreation of a monitor rocksdb

[Victor Rodriguez]

Yes, I did end up destroying and recreating the monitor.

As I wanted to use the same IP it was somewhat tedious as I had to 
restart every OSD so they will catch the new value for mon_host.


Is there any way to tell all OSD that mon_host has a new value without 
restarting them?




On 4/4/22 16:48, Konstantin Shalygin wrote:

Hi,

The fast way to fix quorum issue is redeploy ceph-mon service


k
Sent from my iPhone


On 1 Apr 2022, at 14:43, Victor Rodriguez  wrote:

Hello,

Have a 3 node cluster using Proxmox + ceph version 14.2.22 (nautilus). After a 
power failure one of the monitors does not start. The log states some kind of 
problem with it's rocksdb but I can't really pinpoint the issue. The log is 
available at https://pastebin.com/TZrFrZ1u.

How can I check or repair the rocksdb of this monitor?

Is there anyway to force the replication from another monitor?

Should I just remove that monitor from the cluster and re-add it back?

Should I force something to remove it from the cluster?


I've had problems with rocksdb only once before. Then it was an OSD and simply 
removed it and recreated and Ceph did rebuild/replace all PGs, etc.

Many thanks in advance.


___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io


___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] RuntimeError on activate lvm

[Dominique Ramaekers]

Hi,


I've setup a ceph cluster using cephadmin on three ubuntu servers. Everything 
went great until I tried to activate a osd prepared on a lvm.


I have prepared 4 volumes with the command:

ceph-volume lvm prepare --data vg/lv


Now I try to activate one of them with the command (followed by the output):

root@hvs001:/# ceph-volume lvm activate 0 25bfe96a-4f7a-47e1-8644-b74a4d104dbc
Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-0
Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0
Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir 
--dev /dev/hvs001_sda2/lvol0 --path /var/lib/ceph/osd/ceph-0 --no-mon-config
Running command: /usr/bin/ln -snf /dev/hvs001_sda2/lvol0 
/var/lib/ceph/osd/ceph-0/block
Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-0/block
Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1
Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0
Running command: /usr/bin/systemctl enable 
ceph-volume@lvm-0-25bfe96a-4f7a-47e1-8644-b74a4d104dbc
 stderr: Created symlink 
/etc/systemd/system/multi-user.target.wants/ceph-volume@lvm-0-25bfe96a-4f7a-47e1-8644-b74a4d104dbc.service
 -> /usr/lib/systemd/system/ceph-volume@.service.
Running command: /usr/bin/systemctl enable --runtime ceph-osd@0
 stderr: Created symlink 
/run/systemd/system/ceph-osd.target.wants/ceph-osd@0.service -> 
/usr/lib/systemd/system/ceph-osd@.service.
Running command: /usr/bin/systemctl start ceph-osd@0
 stderr: Failed to connect to bus: No such file or directory
-->  RuntimeError: command returned non-zero exit status: 1


Seems systemd isn't playing along?


Please advice.


Some additional backround info:

root@hvs001:/# ceph status
  cluster:
id: dd4b0610-b4d2-11ec-bb58-d1b32ae31585
health: HEALTH_OK

  services:
mon: 3 daemons, quorum hvs001,hvs002,hvs003 (age 23m)
mgr: hvs001.baejuo(active, since 23m), standbys: hvs002.etijdk
osd: 4 osds: 0 up, 2 in (since 36m)

  data:
pools:   0 pools, 0 pgs
objects: 0 objects, 0 B
usage:   0 B used, 0 B / 0 B avail
pgs:


root@hvs001:/# ceph-volume lvm list


== osd.0 ===

  [block]   /dev/hvs001_sda2/lvol0

  block device  /dev/hvs001_sda2/lvol0
  block uuid6cEw8v-5xIA-K76l-7zIN-V2BK-RNWD-yGwfqp
  cephx lockbox secret
  cluster fsid  dd4b0610-b4d2-11ec-bb58-d1b32ae31585
  cluster name  ceph
  crush device class
  encrypted 0
  osd fsid  25bfe96a-4f7a-47e1-8644-b74a4d104dbc
  osd id0
  osdspec affinity
  type  block
  vdo   0
  devices   /dev/sda2

== osd.1 ===

  [block]   /dev/hvs001_sdb3/lvol1




Greetings,


Dominique.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] memory recommendation for monitors

[Ali Akil]

Hallo everybody,

the official documentation recommends for the monitor nodes 32GB for a
small clusters. Is that per node?
Like i would need 3 nodes with 32GB RAM each in addition to the OSD nodes?

my cluster will consist of 3 replicated OSD nodes (12 OSD each), how can
i calculate the required amount of memory for monitor nodes ?

Best regards,
Ali

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: quincy v17.2.0 QE Validation status

[Dan Mick]

On 4/5/2022 2:47 AM, Ernesto Puerta wrote:

Hi Josh,

I'm stuck with the Grafana (ceph/ceph-grafana) image issue. I'm 
discussing this with Dan & David just to see how to move forward:


  * Our Docker hub credentials are no longer working (it seems we don't
push cephadm images to Docker hub anymore).
  * The Quay.io credentials (Dan's) don't work either (some organization
account limitation enforced by Quay.io)
  * Dan was suggesting to push the Grafana image to Ceph's Quay instead
(https://quay.ceph.io/ )


Some confusion there, probably mine; quay.ceph.io is for CI, not 
release.  I think we're on track now but I just wanted to snip off that 
loose end.




I'm ok with either approach (quay.io  or Ceph Quay), but 
I don't find in Jenkins the required credentials for the Ceph org in 
quay.io  or quay.ceph.io . Can 
anyone help here? Thanks!


Once that's sorted out, it'd be just a matter of updating cephadm 
references to the Grafana tag and/or registry.


Kind Regards,
Ernesto


On Tue, Apr 5, 2022 at 4:15 AM Venky Shankar > wrote:


Hey Josh,

On Tue, Apr 5, 2022 at 4:34 AM Josh Durgin mailto:jdur...@redhat.com>> wrote:
 >
 > Hi Venky and Ernesto, how are the mount fix and grafana container
build looking?

Currently running into various teuthology related issues when testing
out the mount fix.

We'll want a test run without these failures to be really sure that we
aren't missing anything.

 >
 > Josh
 >
 >
 > On Fri, Apr 1, 2022 at 8:22 AM Venky Shankar mailto:vshan...@redhat.com>> wrote:
 >>
 >> On Thu, Mar 31, 2022 at 8:51 PM Venky Shankar
mailto:vshan...@redhat.com>> wrote:
 >> >
 >> > Hi Yuri,
 >> >
 >> > On Wed, Mar 30, 2022 at 11:24 PM Yuri Weinstein
mailto:ywein...@redhat.com>> wrote:
 >> > >
 >> > > We merged rgw, cephadm and core PRs, but some work is still
pending on fs and dashboard components.
 >> > >
 >> > > Seeking approvals for:
 >> > >
 >> > > smoke - Venky
 >> > > fs - Venky
 >> >
 >> > I approved the latest batch for cephfs PRs:
 >> >
https://trello.com/c/Iq3WtUK5/1494-wip-yuri-testing-2022-03-29-0741-quincy

 >> >
 >> > There is one pending (blocker) PR:
 >> > https://github.com/ceph/ceph/pull/45689
 - I'll let you know when the
 >> > backport is available.
 >>
 >> Smoke test passes with the above PR:
 >>

https://pulpito.ceph.com/vshankar-2022-04-01_12:29:01-smoke-wip-vshankar-testing1-20220401-123425-testing-default-smithi/


 >>
 >> Requested Yuri to run FS suite w/ master (jobs were not getting
 >> scheduled in my run). Thanks, Yuri!
 >>
 >> ___
 >> ceph-users mailing list -- ceph-users@ceph.io

 >> To unsubscribe send an email to ceph-users-le...@ceph.io

 >>


-- 
Cheers,

Venky



___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: memory recommendation for monitors

[Anthony D'Atri]

32GB for a dedicated node that only runs mon / mgr daemons; no OSDs.  I’ve 
experienced a cluster that grew over time such that 32GB was enough to run 
steady-state, but as OSDs and PGs were added to the cluster it was no longer 
enough to *boot* the daemons and I had to do emergency upgrades to 64 or 128GB. 
 That was around a thousand OSDs.

If you’re colocating the mon / mgr daemons on your OSD nodes, I would suggest 
reserving that much RAM for those daemons, plus the sum of your osd memory 
targets.  I suggest provisioning 2x the sum of osd memory targets to handle 
spikes.

> Hallo everybody,
> 
> the official documentation recommends for the monitor nodes 32GB for a
> small clusters. Is that per node?
> Like i would need 3 nodes with 32GB RAM each in addition to the OSD nodes?
> 
> my cluster will consist of 3 replicated OSD nodes (12 OSD each), how can
> i calculate the required amount of memory for monitor nodes ?
> 
> Best regards,
> Ali
> 
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Ceph Bluestore tweaks for Bcache

[Richard Bade]

Thanks Igor for the tip. I'll see if I can use this to reduce the
number of tweaks I need.

Rich

On Tue, 5 Apr 2022 at 21:26, Igor Fedotov  wrote:
>
> Hi Richard,
>
> just FYI: one can use "bluestore debug enforce settings=hdd" config
> parameter to manually enforce HDD-related  settings for a BlueStore
>
>
> Thanks,
>
> Igor
>
> On 4/5/2022 1:07 AM, Richard Bade wrote:
> > Hi Everyone,
> > I just wanted to share a discovery I made about running bluestore on
> > top of Bcache in case anyone else is doing this or considering it.
> > We've run Bcache under Filestore for a long time with good results but
> > recently rebuilt all the osds on bluestore. This caused some
> > degradation in performance that I couldn't quite put my finger on.
> > Bluestore osds have some smarts where they detect the disk type.
> > Unfortunately in the case of Bcache it detects as SSD, when in fact
> > the HDD parameters are better suited.
> > I changed the following parameters to match the HDD default values and
> > immediately saw my average osd latency during normal workload drop
> > from 6ms to 2ms. Peak performance didn't change really, but a test
> > machine that I have running a constant iops workload was much more
> > stable as was the average latency.
> > Performance has returned to Filestore or better levels.
> > Here are the parameters.
> >
> >   ; Make sure that we use values appropriate for HDD not SSD - Bcache
> > gets detected as SSD
> >   bluestore_prefer_deferred_size = 32768
> >   bluestore_compression_max_blob_size = 524288
> >   bluestore_deferred_batch_ops = 64
> >   bluestore_max_blob_size = 524288
> >   bluestore_min_alloc_size = 65536
> >   bluestore_throttle_cost_per_io = 67
> >
> >   ; Try to improve responsiveness when some disks are fully utilised
> >   osd_op_queue = wpq
> >   osd_op_queue_cut_off = high
> >
> > Hopefully someone else finds this useful.
> > ___
> > ceph-users mailing list -- ceph-users@ceph.io
> > To unsubscribe send an email to ceph-users-le...@ceph.io
>
> --
> Igor Fedotov
> Ceph Lead Developer
>
> Looking for help with your Ceph cluster? Contact us at https://croit.io
>
> croit GmbH, Freseniusstr. 31h, 81247 Munich
> CEO: Martin Verges - VAT-ID: DE310638492
> Com. register: Amtsgericht Munich HRB 231263
> Web: https://croit.io | YouTube: https://goo.gl/PGE1Bx
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Ceph Bluestore tweaks for Bcache

[Richard Bade]

Thanks, this should help me with some debugging around the setting
Igor suggested.

Rich

On Tue, 5 Apr 2022 at 21:20, Rudenko Aleksandr  wrote:
>
> OSD uses sysfs device parameter "rotational" for detecting device type 
> (HDD/SSD).
>
> You can see it:
>
> ceph osd metadata {osd_id}
>
> On 05.04.2022, 11:49, "Richard Bade"  wrote:
>
> Hi Frank, yes I changed the device class to HDD but there seems to be some
> smarts in the background that apply the different settings that are not
> based on the class but some other internal mechanism.
> However, I did apply the class after creating the osd, rather than during.
> If someone knows how to manually specify this, I'd also be interested to
> know.
>
> I probably should also have said that I am using Nautilus and it may be
> different in newer versions.
>
> Rich
>
>
> On Tue, 5 Apr 2022, 20:39 Frank Schilder,  wrote:
>
> > Hi Richard,
> >
> > I'm planning to use dm_cache with bluestore OSDs on LVM. I was also
> > wondering how the device will be detected. I guess if I build the OSD
> > before assigning dm_cache space it will use the usual HDD defaults. Did 
> you
> > try forcing the OSD to be in class HDD on build? I believe the OSD 
> create
> > commands have a flag for that.
> >
> > If any of the OSD gurus looks at this, could you possibly point to a
> > reference about what parameters might need attention in such scenarios 
> and
> > what the preferred deployment method would be?
> >
> > Thanks and best regards,
> > =
> > Frank Schilder
> > AIT Risø Campus
> > Bygning 109, rum S14
> >
> > 
> > From: Richard Bade 
> > Sent: 05 April 2022 00:07:34
> > To: Ceph Users
> > Subject: [ceph-users] Ceph Bluestore tweaks for Bcache
> >
> > Hi Everyone,
> > I just wanted to share a discovery I made about running bluestore on
> > top of Bcache in case anyone else is doing this or considering it.
> > We've run Bcache under Filestore for a long time with good results but
> > recently rebuilt all the osds on bluestore. This caused some
> > degradation in performance that I couldn't quite put my finger on.
> > Bluestore osds have some smarts where they detect the disk type.
> > Unfortunately in the case of Bcache it detects as SSD, when in fact
> > the HDD parameters are better suited.
> > I changed the following parameters to match the HDD default values and
> > immediately saw my average osd latency during normal workload drop
> > from 6ms to 2ms. Peak performance didn't change really, but a test
> > machine that I have running a constant iops workload was much more
> > stable as was the average latency.
> > Performance has returned to Filestore or better levels.
> > Here are the parameters.
> >
> >  ; Make sure that we use values appropriate for HDD not SSD - Bcache
> > gets detected as SSD
> >  bluestore_prefer_deferred_size = 32768
> >  bluestore_compression_max_blob_size = 524288
> >  bluestore_deferred_batch_ops = 64
> >  bluestore_max_blob_size = 524288
> >  bluestore_min_alloc_size = 65536
> >  bluestore_throttle_cost_per_io = 67
> >
> >  ; Try to improve responsiveness when some disks are fully utilised
> >  osd_op_queue = wpq
> >  osd_op_queue_cut_off = high
> >
> > Hopefully someone else finds this useful.
> > ___
> > ceph-users mailing list -- ceph-users@ceph.io
> > To unsubscribe send an email to ceph-users-le...@ceph.io
> >
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Ceph Bluestore tweaks for Bcache

[Richard Bade]

Just for completeness for anyone that is following this thread. Igor
added that setting in Octopus, so unfortunately I am unable to use it
as I am still on Nautilus.

Thanks,
Rich

On Wed, 6 Apr 2022 at 10:01, Richard Bade  wrote:
>
> Thanks Igor for the tip. I'll see if I can use this to reduce the
> number of tweaks I need.
>
> Rich
>
> On Tue, 5 Apr 2022 at 21:26, Igor Fedotov  wrote:
> >
> > Hi Richard,
> >
> > just FYI: one can use "bluestore debug enforce settings=hdd" config
> > parameter to manually enforce HDD-related  settings for a BlueStore
> >
> >
> > Thanks,
> >
> > Igor
> >
> > On 4/5/2022 1:07 AM, Richard Bade wrote:
> > > Hi Everyone,
> > > I just wanted to share a discovery I made about running bluestore on
> > > top of Bcache in case anyone else is doing this or considering it.
> > > We've run Bcache under Filestore for a long time with good results but
> > > recently rebuilt all the osds on bluestore. This caused some
> > > degradation in performance that I couldn't quite put my finger on.
> > > Bluestore osds have some smarts where they detect the disk type.
> > > Unfortunately in the case of Bcache it detects as SSD, when in fact
> > > the HDD parameters are better suited.
> > > I changed the following parameters to match the HDD default values and
> > > immediately saw my average osd latency during normal workload drop
> > > from 6ms to 2ms. Peak performance didn't change really, but a test
> > > machine that I have running a constant iops workload was much more
> > > stable as was the average latency.
> > > Performance has returned to Filestore or better levels.
> > > Here are the parameters.
> > >
> > >   ; Make sure that we use values appropriate for HDD not SSD - Bcache
> > > gets detected as SSD
> > >   bluestore_prefer_deferred_size = 32768
> > >   bluestore_compression_max_blob_size = 524288
> > >   bluestore_deferred_batch_ops = 64
> > >   bluestore_max_blob_size = 524288
> > >   bluestore_min_alloc_size = 65536
> > >   bluestore_throttle_cost_per_io = 67
> > >
> > >   ; Try to improve responsiveness when some disks are fully utilised
> > >   osd_op_queue = wpq
> > >   osd_op_queue_cut_off = high
> > >
> > > Hopefully someone else finds this useful.
> > > ___
> > > ceph-users mailing list -- ceph-users@ceph.io
> > > To unsubscribe send an email to ceph-users-le...@ceph.io
> >
> > --
> > Igor Fedotov
> > Ceph Lead Developer
> >
> > Looking for help with your Ceph cluster? Contact us at https://croit.io
> >
> > croit GmbH, Freseniusstr. 31h, 81247 Munich
> > CEO: Martin Verges - VAT-ID: DE310638492
> > Com. register: Amtsgericht Munich HRB 231263
> > Web: https://croit.io | YouTube: https://goo.gl/PGE1Bx
> >
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io