[ceph-users] Re: Cheap M.2 2280 SSD for Ceph

[Varun Priolkar]

Thanks. I'll try to source these, little hard to find.

Regards,

Varun Priolkar

On Mon, 15 Nov, 2021, 10:57 am Ján Senko,  wrote:

> In my experience there is only one good model in size of 2280 for Ceph,
> and that is Micron 7300 MAX
> They go up to 800GB in size. 7400 MAX just came out recently and should be
> even better.
>
> Should be around $200-$250 for the 800GB one.
>
> J.
>
> Ján Senko
> Proton AG
> Lead Storage Engineer
>
> ‐‐‐ Original Message ‐‐‐
>
> On Sunday, November 14th, 2021 at 9:55, Varun Priolkar <
> m...@varunpriolkar.com> wrote:
>
> > Hello,
> >
> > I am a home user trying to build a homelab for Ceph+KVM VMs. I have
> >
> > acquired 3 Lenovo Tiny M920q systems for this purpose. I am trying to
> >
> > select not so expensive M.2 NVMe or M.2 SATA SSDs, but it is hard to
> >
> > find information on which drives would have OK performance with Ceph.
> >
> > My systems have a 1x M.2 2280 slot. The 2.5" SATA slot would be
> >
> > unusable after I replace it with a 10/40Gbe network card.
> >
> > From what I read online, I need to look for drives with PLP. Can
> >
> > someone please recommend some M.2 drives to me? I am fine with buying
> >
> > used and my budget is 200 CAD or around 160 USD per drive. Would
> >
> > prefer capacity close to 1TB as usable capacity would be the same
> >
> > divided by 3.
> >
> > Thank you.
> >
> >
> -
> >
> > Regards,
> >
> > Varun Priolkar
> >
> > ceph-users mailing list -- ceph-users@ceph.io
> >
> > To unsubscribe send an email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Cheap M.2 2280 SSD for Ceph

[Varun Priolkar]

Unfortunately the Seagate drives aren't 2280. That is the max size that
fits my build.

Regards,

Varun Priolkar

On Mon, 15 Nov, 2021, 10:17 am Mario Giammarco, 
wrote:

> You can use also consumer drives considering that is an homelab.
> Otherwise try to find seagate nytro xm1441 or xm1440.
> Mario
>
> Il giorno lun 15 nov 2021 alle ore 14:59 Eneko Lacunza 
> ha scritto:
>
>> Hi Varun,
>>
>> That Kingston DC grade model should work (well enough at least for a
>> home lab), it has PLP.  Note I haven't used that model.
>>
>> Just avoid consumer drives.
>>
>> Cheers
>>
>> El 15/11/21 a las 14:35, Varun Priolkar escribió:
>> > Thank you!
>> >
>> > It is hard for me to find that particular model. Kingston DC1000B is
>> > readily available and not very expensive. Would this work well?
>> >
>> > https://www.kingston.com/us/ssd/dc1000b-data-center-boot-ssd
>> > 
>> >
>> > It is marketed as a boot disk. Would it be OK for a home lab?
>> >
>> > Regards,
>> >
>> > Varun Priolkar
>> >
>> > On Mon, 15 Nov, 2021, 4:19 am Eneko Lacunza, > > > wrote:
>> >
>> > Hi Varun,
>> >
>> > El 14/11/21 a las 9:55, Varun Priolkar escribió:
>> > > Hello,
>> > >
>> > > I am a home user trying to build a homelab for Ceph+KVM VMs. I
>> have
>> > > acquired 3 Lenovo Tiny M920q systems for this purpose. I am
>> > trying to
>> > > select not so expensive M.2 NVMe or M.2 SATA SSDs, but it is hard
>> to
>> > > find information on which drives would have OK performance with
>> > Ceph.
>> > > My systems have a 1x M.2 2280 slot. The 2.5" SATA slot would be
>> > > unusable after I replace it with a 10/40Gbe network card.
>> > >
>> > > >From what I read online, I need to look for drives with PLP. Can
>> > > someone please recommend some M.2 drives to me? I am fine with
>> > buying
>> > > used and my budget is 200 CAD or around 160 USD per drive. Would
>> > > prefer capacity close to 1TB as usable capacity would be the same
>> > > divided by 3.
>> >
>> > Any "datacenter" SSD will do, for example:
>> >
>> >
>> https://ark.intel.com/content/www/us/en/ark/products/134913/intel-ssd-d3s4510-series-960gb-m-2-80mm-sata-6gbs-3d2-tlc.html
>> > <
>> https://ark.intel.com/content/www/us/en/ark/products/134913/intel-ssd-d3s4510-series-960gb-m-2-80mm-sata-6gbs-3d2-tlc.html
>> >
>> >
>> > Cheers
>> >
>> >
>> > Eneko Lacunza
>> > Zuzendari teknikoa | Director técnico
>> > Binovo IT Human Project
>> >
>> > Tel. +34 943 569 206 | https://www.binovo.es > >
>> > Astigarragako Bidea, 2 - 2º izda. Oficina 10-11, 20180 Oiartzun
>> >
>> > https://www.youtube.com/user/CANALBINOVO
>> > 
>> > https://www.linkedin.com/company/37269706/
>> > 
>> >
>> > ___
>> > ceph-users mailing list -- ceph-users@ceph.io
>> > 
>> > To unsubscribe send an email to ceph-users-le...@ceph.io
>> > 
>> >
>>
>>   EnekoLacunza
>>
>> CTO | Zuzendari teknikoa
>>
>> Binovo IT Human Project
>>
>> 943 569 206 
>>
>> elacu...@binovo.es 
>>
>> binovo.es 
>>
>> Astigarragako Bidea, 2 - 2 izda. Oficina 10-11, 20180 Oiartzun
>>
>>
>> youtube 
>> linkedin 
>>
>> ___
>> ceph-users mailing list -- ceph-users@ceph.io
>> To unsubscribe send an email to ceph-users-le...@ceph.io
>>
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Cheap M.2 2280 SSD for Ceph

[Varun Priolkar]

Thanks. I think I will buy 1 to benchmark and buy the rest if it works well.

Regards,

Varun Priolkar

On Mon, 15 Nov, 2021, 8:57 am Eneko Lacunza,  wrote:

> Hi Varun,
>
> That Kingston DC grade model should work (well enough at least for a home
> lab), it has PLP.  Note I haven't used that model.
>
> Just avoid consumer drives.
>
> Cheers
>
> El 15/11/21 a las 14:35, Varun Priolkar escribió:
>
> Thank you!
>
> It is hard for me to find that particular model. Kingston DC1000B is
> readily available and not very expensive. Would this work well?
>
> https://www.kingston.com/us/ssd/dc1000b-data-center-boot-ssd
>
> It is marketed as a boot disk. Would it be OK for a home lab?
>
> Regards,
>
> Varun Priolkar
>
> On Mon, 15 Nov, 2021, 4:19 am Eneko Lacunza,  wrote:
>
>> Hi Varun,
>>
>> El 14/11/21 a las 9:55, Varun Priolkar escribió:
>> > Hello,
>> >
>> > I am a home user trying to build a homelab for Ceph+KVM VMs. I have
>> > acquired 3 Lenovo Tiny M920q systems for this purpose. I am trying to
>> > select not so expensive M.2 NVMe or M.2 SATA SSDs, but it is hard to
>> > find information on which drives would have OK performance with Ceph.
>> > My systems have a 1x M.2 2280 slot. The 2.5" SATA slot would be
>> > unusable after I replace it with a 10/40Gbe network card.
>> >
>> > >From what I read online, I need to look for drives with PLP. Can
>> > someone please recommend some M.2 drives to me? I am fine with buying
>> > used and my budget is 200 CAD or around 160 USD per drive. Would
>> > prefer capacity close to 1TB as usable capacity would be the same
>> > divided by 3.
>>
>> Any "datacenter" SSD will do, for example:
>>
>>
>> https://ark.intel.com/content/www/us/en/ark/products/134913/intel-ssd-d3s4510-series-960gb-m-2-80mm-sata-6gbs-3d2-tlc.html
>>
>> Cheers
>>
>>
>> Eneko Lacunza
>> Zuzendari teknikoa | Director técnico
>> Binovo IT Human Project
>>
>> Tel. +34 943 569 206 | https://www.binovo.es
>> Astigarragako Bidea, 2 - 2º izda. Oficina 10-11, 20180 Oiartzun
>>
>> https://www.youtube.com/user/CANALBINOVO
>> https://www.linkedin.com/company/37269706/
>>
>> ___
>> ceph-users mailing list -- ceph-users@ceph.io
>> To unsubscribe send an email to ceph-users-le...@ceph.io
>>
>
> Eneko Lacunza
>
> CTO | Zuzendari teknikoa
>
> Binovo IT Human Project
> 943 569 206
> elacu...@binovo.es
> binovo.es 
> Astigarragako Bidea, 2 - 2 izda. Oficina 10-11, 20180 Oiartzun
> [image: youtube] 
> [image: linkedin] 
>
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] pg inactive+remapped

[Joffrey]

Hi,

I don't understand why my Global Recovery Event never finish...
I have 3 hosts, all osd and hosts are up. My pools are replica*3

# ceph status
  cluster:
id: 0a77af8a-414c-11ec-908a-005056b4f234
health: HEALTH_WARN
Reduced data availability: 1 pg inactive
Degraded data redundancy: 1/1512 objects degraded (0.066%), 1
pg degraded, 1 pg undersized

  services:
mon: 3 daemons, quorum
preprod-ceph1-mon1,preprod-ceph1-mon3,preprod-ceph1-mon2 (age 2d)
mgr: preprod-ceph1-mon3.ssvflc(active, since 3d), standbys:
preprod-ceph1-mon1.giaate, preprod-ceph1-mon2.yducxr
osd: 12 osds: 12 up (since 28m), 12 in (since 29m); 1 remapped pgs

  data:
pools:   2 pools, 742 pgs
objects: 504 objects, 1.8 GiB
usage:   4.4 TiB used, 87 TiB / 92 TiB avail
pgs: 0.135% pgs not active
 1/1512 objects degraded (0.066%)
 741 active+clean
 1   activating+undersized+degraded+remapped

  io:
client:   0 B/s rd, 51 KiB/s wr, 2 op/s rd, 2 op/s wr

  progress:
Global Recovery Event (39m)
  [===.] (remaining: 3s)
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Fwd: pg inactive+remapped

[Joffrey]

# ceph pg 4.a3 query
{
"snap_trimq": "[]",
"snap_trimq_len": 0,
"state": "activating+undersized+degraded+remapped",
"epoch": 640,
"up": [
7,
0,
8
],
"acting": [
7,
8
],
"async_recovery_targets": [
"0"
],
"acting_recovery_backfill": [
"0",
"7",
"8"
],
"info": {
"pgid": "4.a3",
"last_update": "373'783",
"last_complete": "373'783",
"log_tail": "0'0",
"last_user_version": 783,
"last_backfill": "MAX",
"purged_snaps": [],
"history": {
"epoch_created": 281,
"epoch_pool_created": 277,
"last_epoch_started": 461,
"last_interval_started": 460,
"last_epoch_clean": 437,
"last_interval_clean": 436,
"last_epoch_split": 281,
"last_epoch_marked_full": 0,
"same_up_since": 532,
"same_interval_since": 533,
"same_primary_since": 382,
"last_scrub": "373'783",
"last_scrub_stamp": "2021-11-16T09:27:52.382799+",
"last_deep_scrub": "301'93",
"last_deep_scrub_stamp": "2021-11-12T01:22:57.943041+",
"last_clean_scrub_stamp": "2021-11-16T09:27:52.382799+",
"prior_readable_until_ub": 13.203500275
},
"stats": {
"version": "373'783",
"reported_seq": 1321,
"reported_epoch": 640,
"state": "activating+undersized+degraded+remapped",
"last_fresh": "2021-11-16T10:58:15.112866+",
"last_change": "2021-11-16T10:43:18.311446+",
"last_active": "2021-11-16T10:43:16.294751+",
"last_peered": "2021-11-16T10:43:15.363400+",
"last_clean": "2021-11-16T10:33:29.421037+",
"last_became_active": "2021-11-16T10:33:31.489138+",
"last_became_peered": "2021-11-16T10:33:31.489138+",
"last_unstale": "2021-11-16T10:58:15.112866+",
"last_undegraded": "2021-11-16T10:43:18.298347+",
"last_fullsized": "2021-11-16T10:43:18.295033+",
"mapping_epoch": 533,
"log_start": "0'0",
"ondisk_log_start": "0'0",
"created": 281,
"last_epoch_clean": 437,
"parent": "0.0",
"parent_split_bits": 8,
"last_scrub": "373'783",
"last_scrub_stamp": "2021-11-16T09:27:52.382799+",
"last_deep_scrub": "301'93",
"last_deep_scrub_stamp": "2021-11-12T01:22:57.943041+",
"last_clean_scrub_stamp": "2021-11-16T09:27:52.382799+",
"log_size": 783,
"ondisk_log_size": 783,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"stat_sum": {
"num_bytes": 4169728,
"num_objects": 1,
"num_object_clones": 0,
"num_object_copies": 3,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 0,
"num_objects_degraded": 1,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 1,
"num_whiteouts": 0,
"num_read": 782,
"num_read_kb": 0,
"num_write": 783,
"num_write_kb": 34408,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 3,
"num_bytes_recovered": 12435456,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
7,
0,
8
],
"acting": [
7,
8
],
"avail_no_missing": [
   

[ceph-users] Re: mons fail as soon as I attempt to mount

[胡 玮文]

Hi Jeremy. Since you say the mons fail, could you share the logs of the failing 
mons? It is hard to diagnostic with little information.

发件人: Jeremy Hansen
发送时间: 2021年11月16日 19:27
收件人: ceph-users
主题: [ceph-users] Re: mons fail as soon as I attempt to mount

No ideas on this? It has me stumped. I’m going to revert my kernel upgrades if 
there isn’t a more logical explanation.

Thanks

> On Monday, Nov 15, 2021 at 2:33 AM, Jeremy Hansen  (mailto:jer...@skidrow.la)> wrote:
> This post references a kernel issue:
>
> https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/6JDAPD5IR46JI6R6YGWQORDJTZ5Z2FIU/
>
> I recently updated my ceph nodes to 5.15.1. Could this be my issue?
>
> -jeremy
>
>
>
>
> > On Sunday, Nov 14, 2021 at 4:37 PM, Jeremy Hansen  > (mailto:jer...@skidrow.la)> wrote:
> > I’m trying to mount a cephfs volume from a new machine. For some reason, it 
> > looks like all the mons fail when attempting to mount:
> >
> > [root@btc04 ~]# mount -t ceph :/ /mnt/ceph -o name=btc
> > mount error: no mds server is up or the cluster is laggy
> > [root@btc04 ~]# rpm -qa | grep ceph
> > python3-cephfs-16.2.4-0.el8.x86_64
> > ceph-common-16.2.4-0.el8.x86_64
> > cephadm-16.2.4-0.el8.noarch
> > python3-ceph-argparse-16.2.4-0.el8.x86_64
> > libcephfs2-16.2.4-0.el8.x86_64
> > python3-ceph-common-16.2.4-0.el8.x86_64
> >
> >
> > [ 51.105212] libceph: loaded (mon/osd proto 15/24)
> > [ 51.145564] ceph: loaded (mds proto 32)
> > [ 51.164266] libceph: mon3 (1)192.168.30.14:6789 session established
> > [ 70.199453] libceph: mon3 (1)192.168.30.14:6789 socket closed (con state 
> > OPEN)
> > [ 70.199464] libceph: mon3 (1)192.168.30.14:6789 session lost, hunting for 
> > new mon
> > [ 70.204400] libceph: mon0 (1)192.168.30.11:6789 session established
> > [ 70.771652] libceph: mon0 (1)192.168.30.11:6789 socket closed (con state 
> > OPEN)
> > [ 70.771670] libceph: mon0 (1)192.168.30.11:6789 session lost, hunting for 
> > new mon
> > [ 70.774588] libceph: mon4 (1)192.168.30.15:6789 session established
> > [ 71.234037] libceph: mon4 (1)192.168.30.15:6789 socket closed (con state 
> > OPEN)
> > [ 71.234055] libceph: mon4 (1)192.168.30.15:6789 session lost, hunting for 
> > new mon
> > [ 77.904722] libceph: mon3 (1)192.168.30.14:6789 socket closed (con state 
> > V1_BANNER)
> > [ 78.160614] libceph: mon3 (1)192.168.30.14:6789 socket closed (con state 
> > V1_BANNER)
> > [ 78.664602] libceph: mon3 (1)192.168.30.14:6789 socket closed (con state 
> > V1_BANNER)
> > [ 79.824787] libceph: mon3 (1)192.168.30.14:6789 socket closed (con state 
> > V1_BANNER)
> > [ 81.808526] libceph: mon3 (1)192.168.30.14:6789 socket closed (con state 
> > V1_BANNER)
> > [ 85.840430] libceph: mon3 (1)192.168.30.14:6789 socket closed (con state 
> > V1_BANNER)
> >
> >
> > Not really sure why…
> >
> > [ceph: root@cn01 /]# ceph osd pool get cephfs.btc.data all
> > size: 4
> > min_size: 2
> > pg_num: 32
> > pgp_num: 32
> > crush_rule: replicated_rule
> > hashpspool: true
> > nodelete: false
> > nopgchange: false
> > nosizechange: false
> > write_fadvise_dontneed: false
> > noscrub: false
> > nodeep-scrub: false
> > use_gmt_hitset: 1
> > fast_read: 0
> > pg_autoscale_mode: on
> > [ceph: root@cn01 /]# ceph osd pool get cephfs.btc.meta all
> > size: 4
> > min_size: 2
> > pg_num: 32
> > pgp_num: 32
> > crush_rule: replicated_rule
> > hashpspool: true
> > nodelete: false
> > nopgchange: false
> > nosizechange: false
> > write_fadvise_dontneed: false
> > noscrub: false
> > nodeep-scrub: false
> > use_gmt_hitset: 1
> > fast_read: 0
> > recovery_priority: 5
> > pg_autoscale_mode: on
> > pg_num_min: 16
> > pg_autoscale_bias: 4
> >
> >
> > The cluster becomes unhealthy but then clears shortly after the client 
> > times out.
> >
> > [ceph: root@cn01 /]# ceph -s
> > cluster:
> > id: bfa2ad58-c049-11eb-9098-3c8cf8ed728d
> > health: HEALTH_OK
> >
> > services:
> > mon: 5 daemons, quorum cn05,cn02,cn03,cn04,cn01 (age 6m)
> > mgr: cn05.vpuwau(active, since 6d), standbys: cn02.arszct
> > mds: 2/2 daemons up, 4 standby
> > osd: 35 osds: 35 up (since 2d), 35 in (since 6d)
> >
> > data:
> > volumes: 2/2 healthy
> > pools: 6 pools, 289 pgs
> > objects: 6.73M objects, 4.3 TiB
> > usage: 17 TiB used, 108 TiB / 126 TiB avail
> > pgs: 289 active+clean
> >
> > io:
> > client: 0 B/s rd, 105 KiB/s wr, 2 op/s rd, 13 op/s wr
> >
> >
> >
> > -jeremy
> >
> >
> >

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Fwd: pg inactive+remapped

[胡 玮文]

> But my log file for osd 7 is empty

Is it deployed by cephadm? If so, you can try “sudo cephadm logs --name osd.7”, 
which is a wrapper around journalctl.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Fwd: pg inactive+remapped

[Joffrey]

Ok, restart my OSD 0 fix the problem ! Thanks you

Le mar. 16 nov. 2021 à 13:32, Stefan Kooman  a écrit :

> On 11/16/21 13:17, Joffrey wrote:
>
> >  "peer_info": [
> >  {
> >  "peer": "0",
> >  "pgid": "4.a3",
> >  "last_update": "373'783",
> >  "last_complete": "0'0",
> >  "log_tail": "0'0",
> >  "last_user_version": 0,
> >  "last_backfill": "MAX",
> >  "purged_snaps": [],
> >  "history": {
> >  "epoch_created": 0,
> >  "epoch_pool_created": 0,
> >  "last_epoch_started": 0,
> >  "last_interval_started": 0,
> >  "last_epoch_clean": 0,
> >  "last_interval_clean": 0,
> >  "last_epoch_split": 0,
> >  "last_epoch_marked_full": 0,
> >  "same_up_since": 0,
> >  "same_interval_since": 0,
> >  "same_primary_since": 0,
> >  "last_scrub": "0'0",
> >  "last_scrub_stamp": "0.00",
> >  "last_deep_scrub": "0'0",
> >  "last_deep_scrub_stamp": "0.00",
> >  "last_clean_scrub_stamp": "0.00",
> >  "prior_readable_until_ub": 0
> >  },
> >  "stats": {
> >  "version": "0'0",
> >  "reported_seq": 0,
> >  "reported_epoch": 0,
> >  "state": "unknown",
> >  "last_fresh": "0.00",
> >  "last_change": "0.00",
> >  "last_active": "0.00",
> >  "last_peered": "0.00",
> >  "last_clean": "0.00",
> >  "last_became_active": "0.00",
> >  "last_became_peered": "0.00",
> >  "last_unstale": "0.00",
> >  "last_undegraded": "0.00",
> >  "last_fullsized": "0.00",
> >  "mapping_epoch": 0,
> >  "log_start": "0'0",
> >  "ondisk_log_start": "0'0",
> >  "created": 0,
> >  "last_epoch_clean": 0,
> >  "parent": "0.0",
> >  "parent_split_bits": 0,
> >  "last_scrub": "0'0",
> >  "last_scrub_stamp": "0.00",
> >  "last_deep_scrub": "0'0",
> >  "last_deep_scrub_stamp": "0.00",
> >  "last_clean_scrub_stamp": "0.00",
> >  "log_size": 0,
> >  "ondisk_log_size": 0,
> >  "stats_invalid": false,
> >  "dirty_stats_invalid": false,
> >  "omap_stats_invalid": false,
> >  "hitset_stats_invalid": false,
> >  "hitset_bytes_stats_invalid": false,
> >  "pin_stats_invalid": false,
> >  "manifest_stats_invalid": false,
> >  "snaptrimq_len": 0,
> >  "stat_sum": {
> >  "num_bytes": 0,
> >  "num_objects": 0,
> >  "num_object_clones": 0,
> >  "num_object_copies": 0,
> >  "num_objects_missing_on_primary": 0,
> >  "num_objects_missing": 1,
> >  "num_objects_degraded": 0,
> >  "num_objects_misplaced": 0,
> >  "num_objects_unfound": 0,
> >  "num_objects_dirty": 0,
> >  "num_whiteouts": 0,
> >  "num_read": 0,
> >  "num_read_kb": 0,
> >  "num_write": 0,
> >  "num_write_kb": 0,
> >  "num_scrub_errors": 0,
> >  "num_shallow_scrub_errors": 0,
> >  "num_deep_scrub_errors": 0,
> >  "num_objects_recovered": 0,
> >  "num_bytes_recovered": 0,
> >  "num_keys_recovered": 0,
> >  "num_objects_omap": 0,
> >  "num_objects_hit_set_archive": 0,
> >  "num_bytes_hit_set_archive": 0,
> >  "num_flush": 0,
> >  "num_flush_kb": 0,
> >  "num_evict": 0,
> >  "num_evict_kb": 0,
> >  "num_promote": 0,
> >  "num_flush_mode_high": 0,
> >  "num_flush_mode_low": 0,
> >  "num_evict_mode_some": 0,
> >  "num_evict_mode_full": 0,
> >  "num_objects_pinned": 0,
> >  "num_legacy_snapsets": 0,
> >  "num_large_omap_objects": 0,
> >  "num_objects_manifest": 0,
> >  "num_omap_bytes": 0,
> >  "num_omap_keys":

[ceph-users] Re: OSDs get killed by OOM when other host goes down

[Mark Nelson]

Yeah, if it's not memory reported by the mempools, that means it's 
something we aren't tracking.  Perhaps temporary allocations in some 
dark corner of the code, or possibly rocksdb (though 38GB of ram is 
obviously excessive).  heap stats are a good idea.  it's possible if 
neither the heap stats nor the mempool stats are helpful (and if debug 
bluestore = 5 and debug prioritycache =5 doesn't indicate any obvious 
problems with autotuning code), it may require valgrind or some other 
method to figure out where the memory is going.  If the memory is 
growing rapidly it's possible that wallclock profiling may help if you 
can catch where the allocations are being made.



Mark


On 11/16/21 2:42 PM, Josh Baergen wrote:

Hi Marius,


However, docker stats reports 38GB for that container.
There is a huge gap between what RAM is being used by the container what ceph 
daemon osd.xxx dump_mempools reports.

Take a look at "ceph daemon osd.XX heap stats" and see what it says.
You might try "ceph daemon osd.XX heap release"; I didn't think that
was supposed to be necessary with Bluestore, though. This is reaching
the end of the sort of problems I know how to track down, though, so
maybe others have some ideas.


How can I check if trim happens?

I'm not sure how to dig into this, but if your "up" count = your "in"
count in "ceph -s", it should be trimming.

Josh
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io



___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] cephadm / ceph orch : indefinite hang adding hosts to new cluster

[Lincoln Bryant]

Greetings list,

We have a new Ceph cluster we are trying to deploy on EL8 (CentOS Stream) using 
cephadm (+podman), targeting Pacific.

We are successfully able to bootstrap the first host, but attempting to add any 
additional hosts hangs indefinitely. We have confirmed that we are able to SSH 
from the first host to subsequent hosts using the key generated by Ceph.

This is the only logging output we see:
# cephadm shell -- ceph orch host add kvm-mon02 192.168.7.12
Inferring fsid 826b9b36-4729-11ec-99f0-c81f66d05a38
Using recent ceph image 
quay.io/ceph/ceph@sha256:a2c23b6942f7fbc1e15d8cfacd6655a681fe0e44f288e4a158db22030b8d58e3

This command hangs indefinitely until killed via podman kill​.

Inspecting the host we're trying to add, we see that Ceph has launched a python 
process:
root3604  0.0  0.0 164128  6316 ?S16:36   0:00  |   \_ 
sshd: root@notty
root3605  0.0  0.0  31976  8752 ?Ss   16:36   0:00  |   \_ 
python3 -c import sys;exec(eval(sys.stdin.readline()))

Inside of the mgr container, we see 2 SSH connections:
ceph 186  0.0  0.0  44076  6676 ?S22:31   0:00  \_ ssh -C 
-F /tmp/cephadm-conf-s0b8c90d -i /tmp/cephadm-identity-8ku7ib6b 
root@192.168.7.13 python3 -c "import sys;exec(eval(sys.stdin.readline()))"
ceph 211  0.0  0.0  44076  6716 ?S22:36   0:00  \_ ssh -C 
-F /tmp/cephadm-conf-s0b8c90d -i /tmp/cephadm-identity-8ku7ib6b 
root@192.168.7.12 python3 -c "import sys;exec(eval(sys.stdin.readline()))"

where 192.168.1.13 is the IP of the first host in the cluster (which has 
succesfully bootstrapped and is running mgr, mon, and so on), and 196.168.1.12 
is the host we are trying to unsuccessfully add.

The mgr logs show no particularly interesting except for:
debug 2021-11-16T22:39:03.570+ 7fb6e4914700  0 [progress WARNING root] 
complete: ev de058df7-b54a-4429-933a-99abe7796715 does not exist
debug 2021-11-16T22:39:03.571+ 7fb6e4914700  0 [progress WARNING root] 
complete: ev 61fe4998-4ef4-4640-8a13-2b0928da737f does not exist
debug 2021-11-16T22:39:03.571+ 7fb6e4914700  0 [progress WARNING root] 
complete: ev 2323b586-5262-497e-b318-42702c0dc3dc does not exist
debug 2021-11-16T22:39:03.572+ 7fb6e4914700  0 [progress WARNING root] 
complete: ev f882757b-e03a-4798-8fae-4d55e2d1f531 does not exist
debug 2021-11-16T22:39:03.572+ 7fb6e4914700  0 [progress WARNING root] 
complete: ev 3d35e91f-b475-4dbc-a040-65658e82fe67 does not exist
debug 2021-11-16T22:39:03.572+ 7fb6e4914700  0 [progress WARNING root] 
complete: ev 694ed4e6-b741-4bf3-9f7f-b11c549aca87 does not exist
debug 2021-11-16T22:39:03.573+ 7fb6e4914700  0 [progress WARNING root] 
complete: ev 1ce718f4-a23b-4a27-8f7c-bc2610340403 does not exist

We've tried purging/reinstalling several times to no avail, and also tried 
swapping which host was used as the initial bootstrap mon and so on. I've also 
tried the option to disable the monitoring stack and that did not help either.

In any case, we are not sure how to proceed from here. Is there anything we can 
do to turn up logging verbosity, or other things to check? I've tried to find 
the ceph orch​ source code to try to understand what may be happening, but I'm 
not sure where to look.

Thanks,
Lincoln Bryant
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] how to list ceph file size on ubuntu 20.04

[zxcs]

Hi, 

I want to list cephfs directory size on ubuntu 20.04, but when I use ls -alh 
[directory] ,it shows the number of files and directorys under this 
directory(it only count number not size) , i remember when i use ls -alh 
[directory] on ubuntu 16.04, it will shows the size of this directory (include 
it sub directory). i know du -sh can list the directory size, but it very slow 
as our ceph directory has tons of small files.

Would anyone please help to shed some light here? Thanks a ton!


Thanks,
Xiong 
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: how to list ceph file size on ubuntu 20.04

[胡 玮文]

There is a rbytes mount option [1]. Besides, you can use “getfattr -n 
ceph.dir.rbytes /path/in/cephfs”

[1]: https://docs.ceph.com/en/latest/man/8/mount.ceph/#advanced

Weiwen Hu

在 2021年11月17日，10:26，zxcs  写道：

Hi,

I want to list cephfs directory size on ubuntu 20.04, but when I use ls -alh 
[directory] ,it shows the number of files and directorys under this 
directory(it only count number not size) , i remember when i use ls -alh 
[directory] on ubuntu 16.04, it will shows the size of this directory (include 
it sub directory). i know du -sh can list the directory size, but it very slow 
as our ceph directory has tons of small files.

Would anyone please help to shed some light here? Thanks a ton!


Thanks,
Xiong
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: OSDs get killed by OOM when other host goes down

[Marius Leustean]

I did the heap release. It did not free the memory right away, but after
I restarted the OSD, it started with "just" 10GB RAM.
After ~12h the container reports again 30+ GB RAM.
To mention that I lowered pg_log values too:

osd_min_pg_log_entries = 100
osd_max_pg_log_entries = 500
osd_target_pg_log_entries_per_osd = 3

On Tue, Nov 16, 2021 at 10:49 PM Mark Nelson  wrote:

> Yeah, if it's not memory reported by the mempools, that means it's
> something we aren't tracking.  Perhaps temporary allocations in some
> dark corner of the code, or possibly rocksdb (though 38GB of ram is
> obviously excessive).  heap stats are a good idea.  it's possible if
> neither the heap stats nor the mempool stats are helpful (and if debug
> bluestore = 5 and debug prioritycache =5 doesn't indicate any obvious
> problems with autotuning code), it may require valgrind or some other
> method to figure out where the memory is going.  If the memory is
> growing rapidly it's possible that wallclock profiling may help if you
> can catch where the allocations are being made.
>
>
> Mark
>
>
> On 11/16/21 2:42 PM, Josh Baergen wrote:
> > Hi Marius,
> >
> >> However, docker stats reports 38GB for that container.
> >> There is a huge gap between what RAM is being used by the container
> what ceph daemon osd.xxx dump_mempools reports.
> > Take a look at "ceph daemon osd.XX heap stats" and see what it says.
> > You might try "ceph daemon osd.XX heap release"; I didn't think that
> > was supposed to be necessary with Bluestore, though. This is reaching
> > the end of the sort of problems I know how to track down, though, so
> > maybe others have some ideas.
> >
> >> How can I check if trim happens?
> > I'm not sure how to dig into this, but if your "up" count = your "in"
> > count in "ceph -s", it should be trimming.
> >
> > Josh
> > ___
> > ceph-users mailing list -- ceph-users@ceph.io
> > To unsubscribe send an email to ceph-users-le...@ceph.io
> >
>
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Adding a RGW realm to a single cephadm-managed ceph cluster

[Eugen Block]

Hi,

I retried it with your steps, they worked for me.
The first non-default realm runs on the default port 80 on two RGWs,  
the second realm is on the same hosts on port 8081 as configured in  
the spec file. The period commit ran successfully, though, so maybe  
there's something wrong with your setup, it's hard to tell.


Does the single realm config work as expected? The rgw container  
starts successfully?



Zitat von J-P Methot :

I've rebuilt my rgw several times since our last emails and every  
time I get stuck more or less to the same point, so let me share  
with you all what I am doing step by step, along with the specs I use.


1. I create a default RGW  container and put it on my second  
monitor. Default configs are set


ceph orch apply rgw staging staging --placement="1 ceph-monitor2"

2. I create a default realm

radosgw-admin realm create --rgw-realm=staging

3. I create a second realm

radosgw-admin realm create --rgw-realm=test

4. I create a master zonegroup for my second realm

radosgw-admin zonegroup create --rgw-zonegroup=test --rgw-realm=test  
--realm-id=aa7d5bdd-afd7-449e-8f5b-c055877d7fef


5. I create a master zone for my second realm

radosgw-admin zone create --rgw-zonegroup=test --rgw-realm=test  
--realm-id=aa7d5bdd-afd7-449e-8f5b-c055877d7fef --master  
--rgw-zone=test


6. I try to commit my second realm (it fails, usually with a can't  
init error)


radosgw-admin period update --commit  
--realm-id=aa7d5bdd-afd7-449e-8f5b-c055877d7fef


7.I try to create my container with the following spec file

ceph orch apply -i multi-realm.spec

cat ceph orch apply -i multi-realm.spec

service_type: rgw
service_id: test.test
placement:
   label: rgw-aux
   count: 1
spec:
    rgw_realm: test
    rgw_zone: test
    ssl: false
    rgw_frontend_port: 


I've set Cephadm to output to a logfile and that logfile tells me:

Cannot find zone id=35ea43e5-e0d2-4ebe-81d6-9274231bd542 (name=test)

And that is even though the zone exists and is in the realm this  
gateway is set to serve. Is the commit supposed to work before or  
after the gateway is setup?



On 11/16/21 2:45 AM, Eugen Block wrote:

Hi,


Couldn't init storage provider (RADOS)


I usually see this when my rgw config is wrong, can you share your  
rgw spec(s)?



Zitat von J-P Methot :

After searching through logs and figuring out how cephadm works,  
I've figured out that when cephadm tries to create the new systemd  
service and launch the new RGW container, it fails with this error :


Couldn't init storage provider (RADOS)

Systemd then complains that the service doesn't exist :

ceph-04c5d4a4-8815-45fb-b97f-027252d1a...@rgw.test.test.ceph-monitor1.twltpl.service: Main process exited, code=exited,  
status=5/NOTINSTALLED


I can't seem to find further logs anywhere regarding why the  
storage provider fails to init. Googling indicate issues with the  
number of pgs in past versions, but I believe it shouldn't be an  
issue here, since the amount of PG auto-adjusts. How could I find  
out why this happens?


On 11/15/21 11:02 AM, J-P Methot wrote:
Yes, I'm trying to add a RGW container on a second port on the  
same server. For example, I do :


ceph orch apply rgw test test  
--placement="ceph-monitor1:[10.50.47.3:]"


and this results in :

ceph orch ls

NAME RUNNING  REFRESHED  AGE  
PLACEMENT  IMAGE  
NAME   IMAGE ID


rgw.test.test    0/1  2s ago 5s  
ceph-monitor1:[10.50.47.3:] docker.io/ceph/ceph:v15 



the image and container ID being unknown is making me scratch my  
head. A look in the log files show this:


2021-11-15 10:50:12,253 INFO Deploy daemon  
rgw.test.test.ceph-monitor1.rtoiwh ...
2021-11-15 10:50:12,254 DEBUG Running command: /usr/bin/docker  
run --rm --ipc=host --net=host --entrypoint stat -e  
CONTAINER_IMAGE=docker.io/ceph/ceph:v15 -e  
NODE_NAME=ceph-monitor1 docker.io/ceph/ceph:v15 -c %u %g  
/var/lib/ceph

2021-11-15 10:50:12,452 DEBUG stat: stdout 167 167
2021-11-15 10:50:12,525 DEBUG Running command: install -d -m0770  
-o 167 -g 167 /var/run/ceph/04c5d4a4-8815-45fb-b97f-027252d1aea5

2021-11-15 10:50:12,534 DEBUG Running command: systemctl daemon-reload
2021-11-15 10:50:12,869 DEBUG Running command: systemctl stop  
ceph-04c5d4a4-8815-45fb-b97f-027252d1a...@rgw.test.test.ceph-monitor1.rtoiwh 2021-11-15 10:50:12,879 DEBUG Running command: systemctl reset-failed  
ceph-04c5d4a4-8815-45fb-b97f-027252d1a...@rgw.test.test.ceph-monitor1.rtoiwh
2021-11-15 10:50:12,884 DEBUG systemctl: stderr Failed to reset  
failed state of unit  
ceph-04c5d4a4-8815-45fb-b97f-027252d1a...@rgw.test.test.ceph-monitor1.rtoiwh.service: Unit ceph-04c5d4a4-8815-45fb-b97f-027252d1a...@rgw.test.test.ceph-monitor1.rtoiwh.service not  
loaded


journalctl -xe shows the service entered failed state, without  
any real useful information


Nov 15 10:50:24 ceph-monitor1 systemd[1]:  
ceph-04c5d4a4-8815-45fb-b97f-027252d1a...@rgw.test.test.cep

[ceph-users] Re: Why you might want packages not containers for Ceph deployments

[Peter Lieven]

> Am 09.11.2021 um 00:01 schrieb Igor Fedotov :
> 
> Hi folks,
> 
> having a LTS release cycle could be a great topic for upcoming "Ceph User + 
> Dev Monthly meeting".
> 
> The first one is scheduled  on November 18, 2021, 14:00-15:00 UTC
> 
> https://pad.ceph.com/p/ceph-user-dev-monthly-minutes
> 
> Any volunteers to extend the agenda and advocate the idea?

Hi Igor,

do you still think we can add the LTS topic to the agenda? I will attend 
tomorrow and can try to advocate it.

Best,
Peter

> 
> Thanks,
> Igor
> 
>> On 11/8/2021 3:21 PM, Frank Schilder wrote:
>> Hi all,
>> 
>> I followed this thread with great interest and would like to add my 
>> opinion/experience/wishes as well.
>> 
>> I believe the question packages versus containers needs a bit more context 
>> to be really meaningful. This was already mentioned several times with 
>> regards to documentation. I see the following three topics tightly connected 
>> (my opinion/answers included):
>> 
>> 1. Distribution: Packages are compulsory, containers are optional.
>> 2. Deployment: Ceph adm (yet another deployment framework) and ceph (the 
>> actual storage system) should be strictly different projects.
>> 3. Release cycles: The release cadence is way too fast, I very much miss a 
>> ceph LTS branch with at least 10 years back-port support.
>> 
>> These are my short answers/wishes/expectations in this context. I will add 
>> below some more reasoning as optional reading (warning: wall of text ahead).
>> 
>> 
>> 1. Distribution
>> -
>> 
>> I don't think the question is about packages versus containers, because even 
>> if a distribution should decide not to package ceph any more, other 
>> distributors certainly will and the user community just moves away from 
>> distributions without ceph packages. In addition, unless Rad Hat plans to 
>> move to a source-only container where I run the good old configure - make - 
>> make install, it will be package based any ways, so packages are there to 
>> stay.
>> 
>> Therefore, the way I understand this question is about ceph-adm versus other 
>> deployment methods. Here, I think the push to a container-based ceph-adm 
>> only deployment is unlikely to become the no. 1 choice for everyone for good 
>> reasons already mentioned in earlier messages. In addition, I also believe 
>> that development of a general deployment tool is currently not sustainable 
>> as was mentioned by another user. My reasons for this are given in the next 
>> section.
>> 
>> 
>> 2. Deployment
>> -
>> 
>> In my opinion, it is really important to distinguish three components of any 
>> open-source project: development (release cycles), distribution and 
>> deployment. Following the good old philosophy that every tool does exactly 
>> one job and does it well, each of these components are separate projects, 
>> because they correspond to different tools.
>> 
>> This implies immediately that ceph documentation should not contain 
>> documentation about packaging and deployment tools. Each of these ought to 
>> be strictly separate. If I have a low-level problem with ceph and go to the 
>> ceph documentation, I do not want to see ceph-adm commands. Ceph 
>> documentation should be about ceph (the storage system) only. Such a mix-up 
>> is leading to problems and there were already ceph-user cases where people 
>> could not use the documentation for trouble shooting, because it showed 
>> ceph-adm commands but their cluster was not ceph-adm deployed.
>> 
>> In this context, I would prefer if there was a separate ceph-adm-users list 
>> so that ceph-users can focus on actual ceph problems again.
>> 
>> Now to the point that ceph-adm might be an un-sustainable project. Although 
>> at a first glance the idea of a generic deployment tool that solves all 
>> problems with a single command might look appealing, it is likely doomed to 
>> fail for a simple reason that was already indicated in an earlier message: 
>> ceph deployment is subject to a complexity paradox. Ceph has a very large 
>> configuration space and implementing and using a generic tool that covers 
>> and understands this configuration space is more complex than deploying any 
>> specific ceph cluster, each of which uses only a tiny subset of the entire 
>> configuration space.
>> 
>> In other words: deploying a specific ceph cluster is actually not that 
>> difficult.
>> 
>> Designing a - and dimensioning all components of a ceph cluster is difficult 
>> and none of the current deployment tools help here. There is not even a 
>> check for suitable hardware. In addition, technology is moving fast and 
>> adapting a generic tool to new developments in time seems a hopeless task. 
>> For example, when will ceph-adm natively support collocated lvm OSDs with 
>> dm_cache devices? Is it even worth trying to incorporate this?
>> 
>> My wish would be to keep the ceph project clean of any deployment tasks. In 
>> my opinion, the basic ceph tooling is already doing