[ceph-users] Re: Empty /var/lib/ceph/osd/ceph-$osd after reboot

[Isaiah Tang Yue Shun]

Hi,

That's exactly what I meant. I did try to activate but it fails and the problem 
is due to bluestore label gone after reboot, which in the end caused the 
activation fail and /var/lib/ceph/osd/ceph-$osd​ empty

Upgrading from Pacific to Quincy solved the problem for me.

Regards,

Isaiah Tang


From: Konstantin Shalygin 
Sent: Tuesday, December 27, 2022 3:57 PM
To: Isaiah Tang Yue Shun 
Cc: ceph-users@ceph.io 
Subject: Re: [ceph-users] Empty /var/lib/ceph/osd/ceph-$osd after reboot

Hi,

ceph-volume lvm activate --all


k

On 21 Dec 2022, at 13:53, Isaiah Tang Yue Shun  wrote:

From what I understand, after creating an OSD using  "ceph-volume lvm create", 
we will do a "ceph-volume lvm activate" so that the systemd is created.

However, I found that after rebooting a host, some OSDs in the host will have 
empty /var/lib/ceph/osd/ceph-$osd
And I am not able to recover from there. Am I missing any steps?

I am running Oracle Linux 8.7, pacific release, ol8.

Thanks in advance.

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Does Replica Count Affect Tell Bench Result or Not?

[hosseinz8...@yahoo.com]

 Thanks AnthonyI have a cluster with QLC SSD disks (Samsung QVO 860). The 
cluster works for 2 year. Now all OSDs return 12 iops when running tell bench 
which is very slow. But I Buy new QVO disks yesterday, and I added this new 
disk to cluster. For the first 1 hour, I got 100 iops from this new OSD. But 
after 1 Hour, this new disk (OSD) returns to iops 12 again as the same as other 
OLD OSDs.I can not imagine what happening?!!
On Tuesday, December 27, 2022 at 12:18:07 AM GMT+3:30, Anthony D'Atri 
 wrote:  
 
 My understanding is that when you ask an OSD to bench (via the admin socket), 
only that OSD executes, there is no replication.  Replication is a function of 
PGs.

Thus, this is a narrowly-focused tool with both unique advantages and 
disadvantages.



> On Dec 26, 2022, at 12:47 PM, hosseinz8...@yahoo.com wrote:
> 
> Hi experts,I want to know, when I execute ceph tell osd.x bench command, is 
> replica 3 considered in the bench or not? I mean, for example in case of 
> replica 3, when I executing tell bench command, replica 1 of bench data write 
> to osd.x, replica 2 write to osd.y and replica 3 write to osd.z? If this is 
> true, it means that I can not get benchmark of only one of my OSD in the 
> cluster because the IOPS and throughput of 2 other for example slow OSDs will 
> affect the result of tell bench command for my target OSD.Is that true?
> Thanks in advance.
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
  
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Does Replica Count Affect Tell Bench Result or Not?

[Anthony D'Atri]

I do not have *direct* experience with that model but can share some 
speculations:

That is a *consumer* model, known as “client” in the SSD industry.  It’s also 
QLC. It’s optimized for PB/$.

I suspect that at least one of several things is going on.

* Cliffing: Client SSDs are architected for an intermittent duty cycle, not 
24x7, and for various reasons sustained performance can drop after a certain 
amount of sustained time or data
* How full are the OSDs? `ceph osd df` Drives like this appear to allocate an 
SLC cache/staging area that decreases as the drive fills up — a design decision 
that I’ve never fully understood.  I suspect that is at play here.  This 
article touches on this phenomenon:

https://www.tweaktown.com/articles/8819/samsung-860-qvo-ssd-review-rip-consumer-hdds/index.html
 


* Are you using a RAID HBA?  If so, do you have a VD wrapped around each drive, 
or do you have HBA caching enabled?  
* It also likely does not feature power loss protection (PLP), and caching may 
be hurting you

https://www.mail-archive.com/ceph-users@ceph.io/msg04704.html 


https://yourcmc.ru/wiki/index.php?title=Ceph_performance&mobileaction=toggle_view_desktop#Drive_cache_is_slowing_you_down
 



* Have you checked SMART counters for projected lifetime remaining, used vs 
remaining spare blocks, etc?  Please feel free to collect the output from 
`smartctl -a` for all drives, post the text file somewhere, and send a link to 
it.

> On Dec 27, 2022, at 4:40 AM, hosseinz8...@yahoo.com wrote:
> 
> Thanks Anthony
> I have a cluster with QLC SSD disks (Samsung QVO 860). The cluster works for 
> 2 year. Now all OSDs return 12 iops when running tell bench which is very 
> slow. But I Buy new QVO disks yesterday, and I added this new disk to 
> cluster. For the first 1 hour, I got 100 iops from this new OSD. But after 1 
> Hour, this new disk (OSD) returns to iops 12 again as the same as other OLD 
> OSDs.
> I can not imagine what happening?!!
> 
> On Tuesday, December 27, 2022 at 12:18:07 AM GMT+3:30, Anthony D'Atri 
>  wrote:
> 
> 
> My understanding is that when you ask an OSD to bench (via the admin socket), 
> only that OSD executes, there is no replication.  Replication is a function 
> of PGs.
> 
> Thus, this is a narrowly-focused tool with both unique advantages and 
> disadvantages.
> 
> 
> 
> > On Dec 26, 2022, at 12:47 PM, hosseinz8...@yahoo.com 
> >  wrote:
> > 
> > Hi experts,I want to know, when I execute ceph tell osd.x bench command, is 
> > replica 3 considered in the bench or not? I mean, for example in case of 
> > replica 3, when I executing tell bench command, replica 1 of bench data 
> > write to osd.x, replica 2 write to osd.y and replica 3 write to osd.z? If 
> > this is true, it means that I can not get benchmark of only one of my OSD 
> > in the cluster because the IOPS and throughput of 2 other for example slow 
> > OSDs will affect the result of tell bench command for my target OSD.Is that 
> > true?
> > Thanks in advance.
> 
> > ___
> > ceph-users mailing list -- ceph-users@ceph.io 
> > To unsubscribe send an email to ceph-users-le...@ceph.io 
> > 
> 
> ___
> ceph-users mailing list -- ceph-users@ceph.io 
> To unsubscribe send an email to ceph-users-le...@ceph.io 
> 
> 

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Urgent help! RGW Disappeared on Quincy

[Deep Dish]

I built a net-new Quincy cluster (17.2.5) using ceph orch as follows:

2x mgrs
4x rgw
5x mon
4x rgw
5x mds
6x osd hosts w/ 10 drives each --> will be growing to 7 osd hosts in the
coming days.

I migrated all data from my legacy nautilus cluster (via rbd-mirror, rclone
for s3 buckets, etc.).  All moved over successfully without issue.

The cluster went through a series of rebalancing events (adding capacity,
osd nodes, changing fault domain for EC volumes).

It's settled now, however throughout the process all of my RGW nodes are no
longer part of the cluster -- meaning ceph doesn't recognize / detect them,
despite containers, networking, etc. all being setup correctly.   This also
means I'm unable to manage any RGW functions (via the dashboard or cli).
As an example via cli (within Cephadm shell):

# radosgw-admin pools list

could not list placement set: (2) No such file or directory

I have data in buckets, how can I get my RGWs to return online?
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Urgent help! RGW Disappeared on Quincy

[Deep Dish]

Quick update:

- I followed documentation, and ran the following:

# ceph dashboard set-rgw-credentials

Error EINVAL: No RGW credentials found, please consult the documentation on
how to enable RGW for the dashboard.



- I see dashboard credentials configured (all this was working fine before):


# ceph dashboard get-rgw-api-access-key

P?G  (? commented out)



Seems to me like my RGW config is non-existent / corrupted for some
reason.  When trying to curl a RGW directly I get a "connection refused".



On Tue, Dec 27, 2022 at 9:41 AM Deep Dish  wrote:

> I built a net-new Quincy cluster (17.2.5) using ceph orch as follows:
>
> 2x mgrs
> 4x rgw
> 5x mon
> 4x rgw
> 5x mds
> 6x osd hosts w/ 10 drives each --> will be growing to 7 osd hosts in the
> coming days.
>
> I migrated all data from my legacy nautilus cluster (via rbd-mirror,
> rclone for s3 buckets, etc.).  All moved over successfully without issue.
>
> The cluster went through a series of rebalancing events (adding capacity,
> osd nodes, changing fault domain for EC volumes).
>
> It's settled now, however throughout the process all of my RGW nodes are
> no longer part of the cluster -- meaning ceph doesn't recognize / detect
> them, despite containers, networking, etc. all being setup correctly.
> This also means I'm unable to manage any RGW functions (via the dashboard
> or cli).   As an example via cli (within Cephadm shell):
>
> # radosgw-admin pools list
>
> could not list placement set: (2) No such file or directory
>
> I have data in buckets, how can I get my RGWs to return online?
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Object missing in bucket index

[mahnoosh shahidi]

Hi Ceph users,

I have a running cluster in octopus 15.2.12. I found an object in one of my
S3 buckets that does not exist in the bucket list but I can download it
with any client. I also tried to get bucket index data by ```radosgw-admin
bi list --bucket MYBUCKET --object MYOBJECT``` and it returns an empty
list.
What is the reason for this behaviour? How can I fix this?

Regards,
Mahnoosh
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Object missing in bucket index

[Ramin Najjarbashi]

Hi Mahnoosh

try with object locator

On Tue, Dec 27, 2022 at 8:13 PM mahnoosh shahidi 
wrote:

> Hi Ceph users,
>
> I have a running cluster in octopus 15.2.12. I found an object in one of my
> S3 buckets that does not exist in the bucket list but I can download it
> with any client. I also tried to get bucket index data by ```radosgw-admin
> bi list --bucket MYBUCKET --object MYOBJECT``` and it returns an empty
> list.
> What is the reason for this behaviour? How can I fix this?
>
> Regards,
> Mahnoosh
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Urgent help! RGW Disappeared on Quincy

[Deep Dish]

HI Pavin,

Thanks for the reply.   I'm a bit at a loss honestly as this worked
perfectly without any issue up until the rebalance of the cluster.
Orchestrator is great.   Aside from this (which I suspect is not
orchestrator related), I haven't had any issues.

In terms of logs, I'm not sure where to start looking in this new
containerized environment as they pertain to individual ceph processes -- I
assumed everything would be centrally collected within orch.

Connecting into the podman container of a RGW, there are no logs in
/var/log/ceph aside from ceph-volume.   My ceph.conf is minimal with only
monitors defined.  The only log I'm able to pull is as follows:

# podman logs 35d4ac5445ca

INFO:ceph-crash:monitoring path /var/lib/ceph/crash, delay 600s

Traceback (most recent call last):

  File "/usr/bin/ceph-crash", line 113, in 

main()

  File "/usr/bin/ceph-crash", line 109, in main

time.sleep(args.delay * 60)

TypeError: handler() takes 1 positional argument but 2 were given

INFO:ceph-crash:monitoring path /var/lib/ceph/crash, delay 600s



Looks like the RGW daemon is crashing.   How do I get logs to persist?   I
suspect I won't be able to use orchestrator to push down the config, and
would have to manipulate within the container image itself.

I also attempted to redeply the RGW containers without success.

On Tue, Dec 27, 2022 at 10:39 AM Pavin Joseph  wrote:

> Here's the first things I'd check in your situation:
>
> 1. Logs
> 2. Is the RGW HTTP server running on its port?
> 3. Re-check config including authentication.
>
> ceph orch is too new and didn't pass muster in our own internal testing.
> You're braver than most for using it in production.
>
> Pavin.
>
> On 27-Dec-22 8:48 PM, Deep Dish wrote:
> > Quick update:
> >
> > - I followed documentation, and ran the following:
> >
> > # ceph dashboard set-rgw-credentials
> >
> > Error EINVAL: No RGW credentials found, please consult the documentation
> on
> > how to enable RGW for the dashboard.
> >
> >
> >
> > - I see dashboard credentials configured (all this was working fine
> before):
> >
> >
> > # ceph dashboard get-rgw-api-access-key
> >
> > P?G  (? commented out)
> >
> >
> >
> > Seems to me like my RGW config is non-existent / corrupted for some
> > reason.  When trying to curl a RGW directly I get a "connection refused".
> >
> >
> >
> > On Tue, Dec 27, 2022 at 9:41 AM Deep Dish  wrote:
> >
> >> I built a net-new Quincy cluster (17.2.5) using ceph orch as follows:
> >>
> >> 2x mgrs
> >> 4x rgw
> >> 5x mon
> >> 4x rgw
> >> 5x mds
> >> 6x osd hosts w/ 10 drives each --> will be growing to 7 osd hosts in the
> >> coming days.
> >>
> >> I migrated all data from my legacy nautilus cluster (via rbd-mirror,
> >> rclone for s3 buckets, etc.).  All moved over successfully without
> issue.
> >>
> >> The cluster went through a series of rebalancing events (adding
> capacity,
> >> osd nodes, changing fault domain for EC volumes).
> >>
> >> It's settled now, however throughout the process all of my RGW nodes are
> >> no longer part of the cluster -- meaning ceph doesn't recognize / detect
> >> them, despite containers, networking, etc. all being setup correctly.
> >> This also means I'm unable to manage any RGW functions (via the
> dashboard
> >> or cli).   As an example via cli (within Cephadm shell):
> >>
> >> # radosgw-admin pools list
> >>
> >> could not list placement set: (2) No such file or directory
> >>
> >> I have data in buckets, how can I get my RGWs to return online?
> >>
> > ___
> > ceph-users mailing list -- ceph-users@ceph.io
> > To unsubscribe send an email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Object missing in bucket index

[mahnoosh shahidi]

Hi Ramin,

Thanks for your response. Can you explain it a bit more? How is the object
locator supposed to fix this problem? And do you have any idea of the root
cause?

Thank you

On Tue, Dec 27, 2022 at 8:33 PM Ramin Najjarbashi <
ramin.najarba...@gmail.com> wrote:

> Hi Mahnoosh
>
> try with object locator
>
> On Tue, Dec 27, 2022 at 8:13 PM mahnoosh shahidi 
> wrote:
>
>> Hi Ceph users,
>>
>> I have a running cluster in octopus 15.2.12. I found an object in one of
>> my
>> S3 buckets that does not exist in the bucket list but I can download it
>> with any client. I also tried to get bucket index data by ```radosgw-admin
>> bi list --bucket MYBUCKET --object MYOBJECT``` and it returns an empty
>> list.
>> What is the reason for this behaviour? How can I fix this?
>>
>> Regards,
>> Mahnoosh
>> ___
>> ceph-users mailing list -- ceph-users@ceph.io
>> To unsubscribe send an email to ceph-users-le...@ceph.io
>>
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Cannot create CephFS subvolume

[Daniel Kovacs]

Hello!

I'd like to create a CephFS subvol, with these command: ceph fs 
subvolume create cephfs_ssd subvol_1
I got this error: Error EINVAL: invalid value specified for 
ceph.dir.subvolume

If I use another cephfs volume, there were no error reported.

What did I wrong?

Best regards,

Daniel

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Best Disk Brand for Ceph OSD

[hosseinz8...@yahoo.com]

Hello Everyone.I want to create ceph cluster with 50 OSD. I am looking for the 
best enterprise SSD disk model which has high IOPs but acceptable price.Which 
disk Brand & Model do you know to introduce me to buy?
Thanks.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Urgent help! RGW Disappeared on Quincy

[Deep Dish]

Got logging enabled as per
https://ceph.io/en/news/blog/2022/centralized_logging/.   My embedded
grafana doesn't come up in the dashboard, but at least I have log (files)
on my nodes.   Interesting.

Two issues plaguing my cluster:

1 - RGWs not manageable
2 - MDS_SLOW_METADATA_IO warning (impact to cephfs)

Issue 1:

I have 4x RGWs deployed.   All started / processes running.  They all
report similar log entries:

7fcc32b6a5c0  0 deferred set uid:gid to 167:167 (ceph:ceph)

7fcc32b6a5c0  0 ceph version 17.2.5
(98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable), process
radosgw, pid 2

7fcc32b6a5c0  0 framework: beast

7fcc32b6a5c0  0 framework conf key: port, val: 80

7fcc32b6a5c0  1 radosgw_Main not setting numa affinity

7fcc32b6a5c0  1 rgw_d3n: rgw_d3n_l1_local_datacache_enabled=0

7fcc32b6a5c0  1 D3N datacache enabled: 0

7fcc0869a700  0 INFO: RGWReshardLock::lock found lock on reshard.11
to be held by another RGW process; skipping for now

7fcc0bea1700  0 lifecycle: RGWLC::process() failed to acquire lock on lc.1,
sleep 5, try again

7fcc0dea5700  0 lifecycle: RGWLC::process() failed to acquire lock on lc.3,
sleep 5, try again

7fcc0dea5700  0 lifecycle: RGWLC::process() failed to acquire lock on
lc.16, sleep 5, try again

7fcc0dea5700  0 lifecycle: RGWLC::process() failed to acquire lock on
lc.16, sleep 5, try again

7fcc0bea1700  0 lifecycle: RGWLC::process() failed to acquire lock on
lc.16, sleep 5, try again

7fcc0dea5700  0 lifecycle: RGWLC::process() failed to acquire lock on
lc.16, sleep 5, try again

7fcc0bea1700  0 lifecycle: RGWLC::process() failed to acquire lock on
lc.16, sleep 5, try again

7fcc0dea5700  0 lifecycle: RGWLC::process() failed to acquire lock on
lc.16, sleep 5, try again

7fcc0bea1700  0 lifecycle: RGWLC::process() failed to acquire lock on
lc.16, sleep 5, try again

7fcc0dea5700  0 lifecycle: RGWLC::process() failed to acquire lock on
lc.16, sleep 5, try again

7fcc0bea1700  0 lifecycle: RGWLC::process() failed to acquire lock on
lc.16, sleep 5, try again
(repeating)

Seems like a stale lock, not previously cleaned up when the cluster was
busy recovering and rebalancing.

Issue 2:

ceph health detail:

[WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs

mds.fs01.ceph02mon03.rjcxat(mds.0): 8 slow metadata IOs are blocked >
30 secs, oldest blocked for 39485 secs

Log entries from ceph02mon03 MDS host:

 7fe36debb700  1 mds.fs01.ceph02mon03.rjcxat Updating MDS map to version
131271 from mon.4
 7fe36debb700  1 mds.fs01.ceph02mon03.rjcxat Updating MDS map to version
131272 from mon.4
 7fe36debb700  1 mds.fs01.ceph02mon03.rjcxat Updating MDS map to version
131273 from mon.4
 7fe36debb700  1 mds.fs01.ceph02mon03.rjcxat Updating MDS map to version
131274 from mon.4
 7fe36debb700  1 mds.fs01.ceph02mon03.rjcxat Updating MDS map to version
131275 from mon.4
 7fe36c6b8700  0 log_channel(cluster) log [WRN] : 1 slow requests, 1
included below; oldest blocked for > 33.126589 secs
 7fe36c6b8700  0 log_channel(cluster) log [WRN] : slow request 33.126588
seconds old, received at 2022-12-27T19:45:45.952225+:
client_request(client.55009:99980 create
#0x1000bc2/vzdump-qemu-30003-2022_12_27-14_43_43.log
2022-12-27T19:45:45.948045+ caller_uid=0, caller_gid=0{}) currently
submit entry: journal_and_reply
 7fe36debb700  1 mds.fs01.ceph02mon03.rjcxat Updating MDS map to version
131276 from mon.4
 7fe36c6b8700  0 log_channel(cluster) log [WRN] : 1 slow requests, 0
included below; oldest blocked for > 38.126737 secs
 7fe36debb700  1 mds.fs01.ceph02mon03.rjcxat Updating MDS map to version
131277 from mon.4
 7fe36debb700  1 mds.fs01.ceph02mon03.rjcxat Updating MDS map to version
131278 from mon.4
 7fe36debb700  1 mds.fs01.ceph02mon03.rjcxat Updating MDS map to version
131279 from mon.4
 7fe36debb700  1 mds.fs01.ceph02mon03.rjcxat Updating MDS map to version
131280 from mon.4


I suspect that the file in the log above int's the culprit.   How can I get
to the root cause of MDS slowdowns?


On Tue, Dec 27, 2022 at 3:32 PM Pavin Joseph  wrote:

> Interesting, the logs show the crash module [0] itself has crashed.
> Something sent it a SIGINT or SIGTERM and the module didn't handle it
> correctly due to what seems like a bug in the code.
>
> I haven't experienced the crash module itself crashing yet (in Quincy)
> because nothing sent a SIG[INT|TERM] to it yet.
>
> So I'd continue investigating into why these signals were sent to the
> crash module.
>
> To fix the crash module from crashing, go to "/usr/bin/ceph-crash" and
> edit the handler function on line 82 like so:
>
> def handler(signum, frame):
>print('*** Interrupted with signal %d ***' % signum)
>signame = signal.Signals(signum).name
>print(f'Signal handler called with signal {signame} ({signum})')
>print(frame)
>sys.exit(0)
>
> ---
>
> Once the crash module is working, perhaps you could run a "ceph crash ls"
>
> Regarding podman logs, perhaps try this [1].
>
> [0]: https://docs.ceph.com/en

[ceph-users] Re: Cannot create CephFS subvolume

[Milind Changire]

What ceph version are you using?

$ ceph versions


On Wed, Dec 28, 2022 at 3:17 AM Daniel Kovacs 
wrote:

> Hello!
>
> I'd like to create a CephFS subvol, with these command: ceph fs
> subvolume create cephfs_ssd subvol_1
> I got this error: Error EINVAL: invalid value specified for
> ceph.dir.subvolume
> If I use another cephfs volume, there were no error reported.
>
> What did I wrong?
>
> Best regards,
>
> Daniel
>
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
>

-- 
Milind
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Cannot create CephFS subvolume

[Milind Changire]

Also, please list the volumes available on your system:

$ ceph fs volume ls


On Wed, Dec 28, 2022 at 9:09 AM Milind Changire  wrote:

> What ceph version are you using?
>
> $ ceph versions
>
>
> On Wed, Dec 28, 2022 at 3:17 AM Daniel Kovacs 
> wrote:
>
>> Hello!
>>
>> I'd like to create a CephFS subvol, with these command: ceph fs
>> subvolume create cephfs_ssd subvol_1
>> I got this error: Error EINVAL: invalid value specified for
>> ceph.dir.subvolume
>> If I use another cephfs volume, there were no error reported.
>>
>> What did I wrong?
>>
>> Best regards,
>>
>> Daniel
>>
>> ___
>> ceph-users mailing list -- ceph-users@ceph.io
>> To unsubscribe send an email to ceph-users-le...@ceph.io
>>
>>
>
> --
> Milind
>
>

-- 
Milind
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Does Replica Count Affect Tell Bench Result or Not?

[Konstantin Shalygin]

Hi,

The cache was gone, optimize is proceed. This is not enterprise device, you 
should never use it with Ceph 🙂


k
Sent from my iPhone

> On 27 Dec 2022, at 16:41, hosseinz8...@yahoo.com wrote:
> 
>  Thanks AnthonyI have a cluster with QLC SSD disks (Samsung QVO 860). The 
> cluster works for 2 year. Now all OSDs return 12 iops when running tell bench 
> which is very slow. But I Buy new QVO disks yesterday, and I added this new 
> disk to cluster. For the first 1 hour, I got 100 iops from this new OSD. But 
> after 1 Hour, this new disk (OSD) returns to iops 12 again as the same as 
> other OLD OSDs.I can not imagine what happening?!!
>On Tuesday, December 27, 2022 at 12:18:07 AM GMT+3:30, Anthony D'Atri 
>  wrote:  
> 
> My understanding is that when you ask an OSD to bench (via the admin socket), 
> only that OSD executes, there is no replication.  Replication is a function 
> of PGs.
> 
> Thus, this is a narrowly-focused tool with both unique advantages and 
> disadvantages.
> 
> 
> 
>> On Dec 26, 2022, at 12:47 PM, hosseinz8...@yahoo.com wrote:
>> 
>> Hi experts,I want to know, when I execute ceph tell osd.x bench command, is 
>> replica 3 considered in the bench or not? I mean, for example in case of 
>> replica 3, when I executing tell bench command, replica 1 of bench data 
>> write to osd.x, replica 2 write to osd.y and replica 3 write to osd.z? If 
>> this is true, it means that I can not get benchmark of only one of my OSD in 
>> the cluster because the IOPS and throughput of 2 other for example slow OSDs 
>> will affect the result of tell bench command for my target OSD.Is that true?
>> Thanks in advance.
>> ___
>> ceph-users mailing list -- ceph-users@ceph.io
>> To unsubscribe send an email to ceph-users-le...@ceph.io
> 
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
> 
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] CephFS active-active

[Isaiah Tang Yue Shun]

Hi all,

>From the documentation, I can only find Ceph Object Gateway multi-site 
>implementation. I wonder is it if we are using CephFS, how can we achieve 
>active-active setup for production?

Any input is appreciated.

Thanks.

Regards,
Isaiah Tang
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Best Disk Brand for Ceph OSD

[Eneko Lacunza]

Hi,

El 28/12/22 a las 0:55, hosseinz8...@yahoo.com escribió:

Hello Everyone.I want to create ceph cluster with 50 OSD. I am looking for the best 
enterprise SSD disk model which has high IOPs but acceptable price.Which disk Brand 
& Model do you know to introduce me to buy?



Just search for Enterprise/Datacenter SSDs and check price/IOPS. We have 
used Micron, Intel and Samsung with success.


Avoid Kingston DC450R, it doesn't have capacitors and performance for 
Ceph is HDD-like.


When you have the disks check performance with and without write cache 
(see other threads in this list for reference).


Cheers

Eneko Lacunza
Zuzendari teknikoa | Director técnico
Binovo IT Human Project

Tel. +34 943 569 206 |https://www.binovo.es
Astigarragako Bidea, 2 - 2º izda. Oficina 10-11, 20180 Oiartzun

https://www.youtube.com/user/CANALBINOVO
https://www.linkedin.com/company/37269706/
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: CephFS active-active

[Milind Changire]

Isaiah,
I'm trying to understand your requirements for a CephFS Active-Active setup.
What do you want to achieve with a CephFS Active-Active setup ?
Once you list the exact requirements, we can discuss further on how to
achieve them.

There's also something called *CephFS Snapshot Mirroring*:
https://docs.ceph.com/en/latest/cephfs/cephfs-mirroring/
which helps with replication of snapshots to a remote cluster.
This provision is supposed to be used for Disaster Recovery and is not a
real-time replication solution.
So, if the primary site goes down while there is ongoing activity on the
file-system then the remote site may not have all the data replicated to it.
You could then restore data from a snapshot to a volume and start using it
as the active copy until the failed cluster is recovered and recommissioned
into production.


On Wed, Dec 28, 2022 at 11:27 AM Isaiah Tang Yue Shun 
wrote:

> Hi all,
>
> From the documentation, I can only find Ceph Object Gateway multi-site
> implementation. I wonder is it if we are using CephFS, how can we achieve
> active-active setup for production?
>
> Any input is appreciated.
>
> Thanks.
>
> Regards,
> Isaiah Tang
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
>

-- 
Milind
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Does Replica Count Affect Tell Bench Result or Not?

[hosseinz8...@yahoo.com]

 Thanks. I am planning to change all of my disks. But do you know enterprise 
SSD Disk which is best in trade of between cost & iops performance?Which model 
and brand.Thanks in advance.
On Wednesday, December 28, 2022 at 08:44:34 AM GMT+3:30, Konstantin 
Shalygin  wrote:  
 
 Hi,

The cache was gone, optimize is proceed. This is not enterprise device, you 
should never use it with Ceph 🙂


k
Sent from my iPhone

> On 27 Dec 2022, at 16:41, hosseinz8...@yahoo.com wrote:
> 
>  Thanks AnthonyI have a cluster with QLC SSD disks (Samsung QVO 860). The 
> cluster works for 2 year. Now all OSDs return 12 iops when running tell bench 
> which is very slow. But I Buy new QVO disks yesterday, and I added this new 
> disk to cluster. For the first 1 hour, I got 100 iops from this new OSD. But 
> after 1 Hour, this new disk (OSD) returns to iops 12 again as the same as 
> other OLD OSDs.I can not imagine what happening?!!
>    On Tuesday, December 27, 2022 at 12:18:07 AM GMT+3:30, Anthony D'Atri 
> wrote:  
> 
> My understanding is that when you ask an OSD to bench (via the admin socket), 
> only that OSD executes, there is no replication.  Replication is a function 
> of PGs.
> 
> Thus, this is a narrowly-focused tool with both unique advantages and 
> disadvantages.
> 
> 
> 
>> On Dec 26, 2022, at 12:47 PM, hosseinz8...@yahoo.com wrote:
>> 
>> Hi experts,I want to know, when I execute ceph tell osd.x bench command, is 
>> replica 3 considered in the bench or not? I mean, for example in case of 
>> replica 3, when I executing tell bench command, replica 1 of bench data 
>> write to osd.x, replica 2 write to osd.y and replica 3 write to osd.z? If 
>> this is true, it means that I can not get benchmark of only one of my OSD in 
>> the cluster because the IOPS and throughput of 2 other for example slow OSDs 
>> will affect the result of tell bench command for my target OSD.Is that true?
>> Thanks in advance.
>> ___
>> ceph-users mailing list -- ceph-users@ceph.io
>> To unsubscribe send an email to ceph-users-le...@ceph.io
> 
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
> 
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
  
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io