[ceph-users] Re: find PG with large omap object

2023-10-31 Thread Ramin Najjarbashi
https://gist.github.com/RaminNietzsche/b8702014c333f3a44a995d5a6d4a56be On Mon, Oct 16, 2023 at 4:27 PM Frank Schilder wrote: > Hi Eugen, > > the warning threshold is per omap object, not per PG (which apparently has > more than 1 omap object). Still, I misread the numbers by 1 order, which > me

[ceph-users] Connect to Ceph Cluster from other network

2024-03-10 Thread Ramin Najjarbashi
Hello Ceph community, I am currently facing an issue connecting to my Ceph cluster from an external system and would appreciate your guidance. Here are some details about my setup: - Ceph Public Network: 172.0.0.1/24 - Ceph Private Network: 172.10.0.1/24 - Monitor (MON) Nodes: - mon1: 172.0.3.

[ceph-users] crashes after upgrade from octopus to pacific

2022-07-19 Thread Ramin Najjarbashi
Hi My account in "tracker.ceph.com" was not approved after 5 days but anyway I have some problems in Ceph v16.2.7. Accourding this issue [rgw: s3website crashes after upgrade from octopus to pacific](https://tracker.ceph.com/issues/53913), RGW crashed whene s3website calls without subdomain (`s-

[ceph-users] Re: crashes after upgrade from octopus to pacific

2022-07-19 Thread Ramin Najjarbashi
p -rdS ` is needed to interpret this. > On Jul 19, 2022, at 2:16 PM, Ramin Najjarbashi > wrote: > > Hi > My account in "tracker.ceph.com <http://tracker.ceph.com/>" was not approved > after 5 days but anyway I have some problems in Ceph v16.2.7. > Accour

[ceph-users] multisite sync error

2022-12-04 Thread Ramin Najjarbashi
Hi I have two zonegroup inside a realm, each one has a zone like the one below: REALM ++ | +--+ +-+ | | | | | | | | | zonegroup 1 (Master) | | zonegroup 2 | | | | | | | | | | | | | |

[ceph-users] Re: Object missing in bucket index

2022-12-27 Thread Ramin Najjarbashi
Hi Mahnoosh try with object locator On Tue, Dec 27, 2022 at 8:13 PM mahnoosh shahidi wrote: > Hi Ceph users, > > I have a running cluster in octopus 15.2.12. I found an object in one of my > S3 buckets that does not exist in the bucket list but I can download it > with any client. I also tried

[ceph-users] Re: large omap objects in the .rgw.log pool

2023-01-17 Thread Ramin Najjarbashi
I am experiencing the same problem with 'Large omap object' in Ceph, so I wrote a script to find the large objects in the pool, count the number of "omap" in each object, and compare it with the value set for large_omap_object_key_threshold in the Ceph configuration. https://gist.github.com/RaminN

[ceph-users] Inconsistency in rados ls

2023-02-01 Thread Ramin Najjarbashi
Hi I hope this email finds you well. I am reaching out to you because I have encountered an issue with my CEPH Bluestore cluster and I am seeking your assistance. I have a cluster with approximately one billion objects and when I run a PG query, it shows that I have 27,000 objects per PG. I have ru

[ceph-users] Re: Inconsistency in rados ls

2023-02-04 Thread Ramin Najjarbashi
; and it sums up to 10 objects which matches the total stats: > > storage01:~ # ceph df | grep -E "OBJECTS|volumes" > POOL ID PGS STORED OBJECTS USED %USED > MAX AVAIL > volumes38379 B 10 960 KiB 07.5

[ceph-users] Re: Inconsistency in rados ls

2023-02-06 Thread Ramin Najjarbashi
ve any further suggestions or if there is anything else I can assist with. On Mon, Feb 6, 2023 at 4:24 PM Robert Sander wrote: > On 04.02.23 20:54, Ramin Najjarbashi wrote: > > > ceph df | grep mypoo > > > > --- POOLS --- > > > > POOL OBJECTS > > &g

[ceph-users] orphan multipart objects in Ceph cluster

2023-03-27 Thread Ramin Najjarbashi
I hope this email finds you well. I wanted to share a recent experience I had with our Ceph cluster and get your feedback on a solution I came up with. Recently, we had some orphan objects stuck in our cluster that were not visible by any client like s3cmd, boto3, and mc. This caused some confusio

[ceph-users] Re: Ceph Failure and OSD Node Stuck Incident

2023-03-30 Thread Ramin Najjarbashi
On Thu, Mar 30, 2023 at 6:08 PM wrote: > We encountered a Ceph failure where the system became unresponsive with no > IOPS or throughput after encountering a failed node. Upon investigation, it > appears that the OSD process on one of the Ceph storage nodes is stuck, but > ping is still responsiv

[ceph-users] Re: Radosgw-admin bucket list has duplicate objects

2023-04-15 Thread Ramin Najjarbashi
Hi, to verify if the bucket names are really duplicated, please try running the following command: 'radosgw-admin bucket list --allow-unordered | jq -r ".[]" | sort | uniq -c | sort -h | tail' On Sat, Apr 15, 2023 at 7:20 PM Janne Johansson wrote: > Den lör 15 apr. 2023 kl 15:47 skrev mahnoosh

[ceph-users] Re: OSDs remain not in after update to v17

2023-04-15 Thread Ramin Najjarbashi
Hi I think the issue you are experiencing may be related to a bug that has been reported in the Ceph project. Specifically, the issue is documented in https://tracker.ceph.com/issues/58156, and a pull request has been submitted and merged in https://github.com/ceph/ceph/pull/44090. On Fri, Apr 1

[ceph-users] Re: ceph pg stuck - missing on 1 osd how to proceed

2023-04-15 Thread Ramin Najjarbashi
Hi You can use the script available at https://github.com/TheJJ/ceph-balancer/blob/master/placementoptimizer.py to check the status of backfilling and PG state, and also to cancel backfilling using upmap. To view the movement status of all PGs in the backfilling state, you can execute the command

[ceph-users] Unable to Retrieve Usage Logs in Ceph RGW with Boto3

2023-05-13 Thread Ramin Najjarbashi
Hi I am currently working on retrieving user usage and usage logs from Ceph RGW using Boto3. While I am able to successfully obtain user usage and all bucket usage with the following model and code: ```python params = { "start-date": "2023-05-13T00:00:00Z", "end-date": "2023-05-13T23:59:59Z", }

[ceph-users] Creating a bucket with bucket constructor in Ceph v16.2.7

2023-05-17 Thread Ramin Najjarbashi
Hi I'm currently using Ceph version 16.2.7 and facing an issue with bucket creation in a multi-zone configuration. My setup includes two zone groups: ZG1 (Master) and ZG2, with one zone in each zone group (zone-1 in ZG1 and zone-2 in ZG2). The objective is to create buckets in a specific zone gr

[ceph-users] Re: Creating a bucket with bucket constructor in Ceph v16.2.7

2023-05-18 Thread Ramin Najjarbashi
11:13 PM Ramin Najjarbashi > wrote: > > > > Hi > > > > I'm currently using Ceph version 16.2.7 and facing an issue with bucket > > creation in a multi-zone configuration. My setup includes two zone > groups: > > > > ZG1 (Master) and ZG2, with o

[ceph-users] Re: Creating a bucket with bucket constructor in Ceph v16.2.7

2023-05-20 Thread Ramin Najjarbashi
): Location: zg1-api-name Payer: BucketOwner Expiration Rule: none Policy:none CORS: none ACL: development: FULL_CONTROL ``` On Thu, May 18, 2023 at 4:29 PM Ramin Najjarbashi < ramin.najarba...@gmail.com> wrote: > Thank Casey > > Currently, when I crea

[ceph-users] Workload Separation in Ceph RGW Cluster - Recommended or Not?

2023-06-06 Thread Ramin Najjarbashi
Hi I would like to seek your insights and recommendations regarding the practice of workload separation in a Ceph RGW (RADOS Gateway) cluster. I have been facing challenges with large queues in my deployment and would appreciate your expertise in determining whether workload separation is a recomm

[ceph-users] Re: Workload Separation in Ceph RGW Cluster - Recommended or Not?

2023-06-06 Thread Ramin Najjarbashi
r bottlenecks are within RGW not > within the cluster? > > e.g. is your index pool on a large number of NVMe SSDs with sufficient > PGs? Is your bucket data on SSD as well? > > > On Jun 6, 2023, at 13:52, Ramin Najjarbashi > wrote: > > I would like to seek your insight

[ceph-users] Changing bucket owner in a multi-zonegroup Ceph cluster

2023-06-25 Thread Ramin Najjarbashi
Hi all I have a Ceph cluster consisting of two zonegroups with metadata syncing enabled. I need to change the owner of a bucket that is located in the secondary zonegroup. I followed the steps below: Unlinked the bucket from the old user on the secondary zonegroup: bash Copy code $ radosgw-admin

[ceph-users] Slow ACL Changes in Secondary Zone

2023-07-05 Thread Ramin Najjarbashi
Dear Ceph community, I'm facing an issue with ACL changes in the secondary zone of my Ceph cluster after making modifications to the API name in the master zone of my master zonegroup. I would appreciate any insights or suggestions on how to resolve this problem. Here's the background information

[ceph-users] Separating Mons and OSDs in Ceph Cluster

2023-09-09 Thread Ramin Najjarbashi
Hi I am writing to seek guidance and best practices for a maintenance operation in my Ceph cluster. I have an older cluster in which the Monitors (Mons) and Object Storage Devices (OSDs) are currently deployed on the same host. I am interested in separating them while ensuring zero downtime and mi

[ceph-users] Re: Separating Mons and OSDs in Ceph Cluster

2023-09-10 Thread Ramin Najjarbashi
Thank you, Eugen, for suggesting the option of leaving the Mons and OSDs co-located if it's not an actual requirement to redeploy Mons. Your insight into the lightweight nature of MON daemons and the possibility of gradually adding new MONs without interrupting cluster services is well-received. We

[ceph-users] Severe Latency Issues in Ceph Cluster

2025-03-01 Thread Ramin Najjarbashi
Hi We are currently facing severe latency issues in our Ceph cluster, particularly affecting read and write operations. At times, write operations completely stall, leading to significant service degradation. Below is a detailed breakdown of the issue, our observations, and the mitigation steps we

[ceph-users] Re: Severe Latency Issues in Ceph Cluster

2025-03-03 Thread Ramin Najjarbashi
packets on the network. Let me know if you have any insights or suggestions. On Mon, Mar 3, 2025 at 12:36 Stefan Kooman wrote: > On 01-03-2025 15:10, Ramin Najjarbashi wrote: > > Hi > > We are currently facing severe latency issues in our Ceph cluster, > > particularly aff

[ceph-users] Unexpected data sync on deleted bucket instances with sync disabled

2025-05-25 Thread Ramin Najjarbashi
Hi I am running a multi-region Ceph cluster with version 18.2.6. Our system has been operating reliably for a long time and currently holds over 100,000 buckets and approximately 1.5 billion objects. We recently attempted to add a new zone to the primary region. To control the synchronization sco