Re: [ceph-users] CephFS read IO caching, where it is happining?

Thank you guys, I tried to add option "exec_prerun=echo 3 > /proc/sys/vm/drop_caches" as well as "exec_prerun=echo 3 | sudo tee /proc/sys/vm/drop_caches", but despite FIO corresponds that command was executed, there are no changes. But, I caught very strange another behavior. If I will run my FIO

Re: [ceph-users] Experience with 5k RPM/archive HDDs

> Op 3 februari 2017 om 8:39 schreef Christian Balzer : > > > > Hello, > > On Fri, 3 Feb 2017 10:30:28 +0300 Irek Fasikhov wrote: > > > Hi, Maxime. > > > > Linux SMR is only starting with version 4.9 kernel. > > > What Irek said. > > Also, SMR in general is probably a bad match for Ceph. >

[ceph-users] RGW authentication fail with AWS S3 v4

Hi all, I'm using Ceph Object Gateway with S3 API (ceph-radosgw-10.2.5-0.el7.x86_64 on CentOS Linux release 7.3.1611) and I use generate_presigned_url method of boto3 to create rgw url. This url working fine in period of 15 minutes, after 15 minutes I recived *RequestTimeTooSkewed* error. My rados

Re: [ceph-users] RGW authentication fail with AWS S3 v4

> Op 3 februari 2017 om 9:52 schreef Khang Nguyễn Nhật > : > > > Hi all, > I'm using Ceph Object Gateway with S3 API (ceph-radosgw-10.2.5-0.el7.x86_64 > on CentOS Linux release 7.3.1611) and I use generate_presigned_url method > of boto3 to create rgw url. This url working fine in period of 15

Re: [ceph-users] RGW authentication fail with AWS S3 v4

> > Dear Wido, I have used X-Amz-Expires=86400 in url but it doesn't work 2017-02-03 16:00 GMT+07:00 Wido den Hollander : > > > Op 3 februari 2017 om 9:52 schreef Khang Nguyễn Nhật < > nguyennhatkhang2...@gmail.com>: > > > > > > Hi all, > > I'm using Ceph Object Gateway with S3 API (ceph-radosgw

[ceph-users] Monitor repeatedly calling new election

Hi, everyone. Recently, when I was doing some stress test, one of the monitors of my ceph cluster was marked down, and all the monitors repeatedly call new election and the I/O can be finished. There were three monitors in my cluster: rg3-ceph36, rg3-ceph40, rg3-ceph45. It was rg3-ceph40 that w

[ceph-users] Monitor repeatedly calling new election

Hi, everyone. Recently, when I was doing some stress test, one of the monitors of my ceph cluster was marked down, and all the monitors repeatedly call new election and the I/O can be finished. There were three monitors in my cluster: rg3-ceph36, rg3-ceph40, rg3-ceph45. It was rg3-ceph40 that w

Re: [ceph-users] Monitor repeatedly calling new election

On 02/03/2017 09:16 AM, 许雪寒 wrote: Hi, everyone. Recently, when I was doing some stress test, one of the monitors of my ceph cluster was marked down, and all the monitors repeatedly call new election and the I/O can be finished. There were three monitors in my cluster: rg3-ceph36, rg3-ceph40,

Re: [ceph-users] Experience with 5k RPM/archive HDDs

Hi, Interesting feedback! > In my opinion the SMR can be used exclusively for the RGW. > Unless it's something like a backup/archive cluster or pool with little to > none concurrent R/W access, you're likely to run out of IOPS (again) long > before filling these monsters up. That’s exactly

Re: [ceph-users] Experience with 5k RPM/archive HDDs

> Op 3 februari 2017 om 11:03 schreef Maxime Guyot : > > > Hi, > > Interesting feedback! > > > In my opinion the SMR can be used exclusively for the RGW. > > Unless it's something like a backup/archive cluster or pool with little to > none concurrent R/W access, you're likely to run out of

Re: [ceph-users] CephFS read IO caching, where it is happining?

> Op 3 februari 2017 om 9:07 schreef Ahmed Khuraidah : > > > Thank you guys, > > I tried to add option "exec_prerun=echo 3 > /proc/sys/vm/drop_caches" as > well as "exec_prerun=echo 3 | sudo tee /proc/sys/vm/drop_caches", but > despite FIO corresponds that command was executed, there are no cha

Re: [ceph-users] CephFS read IO caching, where it is happining?

On Fri, Feb 3, 2017 at 8:07 AM, Ahmed Khuraidah wrote: > Thank you guys, > > I tried to add option "exec_prerun=echo 3 > /proc/sys/vm/drop_caches" as > well as "exec_prerun=echo 3 | sudo tee /proc/sys/vm/drop_caches", but > despite FIO corresponds that command was executed, there are no changes. >

Re: [ceph-users] 答复: Monitor repeatedly calling new election

On 02/03/2017 09:53 AM, 许雪寒 wrote: Thanks for your quick reply:-) I'm trying to send you more logs. Many of our online clusters has been ]running hammer version for a long time, it's a bit difficult for us to > update those clusters since we are really afraid of encountering problems during u

Re: [ceph-users] slow requests break performance

Hello, On Thu, 02 Feb 2017 10:24:53 +0100 Eugen Block wrote: > Hi, > > thank you very much for your answer! I'm not sure I get all your > points, but I'll try to dig deeper. > I'll repeat myself and say that looking at your nodes with atop during a benchmark and slow request situation should

Re: [ceph-users] Split-brain in a multi-site cluster

On 02/02/2017 04:01 PM, Ilia Sokolinski wrote: Hi, We are testing a multi-site CEPH cluster using 0.94.5 release. There are 2 sites with 2 CEPH nodes in each site. Each node is running a monitor and a bunch of OSDs. The CRUSH rules are configured to require a copy of data in each site. The sites

Re: [ceph-users] RGW authentication fail with AWS S3 v4

It looks like, as it's now coded, the 15 minute time limit is hard coded. It checks that X-Amz-Expires is not exceeded, and then unconditionally checks that the request time is within 15 minutes of now. Daniel On 02/03/2017 04:06 AM, Khang Nguyễn Nhật wrote: Dear Wido, I have used X-Amz

Re: [ceph-users] osd_snap_trim_sleep keeps locks PG during sleep?

After searching the code, osd_snap_trim_cost and osd_snap_trim_priority exist in Master but not in Jewel or Kraken. If osd_snap_trim_sleep was made useless in Jewel by moving snap trimming to the main op thread and no new feature was added to Jewel to allow clusters to throttle snap trimming...

Re: [ceph-users] osd_snap_trim_sleep keeps locks PG during sleep?

They do seem to exist in Jewel. -Sam On Fri, Feb 3, 2017 at 10:12 AM, David Turner wrote: > After searching the code, osd_snap_trim_cost and osd_snap_trim_priority > exist in Master but not in Jewel or Kraken. If osd_snap_trim_sleep was > made useless in Jewel by moving snap trimming to the mai

Re: [ceph-users] osd_snap_trim_sleep keeps locks PG during sleep?

Yep, https://github.com/ceph/ceph/search?utf8=%E2%9C%93 &q=osd_snap_trim_priority And to answer your other question, sort of. I’m using snaps and just about managing with using sleep settings somewhere between

Re: [ceph-users] Experience with 5k RPM/archive HDDs

On 2/3/17, 3:23 AM, "ceph-users on behalf of Wido den Hollander" wrote: > >> Op 3 februari 2017 om 11:03 schreef Maxime Guyot >>: >> >> >> Hi, >> >> Interesting feedback! >> >> > In my opinion the SMR can be used exclusively for the RGW. >> > Unless it's something like a backup/archive clus

Re: [ceph-users] osd_snap_trim_sleep keeps locks PG during sleep?

We found where it is in 10.2.5. It is implemented in the OSD.h file in Jewel, but it is implemented in OSD.cc in Master. We assumed it would be in the same place. We delete over 100TB of snapshots spread across thousands of snapshots every day. We haven't yet found any combination of setting

Re: [ceph-users] osd_snap_trim_sleep keeps locks PG during sleep?

Ok, I'm still working on a branch for master that will introduce limiter on how many pgs can be trimming per osd at once. It should backport trivially to kraken, but jewel will require more work once we've got it in master. Would you be willing to test the master version to determine whether it's

Re: [ceph-users] osd_snap_trim_sleep keeps locks PG during sleep?

We still had blocked requests with osd_snap_trim_cost set to 1GB and osd_snap_trim_priority set to 1 in our test cluster. The test has 20 threads writing to RBD's and 1 thread deleting snapshots on RBD's with an osd_map. The snap_trim_q on the PGs holds at empty on the PGs unless we use osd_sn

Re: [ceph-users] CephFS read IO caching, where it is happining?

I Have opened ticket on http://tracker.ceph.com/ http://tracker.ceph.com/issues/18816 My client and server kernels are the same, here is info: # lsb_release -a LSB Version:n/a Distributor ID: SUSE Description:SUSE Linux Enterprise Server 12 SP2 Release:12.2 Codename: n/a #