...@fisk.me.uk; 'Peter Maloney'
Cc: ceph-users@lists.ceph.com
Subject: RE: [ceph-users] ceph cluster having blocke requests very frequently
Hi Nick,
We have recently increase osd op threads from 2 (default value) to 16 because
CPU usage on DN was very low.
We have the impress
limit its
impact.
Thomas
From: Nick Fisk [mailto:n...@fisk.me.uk]
Sent: mercredi 23 novembre 2016 14:09
To: Thomas Danan; 'Peter Maloney'
Cc: ceph-users@lists.ceph.com
Subject: RE: [ceph-users] ceph cluster having blocke requests very frequently
Hi Thomas,
I’m afraid I can’t off
On 11/26/16 09:52, Peter Maloney wrote:
> On 11/18/16 23:15, Peter Maloney wrote:
>> BTW, my rebalance finished, and I guess the performance is a bit
>> better, with load distributed a bit better, but blocked requests still
>> happen if I use snapshot create + export-diff + delete snapshot, and
>>
On 11/18/16 23:15, Peter Maloney wrote:
>
> BTW, my rebalance finished, and I guess the performance is a bit
> better, with load distributed a bit better, but blocked requests still
> happen if I use snapshot create + export-diff + delete snapshot, and
> make qemu clients hang the same. A 30s sleep
mon_osd_min_down_reports = 10
Thomas
From: David Turner [mailto:david.tur...@storagecraft.com]
Sent: mercredi 23 novembre 2016 21:27
To: n...@fisk.me.uk; Thomas Danan; 'Peter Maloney'
Cc: ceph-users@lists.ceph.com
Subject: RE: [ceph-users] ceph cluster having blocke requests very frequently
T
24 novembre 2016 01:42
To: Thomas Danan
Cc: n...@fisk.me.uk; Peter Maloney; ceph-users@lists.ceph.com
Subject: Re: [ceph-users] ceph cluster having blocke requests very frequently
Hi Thomas,
do you have any RBD created as clone from another snapshot? If yes then this
would mean you still have some
8 pipe(0x1271b000 sd=34 :50711 s=2 pgs=647
> cs=5 l=0 c=0x42798c0).fault, initiating reconnect
>
>
>
> I do not manage to identify anything obvious in the logs.
>
>
>
> Thanks for your help …
>
>
>
> Thomas
>
>
>
>
>
> *From:* Nick Fisk [
age to try that?
Nick
From: Thomas Danan [mailto:thomas.da...@mycom-osi.com]
Sent: 23 November 2016 11:29
To: n...@fisk.me.uk; 'Peter Maloney'
Cc: ceph-users@lists.ceph.com
Subject: RE: [ceph-users] ceph cluster having blocke requests very frequently
Hi all,
Still not able to find any explanation to
...@fisk.me.uk; 'Peter Maloney'
Cc: ceph-users@lists.ceph.com
Subject: RE: [ceph-users] ceph cluster having blocke requests very frequently
Hi all,
Still not able to find any explanation to this issue.
I recently tested the network and I am seeing some retransmit being d
17:12
To: 'n...@fisk.me.uk'; 'Peter Maloney'
Cc: ceph-users@lists.ceph.com
Subject: RE: [ceph-users] ceph cluster having blocke requests very frequently
Hi Nick,
Here are some logs. The system is in IST TZ and I have filtered the logs to get
only 2 last hours during whic
On 11/18/16 18:00, Thomas Danan wrote:
>
> I often read that small IO write and RBD are working better with
> bigger filestore_max_sync_interval than default value.
>
> Default value is 5 sec and I saw many post saying they are using 30 sec.
>
> Also the slow request symptom is often linked to this
online ?
Thanks
Thomas
From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Thomas
Danan
Sent: vendredi 18 novembre 2016 12:42
To: n...@fisk.me.uk; 'Peter Maloney'
Cc: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] ceph cluster having blocke requests very freq
entify anything obvious in the logs.
Thanks for your help …
Thomas
From: Nick Fisk [mailto:n...@fisk.me.uk]
Sent: jeudi 17 novembre 2016 11:02
To: Thomas Danan; n...@fisk.me.uk; 'Peter Maloney'
Cc: ceph-users@lists.ceph.com
Subject: RE: [ceph-users] ceph cluster having blocke requests very f
2016 08:59
To: n...@fisk.me.uk; 'Peter Maloney'
Cc: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] ceph cluster having blocke requests very frequently
Hi,
I have recheck the pattern when slow request are detected.
I have an example with following (primary: 411, secondary
;
Cc: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] ceph cluster having blocke requests very frequently
Hi,
I have recheck the pattern when slow request are detected.
I have an example with following (primary: 411, secondary: 176, 594)
On primary slow requests detected: waiting for subops
n...@fisk.me.uk]
Sent: mercredi 16 novembre 2016 22:13
To: Thomas Danan; n...@fisk.me.uk; 'Peter Maloney'
Cc: ceph-users@lists.ceph.com
Subject: RE: [ceph-users] ceph cluster having blocke requests very frequently
Hi,
Yes, I can’t think of anything else at this stage. Could you maybe
November 2016 17:38
To: n...@fisk.me.uk; 'Peter Maloney'
Cc: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] ceph cluster having blocke requests very frequently
Hi Nick,
We have deleted all Snapshots and observed the system for several hours.
>From what I see this did not help
everything seems fine …
Thomas
From: Nick Fisk [mailto:n...@fisk.me.uk]
Sent: mercredi 16 novembre 2016 14:01
To: Thomas Danan; 'Peter Maloney'
Cc: ceph-users@lists.ceph.com
Subject: RE: [ceph-users] ceph cluster having blocke requests very frequently
The snapshot works by using Copy On Wri
Danan ; n...@fisk.me.uk; 'Peter Maloney'
Cc: ceph-users@lists.ceph.com
Subject: RE: [ceph-users] ceph cluster having blocke requests very frequently
Hi Nick,
Actually I was wondering, is there any difference between Snapshot or simple
RBD image ?
With simple RBD image when doin
From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Thomas
Danan
Sent: mercredi 16 novembre 2016 13:52
To: n...@fisk.me.uk; 'Peter Maloney'
Cc: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] ceph cluster having blocke requests very frequently
Hi Nick,
Yes our a
freeze on client
side.
Thanks
Thomas
From: Nick Fisk [mailto:n...@fisk.me.uk]
Sent: mercredi 16 novembre 2016 13:25
To: Thomas Danan; 'Peter Maloney'
Cc: ceph-users@lists.ceph.com
Subject: RE: [ceph-users] ceph cluster having blocke requests very frequently
From: ceph-users [mailto:
From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Thomas
Danan
Sent: 15 November 2016 21:14
To: Peter Maloney
Cc: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] ceph cluster having blocke requests very frequently
Very interesting ...
Any idea why optimal
6 different ceph RBD clients.
> Snapshoting the RBD image is quite immediate while we are seing the
> issue continuously during the day...
>
> Will check all of this tomorrow . ..
>
> Thanks again
>
> Thomas
>
>
>
> Sent from my Samsung device
>
>
> ---
Original message
From: Peter Maloney
Date: 11/15/16 21:27 (GMT+01:00)
To: Thomas Danan
Cc: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] ceph cluster having blocke requests very frequently
On 11/15/16 14:05, Thomas Danan wrote:
> Hi Peter,
>
> Ceph cluster version is 0.94
On 11/15/16 14:05, Thomas Danan wrote:
> Hi Peter,
>
> Ceph cluster version is 0.94.5 and we are running with Firefly tunables and
> also we have 10KPGs instead of the 30K / 40K we should have.
> The linux kernel version is 3.10.0-327.36.1.el7.x86_64 with RHEL 7.2
>
> On our side we havethe follow
sure I want
to change it base on your experience
Thomas
-Original Message-
From: Peter Maloney [mailto:peter.malo...@brockmann-consult.de]
Sent: mardi 15 novembre 2016 13:44
To: Thomas Danan
Cc: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] ceph cluster having blocke requests very
Which kernel version are you using?
I have a similar issue..ubuntu 14.04 kernel 3.13.0-96-generic, and ceph
jewel 10.2.3.
I get logs like this:
2016-11-15 13:13:57.295067 osd.9 10.3.0.132:6817/24137 98 : cluster
[WRN] 16 slow requests, 5 included below; oldest blocked for > 7.957045 secs
I set o
rs ?
Thanks
Thomas
From: Chris Taylor [mailto:ctay...@eyonic.com]
Sent: mardi 15 novembre 2016 00:54
To: Brad Hubbard
Cc: Thomas Danan; ceph-users@lists.ceph.com
Subject: Re: [ceph-users] ceph cluster having blocke requests very frequently
Maybe a long shot, but have you checked OSD memor
..@gmail.com]
> SENT: lundi 14 novembre 2016 16:23
> TO: Thomas Danan
> CC: ceph-users@lists.ceph.com
> SUBJECT: Re: [ceph-users] ceph cluster having blocke requests very frequently
>
> Without knowing the cluster architecture it's hard to know exactly what may
> be happ
>
>
>
> Thomas
>
>
>
>
>
> *From:* Luis Periquito [mailto:periqu...@gmail.com]
> *Sent:* lundi 14 novembre 2016 16:23
> *To:* Thomas Danan
> *Cc:* ceph-users@lists.ceph.com
> *Subject:* Re: [ceph-users] ceph cluster having blocke requests very
> frequ
ing replicas.
Thomas
From: Luis Periquito [mailto:periqu...@gmail.com]
Sent: lundi 14 novembre 2016 16:23
To: Thomas Danan
Cc: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] ceph cluster having blocke requests very frequently
Without knowing the cluster architecture it's hard to know
Without knowing the cluster architecture it's hard to know exactly what may
be happening.
How is the cluster hardware? Where are the journals? How busy are the disks
(% time busy)? What is the pool size? Are these replicated or EC pools?
Have you tried tuning the deep-scrub processes? Have you tr
32 matches
Mail list logo
