Re: [ceph-users] PG Stuck EC Pool

[Ashley Merrick]

>From this extract from pg query:

"up": [
11,
10,
84,
83,
22,
26,
69,
72,
53,
59,
8,
4,
46
],
"acting": [
2147483647,
2147483647,
84,
83,
22,
26,
69,
72,
53,
59,
8,
4,
46

I am wondering if there is an issue on 11 , 10 causing the current active 
primary "acting_primar": 84" to crash.

But can't see anything that could be causing it.

,Ashley

From: Ashley Merrick
Sent: 01 June 2017 23:39
To: ceph-us...@ceph.com
Subject: RE: PG Stuck EC Pool

Have attached the full pg query for the effected PG encase this shows anything 
of interest.

Thanks

From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Ashley 
Merrick
Sent: 01 June 2017 17:19
To: ceph-us...@ceph.com
Subject: [ceph-users] PG Stuck EC Pool


This sender failed our fraud detection checks and may not be who they appear to 
be. Learn about spoofing

Feedback


Have a PG which is stuck in this state (Is an EC with K=10 M=3)





pg 6.14 is active+undersized+degraded+remapped+inconsistent+backfilling, acting 
[2147483647,2147483647,84,83,22,26,69,72,53,59,8,4,46]



Currently have no-recover set, if I unset no recover both OSD 83 + 84 start to 
flap and go up and down, I see the following in the log's of the OSD.



*
-5> 2017-06-01 10:08:29.658593 7f430ec97700  1 -- 172.16.3.14:6806/5204 <== 
osd.17 172.16.3.3:6806/2006016 57  MOSDECSubOpWriteReply(6.31as0 71513 
ECSubWriteReply(tid=152, last_complete=0'0, committed=0, applied=1)) v1  
67+0+0 (245959818 0 0) 0x563c9db7be00 con 0x563c9cfca480
-4> 2017-06-01 10:08:29.658620 7f430ec97700  5 -- op tracker -- seq: 2367, 
time: 2017-06-01 10:08:29.658620, event: queued_for_pg, op: 
MOSDECSubOpWriteReply(6.31as0 71513 ECSubWriteReply(tid=152, last_complete=0'0, 
committed=0, applied=1))
-3> 2017-06-01 10:08:29.658649 7f4319e11700  5 -- op tracker -- seq: 2367, 
time: 2017-06-01 10:08:29.658649, event: reached_pg, op: 
MOSDECSubOpWriteReply(6.31as0 71513 ECSubWriteReply(tid=152, last_complete=0'0, 
committed=0, applied=1))
-2> 2017-06-01 10:08:29.658661 7f4319e11700  5 -- op tracker -- seq: 2367, 
time: 2017-06-01 10:08:29.658660, event: done, op: 
MOSDECSubOpWriteReply(6.31as0 71513 ECSubWriteReply(tid=152, last_complete=0'0, 
committed=0, applied=1))
-1> 2017-06-01 10:08:29.663107 7f43320ec700  5 -- op tracker -- seq: 2317, 
time: 2017-06-01 10:08:29.663107, event: sub_op_applied, op: 
osd_op(osd.79.66617:8675008 6.82058b1a rbd_data.e5208a238e1f29.00025f3e 
[copy-from ver 4678410] snapc 0=[] 
ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected e71513)
 0> 2017-06-01 10:08:29.663474 7f4319610700 -1 *** Caught signal (Aborted) 
**
 in thread 7f4319610700 thread_name:tp_osd_recov

 ceph version 10.2.7 (50e863e0f4bc8f4b9e31156de690d765af245185)
 1: (()+0x9564a7) [0x563c6a6f24a7]
 2: (()+0xf890) [0x7f4342308890]
 3: (gsignal()+0x37) [0x7f434034f067]
 4: (abort()+0x148) [0x7f4340350448]
 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char 
const*)+0x256) [0x563c6a7f83d6]
 6: (ReplicatedPG::recover_replicas(int, ThreadPool::TPHandle&)+0x62f) 
[0x563c6a2850ff]
 7: (ReplicatedPG::start_recovery_ops(int, ThreadPool::TPHandle&, int*)+0xa8a) 
[0x563c6a2b878a]
 8: (OSD::do_recovery(PG*, ThreadPool::TPHandle&)+0x36d) [0x563c6a131bbd]
 9: (ThreadPool::WorkQueue::_void_process(void*, 
ThreadPool::TPHandle&)+0x1d) [0x563c6a17c88d]
 10: (ThreadPool::worker(ThreadPool::WorkThread*)+0xa9f) [0x563c6a7e8e3f]
 11: (ThreadPool::WorkThread::entry()+0x10) [0x563c6a7e9d70]
 12: (()+0x8064) [0x7f4342301064]
 13: (clone()+0x6d) [0x7f434040262d]
 NOTE: a copy of the executable, or `objdump -rdS ` is needed to 
interpret this.
*




What should my next steps be?



Thanks!
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] PG Stuck EC Pool

[Ashley Merrick]

I have now done some further testing and seeing these errors on 84 / 83 the 
OSD's that crash while backfilling to 10,11

   -60> 2017-06-03 10:08:56.651768 7f6f76714700  1 -- 172.16.3.14:6823/2694 <== 
osd.3 172.16.2.101:0/25361 10  osd_ping(ping e71688 stamp 2017-06-03 
10:08:56.652035) v2  47+0+0 (1097709006 0 0) 0x5569ea88d400 con 
0x5569e900e300
   -59> 2017-06-03 10:08:56.651804 7f6f76714700  1 -- 172.16.3.14:6823/2694 --> 
172.16.2.101:0/25361 -- osd_ping(ping_reply e71688 stamp 2017-06-03 
10:08:56.652035) v2 -- ?+0 0x5569e985fc00 con 0x5569e900e300
-6> 2017-06-03 10:08:56.937156 7f6f5ee4d700  1 -- 172.16.3.14:6822/2694 <== 
osd.53 172.16.3.7:6816/15230 13  MOSDECSubOpReadReply(6.14s3 71688 
ECSubReadReply(tid=83, attrs_read=0)) v1  148+0+0 (2355392791 0 0) 
0x5569e8b22080 con 0x5569e9538f00
-5> 2017-06-03 10:08:56.937193 7f6f5ee4d700  5 -- op tracker -- seq: 2409, 
time: 2017-06-03 10:08:56.937193, event: queued_for_pg, op: 
MOSDECSubOpReadReply(6.14s3 71688 ECSubReadReply(tid=83, attrs_read=0))
-4> 2017-06-03 10:08:56.937241 7f6f8ef8a700  5 -- op tracker -- seq: 2409, 
time: 2017-06-03 10:08:56.937240, event: reached_pg, op: 
MOSDECSubOpReadReply(6.14s3 71688 ECSubReadReply(tid=83, attrs_read=0))
-3> 2017-06-03 10:08:56.937266 7f6f8ef8a700  0 osd.83 pg_epoch: 71688 
pg[6.14s3( v 71685'35512 (68694'30812,71685'35512] local-les=71688 n=15928 
ec=31534 les/c/f 71688/69510/67943 71687/71687/71687) 
[11,10,2147483647,83,22,26,69,72,53,59,8,4,46]/[2147483647,2147483647,2147483647,83,22,26,69,72,53,59,8,4,46]
 r=3 lpr=71687 pi=47065-71686/711 rops=1 bft=10(1),11(0) crt=71629'35509 mlcod 
0'0 active+undersized+degraded+remapped+inconsistent+backfilling NIBBLEWISE] 
failed_push 6:28170432:::rbd_data.e3d8852ae8944a.00047d28:head from 
shard 53(8), reps on  unfound? 0
-2> 2017-06-03 10:08:56.937346 7f6f8ef8a700  5 -- op tracker -- seq: 2409, 
time: 2017-06-03 10:08:56.937345, event: done, op: MOSDECSubOpReadReply(6.14s3 
71688 ECSubReadReply(tid=83, attrs_read=0))
-1> 2017-06-03 10:08:56.937351 7f6f89f80700 -1 osd.83 pg_epoch: 71688 
pg[6.14s3( v 71685'35512 (68694'30812,71685'35512] local-les=71688 n=15928 
ec=31534 les/c/f 71688/69510/67943 71687/71687/71687) 
[11,10,2147483647,83,22,26,69,72,53,59,8,4,46]/[2147483647,2147483647,2147483647,83,22,26,69,72,53,59,8,4,46]
 r=3 lpr=71687 pi=47065-71686/711 bft=10(1),11(0) crt=71629'35509 mlcod 0'0 
active+undersized+degraded+remapped+inconsistent+backfilling NIBBLEWISE] 
recover_replicas: object added to missing set for backfill, but is not in 
recovering, error!
   -42> 2017-06-03 10:08:56.968433 7f6f5f04f700  1 -- 172.16.2.114:6822/2694 
<== client.22857445 172.16.2.212:0/2238053329 56  
osd_op(client.22857445.1:759236283 2.e732321d 
rbd_data.61b4c6238e1f29.0001ea27 [set-alloc-hint object_size 4194304 
write_size 4194304,write 126976~45056] snapc 0=[] ondisk+write e71688) v4  
217+0+45056 (2626314663 0 3883338397) 0x5569ea886b00 con 0x5569ea99c880

From: Ashley Merrick
Sent: 03 June 2017 14:27
To: 'ceph-us...@ceph.com' 
Subject: RE: PG Stuck EC Pool

>From this extract from pg query:

"up": [
11,
10,
84,
83,
22,
26,
69,
72,
53,
59,
8,
4,
46
],
"acting": [
2147483647,
2147483647,
84,
83,
22,
26,
69,
72,
53,
59,
8,
4,
46

I am wondering if there is an issue on 11 , 10 causing the current active 
primary "acting_primar": 84" to crash.

But can't see anything that could be causing it.

,Ashley

From: Ashley Merrick
Sent: 01 June 2017 23:39
To: ceph-us...@ceph.com
Subject: RE: PG Stuck EC Pool

Have attached the full pg query for the effected PG encase this shows anything 
of interest.

Thanks

From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Ashley 
Merrick
Sent: 01 June 2017 17:19
To: ceph-us...@ceph.com
Subject: [ceph-users] PG Stuck EC Pool


This sender failed our fraud detection checks and may not be who they appear to 
be. Learn about spoofing

Feedback


Have a PG which is stuck in this state (Is an EC with K=10 M=3)





pg 6.14 is active+undersized+degraded+remapped+inconsistent+backfilling, acting 
[2147483647,2147483647,84,83,22,26,69,72,53,59,8,4,46]



Currently have no-recover set, if I unset no recover both OSD 83 + 84 start to 
flap and go up and down, I see the following in the log's of the OSD.

Re: [ceph-users] PG Stuck EC Pool

[Ashley Merrick]

Attaching with logging to level 20.

After repeat attempts by removing nobackfill I have got it down to:


recovery 31892/272325586 objects degraded (0.012%)
recovery 2/272325586 objects misplaced (0.000%)

However any further attempts after removing nobackfill just causes an instant 
crash on 83 & 84, at this point I feel there is some corruption on the 
remaining 11 OSD's of the PG however the error's aren't directly saying that, 
however always end the crash with:

-1 *** Caught signal (Aborted) ** in thread 7f716e862700 
thread_name:tp_osd_recov

,Ashley

From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Ashley 
Merrick
Sent: 03 June 2017 17:14
To: ceph-us...@ceph.com
Subject: Re: [ceph-users] PG Stuck EC Pool


This sender failed our fraud detection checks and may not be who they appear to 
be. Learn about spoofing

Feedback

I have now done some further testing and seeing these errors on 84 / 83 the 
OSD's that crash while backfilling to 10,11

   -60> 2017-06-03 10:08:56.651768 7f6f76714700  1 -- 172.16.3.14:6823/2694 <== 
osd.3 172.16.2.101:0/25361 10  osd_ping(ping e71688 stamp 2017-06-03 
10:08:56.652035) v2  47+0+0 (1097709006 0 0) 0x5569ea88d400 con 
0x5569e900e300
   -59> 2017-06-03 10:08:56.651804 7f6f76714700  1 -- 172.16.3.14:6823/2694 --> 
172.16.2.101:0/25361 -- osd_ping(ping_reply e71688 stamp 2017-06-03 
10:08:56.652035) v2 -- ?+0 0x5569e985fc00 con 0x5569e900e300
-6> 2017-06-03 10:08:56.937156 7f6f5ee4d700  1 -- 172.16.3.14:6822/2694 <== 
osd.53 172.16.3.7:6816/15230 13  MOSDECSubOpReadReply(6.14s3 71688 
ECSubReadReply(tid=83, attrs_read=0)) v1  148+0+0 (2355392791 0 0) 
0x5569e8b22080 con 0x5569e9538f00
-5> 2017-06-03 10:08:56.937193 7f6f5ee4d700  5 -- op tracker -- seq: 2409, 
time: 2017-06-03 10:08:56.937193, event: queued_for_pg, op: 
MOSDECSubOpReadReply(6.14s3 71688 ECSubReadReply(tid=83, attrs_read=0))
-4> 2017-06-03 10:08:56.937241 7f6f8ef8a700  5 -- op tracker -- seq: 2409, 
time: 2017-06-03 10:08:56.937240, event: reached_pg, op: 
MOSDECSubOpReadReply(6.14s3 71688 ECSubReadReply(tid=83, attrs_read=0))
-3> 2017-06-03 10:08:56.937266 7f6f8ef8a700  0 osd.83 pg_epoch: 71688 
pg[6.14s3( v 71685'35512 (68694'30812,71685'35512] local-les=71688 n=15928 
ec=31534 les/c/f 71688/69510/67943 71687/71687/71687) 
[11,10,2147483647,83,22,26,69,72,53,59,8,4,46]/[2147483647,2147483647,2147483647,83,22,26,69,72,53,59,8,4,46]
 r=3 lpr=71687 pi=47065-71686/711 rops=1 bft=10(1),11(0) crt=71629'35509 mlcod 
0'0 active+undersized+degraded+remapped+inconsistent+backfilling NIBBLEWISE] 
failed_push 6:28170432:::rbd_data.e3d8852ae8944a.00047d28:head from 
shard 53(8), reps on  unfound? 0
-2> 2017-06-03 10:08:56.937346 7f6f8ef8a700  5 -- op tracker -- seq: 2409, 
time: 2017-06-03 10:08:56.937345, event: done, op: MOSDECSubOpReadReply(6.14s3 
71688 ECSubReadReply(tid=83, attrs_read=0))
-1> 2017-06-03 10:08:56.937351 7f6f89f80700 -1 osd.83 pg_epoch: 71688 
pg[6.14s3( v 71685'35512 (68694'30812,71685'35512] local-les=71688 n=15928 
ec=31534 les/c/f 71688/69510/67943 71687/71687/71687) 
[11,10,2147483647,83,22,26,69,72,53,59,8,4,46]/[2147483647,2147483647,2147483647,83,22,26,69,72,53,59,8,4,46]
 r=3 lpr=71687 pi=47065-71686/711 bft=10(1),11(0) crt=71629'35509 mlcod 0'0 
active+undersized+degraded+remapped+inconsistent+backfilling NIBBLEWISE] 
recover_replicas: object added to missing set for backfill, but is not in 
recovering, error!
   -42> 2017-06-03 10:08:56.968433 7f6f5f04f700  1 -- 172.16.2.114:6822/2694 
<== client.22857445 172.16.2.212:0/2238053329 56  
osd_op(client.22857445.1:759236283 2.e732321d 
rbd_data.61b4c6238e1f29.0001ea27 [set-alloc-hint object_size 4194304 
write_size 4194304,write 126976~45056] snapc 0=[] ondisk+write e71688) v4  
217+0+45056 (2626314663 0 3883338397) 0x5569ea886b00 con 0x5569ea99c880

From: Ashley Merrick
Sent: 03 June 2017 14:27
To: 'ceph-us...@ceph.com' mailto:ceph-us...@ceph.com>>
Subject: RE: PG Stuck EC Pool

>From this extract from pg query:

"up": [
11,
10,
84,
83,
22,
26,
69,
72,
53,
59,
8,
4,
46
],
"acting": [
2147483647,
2147483647,
84,
83,
22,
26,
69,
72,
53,
59,
8,
4,
46

I am wondering if there is an issue on 11 , 10 causing the current active 
primary "acting_primar": 84" to crash.

But can't see anything that could be causing it.

,Ashl

[ceph-users] Reg: Request blocked issue

[Shuresh]

Hi,

Am using the ceph past 3 years, Now Currently in jewel version, I am facing
the issue "request blocked" in my cluster, we already faced these issues on
that time we have removed the problem OSD from the cluster, but now am
receving it from 4 osd at the same time, please find the screent shot of
that,.

Current configuration - 16 OSDs in 8 Node cluster

[image: Inline images 1]

6 ops are blocked > 33554.4 sec on osd.7
100 ops are blocked > 16777.2 sec on osd.9
3 ops are blocked > 33554.4 sec on osd.1
24 ops are blocked > 16777.2 sec on osd.5

Log from OSD.9

2017-06-03 17:20:44.689027 7fcb653c6700  0 -- 10.10.2.131:6800/3782 >>
10.10.2.83:0/2228271579 pipe(0x558fceaea000 sd=289 :6800 s=0 pgs=0 cs=0 l=1
c=0x558fd12c6300).accept replacing existing (lossy) channel (new one
lossy=1)
2017-06-03 17:20:48.102174 7fcb651c4700  0 -- 10.10.2.131:6800/3782 >>
10.10.2.147:0/2527575508 pipe(0x558fa9265400 sd=290 :6800 s=0 pgs=0 cs=0
l=1 c=0x558fd12d7980).accept replacing existing (lossy) channel (new one
lossy=1)
2017-06-03 17:20:53.630144 7fcb64fc2700  0 -- 10.10.2.131:6800/3782 >>
10.10.2.124:6800/989 pipe(0x558fbef0c000 sd=291 :6800 s=0 pgs=0 cs=0 l=1
c=0x558fb8f80300).accept replacing existing (lossy) channel (new one
lossy=1)
2017-06-03 17:21:38.569943 7fcb64dc0700  0 -- 192.168.10.131:6800/3782 >>
192.168.10.128:6802/7686 pipe(0x558fd2198800 sd=292 :6800 s=0 pgs=0 cs=0
l=0 c=0x558faa0aa600).accept connect_seq 3 vs existing 3 state standby
2017-06-03 17:21:38.570449 7fcb64dc0700  0 -- 192.168.10.131:6800/3782 >>
192.168.10.128:6802/7686 pipe(0x558fd2198800 sd=292 :6800 s=0 pgs=0 cs=0
l=0 c=0x558faa0aa600).accept connect_seq 4 vs existing 3 state standby


Regards
Shureshbabu
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Recovering PGs from Dead OSD disk

[James Horner]

Hi All

Thanks in advance for any help, I was wondering if anyone can help me with
a pickle I have gotten myself into!

I was in the process of adding OSD's to my small cluster (6 OSDs) and the
disk died halfway through, unfort I had left the defaults from when I was
bootstrapping the cluster in place which meant that size=1. I have managed
to mount the old OSD disk and copy off all the files in the root and some
of the files under current but when I try and recover the PG's using
ceph-objectstore-tool I get the following:

ceph-objectstore-tool --op export --pgid 7.cb --data-path /mnt/temp
--journal-path /mnt/temp/journal --skip-journal-replay --debug --file
7.cb.export
2017-06-03 14:03:39.906546 7f2b217cc940  0 filestore(/mnt/temp) backend xfs
(magic 0x58465342)
2017-06-03 14:03:39.907153 7f2b217cc940  0
genericfilestorebackend(/mnt/temp) detect_features: FIEMAP ioctl is
disabled via 'filestore fiemap' config option
2017-06-03 14:03:39.907162 7f2b217cc940  0
genericfilestorebackend(/mnt/temp) detect_features: SEEK_DATA/SEEK_HOLE is
disabled via 'filestore seek data hole' config option
2017-06-03 14:03:39.907193 7f2b217cc940  0
genericfilestorebackend(/mnt/temp) detect_features: splice is supported
2017-06-03 14:03:39.999729 7f2b217cc940  0
genericfilestorebackend(/mnt/temp) detect_features: syncfs(2) syscall fully
supported (by glibc and kernel)
2017-06-03 14:03:39.999865 7f2b217cc940  0 xfsfilestorebackend(/mnt/temp)
detect_feature: extsize is disabled by conf
2017-06-03 14:03:40.000392 7f2b217cc940 -1 filestore(/mnt/temp) mount
initial op seq is 0; something is wrong
Mount failed with '(22) Invalid argument'

I know there is data in pg 7.cb as:
$ du -sh  /mnt/temp/current/7.cb*
15G/mnt/temp/current/7.cb_head
0/mnt/temp/current/7.cb_TEMP

I am unsure why it is trying to mount anything as the disk is already
mounted. Is one of the disabled config options stopping me from completing
this or am I missing something else? This pool was used for CephFS data
(pool 6 is the metadata and in a similar state).  I have only copied the pg
folders for the relevant pools, there are others on there but I think they
might be stored on the damaged section of the disk.

Thanks for any help
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] is there any way to speed up cache evicting?

[David Turner]

I set all of my target ratios to 0.0 and that is the one I meant.

On Fri, Jun 2, 2017, 11:11 PM jiajia zhong  wrote:

> david,
>
> 2017-06-02 21:41 GMT+08:00 David Turner :
>
>> I'm thinking you have erasure coding in cephfs and only use cache tiring
>> because you have to, correct? What is your use case for repeated file
>> accesses? How much data is written into cephfs at a time?
>>
> these days, up to ten-millions of tiny files were written into  cephfs
> everyday; not directly but via samba, some processes are running on
> windows, so there are some linux systems run samba service, windows
> processes writes through samba. after data written,  they are rarely
> accessed; at last, they were uploaded to a cloud storage outside.
>
>> For me, my files are infrequently accessed after they are written or read
>> from the EC back-end pool.  I set my cache pool to never leave any data in
>> it older than an hour. I have a buddy with a similar setup with the
>> difference that files added to cephfs will be heavily accessed and modified
>> for the first day, but then intently accessed. He has his settings to keep
>> all of his accessed files in cache for 24 hours before they are all cleaned
>> up.
>>
>> We do that by having a target_max_ratio of 0.0 and min_evict and
>> min_flush ages set appropriately. The cluster is never chosing what to
>> flush or evict based on maintaining the full ratio. As soon as the minimum
>> age are met, it does it is added to the queue to process.
>>
> I can't find target_max_ratio, do you mean cache_target_full_ratio ?
>
> I though cache_target_full_ratio only  concerned the clean objects
> (evict), any misunderstanding ?
>
> This fixed our cluster speeds during times when the cache pool was
>> cleaning up. The problem we hypothesized was that it was the prices of
>> choosing what to clean vs what to keep was causing.
>>
>> On Fri, Jun 2, 2017, 4:54 AM jiajia zhong  wrote:
>>
>>> thank you for your guide :)， It's making sense.
>>>
>>> 2017-06-02 16:17 GMT+08:00 Christian Balzer :
>>>

 Hello,

 On Fri, 2 Jun 2017 14:30:56 +0800 jiajia zhong wrote:

 > christian, thanks for your reply.
 >
 > 2017-06-02 11:39 GMT+08:00 Christian Balzer :
 >
 > > On Fri, 2 Jun 2017 10:30:46 +0800 jiajia zhong wrote:
 > >
 > > > hi guys:
 > > >
 > > > Our ceph cluster is working with tier cache.
 > > If so, then I suppose you read all the discussions here as well and
 not
 > > only the somewhat lacking documentation?
 > >
 > > > I am running "rados -p data_cache cache-try-flush-evict-all" to
 evict all
 > > > the objects.
 > > Why?
 > > And why all of it?
 >
 >
 > we found that when the threshold(flush/evict) was triggered, the
 > performance would make us a bit upset :), so I wish to flush/evict
 the tier
 > in a spare time,eg, middle night，In this scenario，the tier could not
 pay
 > any focus on flush/evict while the great w/r operations on cephfs
 which we
 > are using.
 >
 As I said, eviction (which is basically zeroing the data in cache) has
 very little impact.
 Flushing, moving data from the cache tier to the main pool is another
 story.

 But what you're doing here is completely invalidating your cache (the
 eviction part), so the performance will be very bad after this as well.

 If you have low utilization periods, consider a cron job that lowers the
 dirty ratio (causing only flushes to happen) and then after a while (few
 minutes should do, experiment) restore the old setting.

 For example:
 ---
 # Preemptive Flushing before midnight
 45 23 * * * root ceph osd pool set cache cache_target_dirty_ratio 0.52

 # And back to normal levels
 55 23 * * * root ceph osd pool set cache cache_target_dirty_ratio 0.60
 ---

 This will of course only help if the amount of data promoted into your
 cache per day is small enough to fit into the flushed space.

 Otherwise your cluster has no other choice to start flushing when things
 get full.

 Christian
 > >
 > > > But It a bit slow
 > > >
 > > Define slow, but it has to do a LOT of work and housekeeping to do
 this,
 > > so unless your cluster is very fast (probably not, or you wouldn't
 > > want/need a cache tier) and idle, that's the way it is.
 > >
 > > > 1. Is there any way to speed up the evicting?
 > > >
 > > Not really, see above.
 > >
 > > > 2. Is evicting triggered by itself good enough for cluster ?
 > > >
 > > See above, WHY are you manually flushing/evicting?
 > >
 > explained above.
 >
 >
 > > Are you aware that flushing is the part that's very I/O intensive,
 while
 > > evicting is a very low cost/impact operation?
 > >
 > not very sure, my instinct believed those.
 >
 >
 > > In normal produ

Re: [ceph-users] Bug report: unexpected behavior when executing Lua object class

[Noah Watkins]

Comments inline

> -- Forwarded message --
> From: Zheyuan Chen 
> Date: Sat, Jun 3, 2017 at 1:45 PM
> Subject: [ceph-users] Bug report: unexpected behavior when executing
> Lua object class
> To: ceph-users@lists.ceph.com
>
> Bug 1: I can not get returned output in the first script. "data" is
> always empty.
>
>> import rados, json
>> cluster = rados.Rados(conffile='')
>> cluster.connect()
>> ioctx = cluster.open_ioctx('data')
>> cmd = {
>> "script": """
>> function test(input, output)
>> data = bufferlist.new()
>> data:append("test")
>> objclass.write_full(data, #data)
>> output:append("Here is the return value")
>> end
>> objclass.register(test)
>> """,
>> "handler": "test",
>> "input": "",
>> }
>> # I can not get returned data
>> ret, data = ioctx.execute('test', 'lua', 'eval_json', json.dumps(cmd))
>> print ret, data

Unfortunately, this isn't a bug. Rados clears any returned data from
an object class method if the operation also writes to the object.

> Bug 2:
> The returned mtime is always 0.

This might be a bug, but there are a few things to rule out. Did you
write to the object before running stat? There is also a test case
here: can you try to run them?

https://github.com/ceph/ceph/blob/master/src/test/cls_lua/test_cls_lua.cc#L679

- Noah
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com