When checking the RGW logs I can confirm that it is in fact the same issue as the one in the issue.
2023-09-20T12:52:06.670+0000 7f216d702700 1 -- xxx.xxx.58.15:0/758879303 --> [v2:xxx.xxx.58.2:6816/8556,v1:xxx.xxx.58.2:6817/8556] -- osd_op(unknown.0.0:238 18.651 18:8a75a7b2:::39078a70-7768-48c8-96a5-1e13ced83b5b.58017020.1_videos%2f7.mp4:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e60419) v8 -- 0x7f21dc00a420 con 0x7f21dc007820 You can find the OSD part of the log here - https://pastebin.com/nGQw4ugd For the record the version of the cluster when I’m able to replicate this is ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable) Regards, Ondrej > On 20. 9. 2023, at 11:25, Ondřej Kukla <ond...@kuuk.la> wrote: > > I was checking the tracker again and I found already fixed issue that seems > to be connected with this issue. > > https://tracker.ceph.com/issues/44508 > > Here is the PR that fixes it https://github.com/ceph/ceph/pull/33807 > > What I’m still not understanding is why this is only happening when using > s3website api. > > Is there someone who could shed some light on this? > > Regards, > > Ondrej > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io > On 19. 9. 2023, at 10:49, Ondřej Kukla <ond...@kuuk.la> wrote: > > Hello, > > In our deployment we are using the mix of s3 and s3website RGW. I’ve noticed > strange behaviour when sending range requests to the s3website RGWs that I’m > not able to replicate on the s3 ones. > > I’ve created a simple wrk LUA script to test sending range requests on tiny > ranges so the issue is easily seen. > > When sending these requests against s3 RGW I can see that the amount of data > read from Ceph is ± equivalent to what the RGW sends to the client. This > change very dramatically when I’m doing the same test against s3website RGW. > The read from Ceph is huge (3Gb/s compared to ~22Mb/s on s3 RGW) I seems to > me like the RGW is reading the whole files and then sending just the range > which is different then what s3 does. > > I do not understand why would s3website need to read that much from Ceph and > I believe this is a bug - I was looking through the tracker and wasn’t able > to find anything related to s3website and range requests. > > Did anyone else noticed this issue? > > You can replicate it by running this wrk command wrk -t56 -c500 -d5m > http://${rgwipaddress}:8080/${bucket}/videos/ -s wrk-range-small.lua > > wrk script > > -- Initialize the pseudo random number generator > math.randomseed( os.time()) > math.random(); math.random(); math.random() > > i = 1 > > function request() > if i == 8 > then > i = 1 > end > > local nrangefrom = math.random() > local nrangeto = math.random(100) > local path = wrk.path > url = path..i..".mp4" > wrk.headers["Range"] = nrangefrom.."-"..nrangeto > i = i+1 > return wrk.format(nil, url) > end > > Kind regards, > > Ondrej > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
