Hi clay I just using the command like `swift-ring-builder object.builder set_weight d234 2220`. Then execute `swift-ring-builder object.builder rebalance` and push the rings all object node.
Here is more detail information. I/O scheduler : deadline OS : CentOS 7.4 Kernel version : 3.10.0-693.11.1.el7.x86_64 I will try decrease the node_timeout from 3 to 1. (Do you have any recommend value?) On Tue, Oct 30, 2018 at 1:42 AM Clay Gerrard <clay.gerr...@gmail.com> wrote: > Obviously a re-balance will cost some IO, but it's normally perceptible to > the client unless you were already on a razor thin line. > > Two config options seem obvious to think about experimenting with: > > You could decrease the node_timeout and let the proxy try to write more to > handoffs > You could try to use some of the ionice options (or other tuning options) > to make the replicators hammer the disks a little less > > It might having something to do with how you're organizing your ring > weight changes - could you describe how you're managing rings? > Could also be a io scheduling issue - are you using noop/deadline or cfq? > What kernel version? > > -Clay > > > On Mon, Oct 29, 2018 at 9:55 AM SeongSoo Cho <ppiya...@printf.kr> wrote: > >> For more information, I use ocata version. >> >> 2018년 10월 29일 (월) 오후 10:07, SeongSoo Cho <ppiya...@printf.kr>님이 작성: >> >>> Hello, All >>> >>> I have a terrible problem with object server. >>> Here is the case. >>> 1. User upload an object to proxy-server >>> 2. Proxy server try to connect with object-server >>> 3. If one of object-server is slow to respond, proxy-server is waiting >>> for response. >>> 3.1 While waiting for response, proxy-server can't do anything >>> 4. So, The response of client request will be delayed. >>> >>> In my opinion, this code seems to be a problem >>> ( >>> https://github.com/openstack/swift/blob/stable/rocky/swift/proxy/controllers/obj.py#L1734 >>> ) >>> >>> ``` >>> with ResponseTimeout(node_timeout): >>> resp = conn.getexpect() >>> ``` >>> >>> If node_timeout's value is 3 and object-server respond after 2 seconds, >>> proxy-server wait 2 seconds. >>> >>> Because proxy-server wait for the above response, the execution of the >>> following code is delayed. >>> ( >>> https://github.com/openstack/swift/blob/stable/rocky/swift/proxy/controllers/obj.py#L627 >>> ) >>> >>> ``` >>> for node in nodes: >>> try: >>> putter = self._make_putter(node, part, req, headers) >>> self.app.set_node_timing(node, putter.connect_duration) >>> return putter >>> ``` >>> >>> This problem occurs when i do a ring rebalance. >>> When object-replicator delete a partition directory that are no longer >>> mine, the disk becomes very busy (Because of xfsaild daemon) >>> Because the disk are busy, object-server can't create diskfile during >>> PUT operation. >>> >>> Is there anyone who is having problems like me? >>> How can I solve this problem? >>> >>> I need everyone's help. >>> Thanks. >>> >>> Best Regards >>> SeongSoo Cho >>> >>> ------ >>> SeongSoo Cho (South Korea) >>> >>> _______________________________________________ >> Mailing list: >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack >> Post to : openstack@lists.openstack.org >> Unsubscribe : >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack >> >
_______________________________________________ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
