Hi Clay, Hi All,
Configured my cluster from ground up with one proxy node and one storage
node.
Now I am getting two types of errors.
Large files:
Feb 3 18:12:33 centos7-swift-proxy1 swift-proxy-server: ERROR with Object
server 192.168.47.128:6201/sda re: Trying to write to /AUTH_admin/ara1/abc:
#012Traceback (most recent call last):#012 File
"/usr/lib/python2.7/site-packages/swift/proxy/controllers/obj.py", line
1617, in _send_file#012 self.conn.send(to_send)#012 File
"/usr/lib64/python2.7/httplib.py", line 840, in send#012
self.sock.sendall(data)#012 File
"/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 393, in
sendall#012 tail += self.send(data[tail:], flags)#012 File
"/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 384, in
send#012 return self._send_loop(self.fd.send, data, flags)#012 File
"/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 371, in
_send_loop#012 return send_method(data, *args)#012error: [Errno 32]
Broken pipe
Feb 3 18:12:33 centos7-swift-proxy1 swift-proxy-server: Object 1 PUT
exceptions during send, 0/1 required connections (txn:
tx0b01a226078f47b4b3593-005a7641e1) (client_ip: 192.168.47.132)
Large file error on storage node:
Feb 4 19:15:51 centos7-swift-node1 container-server: STDERR: (112666)
accepted ('192.168.47.132', 35624)
Feb 4 19:15:51 centos7-swift-node1 container-server: 192.168.47.132 - -
[05/Feb/2018:00:15:51 +0000] "PUT /sda/89/AUTH_admin/ara1" 202 - "PUT
http://192.168.47.132:8080/v1/AUTH_admin/ara1";
"tx158875c238214209bdfc3-005a7646ab" "proxy-server 44255" 0.0063 "-" 112666
0
Feb 4 19:15:51 centos7-swift-node1 container-server: STDERR:
192.168.47.132 - - [05/Feb/2018 00:15:51] "PUT /sda/89/AUTH_admin/ara1
HTTP/1.1" 202 252 0.006706 (txn: tx158875c238214209bdfc3-005a7646ab)
Feb 4 19:15:51 centos7-swift-node1 container-server: STDERR: (112666)
accepted ('192.168.47.132', 35630)
Feb 4 19:15:51 centos7-swift-node1 container-server: 192.168.47.132 - -
[05/Feb/2018:00:15:51 +0000] "HEAD /sda/89/AUTH_admin/ara1" 204 - "HEAD
http://192.168.47.132:8080/v1/AUTH_admin/ara1";
"txc23a59c70e8940bf85101-005a7646ab" "proxy-server 44253" 0.0011 "-" 112666
0
Feb 4 19:15:51 centos7-swift-node1 container-server: STDERR:
192.168.47.132 - - [05/Feb/2018 00:15:51] "HEAD /sda/89/AUTH_admin/ara1
HTTP/1.1" 204 521 0.001422 (txn: txc23a59c70e8940bf85101-005a7646ab)
Feb 4 19:15:51 centos7-swift-node1 container-server: STDERR: (112666)
accepted ('192.168.47.132', 35632)
Feb 4 19:15:51 centos7-swift-node1 container-server: 192.168.47.132 - -
[05/Feb/2018:00:15:51 +0000] "HEAD /sda/75/AUTH_admin/ara1/abc" 404 - "HEAD
http://192.168.47.132:8080/v1/AUTH_admin/ara1/abc";
"txc23a59c70e8940bf85101-005a7646ab" "proxy-server 44253" 0.0003 "-" 112666
0
Feb 4 19:15:51 centos7-swift-node1 container-server: STDERR:
192.168.47.132 - - [05/Feb/2018 00:15:51] "HEAD /sda/75/AUTH_admin/ara1/abc
HTTP/1.1" 404 351 0.000549 (txn: txc23a59c70e8940bf85101-005a7646ab)
Feb 4 19:15:51 centos7-swift-node1 container-server: STDERR: (112666)
accepted ('192.168.47.132', 35634)
Feb 4 19:15:51 centos7-swift-node1 container-server: 192.168.47.132 - -
[05/Feb/2018:00:15:51 +0000] "PUT /sda/75/AUTH_admin/ara1/abc" 404 - "PUT
http://192.168.47.132:8080/v1/AUTH_admin/ara1/abc";
"tx5138df5cf1f840808d2b2-005a7646ab" "proxy-server 44253" 0.0002 "-" 112666
0
Feb 4 19:15:51 centos7-swift-node1 container-server: STDERR:
192.168.47.132 - - [05/Feb/2018 00:15:51] "PUT /sda/75/AUTH_admin/ara1/abc
HTTP/1.1" 404 212 0.000506 (txn: tx5138df5cf1f840808d2b2-005a7646ab)
Feb 4 19:15:52 centos7-swift-node1 container-server: STDERR: (112666)
accepted ('192.168.47.132', 35636)
Feb 4 19:15:52 centos7-swift-node1 container-server: 192.168.47.132 - -
[05/Feb/2018:00:15:52 +0000] "PUT /sda/75/AUTH_admin/ara1/abc" 404 - "PUT
http://192.168.47.132:8080/v1/AUTH_admin/ara1/abc";
"txeb1c76b896854e4885009-005a7646ac" "proxy-server 44253" 0.0002 "-" 112666
0
Feb 4 19:15:52 centos7-swift-node1 container-server: STDERR:
192.168.47.132 - - [05/Feb/2018 00:15:52] "PUT /sda/75/AUTH_admin/ara1/abc
HTTP/1.1" 404 212 0.000589 (txn: txeb1c76b896854e4885009-005a7646ac)
Feb 4 19:15:54 centos7-swift-node1 container-server: STDERR: (112666)
accepted ('192.168.47.132', 35638)
Feb 4 19:15:54 centos7-swift-node1 container-server: 192.168.47.132 - -
[05/Feb/2018:00:15:54 +0000] "PUT /sda/75/AUTH_admin/ara1/abc" 404 - "PUT
http://192.168.47.132:8080/v1/AUTH_admin/ara1/abc";
"tx1a011b125608447ca36a8-005a7646ae" "proxy-server 44253" 0.0002 "-" 112666
0
Feb 4 19:15:54 centos7-swift-node1 container-server: STDERR:
192.168.47.132 - - [05/Feb/2018 00:15:54] "PUT /sda/75/AUTH_admin/ara1/abc
HTTP/1.1" 404 212 0.000569 (txn: tx1a011b125608447ca36a8-005a7646ae)
or
Smaller files:
Feb 3 18:30:40 centos7-swift-proxy1 swift-proxy-server: ERROR Unhandled
exception in request: #012Traceback (most recent call last):#012 File
"/usr/lib/python2.7/site-packages/swift/proxy/server.py", line 521, in
handle_request#012 return handler(req)#012 File
"/usr/lib/python2.7/site-packages/swift/proxy/controllers/base.py", line
283, in wrapped#012 return func(*a, **kw)#012 File
"/usr/lib/python2.7/site-packages/swift/proxy/controllers/obj.py", line
745, in PUT#012 req, data_source, nodes, partition,
outgoing_headers)#012 File
"/usr/lib/python2.7/site-packages/swift/proxy/controllers/obj.py", line
949, in _store_object#012 self._get_put_responses(req, putters,
len(nodes))#012 File
"/usr/lib/python2.7/site-packages/swift/proxy/controllers/obj.py", line
406, in _get_put_responses#012 _handle_response(putter, response)#012
File "/usr/lib/python2.7/site-packages/swift/proxy/controllers/obj.py",
line 402, in _handle_response#012
etags.add(response.getheader('etag').strip('"'))#012AttributeError:
'NoneType' object has no attribute 'strip' (txn:
tx029f271d16a84de09d4fb-005a764620) (client_ip: 192.168.47.132)
Smaller file error on storage node:
Feb 4 19:17:34 centos7-swift-node1 container-server: STDERR: (112666)
accepted ('192.168.47.132', 35696)
Feb 4 19:17:34 centos7-swift-node1 container-server: 192.168.47.132 - -
[05/Feb/2018:00:17:34 +0000] "PUT /sda/89/AUTH_admin/ara1" 202 - "PUT
http://192.168.47.132:8080/v1/AUTH_admin/ara1";
"txb8d792ec40844162978cc-005a764712" "proxy-server 44253" 0.0077 "-" 112666
0
Feb 4 19:17:34 centos7-swift-node1 container-server: STDERR:
192.168.47.132 - - [05/Feb/2018 00:17:34] "PUT /sda/89/AUTH_admin/ara1
HTTP/1.1" 202 252 0.008124 (txn: txb8d792ec40844162978cc-005a764712)
Feb 4 19:17:34 centos7-swift-node1 container-server: STDERR: (112666)
accepted ('192.168.47.132', 35702)
Feb 4 19:17:34 centos7-swift-node1 container-server: 192.168.47.132 - -
[05/Feb/2018:00:17:34 +0000] "HEAD /sda/89/AUTH_admin/ara1" 204 - "HEAD
http://192.168.47.132:8080/v1/AUTH_admin/ara1";
"tx672029e079574e78b1c1b-005a764712" "proxy-server 44254" 0.0011 "-" 112666
0
Feb 4 19:17:34 centos7-swift-node1 container-server: STDERR:
192.168.47.132 - - [05/Feb/2018 00:17:34] "HEAD /sda/89/AUTH_admin/ara1
HTTP/1.1" 204 521 0.001465 (txn: tx672029e079574e78b1c1b-005a764712)
Feb 4 19:17:34 centos7-swift-node1 container-server: STDERR: (112666)
accepted ('192.168.47.132', 35704)
Feb 4 19:17:34 centos7-swift-node1 container-server: 192.168.47.132 - -
[05/Feb/2018:00:17:34 +0000] "HEAD /sda/26/AUTH_admin/ara1/swift.conf" 404
- "HEAD http://192.168.47.132:8080/v1/AUTH_admin/ara1/swift.conf";
"tx672029e079574e78b1c1b-005a764712" "proxy-server 44254" 0.0003 "-" 112666
0
Feb 4 19:17:34 centos7-swift-node1 container-server: STDERR:
192.168.47.132 - - [05/Feb/2018 00:17:34] "HEAD
/sda/26/AUTH_admin/ara1/swift.conf HTTP/1.1" 404 351 0.000547 (txn:
tx672029e079574e78b1c1b-005a764712)
Feb 4 19:17:34 centos7-swift-node1 container-server: STDERR: (112666)
accepted ('192.168.47.132', 35706)
Feb 4 19:17:34 centos7-swift-node1 container-server: 192.168.47.132 - -
[05/Feb/2018:00:17:34 +0000] "PUT /sda/26/AUTH_admin/ara1/swift.conf" 404 -
"PUT http://192.168.47.132:8080/v1/AUTH_admin/ara1/swift.conf";
"tx0ec94217b5584b7ea4e21-005a764712" "proxy-server 44254" 0.0002 "-" 112666
0
Feb 4 19:17:34 centos7-swift-node1 container-server: STDERR:
192.168.47.132 - - [05/Feb/2018 00:17:34] "PUT
/sda/26/AUTH_admin/ara1/swift.conf HTTP/1.1" 404 212 0.000513 (txn:
tx0ec94217b5584b7ea4e21-005a764712)
Feb 4 19:17:35 centos7-swift-node1 container-server: STDERR: (112666)
accepted ('192.168.47.132', 35708)
Feb 4 19:17:35 centos7-swift-node1 container-server: 192.168.47.132 - -
[05/Feb/2018:00:17:35 +0000] "PUT /sda/26/AUTH_admin/ara1/swift.conf" 404 -
"PUT http://192.168.47.132:8080/v1/AUTH_admin/ara1/swift.conf";
"tx8e93f98453804972b39d8-005a764713" "proxy-server 44254" 0.0004 "-" 112666
0
Feb 4 19:17:35 centos7-swift-node1 container-server: STDERR:
192.168.47.132 - - [05/Feb/2018 00:17:35] "PUT
/sda/26/AUTH_admin/ara1/swift.conf HTTP/1.1" 404 212 0.000791 (txn:
tx8e93f98453804972b39d8-005a764713)
Feb 4 19:17:37 centos7-swift-node1 container-server: STDERR: (112666)
accepted ('192.168.47.132', 35710)
Feb 4 19:17:37 centos7-swift-node1 container-server: 192.168.47.132 - -
[05/Feb/2018:00:17:37 +0000] "PUT /sda/26/AUTH_admin/ara1/swift.conf" 404 -
"PUT http://192.168.47.132:8080/v1/AUTH_admin/ara1/swift.conf";
"txd44faaca5b5b408da4fea-005a764715" "proxy-server 44254" 0.0004 "-" 112666
0
Feb 4 19:17:37 centos7-swift-node1 container-server: STDERR:
192.168.47.132 - - [05/Feb/2018 00:17:37] "PUT
/sda/26/AUTH_admin/ara1/swift.conf HTTP/1.1" 404 212 0.000716 (txn:
txd44faaca5b5b408da4fea-005a764715)
Feb 4 19:17:41 centos7-swift-node1 container-server: STDERR: (112666)
accepted ('192.168.47.132', 35712)
Feb 4 19:17:41 centos7-swift-node1 container-server: 192.168.47.132 - -
[05/Feb/2018:00:17:41 +0000] "PUT /sda/26/AUTH_admin/ara1/swift.conf" 404 -
"PUT http://192.168.47.132:8080/v1/AUTH_admin/ara1/swift.conf";
"tx6e5c49ba6d54489aa1de5-005a764719" "proxy-server 44254" 0.0002 "-" 112666
0
Feb 4 19:17:41 centos7-swift-node1 container-server: STDERR:
192.168.47.132 - - [05/Feb/2018 00:17:41] "PUT
/sda/26/AUTH_admin/ara1/swift.conf HTTP/1.1" 404 212 0.000590 (txn:
tx6e5c49ba6d54489aa1de5-005a764719)
Directories created on storage node are
We are really stuck on the issue and not able to use swift in our
environment. Any help will be really appreciated.
Aravind M D
On Thu, Feb 8, 2018 at 2:43 AM, Clay Gerrard <clay.gerr...@gmail.com> wrote:
> One replica is a little strange. Do the uploads *always* fail - in the
> same way? Or is this just one example of a PUT that returned 503? Are you
> doing a lot of concurrent PUTs to the same object/name/disk?
>
> The error from the log (EPIPE) means the object-server closed the
> connection as the proxy was writing to it... which is a little strange.
> There should be a corresponding exception/error from the object-server
> service - you should make sure the object-servers are running and find
> where they are logging - then grep all the logs for the transaction-id to
> get a better picture of the whole distributed transaction.
>
> If you keep digging I know you can find the problem. Let us know what you
> find.
>
> Good luck,
>
> -Clay
>
>
> On Wed, Feb 7, 2018 at 1:58 AM, aRaviNd <ambadiarav...@gmail.com> wrote:
>
>> Hi All,
>>
>> We have created an openstack cluster with one proxy server and three
>> storage nodes. Configuration consist of two regions and three zones.
>>
>> [image: enter image description here]
>> <https://i.stack.imgur.com/w9EWG.png>
>>
>> We are able to create containers
>>
>> [image: enter image description here]
>> <https://i.stack.imgur.com/QgQQM.png>
>>
>> But while trying to upload files we are getting 503 service unavailable
>> and seeing below logs in swift.log
>>
>> [image: enter image description here]
>> <https://i.stack.imgur.com/7ROMm.png>
>> - Aravind
>>
>> _______________________________________________
>> Mailing list: http://lists.openstack.org/cgi
>> -bin/mailman/listinfo/openstack
>> Post to : openstack@lists.openstack.org
>> Unsubscribe : http://lists.openstack.org/cgi
>> -bin/mailman/listinfo/openstack
>>
>>
>
_______________________________________________
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
