[Openstack] [keystone] Keystone service creation (HTTP 500)
py", line 449, in __init__ 2016-02-24 15:18:29.100 1123 ERROR keystone.common.wsgi self.connection = self.__connect() 2016-02-24 15:18:29.100 1123 ERROR keystone.common.wsgi File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 607, in __connect 2016-02-24 15:18:29.100 1123 ERROR keystone.common.wsgi connection = self.__pool._invoke_creator(self) 2016-02-24 15:18:29.100 1123 ERROR keystone.common.wsgi File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 97, in connect 2016-02-24 15:18:29.100 1123 ERROR keystone.common.wsgi return dialect.connect(*cargs, **cparams) 2016-02-24 15:18:29.100 1123 ERROR keystone.common.wsgi File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 385, in connect 2016-02-24 15:18:29.100 1123 ERROR keystone.common.wsgi return self.dbapi.connect(*cargs, **cparams) 2016-02-24 15:18:29.100 1123 ERROR keystone.common.wsgi File "/usr/lib64/python2.7/site-packages/MySQLdb/__init__.py", line 81, in Connect 2016-02-24 15:18:29.100 1123 ERROR keystone.common.wsgi return Connection(*args, **kwargs) 2016-02-24 15:18:29.100 1123 ERROR keystone.common.wsgi File "/usr/lib64/python2.7/site-packages/MySQLdb/connections.py", line 187, in __init__ 2016-02-24 15:18:29.100 1123 ERROR keystone.common.wsgi super(Connection, self).__init__(*args, **kwargs2) 2016-02-24 15:18:29.100 1123 ERROR keystone.common.wsgi OperationalError: (_mysql_exceptions.OperationalError) (1045, "Access denied for user 'keystone'@'controller' (using password: YES)") 2016-02-24 15:18:29.100 1123 ERROR keystone.common.wsgi Please, help me to figure out what is going on. Sincerely, Alexandr Porunov ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Re: [Openstack] [keystone] Keystone service creation (HTTP 500)
Mohammed, Also I haven't mentioned that "systemctl start openstack-keystone" doesn't want to start just because httpd already running and if I understand they share the same port On Wed, Feb 24, 2016 at 5:07 PM, Mohammed Naser wrote: > Alexandr, > > You should make sure to hit “Reply all” so that it goes back to the > mailing list, but I’m not sure why it’s not connected. I’d try > connecting from the mysql CLI client and see what happens > > Mohammed > > On Wed, Feb 24, 2016 at 10:04 AM, Alexandr Porunov > wrote: > > Hello Mohammed, > > > > I don't know why my keystone is not able to authenticate with the MySQL > > server. I have created a user like in the tutorial: > > > > GRANT ALL PRIVILEGES ON keystone.* TO 'keystone'@'localhost' IDENTIFIED > BY > > 'KEYSTONE_DBPASS'; > > GRANT ALL PRIVILEGES ON keystone.* TO 'keystone'@'%' IDENTIFIED BY > > 'KEYSTONE_DBPASS' > > > > But it doesn't work > > > > On Wed, Feb 24, 2016 at 4:26 PM, Mohammed Naser > wrote: > >> > >> Hi Alexandr, > >> > >> The key error to check: > >> > >> 2016-02-24 15:18:29.100 1123 ERROR keystone.common.wsgi > OperationalError: > >> (_mysql_exceptions.OperationalError) (1045, "Access denied for user > >> 'keystone'@'controller' (using password: YES)") > >> > >> You'll need to make sure you address and resolve this, your keystone is > >> not able to authenticate with your MySQL server. > >> > >> Mohammed > >> > >> On Wednesday, 24 February 2016, Alexandr Porunov > >> wrote: > >>> > >>> Hello everyone, > >>> > >>> I am trying to pass this tutorial: > >>> > >>> > http://docs.openstack.org/liberty/install-guide-rdo/keystone-services.html > >>> but without success. > >>> > >>> It is my /etc/hosts file: > >>> 127.0.0.1 me > >>> 192.168.56.111 controller > >>> 192.168.56.131 compute1 > >>> 192.168.56.141 block1 > >>> 192.168.56.151 object1 > >>> 192.168.56.152 object2 > >>> 192.168.56.50 ns1.porunov.com ns1 > >>> > >>> I am using CentOS 7 in all nodes. > >>> > >>> When I try to create the service entity for the Identity service I > always > >>> gets an error 500: > >>> openstack service create --name keystone --description "OpenStack > >>> Identity" identity > >>> > >>> An unexpected error prevent the server from fulfilling your request. > >>> (HTTP 500) (Request-ID: req-12dc5717-71b4-4311-87e9-b0cfd4e90837) > >>> > >>> Request-ID always changes. > >>> > >>> This is my /var/log/keystone/keystone.log: > >>> 2016-02-24 15:09:45.922 4518 INFO oslo_service.service [-] Caught > >>> SIGTERM, stopping children > >>> 2016-02-24 15:09:45.923 4518 INFO oslo_service.service [-] Waiting on 4 > >>> children to exit > >>> 2016-02-24 15:09:45.926 4528 INFO eventlet.wsgi.server > >>> [req-550473ac-acff-4581-af0f-1e02a66e4f66 - - - - -] (4528) wsgi > exited, > >>> is_accepting=True > >>> 2016-02-24 15:09:45.931 4529 INFO eventlet.wsgi.server [-] (4529) wsgi > >>> exited, is_accepting=True > >>> 2016-02-24 15:09:45.933 4530 INFO eventlet.wsgi.server [-] (4530) wsgi > >>> exited, is_accepting=True > >>> 2016-02-24 15:09:45.936 4531 INFO eventlet.wsgi.server [-] (4531) wsgi > >>> exited, is_accepting=True > >>> 2016-02-24 15:09:45.942 4518 INFO oslo_service.service [-] Child 4528 > >>> exited with status 0 > >>> 2016-02-24 15:09:45.943 4518 INFO oslo_service.service [-] Child 4529 > >>> exited with status 0 > >>> 2016-02-24 15:09:45.943 4518 INFO oslo_service.service [-] Child 4530 > >>> exited with status 0 > >>> 2016-02-24 15:09:45.944 4518 INFO oslo_service.service [-] Child 4531 > >>> exited with status 0 > >>> 2016-02-24 15:09:45.947 4518 INFO oslo_service.service [-] Caught > >>> SIGTERM, stopping children > >>> 2016-02-24 15:18:29.074 1123 INFO keystone.common.wsgi > >>> [req-00fd181c-00ea-4b74-a16a-3ab8c1c38e24 - - - - -] POST > >>> http://controller:35357/v3/services > >>> 2016-02-24 15:18:29.075 1123 WARNING keystone.common.controller > >>> [req-00fd181c-00ea-
Re: [Openstack] [keystone] Keystone service creation (HTTP 500)
Also there is a strange thing. This command: su -s /bin/sh -c "keystone-manage db_sync" keystone shows: No handlers could be found for logger "oslo_config.cfg" On Wed, Feb 24, 2016 at 11:27 PM, Alexandr Porunov < alexandr.poru...@gmail.com> wrote: > Mohammed, > > Do you know what values I have to assign to these variables?: > export OS_USERNAME=??? > export OS_PASSWORD=??? > export OS_TENANT_NAME=??? > > I have assigned my username and password of controller node. This is my > full list of variables: > export OS_TOKEN=4b8548f5298c0d4c9049 > export OS_URL=http://controller:35357/v3 > export OS_IDENTITY_API_VERSION=3 > export OS_SERVICE_TOKEN=$OS_TOKEN > export OS_SERVICE_ENDPOINT=$OS_URL > export OS_USERNAME=root > export OS_PASSWORD=root > export OS_TENANT_NAME=root > export OS_AUTH_URL=$OS_URL > > But I can't run this command: > openstack service create --name keystone --description "OpenStack > Identity" identity > > It always shows: > An unexpected error prevented the server from fulfilling your request. > (HTTP 500) (Request-ID: req-ec9569be-6698-47f9-b338-309c58eb1820) > > /var/log/keystone/keystone.log as always shows: > ... > 016-02-24 23:14:29.889 1053 ERROR keystone.common.wsgi OperationalError: > (_mysql_exceptions.OperationalError) (1045, "Access denied for user > 'keystone'@'controller' (using password: YES)") > ... > > It is the full log file: > http://paste.openstack.org/show/488107/ > > Best Regards, Alexandr > > On Wed, Feb 24, 2016 at 11:25 PM, Alexandr Porunov < > alexandr.poru...@gmail.com> wrote: > >> Mohammed, >> >> Do you know what values I have to assign to these variables?: >> export OS_USERNAME=??? >> export OS_PASSWORD=??? >> export OS_TENANT_NAME=??? >> >> I have assigned my username and password of controller node. This is my >> full list of variables: >> export OS_TOKEN=4b8548f5298c0d4c9049 >> export OS_URL=http://controller:35357/v3 >> export OS_IDENTITY_API_VERSION=3 >> export OS_SERVICE_TOKEN=$OS_TOKEN >> export OS_SERVICE_ENDPOINT=$OS_URL >> export OS_USERNAME=root >> export OS_PASSWORD=root >> export OS_TENANT_NAME=root >> export OS_AUTH_URL=$OS_URL >> >> But I can't run this command: >> openstack service create --name keystone --description "OpenStack >> Identity" identity >> >> It always shows: >> An unexpected error prevented the server from fulfilling your request. >> (HTTP 500) (Request-ID: req-ec9569be-6698-47f9-b338-309c58eb1820) >> >> /var/log/keystone/keystone.log as always shows: >> ... >> 016-02-24 23:14:29.889 1053 ERROR keystone.common.wsgi OperationalError: >> (_mysql_exceptions.OperationalError) (1045, "Access denied for user >> 'keystone'@'controller' (using password: YES)") >> ... >> >> It is the full log file: >> http://paste.openstack.org/show/488107/ >> >> Best Regards, Alexandr >> >> >> >> On Wed, Feb 24, 2016 at 5:17 PM, Mohammed Naser >> wrote: >> >>> Alexandr, >>> >>> That’s fine. The suggested way of deploying Keystone is via Apache >>> now, so you should just let it live in a WSGI application as it should >>> >>> M >>> >>> On Wed, Feb 24, 2016 at 10:17 AM, Alexandr Porunov >>> wrote: >>> > Mohammed, >>> > >>> > Also I haven't mentioned that "systemctl start openstack-keystone" >>> doesn't >>> > want to start just because httpd already running and if I understand >>> they >>> > share the same port >>> > >>> > On Wed, Feb 24, 2016 at 5:07 PM, Mohammed Naser >>> wrote: >>> >> >>> >> Alexandr, >>> >> >>> >> You should make sure to hit “Reply all” so that it goes back to the >>> >> mailing list, but I’m not sure why it’s not connected. I’d try >>> >> connecting from the mysql CLI client and see what happens >>> >> >>> >> Mohammed >>> >> >>> >> On Wed, Feb 24, 2016 at 10:04 AM, Alexandr Porunov >>> >> wrote: >>> >> > Hello Mohammed, >>> >> > >>> >> > I don't know why my keystone is not able to authenticate with the >>> MySQL >>> >> > server. I have created a user like in the tutorial: >>> >> > >>> >> > GRANT ALL PRIVILEGES ON keystone.* T
Re: [Openstack] [keystone] Keystone service creation (HTTP 500)
Mohammed, Do you know what values I have to assign to these variables?: export OS_USERNAME=??? export OS_PASSWORD=??? export OS_TENANT_NAME=??? I have assigned my username and password of controller node. This is my full list of variables: export OS_TOKEN=4b8548f5298c0d4c9049 export OS_URL=http://controller:35357/v3 export OS_IDENTITY_API_VERSION=3 export OS_SERVICE_TOKEN=$OS_TOKEN export OS_SERVICE_ENDPOINT=$OS_URL export OS_USERNAME=root export OS_PASSWORD=root export OS_TENANT_NAME=root export OS_AUTH_URL=$OS_URL But I can't run this command: openstack service create --name keystone --description "OpenStack Identity" identity It always shows: An unexpected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-ec9569be-6698-47f9-b338-309c58eb1820) /var/log/keystone/keystone.log as always shows: ... 016-02-24 23:14:29.889 1053 ERROR keystone.common.wsgi OperationalError: (_mysql_exceptions.OperationalError) (1045, "Access denied for user 'keystone'@'controller' (using password: YES)") ... It is the full log file: http://paste.openstack.org/show/488107/ Best Regards, Alexandr On Wed, Feb 24, 2016 at 11:25 PM, Alexandr Porunov < alexandr.poru...@gmail.com> wrote: > Mohammed, > > Do you know what values I have to assign to these variables?: > export OS_USERNAME=??? > export OS_PASSWORD=??? > export OS_TENANT_NAME=??? > > I have assigned my username and password of controller node. This is my > full list of variables: > export OS_TOKEN=4b8548f5298c0d4c9049 > export OS_URL=http://controller:35357/v3 > export OS_IDENTITY_API_VERSION=3 > export OS_SERVICE_TOKEN=$OS_TOKEN > export OS_SERVICE_ENDPOINT=$OS_URL > export OS_USERNAME=root > export OS_PASSWORD=root > export OS_TENANT_NAME=root > export OS_AUTH_URL=$OS_URL > > But I can't run this command: > openstack service create --name keystone --description "OpenStack > Identity" identity > > It always shows: > An unexpected error prevented the server from fulfilling your request. > (HTTP 500) (Request-ID: req-ec9569be-6698-47f9-b338-309c58eb1820) > > /var/log/keystone/keystone.log as always shows: > ... > 016-02-24 23:14:29.889 1053 ERROR keystone.common.wsgi OperationalError: > (_mysql_exceptions.OperationalError) (1045, "Access denied for user > 'keystone'@'controller' (using password: YES)") > ... > > It is the full log file: > http://paste.openstack.org/show/488107/ > > Best Regards, Alexandr > > > > On Wed, Feb 24, 2016 at 5:17 PM, Mohammed Naser > wrote: > >> Alexandr, >> >> That’s fine. The suggested way of deploying Keystone is via Apache >> now, so you should just let it live in a WSGI application as it should >> >> M >> >> On Wed, Feb 24, 2016 at 10:17 AM, Alexandr Porunov >> wrote: >> > Mohammed, >> > >> > Also I haven't mentioned that "systemctl start openstack-keystone" >> doesn't >> > want to start just because httpd already running and if I understand >> they >> > share the same port >> > >> > On Wed, Feb 24, 2016 at 5:07 PM, Mohammed Naser >> wrote: >> >> >> >> Alexandr, >> >> >> >> You should make sure to hit “Reply all” so that it goes back to the >> >> mailing list, but I’m not sure why it’s not connected. I’d try >> >> connecting from the mysql CLI client and see what happens >> >> >> >> Mohammed >> >> >> >> On Wed, Feb 24, 2016 at 10:04 AM, Alexandr Porunov >> >> wrote: >> >> > Hello Mohammed, >> >> > >> >> > I don't know why my keystone is not able to authenticate with the >> MySQL >> >> > server. I have created a user like in the tutorial: >> >> > >> >> > GRANT ALL PRIVILEGES ON keystone.* TO 'keystone'@'localhost' >> IDENTIFIED >> >> > BY >> >> > 'KEYSTONE_DBPASS'; >> >> > GRANT ALL PRIVILEGES ON keystone.* TO 'keystone'@'%' IDENTIFIED BY >> >> > 'KEYSTONE_DBPASS' >> >> > >> >> > But it doesn't work >> >> > >> >> > On Wed, Feb 24, 2016 at 4:26 PM, Mohammed Naser > > >> >> > wrote: >> >> >> >> >> >> Hi Alexandr, >> >> >> >> >> >> The key error to check: >> >> >> >> >> >> 2016-02-24 15:18:29.100 1123 ERROR keystone.common.wsgi >> >> >> OperationalError: >&
Re: [Openstack] [keystone] Keystone service creation (HTTP 500)
Hello everyone, Thanks for helping. mysql -u keystone -p KEYSTONE_DBPASS //loging successfully "keystone" database is created but there no exists any tables. "su -s /bin/sh -c "keystone-manage db_sync" keystone" or "/bin/sh -c "keystone-manage db_sync" keystone" commands do nothing. I have tried reexecute this commands: GRANT ALL PRIVILEGES ON keystone.* TO 'keystone'@'localhost' \ IDENTIFIED BY 'KEYSTONE_DBPASS'; GRANT ALL PRIVILEGES ON keystone.* TO 'keystone'@'%' \ IDENTIFIED BY 'KEYSTONE_DBPASS'; My exported variables: export OS_TOKEN=4b8548f5298c0d4c9049 export OS_URL=http://controller:35357/v3 export OS_IDENTITY_API_VERSION=3 export OS_SERVICE_TOKEN=$OS_TOKEN export OS_SERVICE_ENDPOINT=$OS_URL export OS_USERNAME=keystone export OS_PASSWORD=KEYSTONE_DBPASS export OS_TENANT_NAME=keystone export OS_AUTH_URL=$OS_URL /etc/keystone/keystone.conf: [DEFAULT] admin_token = 4b8548f5298c0d4c9049 ... [database] ... connection = mysql://keystone:KEYSTONE_DBPASS@controller/keystone ... [memcache] servers = localhost:11211 ... [token] provider = uuid driver = memcache ... [revoke] driver = sql ... command still do not work.. Log file as always: http://paste.openstack.org/show/488107/ Best Regards, Alexandr On Thu, Feb 25, 2016 at 6:13 AM, Adam Young wrote: > On 02/24/2016 04:29 PM, Alexandr Porunov wrote: > > Also there is a strange thing. This command: > su -s /bin/sh -c "keystone-manage db_sync" keystone > > shows: > No handlers could be found for logger "oslo_config.cfg" > > > That is just a warning, and can be silenced by setting verboes = false in > the keystone conf file. > > > On Wed, Feb 24, 2016 at 11:27 PM, Alexandr Porunov < > alexandr.poru...@gmail.com> wrote: > >> Mohammed, >> >> Do you know what values I have to assign to these variables?: >> export OS_USERNAME=??? >> export OS_PASSWORD=??? >> export OS_TENANT_NAME=??? >> >> I have assigned my username and password of controller node. This is my >> full list of variables: >> export OS_TOKEN=4b8548f5298c0d4c9049 >> export OS_URL=http://controller:35357/v3 >> export OS_IDENTITY_API_VERSION=3 >> export OS_SERVICE_TOKEN=$OS_TOKEN >> export OS_SERVICE_ENDPOINT=$OS_URL >> export OS_USERNAME=root >> export OS_PASSWORD=root >> export OS_TENANT_NAME=root >> export OS_AUTH_URL=$OS_URL >> >> But I can't run this command: >> openstack service create --name keystone --description "OpenStack >> Identity" identity >> >> It always shows: >> An unexpected error prevented the server from fulfilling your request. >> (HTTP 500) (Request-ID: req-ec9569be-6698-47f9-b338-309c58eb1820) >> >> /var/log/keystone/keystone.log as always shows: >> ... >> 016-02-24 23:14:29.889 1053 ERROR keystone.common.wsgi OperationalError: >> (_mysql_exceptions.OperationalError) (1045, "Access denied for user >> 'keystone'@'controller' (using password: YES)") >> ... >> >> It is the full log file: >> <http://paste.openstack.org/show/488107/> >> http://paste.openstack.org/show/488107/ >> >> Best Regards, Alexandr >> >> On Wed, Feb 24, 2016 at 11:25 PM, Alexandr Porunov < >> alexandr.poru...@gmail.com> wrote: >> >>> Mohammed, >>> >>> Do you know what values I have to assign to these variables?: >>> export OS_USERNAME=??? >>> export OS_PASSWORD=??? >>> export OS_TENANT_NAME=??? >>> >>> I have assigned my username and password of controller node. This is my >>> full list of variables: >>> export OS_TOKEN=4b8548f5298c0d4c9049 >>> export OS_URL=http://controller:35357/v3 >>> export OS_IDENTITY_API_VERSION=3 >>> export OS_SERVICE_TOKEN=$OS_TOKEN >>> export OS_SERVICE_ENDPOINT=$OS_URL >>> export OS_USERNAME=root >>> export OS_PASSWORD=root >>> export OS_TENANT_NAME=root >>> export OS_AUTH_URL=$OS_URL >>> >>> But I can't run this command: >>> openstack service create --name keystone --description "OpenStack >>> Identity" identity >>> >>> It always shows: >>> An unexpected error prevented the server from fulfilling your request. >>> (HTTP 500) (Request-ID: req-ec9569be-6698-47f9-b338-309c58eb1820) >>> >>> /var/log/keystone/keystone.log as always shows: >>> ... >>> 016-02-24 23:14:29.889 1053 ERROR keystone.common.wsgi OperationalError: >>> (_mysql_exceptions.OperationalError) (1045, "Access denied for
Re: [Openstack] [keystone] Keystone service creation (HTTP 500)
Nasir, Thank you a lot! I couldn't access with this command: mysql -u keystone -h controller -p I don't know why this command hasn't given me access: GRANT ALL PRIVILEGES ON keystone.* TO 'keystone'@'%' IDENTIFIED BY 'KEYSTONE_DBPASS'; but this command has given me access: GRANT ALL PRIVILEGES ON keystone.* TO 'keystone'@'192.168.56.111' IDENTIFIED BY 'KEYSTONE_DBPASS'; Now I can create create the service entity for the Identity service. Thanks for helping. Sincerely, Alexandr On Thu, Feb 25, 2016 at 10:20 AM, Alexandr Porunov < alexandr.poru...@gmail.com> wrote: > Nasir, > > Thank you a lot! > I couldn't access with this command: > mysql -u keystone -h controller -p > > I don't know why this command hasn't given me access: > GRANT ALL PRIVILEGES ON keystone.* TO 'keystone'@'%' IDENTIFIED BY > 'KEYSTONE_DBPASS'; > > but this command has given me access: > GRANT ALL PRIVILEGES ON keystone.* TO 'keystone'@'192.168.56.111' > IDENTIFIED BY 'KEYSTONE_DBPASS'; > > Now I can create create the service entity for the Identity service. > > Thanks for helping. > > Sincerely, Alexandr > > On Thu, Feb 25, 2016 at 9:19 AM, Nasir Mahmood > wrote: > >> Last line of your logs says that keystone cant login to mysql server at >> controller node which looks to me like a node recognition issue, is >> controller node defined appropriately and resolves back to a good ip >> address? >> Try loging into mysql uusing >> >> mysql -u keystone -h controller -p >> On Feb 25, 2016 12:07 PM, "Alexandr Porunov" >> wrote: >> >>> Hello everyone, >>> Thanks for helping. >>> >>> mysql -u keystone -p KEYSTONE_DBPASS >>> >>> //loging successfully >>> >>> "keystone" database is created but there no exists any tables. >>> >>> "su -s /bin/sh -c "keystone-manage db_sync" keystone" or "/bin/sh -c >>> "keystone-manage db_sync" keystone" commands do nothing. >>> >>> I have tried reexecute this commands: >>> GRANT ALL PRIVILEGES ON keystone.* TO 'keystone'@'localhost' \ >>> IDENTIFIED BY 'KEYSTONE_DBPASS'; >>> GRANT ALL PRIVILEGES ON keystone.* TO 'keystone'@'%' \ >>> IDENTIFIED BY 'KEYSTONE_DBPASS'; >>> >>> My exported variables: >>> export OS_TOKEN=4b8548f5298c0d4c9049 >>> export OS_URL=http://controller:35357/v3 >>> export OS_IDENTITY_API_VERSION=3 >>> export OS_SERVICE_TOKEN=$OS_TOKEN >>> export OS_SERVICE_ENDPOINT=$OS_URL >>> export OS_USERNAME=keystone >>> export OS_PASSWORD=KEYSTONE_DBPASS >>> export OS_TENANT_NAME=keystone >>> export OS_AUTH_URL=$OS_URL >>> >>> /etc/keystone/keystone.conf: >>> [DEFAULT] >>> admin_token = 4b8548f5298c0d4c9049 >>> ... >>> [database] >>> ... >>> connection = mysql://keystone:KEYSTONE_DBPASS@controller/keystone >>> ... >>> [memcache] >>> servers = localhost:11211 >>> ... >>> [token] >>> provider = uuid >>> driver = memcache >>> ... >>> [revoke] >>> driver = sql >>> ... >>> >>> command still do not work.. >>> Log file as always: >>> http://paste.openstack.org/show/488107/ >>> >>> Best Regards, Alexandr >>> >>> On Thu, Feb 25, 2016 at 6:13 AM, Adam Young wrote: >>> >>>> On 02/24/2016 04:29 PM, Alexandr Porunov wrote: >>>> >>>> Also there is a strange thing. This command: >>>> su -s /bin/sh -c "keystone-manage db_sync" keystone >>>> >>>> shows: >>>> No handlers could be found for logger "oslo_config.cfg" >>>> >>>> >>>> That is just a warning, and can be silenced by setting verboes = false >>>> in the keystone conf file. >>>> >>>> >>>> On Wed, Feb 24, 2016 at 11:27 PM, Alexandr Porunov < >>>> alexandr.poru...@gmail.com> wrote: >>>> >>>>> Mohammed, >>>>> >>>>> Do you know what values I have to assign to these variables?: >>>>> export OS_USERNAME=??? >>>>> export OS_PASSWORD=??? >>>>> export OS_TENANT_NAME=??? >>>>> >>>>> I have assigned my username and password of controller nod
[Openstack] [Glance] Error 503 when creating image
Hello everyone, I am trying to upload the image to the Image service like in this tutorial: http://docs.openstack.org/liberty/install-guide-rdo/glance-verify.html I have done all steps, but this step fails: glance image-create --name "cirros" --file cirros-0.3.4-x86_64-disk.img --disk-format qcow2 --container-format bare --visibility public --progress 503 Service Unavailable: The server is currently unavailable. Please try again at a later time. (HTTP 503) It is my /var/log/glance/api.log: http://paste.openstack.org/show/488205/ It is my /var/log/glance/registry.log: http://paste.openstack.org/show/488207/ It is my /var/log/keystone/keystone.log: http://paste.openstack.org/show/488208/ It is my variables: export OS_PROJECT_DOMAIN_ID=default export OS_USER_DOMAIN_ID=default export OS_PROJECT_NAME=admin export OS_TENANT_NAME=admin export OS_USERNAME=admin export OS_PASSWORD=admin export OS_AUTH_URL=http://controller:35357/v3 export OS_IDENTITY_API_VERSION=3 export OS_IMAGE_API_VERSION=2 May somebody help me, please? Best Regards, Alexandr ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Re: [Openstack] [Glance] Error 503 when creating image
But I have put admin's variables. Haven't I? On Thu, Feb 25, 2016 at 6:15 PM, Remo Mattei wrote: > Only admin can use public > > It will not work if that is a normal tenant > > Inviato da iPhone > > Il giorno 25 feb 2016, alle ore 07:56, Alexandr Porunov < > alexandr.poru...@gmail.com> ha scritto: > > Hello everyone, > > I am trying to upload the image to the Image service like in this tutorial: > http://docs.openstack.org/liberty/install-guide-rdo/glance-verify.html > > I have done all steps, but this step fails: > glance image-create --name "cirros" --file cirros-0.3.4-x86_64-disk.img > --disk-format qcow2 --container-format bare --visibility public --progress > > 503 Service Unavailable: The server is currently unavailable. Please try > again at a later time. (HTTP 503) > > It is my /var/log/glance/api.log: > http://paste.openstack.org/show/488205/ > > It is my /var/log/glance/registry.log: > http://paste.openstack.org/show/488207/ > > It is my /var/log/keystone/keystone.log: > http://paste.openstack.org/show/488208/ > > It is my variables: > export OS_PROJECT_DOMAIN_ID=default > export OS_USER_DOMAIN_ID=default > export OS_PROJECT_NAME=admin > export OS_TENANT_NAME=admin > export OS_USERNAME=admin > export OS_PASSWORD=admin > export OS_AUTH_URL=http://controller:35357/v3 > export OS_IDENTITY_API_VERSION=3 > export OS_IMAGE_API_VERSION=2 > > May somebody help me, please? > > Best Regards, Alexandr > !DSPAM:1,56cf283723854106764879! > > ___ > Mailing list: > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack > Post to : openstack@lists.openstack.org > Unsubscribe : > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack > > > !DSPAM:1,56cf283723854106764879! > > ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Re: [Openstack] [Glance] Error 503 when creating image
On Thu, Feb 25, 2016 at 11:21 PM, Alexandr Porunov < alexandr.poru...@gmail.com> wrote: > The same error. > > glance image-create --progress --file ./cirros-0.3.4-x86_64-disk.img > --name cirros --disk-format qcow2 --container-format bare > 503 Service Unavailable: The server is currently unavailable. Please try > again at a later time. (HTTP 503) > > =( > > > On Thu, Feb 25, 2016 at 11:20 PM, Remo Mattei wrote: > >> try this >> >> glance image-create --progress --file ./cirros-0.3.4-x86_64-disk.img >> --name cirros --disk-format qcow2 --container-format bare >> >> >> On Feb 25, 2016, at 13:19, Remo Mattei wrote: >> >> did you try without public ? >> >> On Feb 25, 2016, at 13:11, Alexandr Porunov >> wrote: >> >> But I have put admin's variables. Haven't I? >> >> On Thu, Feb 25, 2016 at 6:15 PM, Remo Mattei wrote: >> >>> Only admin can use public >>> >>> It will not work if that is a normal tenant >>> >>> Inviato da iPhone >>> >>> Il giorno 25 feb 2016, alle ore 07:56, Alexandr Porunov < >>> alexandr.poru...@gmail.com> ha scritto: >>> >>> Hello everyone, >>> >>> I am trying to upload the image to the Image service like in this >>> tutorial: >>> http://docs.openstack.org/liberty/install-guide-rdo/glance-verify.html >>> >>> I have done all steps, but this step fails: >>> glance image-create --name "cirros" --file cirros-0.3.4-x86_64-disk.img >>> --disk-format qcow2 --container-format bare --visibility public --progress >>> >>> 503 Service Unavailable: The server is currently unavailable. Please try >>> again at a later time. (HTTP 503) >>> >>> It is my /var/log/glance/api.log: >>> http://paste.openstack.org/show/488205/ >>> >>> It is my /var/log/glance/registry.log: >>> http://paste.openstack.org/show/488207/ >>> >>> It is my /var/log/keystone/keystone.log: >>> http://paste.openstack.org/show/488208/ >>> >>> It is my variables: >>> export OS_PROJECT_DOMAIN_ID=default >>> export OS_USER_DOMAIN_ID=default >>> export OS_PROJECT_NAME=admin >>> export OS_TENANT_NAME=admin >>> export OS_USERNAME=admin >>> export OS_PASSWORD=admin >>> export OS_AUTH_URL=http://controller:35357/v3 >>> export OS_IDENTITY_API_VERSION=3 >>> export OS_IMAGE_API_VERSION=2 >>> >>> May somebody help me, please? >>> >>> Best Regards, Alexandr >>> >>> ___ >>> Mailing list: >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack >>> Post to : openstack@lists.openstack.org >>> Unsubscribe : >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack >>> >>> >>> !DSPAM:1,56cf283723854106764879! >>> >>> >> !DSPAM:1,56cf6f7e268051706116401! >> >> >> >> > ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Re: [Openstack] [Glance] Error 503 when creating image
There are my logs:
It is my /var/log/glance/api.log:
http://paste.openstack.org/show/488205/
It is my /var/log/glance/registry.log:
http://paste.openstack.org/show/488207/
It is my /var/log/keystone/keystone.log:
http://paste.openstack.org/show/488208/
Nothing have changed. There only this lines interesting in api.log:
2016-02-25 17:44:11.524 2462 WARNING keystonemiddleware.auth_token [-]
Identity response: {"error": {"message": "The request you have made
requires authentication.", "code": 401, "title": "Unauthorized"}}
2016-02-25 17:44:12.335 2462 WARNING keystonemiddleware.auth_token [-]
Identity response: {"error": {"message": "The request you have made
requires authentication.", "code": 401, "title": "Unauthorized"}}
2016-02-25 17:44:12.336 2462 CRITICAL keystonemiddleware.auth_token
[-] Unable to validate token: Identity server rejected authorization
necessary to fetch token data
But I don't know how to repair it
On Thu, Feb 25, 2016 at 11:22 PM, Remo Mattei wrote:
> Check your logs and see what it says you may want to add —debug to that
> and see where it fails
>
>
> On Feb 25, 2016, at 13:21, Alexandr Porunov
> wrote:
>
> The same error.
>
> glance image-create --progress --file ./cirros-0.3.4-x86_64-disk.img
> --name cirros --disk-format qcow2 --container-format bare
> 503 Service Unavailable: The server is currently unavailable. Please try
> again at a later time. (HTTP 503)
>
> =(
>
>
> On Thu, Feb 25, 2016 at 11:20 PM, Remo Mattei wrote:
>
>> try this
>>
>> glance image-create --progress --file ./cirros-0.3.4-x86_64-disk.img
>> --name cirros --disk-format qcow2 --container-format bare
>>
>>
>> On Feb 25, 2016, at 13:19, Remo Mattei wrote:
>>
>> did you try without public ?
>>
>> On Feb 25, 2016, at 13:11, Alexandr Porunov
>> wrote:
>>
>> But I have put admin's variables. Haven't I?
>>
>> On Thu, Feb 25, 2016 at 6:15 PM, Remo Mattei wrote:
>>
>>> Only admin can use public
>>>
>>> It will not work if that is a normal tenant
>>>
>>> Inviato da iPhone
>>>
>>> Il giorno 25 feb 2016, alle ore 07:56, Alexandr Porunov <
>>> alexandr.poru...@gmail.com> ha scritto:
>>>
>>> Hello everyone,
>>>
>>> I am trying to upload the image to the Image service like in this
>>> tutorial:
>>> http://docs.openstack.org/liberty/install-guide-rdo/glance-verify.html
>>>
>>> I have done all steps, but this step fails:
>>> glance image-create --name "cirros" --file cirros-0.3.4-x86_64-disk.img
>>> --disk-format qcow2 --container-format bare --visibility public --progress
>>>
>>> 503 Service Unavailable: The server is currently unavailable. Please try
>>> again at a later time. (HTTP 503)
>>>
>>> It is my /var/log/glance/api.log:
>>> http://paste.openstack.org/show/488205/
>>>
>>> It is my /var/log/glance/registry.log:
>>> http://paste.openstack.org/show/488207/
>>>
>>> It is my /var/log/keystone/keystone.log:
>>> http://paste.openstack.org/show/488208/
>>>
>>> It is my variables:
>>> export OS_PROJECT_DOMAIN_ID=default
>>> export OS_USER_DOMAIN_ID=default
>>> export OS_PROJECT_NAME=admin
>>> export OS_TENANT_NAME=admin
>>> export OS_USERNAME=admin
>>> export OS_PASSWORD=admin
>>> export OS_AUTH_URL=http://controller:35357/v3
>>> export OS_IDENTITY_API_VERSION=3
>>> export OS_IMAGE_API_VERSION=2
>>>
>>> May somebody help me, please?
>>>
>>> Best Regards, Alexandr
>>>
>>> ___
>>> Mailing list:
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>> Post to : openstack@lists.openstack.org
>>> Unsubscribe :
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>>
>>>
>>>
>>>
>> !DSPAM:1,56cf6f7e268051706116401!
>>
>>
>>
>>
> !DSPAM:1,56cf7078270641806488744!
>
>
>
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
[Openstack] [neutron] 503 Service Unavailable
Hello everyone, I am trying to install neutron. I have passed all steps except the verify step: http://docs.openstack.org/liberty/install-guide-rdo/neutron-verify.html $ neutron ext-list 503 Service Unavailable The server is currently unavailable. Please try again at a later time. There is logs from the controller node: /var/log/neutron/server.log: http://paste.openstack.org/show/488423/ There is logs from the compute node: /var/log/neutron/linuxbridge-agent.log: http://paste.openstack.org/show/488424/ And there is additional logs from the controller node: /var/log/neutron/metadata-agent.log: http://paste.openstack.org/show/488422/ /var/log/neutron/linuxbridge-agent.log: http://paste.openstack.org/show/488421/ /var/log/neutron/l3-agent.log: http://paste.openstack.org/show/488419/ /var/log/neutron/dhcp-agent.log: http://paste.openstack.org/show/488417/ Please, help me figure out why neutron doesn't work. Best Regards, Alexandr ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Re: [Openstack] [neutron] 503 Service Unavailable
Thank you, Neil Yes, it was just because I had incorrect password of neutron user Best Regards, Alexandr On Fri, Feb 26, 2016 at 11:31 PM, Neil Jerram wrote: > I think the Neutron server log is indicating invalid credentials. Suggest > checking the auth settings in your neutron.conf. > > Neil > > > *From: *Alexandr Porunov > *Sent: *Friday, 26 February 2016 20:39 > *To: *Openstack > *Subject: *[Openstack] [neutron] 503 Service Unavailable > > Hello everyone, > > I am trying to install neutron. I have passed all steps except the verify > step: > http://docs.openstack.org/liberty/install-guide-rdo/neutron-verify.html > > $ neutron ext-list > > 503 Service Unavailable > > The server is currently unavailable. Please try again at a later time. > > There is logs from the controller node: > /var/log/neutron/server.log: http://paste.openstack.org/show/488423/ > > There is logs from the compute node: > /var/log/neutron/linuxbridge-agent.log: > http://paste.openstack.org/show/488424/ > > And there is additional logs from the controller node: > /var/log/neutron/metadata-agent.log: > http://paste.openstack.org/show/488422/ > /var/log/neutron/linuxbridge-agent.log: > http://paste.openstack.org/show/488421/ > /var/log/neutron/l3-agent.log: http://paste.openstack.org/show/488419/ > /var/log/neutron/dhcp-agent.log: http://paste.openstack.org/show/488417/ > > Please, help me figure out why neutron doesn't work. > > Best Regards, Alexandr > ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
[Openstack] [neutron] all agents are dead
Hello everyone, I am trying to install neutron. I have passed all steps except the verify step: http://docs.openstack.org/liberty/install-guide-rdo/neutron-verify-option2.html "neutron agent-list" shows agents but all agents aren't alive. All agents have parameter alive: xxx. But in the verify step we must have parameter alive: :-) I have looked all logs and found nothing special. Some errors like: 2016-02-27 13:36:17.250 846 ERROR oslo.messaging._drivers.impl_rabbit [req-62205e8a-47e6-49b5-accc-f774070a9b9b - - - - -] AMQP server on controller:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. 2016-02-27 13:36:18.269 846 ERROR oslo.messaging._drivers.impl_rabbit [req-62205e8a-47e6-49b5-accc-f774070a9b9b - - - - -] AMQP server on controller:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. 2016-02-27 13:36:20.309 846 ERROR oslo.messaging._drivers.impl_rabbit [req-62205e8a-47e6-49b5-accc-f774070a9b9b - - - - -] AMQP server on controller:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. But after errors all agents are connected: 2016-02-27 13:36:22.358 846 INFO oslo.messaging._drivers.impl_rabbit [req-62205e8a-47e6-49b5-accc-f774070a9b9b - - - - -] Reconnected to AMQP server on controller:5672 2016-02-27 13:36:22.358 846 INFO oslo.messaging._drivers.impl_rabbit [req-62205e8a-47e6-49b5-accc-f774070a9b9b - - - - -] Connected to AMQP server on controller:5672 2016-02-27 13:36:22.376 846 INFO oslo.messaging._drivers.impl_rabbit [req-62205e8a-47e6-49b5-accc-f774070a9b9b - - - - -] Connecting to AMQP server on controller:5672 2016-02-27 13:36:22.383 846 INFO oslo.messaging._drivers.impl_rabbit [-] Connecting to AMQP server on controller:5672 2016-02-27 13:36:22.526 846 INFO oslo.messaging._drivers.impl_rabbit [-] Connected to AMQP server on controller:5672 2016-02-27 13:36:22.526 846 INFO oslo.messaging._drivers.impl_rabbit [req-62205e8a-47e6-49b5-accc-f774070a9b9b - - - - -] Connected to AMQP server on controller:5672 How to bring agents to life? Please, help me figure out what is going on Sincerely, Alexandr ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Re: [Openstack] [neutron] all agents are dead
Hello, Nasir If I restart all agents then they become working. E.g. : Controller node: systemctl restart neutron-linuxbridge-agent.service neutron-dhcp-agent.service neutron-metadata-agent.service neutron-l3-agent.service Compute node. systemctl restart neutron-linuxbridge-agent.service But it is strange because they all are enabled and are started on boot. But after boot only "L3 agent" works. Also On compute node "Linux bridge agent" works several seconds and then dies. Also problem that even if I restart all agents I can't watch networks through the "horizon dashboard". Horizon works nice but when I click on "Networks" it shows: "Something went wrong! An unexpected error has occurred. Try refreshing the page. If that doesn't help, contact your local administrator." Console shows: "Failed to load resource: the server responded with a status of 500 (INTERNAL SERVER ERROR)" Dashboard logs on the controller node are empty. Do you know why it can happens? Best Regards, Alexandr On Sat, Feb 27, 2016 at 1:13 PM, Nasir Mahmood wrote: > Looks looks your message queue server is nit responding , try restating > the services on controller node > On Feb 27, 2016 3:24 PM, "Alexandr Porunov" > wrote: > >> Hello everyone, >> >> I am trying to install neutron. I have passed all steps except the verify >> step: >> >> http://docs.openstack.org/liberty/install-guide-rdo/neutron-verify-option2.html >> >> "neutron agent-list" shows agents but all agents aren't alive. All agents >> have parameter alive: xxx. >> >> But in the verify step we must have parameter alive: :-) >> >> I have looked all logs and found nothing special. Some errors like: >> 2016-02-27 13:36:17.250 846 ERROR oslo.messaging._drivers.impl_rabbit >> [req-62205e8a-47e6-49b5-accc-f774070a9b9b - - - - -] AMQP server on >> controller:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 >> seconds. >> 2016-02-27 13:36:18.269 846 ERROR oslo.messaging._drivers.impl_rabbit >> [req-62205e8a-47e6-49b5-accc-f774070a9b9b - - - - -] AMQP server on >> controller:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 >> seconds. >> 2016-02-27 13:36:20.309 846 ERROR oslo.messaging._drivers.impl_rabbit >> [req-62205e8a-47e6-49b5-accc-f774070a9b9b - - - - -] AMQP server on >> controller:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 >> seconds. >> >> But after errors all agents are connected: >> 2016-02-27 13:36:22.358 846 INFO oslo.messaging._drivers.impl_rabbit >> [req-62205e8a-47e6-49b5-accc-f774070a9b9b - - - - -] Reconnected to AMQP >> server on controller:5672 >> 2016-02-27 13:36:22.358 846 INFO oslo.messaging._drivers.impl_rabbit >> [req-62205e8a-47e6-49b5-accc-f774070a9b9b - - - - -] Connected to AMQP >> server on controller:5672 >> 2016-02-27 13:36:22.376 846 INFO oslo.messaging._drivers.impl_rabbit >> [req-62205e8a-47e6-49b5-accc-f774070a9b9b - - - - -] Connecting to AMQP >> server on controller:5672 >> 2016-02-27 13:36:22.383 846 INFO oslo.messaging._drivers.impl_rabbit [-] >> Connecting to AMQP server on controller:5672 >> 2016-02-27 13:36:22.526 846 INFO oslo.messaging._drivers.impl_rabbit [-] >> Connected to AMQP server on controller:5672 >> 2016-02-27 13:36:22.526 846 INFO oslo.messaging._drivers.impl_rabbit >> [req-62205e8a-47e6-49b5-accc-f774070a9b9b - - - - -] Connected to AMQP >> server on controller:5672 >> >> How to bring agents to life? >> >> Please, help me figure out what is going on >> >> Sincerely, Alexandr >> >> ___ >> Mailing list: >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack >> Post to : openstack@lists.openstack.org >> Unsubscribe : >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack >> >> ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
[Openstack] Which storage is the best for temporary store?
Hello, I need a temporary storage for users videos. This is my scenario: 1. User uploads a video file chunk by chunk. 2. My application stores each chunk in the temporary storage. 3. When video loading is finished then my transformation server transforms this video and stores into permanent storage. 4. After video was saved in permanent storage it is deleted from temporary storage. Which storage is the best for read/write/delete only once and has the ability to save files chunk by chunk? Swift? Cinder? or something else? Sincerely, Alexandr ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
[Openstack] Fwd: Which storage is the best for temporary store?
Hello Hugo, Thank you for your response. It is example of my architecture of video upload: [image: Inline image 1] 1. User uploads video chunks to an upload server. 2. The upload server uploads each chunk to the temporary storage. (If connection breaks then the user will be able to recover it and continue download his video because his chunks already saved in the temporary storage) 3. After the video is fully saved, a upload server will send a task to a queue to transform the video. 4. A worker takes the task and transforms the video. 5. The worker uploads the video to the permanent storage. 6. The worker removes the video from temporary storage. Each chunk of the video is about 1 mb. Maximum size of the video is 8 GB. The upload server and the worker are written on Java. Permanent storage - OpenStack Swift. Transformation will starts only when full video will be uploaded. It will be great to use Swift as a temporary storage but I am not sure that I can manage chunks in Swift. If it is possible, please, give me any link to read about it. Best Regards, Alexandr On Fri, Mar 11, 2016 at 6:15 AM, Kuo Hugo wrote: > Hi Alexandr, > > The OpenStack Swift is a good choice as permanent storage pool for your > video case. > As for temporary storage media in your case, Swift could be an option. But > you need to consider several factors. > >- How will user upload video to your site ? >- How’s the speed do you need to store video chunks into temporary >tier ? >- What protocol is supported to store video by your application ? >- Does your transformation server pull the data to it’s memory/local >media to process ? >- How many data will you have in your estimation ? >- How’s the size of each video ingeneral ? > > Here’s a list of benefit by using Swift as both temporary & permanent > storage pool : > >- User can upload video to Swift directly via HTTP. No need an >additional server between your Swift pool and client. >- Swift has built-in function for authentication/authorization. >- If you’d like to move video cross temporary and permanent locations, >it could be done in swift’s internal network. >- The metadata can associate with object itself. >- HTTP protocol gives high flexibility for various ways. >- No video size limitation. >- You can have 2 copies policy for temporary tier and 3 copies for >permanent tier to get better data availability and durability. > > Here’s a list of potential weaknesses for temporary tier : > >- If your transformation server needs to edit data online and change >bits on each chunk, it’s not good for Swift. > > If the temporary tier doesn’t need too much space (lower than 3TB), you > can just use several high speed disks as a raid0 pool to have nice IOPs. > > Regard // Hugo > > > 2016-03-10 18:54 GMT+08:00 Alexandr Porunov : > >> Hello, >> >> I need a temporary storage for users videos. >> This is my scenario: >> 1. User uploads a video file chunk by chunk. >> 2. My application stores each chunk in the temporary storage. >> 3. When video loading is finished then my transformation server >> transforms this video and stores into permanent storage. >> 4. After video was saved in permanent storage it is deleted from >> temporary storage. >> >> Which storage is the best for read/write/delete only once and has the >> ability to save files chunk by chunk? Swift? Cinder? or something else? >> >> Sincerely, Alexandr >> >> ___ >> Mailing list: >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack >> Post to : openstack@lists.openstack.org >> Unsubscribe : >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack >> >> > ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
[Openstack] [Controller] Questions about horizontal scalability of controller
Hello, I have finished the installation guide: http://docs.openstack.org/liberty/install-guide-rdo/ But now I have a lot of questions about controllers scalability. Please, answer if you can. If I understood correctly, all requests to any node (compute or storage) will pass through controller nodes and this requests can write some data into mysql or rabbitmq. Isn't it a bottleneck? We use Mysql Galera Cluster on the controller nodes, so each node will keep all data from all nodes. That is why we must scale each node vertically. I am not sure but we can stretch against into disk space on controller nodes. The same story about RabbitMQ. Have I correctly understood it? If no (I hope), can you point me on the documentation/article where I can read about it? If yes, can you suggest me how to build a good cluster? The second question is about compute nodes requests. How will go one request from the first compute node to the second compute node? Does it go straight from the first compute node to the second or it go from the first compute node to the controller node (which?) and then from the controller node to the second compute node? Please, share any information which you know about it or point me to right articles where I can read about it Best Regards, Alexandr ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
[Openstack] [Fuel] Can we use different compute nodes with different capacity?
Hello, I want to use totally different compute nodes for different purposes. Is it normal practice? Also, if I have a server with a lot of RAM and a lot of disk space is it normal practice to install swift and nova on this server to use maximum of the server? Application will use only RAM and storage will use only disks. Sincerely, Alexandr ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
[Openstack] [OpenStack] [Fuel] [Networks] Verification failed
Hello, I have created two networks in VirtualBox: vboxnet0 : 192.168.56.0 /24 vboxnet1 : 192.168.57.0 /24 vboxnet0 is a public network (gateway on 192.168.56.10) vboxnet1 is a private network Each virtual machine is connected to the private and public networks (Fuel Master and two Fuel Slaves (Controller, Compute) ). I have installed Fuel Master and run it. Then I have logged into UI. Then I have created a new Environment and added two nodes (Controller, Compute). I have not changed networks. So, I tried to Verify Networks but without success. It says: Verification failed. Repo availability verification using public network failed on following nodes Controller1. Following repos are not available - http://archive.ubuntu.com/ubuntu/, http://mirror.fuel-infra.org/mos-repos/ubuntu/8.0/ . Check your public network settings and availability of the repositories from public network. Please examine nailgun and astute logs for additional details. I have tested Internet connectivity on the Fuel Master. It works. I understand that I have to change networks but I don't know how to do it. The problem is I have to specify public, storage, management, baremetal and internal networks. But phisically I have only private and public networks. If I try to use 192.168.57.0 /24 in the private network then I can't use this network as a storage or management and vice versa. I need to use a network 192.168.56.0 /24 only for public usage (like endpoint for controllers) and a network 192.168.56.0 /24 for other usage (like private, storage, management, baremetal). Can you help me to figure out how to describe these networks? Sincerely, Alexandr ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
[Openstack] Can we send requests directly on a bare metal machine?
Hello, I want to use only these projects: - Object Storage (Swift) - Bare Metal (Ironic) - Telemetry (Ceilometer) - Image Service (Glance) Do I need Controllers and Neutron nodes for using them? Can we send requests directly on a bare metal machine without passing them through controller/neutron nodes? Sincerely, Alexandr ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
[Openstack] [Ironic] Can we send requests directly on a bare metal machine?
On Fri, Apr 8, 2016 at 8:56 PM, Alexandr Porunov wrote: > Hello, > > I want to use only these projects: > - Object Storage (Swift) > - Bare Metal (Ironic) > - Telemetry (Ceilometer) > - Image Service (Glance) > > Do I need Controllers and Neutron nodes for using them? > Can we send requests directly on a bare metal machine without passing them > through controller/neutron nodes? > > Sincerely, > Alexandr > ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
[Openstack] [Swift] How to add new regions/zones into a swift cluster?
Hello everyone! I am trying to install swift without others products but I have problem with adding drives to the swift ring. I use this command: swift-ring-builder account.builder add --region USA --zone New_York --ip 192.168.57.51 --port 6002 --device d1 --weight 100 It shows me: "Usage: swift-ring-builder [options]. swift-ring-builder: error: option --region: invalid integer value: 'USA'". What's wrong with a name of the region? How can I add new regions/zones into swift? Please help me Best regards, Alexandr ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Re: [Openstack] [Swift] How to add new regions/zones into a swift cluster?
Thank you for help! It's work! Sincerely, Alexandr On Thu, May 19, 2016 at 1:47 PM, Kota TSUYUZAKI < tsuyuzaki.k...@lab.ntt.co.jp> wrote: > Hello Alexandr, > > As the error message described, region and zone take an integer value[1] > so probably you should set the value like as: > > swift-ring-builder account.builder add --region 1 --zone 1 --ip > 192.168.57.51 --port 6002 --device d1 --weight 100 > > Although you need to keep your own map like "region 1 means USA and zone 1 > means NewYork" by yourself. > > Best, > Kota > > 1: > https://github.com/openstack/swift/blob/master/swift/common/ring/utils.py#L559-L562 > (2016/05/19 19:13), Alexandr Porunov wrote: > > Hello everyone! > > > > I am trying to install swift without others products but I have problem > > with adding drives to the swift ring. > > I use this command: > > swift-ring-builder account.builder add --region USA --zone New_York --ip > > 192.168.57.51 --port 6002 --device d1 --weight 100 > > > > It shows me: > > "Usage: swift-ring-builder [options]. swift-ring-builder: error: option > > --region: invalid integer value: 'USA'". > > > > What's wrong with a name of the region? How can I add new regions/zones > > into swift? Please help me > > > > Best regards, > > Alexandr > > > > > > > > ___ > > Mailing list: > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack > > Post to : openstack@lists.openstack.org > > Unsubscribe : > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack > > > > > > > > ___ > Mailing list: > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack > Post to : openstack@lists.openstack.org > Unsubscribe : > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack > ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
[Openstack] [Swift] LookupError: Entry point 'copy' not found
I try to install swift but without success. When I run this command: swift-init proxy start It show me an error: Traceback (most recent call last): File "/usr/bin/swift-proxy-server", line 23, in sys.exit(run_wsgi(conf_file, 'proxy-server', **options)) File "/usr/lib/python2.7/site-packages/swift/common/wsgi.py", line 896, in run_wsgi loadapp(conf_path, global_conf=global_conf) File "/usr/lib/python2.7/site-packages/swift/common/wsgi.py", line 383, in loadapp ctx = loadcontext(loadwsgi.APP, conf_file, global_conf=global_conf) File "/usr/lib/python2.7/site-packages/swift/common/wsgi.py", line 367, in loadcontext global_conf=global_conf) File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 296, in loadcontext global_conf=global_conf) File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 320, in _loadconfig return loader.get_context(object_type, name, global_conf) File "/usr/lib/python2.7/site-packages/swift/common/wsgi.py", line 64, in get_context object_type, name=name, global_conf=global_conf) File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 450, in get_context global_additions=global_additions) File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 562, in _pipeline_app_context for name in pipeline[:-1]] File "/usr/lib/python2.7/site-packages/swift/common/wsgi.py", line 64, in get_context object_type, name=name, global_conf=global_conf) File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 454, in get_context section) File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 476, in _context_from_use object_type, name=use, global_conf=global_conf) File "/usr/lib/python2.7/site-packages/swift/common/wsgi.py", line 64, in get_context object_type, name=name, global_conf=global_conf) File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 406, in get_context global_conf=global_conf) File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 296, in loadcontext global_conf=global_conf) File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 328, in _loadegg return loader.get_context(object_type, name, global_conf) File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 620, in get_context object_type, name=name) File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 659, in find_egg_entry_point for prot in protocol_options] or '(no entry points)' LookupError: Entry point 'copy' not found in egg 'swift' (dir: /usr/lib/python2.7/site-packages; protocols: paste.filter_factory, paste.filter_app_factory; entry_points: ) My OS is : CentOS 7 I install swift without other openstack products. I install it by using the book "OpenStack Swift O'REILLY" author Joe Arnold & members of the SwiftStack team. I have reach a page 139 "Starting the Proxy Server". But proxy server doesn't want to start. Here my configuration files: /etc/swift/swift.conf http://paste.openstack.org/show/497857/ /etc/swift/proxy-server.conf http://paste.openstack.org/show/497858/ /etc/swift/account-server.conf http://paste.openstack.org/show/497859/ /etc/swift/container-server.conf http://paste.openstack.org/show/497860/ /etc/swift/object-server.conf http://paste.openstack.org/show/497861/ There is no any log file in /var/log/swift/ Please help me to figure out what I am doing wrong Best regards, Alexandr ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Re: [Openstack] [Swift] LookupError: Entry point 'copy' not found
Hello Mark, I have tried to remove 'copy' from the pipeline but it don't work. I have tried to install swift like this: python setup.py develop pip install -r test-requirements.txt and it works! I really don't know why it works. But I started my proxy servers. Sincerely, Alexandr On Sat, May 21, 2016 at 3:19 AM, Mark Kirkwood < mark.kirkw...@catalyst.net.nz> wrote: > On 20/05/16 20:37, Alexandr Porunov wrote: > >> I try to install swift but without success. >> When I run this command: >> swift-init proxy start >> >> It show me an error: >> Traceback (most recent call last): (snippage) >> >> File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line >> 659, in find_egg_entry_point >> for prot in protocol_options] or '(no entry points)' >> LookupError: Entry point 'copy' not found in egg 'swift' (dir: >> /usr/lib/python2.7/site-packages; protocols: paste.filter_factory, >> paste.filter_app_factory; entry_points: ) >> >> >> My OS is : CentOS 7 >> > > I think Centos7 will install Swift 2.5 (i,e Liberty). Looking at a Swift > 2.5 setup I have here: > > $ cd /usr/lib/python2.7/dist-packages/swift-2.5.0.egg-info > $ grep copy entry_points.txt > (nothing) > > So there is no explicit copy in there. Remove copy from the pipeline and > filter specification and it should start (I'm guessing that copy is enabled > by default in these later swift versions). > > I install swift without other openstack products. I install it by using >> the book "OpenStack Swift O'REILLY" author Joe Arnold & members of the >> SwiftStack team. >> >> I have reach a page 139 "Starting the Proxy Server". But proxy server >> doesn't want to start. >> >> >> > Probably better to use the install guide for your specific version: > > http://docs.openstack.org/liberty/install-guide-obs/swift.html > > regards > > Mark > > ___ > Mailing list: > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack > Post to : openstack@lists.openstack.org > Unsubscribe : > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack > ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
[Openstack] [Swift] How to configure multi-region replication?
I have to configure special multi-region replication. If I put an object into region A or B I need to create 3 local replicas and 1 replica in other regions. If I put an object into region C I need to create 2 local replicas and 1 replica in other regions. Does it possible to configure replication like this? If yes then how? Best regards, Alexandr ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Re: [Openstack] [Swift] How to configure multi-region replication?
Thank you for the response. I haven't any cluster yet. I just learn a documentation about swift. As a user I want to be able to start with a small cluster in a single region and then increase a size of the cluster and a count of regions. I try to understand if it is possible with swift. I know that we can easily increase the size of the cluster but I am not sure about increasing the count of regions because of replication problems. It would be really great if we were able to tell our cluster how many replicas we want to store in different regions. Sincerely, Alexandr On Wed, Jun 29, 2016 at 10:04 PM, Clay Gerrard wrote: > No, that's not currently supported, but it's a reasonable use-case to keep > in mind for ongoing development of composite rings. Thanks for asking > about it. > > For now the closest approximation is to create a 3 region ring with ~5 > replicas - placement is based on capacity - regions with more capacity will > have two replicas of *most* partitions, regions with less capacity will > have only one. But each individual partitions replicas will have different > placement. The non-determinism is probably a poor match for you very > specific goals. > > The 3 region use-case is currently idealized with 6 replicas and a high > overload to force 2 replicas per region then tune/monitor capacity in each > region to keep them roughly even. > > It might be worth noting that you can create multiple storage policies > with different rings, in region A & B the default storage policy could have > four replicas, but even with careful capacity management placement would > flop between 2/2 and 3/1 for any given part. In region C you could use 3 > and get 2/1 and 1/2 depending on how closely you can manage capacity... > > None of these options consider "the point of entry" as a persistent > artifact that classifies the storage class of the object tho... generally > objects in swift are grouped by container - so regardless of the region of > the proxy that originally wrote the object, once it's commited to the > storage layer the part placement in the ring for the container holding the > object will determine where the object should live... you may have to > elaborate on the use-case there - I'm not sure anyone working on Swift is > currently thinking about that part specifically? > > -Clay > > On Wed, Jun 29, 2016 at 11:33 AM, Alexandr Porunov < > alexandr.poru...@gmail.com> wrote: > >> I have to configure special multi-region replication. >> If I put an object into region A or B I need to create 3 local replicas >> and 1 replica in other regions. >> If I put an object into region C I need to create 2 local replicas and 1 >> replica in other regions. >> >> Does it possible to configure replication like this? If yes then how? >> >> Best regards, >> Alexandr >> >> ___ >> Mailing list: >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack >> Post to : openstack@lists.openstack.org >> Unsubscribe : >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack >> >> > ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
[Openstack] [Swift] Is it possible to change a replication factor?
Is it possible to change a replication factor from 2 to 3 or vice versa from 3 to 2 in a running cluster? Best regards, Alexandr ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
[Openstack] [Swift] Is it possible to collect metrics from Swift without Statsd?
Hello, I need to collect different metrics from swift servers and store those metrics into OpenTSDB. I found that there is way to collect metrics with statsd and graphite but it isn't suitable for me because I have to store metrics into OpenTSDB. Is there any way to store metrics from openstack swift into OpenTSDB? I haven't found cli commands to get metrics out from Openstack swift. Sincerely, Alexandr ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Re: [Openstack] [Swift] Is it possible to collect metrics from Swift without Statsd?
But If I use statsd logs then I won't have an opportunity to choose which stats I want to send. Also I will not have an ability to change the name of stats. I want to have some ability to use the simple script on each swift node which will get stats throug the cli and send it to OpenTSDB. On Tue, Aug 9, 2016 at 7:34 PM, John Dickinson wrote: > Any way we could get the stats you need upstream in swift (emitted via > statsd) and then use something like https://github.com/danslimmon/ > statsd-opentsdb-backend to get it in to opentsdb? > > --John > > > > > On 9 Aug 2016, at 8:59, Alexandr Porunov wrote: > > > Hello, > > > > I need to collect different metrics from swift servers and store those > > metrics into OpenTSDB. > > I found that there is way to collect metrics with statsd and graphite but > > it isn't suitable for me because I have to store metrics into OpenTSDB. > > Is there any way to store metrics from openstack swift into OpenTSDB? > > I haven't found cli commands to get metrics out from Openstack swift. > > > > Sincerely, > > Alexandr > > ___ > > Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/ > openstack > > Post to : openstack@lists.openstack.org > > Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/ > openstack > ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
[Openstack] [OpenStack] [Swift] Is it possible to create chunked upload?
Hello, My server accepts files in chunks (4 Kbytes each chunk. File's size can be till 8 GB). Is it possible somehow store those chunks in Swift like a single file? Does somebody know any solution to solve this problem? Sincerely, Alexandr ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Re: [Openstack] [OpenStack] [Swift] Is it possible to create chunked upload?
Hello Jay, Unfortunately I don't know python. I am writing on java. Could you explain me briefly what I have to use to send chunks in the Swift and then combine those chunks? Or maybe we can append bytes to file somehow? If I understand correctly I need to store my first chunk as an individual object (0 - 4095 bytes) and then when I recieve the second chunk (with 4096 - 8192 bytes) I have to append that chunk to the first object. Is it correct? How can we append second chunk? Sincerely, Alexandr On Tue, Aug 23, 2016 at 5:49 PM, Jay Pipes wrote: > On 08/23/2016 10:13 AM, Alexandr Porunov wrote: > >> Hello, >> >> My server accepts files in chunks (4 Kbytes each chunk. File's size can >> be till 8 GB). Is it possible somehow store those chunks in Swift like a >> single file? Does somebody know any solution to solve this problem? >> > > Yes, you can do this. > > The Glance Swift driver shows an example of how to do this: > > https://github.com/openstack/glance_store/blob/master/glance > _store/_drivers/swift/store.py#L829-L944 > > Best, > -jay > > ___ > Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstac > k > Post to : openstack@lists.openstack.org > Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstac > k > ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Re: [Openstack] [OpenStack] [Swift] Is it possible to create chunked upload?
Thank you all! I will try these methods. Sincerely, Alexandr On Tue, Aug 23, 2016 at 8:14 PM, John Dickinson wrote: > There's 2 ways you can do this: > > 1) Use chunked transfer encoding > 2) Use Swift's large object manifest objects > > > For the first, it's the standard HTTP semantics. You can send chunks of > data to Swift to be stored as one object without knowing the full object > size up front. Note that the per-object size limits still apply here > (default is 5GB). > > For the second, see http://docs.openstack.org/developer/swift/overview_ > large_objects.html, specifically "Static Large Objects" on that page. > This will allow you to upload discrete objects and then stitch them > together into a single logical object of arbitrary size. > > --John > > > > > > On 23 Aug 2016, at 10:13, Alexandr Porunov wrote: > > > Hello, > > > > My server accepts files in chunks (4 Kbytes each chunk. File's size can > be > > till 8 GB). Is it possible somehow store those chunks in Swift like a > > single file? Does somebody know any solution to solve this problem? > > > > Sincerely, > > Alexandr > > ___ > > Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/ > openstack > > Post to : openstack@lists.openstack.org > > Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/ > openstack > ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
[Openstack] [OpenStack] [Swift] Is it possible to stream a file from a Java server to OpenStack Swift?
Hello, Is it possible to stream a file from a Java server to OpenStack Swift? I need to store very large files in the OpenStack swift through the Java server. Java server performs authentication, validation and other special logic after that it sends file into OpenStack Swift. The problem is that I don't need to store big files localy on the Java server. I don't need them there. So, Java server has to act like a pipe. It has to accept an input stream from the client and create an output stream to the OpenStack Swift Proxy after that it has to read small parts of data from the input stream and send them to the output stream. Are there any APIs which let us create such a Java server? Sincerely, Alexandr ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
[Openstack] [OpenStack] [Swift] Segments count minimization.
Hello, Does somebody know how to minimize segments count of a file? I have to implement an upload file service with the chunked upload feature. If a client lose connection for some time we have to be able continue to upload the file after connection is established (From the last successfuly uploaded chunk). For it I will use "Static Large Objects" to create small files each with 4 KBytes. The problem is that if we upload very big files then we will create too many small 4 KBytes segments of the big file. My service have to be able to store files up to 120GB. If we want to store a file with 120GB then we will have 31457280 segments of the file in the object storage. It isn't nice because of additional cost for metadate overhead of each file + big metadata file + decrease of performance. It will be much better if we could combine all that segments in a bigger segments each 1-2 GB. So, that after combination we would get maximum of 60-120 segments of the file. I know that we can combine all segments into one by using "copy" command to copy all segments into the new object but the problem is that if our object will be more than 5 GB then we can't use "copy" command (because it will show an error). Does somebody have any ideas how to develop chunk upload or minimize the count of segments? Sincerely, Alexandr ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Re: [Openstack] [OpenStack] [Swift] Segments count minimization.
Hello Clay, Thank you for your suggestions. I need to use Swift in several purposes. 1. As a temporary storage for uncompressed video scenes (y4m, mp4) up to 120GB. (After conversion they will be deleted). 2. As a persistent storage for compressed video scenes (mp4) up to 120GB. Most of the compressed files will be much smaller (up to 15GB) but I need to be able to store files up to 120GB in some rare situations. Thanks for suggestions again. I will increase the size of segments to 1MB. I haven't thought about two times segmentation. I think it will work. Correct me if I am wrong. Algorithm is the next: 1. Upload 1MB sub-segments (up to 500 sub-segments per segment). After they will be uploaded I will use "copy" request to create one 500MB segment. After that I will delete useless 1MB segments. 2. Upload up to 240 segments with 500MB and create a manifest for them. Is it correct? Will this algorithm be suitable for this situation? Best regards, Alexandr On Tue, Sep 13, 2016 at 7:24 PM, Clay Gerrard wrote: > 120GB is a rather largish file. A standard 4TB drive is only gunna be > able to hold a few tens of these. My macbook could only hold two. > > 4KB at the bottom end of where you want to be in object storage - we're > talking about the block device sector size at that point. > > The sweet spot for objects in Swift is probably in the 10MB-100MB range > (2MB-200MB is fine). Unless your primary platform is mobile there's a good > chance the lower end of that range is reasonable target for resuming upload > (oh, my last 500K of my 2MB upload crapped out - I'll just restart the > other 1.5M, no bigz) > > But 120GB is a rather largish file. The default SLO segment size is only > like 1000 - but it'd be perfectly reasonable to bump that up an order of > magnitude if you control the config for that cluster - then you could > upload 20MB objects and call it done - but that doesn't give you much room > to scale. > > I would suggest you consider breaking the 120GB upload into ~500MB > sub-segments (each with ~500x1MB objects) - then assemble the final SLO > manifest from the sub-segments [1]. > > Great to hear about people solving big storage problems with Swift - would > you care to share anymore about your use-case? Come find swifty folks in > #openstack-swift on Freenode if you wanna chat! > > -Clay > > 1. final paragraph of http://docs.openstack.org/developer/swift/overview_ > large_objects.html#uploading-the-manifest > > On Tue, Sep 13, 2016 at 5:21 AM, Alexandr Porunov < > alexandr.poru...@gmail.com> wrote: > >> Hello, >> >> Does somebody know how to minimize segments count of a file? >> I have to implement an upload file service with the chunked upload >> feature. If a client lose connection for some time we have to be able >> continue to upload the file after connection is established (From the last >> successfuly uploaded chunk). For it I will use "Static Large Objects" to >> create small files each with 4 KBytes. >> >> The problem is that if we upload very big files then we will create too >> many small 4 KBytes segments of the big file. >> >> My service have to be able to store files up to 120GB. >> If we want to store a file with 120GB then we will have 31457280 segments >> of the file in the object storage. It isn't nice because of additional cost >> for metadate overhead of each file + big metadata file + decrease of >> performance. >> >> It will be much better if we could combine all that segments in a bigger >> segments each 1-2 GB. So, that after combination we would get maximum of >> 60-120 segments of the file. >> >> I know that we can combine all segments into one by using "copy" command >> to copy all segments into the new object but the problem is that if our >> object will be more than 5 GB then we can't use "copy" command (because it >> will show an error). >> >> Does somebody have any ideas how to develop chunk upload or minimize the >> count of segments? >> >> Sincerely, >> Alexandr >> >> ___ >> Mailing list: http://lists.openstack.org/cgi >> -bin/mailman/listinfo/openstack >> Post to : openstack@lists.openstack.org >> Unsubscribe : http://lists.openstack.org/cgi >> -bin/mailman/listinfo/openstack >> >> > ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Re: [Openstack] [OpenStack] [Swift] Segments count minimization.
Thank you Clay! I use ffmpeg with remote input and local output for first convertion. Here is a simplified algorithm: 1. Convert with h264 in the highest quality (till 4K) and save it in a transformer server. 2. Convert into lower quality locally. And get additional data (thumbnails and so on). 3. Upload all converted videos and additional data into Swift (as you suggested with sub-segments and segments). Compressed videos then are available to the client throug the transmuxing servers. Best regards, Alexandr On Tue, Sep 13, 2016 at 8:55 PM, Clay Gerrard wrote: > > On Tue, Sep 13, 2016 at 10:37 AM, Alexandr Porunov < > alexandr.poru...@gmail.com> wrote: > >> >> Correct me if I am wrong. Algorithm is the next: >> 1. Upload 1MB sub-segments (up to 500 sub-segments per segment). After >> they will be uploaded I will use "copy" request to create one 500MB >> segment. After that I will delete useless 1MB segments. >> 2. Upload up to 240 segments with 500MB and create a manifest for them. >> >> > Is it correct? Will this algorithm be suitable for this situation? >> >> > That sounds correct - I think the COPY/bulk-delete is optional - you could > also just have the top level SLO point to the sub-segment SLO's manifest > directly and leave all of the 1MB chunks stored in Swift as is - could > potentially help smooth out the evenness of diskfullness overall. I think > 1MB is big enough to smooth out any connection overhead - but you could > test with COPY consolidation of the backend subsegments to objects too. > > Whatever works. > > Regardless it'll still take awhile to download a 120GB object. > > Do you stream the uncompressed videos directly into the encoder as you > upload the compressed format - or stage to disk and re-upload? What about > the playback of compressed videos - mp4 should http-pseudo stream directly > to an html5 browser just fine!? > > Cool use case! Good luck! > > -Clay > > > ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Re: [Openstack] [OpenStack] [Swift] Is it possible to stream a file from a Java server to OpenStack Swift?
Thanks again Clay! I will try JOSS Now I am joined to #openstack-swift Best regards, Alexandr On Wed, Sep 14, 2016 at 12:21 AM, Alexandr Porunov < alexandr.poru...@gmail.com> wrote: > Thanks again Clay! > > I will try JOSS > > Now I am joined to #openstack-swift > > Best regards, > Alexandr > > On Tue, Sep 13, 2016 at 10:34 PM, Clay Gerrard > wrote: > >> heh, didn't see this one when responding to the other message :P >> >> My favorite client bindings for the java's is JOSS -> >> http://joss.javaswift.org/ >> >> For streaming upload; maybe here: >> >> https://github.com/javaswift/tutorial-joss-streaming/blob/e9 >> 7a302e42b8964b4c87749fc2a5d28a9bb4d32a/src/main/java/org/jav >> aswift/joss/tutorial/StreamingController.java#L174 >> >> -Clay >> >> IMHO, if you're actively working on this project and find you have a >> bunch of these questions you should totally jump into #openstack-swift on >> Freenode! >> >> On Tue, Sep 13, 2016 at 2:55 AM, Alexandr Porunov < >> alexandr.poru...@gmail.com> wrote: >> >>> Hello, >>> >>> Is it possible to stream a file from a Java server to OpenStack Swift? >>> >>> I need to store very large files in the OpenStack swift through the Java >>> server. Java server performs authentication, validation and other special >>> logic after that it sends file into OpenStack Swift. >>> >>> The problem is that I don't need to store big files localy on the Java >>> server. I don't need them there. So, Java server has to act like a pipe. It >>> has to accept an input stream from the client and create an output stream >>> to the OpenStack Swift Proxy after that it has to read small parts of data >>> from the input stream and send them to the output stream. >>> >>> Are there any APIs which let us create such a Java server? >>> >>> Sincerely, >>> Alexandr >>> >>> ___ >>> Mailing list: http://lists.openstack.org/cgi >>> -bin/mailman/listinfo/openstack >>> Post to : openstack@lists.openstack.org >>> Unsubscribe : http://lists.openstack.org/cgi >>> -bin/mailman/listinfo/openstack >>> >>> >> > ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
[Openstack] [OpenStack] [Swift] Can we use tempauth with version 2 or 3 of identity?
Hello,
I am using tempauth for authentication. Here what I use:
curl -v -H 'X-Auth-User: admin:admin' -H 'X-Auth-Key: admin'
http://192.168.0.58:8080/auth/v1.0/
But I need to use v2 or v3 because I use a Java Client (jclouds) which can
use only v2 or v3 authentication. I tried:
curl -H "Content-Type: application/json" -X POST -d '{"auth" :
{"passwordCredentials" : {"username" : "admin","password" :
"admin"},"tenantName" : "admin"}}' http://192.168.0.58:8080/auth/v2.0/
unfortunately it doesn't work.
Does somebody have any ideas?
Sincerely,
Alexandr
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
[Openstack] [OpenStack] [keystone] How to make keystone highly available?
Hello, I am thinking about using the keystone as an authentication system but I am afraid about failures which can affect all the cluster. In fact if the keystone server dies then our full cluster will stop. It would be better if we could use HA with the keystone. Then if our primary keystone server dies we have to elect a new primary keystoe server. Are there some tools which can be used in HA deployment? Any piece of advice will be valuable Sincerely, Alexandr ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Re: [Openstack] [OpenStack] [keystone] How to make keystone highly available?
Thank you for pointing on it. I haven't yet use pacemaker but I will try to configure it. Sincerely, Alexandr On Mon, Sep 19, 2016 at 10:23 PM, Remo Mattei wrote: > There is already ha options with pacemaker and corosync. Have you looked > at that? > > Inviato da iPhone > > Il giorno 19 set 2016, alle ore 11:46, Alexandr Porunov < > alexandr.poru...@gmail.com> ha scritto: > > Hello, > > I am thinking about using the keystone as an authentication system but I > am afraid about failures which can affect all the cluster. In fact if the > keystone server dies then our full cluster will stop. It would be better if > we could use HA with the keystone. Then if our primary keystone server dies > we have to elect a new primary keystoe server. Are there some tools which > can be used in HA deployment? > > Any piece of advice will be valuable > > Sincerely, > Alexandr > !DSPAM:1,57e036cd282281177211101! > > ___ > Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/ > openstack > Post to : openstack@lists.openstack.org > Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/ > openstack > > > !DSPAM:1,57e036cd282281177211101! > > ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Re: [Openstack] [OpenStack] [keystone] How to make keystone highly available?
Hello everyone, Thank you all for your advice! In my case one keystone server can easily hold a load. I don't need to balance a load between two or more keystone servers. However I need two keystone servers for high availability. So, I decided just to use two keystone servers with the same virtual IP address. Pros: - I can easily configure HA - I don't need to use a master/master replication for a database. - I don't need to use load balancers. - I don't need to learn peacemaker Cons: - It will not increase the overall throughput - One of the keystone servers will always be at a cold state However. We can rid of cons by using two virtual IP adresses. So, we can use a half proxy servers with keystone1 and another half of proxy servers with keystone2. If one of the keystone servers will be dead then keepalived will assign both virtual ip addresses to the alive node. After node will be brought back to live keepalived will reassign IP addresses back (It isn't the load balancing but our throughput will increase). If our clients will use proxy servers with both keystone1 and keystone 2 then we have to use master/master replication for the database. Best regards, Alexandr On Tue, Sep 20, 2016 at 8:49 AM, Van Leeuwen, Robert wrote: > > > > > Hello, > > > > > > I am thinking about using the keystone as an authentication system but I > am afraid about failures which can affect all the cluster. In fact if the > keystone server dies then our full cluster will stop. It would be better if > we could use HA with the keystone. Then if > our primary keystone server > dies we have to elect a new primary keystoe server. Are there some tools > which can be used in HA deployment? > > > > > > Any piece of advice will be valuable > > > > IMHO: get a proper ha load-balancer solution, make sure you install at > least 2 of all APIS and load-balance in active-active mode > > The OpenStack APIs are stateless so you can run multiple servers at the > same time as long as they can connect to the same database backend. > > > > The load-balancer is usable for all APIs, you can use it as the HA-IP for > mysql and can also be used to offload SSL so you just have one place to > configure your certificates. > > (for galera, make use of a “sorry-server” and not active-active > load-balancing for writes) > > > > The great thing about a load-balancer is that you remove all HA complexity > out of the OpenStack setup. > > In my personal opinion the pacemaker setups can function OK but pacemaker > is a complex piece of software and it is not unlikely to cause downtime > either to misconfiguration or inexperienced people operating it. > > Especially if pacemaker is also starting/stopping mysql/rabbit/openstack > services and not only moving around a few IPs. > > If you are going the pacemaker way make sure you play around with it quite > a bit and do failure tests so you are comfortable with the commands and > know what to look for when things go wrong. > > Do not forget: any other people operating the pacemaker cluster will need > that knowledge and a 2 node cluster is not a real cluster (split-brains). > > > > Cheers, > > Robert van Leeuwen > ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Re: [Openstack] [OpenStack] [keystone] How to make keystone highly available?
I think that I haven't understood your situation correctly but I will try to suggest something: If you have a big load on your keystone server you can spread the load between two or more servers by using load balancers like HAProxy. But it wouldn't work if keystone instances not shared tokens with each other. For it you have to share databse to store all tokens which are generated by your keystone servers. Personally I prefer MariaDB Galera Cluster because it isn't hard to install and works very nice. To reach high availability for load balancing you need to have one more load balancer managed by keepalived. (I am assuming that you don't have such a load) If you care about high availability (as I do) then you need to have additional keystone instance which will prevent your cluster from SPOF. For it I use the same virtual IP address for both keystone instances managed by keepalived. Also you can use peacemaker and other stuff to reach high availability but I can't give you advice for those tools because I haven't used them. In your situation I would installed additional keystone instance in one of your node and keepalived on both nodes to prevent SPOF. Sincerely, Alexandr On Tue, Sep 20, 2016 at 10:56 PM, Turbo Fredriksson wrote: > On Sep 20, 2016, at 3:09 PM, Alexandr Porunov wrote: > > > So, I decided just to use two keystone servers with the same virtual IP > address. > > Now that you've made your decision, I'd like to ask some > followup questions for my own decision if I may. > > > Most everyone have talked about some kind of load balancer or > HA solution. > > My setup is such that that wouldn't be .. "workable". > > My whole Openstack setup is installed on a HP Bladecenter > with 16 half hight nodes so all communication etc is pretty > much internal in the blade center. I.e, the only external > link is the one to the rest of the flat and then to the > gatway/firewall/NATbox. > > Also, having a LB in front of MySQL (and possibly one more > in front of Keystone), that will introduce a "single point > of failure" that I'd prefer not to have (even if it was > practically possible - I don't want to dedicate a whole > BC node just for that, they're to precious). > > > What would be the impact and/or problem by using DNS round > robin for MySQL and Keystone for example? And then have > MySQL in a master-master setup. I've never done that either, > so I'm not sure how good idea that would be.. Any pointers? > > The only DB I've ever had the .. "displeasure" to try to get > to work in a master-master setup is OpenLDAP and I can remember > (vividly!) the discussions that's been on the OpenLDAP lists > about this over the years! In essence, "don't, for the love of > whoever, do it!!". > -- > Choose a job you love, and you will never have > to work a day in your life. > > > ___ > Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/ > openstack > Post to : openstack@lists.openstack.org > Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/ > openstack > ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Re: [Openstack] [OpenStack] [keystone] How to make keystone highly available?
> That is more like a hot-standby (only one server is used at any one time) > and I guess that's an option as well. But because I have limited resources, > I'd prefer, if possible, to use all of them all the time. Wrong. It depends on how you configured you cluster. You can point half of your machines to use keystone1 and other half machines to use keystone2. For it you have to configure your keepalived to manage two virtual ip address with different priority. Example: keystone1: 192.168.0.61, proirity 101 keystone1: 192.168.0.62, priority 100 keystone2: 192.168.0.61 proirity 100 keystone2: 192.168.0.62, priority 101 your 8 machines (nove, swift, glance, and so on) will use 192.168.0.61 your other 8 machines will use 192.168.0.62 Assuming your keystone1 is dead then keepalived will assign both IPs for keystone2 (i.e. 192.168.0.61, 192.168.0.62). Your all machines then automatically will use keystone2. After your keystone1 is returned back to life your keepalived will reassign IP address 192.168.0.61 back to keystone1 bacause 192.168.0.61 has higher priority for keystone1 than keystone2. Sincerely, Alexandr On Wed, Sep 21, 2016 at 12:44 AM, Turbo Fredriksson wrote: > On Sep 20, 2016, at 10:06 PM, Alexandr Porunov wrote: > > > If you care about high availability (as I do) then you need to have > > additional keystone instance which will prevent your cluster from SPOF. > > That was the idea. One node is already dedicated for that, but I haven't > installed it yet, because I'm not sure what the right way to do this is. > > I've been reading the high availability docs for the last couple of weeks > now, but they don't talk about the most basic things :(. For once, they > get right down to it (which I've complained about other docs don't in the > past :D). > > > IF (!) I understand things correctly, services report themselves into > the catalog (which is basically the *SQL server). This catalog is Keystone. > > So my first Keystone server have registered itself as: > > bladeA01:~# openstack endpoint list | grep keystone > | 26855d6e55284651a0fcaa5cf25b3d90 | europe-london | keystone | > identity| True| internal | http://10.0.4.1:5000/v2.0 > | > | 72eb76cdb6cf4db3813eac4a683e4e34 | europe-london | keystone | > identity| True| public| http://10.0.4.1:5000/v2.0 > | > | c95ce66a4efa46b4855185d088279824 | europe-london | keystone | > identity| True| admin | http://10.0.4.1:35357/v2.0 > | > > Now, I'm assuming that the second one will do the same (on ITS IP of > course). So "anyone" needing to contact Keystone, will, I assume, > consult this catalog and "pick one". > > > So if I use a DNS round-robin for the two keystones: > > openstack.domain.tld.1 IN A 10.0.4.1 > openstack.domain.tld.1 IN A 10.0.4.9 > > the entry for 'openstack.domain.tld.' will be invalidated every second, > practically making sure that a new request will be given every time. > The downside is that if the .1 keystone is down, the service will have > to wait for the time out before it can ask the next one. > > Round-robin is "poor mans load balancer", and it have many flaws, but > it will at least give some form of using all available resources at any > one time. > > > For it I use the same virtual IP address for both keystone instances > > managed by keepalived. > > That is more like a hot-standby (only one server is used at any one time) > and I guess that's an option as well. But because I have limited resources, > I'd prefer, if possible, to use all of them all the time. > > Balancing the load is less of an issue than high availability is, but > if possible, I'd like to solve both of them :). > > > Also you can use peacemaker and other stuff to reach high availability > > Yes, but I'm guessing those need _another_ machine in front of the ones > I want to load balance. And if that goes down, EVERYTHING stops working. > Unless they are clustered, which require _even more_ machines! > > Which I don't have. I don't want to dedicate a dual CPU, eight core > Intel Xeon E5530 @ 2.40GHz just to swap traffic around! It's a huge > waste of precious resources. And if I add smaller machines outside of > the blade center, then ALL traffic needs to go out in the rack and then > back in, which will affect performance (which is already kind'a bad > because it's an older setup with only Gbps links). > > I had to dedicate one whole switch (there's two Cisco 3020 in the > blade center) just for the truncated link down to the storage. Which > is anothe
Re: [Openstack] [OpenStack] [keystone] How to make keystone highly available?
Also, if you haven't changed default parameter for token cache time then it has 5 min of cache (you can increase this time if you want). So, your 15-16 machines (or less if some of them doesn't use keystone like swift object, container, account nodes) will use requests to keystone only once in 5 minutes. It is about one request in 18-19 seconds. If you will use that scheme like I said it kill be about one request in 36-38 seconds. I don't recomend you to use load balancer on both your keystone servers because it isn't really effective. Half of your requests will ask one keystone server to send request to another keystone server to get a token. It isn't really effective because you send your request to both keystone servers rather than one keystone server. But in your situation I think it doesn't matter. Sincerely, Alexandr On Wed, Sep 21, 2016 at 9:29 AM, Alexandr Porunov < alexandr.poru...@gmail.com> wrote: > > That is more like a hot-standby (only one server is used at any one time) > > and I guess that's an option as well. But because I have limited > resources, > > I'd prefer, if possible, to use all of them all the time. > > Wrong. It depends on how you configured you cluster. You can point half of > your machines to use keystone1 and other half machines to use keystone2. > For it you have to configure your keepalived to manage two virtual ip > address with different priority. Example: > keystone1: 192.168.0.61, proirity 101 > keystone1: 192.168.0.62, priority 100 > keystone2: 192.168.0.61 proirity 100 > keystone2: 192.168.0.62, priority 101 > > your 8 machines (nove, swift, glance, and so on) will use 192.168.0.61 > your other 8 machines will use 192.168.0.62 > > Assuming your keystone1 is dead then keepalived will assign both IPs for > keystone2 (i.e. 192.168.0.61, 192.168.0.62). Your all machines then > automatically will use keystone2. After your keystone1 is returned back to > life your keepalived will reassign IP address 192.168.0.61 back to > keystone1 bacause 192.168.0.61 has higher priority for keystone1 than > keystone2. > > Sincerely, > Alexandr > > On Wed, Sep 21, 2016 at 12:44 AM, Turbo Fredriksson > wrote: > >> On Sep 20, 2016, at 10:06 PM, Alexandr Porunov wrote: >> >> > If you care about high availability (as I do) then you need to have >> > additional keystone instance which will prevent your cluster from SPOF. >> >> That was the idea. One node is already dedicated for that, but I haven't >> installed it yet, because I'm not sure what the right way to do this is. >> >> I've been reading the high availability docs for the last couple of weeks >> now, but they don't talk about the most basic things :(. For once, they >> get right down to it (which I've complained about other docs don't in the >> past :D). >> >> >> IF (!) I understand things correctly, services report themselves into >> the catalog (which is basically the *SQL server). This catalog is >> Keystone. >> >> So my first Keystone server have registered itself as: >> >> bladeA01:~# openstack endpoint list | grep keystone >> | 26855d6e55284651a0fcaa5cf25b3d90 | europe-london | keystone | >> identity| True| internal | http://10.0.4.1:5000/v2.0 >> | >> | 72eb76cdb6cf4db3813eac4a683e4e34 | europe-london | keystone | >> identity| True| public| http://10.0.4.1:5000/v2.0 >> | >> | c95ce66a4efa46b4855185d088279824 | europe-london | keystone | >> identity| True| admin | http://10.0.4.1:35357/v2.0 >>| >> >> Now, I'm assuming that the second one will do the same (on ITS IP of >> course). So "anyone" needing to contact Keystone, will, I assume, >> consult this catalog and "pick one". >> >> >> So if I use a DNS round-robin for the two keystones: >> >> openstack.domain.tld.1 IN A 10.0.4.1 >> openstack.domain.tld.1 IN A 10.0.4.9 >> >> the entry for 'openstack.domain.tld.' will be invalidated every second, >> practically making sure that a new request will be given every time. >> The downside is that if the .1 keystone is down, the service will have >> to wait for the time out before it can ask the next one. >> >> Round-robin is "poor mans load balancer", and it have many flaws, but >> it will at least give some form of using all available resources at any >> one time. >> >> > For it I use the same virtual IP address for both keystone instances >> > managed by keepalived. &
[Openstack] [OpenStack] [Keystone] How to manage permissions for a role in keystone?
Hello, I have installed Swift and Keystone. Now I want to create several users with different permissions: reader - can read from the next containers: "video", "audio", "subtitles", "photos" media_manager - can do anything in the next containers: "video", "audio", "subtitles", "photos" crypt_manager - can not do anything in Swift but can get tokens directly from keystone (it is for other usage). There are a lot of things in keystone (user, role, project, service, endpoint, region-id, admin-url, public-url, internal-url) and it is little bit confusing. Can somebody explain me how to configure such users with those roles? I haven't bootstrap the keystone, so I haven't the admin role yet. I am worried about security with an administrator user. Do we need to define it? I have read examples which says that firstly you have to bootstrap your keystone and it will create the admin user with the admin role: keystone-manage bootstrap --bootstrap-password s3cr3t Also the full command for define all things is: keystone-manage bootstrap \ --bootstrap-password s3cr3t \ --bootstrap-username admin \ --bootstrap-project-name admin \ --bootstrap-role-name admin \ --bootstrap-service-name keystone \ --bootstrap-region-id RegionOne \ --bootstrap-admin-url http://localhost:35357 \ --bootstrap-public-url http://localhost:5000 \ --bootstrap-internal-url http://localhost:5000 What is "role"? It is little bit confusing because it has name "admin". Which roles we can use except admin? What permissions they can give to the user? Also we can create additional roles: keystone role-create --name my_new_role But what this role mean? How to set some permissions on this role (i.e. if I want to set readonly permission for all in swift but write only for some containers?) What we should specify in a region-id? What we should specify in admin,public,internal url? What they mean? Sorry for a lot of questions Sincerely, Alexandr ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
[Openstack] [OpenStack] [Keystone] How to use two keystone servers?
Hello, I have next nodes: swift_proxy1 - 192.168.0.11 swift_proxy2 - 192.168.0.12 keystone1 - 192.168.0.21 keystone2 - 192.168.0.22 I wonder to know if it is possible to use two keystone servers if we use "uuid" or "fernet" tokens. With uuid I can use Galera Cluster to use the same database. The problem is I don't know what to write in endpoints. We can create an endpoint only for one the keystone server as I understand. i.e. : openstack endpoint create --region RegionOne identity public http://192.168.0.11:5000/v3 openstack endpoint create --region RegionOne identity internal http://192.168.0.11:5000/v3 openstack endpoint create --region RegionOne identity admin http://192.168.0.11:35357/v3 Also what should I use when I create a swift endpoints? Does he have to point on itself or on the keystone server? openstack endpoint create --region RegionOne \ object-store public http://192.168.0.11:8080/v1/AUTH_%\(tenant_id\)s openstack endpoint create --region RegionOne \ object-store internal http://192.168.0.11:8080/v1/AUTH_%\(tenant_id\)s openstack endpoint create --region RegionOne \ object-store admin http://192.168.0.11:8080/v1 i.e. if it points on the keystone then it is OK but if he point on itself then which proxy IP address I should use? Also when we use fernet tokens, how to synchronize tokens between nodes (they should always use the same tokens). My aim is to connect the keystone 1 to the proxy 1 and the keystone 2 to the proxy 2 to. i.e. : Proxy 1 should always check tokens only in the keystone1 server and the proxy 2 should always check tokens only in the keystone 2 server. But I want to be able to receive tokens from any keystone server (a user can receive a token from the keystone 1 and be authenticated in the proxy 2 with that token). Sincerely, Alexandr ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Re: [Openstack] [OpenStack] [keystone] How to make keystone highly available?
> How do you handle database high availability and replication? Especially in my case, I don't care about tokens which will be lost after first keystone server dies. My services can authenticate again and get new tokens. It isn't critical. But if in your case it isn't acceptable then I would have used fernet tokens on your place (but they are a little bit bigger than uuid tokens). If you need small tokens, fast checks and you can't lose tokens then I would have used MariaDB Galera Cluster for the token replication. On Fri, Sep 30, 2016 at 3:41 PM, Clint Byrum wrote: > Excerpts from Alexandr Porunov's message of 2016-09-20 17:09:06 +0300: > > Hello everyone, > > > > Thank you all for your advice! > > In my case one keystone server can easily hold a load. I don't need to > > balance a load between two or more keystone servers. However I need two > > keystone servers for high availability. So, I decided just to use two > > keystone servers with the same virtual IP address. > > Pros: > > - I can easily configure HA > > - I don't need to use a master/master replication for a database. > > How do you handle database high availability and replication? > > ___ > Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/ > openstack > Post to : openstack@lists.openstack.org > Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/ > openstack > ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
[Openstack] [Keystone] How to use several endpoints for Swift proxy nodes?
Hello, I am using a Java client (Openstack4j) for send objects to the openstack swift. Openstack4j doesn't allow to manually set proxy node IP addresses. You have to use keystone endpoint for it and it will create a token and will use the endpoint for proxy which was returned by keystone. Sometimes proxy nodes can crashes and clients won't reconnect to the available nodes. I want to know if it is possible to manually set several endpoints for each proxy server in keystone. Now I use the next public endpoint for proxy nodes: http://proxy:8080/v1/AUTH_%\(tenant_id\)s On each client machine I have to set manually IP address for example: if I want that clients A, B and C use proxy with ip 192.168.0.41 then I have to set next line in /etc/hosts: 192.168.0.41 proxy If I want that clints D, E and F use proxy with ip 192.168.0.42 then I have to set next line in /etc/hosts: 192.168.0.42 proxy So, clients A, B and C will fail if proxy with the IP 192.168.0.41 crashes. They can't use proxy with IP 192.168.0.42 because Openstack4j doesn't support to set proxy IPs manually. Does somebody know if it is possible to set several endpoints for swift proxy nodes in keystone? Sincerely, Alexandr ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
[Openstack] [OpenStack] [Swift] Does Swift3 supports SLO (Manifest creation)?
Hello, I want to use S3 API for OpenStack Swift but I need chunked upload for my files. I want to save many small object (1-10 MB) and retrieve them as a single object. For it I need to be able to create a manifest file. Does Swift3 support manifest creation? If yes then where can I see an example? Sincerely, Alexandr ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Re: [Openstack] [OpenStack] [Swift] Does Swift3 supports SLO (Manifest creation)?
Hello, It is what I need. Thank you very much Sincerely, Alexandr On Wed, Oct 12, 2016 at 4:04 AM, Kota TSUYUZAKI < tsuyuzaki.k...@lab.ntt.co.jp> wrote: > Hello Alexandr, > > Swift3 supports Multipart Upload API for actual Amazon S3[1] which has > similar functionality with Swift SLO manifest so that you can use any SDK > for s3 multiupload api. > It needs, > > 1. init a multipart Upload > 2. upload segments (chunk you may call) > 3. complete the multipart upload > > > boto[2] known as python sdk also supports it. And it may be helpful to > see swift3 functional test[3] to know how we can call the apis. > > Thanks, > Kota > > > 1: http://docs.aws.amazon.com/AmazonS3/latest/dev/mpuAndPermissions.html > 2: http://boto.cloudhackers.com/en/latest/ref/s3.html > 3: https://github.com/openstack/swift3/blob/master/swift3/ > test/functional/test_multi_upload.py > > (2016/10/12 4:32), Alexandr Porunov wrote: > > Hello, > > > > I want to use S3 API for OpenStack Swift but I need chunked upload for my > > files. I want to save many small object (1-10 MB) and retrieve them as a > > single object. For it I need to be able to create a manifest file. Does > > Swift3 support manifest creation? If yes then where can I see an example? > > > > Sincerely, > > Alexandr > > > > > > > > ___ > > Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/ > openstack > > Post to : openstack@lists.openstack.org > > Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/ > openstack > > > > > -- > -- > Kota Tsuyuzaki(露﨑 浩太) > NTT Software Innovation Center > Cloud Solution Project > Phone 0422-59-2837 > Fax0422-59-2965 > --- > > > ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
[Openstack] [OpenStack] [Swift] How to install Swift3 middleware?
Hello, I want to use S3 api for Swift. I have installed swift3 as here: git clone https://github.com/openstack/swift3 cd swift3 python setup.py install Then I have configured my proxy-server.conf as here: [pipeline:main] pipeline = catch_errors gatekeeper healthcheck proxy-logging cache container_sync bulk ratelimit swift3 s3token authtoken keystoneauth container-quotas account-quotas slo dlo versioned_writes proxy-logging proxy-server [filter:swift3] use = egg:swift3#swift3 [filter:s3token] use = egg:swift3#s3token auth_uri = http://keystone:35357/ reseller_prefix = AUTH_ Then I tried to restart the proxy: systemctl restart openstack-swift-proxy And it failes. systemctl status openstack-swift-proxy -l shows next: ● openstack-swift-proxy.service - OpenStack Object Storage (swift) - Proxy Server Loaded: loaded (/usr/lib/systemd/system/openstack-swift-proxy.service; enabled; vendor preset: disabled) Active: failed (Result: exit-code) since Wed 2016-10-12 23:17:06 EEST; 3min 0s ago Process: 2213 ExecStart=/usr/bin/swift-proxy-server /etc/swift/proxy-server.conf (code=exited, status=1/FAILURE) Main PID: 2213 (code=exited, status=1/FAILURE) Oct 12 23:17:06 proxy2 systemd[1]: Started OpenStack Object Storage (swift) - Proxy Server. Oct 12 23:17:06 proxy2 systemd[1]: Starting OpenStack Object Storage (swift) - Proxy Server... Oct 12 23:17:06 proxy2 liberasurecode[2213]: liberasurecode_backend_open: dynamic linking error libJerasure.so.2: cannot open shared object file: No such file or directory Oct 12 23:17:06 proxy2 liberasurecode[2213]: liberasurecode_backend_open: dynamic linking error libJerasure.so.2: cannot open shared object file: No such file or directory Oct 12 23:17:06 proxy2 liberasurecode[2213]: liberasurecode_backend_open: dynamic linking error libisal.so.2: cannot open shared object file: No such file or directory Oct 12 23:17:06 proxy2 liberasurecode[2213]: liberasurecode_backend_open: dynamic linking error libshss.so.1: cannot open shared object file: No such file or directory Oct 12 23:17:06 proxy2 systemd[1]: openstack-swift-proxy.service: main process exited, code=exited, status=1/FAILURE Oct 12 23:17:06 proxy2 systemd[1]: Unit openstack-swift-proxy.service entered failed state. Oct 12 23:17:06 proxy2 systemd[1]: openstack-swift-proxy.service failed. If I remove "swift3 s3token" form pipeline then it works.. Why my proxy crashes with swift3? I have an OpenStack Swift Mitaka Sincerely, Alexandr ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Re: [Openstack] [OpenStack] [Swift] How to install Swift3 middleware?
No, I have only OpenStack Swift and Keystone. I used simple installation and configuration from this tutorials: http://docs.openstack.org/mitaka/install-guide-rdo/ On Wed, Oct 12, 2016 at 8:44 PM, John van Ommen wrote: > Are you installing OpenStack from trunk? > > I've installed Swift3 on Helion Open Stack 2.0, and the published > instructions didn't work for me at all. > > On Wed, Oct 12, 2016 at 10:26 AM, Alexandr Porunov > wrote: > > Hello, > > > > I want to use S3 api for Swift. > > > > I have installed swift3 as here: > > > > git clone https://github.com/openstack/swift3 > > cd swift3 > > python setup.py install > > > > Then I have configured my proxy-server.conf as here: > > > > [pipeline:main] > > pipeline = catch_errors gatekeeper healthcheck proxy-logging cache > > container_sync bulk ratelimit swift3 s3token authtoken keystoneauth > > container-quotas account-quotas slo dlo versioned_writes proxy-logging > > proxy-server > > > > [filter:swift3] > > use = egg:swift3#swift3 > > > > [filter:s3token] > > use = egg:swift3#s3token > > auth_uri = http://keystone:35357/ > > reseller_prefix = AUTH_ > > > > Then I tried to restart the proxy: > > systemctl restart openstack-swift-proxy > > > > And it failes. > > > > systemctl status openstack-swift-proxy -l > > shows next: > > > > ● openstack-swift-proxy.service - OpenStack Object Storage (swift) - > Proxy > > Server > >Loaded: loaded (/usr/lib/systemd/system/ > openstack-swift-proxy.service; > > enabled; vendor preset: disabled) > >Active: failed (Result: exit-code) since Wed 2016-10-12 23:17:06 EEST; > > 3min 0s ago > > Process: 2213 ExecStart=/usr/bin/swift-proxy-server > > /etc/swift/proxy-server.conf (code=exited, status=1/FAILURE) > > Main PID: 2213 (code=exited, status=1/FAILURE) > > > > Oct 12 23:17:06 proxy2 systemd[1]: Started OpenStack Object Storage > (swift) > > - Proxy Server. > > Oct 12 23:17:06 proxy2 systemd[1]: Starting OpenStack Object Storage > (swift) > > - Proxy Server... > > Oct 12 23:17:06 proxy2 liberasurecode[2213]: liberasurecode_backend_open: > > dynamic linking error libJerasure.so.2: cannot open shared object file: > No > > such file or directory > > Oct 12 23:17:06 proxy2 liberasurecode[2213]: liberasurecode_backend_open: > > dynamic linking error libJerasure.so.2: cannot open shared object file: > No > > such file or directory > > Oct 12 23:17:06 proxy2 liberasurecode[2213]: liberasurecode_backend_open: > > dynamic linking error libisal.so.2: cannot open shared object file: No > such > > file or directory > > Oct 12 23:17:06 proxy2 liberasurecode[2213]: liberasurecode_backend_open: > > dynamic linking error libshss.so.1: cannot open shared object file: No > such > > file or directory > > Oct 12 23:17:06 proxy2 systemd[1]: openstack-swift-proxy.service: main > > process exited, code=exited, status=1/FAILURE > > Oct 12 23:17:06 proxy2 systemd[1]: Unit openstack-swift-proxy.service > > entered failed state. > > Oct 12 23:17:06 proxy2 systemd[1]: openstack-swift-proxy.service failed. > > > > > > If I remove "swift3 s3token" form pipeline then it works.. Why my proxy > > crashes with swift3? > > > > I have an OpenStack Swift Mitaka > > > > Sincerely, > > Alexandr > > > > ___ > > Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/ > openstack > > Post to : openstack@lists.openstack.org > > Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/ > openstack > > > ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Re: [Openstack] [OpenStack] [Swift] How to install Swift3 middleware?
Hello, It still doesn't work... The same error is shown. Maybe I missed something? Maybe before the installation I had to done something or something else. I tried different things to run it. But when we do have swift3 or/and s3token in pipeline it doesn't work. Sincerely, Alexandr On Thu, Oct 13, 2016 at 12:17 AM, Mark Kirkwood < mark.kirkw...@catalyst.net.nz> wrote: > > > On 13/10/16 06:26, Alexandr Porunov wrote: > >> Hello, >> >> I want to use S3 api for Swift. >> >> I have installed swift3 as here: >> >> git clone https://github.com/openstack/swift3 >> cd swift3 >> python setup.py install >> >> Then I have configured my proxy-server.conf as here: >> >> [pipeline:main] >> pipeline = catch_errors gatekeeper healthcheck proxy-logging cache >> container_sync bulk ratelimit swift3 s3token authtoken keystoneauth >> container-quotas account-quotas slo dlo versioned_writes proxy-logging >> proxy-server >> >> [filter:swift3] >> use = egg:swift3#swift3 >> >> [filter:s3token] >> use = egg:swift3#s3token >> auth_uri = http://keystone:35357/ >> reseller_prefix = AUTH_ >> >> Then I tried to restart the proxy: >> systemctl restart openstack-swift-proxy >> >> And it failes. >> >> >> > Yeah - for Mitaka (or thereabouts) onwards that s3token config needs to > change: > > [filter:s3token] > paste.filter_factory = keystoneclient.middleware.s3_token:filter_factory > auth_uri = http://keystone:35357/ > reseller_prefix = AUTH_ > > Alternatively the older port/protocol/host variables still work instead of > auth_uri: > > [filter:s3token] > paste.filter_factory = keystoneclient.middleware.s3_token:filter_factory > auth_port = 35357 > auth_host = keystone > auth_protocol = http > > The swift3 docs need updating, as it is not obvious what has gone wrong > when you hit this. > > regards > > Mark > > > > > > > ___ > Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstac > k > Post to : openstack@lists.openstack.org > Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstac > k > ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Re: [Openstack] [OpenStack] [Swift] How to install Swift3 middleware?
No, I didn't. I ran "python setup.py install" one more time. Here is the output: http://paste.openstack.org/show/585535/ On Thu, Oct 13, 2016 at 8:21 AM, John van Ommen wrote: > To me, the errors you're receiving are indicative of the swift3 > package not installing correctly. > > When you ran that first step (setup.py), did you get errors? > > John > > On Wed, Oct 12, 2016 at 10:03 PM, Alexandr Porunov > wrote: > > Hello, > > > > It still doesn't work... The same error is shown. Maybe I missed > something? > > Maybe before the installation I had to done something or something else. > I > > tried different things to run it. But when we do have swift3 or/and > s3token > > in pipeline it doesn't work. > > > > Sincerely, > > Alexandr > > > > On Thu, Oct 13, 2016 at 12:17 AM, Mark Kirkwood > > wrote: > >> > >> > >> > >> On 13/10/16 06:26, Alexandr Porunov wrote: > >>> > >>> Hello, > >>> > >>> I want to use S3 api for Swift. > >>> > >>> I have installed swift3 as here: > >>> > >>> git clone https://github.com/openstack/swift3 > >>> cd swift3 > >>> python setup.py install > >>> > >>> Then I have configured my proxy-server.conf as here: > >>> > >>> [pipeline:main] > >>> pipeline = catch_errors gatekeeper healthcheck proxy-logging cache > >>> container_sync bulk ratelimit swift3 s3token authtoken keystoneauth > >>> container-quotas account-quotas slo dlo versioned_writes proxy-logging > >>> proxy-server > >>> > >>> [filter:swift3] > >>> use = egg:swift3#swift3 > >>> > >>> [filter:s3token] > >>> use = egg:swift3#s3token > >>> auth_uri = http://keystone:35357/ > >>> reseller_prefix = AUTH_ > >>> > >>> Then I tried to restart the proxy: > >>> systemctl restart openstack-swift-proxy > >>> > >>> And it failes. > >>> > >>> > >> > >> Yeah - for Mitaka (or thereabouts) onwards that s3token config needs to > >> change: > >> > >> [filter:s3token] > >> paste.filter_factory = keystoneclient.middleware.s3_ > token:filter_factory > >> auth_uri = http://keystone:35357/ > >> reseller_prefix = AUTH_ > >> > >> Alternatively the older port/protocol/host variables still work instead > of > >> auth_uri: > >> > >> [filter:s3token] > >> paste.filter_factory = keystoneclient.middleware.s3_ > token:filter_factory > >> auth_port = 35357 > >> auth_host = keystone > >> auth_protocol = http > >> > >> The swift3 docs need updating, as it is not obvious what has gone wrong > >> when you hit this. > >> > >> regards > >> > >> Mark > >> > >> > >> > >> > >> > >> > >> ___ > >> Mailing list: > >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack > >> Post to : openstack@lists.openstack.org > >> Unsubscribe : > >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack > > > > > > > > ___ > > Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/ > openstack > > Post to : openstack@lists.openstack.org > > Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/ > openstack > > > ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Re: [Openstack] [OpenStack] [Swift] How to install Swift3 middleware?
Thank you all for your help! It have started the proxy finally. Here is the configuration that I used: [filter:swift3] use = egg:swift3#swift3 [filter:s3token] paste.filter_factory = swift3.s3_token_middleware:filter_factory auth_uri = http://keystone:35357/ Also I have installed dependencies: pip install -r requirements.txt pip install -r test-requirements.txt It seems that swift3 has to update the documentation on github Sincerely, Alexandr On Thu, Oct 13, 2016 at 9:23 AM, Kota TSUYUZAKI < tsuyuzaki.k...@lab.ntt.co.jp> wrote: > Hi Alexandr, > > Though I cannot see the actual reason why the proxy-server failed (your > report looks just proxy-server exist with status code 1), > looking at your procedure, you might need to install dependencies (e.g. > lxml) for swift3. I mean, > > >>>>> git clone https://github.com/openstack/swift3 > >>>>> cd swift3 > >>>>> python setup.py install > > sudo pip install -r requirements.txt > (you may need also "sudo pip install -r test-requirements.txt" for testing) > > If you could share sort of logs why proxy-server got failed, it might be > helpful to figure out more. > > Thanks, > Kota > > > (2016/10/13 14:34), Alexandr Porunov wrote: > > No, I didn't. > > > > I ran "python setup.py install" one more time. Here is the output: > > http://paste.openstack.org/show/585535/ > > > > On Thu, Oct 13, 2016 at 8:21 AM, John van Ommen > > > wrote: > > > >> To me, the errors you're receiving are indicative of the swift3 > >> package not installing correctly. > >> > >> When you ran that first step (setup.py), did you get errors? > >> > >> John > >> > >> On Wed, Oct 12, 2016 at 10:03 PM, Alexandr Porunov > >> wrote: > >>> Hello, > >>> > >>> It still doesn't work... The same error is shown. Maybe I missed > >> something? > >>> Maybe before the installation I had to done something or something > else. > >> I > >>> tried different things to run it. But when we do have swift3 or/and > >> s3token > >>> in pipeline it doesn't work. > >>> > >>> Sincerely, > >>> Alexandr > >>> > >>> On Thu, Oct 13, 2016 at 12:17 AM, Mark Kirkwood > >>> wrote: > >>>> > >>>> > >>>> > >>>> On 13/10/16 06:26, Alexandr Porunov wrote: > >>>>> > >>>>> Hello, > >>>>> > >>>>> I want to use S3 api for Swift. > >>>>> > >>>>> I have installed swift3 as here: > >>>>> > >>>>> git clone https://github.com/openstack/swift3 > >>>>> cd swift3 > >>>>> python setup.py install > >>>>> > >>>>> Then I have configured my proxy-server.conf as here: > >>>>> > >>>>> [pipeline:main] > >>>>> pipeline = catch_errors gatekeeper healthcheck proxy-logging cache > >>>>> container_sync bulk ratelimit swift3 s3token authtoken keystoneauth > >>>>> container-quotas account-quotas slo dlo versioned_writes > proxy-logging > >>>>> proxy-server > >>>>> > >>>>> [filter:swift3] > >>>>> use = egg:swift3#swift3 > >>>>> > >>>>> [filter:s3token] > >>>>> use = egg:swift3#s3token > >>>>> auth_uri = http://keystone:35357/ > >>>>> reseller_prefix = AUTH_ > >>>>> > >>>>> Then I tried to restart the proxy: > >>>>> systemctl restart openstack-swift-proxy > >>>>> > >>>>> And it failes. > >>>>> > >>>>> > >>>> > >>>> Yeah - for Mitaka (or thereabouts) onwards that s3token config needs > to > >>>> change: > >>>> > >>>> [filter:s3token] > >>>> paste.filter_factory = keystoneclient.middleware.s3_ > >> token:filter_factory > >>>> auth_uri = http://keystone:35357/ > >>>> reseller_prefix = AUTH_ > >>>> > >>>> Alternatively the older port/protocol/host variables still work > instead > >> of > >>>> auth_uri: > >>>> > >>>> [filter:s3token] > >>>> paste.filter_factory = keystoneclient.middleware.s3_ > >> token:filt
[Openstack] [OpenStack] [Swift] How to use s3curl or s3cmd with Swift3?
Hello,
I try to use s3curl to test swift3 but I miss something.
Here is the command which I use:
./s3curl.pl - 'swift' -key '3d8b70ad194a4cf6857e59ec638d' -get - -s -v
http://proxy:8080/v1/AUTH_3f842db875cc48b99d7ff246c27a2e6a
It shows:
Can't locate Digest/HMAC_SHA1.pm in @INC (@INC contains:
/usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl
/usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 .) at ./
s3curl.pl line 20.
BEGIN failed--compilation aborted at ./s3curl.pl line 20.
I am not sure that I write the command correctly.
*swift* - is the account
*3d8b70ad194a4cf6857e59ec638d* - is the X-Subject-Token I have got in
the header with this command:
curl -i \
-H "Content-Type: application/json" \
-d '
{ "auth": {
"identity": {
"methods": ["password"],
"password": {
"user": {
"name": "swift",
"domain": { "name": "default" },
"password": "swift"
}
}
},
"scope": {
"project": {
"name": "service",
"domain": { "name": "default" }
}
}
}
}' \
http://keystone:5000/v3/auth/tokens ; echo
*http://proxy:8080/v1/AUTH_3f842db875cc48b99d7ff246c27a2e6a
* - is the
public endpoint
Am I doing correctly?
Why doesn't it work?
Sincerely,
Alexandr
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Re: [Openstack] [OpenStack] [Swift] How to use s3curl or s3cmd with Swift3?
The same error..(
Can't locate Digest/HMAC_SHA1.pm in @INC (@INC contains:
/usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl
/usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 .) at ./
s3curl.pl line 20.
BEGIN failed--compilation aborted at ./s3curl.pl line 20.
On Thu, Oct 13, 2016 at 12:01 PM, Kota TSUYUZAKI <
tsuyuzaki.k...@lab.ntt.co.jp> wrote:
> Hello again,
>
> I'm not familier with s3curl in detail though,
>
> > ./s3curl.pl - 'swift' -key '3d8b70ad194a4cf6857e59ec638d' -get - -s
> -v
> > http://proxy:8080/v1/AUTH_3f842db875cc48b99d7ff246c27a2e6a
> >
>
> could you try:
>
> ./s3curl.pl - 'swift' -key 'swift' -get - -s -v http://proxy:8080/
>
> IIRC, secret_key is not a swift token but a password you set in your
> keystone and swift3 doesn't require Swift url so you can request to the
> host directly.
>
> Regards,
> Kota
>
>
> (2016/10/13 17:29), Alexandr Porunov wrote:
> > Hello,
> >
> > I try to use s3curl to test swift3 but I miss something.
> >
> > Here is the command which I use:
> > ./s3curl.pl - 'swift' -key '3d8b70ad194a4cf6857e59ec638d' -get - -s
> -v
> > http://proxy:8080/v1/AUTH_3f842db875cc48b99d7ff246c27a2e6a
> >
> > It shows:
> > Can't locate Digest/HMAC_SHA1.pm in @INC (@INC contains:
> > /usr/local/lib64/perl5 /usr/local/share/perl5
> /usr/lib64/perl5/vendor_perl
> > /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 .) at ./
> > s3curl.pl line 20.
> > BEGIN failed--compilation aborted at ./s3curl.pl line 20.
> >
> > I am not sure that I write the command correctly.
> >
> > *swift* - is the account
> > *3d8b70ad194a4cf6857e59ec638d* - is the X-Subject-Token I have got
> in
> > the header with this command:
> >
> > curl -i \
> > -H "Content-Type: application/json" \
> > -d '
> > { "auth": {
> > "identity": {
> > "methods": ["password"],
> > "password": {
> > "user": {
> > "name": "swift",
> > "domain": { "name": "default" },
> > "password": "swift"
> > }
> > }
> > },
> > "scope": {
> > "project": {
> > "name": "service",
> > "domain": { "name": "default" }
> > }
> > }
> > }
> > }' \
> > http://keystone:5000/v3/auth/tokens ; echo
> >
> > *http://proxy:8080/v1/AUTH_3f842db875cc48b99d7ff246c27a2e6a
> > <http://proxy:8080/v1/AUTH_3f842db875cc48b99d7ff246c27a2e6a>* - is the
> > public endpoint
> >
> > Am I doing correctly?
> >
> > Why doesn't it work?
> >
> > Sincerely,
> > Alexandr
> >
> >
> >
> > ___
> > Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/
> openstack
> > Post to : openstack@lists.openstack.org
> > Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/
> openstack
> >
>
>
> --
> --
> Kota Tsuyuzaki(露﨑 浩太)
> NTT Software Innovation Center
> Cloud Solution Project
> Phone 0422-59-2837
> Fax0422-59-2965
> ---
>
>
>
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Re: [Openstack] [OpenStack] [Swift] How to use s3curl or s3cmd with Swift3?
To get token I use the next command:
curl -i \
-H "Content-Type: application/json" \
-d '
{ "auth": {
"identity": {
"methods": ["password"],
"password": {
"user": {
"name": "swift",
"domain": { "name": "default" },
"password": "swift"
}
}
},
"scope": {
"project": {
"name": "service",
"domain": { "name": "default" }
}
}
}
}' \
http://keystone:5000/v3/auth/tokens ; echo
Here is the latest output:
HTTP/1.1 201 Created
Date: Thu, 13 Oct 2016 12:40:53 GMT
Server: Apache/2.4.6 (CentOS) mod_wsgi/3.4 Python/2.7.5
X-Subject-Token: 21d9ea93e9324fedb8853c94e6698d79
Vary: X-Auth-Token
x-openstack-request-id: req-eaf0c587-fcaa-4294-bee6-507385eea1e8
Content-Length: 1715
Content-Type: application/json
{"token": {"methods": ["password"], "roles": [{"id":
"908072b0551448a29793277b4ad71688", "name": "admin"}], "expires_at":
"2016-10-13T13:40:54.107598Z", "project": {"domain": {"id":
"839eff379a694db0b4c69716b7ee7c32", "name": "default"}, "id":
"3f842db875cc48b99d7ff246c27a2e6a", "name": "service"}, "catalog":
[{"endpoints": [{"region_id": "RegionOne", "url": "
http://proxy:8080/v1/AUTH_3f842db875cc48b99d7ff246c27a2e6a";, "region":
"RegionOne", "interface": "internal", "id":
"19fe381617d84ca7a7953e1dfd89f181"}, {"region_id": "RegionOne", "url": "
http://proxy:8080/v1";, "region": "RegionOne", "interface": "admin", "id":
"6ff73960ff4249e7a3f5d9fa7cbe90e0"}, {"region_id": "RegionOne", "url": "
http://proxy:8080/v1/AUTH_3f842db875cc48b99d7ff246c27a2e6a";, "region":
"RegionOne", "interface": "public", "id":
"ec5e71ac89fc4a66a026ad96d8c0df49"}], "type": "object-store", "id":
"83436928701349379c7e643954acbb8d", "name": "swift"}, {"endpoints":
[{"region_id": "USA_east_1", "url": "http://keystone:5000/v3";, "region":
"USA_east_1", "interface": "public", "id":
"2b552dffdf564d7dab281d6676a5e86e"}, {"region_id": "USA_east_1", "url": "
http://keystone:5000/v3";, "region": "USA_east_1", "interface": "internal",
"id": "8a73678a836a47bbb16a5df7e41576df"}, {"region_id": "USA_east_1",
"url": "http://keystone:35357/v3";, "region": "USA_east_1", "interface":
"admin", "id": "964dbc7a4920416588bc754bfed583b7"}], "type": "identity",
"id": "77366d6be47a46ff94d56e1436fab2aa", "name": "keystone"}], "user":
{"domain": {"id": "839eff379a694db0b4c69716b7ee7c32", "name": "default"},
"id": "1fb1511494744101b9cfab7b68d9098e", "name": "swift"}, "audit_ids":
["lGLvUwAXQo2f2L6WNxvl-Q"], "issued_at": "2016-10-13T12:40:54.107633Z"}}
On Thu, Oct 13, 2016 at 12:40 PM, Alexandr Porunov <
alexandr.poru...@gmail.com> wrote:
> Thank you. You were right.
>
> Now s3curl works but I don't know how to use it.. I try to check if swift3
> is working with the next command:
> ./s3curl.pl --id 'swift' -key '384c48d6a26244aebcfa26cff4567bc0' -get
> -s -v http://proxy:8080/v1/AUTH_3f842db875cc48b99d7ff246c27a2e6a
>
> and
>
> ./s3curl.pl --id 'swift' -key 'swift' -get -s -v
> http://proxy:8080/v1/AUTH_3f842db875cc48b99d7ff246c27a2e6a
>
> in both cases I get the same error:
>
> InvalidBucketNameThe specified bucket is not
> valid.tx75447f12c8434120b20cd-0057ff7ffe<
> BucketName>v1
>
> Do you know how to resolve it?
>
> On Thu, Oct 13, 2016 at 12:10 PM, Turbo Fredriksson
> wrote:
>
>> On Oct 13, 2016, at 9:29 AM, Alexandr Porunov wrote:
>>
>> > Can't locate Digest/HMAC_SHA1.pm in @INC (@INC contains:
>>
>> You need to install the "libdigest-hmac-perl" package..
>> --
>> Try not. Do. Or do not. There is no try!
>> - Yoda
>>
>>
>> ___
>> Mailing list: http://lists.openstack.org/cgi
>> -bin/mailman/listinfo/openstack
>> Post to : openstack@lists.openstack.org
>> Unsubscribe : http://lists.openstack.org/cgi
>> -bin/mailman/listinfo/openstack
>>
>
>
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Re: [Openstack] [OpenStack] [Swift] How to use s3curl or s3cmd with Swift3?
Thank you. You were right. Now s3curl works but I don't know how to use it.. I try to check if swift3 is working with the next command: ./s3curl.pl --id 'swift' -key '384c48d6a26244aebcfa26cff4567bc0' -get -s -v http://proxy:8080/v1/AUTH_3f842db875cc48b99d7ff246c27a2e6a and ./s3curl.pl --id 'swift' -key 'swift' -get -s -v http://proxy:8080/v1/AUTH_3f842db875cc48b99d7ff246c27a2e6a in both cases I get the same error: InvalidBucketNameThe specified bucket is not valid.tx75447f12c8434120b20cd-0057ff7ffev1 Do you know how to resolve it? On Thu, Oct 13, 2016 at 12:10 PM, Turbo Fredriksson wrote: > On Oct 13, 2016, at 9:29 AM, Alexandr Porunov wrote: > > > Can't locate Digest/HMAC_SHA1.pm in @INC (@INC contains: > > You need to install the "libdigest-hmac-perl" package.. > -- > Try not. Do. Or do not. There is no try! > - Yoda > > > ___ > Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/ > openstack > Post to : openstack@lists.openstack.org > Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/ > openstack > ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Re: [Openstack] [OpenStack] [Swift] How to use s3curl or s3cmd with Swift3?
I did it.
First I need to create ec2 credentials:
openstack ec2 credentials create
Then I had to change "my @endpoints" parameter in s3curl.pl as following:
my @endpoints = ( 'proxy');
And then just use my credentials with s3curl.pl
For example. For containers list:
./s3curl.pl --id access_key --key secret_key http://proxy:8080
For objects list in a container:
./s3curl.pl --id access_key --key secret_key http://proxy:8080/my_container
For stream an object:
./s3curl.pl --id access_key --key secret_key
http://proxy:8080/my_container/my_object
Sincerely,
Alexandr
On Thu, Oct 13, 2016 at 12:42 PM, Alexandr Porunov <
alexandr.poru...@gmail.com> wrote:
> To get token I use the next command:
>
> curl -i \
> -H "Content-Type: application/json" \
> -d '
> { "auth": {
> "identity": {
> "methods": ["password"],
> "password": {
> "user": {
> "name": "swift",
> "domain": { "name": "default" },
> "password": "swift"
> }
> }
> },
> "scope": {
> "project": {
> "name": "service",
> "domain": { "name": "default" }
> }
> }
> }
> }' \
> http://keystone:5000/v3/auth/tokens ; echo
>
> Here is the latest output:
>
> HTTP/1.1 201 Created
> Date: Thu, 13 Oct 2016 12:40:53 GMT
> Server: Apache/2.4.6 (CentOS) mod_wsgi/3.4 Python/2.7.5
> X-Subject-Token: 21d9ea93e9324fedb8853c94e6698d79
> Vary: X-Auth-Token
> x-openstack-request-id: req-eaf0c587-fcaa-4294-bee6-507385eea1e8
> Content-Length: 1715
> Content-Type: application/json
>
> {"token": {"methods": ["password"], "roles": [{"id": "
> 908072b0551448a29793277b4ad71688", "name": "admin"}], "expires_at":
> "2016-10-13T13:40:54.107598Z", "project": {"domain": {"id": "
> 839eff379a694db0b4c69716b7ee7c32", "name": "default"}, "id": "
> 3f842db875cc48b99d7ff246c27a2e6a", "name": "service"}, "catalog":
> [{"endpoints": [{"region_id": "RegionOne", "url": "
> http://proxy:8080/v1/AUTH_3f842db875cc48b99d7ff246c27a2e6a";, "region":
> "RegionOne", "interface": "internal", "id": "
> 19fe381617d84ca7a7953e1dfd89f181"}, {"region_id": "RegionOne", "url": "
> http://proxy:8080/v1";, "region": "RegionOne", "interface": "admin", "id":
> "6ff73960ff4249e7a3f5d9fa7cbe90e0"}, {"region_id": "RegionOne", "url": "
> http://proxy:8080/v1/AUTH_3f842db875cc48b99d7ff246c27a2e6a";, "region":
> "RegionOne", "interface": "public", "id":
> "ec5e71ac89fc4a66a026ad96d8c0df49"}],
> "type": "object-store", "id": "83436928701349379c7e643954acbb8d", "name":
> "swift"}, {"endpoints": [{"region_id": "USA_east_1", "url": "
> http://keystone:5000/v3";, "region": "USA_east_1", "interface": "public",
> "id": "2b552dffdf564d7dab281d6676a5e86e"}, {"region_id": "USA_east_1",
> "url": "http://keystone:5000/v3";, "region": "USA_east_1", "interface":
> "internal", "id": "8a73678a836a47bbb16a5df7e41576df"}, {"region_id":
> "USA_east_1", "url": "http://keystone:35357/v3";, "region": "USA_east_1",
> "interface": "admin", "id": "964dbc7a4920416588bc754bfed583b7"}], "type":
> "identity", "id": "77366d6be47a46ff94d56e1436fab2aa", "name":
> "keystone"}], "user": {"domain": {"id": "839eff379a694db0b4c69716b7ee7c32",
> "name": "default"}, "id": "1fb1511494744101b9cfab7b68d9098e", "name":
> "swift"}, "audit_ids": ["lGLvUwAXQo2f2L6WNxvl-Q"], "issued_at":
> "2016-10-13T12:40:54.107633Z"}}
>
>
> On Thu, Oct 13, 2016 at 12:40 PM, Alexandr Porunov <
> alexandr.poru...@gmail.com> wrote:
>
>> Thank you. You were right.
>>
>> Now s3curl works but I don't know how to use it.. I try to check if
>> swift3 is working with the next command:
>> ./s3curl.pl --id 'swift' -key '384c48d6a26244aebcfa26cff4567bc0' -get
>> -s -v http://proxy:8080/v1/AUTH_3f842db875cc48b99d7ff246c27a2e6a
>>
>> and
>>
>> ./s3curl.pl --id 'swift' -key 'swift' -get -s -v
>> http://proxy:8080/v1/AUTH_3f842db875cc48b99d7ff246c27a2e6a
>>
>> in both cases I get the same error:
>>
>> InvalidBucketNameThe specified bucket is
>> not valid.tx75447f12c8434120b20cd-0057ff7ff
>> ev1
>>
>> Do you know how to resolve it?
>>
>> On Thu, Oct 13, 2016 at 12:10 PM, Turbo Fredriksson
>> wrote:
>>
>>> On Oct 13, 2016, at 9:29 AM, Alexandr Porunov wrote:
>>>
>>> > Can't locate Digest/HMAC_SHA1.pm in @INC (@INC contains:
>>>
>>> You need to install the "libdigest-hmac-perl" package..
>>> --
>>> Try not. Do. Or do not. There is no try!
>>> - Yoda
>>>
>>>
>>> ___
>>> Mailing list: http://lists.openstack.org/cgi
>>> -bin/mailman/listinfo/openstack
>>> Post to : openstack@lists.openstack.org
>>> Unsubscribe : http://lists.openstack.org/cgi
>>> -bin/mailman/listinfo/openstack
>>>
>>
>>
>
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
[Openstack] [OpenStack] [Swift] Does Swift3 support v4 signature for S3?
Hello, On gitgub they tell us: Support AWS Signature Version 2 (Version 4 is not ready) But in changelog they tell that they support v4: Amazon S3 Signature V4 support I tried to use aws java s3 client and mino java client to send requests to swift but without success. Both show either 403 or 400 error.. When I use command line tool (s3curl.pl) it works. Does Swift3 support v4? What can be wrong? Sincerely, Alexandr ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
[Openstack] [OpenStack] [Swift] How to create alias for a container?
Hello, Is it possible to create aliases for a container? I have a container "ContainerA" I need to work with (put/remove/copy/get objects) but using the name "ContainerB". Is it possible to create such aliases? If yes then how? Sincerely, Alexandr ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Re: [Openstack] [OpenStack] [Swift] Does Swift3 support v4 signature for S3?
Hello Kota, Thank you very much! Sincerely, Alexandr On Mon, Oct 17, 2016 at 9:31 AM, Kota TSUYUZAKI < tsuyuzaki.k...@lab.ntt.co.jp> wrote: > Hello Alexandr, > > Signature V4 has been supported since > > https://github.com/openstack/swift3/commit/a1cc181bd8ef891d3e3b8ed71db9b4 > 1c179ced0e > > It has been restricted with only keystone authentication though. > The docs in README.rst is wrong. I pushed the fix for the docs, > https://review.openstack.org/#/c/387126/ > > Thanks, > Kota > > (2016/10/15 6:00), Alexandr Porunov wrote: > > Hello, > > > > On gitgub they tell us: > > Support AWS Signature Version 2 (Version 4 is not ready) > > > > But in changelog they tell that they support v4: > > Amazon S3 Signature V4 support > > > > I tried to use aws java s3 client and mino java client to send requests > to > > swift but without success. Both show either 403 or 400 error.. > > > > When I use command line tool (s3curl.pl) it works. > > > > Does Swift3 support v4? What can be wrong? > > > > Sincerely, > > Alexandr > > > > > > > > ___ > > Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/ > openstack > > Post to : openstack@lists.openstack.org > > Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/ > openstack > > > > > -- > -- > Kota Tsuyuzaki(露﨑 浩太) > NTT Software Innovation Center > Cloud Solution Project > Phone 0422-59-2837 > Fax0422-59-2965 > --- > > > ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
[Openstack] [OpenStack] [Keystone] Performance difference between ferret and uuid tokens
Hello, I read a blog about performance comparison between fernet and uuid tokens. They said that fernet tokens is 30% faster for creation but 400% slower for validation. Is it true? I want to use Keystone for Swift. I will have many requests with the same tokens so I need faster validation than faster creation. I would use uuid tokens but fernet tokens give us very good pros (we don't need to use a database). So, I decided to cache all fernet tokens on the Swift Proxy side for 30 minutes. Will the performance be the same for checking tokens in a cache or fernet tokens will still be 400% slower? Sincerely, Alexandr ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Re: [Openstack] [OpenStack] [Keystone] Performance difference between ferret and uuid tokens
Thank you very much for the explanation and for the articles. I will use Newton release of Keystone and Swift. Thanks a lot Best regards, Alexandr On Sat, Dec 10, 2016 at 11:13 PM, Steve Martinelli wrote: > Re-sending Matt's reply since he is not subscribed to this mailing list > and it bounced back. > > --- > > Matt Fischer wrote: > > Alexandr, > > I would not use Fernet without caching, but that said I strongly recommend > against UUID tokens for any reason. Make sure you setup caching on the > swift side & the keystone side. You could consider than an L1 and L2 cache > where from Swift's POV it's authtoken cache is L1 and keystone's cache is > L2. If you do that I believe the performance will be acceptable. > > The slowness comes when Keystone has to attempt to decrypt the uncached > tokens. > > Also if you're looking to squeeze out the last bit of performance from > your keystone, using the deprecated (and not tested in the gate) > Python-MySQL driver instead of pymsql is about 6% faster. That carries > risks as it's untested and becoming less widely used. We switched to > pymysql as has openstack-ansible and several other deployments. > > On Sat, Dec 10, 2016 at 3:23 PM, Steve Martinelli > wrote: > >> On Sat, Dec 10, 2016 at 10:59 AM, Alexandr Porunov < >> alexandr.poru...@gmail.com> wrote: >> >>> Hello, >>> >>> I read a blog about performance comparison between fernet and uuid >>> tokens. They said that fernet tokens is 30% faster for creation but 400% >>> slower for validation. Is it true? >>> >>> >> I assume you are reading Dolph's blog post [1], that data is based off of >> the kilo branch, we've made some improvements to performance since then, he >> should probably do a follow up post for how the same performance tests run >> on Newton ;) >> >> Token validation can be improved using caching, which we worked on in >> Liberty, Mitaka and Newton (the latest Mitaka release (9.2.0) includes a >> critical performance fix, it was not backported to Liberty). Revocation >> events are still an issue for performance, but we've been addressing that >> in Ocata. I don't think we'll be able to backport the fixes for poor >> revocation performance though, unfortunately it goes against the backport >> policy. >> >> >> FWIW, Matt Fischer has 4 blog posts about using fernet tokens in >> production [2], they are very detailed and performance oriented. I really >> recommend reading them, it's great stuff. >> >> >> [1] http://dolphm.com/benchmarking-openstack-keystone-token-formats/ >> [2] https://www.mattfischer.com/blog/?tag=fernet >> >> >> stevemar >> >> >> >>> I want to use Keystone for Swift. I will have many requests with the >>> same tokens so I need faster validation than faster creation. I would use >>> uuid tokens but fernet tokens give us very good pros (we don't need to use >>> a database). So, I decided to cache all fernet tokens on the Swift Proxy >>> side for 30 minutes. Will the performance be the same for checking tokens >>> in a cache or fernet tokens will still be 400% slower? >>> >>> Sincerely, >>> Alexandr >>> >> >> > ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
[Openstack] [Swift] How to use Swift3 module with Nginx proxy for version 4 signature?
Hello, I want to use Nginx as a proxy for OpenStack Swift with Swift3 module (S3 api). I read that it supports verison 4 signature but I am not sure how to use it. I am using a client which work only with version 4 signature and sends requests like this: ./ Does somebody know how to configure Nginx to proxy requests from: https://./ to local node (like 192.168.0.45)? Sincerely, Alexandr ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
[Openstack] [OpenStack] [Swift] Bug: Swift doesn't save last "end of file" characters
Hello, I am using Openstack Newton. I found a small bug in the Swift. It doesn't copy last "end of line" characters. Here is my test (I copy my.cnf file): # cat -A my.cnf [client-server]$ !includedir /etc/my.cnf.d$ $ # openstack object create mybucket my.cnf # openstack object save mybucket my.cnf # cat -A my.cnf [client-server]$ !includedir /etc/my.cnf.d As we see last two characters haven't been copied (i.e. $). In my case it is a big issue. Will be happy if you suggest me how to copy last invisible characters. Sincerely, Alexandr ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
[Openstack] [OpenStack] [Keystone] Is it possible to cache ec2 credentials?
Hello, I want to use ec2 credentials. But I don't know about it's route of validating. Is it much slower than tokens validation? I don't know exactly what is the route of ec2 validation but here is what I think: 1) Service node accept a request with ec2 credentials. 2) Service asks Keystone to check credentials. 3) Keystone search credentials in MySQL. 4) If credentials are the same as in MySQL then Keystone will return OK to the service. If the route is the same as I described above then the performance will be really bad because we will go and check credentials in MySQL for each request. Is it possible to cache ec2 credentials on the Keystone side or on the Service side to don't ask MySQL for each request? Best regards, Alexandr ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
[Openstack] [Keystone] What is credential-keys? Do we need to rotate them?
Hello, I found not only fernet_keys but also credential-keys in /etc/keystone/ What is it? Do we need to rotate them? Sincerely, Alexandr ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Re: [Openstack] [Keystone] What is credential-keys? Do we need to rotate them?
Thank you very much for the link! Best regards, Alexandr On Fri, Jan 20, 2017 at 11:01 PM, Boris Bobrov wrote: > Hi, > > Yes, you need to setup them and rotate if necessary: > http://docs.openstack.org/developer/keystone/configuration.html?highlight= > credential_migrate#credential-encryption > > On 01/20/2017 07:49 PM, Alexandr Porunov wrote: > > Hello, > > > > I found not only fernet_keys but also credential-keys in /etc/keystone/ > > > > What is it? Do we need to rotate them? > > > > Sincerely, > > Alexandr > > > > > > ___ > > Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/ > openstack > > Post to : openstack@lists.openstack.org > > Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/ > openstack > > > > ___ > Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/ > openstack > Post to : openstack@lists.openstack.org > Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/ > openstack > ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
[Openstack] [Swift] How to add a new node without ring copy?
Hello, I read different articles about how to add a new node and they all says this: 1) Add a node to ring builder 2) Rebalance ring 3) Copy ring to each node. Is it possible to skip "copy" step by doing first and second step on each node? Will then all nodes have the same ring? I.e. I mean to do like this: 1) Add a node to the ring builder (Execute this step on each node) 2) Rebalance the ring (Execute this step on each node) Will the ring be the same or we still have to copy the ring from one node to all others? Best regards, Alexandr ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Re: [Openstack] [Swift] How to add a new node without ring copy?
Thanks for your suggestions! Mark was right about automated provisioning tools. I am using Saltstack for it. The problem I try to solve is the automatic deployment. Adding and removing new nodes is the problematic thing because we can't just add or remove a full node at once (it can cause storm of data movement). We need to add / remove weight around 50 GB each 1 hour. So, 2 TB disk will be added or removed in 40 hours. It is a problematic to run a crond job on one machine because this job can fail (if the machine die or something like this). Right now the workaround which I have is the next: Every hour salt masters run a job to provision new nodes. The job consists of several steps but the main idea is the next: 1) Node tries to get a global lock to change rings (in my case I check the time of last lock in MariaDB Galera cluster. If the last lock was more then 1hour ago then I can get a lock if no then I can not get a lock). 2) If the lock was obtained then I download the latest version of rings from OpenStack Swift and get an information about all nodes from rings. If any node has a different wight from the wight which I have set in Saltstack (Pillar) then I add / remove 50 weight. If the node has 0 weight then I remove it from the cluster. If I have point new nodes in Saltstack which hasn't been added to the cluster yet then I add them with weigth 50. 3) Rebalance the ring if any changes have occured. Update the ring in OpenStack Swift. Also every hour the new ring will be copied to any node if it has a different ring. I don't know whether it is a normal workaround or a strange one but I haven't found any other solutions. Please share if you have any ideas. Best regards, Alexandr On Mon, Jan 23, 2017 at 4:37 AM, Mark Kirkwood < mark.kirkw...@catalyst.net.nz> wrote: > Yes, one set of identical ring files is definitely the best option! Also > the recon md5 ring check will likely complain forever if these alternative > methods (for avoiding copying) are tried. > > > More interesting is why people want to avoid the copy? I'm guessing it is > because this type of thing is hard to do in some of the automated > provisioning tools (ahem..looking at you puppet): > > - add disk on storage node > > - tell ring generation/master node(s) about it > > - make new rings on the node(s) above if new disks detected > > - copy new rings from node(s) in step 2 (for every node in cluster) > > > Now the puppet-swift module does tell you how to do this (via exported > resources), but it is not that simple (and in fact the examples in the > module have mistakes that result in non identical rings proxy nodes...). > Other complexities that make this tricky are multiple regions/geo > replication - if each has its own puppet server...you can't (easily) use > exported resources at all. > > Keeping the rings in swift is quite a cool idea - but can't be used when > bootstrapping the cluster initially (probably not that difficult to work > around tho). > > > We are currently > > - keeping boostrap ring in puppet > - add new disks/nodes manually (with helper scripts) > - keep 'master' rings on all proxies > - run ring rsync servers on all proxies > - storage nodes get rings from proxies (puppet or manually) > > We run the recon md5 ring check as part of standard monitoring to pick up > any mistakes we might do add devices. > > cheers > > Mark > > On 23/01/17 03:16, Vladimir Maliaev wrote: > >> Anyway you will copy ring files to new node. Which node you are going to >> copy from? >> I think that to have only one set of ring files is helpful in >> troubleshooting if something goes wrong. >> >> On Jan 22, 2017 6:08 AM, "Alexandr Porunov" > <mailto:alexandr.poru...@gmail.com>> wrote: >> >> Hello, >> >> I read different articles about how to add a new node and they all >> says this: >> 1) Add a node to ring builder >> 2) Rebalance ring >> 3) Copy ring to each node. >> >> Is it possible to skip "copy" step by doing first and second step >> on each node? Will then all nodes have the same ring? >> I.e. I mean to do like this: >> 1) Add a node to the ring builder (Execute this step on each node) >> 2) Rebalance the ring (Execute this step on each node) >> >> Will the ring be the same or we still have to copy the ring from >> one node to all others? >> >> Best regards, >> Alexandr >> >> ___ >> Mailing list: >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack >> <http://lists.openstack.org/cgi-bin/mailm
[Openstack] [Swift] How to get a list of devices from the ring in Java / Bash?
Hello, I need to get a list / array of devices in Java or Bash. Is it possible to get? Are there any API or something like that? I know we can obtain a list with the next command: swift-ring-builder object.builder But it return a string with the list of devices which are quite hard to parse. In the documentation there is an example how to get a list of devices with python: devices = list(self._iter_devs()) But it doesn't show which library to use to get a list of devices and how to use that library. Sincerely, Alexandr ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
[Openstack] [Swift] Bug: swift-ring-builder set_weight doesn't work
Hello, I can't change a weight of the device. Here is a list of my devices: Devices:id region zone ip address port replication ip replication port name weight partitions balance meta 0 1 1 host1 6000 host1 6000d1 1000.00 10240.00 bla bla1 bla2 1 1 1 host1 6000 host1 6000d2 1000.00 10240.00 bla bla1 bla2 2 1 1 host1 6000 host1 6000d3 1000.00 10240.00 bla bla1 bla2 Here is what I tried to change a weight: swift-ring-builder object.builder set_weight --region 1 --zone 1 --ip host1 --port 6000 --replication-ip host1 --replication-port 6000 --device d1 --meta "bla bla1 bla2" --weight 1000 swift-ring-builder object.builder set_weight r1z1-host1:6000Rhost1:6000 d1_"bla bla1 bla2" 2000 In both cases it doesn't work and shows the next information: swift-ring-builder set_weight [ set_weight --region --zone --ip --port --replication-ip --replication-port --device --meta --weight [--yes] Where , and are replication ip, hostname and port. Any of the options are optional in both cases. Resets the devices' weights. No partitions will be reassigned to or from the device until after running 'rebalance'. This is so you can make multiple device changes and rebalance them all just once. Option --yes assume a yes response to all questions. Please, correct me if it is my fault. Best regards, Alexandr ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Re: [Openstack] [Swift] Bug: swift-ring-builder set_weight doesn't work
Hello Matt, Thank you very much for your help! It's awesome. Now everything works! Best regards, Alexandr On Mon, Jan 30, 2017 at 12:48 AM, Matthew Oliver wrote: > Hey Alexandr, > > Thanks for the email. I can help you in this instance, and I'll push up a > patch to correct the usage documentation for the long version of the > set_weight command. > > > On Sun, Jan 29, 2017 at 8:33 AM, Alexandr Porunov < > alexandr.poru...@gmail.com> wrote: > >> Hello, >> >> I can't change a weight of the device. >> Here is a list of my devices: >> >> Devices:id region zone ip address port replication ip >> replication port name weight partitions balance meta >> 0 1 1 host1 6000 host1 >>6000d1 1000.00 10240.00 bla bla1 bla2 >> 1 1 1 host1 6000 host1 >>6000d2 1000.00 10240.00 bla bla1 bla2 >> 2 1 1 host1 6000 host1 >>6000d3 1000.00 10240.00 bla bla1 bla2 >> >> Here is what I tried to change a weight: >> swift-ring-builder object.builder set_weight --region 1 --zone 1 --ip >> host1 --port 6000 --replication-ip host1 --replication-port 6000 --device >> d1 --meta "bla bla1 bla2" --weight 1000 >> > > There is a problem with the command as you've sent it. But I agree it > isn't obvious from the usage error. Using the long form (--region, > --device, etc) is just defining search criteria for what you want to > change. Not specifying what you want to change. > > So in your case above your looking for a device that already has a weight > of 1000, and your not specifying what the change it to. To fix your > command, you'd need to: > > swift-ring-builder object.builder set_weight --region 1 --zone 1 --ip > host1 --port 6000 --replication-ip host1 --replication-port 6000 --device > d1 --meta "bla bla1 bla2" --weight 1024 1000 > > NOTE: I am sending in the 1000 as the last option. Obviously as these are > search terms you don't actually have to send them all in, only the ones > that identify the devices or device you want to change. > > >> swift-ring-builder object.builder set_weight r1z1-host1:6000Rhost1:6000 >> d1_"bla bla1 bla2" 2000 >> > > The short version above is failing, as you forgot to add the '/' seperator > between the port and the device, given your example it should have been: > > swift-ring-builder object.builder set_weight > r1z1-host1:6000Rhost1:6000/d1_"bla bla1 bla2" 2000 > > NOTE: the '/' before the d1. > > What your putting in there is just a search for devices to change. You > don't have to put the full string in. I guess if you have a tool that will > use the full definition then fine. I find when wanting to update just 1 > host it's easier to do so via the device id: > > swift-ring-builder set_weight d1 2000 > > In both cases it doesn't work and shows the next information: >> >> swift-ring-builder set_weight >> >> [ > >> [--yes] >> >> >> >> or >> >> >> >> swift-ring-builder set_weight >> >> --region --zone --ip --port >> >> --replication-ip --replication-port >> >> --device --meta --weight >> >> [--yes] >> >> > > This usage isn't been very useful here, as it's missing the actual adding > the weight. I'll raise a bug and patch to address this. > > >> >> >> Where , and are replication ip, hostname >> >> and port. >> >> Any of the options are optional in both cases. >> >> >> >> Resets the devices' weights. No partitions will be reassigned to or >> from >> the device until after running 'rebalance'. This is so you can make >> >> multiple device changes and rebalance them all just once. >> >> >> >> Option --yes assume a yes response to all questions. >> >> >> Please, correct me if it is my fault. >> >> Best regards, >> Alexandr >> >> ___ >> Mailing list: http://lists.openstack.org/cgi >> -bin/mailman/listinfo/openstack >> Post to : openstack@lists.openstack.org >> Unsubscribe : http://lists.openstack.org/cgi >> -bin/mailman/listinfo/openstack >> >> > regards, > Matt > > ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Re: [Openstack] [Swift] How to get a list of devices from the ring in Java / Bash?
Hello Matt,
Wow! It's a perfect solution! You've changed my day :)
Thank you very much again!
Best regards,
Alexandr
On Tue, Jan 31, 2017 at 12:32 AM, Matthew Oliver wrote:
> Hey Alexandr,
>
> If you wanted to do it using python, and you didn't have swift installed
> you can use pickle and unpickle the builder. something like:
>
> import pickle
> builder = pickle.load(open('')
>
> Then you'd have a python dict structure, where I think you can
> 'builder['devs']'
>
> or with swift installed you can use the RingBuilder class:
>
> from swift.common.ring.builder import RingBuilder
> builder = RingBuilder.load('')
> builder.devs
>
> Where builder.devs will be a list of dictionary objects. You could then
> json-ify and output for java to read or put it into an easier to pass list.
>
> There could be a much simpler way, but I'm a Python dev, so this is what
> I'd do ;)
>
> Regards,
> Matt
>
> On Sat, Jan 28, 2017 at 6:21 AM, Alexandr Porunov <
> alexandr.poru...@gmail.com> wrote:
>
>> Hello,
>>
>> I need to get a list / array of devices in Java or Bash. Is it possible
>> to get? Are there any API or something like that?
>>
>> I know we can obtain a list with the next command:
>> swift-ring-builder object.builder
>>
>> But it return a string with the list of devices which are quite hard to
>> parse. In the documentation there is an example how to get a list of
>> devices with python:
>> devices = list(self._iter_devs())
>>
>> But it doesn't show which library to use to get a list of devices and how
>> to use that library.
>>
>> Sincerely,
>> Alexandr
>>
>> ___
>> Mailing list: http://lists.openstack.org/cgi
>> -bin/mailman/listinfo/openstack
>> Post to : openstack@lists.openstack.org
>> Unsubscribe : http://lists.openstack.org/cgi
>> -bin/mailman/listinfo/openstack
>>
>>
>
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
[Openstack] [OpenStack] [Swift] How to set maximum size of the log file?
Hello, I found out that my log file is almost 3GB. How to set a size limit? Best regards, Alexandr ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
