Hello,
I'm trying to create / delete files on google cloudstorage with unicode
characters in the name. GCS seems to support this alright, however the
client APIs seem to have problems.
- Environment: Google Appengine Standard with Python 2.7. Tested locally
with dev_server 1.9.50. The library used (both locally and in production)
is the latest GoogleAppEngineCloudStorageClient [1] (1.9.22.1)
- Some code snippets:
fn = u'/%s/á' % GS_BUCKET
with gcs.open(fn, 'w') as f_out: f_out.write('old')
This gets me the traceback at [2]
- Ok, how about manually encoding it?
fn = u'/%s/á' % GS_BUCKET
with gcs.open(fn.encode('utf-8'), 'w') as f_out: f_out.write('old')
This just gets me several retries until the request is aborted [3]
- There is the option of encoding it, however I don't believe this is
correct (this is just double quoting the filename):
fn = u'/%s/á' % GS_BUCKET
with gcs.open(urllib.quote(fn.encode('utf-8')), 'w') as f_out:
f_out.write('old')
- I believe gcs.delete is also affected since I have the following
traceback in production [4]
How are unicode filenames supposed to be used with the cloudstorage
library? Also, is the cloudstorage library supposed to be used at all?
Looking around I found the google-cloud-storage library on PyPi
(https://pypi.python.org/pypi/google-cloud-storage) which also seems to be
an official Google project and perhaps has better support for unicode?
Attila
[1] https://pypi.python.org/pypi/GoogleAppEngineCloudStorageClient
[2] Traceback with u'...'
/usr/local/google_appengine_1.9.50/google/appengine/dist27/urllib.py:1277:
UnicodeWarning: Unicode equal comparison failed to convert both arguments
to Unicode - interpreting them as being unequal
return ''.join(map(quoter, s))
ERROR 2017-02-13 13:52:30,401 webapp2.py:1552] u'\xe1'
Traceback (most recent call last):
File "/usr/local/google_appengine_1.9.50/lib/webapp2-2.5.2/webapp2.py",
line 1535, in __call__
rv = self.handle_exception(request, response, e)
File "/usr/local/google_appengine_1.9.50/lib/webapp2-2.5.2/webapp2.py",
line 1529, in __call__
rv = self.router.dispatch(request, response)
File "/usr/local/google_appengine_1.9.50/lib/webapp2-2.5.2/webapp2.py",
line 1278, in default_dispatcher
return route.handler_adapter(request, response)
File "/usr/local/google_appengine_1.9.50/lib/webapp2-2.5.2/webapp2.py",
line 1102, in __call__
return handler.dispatch()
File "/usr/local/google_appengine_1.9.50/lib/webapp2-2.5.2/webapp2.py",
line 572, in dispatch
return self.handle_exception(e, self.app.debug)
File "/usr/local/google_appengine_1.9.50/lib/webapp2-2.5.2/webapp2.py",
line 570, in dispatch
return method(*args, **kwargs)
File "/tmp/gcs_test_unicode_names/main.py", line 14, in get
with gcs.open(fn, 'w') as f_out: f_out.write('old')
File "/tmp/gcs_test_unicode_names/lib/cloudstorage/cloudstorage_api.py",
line 91, in open
filename = api_utils._quote_filename(filename)
File "/tmp/gcs_test_unicode_names/lib/cloudstorage/api_utils.py", line
94, in _quote_filename
return urllib.quote(filename)
File
"/usr/local/google_appengine_1.9.50/google/appengine/dist27/urllib.py",
line 1277, in quote
return ''.join(map(quoter, s))
KeyError: u'\xe1'
----
[3] Traceback with manually encoded filename
---
ERROR 2017-02-13 13:53:44,679 module.py:892] Request to
'/_ah/gcs/app_default_bucket/\xc3\xa1' failed
INFO 2017-02-13 13:53:44,680 module.py:806] default: "POST
/_ah/gcs/app_default_bucket/%C3%A1 HTTP/1.1" 500 -
ERROR 2017-02-13 13:53:44,790 module.py:892] Request to
'/_ah/gcs/app_default_bucket/\xc3\xa1' failed
INFO 2017-02-13 13:53:44,791 module.py:806] default: "POST
/_ah/gcs/app_default_bucket/%C3%A1 HTTP/1.1" 500 -
ERROR 2017-02-13 13:53:45,003 module.py:892] Request to
'/_ah/gcs/app_default_bucket/\xc3\xa1' failed
INFO 2017-02-13 13:53:45,004 module.py:806] default: "POST
/_ah/gcs/app_default_bucket/%C3%A1 HTTP/1.1" 500 -
ERROR 2017-02-13 13:53:45,416 module.py:892] Request to
'/_ah/gcs/app_default_bucket/\xc3\xa1' failed
INFO 2017-02-13 13:53:45,416 module.py:806] default: "POST
/_ah/gcs/app_default_bucket/%C3%A1 HTTP/1.1" 500 -
ERROR 2017-02-13 13:53:46,255 module.py:892] Request to
'/_ah/gcs/app_default_bucket/\xc3\xa1' failed
INFO 2017-02-13 13:53:46,255 module.py:806] default: "POST
/_ah/gcs/app_default_bucket/%C3%A1 HTTP/1.1" 500 -
ERROR 2017-02-13 13:53:47,868 module.py:892] Request to
'/_ah/gcs/app_default_bucket/\xc3\xa1' failed
INFO 2017-02-13 13:53:47,868 module.py:806] default: "POST
/_ah/gcs/app_default_bucket/%C3%A1 HTTP/1.1" 500 -
ERROR 2017-02-13 13:53:51,082 module.py:892] Request to
'/_ah/gcs/app_default_bucket/\xc3\xa1' failed
INFO 2017-02-13 13:53:51,082 module.py:806] default: "POST
/_ah/gcs/app_default_bucket/%C3%A1 HTTP/1.1" 500 -
ERROR 2017-02-13 13:53:51,084 webapp2.py:1552] Expect status [201] from
Google Storage. But got status 500.
Path: '/app_default_bucket/%C3%A1'.
Request headers: {'x-goog-api-version': '2', 'x-goog-resumable': 'start',
'accept-encoding': 'gzip, *'}.
Response headers: {'server': 'Development/2.0', 'date': 'Mon, 13 Feb 2017
13:53:51 GMT', 'transfer-encoding': 'chunked'}.
Body: ''.
Extra info: None.
Traceback (most recent call last):
File "/usr/local/google_appengine_1.9.50/lib/webapp2-2.5.2/webapp2.py",
line 1535, in __call__
rv = self.handle_exception(request, response, e)
File "/usr/local/google_appengine_1.9.50/lib/webapp2-2.5.2/webapp2.py",
line 1529, in __call__
rv = self.router.dispatch(request, response)
File "/usr/local/google_appengine_1.9.50/lib/webapp2-2.5.2/webapp2.py",
line 1278, in default_dispatcher
return route.handler_adapter(request, response)
File "/usr/local/google_appengine_1.9.50/lib/webapp2-2.5.2/webapp2.py",
line 1102, in __call__
return handler.dispatch()
File "/usr/local/google_appengine_1.9.50/lib/webapp2-2.5.2/webapp2.py",
line 572, in dispatch
return self.handle_exception(e, self.app.debug)
File "/usr/local/google_appengine_1.9.50/lib/webapp2-2.5.2/webapp2.py",
line 570, in dispatch
return method(*args, **kwargs)
File "/tmp/gcs_test_unicode_names/main.py", line 14, in get
with gcs.open(fn.encode('utf-8'), 'w') as f_out: f_out.write('old')
File "/tmp/gcs_test_unicode_names/lib/cloudstorage/cloudstorage_api.py",
line 95, in open
return storage_api.StreamingBuffer(api, filename, content_type, options)
File "/tmp/gcs_test_unicode_names/lib/cloudstorage/storage_api.py", line
699, in __init__
body=content)
File "/tmp/gcs_test_unicode_names/lib/cloudstorage/errors.py", line 141,
in check_status
raise ServerError(msg)
ServerError: Expect status [201] from Google Storage. But got status 500.
Path: '/app_default_bucket/%C3%A1'.
Request headers: {'x-goog-api-version': '2', 'x-goog-resumable': 'start',
'accept-encoding': 'gzip, *'}.
Response headers: {'server': 'Development/2.0', 'date': 'Mon, 13 Feb 2017
13:53:51 GMT', 'transfer-encoding': 'chunked'}.
Body: ''.
Extra info: None.
---
[4] Traceback from production for .delete
---
Traceback (most recent call last):
File
"/base/data/home/runtimes/python27_experiment/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py",
line 1535, in __call__
rv = self.handle_exception(request, response, e)
...
gcs.delete(entry.filename)
...
filename = api_utils._quote_filename(filename)
...
return urllib.quote(filename)
File
"/base/data/home/runtimes/python27_experiment/python27_dist/lib/python2.7/urllib.py",
line 1277, in quote
return ''.join(map(quoter, s))
KeyError: u'\u0411'
---
--
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit
https://groups.google.com/d/msgid/google-appengine/14ddbf01-4c28-4b10-86a6-7a67a9d24941%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
