Hi John,
That indeed helped. But I'm stuck in using the python interface to make it
work:
_opts = {'object_uu_threads': 10, 'segment_size': 1048576}
with SwiftService(options=_opts) as swift:
Seems not working.
Also my another question is about the relationship with uu_threads and
segment, my understanding is that if we upload by segments, we should use
more than 1 threads so that parallelism is possible, right?
Best,
Jialin
On Thu, Oct 5, 2017 at 11:32 AM, John Dickinson <m...@not.mn> wrote:
> I'm not familiar with that format, but in general, if you're dealing with
> large objects, you can get better performance by splitting the data in the
> client, uploading each separate segment concurrently, and then creating a
> large object manifest to tie the segments together (allowing future access
> by either segments or as a logical whole).
>
> https://docs.openstack.org/swift/latest/overview_large_
> objects.html#module-swift.common.middleware.slo
>
> --John
>
>
>
> On 5 Oct 2017, at 11:24, Jialin Liu wrote:
>
> Thank you John,
>
> I think I figured out the way. My case is a little bit rare as I'm dealing
> with the HDF5 file format, I can not think of any way to retrieve the
> entire in memory file as a single python object and then serialize it.
> What I did is to use the hdf5.get_file_image function to get a memory
> image of the file, and then use, io.ByteIO(image) to make it a file-like
> object.
>
> So far, it works well, but I believe it is not the best way in terms of
> performance.
>
> Best,
> Jialin
>
> On Thu, Oct 5, 2017 at 8:25 AM, John Dickinson <m...@not.mn> wrote:
>
>> If you've got an arbitrary object in Python, you'll need to serialize it
>> to a file-like object. You could keep it in memory and use a StringIO
>> type, or you could serialize it to disk and open() it like any other
>> file.
>>
>> Ultimately, Swift is storing arbitrary bytes and doesn't care what they
>> are. You, as the Swift client (i.e. API user), need to dump those bytes on
>> the network to send them to Swift. As long as you're transforming your
>> Python object[s] in some regular way that makes sense in your application,
>> it doesn't matter what bytes you send to Swift.
>>
>> --John
>>
>> On 5 Oct 2017, at 8:07, Jialin Liu wrote:
>>
>> Hi,
>> It seems to me that openstack swift only supports file upload/download, is
>> it possible to put a python object to swift store?
>> The doc says we could use file-like object, e.g., StringIO, but this is
>> very limited. I'd like to write a numpy array or other python object into
>> the swift store, can anybody tell me the solution?
>>
>> Best,
>> Jialin
>> ____________________________________________________________
>> ______________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscrib
>> e
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>> ____________________________________________________________
>> ______________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscrib
>> e
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
