Hello Clay, Thank you for your suggestions. I need to use Swift in several purposes. 1. As a temporary storage for uncompressed video scenes (y4m, mp4) up to 120GB. (After conversion they will be deleted). 2. As a persistent storage for compressed video scenes (mp4) up to 120GB. Most of the compressed files will be much smaller (up to 15GB) but I need to be able to store files up to 120GB in some rare situations.
Thanks for suggestions again. I will increase the size of segments to 1MB. I haven't thought about two times segmentation. I think it will work. Correct me if I am wrong. Algorithm is the next: 1. Upload 1MB sub-segments (up to 500 sub-segments per segment). After they will be uploaded I will use "copy" request to create one 500MB segment. After that I will delete useless 1MB segments. 2. Upload up to 240 segments with 500MB and create a manifest for them. Is it correct? Will this algorithm be suitable for this situation? Best regards, Alexandr On Tue, Sep 13, 2016 at 7:24 PM, Clay Gerrard <clay.gerr...@gmail.com> wrote: > 120GB is a rather largish file. A standard 4TB drive is only gunna be > able to hold a few tens of these. My macbook could only hold two. > > 4KB at the bottom end of where you want to be in object storage - we're > talking about the block device sector size at that point. > > The sweet spot for objects in Swift is probably in the 10MB-100MB range > (2MB-200MB is fine). Unless your primary platform is mobile there's a good > chance the lower end of that range is reasonable target for resuming upload > (oh, my last 500K of my 2MB upload crapped out - I'll just restart the > other 1.5M, no bigz) > > But 120GB is a rather largish file. The default SLO segment size is only > like 1000 - but it'd be perfectly reasonable to bump that up an order of > magnitude if you control the config for that cluster - then you could > upload 20MB objects and call it done - but that doesn't give you much room > to scale. > > I would suggest you consider breaking the 120GB upload into ~500MB > sub-segments (each with ~500x1MB objects) - then assemble the final SLO > manifest from the sub-segments [1]. > > Great to hear about people solving big storage problems with Swift - would > you care to share anymore about your use-case? Come find swifty folks in > #openstack-swift on Freenode if you wanna chat! > > -Clay > > 1. final paragraph of http://docs.openstack.org/developer/swift/overview_ > large_objects.html#uploading-the-manifest > > On Tue, Sep 13, 2016 at 5:21 AM, Alexandr Porunov < > alexandr.poru...@gmail.com> wrote: > >> Hello, >> >> Does somebody know how to minimize segments count of a file? >> I have to implement an upload file service with the chunked upload >> feature. If a client lose connection for some time we have to be able >> continue to upload the file after connection is established (From the last >> successfuly uploaded chunk). For it I will use "Static Large Objects" to >> create small files each with 4 KBytes. >> >> The problem is that if we upload very big files then we will create too >> many small 4 KBytes segments of the big file. >> >> My service have to be able to store files up to 120GB. >> If we want to store a file with 120GB then we will have 31457280 segments >> of the file in the object storage. It isn't nice because of additional cost >> for metadate overhead of each file + big metadata file + decrease of >> performance. >> >> It will be much better if we could combine all that segments in a bigger >> segments each 1-2 GB. So, that after combination we would get maximum of >> 60-120 segments of the file. >> >> I know that we can combine all segments into one by using "copy" command >> to copy all segments into the new object but the problem is that if our >> object will be more than 5 GB then we can't use "copy" command (because it >> will show an error). >> >> Does somebody have any ideas how to develop chunk upload or minimize the >> count of segments? >> >> Sincerely, >> Alexandr >> >> _______________________________________________ >> Mailing list: http://lists.openstack.org/cgi >> -bin/mailman/listinfo/openstack >> Post to : openstack@lists.openstack.org >> Unsubscribe : http://lists.openstack.org/cgi >> -bin/mailman/listinfo/openstack >> >> >
_______________________________________________ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
