Hi Raquel, Are u using a compressed filesystem? I recently moved everything including /home directory to ZFS - which gave ~ 1.4X compression for old adsc images. Remember vaguely, years before, James suggested to use aufs/unionfs. You could even enable data-deduplication to save redundant images.
In additon to James' suggestion of amazon glacier i would recommend 'backblaze'. I mainly use it for personal backup $50/year - but storage is unlimited. You can also get a drive FedEx -ed for retrieval. But be warned thst the GUI sucks. One could go for the business plan - clean commandline API based upload/download ~ about 350 per year. If anyone is interested 'backblaze' produce fantastic harddrive statistics. https://www.backblaze.com/blog/2018-hard-drive-failure-rates/ Markus On Friday, November 30, 2018, James Holton < 0000270165b9f4cf-dmarc-requ...@jiscmail.ac.uk> wrote: > The answer depends a lot on what you mean by "long-term storage". Do you > want the data to be available all the time on a mountable volume? Or is > putting it away on a shelf OK? Do you want the storage to be as > bulletproof and worry-free as possible? Or are you OK with the fate of > your data being somewhat nebulous, like in "the cloud"? The price points > for all these things are very different. > > You can now buy a single 8 TB drive for $230 USD. LTO6 tapes are > currently at ~7 USD/TB. Both of these are the current lowest price/TB for > disk and tape. Using the media, of course, generally requires attaching it > to a server that costs ~$5k-$10k USD. Amazon Glacier is free for uploads > and essentially free for downloading it back as long as you don't want more > than 1 GB per month. The other extreme is a NetApp, where you just want a > turnkey system that keeps your data as safe as possible, but is also really > fast. > > What do I do? I am currently deploying a RAID6 array of 8 TB drives for > high-performance storage. For archiving I used to use DVD-R, but that > can't keep up with a Pilatus, so now I'm on LTO6 tapes for off-line > backups. I know tapes are famous as "write-only media", but so far over > the last 10 years I haven't had any real trouble reading back an old LTO > tape. <touch wood> > > -James Holton > MAD Scientist > > > On 11/29/2018 12:54 PM, Lieberman, Raquel L wrote: > > Dear All, > > How do your labs handle long-term raw data backups? My lab is maxing out > our 6TB RAID backup (with two off-site mirrors) so I am investigating our > next long term solution. The vast majority of the data sets are published > structures (i.e. processed data deposited in PDB) or redundant/unusable so > immediate access is not anticipated, but the size of data sets is > increasing quickly with time, so I am looking for a scalable-yet-affordable > solution. > > Would be grateful for input into various options, e.g. bigger HD/RAIDs, > cloud backup, tape, anything else. > > I will compile. > > Thank you, > > Raquel > ------ > Raquel L. Lieberman, Ph.D. > Professor > School of Chemistry and Biochemistry > Georgia Institute of Technology > > > > ------------------------------ > > To unsubscribe from the CCP4BB list, click the following link: > https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB&A=1 > > > > ------------------------------ > > To unsubscribe from the CCP4BB list, click the following link: > https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB&A=1 > ######################################################################## To unsubscribe from the CCP4BB list, click the following link: https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB&A=1
