MooseFS is the way to go here.
I have it working on android SD cards and of course normal Linux servers over the internet and over Yggdrasil-network. One of my in-progress anarchy projects is a global hard drive for all of humanity’s knowledge. I would LOVE to get involved with this preservation project technically in a volunteer capacity. I can build a cutting edge resilient distributed storage system for cheaper than anything currently on the market. Please reach out or pass along my email. Alex On Sun, Apr 6, 2025 at 11:08 PM Linas Vepstas <linasveps...@gmail.com> wrote: > OK what you will read below might sound insane but I am obliged to ask. > > There are 275 petabytes of NIH data at risk of being deleted. Cancer > research, medical data, HIPAA type stuff. Currently unclear where it's > located, how it's managed, who has access to what, but lets ignore > that for now. It's presumably splattered across data centers, cloud, > AWS, supercomputing labs, who knows. Everywhere. > > I'm talking to a biomed person in Australias that uses NCBI data > daily, she's in talks w/ Australian govt to copy and preserve the > datasets they use. Some multi-petabytes of stuff. I don't know. > > While bouncing around tech ideas, IPFS and Ceph came up. My experience > with IPFS is that it's not a serious contender for anything. My > experience with Ceph is that it's more-or-less A-list. > > OK. So here's the question: is it possible to (has anyone tried) set > up an internet-wide Ceph cluster? Ticking off the typical checkboxes > for "decentralized storage"? Stuff, like: internet connections need to > be encrypted. Connections go down, come back up. Slow. Sure, national > labs may have multi-terabit fiber, but little itty-bitty participants > trying to contribute a small collection of disks to a large pool might > only have a gigabit connection, of which maybe 10% is "usable". > Barely. So, a hostile networking environment. > > Is this like, totally insane, run away now, can't do that, it won't > work idea, or is there some glimmer of hope? > > Am I misunderstanding something about IPFS that merits taking a second > look at it? > > Is there any other way of getting scalable reliable "decentralized" > internet-wide storage? > > I mean, yes, of course, the conventional answer is that it could be > copied to AWS or some national lab or two somewhere in the EU or Aus > or UK or where-ever, That's the "obvious" answer. I'm looking for a > non-obvious answer, an IPFS-like thing, but one that actually works. > Could it work? > > -- Linas > > > -- > Patrick: Are they laughing at us? > Sponge Bob: No, Patrick, they are laughing next to us. > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io > _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
