Additional features.
* No "master server". No Single Point of Failure.
* Resource location. A small number of master servers kept in sync like
DNS with tiers of secondary resources. I think blockchains also have a
similar setup?
* Resource identification. A scheme like LDAP. For example:
cn=library,catalog=dewey,filing=504.3,...
cn=library,country=us,catalog=libraryofcongress,...
country=us,agency=nih,department=...
cn=upc,isbn=...
A document/document set should have a canonical name, but allow
alternate names for ease of location, such as author searches, general
topics and the like.
I considered OIDs as an alternative, but LDAP names are more
human-friendly and easier to add sub-domains to without petitioning a
master registrar. Also there's a better option for adding attributes to
the entry description.
On 4/7/25 09:39, Tim Holloway wrote:
Yeah, Ceph in its current form doesn't seem like a good fit.
I think that what we need to support the world's knowledge in the face
of enstupidification is some sort of distributed holographic
datastore. so, like Ceph's PG replication, a torrent-like ability to
pull from multiple unreliable sources, a good indexing mechanism and,
protections against tampering. Probably with a touch of git as well.
I'm sure there's more, but those are items that immediately occur to me.
Tim
On 4/7/25 09:10, Alex Buie wrote:
MooseFS is the way to go here.
I have it working on android SD cards and of course normal Linux servers
over the internet and over Yggdrasil-network.
One of my in-progress anarchy projects is a global hard drive for all of
humanity’s knowledge.
I would LOVE to get involved with this preservation project
technically in
a volunteer capacity. I can build a cutting edge resilient distributed
storage system for cheaper than anything currently on the market.
Please reach out or pass along my email.
Alex
On Sun, Apr 6, 2025 at 11:08 PM Linas Vepstas <linasveps...@gmail.com>
wrote:
OK what you will read below might sound insane but I am obliged to ask.
There are 275 petabytes of NIH data at risk of being deleted. Cancer
research, medical data, HIPAA type stuff. Currently unclear where it's
located, how it's managed, who has access to what, but lets ignore
that for now. It's presumably splattered across data centers, cloud,
AWS, supercomputing labs, who knows. Everywhere.
I'm talking to a biomed person in Australias that uses NCBI data
daily, she's in talks w/ Australian govt to copy and preserve the
datasets they use. Some multi-petabytes of stuff. I don't know.
While bouncing around tech ideas, IPFS and Ceph came up. My experience
with IPFS is that it's not a serious contender for anything. My
experience with Ceph is that it's more-or-less A-list.
OK. So here's the question: is it possible to (has anyone tried) set
up an internet-wide Ceph cluster? Ticking off the typical checkboxes
for "decentralized storage"? Stuff, like: internet connections need to
be encrypted. Connections go down, come back up. Slow. Sure, national
labs may have multi-terabit fiber, but little itty-bitty participants
trying to contribute a small collection of disks to a large pool might
only have a gigabit connection, of which maybe 10% is "usable".
Barely. So, a hostile networking environment.
Is this like, totally insane, run away now, can't do that, it won't
work idea, or is there some glimmer of hope?
Am I misunderstanding something about IPFS that merits taking a second
look at it?
Is there any other way of getting scalable reliable "decentralized"
internet-wide storage?
I mean, yes, of course, the conventional answer is that it could be
copied to AWS or some national lab or two somewhere in the EU or Aus
or UK or where-ever, That's the "obvious" answer. I'm looking for a
non-obvious answer, an IPFS-like thing, but one that actually works.
Could it work?
-- Linas
--
Patrick: Are they laughing at us?
Sponge Bob: No, Patrick, they are laughing next to us.
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
