I would suggest also considering decentralized technologies. IPFS (https://ipfs.io/) looks like a good fit for the task.
- It is distributed, every node used to access the repository will contribute to it's availability. - Directory trees in IPFS work like Clojure's persistent data structures: they are immutable and share identical substructures. - IPFS is content-addressable: to access the repository one will only need to know the hash of the current version of the root directory. The content can not be changed without also changing the hash. - The current hash can be published using IPNS (https://github.com/ipfs/examples/tree/master/examples/ipns) or using a special DNS TXT record ("dnslink=/ipfs/<hash>") on clojars.org domain. Then the current version of the repository (at least the files other nodes have copies of) will be accessible regardless of the availability of the main server - via public IPFS gateway: https://ipfs.io/ipns/clojars.org/ - or local gateway: http://localhost:8080/ipns/clojars.org/ - or local fuse mount at /ipns/clojars.org/ Toby Crawley <t...@tcrawley.org> writes: > Given the recent DDoS-triggered outages at linode (including the one > today that has been the worst yet, currently 10 hours at the time I'm > writing this), I've been giving some more thought to how we can make > future outages less painful for the community. > > I have an open issue[1] (but no code yet) to move the repository off > of the server and on to a block store (s3, etc), with the goal there > to make repo reads (which is what we use clojars for 99.9% of the > time) independent of the status of the server. But I'm not sure that > really solves the problem we are seeing today. Currently, we have two > points of failure for repo reads: > > (1) the server itself (hosted on linode) > (2) DNS for the clojars.org domain (also hosted on linode) > > moving the repo off of the server to a block store still has two > points of failure: > > (1) the block store (aws, rackspace, etc) > (2) DNS for the clojars.org domain, since we would CNAME the block > store (hosted on linode) > > Though the block store provider would probably be better distributed, > and have more resources to withstand a DDoS (but do any block store > providers have 100% uptime?). > > The block store solution is complex - it introduces more moving parts > into clojars, and requires reworking the way we generate usage stats, > and how the api gets its data. It also requires reworking the way we > administer the repo (deletion requests, cleaning up failed/partial > deploys). And it may not solve the availability problem at all, since > we still have two points of failure. > > I think a better solution may be to have multiple mirrors of the repo, > either run by concerned citizens or maintained by the clojars staff. I > know some folks in the community already run internal caching proxies > or rsynced mirrors (and are probably chuckling knowingly at those of > us affected by the outage), but those proxies don't really help those > in the community that don't have that internal infrastructure. And I > don't want to recommend that everyone set up a private mirror - that > seems like a lot of wasted effort. > > Ideally, it would be nice if we had a turn-key tool for creating a > mirror of clojars. We currently provide a way to rsync the repo[2], so > the seed for a mirror could be small, and could then slurp down the > full repo (and could continue to do so on a schedule to remain up to > date). We could then publish a list of mirrors that the community > could turn to in times of need (or use all the time, if they are > closer geographically or just generally more responsive). Any deploys > would still need to hit the primary server, but deploys are are > dwarfed by reads. > > There are a few issues with using mirrors: > > (1) security - with artifacts in more places, there are more > opportunities to to introduce malicious versions. This could be > prevented if we had better tools for verifying that the artifacts > are signed by trusted keys, and we required that all artifacts be > signed, but that's not the case currently. But if we had a regular > process that crawled all of the mirrors and the canonical repo to > verify that the checksums every artifact are identical, this could > actually improve security, since we could detect if any checksum > had been changed (a malicious party would have to change the > checksum of a modified artifact, since maven/lein/boot all confirm > checksums by default). > > (2) download stats - any downloads from a mirror wouldn't get > reflected in the stats for the artifact unless we had some way to > report those stats back to clojars.org. We currently generate the > stats by parsing the nginx access logs, mirrors could do the same > and report stats back to clojars.org if we care enough about > this. We don't get stats from the existing private mirrors, and > the stats aren't critical, so this may be a non-issue, and > definitely isn't something that has to be solved right away, if > ever. > > The repo is just served as static files, so I think a mirror could > simply be: > > (1) a webserver (preferably (required to be?) HTTPS) > (2) a cronjob that rsyncs every N minutes > > And the cronjob would just need the rsync command in [2], so, to get > this started, we just need: > > (1) linode to be up > (2) people willing to run mirrors > > (I would say "(3) add a page to the wiki on how to use a mirror", but > that would destroy the symmetry of all the other 2-item lists in this > message) > > And it would be nice to have the process in place to verify checksums > soon - that would actually be a boon if we had another linode > compromise[3]. > > Does anyone see any issues with this plan - I'm curious if there are > security implications (or anything else) that I haven't thought of? > > Are you willing to run a mirror? > > One issue that comes to mind is if we do decide to move the repo to a > block store, it actually makes mirroring more difficult unless we keep > a copy of the repo on disk on clojars.org as well. But I would like to > have mirrors in place as soon as possible, and worry about that later. > > - Toby > > [1]: https://github.com/clojars/clojars-web/issues/433 > [2]: > https://github.com/clojars/clojars-web/wiki/Data#rsync-the-whole-classic-repository > [3]: > https://groups.google.com/d/msg/clojars-maintainers/uAVJVwRAnSU/WISqQn5E9KIJ > > -- > You received this message because you are subscribed to the Google > Groups "Clojure" group. > To post to this group, send email to clojure@googlegroups.com > Note that posts from new members are moderated - please be patient with your > first post. > To unsubscribe from this group, send email to > clojure+unsubscr...@googlegroups.com > For more options, visit this group at > http://groups.google.com/group/clojure?hl=en > --- > You received this message because you are subscribed to the Google Groups > "Clojure" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to clojure+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. -- Mikhail -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
signature.asc
Description: PGP signature
