I've spent some time looking into both Cloudflare and Fastly over the weekend. Fastly seems to have a sophisticated purging mechanism which the ticket mentions would be a requirement. See https://docs.fastly.com/guides/purging/
Initial setup is dead easy (for both), basically requiring a signup and a change to the DNS record, adding a CNAME. Fastly charges for bandwidth and caches everything. Cloudflare charges monthly flat rates but only caches the most popular assets, unless the subscriber pays $200 a month. In a nutshell, you have full control over the content cached in the CDN with Fastly and full control of the price paid, but not the service rendered, with Cloudflare. Aria Media Sagl Via Rompada 40 6987 Caslano Switzerland +41 (0)91 600 9601 +41 (0)76 303 4477 cell skype: ariamedia On Sun, Jan 3, 2016 at 8:00 PM, Toby Crawley <t...@tcrawley.org> wrote: > Cloudflare (or a similar CDN) would be useful - we have an open issue > to implement that, but haven't had a chance to get to it: > https://github.com/clojars/clojars-web/issues/434 > > - Toby > > On Sat, Jan 2, 2016 at 4:30 AM, Nando Breiter <na...@aria-media.com> > wrote: > > Would CloudFlare help on the short term? I haven't used the service yet, > I > > just ran across it researching DDoS solutions, but judging from the > overview > > of how it works, it might be able to cache all clojars.org assets in a > > distributed manner and handle the DNS issue as well. > > https://www.cloudflare.com/ If it would work, the advantage is a very > quick > > initial setup. All you need to do is let them handle the DNS. > > > > > > > > > > > > Aria Media Sagl > > Via Rompada 40 > > 6987 Caslano > > Switzerland > > > > +41 (0)91 600 9601 > > +41 (0)76 303 4477 cell > > skype: ariamedia > > > > On Sat, Jan 2, 2016 at 4:31 AM, Toby Crawley <t...@tcrawley.org> wrote: > >> > >> Given the recent DDoS-triggered outages at linode (including the one > >> today that has been the worst yet, currently 10 hours at the time I'm > >> writing this), I've been giving some more thought to how we can make > >> future outages less painful for the community. > >> > >> I have an open issue[1] (but no code yet) to move the repository off > >> of the server and on to a block store (s3, etc), with the goal there > >> to make repo reads (which is what we use clojars for 99.9% of the > >> time) independent of the status of the server. But I'm not sure that > >> really solves the problem we are seeing today. Currently, we have two > >> points of failure for repo reads: > >> > >> (1) the server itself (hosted on linode) > >> (2) DNS for the clojars.org domain (also hosted on linode) > >> > >> moving the repo off of the server to a block store still has two > >> points of failure: > >> > >> (1) the block store (aws, rackspace, etc) > >> (2) DNS for the clojars.org domain, since we would CNAME the block > >> store (hosted on linode) > >> > >> Though the block store provider would probably be better distributed, > >> and have more resources to withstand a DDoS (but do any block store > >> providers have 100% uptime?). > >> > >> The block store solution is complex - it introduces more moving parts > >> into clojars, and requires reworking the way we generate usage stats, > >> and how the api gets its data. It also requires reworking the way we > >> administer the repo (deletion requests, cleaning up failed/partial > >> deploys). And it may not solve the availability problem at all, since > >> we still have two points of failure. > >> > >> I think a better solution may be to have multiple mirrors of the repo, > >> either run by concerned citizens or maintained by the clojars staff. I > >> know some folks in the community already run internal caching proxies > >> or rsynced mirrors (and are probably chuckling knowingly at those of > >> us affected by the outage), but those proxies don't really help those > >> in the community that don't have that internal infrastructure. And I > >> don't want to recommend that everyone set up a private mirror - that > >> seems like a lot of wasted effort. > >> > >> Ideally, it would be nice if we had a turn-key tool for creating a > >> mirror of clojars. We currently provide a way to rsync the repo[2], so > >> the seed for a mirror could be small, and could then slurp down the > >> full repo (and could continue to do so on a schedule to remain up to > >> date). We could then publish a list of mirrors that the community > >> could turn to in times of need (or use all the time, if they are > >> closer geographically or just generally more responsive). Any deploys > >> would still need to hit the primary server, but deploys are are > >> dwarfed by reads. > >> > >> There are a few issues with using mirrors: > >> > >> (1) security - with artifacts in more places, there are more > >> opportunities to to introduce malicious versions. This could be > >> prevented if we had better tools for verifying that the artifacts > >> are signed by trusted keys, and we required that all artifacts be > >> signed, but that's not the case currently. But if we had a regular > >> process that crawled all of the mirrors and the canonical repo to > >> verify that the checksums every artifact are identical, this could > >> actually improve security, since we could detect if any checksum > >> had been changed (a malicious party would have to change the > >> checksum of a modified artifact, since maven/lein/boot all confirm > >> checksums by default). > >> > >> (2) download stats - any downloads from a mirror wouldn't get > >> reflected in the stats for the artifact unless we had some way to > >> report those stats back to clojars.org. We currently generate the > >> stats by parsing the nginx access logs, mirrors could do the same > >> and report stats back to clojars.org if we care enough about > >> this. We don't get stats from the existing private mirrors, and > >> the stats aren't critical, so this may be a non-issue, and > >> definitely isn't something that has to be solved right away, if > >> ever. > >> > >> The repo is just served as static files, so I think a mirror could > >> simply be: > >> > >> (1) a webserver (preferably (required to be?) HTTPS) > >> (2) a cronjob that rsyncs every N minutes > >> > >> And the cronjob would just need the rsync command in [2], so, to get > >> this started, we just need: > >> > >> (1) linode to be up > >> (2) people willing to run mirrors > >> > >> (I would say "(3) add a page to the wiki on how to use a mirror", but > >> that would destroy the symmetry of all the other 2-item lists in this > >> message) > >> > >> And it would be nice to have the process in place to verify checksums > >> soon - that would actually be a boon if we had another linode > >> compromise[3]. > >> > >> Does anyone see any issues with this plan - I'm curious if there are > >> security implications (or anything else) that I haven't thought of? > >> > >> Are you willing to run a mirror? > >> > >> One issue that comes to mind is if we do decide to move the repo to a > >> block store, it actually makes mirroring more difficult unless we keep > >> a copy of the repo on disk on clojars.org as well. But I would like to > >> have mirrors in place as soon as possible, and worry about that later. > >> > >> - Toby > >> > >> [1]: https://github.com/clojars/clojars-web/issues/433 > >> [2]: > >> > https://github.com/clojars/clojars-web/wiki/Data#rsync-the-whole-classic-repository > >> [3]: > >> > https://groups.google.com/d/msg/clojars-maintainers/uAVJVwRAnSU/WISqQn5E9KIJ > >> > >> -- > >> You received this message because you are subscribed to the Google > >> Groups "Clojure" group. > >> To post to this group, send email to clojure@googlegroups.com > >> Note that posts from new members are moderated - please be patient with > >> your first post. > >> To unsubscribe from this group, send email to > >> clojure+unsubscr...@googlegroups.com > >> For more options, visit this group at > >> http://groups.google.com/group/clojure?hl=en > >> --- > >> You received this message because you are subscribed to the Google > Groups > >> "Clojure" group. > >> To unsubscribe from this group and stop receiving emails from it, send > an > >> email to clojure+unsubscr...@googlegroups.com. > >> For more options, visit https://groups.google.com/d/optout. > > > > > > -- > > You received this message because you are subscribed to the Google > > Groups "Clojure" group. > > To post to this group, send email to clojure@googlegroups.com > > Note that posts from new members are moderated - please be patient with > your > > first post. > > To unsubscribe from this group, send email to > > clojure+unsubscr...@googlegroups.com > > For more options, visit this group at > > http://groups.google.com/group/clojure?hl=en > > --- > > You received this message because you are subscribed to the Google Groups > > "Clojure" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to clojure+unsubscr...@googlegroups.com. > > For more options, visit https://groups.google.com/d/optout. > > -- > You received this message because you are subscribed to the Google > Groups "Clojure" group. > To post to this group, send email to clojure@googlegroups.com > Note that posts from new members are moderated - please be patient with > your first post. > To unsubscribe from this group, send email to > clojure+unsubscr...@googlegroups.com > For more options, visit this group at > http://groups.google.com/group/clojure?hl=en > --- > You received this message because you are subscribed to the Google Groups > "Clojure" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to clojure+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
