Good info. Now that we've performed the initial clojars drive, which was performed at a very fortuitous time, do you think that the problem is primarily one of money, man poweror, or both? I realise that there's a lot of kI'm happy to help in I'm one of Rikkkkeeee way, because I think we definitely want to avoid some of the past issues in Node JS - which I think they have mostly solved now
Lucas > On 4 Jan 2016, at 5:14 AM, Nando Breiter <na...@aria-media.com> wrote: > > I've spent some time looking into both Cloudflare and Fastly over the > weekend. Fastly seems to have a sophisticated purging mechanism which the > ticket mentions would be a requirement. See > https://docs.fastly.com/guides/purging/ > > Initial setup is dead easy (for both), basically requiring a signup and a > change to the DNS record, adding a CNAME. Fastly charges for bandwidth and > caches everything. Cloudflare charges monthly flat rates but only caches the > most popular assets, unless the subscriber pays $200 a month. In a nutshell, > you have full control over the content cached in the CDN with Fastly and full > control of the price paid, but not the service rendered, with Cloudflare. > > > > Aria Media Sagl > Via Rompada 40 > 6987 Caslano > Switzerland > > +41 (0)91 600 9601 > +41 (0)76 303 4477 cell > skype: ariamedia > >> On Sun, Jan 3, 2016 at 8:00 PM, Toby Crawley <t...@tcrawley.org> wrote: >> Cloudflare (or a similar CDN) would be useful - we have an open issue >> to implement that, but haven't had a chance to get to it: >> https://github.com/clojars/clojars-web/issues/434 >> >> - Toby >> >> On Sat, Jan 2, 2016 at 4:30 AM, Nando Breiter <na...@aria-media.com> wrote: >> > Would CloudFlare help on the short term? I haven't used the service yet, I >> > just ran across it researching DDoS solutions, but judging from the >> > overview >> > of how it works, it might be able to cache all clojars.org assets in a >> > distributed manner and handle the DNS issue as well. >> > https://www.cloudflare.com/ If it would work, the advantage is a very quick >> > initial setup. All you need to do is let them handle the DNS. >> > >> > >> > >> > >> > >> > Aria Media Sagl >> > Via Rompada 40 >> > 6987 Caslano >> > Switzerland >> > >> > +41 (0)91 600 9601 >> > +41 (0)76 303 4477 cell >> > skype: ariamedia >> > >> > On Sat, Jan 2, 2016 at 4:31 AM, Toby Crawley <t...@tcrawley.org> wrote: >> >> >> >> Given the recent DDoS-triggered outages at linode (including the one >> >> today that has been the worst yet, currently 10 hours at the time I'm >> >> writing this), I've been giving some more thought to how we can make >> >> future outages less painful for the community. >> >> >> >> I have an open issue[1] (but no code yet) to move the repository off >> >> of the server and on to a block store (s3, etc), with the goal there >> >> to make repo reads (which is what we use clojars for 99.9% of the >> >> time) independent of the status of the server. But I'm not sure that >> >> really solves the problem we are seeing today. Currently, we have two >> >> points of failure for repo reads: >> >> >> >> (1) the server itself (hosted on linode) >> >> (2) DNS for the clojars.org domain (also hosted on linode) >> >> >> >> moving the repo off of the server to a block store still has two >> >> points of failure: >> >> >> >> (1) the block store (aws, rackspace, etc) >> >> (2) DNS for the clojars.org domain, since we would CNAME the block >> >> store (hosted on linode) >> >> >> >> Though the block store provider would probably be better distributed, >> >> and have more resources to withstand a DDoS (but do any block store >> >> providers have 100% uptime?). >> >> >> >> The block store solution is complex - it introduces more moving parts >> >> into clojars, and requires reworking the way we generate usage stats, >> >> and how the api gets its data. It also requires reworking the way we >> >> administer the repo (deletion requests, cleaning up failed/partial >> >> deploys). And it may not solve the availability problem at all, since >> >> we still have two points of failure. >> >> >> >> I think a better solution may be to have multiple mirrors of the repo, >> >> either run by concerned citizens or maintained by the clojars staff. I >> >> know some folks in the community already run internal caching proxies >> >> or rsynced mirrors (and are probably chuckling knowingly at those of >> >> us affected by the outage), but those proxies don't really help those >> >> in the community that don't have that internal infrastructure. And I >> >> don't want to recommend that everyone set up a private mirror - that >> >> seems like a lot of wasted effort. >> >> >> >> Ideally, it would be nice if we had a turn-key tool for creating a >> >> mirror of clojars. We currently provide a way to rsync the repo[2], so >> >> the seed for a mirror could be small, and could then slurp down the >> >> full repo (and could continue to do so on a schedule to remain up to >> >> date). We could then publish a list of mirrors that the community >> >> could turn to in times of need (or use all the time, if they are >> >> closer geographically or just generally more responsive). Any deploys >> >> would still need to hit the primary server, but deploys are are >> >> dwarfed by reads. >> >> >> >> There are a few issues with using mirrors: >> >> >> >> (1) security - with artifacts in more places, there are more >> >> opportunities to to introduce malicious versions. This could be >> >> prevented if we had better tools for verifying that the artifacts >> >> are signed by trusted keys, and we required that all artifacts be >> >> signed, but that's not the case currently. But if we had a regular >> >> process that crawled all of the mirrors and the canonical repo to >> >> verify that the checksums every artifact are identical, this could >> >> actually improve security, since we could detect if any checksum >> >> had been changed (a malicious party would have to change the >> >> checksum of a modified artifact, since maven/lein/boot all confirm >> >> checksums by default). >> >> >> >> (2) download stats - any downloads from a mirror wouldn't get >> >> reflected in the stats for the artifact unless we had some way to >> >> report those stats back to clojars.org. We currently generate the >> >> stats by parsing the nginx access logs, mirrors could do the same >> >> and report stats back to clojars.org if we care enough about >> >> this. We don't get stats from the existing private mirrors, and >> >> the stats aren't critical, so this may be a non-issue, and >> >> definitely isn't something that has to be solved right away, if >> >> ever. >> >> >> >> The repo is just served as static files, so I think a mirror could >> >> simply be: >> >> >> >> (1) a webserver (preferably (required to be?) HTTPS) >> >> (2) a cronjob that rsyncs every N minutes >> >> >> >> And the cronjob would just need the rsync command in [2], so, to get >> >> this started, we just need: >> >> >> >> (1) linode to be up >> >> (2) people willing to run mirrors >> >> >> >> (I would say "(3) add a page to the wiki on how to use a mirror", but >> >> that would destroy the symmetry of all the other 2-item lists in this >> >> message) >> >> >> >> And it would be nice to have the process in place to verify checksums >> >> soon - that would actually be a boon if we had another linode >> >> compromise[3]. >> >> >> >> Does anyone see any issues with this plan - I'm curious if there are >> >> security implications (or anything else) that I haven't thought of? >> >> >> >> Are you willing to run a mirror? >> >> >> >> One issue that comes to mind is if we do decide to move the repo to a >> >> block store, it actually makes mirroring more difficult unless we keep >> >> a copy of the repo on disk on clojars.org as well. But I would like to >> >> have mirrors in place as soon as possible, and worry about that later. >> >> >> >> - Toby >> >> >> >> [1]: https://github.com/clojars/clojars-web/issues/433 >> >> [2]: >> >> https://github.com/clojars/clojars-web/wiki/Data#rsync-the-whole-classic-repository >> >> [3]: >> >> https://groups.google.com/d/msg/clojars-maintainers/uAVJVwRAnSU/WISqQn5E9KIJ >> >> >> >> -- >> >> You received this message because you are subscribed to the Google >> >> Groups "Clojure" group. >> >> To post to this group, send email to clojure@googlegroups.com >> >> Note that posts from new members are moderated - please be patient with >> >> your first post. >> >> To unsubscribe from this group, send email to >> >> clojure+unsubscr...@googlegroups.com >> >> For more options, visit this group at >> >> http://groups.google.com/group/clojure?hl=en >> >> --- >> >> You received this message because you are subscribed to the Google Groups >> >> "Clojure" group. >> >> To unsubscribe from this group and stop receiving emails from it, send an >> >> email to clojure+unsubscr...@googlegroups.com. >> >> For more options, visit https://groups.google.com/d/optout. >> > >> > >> > -- >> > You received this message because you are subscribed to the Google >> > Groups "Clojure" group. >> > To post to this group, send email to clojure@googlegroups.com >> > Note that posts from new members are moderated - please be patient with >> > your >> > first post. >> > To unsubscribe from this group, send email to >> > clojure+unsubscr...@googlegroups.com >> > For more options, visit this group at >> > http://groups.google.com/group/clojure?hl=en >> > --- >> > You received this message because you are subscribed to the Google Groups >> > "Clojure" group. >> > To unsubscribe from this group and stop receiving emails from it, send an >> > email to clojure+unsubscr...@googlegroups.com. >> > For more options, visit https://groups.google.com/d/optout. >> >> -- >> You received this message because you are subscribed to the Google >> Groups "Clojure" group. >> To post to this group, send email to clojure@googlegroups.com >> Note that posts from new members are moderated - please be patient with your >> first post. >> To unsubscribe from this group, send email to >> clojure+unsubscr...@googlegroups.com >> For more options, visit this group at >> http://groups.google.com/group/clojure?hl=en >> --- >> You received this message because you are subscribed to the Google Groups >> "Clojure" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to clojure+unsubscr...@googlegroups.com. >> For more options, visit https://groups.google.com/d/optout. > > -- > You received this message because you are subscribed to the Google > Groups "Clojure" group. > To post to this group, send email to clojure@googlegroups.com > Note that posts from new members are moderated - please be patient with your > first post. > To unsubscribe from this group, send email to > clojure+unsubscr...@googlegroups.com > For more options, visit this group at > http://groups.google.com/group/clojure?hl=en > --- > You received this message because you are subscribed to the Google Groups > "Clojure" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to clojure+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
