Any tooling would also have to upgrade to clj-http 2.0.0 and/or HttpClient 4.5, because before that SNI was broken even on Java 8:
https://issues.apache.org/jira/browse/HTTPCLIENT-1613?devStatusDetailDialog=repository Supposedly fixed in 4.5 of HttpClient, which 2.0.0 of clj-http pulls in, but I haven't tested to confirm. -ken -- ----- On Fri, Jan 01, 2016 at 10:49:13PM -0500, Toby Crawley wrote: > One potential issue with the mirrors is java 6 and HTTPS - the mirrors > couldn't use 2048-bit dhparams[1] or SNI[2], since neither are > supported in java 6. Yes, we all should be on java 7 or 8 at this > point, but I believe Intellij still uses java 6 on MacOS, which would > mean Cursive couldn't download from the mirrors. > > [1]: https://weakdh.org/sysadmin.html > [2]: https://en.wikipedia.org/wiki/Server_Name_Indication > > On Fri, Jan 1, 2016 at 10:31 PM, Toby Crawley <t...@tcrawley.org> wrote: > > Given the recent DDoS-triggered outages at linode (including the one > > today that has been the worst yet, currently 10 hours at the time I'm > > writing this), I've been giving some more thought to how we can make > > future outages less painful for the community. > > > > I have an open issue[1] (but no code yet) to move the repository off > > of the server and on to a block store (s3, etc), with the goal there > > to make repo reads (which is what we use clojars for 99.9% of the > > time) independent of the status of the server. But I'm not sure that > > really solves the problem we are seeing today. Currently, we have two > > points of failure for repo reads: > > > > (1) the server itself (hosted on linode) > > (2) DNS for the clojars.org domain (also hosted on linode) > > > > moving the repo off of the server to a block store still has two > > points of failure: > > > > (1) the block store (aws, rackspace, etc) > > (2) DNS for the clojars.org domain, since we would CNAME the block > > store (hosted on linode) > > > > Though the block store provider would probably be better distributed, > > and have more resources to withstand a DDoS (but do any block store > > providers have 100% uptime?). > > > > The block store solution is complex - it introduces more moving parts > > into clojars, and requires reworking the way we generate usage stats, > > and how the api gets its data. It also requires reworking the way we > > administer the repo (deletion requests, cleaning up failed/partial > > deploys). And it may not solve the availability problem at all, since > > we still have two points of failure. > > > > I think a better solution may be to have multiple mirrors of the repo, > > either run by concerned citizens or maintained by the clojars staff. I > > know some folks in the community already run internal caching proxies > > or rsynced mirrors (and are probably chuckling knowingly at those of > > us affected by the outage), but those proxies don't really help those > > in the community that don't have that internal infrastructure. And I > > don't want to recommend that everyone set up a private mirror - that > > seems like a lot of wasted effort. > > > > Ideally, it would be nice if we had a turn-key tool for creating a > > mirror of clojars. We currently provide a way to rsync the repo[2], so > > the seed for a mirror could be small, and could then slurp down the > > full repo (and could continue to do so on a schedule to remain up to > > date). We could then publish a list of mirrors that the community > > could turn to in times of need (or use all the time, if they are > > closer geographically or just generally more responsive). Any deploys > > would still need to hit the primary server, but deploys are are > > dwarfed by reads. > > > > There are a few issues with using mirrors: > > > > (1) security - with artifacts in more places, there are more > > opportunities to to introduce malicious versions. This could be > > prevented if we had better tools for verifying that the artifacts > > are signed by trusted keys, and we required that all artifacts be > > signed, but that's not the case currently. But if we had a regular > > process that crawled all of the mirrors and the canonical repo to > > verify that the checksums every artifact are identical, this could > > actually improve security, since we could detect if any checksum > > had been changed (a malicious party would have to change the > > checksum of a modified artifact, since maven/lein/boot all confirm > > checksums by default). > > > > (2) download stats - any downloads from a mirror wouldn't get > > reflected in the stats for the artifact unless we had some way to > > report those stats back to clojars.org. We currently generate the > > stats by parsing the nginx access logs, mirrors could do the same > > and report stats back to clojars.org if we care enough about > > this. We don't get stats from the existing private mirrors, and > > the stats aren't critical, so this may be a non-issue, and > > definitely isn't something that has to be solved right away, if > > ever. > > > > The repo is just served as static files, so I think a mirror could > > simply be: > > > > (1) a webserver (preferably (required to be?) HTTPS) > > (2) a cronjob that rsyncs every N minutes > > > > And the cronjob would just need the rsync command in [2], so, to get > > this started, we just need: > > > > (1) linode to be up > > (2) people willing to run mirrors > > > > (I would say "(3) add a page to the wiki on how to use a mirror", but > > that would destroy the symmetry of all the other 2-item lists in this > > message) > > > > And it would be nice to have the process in place to verify checksums > > soon - that would actually be a boon if we had another linode > > compromise[3]. > > > > Does anyone see any issues with this plan - I'm curious if there are > > security implications (or anything else) that I haven't thought of? > > > > Are you willing to run a mirror? > > > > One issue that comes to mind is if we do decide to move the repo to a > > block store, it actually makes mirroring more difficult unless we keep > > a copy of the repo on disk on clojars.org as well. But I would like to > > have mirrors in place as soon as possible, and worry about that later. > > > > - Toby > > > > [1]: https://github.com/clojars/clojars-web/issues/433 > > [2]: > > https://github.com/clojars/clojars-web/wiki/Data#rsync-the-whole-classic-repository > > [3]: > > https://groups.google.com/d/msg/clojars-maintainers/uAVJVwRAnSU/WISqQn5E9KIJ > > -- > You received this message because you are subscribed to the Google > Groups "Clojure" group. > To post to this group, send email to clojure@googlegroups.com > Note that posts from new members are moderated - please be patient with your > first post. > To unsubscribe from this group, send email to > clojure+unsubscr...@googlegroups.com > For more options, visit this group at > http://groups.google.com/group/clojure?hl=en > --- > You received this message because you are subscribed to the Google Groups > "Clojure" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to clojure+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
