I concur that this is a serious problem and at the moment anyone can censor any hidden service at will. These attacks use little bandwidth, and seem to involve each request having a new rendezvous for each attempt, using lots of resources. can't imagine a fix myself at this point.
[email protected] wrote: > On 2014-12-02 21:45, l.m wrote: >>> Perhaps the new implementation of the hidden services will be better >> ? >>> How is it going ? >> >> I don't see anything in the improvements suggested for hidden services >> that would help this situation. Though I would be grateful for being >> corrected. >> >> First, I just want to say I only meant sheep(s) to emphasize that you >> don't know how many black sheep you have participating. I mentioned >> the part about this potentially being an attack external to Tor out of >> concern for your participation in a de-anonymizing attack on your >> hosted HS. I see your HS's are offline while you troubleshoot this so >> that's good. Next, I'm confused by what you describe. Sorry I deleted >> your previous email so I may repeat some things you already said. >> >> - no evidence of any HS being flooded from logs. No evidence of a load >> on any particular HS. >> and >> - next to no bandwidth consumption. Does this include no processor >> use? I don't recall if that was mentioned before. >> >> - no evidence is apparent from checking REND_QUERY=HS. So no request >> to rendezvous. Might make sense given little/no traffic. >> >> - your guards go offline. This is contradictory. If the attack is >> within Tor via a HS it implies the HS traffic *reliably* makes it to >> at least your guard before you experience the symptomatic overload and >> timeout. Meaning there must be traffic you can detect. Otherwise the >> attacker would likely lose their connection to the rendezvous point >> (at least sometimes) by committing to the attack. What I mean is in >> order for this to be an attack via malicious HS it would need to >> succeed in not timing out until the traffic reaches your guard and >> server. That's two circuits that must work before failing at your >> guard. Not to mention you already tried changing the guards. It just >> seems implausible to occur reliably enough to take your server down. >> This assumes little/no traffic and no heavy cpu usage. >> >> -- Now I don't know how you setup your logging but I assume you would >> know if there was a load on any particular site or flood. I can >> suggest beefing up this part of the auditing trail. You could use >> proxy (on the same server) in your server blocks (Nginx?) for each HS >> (or in batches). Then you can use SPI to analyses the traffic of each >> proxy for a build up of use that might be causing your timeouts. >> Though I don't see the use if your logging is as good as you >> mentioned. >> >> If there's no traffic, no cpu usage, no evidence of HS load except >> your guards are timing out--I'm back to the implausible. Two circuits >> that reliably take down your guards. There *has* to be traffic on your >> side you can measure or some load indicator. Either that or the attack >> is external to Tor. On the other hand you could reply and say 'yeh >> lots of cpu use'. In which case sorry for wasting your time. If there >> is alot of cpu use the VM-partitioning solution is the best solution >> as it would guarantee at least some guards available to your other >> HS's. It also provides you more granular control over hardware >> allocation. Either way you have to assume at some point you will be >> targeted externally (from Tor) to de-anonymize your HS's. Shared >> hosting.. many HS's.. you're an eavesdropping goldmine. >> >> -- leeroy bearr > > > Hi, thanks for supporting. > > As a reminder, here is the archive for this thread: > https://lists.torproject.org/pipermail/tor-talk/2014-December/035807.html > > And that's December: > https://lists.torproject.org/pipermail/tor-talk/2014-December/thread.html > > Problem is still going on, let's recap: > > Tor goes 100% CPU only when (re)starting and publishing the HS, this > takes around 5 minutes, > after that, it uses about 1% - 5% of 1 core. > > We _are_ able to see the access.log and error.log of each virtual host, > since they are simple nginx vhosts. > We also have a script that records the access.log size on mysql, we use > that script to delete unused accounts. > We confirm that both access.log and error.log doesn't move of an inch, > basically there is no http traffic. > > Bandwidth usage is consistently around 5KB - 15KB / second. > > Currently we disabled from torrc the most visited vhosts, have a look at > this: > > http://fuckyouhotwkd3xh.onion/ (working) > http://ijaw6sx25rzose3c.onion/ (working) > http://3xsrosyv52vefh2l.onion/ (working) > http://v5uhidj456rn4cra.onion/ (working) > http://za3uovobxijq4grb.onion/ (not working) > http://4vxnpigkblvmud4i.onion/ (not working) > http://3jfbvyg5pggg7mfg.onion/ (not working) > > this is a list of the HS we are currently hosting. We tried more > addresses but in short, it looks like only 10% of the HS are working. > > Those are the Tor logs of the current session: > > Dec 03 XXX [notice] Bootstrapped 90%: Establishing a Tor circuit. > Dec 03 XXX [notice] Tor has successfully opened a circuit. Looks like > client functionality is working. > Dec 03 XXX [notice] Bootstrapped 100%: Done. > Dec 03 XXX [notice] Your Guard regar42 (XXX) is failing more circuits > than usual. Most likely this means the Tor network is overloaded. > Success counts are 120/181. Use counts are 453/453. 120 circuits > completed, 0 were unusable, 0 collapsed, and 0 timed out. For reference, > your timeout cutoff is 91 seconds. > Dec 03 XXX [warn] Your Guard regar42 (XXX) is failing a very large > amount of circuits. Most likely this means the Tor network is > overloaded, but it could also mean an attack against you or potentially > the guard itself. Success counts are 120/241. Use counts are 453/453. > 120 circuits completed, 0 were unusable, 0 collapsed, and 0 timed out. > For reference, your timeout cutoff is 91 seconds. > Dec 03 XXX [warn] Your Guard regar42 (XXX) is failing an extremely large > amount of circuits. This could indicate a route manipulation attack, > extreme network overload, or a bug. Success counts are 60/241. Use > counts are 453/453. 60 circuits completed, 0 were unusable, 0 collapsed, > and 0 timed out. For reference, your timeout cutoff is 91 seconds. > Dec 03 XXX [notice] We'd like to launch a circuit to handle a > connection, but we already have 32 general-purpose client circuits > pending. Waiting until some finish. > Dec 03 XXX [notice] Your Guard TorKuato (XXX) is failing more circuits > than usual. Most likely this means the Tor network is overloaded. > Success counts are 105/151. Use counts are 81/81. 105 circuits > completed, 0 were unusable, 0 collapsed, and 2063 timed out. For > reference, your timeout cutoff is 60 seconds. > (after 5 minutes) > Dec 03 XXX [notice] We'd like to launch a circuit to handle a > connection, but we already have 32 general-purpose client circuits > pending. Waiting until some finish. [749254 similar message(s) > suppressed in last 600 seconds] > (after another 5 minutes) > Dec 03 XXX [notice] We'd like to launch a circuit to handle a > connection, but we already have 32 general-purpose client circuits > pending. Waiting until some finish. [398 similar message(s) suppressed > in last 600 seconds] > (after 20 minutes) > Dec 03 XXX [notice] We'd like to launch a circuit to handle a > connection, but we already have 32 general-purpose client circuits > pending. Waiting until some finish. [6 similar message(s) suppressed in > last 600 seconds] > > We are running all this on a small VPS with 1GB of RAM. > The RAM finishes quickly but there is plenty of free swap. > The service was running fine with much more hidden services, on the same > hardware. > We could increase the RAM but doubt that's the problem. > > Again .. any Tor developer willing to dig into this ? > Ours is an experimental project, there is nothing business-oriented here. > If we are pushing the limits of hidden services capabilities, or really > are under a low-level Tor based attack, isn't this a good chance to > improve Tor's hidden service code > or just improve Tor's hidden service _logging_ code ? > > Perhaps we should post this on tor-dev ? -- CYRUSERV Onionland Hosting: http://cyruservvvklto2l.onion/ PGP public key: http://cyruservvvklto2l.onion/contact This email is just for mailing lists and private correspondence. Please use [email protected] for business inquiries. -- tor-talk mailing list - [email protected] To unsubscribe or change other settings go to https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-talk
