[tor-relays] Ensure servers with >2 relays per IP do not get hit by rate limiting firewalls (by other relays)

[mail--- via tor-relays]

Hi Tor operators,

Some of us took/will take advantage of the increase in allowed Tor relays per 
IPv4 address[1] to reduce costs for running Tor relays. This change will result 
in more relays sharing the same source IP address than before, which means 
other relays using rate limits on their ORPorts might need to make sure they do 
not
unintentionally block relay to relay connectivity.

Many relay operators deploy TCP SYN rate limiting packet filters theses days 
due to the ongoing DDoS issues. With the increase in Tor relays per IPv4 
address, there might be more (new) connection coming from the same source IP.

If you have strict TCP SYN rate limits per source IP, please ensure that this 
change does not result in blacklisting relay to relay traffic. You could for 
example whitelist relay IP addresses or have less strict rate limits for them.

Thanks for reading,

https://applied-privacy.net
https://nothingtohide.nl

[1] https://gitlab.torproject.org/tpo/core/tor/-/issues/40744
___
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays

Re: [tor-relays] Relay requirements

[mail--- via tor-relays]

Hi,

> No network experience but already running 2 TOR instances: 1 TOR service + 1 
> bridge.

Very nice! You're asking some great questions and I'll try to answer as many as 
I can :).

> My preferred choice at the moment is [3] where I would like to have 3 relays 
> (one of each type; 3.5 GB RAM needed).

In most cases guard and middle relays are the same thing (they are both at the 
same time in most cases). So in your case, instead of hosting one guard, one 
middle and one exit, it probably makes more sense to host two guard/middle 
relays and one exit relays (or one guard/middle and two exits, which probably 
is more useful tot he network).

> The [TOR-friendly] ISP I'm looking at have these 4 offers available:

Are these monthly bandwidth allowances tx/rx (send + receive traffic) combined 
or based on the highest of the two? In the former, you would only have half of 
it effectively since Tor's traffic is more or less symmetrical in nature. 
That's good to know beforehand.

I'm running a few exit relays on a basic VPS with one similar core and 2GB of 
ram and those move ~23 TB a month (rx+tx combined). Do note that this VPS 
provider has extremely poor CPU performance that doesn't even come remotely 
close to proper virtualized CPU performance (let alone bare metal). But this 
being the case and assuming the VPS provider you are looking at has similar or 
better CPU performance, you will easily move more traffic than the monthly 
allowance on any of their offerings. In other words: without a increase in 
monthly allowance your CPU will either be idle for a large portion of the month 
(e.g. AccountingMax) or will be idling a lot (e.g. BandwidthRate).

If you think that's undesirable, you might inquiry about increasing the monthly 
allowance beforehand or look for another provider. But on the other hand, it 
will still be useful to the Tor network and you will also be able to learn and 
get experience from such a setup (which is one of your goals) so there also is 
nothing wrong with going with one of the options you mentioned :-).

> From relay requirements 
> , my 
> understanding is that [1] is what is expected for a guard or middle relay and 
> [2] for an exit relay.

Although these memory requirements are fine to use as a 'bare minimum', it will 
change a lot when multiple relays are being hosted. For example, if I would 
take the requirement page literally I would need 60 GB of ram for my Tor relays 
(40 per server), but in reality they never come even close to 30% of that. 
Running a few relays on 1-2 CPU cores with limited RAM is fine, but just keep 
an eye on it and don't run other memory intensive stuff on the server (like DNS 
query caching, which can take quite some RAM as well).

> I guess my fundamental question is what is the advantage of running multiple 
> relays of the same type, on the same server? I see some operators running 
> dozens of them, all in the same country, same ISP. Why not just a single 
> relay running with a large capacity?

Sadly the current Tor relay software can't use multiple cores effectively, so 
it won't scale at all on multi thread CPU architectures. Ideally I would run 
one relay per server, but because of this limitation in Tor's architecture the 
only other way to utilize the CPU to its full potential is by running many 
relays.

> Also, is there a requirement for the number of relays per core? (Maybe this 
> is the answer to my question.)

Not really, unless you're really optimizing for max. bandwidth per CPU. You 
could run one relay per physical core, one relay per thread or even multiple 
relays per thread on modern hardware and it wouldn't matter much as long as 
there is enough headroom for the memory overhead of multiple processes. In my 
experience, one relay per CPU core isn't enough to saturate a modern CPU. Even 
running one relay per thread (in the case of SMT) often isn't enough.

But in a lot of cases people run Tor relays on type 1 or type 2 hypervisors 
with OS virtualization (typical VPS providers), and then the OS that you use 
for Tor won't have control over the CPU threads and hence can't optimize this 
meaningfully. In that case it depends more on the amount of RAM (which is often 
limited on cheap VPS) you have available for running additional relays.

> I know my bridge is currently keeping one core of my 2-core server constantly 
> under load.

If your monthly bandwidth allowance, bandwidth and electricity consumption 
allow for it, then I would always advise to run more relays to use the CPU more 
effectively. It's a waste of good CPU cycles to not saturate it! ^_^

Hopefully this was useful to you and don't hesitate to ask more or follow-up 
questions!

Cheers,

NTH

Mar 7, 2023, 08:14 by sydney+...@liaison.club:

> Newbie here.  No network experience but already running 2 TOR instances: 1 
> TOR service + 1 bridge.
>
> I would like to "upgrade" to TOR relays but have a

Re: [tor-relays] AROI not getting verified

[mail--- via tor-relays]

Hi chani,

When I try to access 
https://vagabyte.com/.well-known/tor-relay/rsa-fingerprint.txt I get a 404 Not 
Found. So my guess is that this is (at least one of the) problem(s). See 
https://nusenu.github.io/tor-relay-well-known-uri-spec/#well-knowntor-relayrsa-fingerprinttxt
 as well for more information.

Cheers,

tornth


Jun 14, 2023, 09:21 by tor-relays@lists.torproject.org:

> Hey all,
>
> I'm running a relay and wanted to get AROI setup. I tried to debug it now for 
> quite some time, but I just cant get it to get verified. I checked that my 
> contact info matches the well known file, and if the well known file is 
> reachable. I also cant find anyone except myself, querying the well known 
> file on my webserver for the last few days. Am I missing something?
>
> relay fingerprint: CA1D49840F37E0C9D5F1770A5F033D80EB6AF369
>
> ___
> tor-relays mailing list
> tor-relays@lists.torproject.org
> https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
>

___
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays

Re: [tor-relays] Setting up HTML page for exit relay

[mail--- via tor-relays]

Hi,

> If I want to serve an HTML page for my exit node do I need Apache2/nginx or 
> can I just modify my torrc?

You don't need a dedicated webserver as long as unencrypted HTTP is acceptable. 
You can read more about the default HTML exit notice for Tor exit relays here: 
https://community.torproject.org/relay/setup/exit/. To quote:

"To make it even more obvious that this is a Tor exit relay you should serve a 
Tor exit notice HTML page.Tor can do that for you: if your DirPort is on TCP 
port 80, you can make use of tor's DirPortFrontPage feature to display an HTML 
file on that port.This file will be shown to anyone directing their browser to 
your Tor exit relay IP address."

And a sample HTML page can be found here: 
https://gitlab.torproject.org/tpo/core/tor/-/raw/HEAD/contrib/operator-tools/tor-exit-notice.html.

But this doesn't scale well on many relays and doesn't provide TLS, so if you 
run many relays and/or want TLS I'd advise to still use a dedicated webserver 
(Apache, Nginx, Caddy etc.) that redirects to a single page on your Tor domain. 
For example, my IP addresses redirect to https://nothingtohide.nl/tor-relay/.

Do note though that adding dedicated webservers to a OS that runs Tor also adds 
attack surface (both for hacking/breaching attempts and DDoS) and complexity. 
Make sure to harden and maintain it properly. For example with Apache the 
following setup might be acceptable:

- Run it as a dedicated user
- Disable ServerSignature
- Production mode for ServerTokens
- No mod_rewrite but basic Redirect 301 / https:// 
domain.tld/tor-relay 

- Disable any other unneeded modules
- Disable directory listing
- Disable access to all directories
- HSTS and proper security headers
- Use options such as -ExecCGI, -FollowSymlinks (or +SymLinksIfOwnerMatch if 
you really need it), -Includes etc. etc.

And if DDoS becomes too big of a problem, you might also want to look in 
mitigation for that as well.

Cheers and good luck!

tornth


Jan 29, 2024, 09:13 by tor-relays@lists.torproject.org:

>
> If I want to serve an HTML page for my exit node do I need Apache2/nginx or 
> can I just modify my torrc?
>
>

___
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays

Re: [tor-relays] Way to be notified when relay goes offline?

[mail--- via tor-relays]

Hi,

> Is there a way to be notified when a relay goes offline? Thanks.

For a few relays you could make an account on weather.torproject.org and add 
your email address and relays. But pretty much any other remote monitoring tool 
will suffice as well.

Cheers,

tornth


Feb 4, 2024, 22:30 by keifer@gmail.com:

> Hi,
>
> So my relay at  > 
> https://metrics.torproject.org/rs.html#details/79E3B585803DE805CCBC00C1EF36B1E74372861D>
>    went offline for a few days, and I was unaware. Is there a way to be 
> notified when a relay goes offline? Thanks.
> --Keifer
>

___
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays

Re: [tor-relays] Request for Feedback on Relay Node Update Management

[mail--- via tor-relays]

Hi Allessandro,

My two cents:
I deliberately don't use automatic updates to improve relay stability and only 
update when it's necessary. And this applies to any updates including the 
FreeBSD/OpenBSD kernel (which requires a reboot as well). Not all updates are 
security patches or are relevant enough to immediately effectuate. Sometimes 
they can go ~60 days without needing a Tor restart.

But your mileage may vary. For example some operators compile Tor daily and 
then a restart may be required more often. I do feel something is wrong in your 
configuration though. Tor certainly has updates occasionally, but not every few 
hours. Also in Tor Project's Debian repo[1] it looks like the last update was 
from 22-03 (and not a few hours ago). So you may want to check your apt 
configuration. I don't use Debian/Linux so someone else could maybe chime in to 
help you with setting that up properly.

Regards,

tornth

[1] https://deb.torproject.org/torproject.org/dists/


Mar 26, 2024, 11:03 by tor-relays@lists.torproject.org:

> Dear Tor-Relays Mailing List Members,
>
> I hope this email finds you well. I'm reaching out to share some observations 
> and pose some questions regarding the management of relay node updates, 
> particularly concerning their impact on stability and security of the service 
> provided.
>
> Recently, I've noticed an interesting pattern with my relay node (ID: 
> 47B72187844C00AA5D524415E52E3BE81E63056B [1]). I've followed TorProject's 
> recommendations [2] and configured automatic updates, which has led to 
> frequent restarts of the node to keep the Tor software up-to-date. While this 
> practice ensures high security by keeping the software updated, it seems to 
> compromise the stability of the node itself. The Uptime value of my node has 
> remained at a maximum of a few hours.
>
> This situation has prompted me to reflect on what might be the best strategy 
> to adopt. On one hand, frequent updates ensure optimal security, while on the 
> other hand, continuous restarts may affect the user experience for those 
> relying on the node's stability for their Tor activities.
>
> As such, I'd like to pose some questions to the community to gather feedback 
> and assess best practices:
>
> 1. In your opinion, is it preferable to maintain automatic updates to ensure 
> maximum security, even if frequent restarts may compromise the node's 
> stability?
> 2. Or would it be more sensible to adjust the update frequency, perhaps 
> performing them once or twice a week, to ensure greater stability of the node 
> without excessively compromising security?
> 3. Have you had similar experiences with your relay nodes? How have you 
> addressed this challenge and what were the outcomes?
>
> Thank you in advance for your time and cooperation.
>
> Best regards,
> Aleff.
>
> [1] 
> https://metrics.torproject.org/rs.html#details/47B72187844C00AA5D524415E52E3BE81E63056B
> [2] https://community.torproject.org/relay/setup/guard/debian-ubuntu/updates/
>
> ---
>
> Browse my WebSite: aleff-gitlab.gitlab.io
> Use my PGP Public Key: pgp.mit.edu/pks/lookup?op=get&search=0x7CFCE404A2168C85
> Join to support:
> - Free Software Foundation! (my.fsf.org/join?referrer=6202114)
> - Electronic Frontier Foundation! (eff.org)
> - Tor-Project (torproject.org)
> - Signal (signal.org)
>

___
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays

[tor-relays] Quick Assist Technology and Tor?

[mail--- via tor-relays]

Hi o/,

During the Tor Operator Meetup I asked about Quick Assist Technology (QAT) 
support and was asked to bring it to the tor-relays mailing list so the network 
team can take a look at the question.

In 2025 we're going to build one or more new servers and we're looking in to 
optimizing the performance per watt ratio since some of our current servers are 
rather power hungry ;-).

I'm wondering whether QAT works for Tor to offload compression, hashing and 
encryption. In theory, looking at the nature of Tor (a lot encryption), this 
could result in a huge performance boost of 100-300% (based on other hashing, 
cryptographic and compression offload benchmarks). Support for QAT also has 
improved considerably over the years so many programs/workloads already work 
nicely with it, but I'm not sure about Tor.

It looks like Tor uses [1] RSA-1024, AES-CBC, AES-CTR, Curve25519, Ed25519, 
SHA1, AES256, AES3-256. Most (no Curve- and Ed25519) should in theory also work 
with QAT [2] (although I guess only a few would impact performance 
significantly when offloaded). But the question is: does it really work? If 
not, what would be needed to make it work? Are there Tor operators who already 
utilize QAT? Does the Network Team have some insight in to this? :)

Some of the potential advantages when comparing a similar amount of traffic:
- Lower power consumption (much cheaper to run in expensive European countries).
- Less CPU cycles required (= cheaper CPUs).
- Less heat/cooling required (easier to put in distribution boxes and other 
small places).
- Smaller physical footprint (easier to put in distribution boxes and other 
small places).
- Alleviates some of the issues and challenges caused by Tor's single threaded 
architecture by effectively increasing bandwidth per CPU core considerably.

With regards,

tornth

[1] 
https://spec.torproject.org/tor-spec/preliminaries.html?highlight=cipher#ciphers
[2] 
https://www.intel.com/content/www/us/en/support/articles/93843/technologies/intel-quickassist-technology-intel-qat.html

___
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays

Re: [tor-relays] Quick Assist Technology and Tor?

[mail--- via tor-relays]

Hi,

Thanks for your reply Alex! That mailing list thread is great and contains 
quite some relevant pointers.

Some of my current Tor hardware (Intel C3958) is actually QAT compatible (it's 
the one mentioned even) so based on the information in the thread I activated 
to experiment:
- Kernel TLS
- Kernel TLS for AES-CBC
- QAT kernel module
- QAT itself
- HardwareAccel 1 in torrc

I'll monitor the differences, although I doubt it will be this simple. I also 
have to look in to how I can even verify that QAT and/or kTLS are being used. 
It's a new territory for me :). Looks like there are stats available for kTLS 
at least in sysctl.

In between the previous mailing list thread and this one, support for OpenSSL 
3.0 has been greatly improved it seems by the way.

Also about RSA-1024: Intel documentation says it's "Opt-in" [1]. Any idea how 
one can opt-in? I can't find any sysctl parameters for this in FreeBSD.

Cheers and thanks again,

tornth

[1] 
https://www.intel.com/content/www/us/en/support/articles/93843/technologies/intel-quickassist-technology-intel-qat.html




Jun 24, 2024, 21:53 by tor-relays@lists.torproject.org:

> Excerpts from mail--- via tor-relays's message of June 22, 2024 5:14 pm:
>
>> Hi o/,
>>
>> During the Tor Operator Meetup I asked about Quick Assist Technology (QAT) 
>> support and was asked to bring it to the tor-relays mailing list so the 
>> network team can take a look at the question.
>>
>> In 2025 we're going to build one or more new servers and we're looking in to 
>> optimizing the performance per watt ratio since some of our current servers 
>> are rather power hungry ;-).
>>
>> I'm wondering whether QAT works for Tor to offload compression, hashing and 
>> encryption. In theory, looking at the nature of Tor (a lot encryption), this 
>> could result in a huge performance boost of 100-300% (based on other 
>> hashing, cryptographic and compression offload benchmarks). Support for QAT 
>> also has improved considerably over the years so many programs/workloads 
>> already work nicely with it, but I'm not sure about Tor.
>>
>> It looks like Tor uses [1] RSA-1024, AES-CBC, AES-CTR, Curve25519, Ed25519, 
>> SHA1, AES256, AES3-256. Most (no Curve- and Ed25519) should in theory also 
>> work with QAT [2] (although I guess only a few would impact performance 
>> significantly when offloaded). But the question is: does it really work? If 
>> not, what would be needed to make it work? Are there Tor operators who 
>> already utilize QAT? Does the Network Team have some insight in to this? :)
>>
>> Some of the potential advantages when comparing a similar amount of traffic:
>> - Lower power consumption (much cheaper to run in expensive European 
>> countries).
>> - Less CPU cycles required (= cheaper CPUs).
>> - Less heat/cooling required (easier to put in distribution boxes and other 
>> small places).
>> - Smaller physical footprint (easier to put in distribution boxes and other 
>> small places).
>> - Alleviates some of the issues and challenges caused by Tor's single 
>> threaded architecture by effectively increasing bandwidth per CPU core 
>> considerably.
>>
>> With regards,
>>
>> tornth
>>
>> [1] 
>> https://spec.torproject.org/tor-spec/preliminaries.html?highlight=cipher#ciphers
>> [2] 
>> https://www.intel.com/content/www/us/en/support/articles/93843/technologies/intel-quickassist-technology-intel-qat.html
>>
>> ___
>> tor-relays mailing list
>> tor-relays@lists.torproject.org
>> https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
>>
>
> I previously answered this at 
> https://lists.torproject.org/pipermail/tor-relays/2022-April/020495.html.
> In principle, it should work if you set HardwareAccel 1. However, based 
> on my profiling, the actual AES encryption doesn't use that much CPU 
> when using regular AES instructions. I couldn't find any independent QAT 
> benchmarks from an internet search, but 
> https://calomel.org/aesni_ssl_performance.html says AES-NI can reach 
> over 1 GB/s per core, which is far more than Tor can use.
>
> Cheers,
> Alex.
> ___
> tor-relays mailing list
> tor-relays@lists.torproject.org
> https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
>

___
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays

Re: [tor-relays] Hardware sizing for physical exit node

[mail--- via tor-relays]

Hi,

Imo mobile chips with mostly low-performance 'efficient cores' are a suboptimal 
fit for running Tor at scale. A couple of reasons:

- Performance between the different core types (efficient vs. performance) is 
very different. You would need to lock Tor processes to specific cores and 
accept that you can only run 2 fast relays (and 8 slow ones), or have 10 of 
them capped at the efficient core's performance profile (because the Tor 
processes would get swapped around all the time).
- These CPUs aren't build for sustained loads so their real world sustained 
clock speed is significantly lower than their turbo speed, hampering their 
performance (especially compared to benchmarks).

That being said, especially the 2 performance cores are quite decent and should 
be able to push some nice amount of Tor traffic (with some help from the 
efficient cores). My guess is that the i7 would not saturate a 1 Gb/s link 
(probably 600-650 Mb/s when utilizing all cores effectively) if your relays 
have the guard flag. Middle only relays would push more traffic. So with some 
quirks/workarounds, I think you could certainly make these CPUs work (but I 
still wouldn't recommend them).

Do you need SFP or SFP+? Both aren't a feature of the CPU (except for some 
embedded SoCs) and are generally easily/cheaply available. For example the 
Intel X520-DA2 and Mellanox ConnectX-3 are extremely affordable second hand or 
refurbished and both of the CPUs have enough PCI-e lanes to run them at full 
speed.

Regarding i5 vs i7: the i7 effectively only has a higher sustained clock speed 
and marginally better GPU (which is irrelevant for Tor) but otherwise is pretty 
much identical. If the difference would be small (i.e. ~20 euro/dollar) I would 
go for the i7 but otherwise I wouldn't bother. Your mileage may vary of course 
:-).

About RAM: Tor is (for the most part) single threaded so quite a few operators 
run one Tor process per physical core or per thread. In your case this would 
mean 10 or 12 Tor relays. Especially guard relays can go out of memory very 
easily nowadays because of all sorts of attacks, so I wouldn't recommend less 
than 4 GB of available RAM per relay (so 40-48 GB for Tor + X GB for 
OS/networking/routing/firewall/services/etc.). 90% of the time you don't need 
it, but when you do need it it makes sure Tor doesn't crash, your kernel 
doesn't panic and your server stays responsive (e.g. ssh) etc. You could also 
run less relays (or limit their bandwidth) and keep it at 32 GB of RAM, but 
then you would have a harder time to saturate the uplink.

Cheers,

tornth


Jul 10, 2024, 11:30 by tor-relays@lists.torproject.org:

> Hi everyone,
> we are planning to get some hardware to run a physical Tor exit node, 
> starting with a 1Gbps dedicated, unmetered uplink (10Gbps downlink). We will 
> also route a /24 on it, so we will have large availability of addresses to 
> run multiple instances. We have been running a few exit nodes so far, but 
> never on our own hardware.
>
> Which is the bandwith limit per core/Tore instance? Or what can we expect to 
> be the bottleneck?
>
> Due to some other requirements we need for some experiments (SFP ports, 
> coreboot support, etc) we can mainly choose between these 2 CPUs:
>  Intel i5-1235U
>  Intel i7-1255U
>
> The cost between the two models is significant enough in our case to pick the 
> i7 only if it's really useful.
>
> In both cases with 32GB of DDR5 RAM (we can max to 64 if needed, but is it?).
>
> Should this allow us to saturate the uplink?
>
> To summarize, with this bandwith, this hardware and a /24 how many Tor exit 
> nodes should be ideal to run considering that each of them could have their 
> own address?
>
> Thanks!
> ___
> tor-relays mailing list
> tor-relays@lists.torproject.org
> https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
>

___
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays

Re: [tor-relays] a couple noob questions

[mail--- via tor-relays]

Hi,

To give some answers:
Yes this is completely normal! It can take 2-3 months for a new guard relay to 
finally be 'ramped up'. Here you can read more about it: 
https://blog.torproject.org/lifecycle-of-a-new-relay/. Exit relays ramp up much 
faster in terms of bandwidth, but come with their own headaches/challenges vs. 
guard relays.
This is normal as well. I don't know speedtest-cli specifically, but I imagine 
it's a regular tcp/udp speedtest while Tor traffic is more likely to be CPU 
limited instead of network throughput limited. Just pushing some udp/tcp 
traffic with a speedtest is a great to find the limitations of your server's 
uplink (for example 1000 Mb/s down but 200 Mb/s up on certain VPS providers, 
which means Tor can use up to 200 Mb/s in practice). But the real bandwidth Tor 
will be able to push is more often limited by the CPU. You should slowly see 
your CPU utilization go towards 100% in the next couple of months (if the relay 
stays stable enough).
The Tor Project doesn't ask this of relay operators. Personally I regard VPS 
hosted Tor Relays as 'not/less secure' because of the trivial access parties 
have to these relays, their keys/certificates and their traffic, but as far as 
I know the Tor community/Project considers all relays to be equal however they 
are hosted. Do note that if you're running the relay from a legal entity in the 
EEA, that your VPS provider is a processor and needs to be registered as such 
in your own registry and stuff. Tor's data being encrypted doesn't exclude it 
from the GDPR because encryption is equal to pseudonymization instead of 
anonymization in the EEA. But if you're just a person running some Tor relays 
then you don't need to bother with this :-).
Sure it is! This way everyone can chime in and learn/contribute. On IRC/Element 
there is also a #tor-relay channel for chatting with other Tor operators if 
that's what you like to use.
Looking at your relay I see nothing out of the ordinary. If anything, I feel ~2 
MB/s of traffic after a few days already is quite nice and also a bit more than 
many other guard relays expect in the first couple of days (it's a bit random 
in my experience) :). The question is whether the 7.66 MB/s advertised 
bandwidth based on the tests of the Tor network is true to your machine's 
capabilities. And you will soon find that out if you wait for the guard relay 
to age a bit.

One final thing: do note that Tor's single threaded architecture only let's it 
utilize a single CPU core (plus a bit of overhead on a second core). So let's 
say you have 4 vcores in your VPS, then you actually would need to run 2-4 Tor 
relays to be able to utilize those fully. Otherwise it will just cap at 1.2-1.4 
cores. You can run up to 8 relays per IP address. If you have more than 2 
cores, I would advice to already create 1 or more additional guard relays 
because they take months to ramp up. It's easier to delete a few redundant 
relays later than having to wait for new ones to ramp up :-).

Good luck running the relay and don't hesitate to ask more questions when you 
have them!

Cheers,

tornth



Aug 20, 2024, 07:50 by tor-relays@lists.torproject.org:

> Dear fellow relay operators,
>
> I've been hosting a tor relay on a VPS (strato) for a couple days now. I've 
> never done this before, so I have a couple of questions:
>
> Is it normal for my relay to only use up only about 2 MB/s after nearly 5 
> days of uptime despite bandwidth speed tests done with speedtest-cli 
> generally reporting significantly higher bandwidth?
> Is it normal for my relay to advertise > 7.37 MiB/s on its relay search page 
> despite > bandwidth speed tests done with speedtest-cli generally reporting 
> significantly higher bandwidth?
> Do I have to mark somewhere that I'm hosting the tor relay on a VPS (which 
> means > technically>  a company also has control over the relay)?
> Is this mailing list a good place to ask these questions? Or am I reaching 
> too many people by doing this?
>
> When I say I performed a bandwidth test, I mean I performed a bandwidth test 
> with the speedtest-cli debian package. The test generally tells me I have 
> about 217 MB/s download speed and 113 MB/s upload speed. The nickname of my 
> relay is observatory123 and its fingerprint is:
> 45431BF8AB66C673942C1E344BB520625CE5F817
>
> Kind regards,
> observatory123
>

___
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays

[tor-relays] Re: Guidance on optimal Tor relay server configurations

[mail--- via tor-relays]

Hi,

> Summary from your email - did I miss anything?

Yes, with the general disclaimer (not to sound like a lawyer) that your mileage 
may vary. For example we run everything bare metal on FreeBSD and run a mix of 
guard/middle/exit relays. Running the same workload virtualized or on another 
operating system may impact the performance/overhead (either positively or 
negatively). Also your RAM budget of 4 GB per relay may be a bit on the safe 
side, I don't think it would hurt to lower this.
> What are the primary factors that justify running up to two Tor relays per 
>physical core (leveraging SMT) versus a one-to-one mapping?

Tor relays sadly don't scale well. They fluctuate on a daily basis (the Tor 
network as a whole does) and even their general utilization is kind of 
unpredictable. So I think there are two approaches to this:

1) Run 1 relay per physical core, accepting that your CPU will idle a large 
amount of the time (50%+ in our case).

2) Run multiple Tor relays per physical core until you saturate 90-95% of your 
CPU cycles, accepting additional system overhead/congestion.

There is no right or wrong here. In our case we went with running multiple 
relays per core because we want to utilize the (very expensive) hardware we run 
on as much as possible. Every CPU cycle not spent on privacy enhancing services 
is a wasted CPU cycle from our point of view ;).

> Is one-to-one mapping of Tor relay to core/thread the most compute- and 
>system-efficient approach?

Yes, this should lower the amount of congestion (interrupts and stuff). In this 
sense it can also be beneficial to lock your NIC/irq threads to specific cores.

> Are the clock speeds you listed base or turbo numbers (3.4, 2.25 and 2.0 Ghz)?

Base indeed. No CPU is able to consistently maintain their turbo speed on this 
many cores. When all cores are utilized, the base speed pretty much is the max 
speed in practice.

> From your real world scenario #2 and advice for the "fastest/best hardware", 
>would this type of server work well for a $20k budget?

Looks like a capable server. That CPU looks powerful enough but keep in mind 
that it has a rather low clockspeed, so you will be running many medium speed 
relays. Nothing wrong with that since CPUs with this many cores simply 
don't/can't have high base clocks. Also I think 512 GB of RAM would be enough 
unless you run a *lot* of relays on it (which may be a viable strategy to 
utilize your CPU fully).

Just a note: in my experience the Epyc platform (especially when self-build) 
provides a bit more bang for your buck. For example a AMD Epyc 9969 with 192 
cores/384 threads@2.25 Ghz baseclock will probably outperform the Intel 6980P 
considerably (for Tor workloads at least), while being much cheaper (listing 
price at least). But of course this greatly depends on where you buy the server 
or parts so your mileage may vary. When I look around here locally a complete 
self-build system with the 192 core Epyc, 512 GB RAM and a 100 Gb/s NIC would 
cost ~12k excluding VAT before any tax benefits. But your proposed server will 
work perfectly fine as well so if you prefer a brand, go for it :).
> Assuming one relay per core/thread, would this setup be capable of saturating 
>40 Gbps, given that 10 Gbps typically saturated with ~31.5 physical cores + 
>~31.5 SMT threads at 2 Ghz and thus 256 relays vs ~63 relays translates 
>roughly to 4×10 Gbps = 40 Gbps?

Assuming you get the Tor relays to saturate their cores, yes. That CPU should 
be able to push 40 Gb/s of Tor traffic. Our ~2019 64 core/128 thread Epyc 
pushes 10 Gb/s of Tor traffic on a bit less than half it's capacity. And your 
CPU is newer (better IPC hopefully if Intel finally stepped up their game since 
2019), has much more cache and runs on DDR5 while having a bit lower base 
clock. So it should perform at least similar but probably better than ours.


Do you have the network capacity covered already? If you plan to do 40 Gb/s, 
then you also need enough peers/upstream capacity. The required networking 
equipment and connections themselves for this can also be costly.
Cheers,

tornth

Feb 20, 2025, 08:42 by t...@1aeo.com:

> Excellent information, especially the real world scenarios! Exactly what I 
> was looking for!
>
> Summary from your email - did I miss anything?
>
> To saturate 10 Gbps connection:
> 1) IPv4 Allocation: Use between 5 and 20. Much lower than 256 in a /24!
> 2) Tor Relay Count:  Run roughly ~40 to ~150, depending on CPU clock speed, 
> i.e. faster clock, fewer relays needed.
> 3) CPU Utilization: 1 Tor relay per physical core preferred but okay to scale 
> to 1 Tor relay per threads/SMT as well, up to 2x Tor relays per core/thread
> 4) RAM requirements: Maintain a 4:1 RAM-to-core/relay ratio (4GB per 
> core/relay), including extra 32GB per server to cover DoS, OS, networking, 
> etc. overheads
>
> In general, some ideals but not required:
> CPU clock speed: Higher CPU clock speed, better relay performance
>

[tor-relays] Re: Guidance on optimal Tor relay server configurations

[mail--- via tor-relays]

Hi,

Many people already replied, but here are my (late) two cents.

> 1) If a full IPv4 /24 Class C was available to host Tor relays, what are some 
>optimal ways to allocate bandwidth, CPU cores and RAM to maximize utilization 
>of the IPv4 /24 for Tor?

"Optimal" depends on your preferences and goals. Some examples:

- IP address efficiency: run 8 relays per IPv4 address.
- Use the best ports: 256 relays (443) or 512 relays (443+80).
- Lowest kernel/system congestion: 1 locked relay per core/SMT thread 
combination, ideally on high clocked CPUs.
- Easiest to manage: as few relays as possible.
- Memory efficiency: only run middle relays on very high clocked CPUs (4-5 Ghz).
- Cost efficiency: run many relays on 1-2 generations old Epyc CPUs with a high 
core count (64 or more).

There are always constraints. The hardware/CPU/memory and bandwidth/routing 
capability available to you are probably not infinite. Also the Tor Project 
maximizes bandwidth contributions to 20% and 10% for exit relay and overall 
consensus weight respectively.

With 256 IP addresses on modern hardware, it will be very hard to not run in to 
one of these limitations long before you can make it 'optimal'. Hardware wise, 
one modern/current gen high performance server only running exit relays will 
easily push enough Tor traffic to do more than half of the total exit bandwidth 
of the Tor network.

My advice would be:
1) Get the fastest/best hardware with current-ish generation CPU IPC 
capabilities you can get within your budget. To lower complexity with keeping 
congestion in control, one socket is easier to deal with than a dual socket 
system.

(tip for NIC: if your switch/router has 10 Gb/s or 25 Gb/s ports, get some of 
the older Mellanox cards. They are very stable (more so than their Intel 
counterparts in my experience) and extremely affordable nowadays because of all 
the organizations that throw away their digital sovereignty and privacy of 
their employees/users to move to the cloud).

3) Start with 1 Tor relay per physical core (ignoring SMT). When the Tor relays 
have ramped up (this takes 2-3 months for guard relays) and there still is 
considerable headroom on the CPU (Tor runs extremely poorly at scale sadly, so 
this would be my expectation) then move to 1 Tor relay per thread (SMT 
included).

(tip: already run/'train' some Tor relays with a very limited bandwidth (2 MB/s 
or something) parallel to your primary ones and pin them all to 1-2 cores to 
let them ramp up in parallel to your primary ones. This makes it *much* less 
cumbersome to scale up your Tor contribution when you need/want/can do that in 
the future).

4) Assume at least 1 GB of RAM per relay on modern CPUs + 32 GB additionally 
for OS, DNS, networking and to have some headroom for DoS attacks. This may 
sound high, especially considering the advice in the Tor documentation. But on 
modern CPUs (especially with a high clockspeed) guard relays can use a lot more 
than 512 MB of RAM, especially when they are getting attacked. Middle and exit 
relays require less RAM.

Don't skimp out on system memory capacity. DDR4 RDIMMs with decent clockspeeds 
are so cheap nowadays. For reference: we ran our smaller Tor servers 
(16C@3.4Ghz) with 64 GB of RAM and had to upgrade it to 128 GB because during 
attacks RAM usage exceeded the amount available and killed processes.

5) If you have the IP space available, use one IPv4 address per relay and use 
all the good ports such as 443. If IP addresses are more scarce, it's also not 
bad to run 4 or 8 relays per IP address. Especially for middle and exit relays 
the port doesn't matter (much). Guard relays should ideally always run on a 
generally used (and generally unblocked) port.


> 2) If a full 10 Gbps connection was available for Tor relays, how many CPU 
>cores, RAM and IPv4 addresses would be required to saturate the 10 Gbps 
>connection?

That greatly depends on the CPU and your configuration. I can offer 3 
references based on real world examples. They all run a mix of 
guard/middle/exit relays.

1) Typical low core count (16+SMT) with higher clockspeed (3.4 Ghz) saturates a 
10 Gb/s connection with ~18.5 physical cores + SMT.
2) Typical higher core count (64+SMT) with lower clockspeed (2.25 Ghz) 
saturates a 10 Gb/s connection with ~31.5 physical cores + SMT.
3) Typical energy efficient/low performance CPU with low core count (16) with 
very low clockspeed (2.0 Ghz) used often in networking appliances saturates a 
10 Gb/s connection with ~75 physical cores (note: no SMT).

The amount of IP addresses required also depends on multiple factors. But I'd 
say that you would need between the amount and double the amount of relays of 
the mentioned core+SMT count in order to saturate 10 Gb/s. This would be 37-74, 
63-126 and 75-150 relays respectively. So between 5 and 19 IPv4 addresses would 
be required at minimum, depending on CPU performance level.
RAM wise the more relays you run, the more RAM overhead you