Re: Measuring DNS Performance & Graphing Logs

[Andrew Smith]

Smokeping (http://oss.oetiker.ch/smokeping/) can graph DNS response latency
via dig.

ThousandEyes (https://www.thousandeyes.com/) has some commercial options
for monitoring DNS server responsiveness, and zone performance from
different vantage points throughout the globe.



On Tue, May 19, 2015 at 12:34 PM, Zayed Mahmud 
wrote:

> Hello!
> This is my first message to NANOG's mailing list. I hope someone can help
> me.
>
> I was wondering which tool(s) can I use to measure the performance of my 3
> DNS servers (1 primary, 1 secondary, 1 solely cacheDNS)? From the stats I
> would like to know if my DNS server is serving as it should be or if any of
> it's options are set inappropriately and others alike.
>
> I looked for a while but could not find any. Any help would be highly
> appreciated. I am running bind9 on UNIX platform.
>
> Question 2) I would also like to know how can I graph my DNS logs? And how
> can I integrate it to my CACTI server as well? I couldn't find any suitable
> plugin. Any suggestion?
>
> --
>
> --
> Best Regards,
>
> *Zayed Mahmud*
>
> *Senior Core & IP Network Team,*
>
> *Banglalion Communications Limited, Bangladesh.*
>

Re: Measuring DNS Performance & Graphing Logs

[Denis Fondras]

> I was wondering which tool(s) can I use to measure the performance of my 3
> DNS servers (1 primary, 1 secondary, 1 solely cacheDNS)? From the stats I
> would like to know if my DNS server is serving as it should be or if any of
> it's options are set inappropriately and others alike.

Perhaps http://dns.measurement-factory.com/tools/dsc/ (used by AS112) can help.

Denis

Re: Low Cost 10G Router

[Eduardo Meyer]

On Tuesday, May 19, 2015, Warsaw wrote:

> > > On May 19, 2015, at 10:22, Colton Conor  > wrote:
> > >
> > > What options are available for a small, low cost router that has at
> least
> > > four 10G ports, and can handle full BGP routes? All that I know of are
> the
> > > Juniper MX80, and the Brocade CER line. What does Cisco and others have
> > > that compete with these two? Any other vendors besides Juniper,
> Brocade,
> > > and Cisco to look at?
>
> I have two ServerU L-800 boxes routing BGP and OSPF, one of those has
> 4x10G SFP+ port and the


I'm good w/ ServerU L-800 as well running BGP with FreeBSD in a location
and VyOS in a couple other.

I still dont know how much traffic Mr Conor needs to forward, if it's a 10G
base or just needs 10G ports. Without Chelsio ASICS I route 4Gb/s on this
router and I second the suggestion for L-800 if the desired forwarding rate
is around 4Gbit.

I didnt know Chelsio expansions could do forwarding directly on the card.
just heard about its low rate of interruption requests. Sounds like it
worths further investigation thanks on that..

As for L-800 I run it for over one year now doing BGP and firewalling.
Great value for a twelve hundred bucks purchase.


> It's a 1,200 USD starting cost for a very decent router which promisses to
> delivery a good pps and bps rate specially when compared to Mikrotik's CCR
> and other Cisco/Brocade routers on this same grade. Add to it a couple
> hundred extra bucks to have a very decent Chelsio T5 ASICS expansion to
> L800 chassis and you pretty much have a system that, according to Chelsion
> data sheet, promisses to delivery 27 milion packets per second filtered and
> forwarded. Pretty much Line Rate for 10G ports.
>
> I don't know about the expected 27Mpps per port, but I can confirm 4.8Mpps
> peaking / 4.2Mpps avging on my rack everyday, and for the price I pay on
> this ServerU + FreeBSD setup I can't avoid to suggest it worths pretty much
> a try!
>
> http://www.serveru.us/en/netmapl800
>
> If you buy a Chelsio card or already have it, or have it at a better price
> (sometimes we find very good 300.00 USD deals on chelsio T5, while their
> list price is ~900.00 USD) talk to 'em first, they have Chelsio front
> expansions by default but if you buy a Chelsio x8 PCIe card your own they
> need to arrange ServerU L-800 to have it perfectly fitted in their L-800
> chassis, and usually it requires rear raiser replacement in their router,
> so talk to them first... I learned it the bad way ;] bought the chelsio
> card myself and found out I could not use it, since this L-800 router comes
> with raisers for front expansions. They were gentle enough to upgrade the
> raiser for free but I had to ship the box back to Florida. So talk to
> them...
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>



-- 
===
Eduardo Meyer
pessoal: dudu.me...@gmail.com
profissional: ddm.farmac...@saude.gov.br

Re: Low Cost 10G Router

[Ray Soucy]

You're right I dropped down to the v2 for pricing reasons:

- Supermicro SuperServer 5017R-MTRF
- 4x SATA
- 8x DDR3
- 400W Redundant
- Eight-Core Intel Xeon Processor E5-2640 v2 2.00GHz 20MB Cache (95W)
- 4 x SAMSUNG 2GB PC3-12800 DDR3-160
- 2 x 500GB SATA 6.0Gb/s 7200RPM - 3.5" - Western Digital RE4 WD5003ABYZ
- Supermicro System Cabinet Front Bezel CSE-PTFB-813B with Lock and Filter
(Black)
- No Windows Operating System (Hardware Warranty Only, No Software Support)
- Three Year Warranty with Advanced Parts Replacement

FWIW I used Sourcecode as the system builder.  They've been great to work
with.

On Tue, May 19, 2015 at 4:46 PM, Joe Greco  wrote:

> > How cheap is cheap and what performance numbers are you looking for?
> >
> > About as cheap as you can get:
> >
> > For about $3,000 you can build a Supermicro OEM system with an 8-core
> Xeon
> > E5 V3 and 4-port 10G Intel SFP+ NIC with 8G of RAM running VyOS.  The pro
> > is that BGP convergence time will be good (better than a 7200 VXR), and
> > number of tables likely won't be a concern since RAM is cheap.  The con
> is
> > that you're not doing things in hardware, so you'll have higher latency,
> > and your PPS will be lower.
>
> What 8 core Xeon E5 v3 would that be?  The 26xx's are hideously pricey,
> and for a router, you're probably better off with something like a
> Supermicro X10SRn fsvo "n" with a Xeon E5-1650v3.  Board is typically
> around $300, 1650 is around $550, so total cost I'm guessing closer to
> $1500-$2000 that route.
>
> The edge you get there is the higher clock on the CPU.  Only six cores
> and only 15M cache, but 3.5GHz.  The E5-2643v3 is three times the cost
> for very similar performance specs.  Costwise, E5 single socket is the
> way to go unless you *need* more.
>
> ... JG
> --
> Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net
> "We call it the 'one bite at the apple' rule. Give me one chance [and]
> then I
> won't contact you again." - Direct Marketing Ass'n position on e-mail
> spam(CNN)
> With 24 million small businesses in the US alone, that's way too many
> apples.
>



-- 
Ray Patrick Soucy
Network Engineer
University of Maine System

T: 207-561-3526
F: 207-561-3531

MaineREN, Maine's Research and Education Network
www.maineren.net

Re: Low Cost 10G Router

[Ray Soucy]

P.S I went through HotLava Systems for the Intel-based SFP+ NICs to add to
those, http://hotlavasystems.com/ (not trying to plug; these are just hard
to find)

On Wed, May 20, 2015 at 9:08 AM, Ray Soucy  wrote:

> You're right I dropped down to the v2 for pricing reasons:
>
> - Supermicro SuperServer 5017R-MTRF
> - 4x SATA
> - 8x DDR3
> - 400W Redundant
> - Eight-Core Intel Xeon Processor E5-2640 v2 2.00GHz 20MB Cache (95W)
> - 4 x SAMSUNG 2GB PC3-12800 DDR3-160
> - 2 x 500GB SATA 6.0Gb/s 7200RPM - 3.5" - Western Digital RE4 WD5003ABYZ
> - Supermicro System Cabinet Front Bezel CSE-PTFB-813B with Lock and Filter
> (Black)
> - No Windows Operating System (Hardware Warranty Only, No Software Support)
> - Three Year Warranty with Advanced Parts Replacement
>
> FWIW I used Sourcecode as the system builder.  They've been great to work
> with.
>
> On Tue, May 19, 2015 at 4:46 PM, Joe Greco  wrote:
>
>> > How cheap is cheap and what performance numbers are you looking for?
>> >
>> > About as cheap as you can get:
>> >
>> > For about $3,000 you can build a Supermicro OEM system with an 8-core
>> Xeon
>> > E5 V3 and 4-port 10G Intel SFP+ NIC with 8G of RAM running VyOS.  The
>> pro
>> > is that BGP convergence time will be good (better than a 7200 VXR), and
>> > number of tables likely won't be a concern since RAM is cheap.  The con
>> is
>> > that you're not doing things in hardware, so you'll have higher latency,
>> > and your PPS will be lower.
>>
>> What 8 core Xeon E5 v3 would that be?  The 26xx's are hideously pricey,
>> and for a router, you're probably better off with something like a
>> Supermicro X10SRn fsvo "n" with a Xeon E5-1650v3.  Board is typically
>> around $300, 1650 is around $550, so total cost I'm guessing closer to
>> $1500-$2000 that route.
>>
>> The edge you get there is the higher clock on the CPU.  Only six cores
>> and only 15M cache, but 3.5GHz.  The E5-2643v3 is three times the cost
>> for very similar performance specs.  Costwise, E5 single socket is the
>> way to go unless you *need* more.
>>
>> ... JG
>> --
>> Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net
>> "We call it the 'one bite at the apple' rule. Give me one chance [and]
>> then I
>> won't contact you again." - Direct Marketing Ass'n position on e-mail
>> spam(CNN)
>> With 24 million small businesses in the US alone, that's way too many
>> apples.
>>
>
>
>
> --
> Ray Patrick Soucy
> Network Engineer
> University of Maine System
>
> T: 207-561-3526
> F: 207-561-3531
>
> MaineREN, Maine's Research and Education Network
> www.maineren.net
>



-- 
Ray Patrick Soucy
Network Engineer
University of Maine System

T: 207-561-3526
F: 207-561-3531

MaineREN, Maine's Research and Education Network
www.maineren.net

Re: Low Cost 10G Router

[Pavel Odintsov]

Hello!

Ray, I could suggest switch from multi physical CPU configuration to
single. Like Intel Xeon E5-1650/1660/1680 or even Xeon E3 platforms.
Because multi processor systems need really huge amount of knowledge
for NUMA configuration and PCI-E devices assignment for each NUMA.

Secondly, I could vote many times for Supermicro! :) Dell or HP are
really ugly systems for soft routers. CPU frequency tuning, PCM
debugging are real nightmare on this systems. Please beware of they!

Supermicro is very clear and do not block useful functions of platform.


On Wed, May 20, 2015 at 4:08 PM, Ray Soucy  wrote:
> You're right I dropped down to the v2 for pricing reasons:
>
> - Supermicro SuperServer 5017R-MTRF
> - 4x SATA
> - 8x DDR3
> - 400W Redundant
> - Eight-Core Intel Xeon Processor E5-2640 v2 2.00GHz 20MB Cache (95W)
> - 4 x SAMSUNG 2GB PC3-12800 DDR3-160
> - 2 x 500GB SATA 6.0Gb/s 7200RPM - 3.5" - Western Digital RE4 WD5003ABYZ
> - Supermicro System Cabinet Front Bezel CSE-PTFB-813B with Lock and Filter
> (Black)
> - No Windows Operating System (Hardware Warranty Only, No Software Support)
> - Three Year Warranty with Advanced Parts Replacement
>
> FWIW I used Sourcecode as the system builder.  They've been great to work
> with.
>
> On Tue, May 19, 2015 at 4:46 PM, Joe Greco  wrote:
>
>> > How cheap is cheap and what performance numbers are you looking for?
>> >
>> > About as cheap as you can get:
>> >
>> > For about $3,000 you can build a Supermicro OEM system with an 8-core
>> Xeon
>> > E5 V3 and 4-port 10G Intel SFP+ NIC with 8G of RAM running VyOS.  The pro
>> > is that BGP convergence time will be good (better than a 7200 VXR), and
>> > number of tables likely won't be a concern since RAM is cheap.  The con
>> is
>> > that you're not doing things in hardware, so you'll have higher latency,
>> > and your PPS will be lower.
>>
>> What 8 core Xeon E5 v3 would that be?  The 26xx's are hideously pricey,
>> and for a router, you're probably better off with something like a
>> Supermicro X10SRn fsvo "n" with a Xeon E5-1650v3.  Board is typically
>> around $300, 1650 is around $550, so total cost I'm guessing closer to
>> $1500-$2000 that route.
>>
>> The edge you get there is the higher clock on the CPU.  Only six cores
>> and only 15M cache, but 3.5GHz.  The E5-2643v3 is three times the cost
>> for very similar performance specs.  Costwise, E5 single socket is the
>> way to go unless you *need* more.
>>
>> ... JG
>> --
>> Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net
>> "We call it the 'one bite at the apple' rule. Give me one chance [and]
>> then I
>> won't contact you again." - Direct Marketing Ass'n position on e-mail
>> spam(CNN)
>> With 24 million small businesses in the US alone, that's way too many
>> apples.
>>
>
>
>
> --
> Ray Patrick Soucy
> Network Engineer
> University of Maine System
>
> T: 207-561-3526
> F: 207-561-3531
>
> MaineREN, Maine's Research and Education Network
> www.maineren.net



-- 
Sincerely yours, Pavel Odintsov

Re: Low Cost 10G Router

[Cody Grosskopf]

I haven't tried myself but some of the stuff Cumulus Linux is doing is
pretty amazing, not certain quagga can or should handle full bgp table but
you could probably get a Penguin 10gbe for less than 8k.

On Tue, May 19, 2015, 10:25 AM Colton Conor  wrote:

> What options are available for a small, low cost router that has at least
> four 10G ports, and can handle full BGP routes? All that I know of are the
> Juniper MX80, and the Brocade CER line. What does Cisco and others have
> that compete with these two? Any other vendors besides Juniper, Brocade,
> and Cisco to look at?
>

Re: Low Cost 10G Router

[Nick Hilliard]

On 20/05/2015 14:32, Cody Grosskopf wrote:
> I haven't tried myself but some of the stuff Cumulus Linux is doing is
> pretty amazing, not certain quagga can or should handle full bgp table but
> you could probably get a Penguin 10gbe for less than 8k.

quagga (or whatever RIB manager you want, e.g. bird) isn't the issue.  The
issue is that these switches have limited hardware FIB capacity and if you
attempt to put a full table on them, they won't accept it.

Nick

Re: Low Cost 10G Router

[Pavel Odintsov]

I have tried Cumulus. It's awesome! :) You definitely could run
Quagga, Bird or even ExaBGP https://github.com/Exa-Networks/exabgp and
build full feature router from 10GE switch.

On Wed, May 20, 2015 at 4:32 PM, Cody Grosskopf  wrote:
> I haven't tried myself but some of the stuff Cumulus Linux is doing is
> pretty amazing, not certain quagga can or should handle full bgp table but
> you could probably get a Penguin 10gbe for less than 8k.
>
> On Tue, May 19, 2015, 10:25 AM Colton Conor  wrote:
>
>> What options are available for a small, low cost router that has at least
>> four 10G ports, and can handle full BGP routes? All that I know of are the
>> Juniper MX80, and the Brocade CER line. What does Cisco and others have
>> that compete with these two? Any other vendors besides Juniper, Brocade,
>> and Cisco to look at?
>>



-- 
Sincerely yours, Pavel Odintsov

Re: Low Cost 10G Router

[Pavel Odintsov]

We could cut full BGP and select only important prefixes with ExaBGP.

On Wed, May 20, 2015 at 4:41 PM, Nick Hilliard  wrote:
> On 20/05/2015 14:32, Cody Grosskopf wrote:
>> I haven't tried myself but some of the stuff Cumulus Linux is doing is
>> pretty amazing, not certain quagga can or should handle full bgp table but
>> you could probably get a Penguin 10gbe for less than 8k.
>
> quagga (or whatever RIB manager you want, e.g. bird) isn't the issue.  The
> issue is that these switches have limited hardware FIB capacity and if you
> attempt to put a full table on them, they won't accept it.
>
> Nick
>
>



-- 
Sincerely yours, Pavel Odintsov

Re: Low Cost 10G Router

[Nick Hilliard]

On 20/05/2015 14:46, Pavel Odintsov wrote:
> We could cut full BGP and select only important prefixes with ExaBGP.

exabgp is rib mgmt only and doesn't program the fib.  you will need quagga
/ bird / etc for this.

Nick

Re: Low Cost 10G Router

[Pavel Odintsov]

Yes, right! But ExaBGP could receive full BGP table, drop some rules
and reflect they to Quagga which could load FIB on the Cumulus.

On Wed, May 20, 2015 at 4:53 PM, Nick Hilliard  wrote:
> On 20/05/2015 14:46, Pavel Odintsov wrote:
>> We could cut full BGP and select only important prefixes with ExaBGP.
>
> exabgp is rib mgmt only and doesn't program the fib.  you will need quagga
> / bird / etc for this.
>
> Nick
>



-- 
Sincerely yours, Pavel Odintsov

Re: Low Cost 10G Router

[Nick Hilliard]

On 20/05/2015 14:56, Pavel Odintsov wrote:
> Yes, right! But ExaBGP could receive full BGP table, drop some rules
> and reflect they to Quagga which could load FIB on the Cumulus.

or you could not bother with exabgp and do your route filtering on quagga.

Nothing wrong with exabgp, btw.  Great product.  It's just the wrong tool
for the job here.

Nick

Re: Low Cost 10G Router

[Pavel Odintsov]

Yes, you could do filtering with Quagga. But Quagga is pretty old tool
without multiple dynamic features. But with ExaBGP you could do really
any significant route table transformations with Python in few lines
of code. But it's definitely add additional point of failure/bug.


On Wed, May 20, 2015 at 4:57 PM, Nick Hilliard  wrote:
> On 20/05/2015 14:56, Pavel Odintsov wrote:
>> Yes, right! But ExaBGP could receive full BGP table, drop some rules
>> and reflect they to Quagga which could load FIB on the Cumulus.
>
> or you could not bother with exabgp and do your route filtering on quagga.
>
> Nothing wrong with exabgp, btw.  Great product.  It's just the wrong tool
> for the job here.
>
> Nick
>
>



-- 
Sincerely yours, Pavel Odintsov

Re: Low Cost 10G Router

[Aled Morris]

On 20 May 2015 at 15:00, Pavel Odintsov  wrote:

> Yes, you could do filtering with Quagga. But Quagga is pretty old tool
> without multiple dynamic features. But with ExaBGP you could do really
> any significant route table transformations with Python in few lines
> of code. But it's definitely add additional point of failure/bug.
>

Couldn't your back-end scripts running under ExaBGP also manage the FIB,
using standard Unix tools/APIs?

Managing the FIB is basically just "route add" and "route delete" right?

Aled

Re: Low Cost 10G Router

[charles]

On 2015-05-20 08:17, Pavel Odintsov wrote:

Hello!

Ray, I could suggest switch from multi physical CPU configuration to
single. Like Intel Xeon E5-1650/1660/1680 or even Xeon E3 platforms.
Because multi processor systems need really huge amount of knowledge
for NUMA configuration and PCI-E devices assignment for each NUMA.



Not really. Well that's opinion I suppose. It didn't seem like that 
steep of a learning curve. Just need to play with taskset and do some 
reading. If you are just starting out and experimenting, then sure a 
single CPU system would probably be the way to go.




Secondly, I could vote many times for Supermicro! :) Dell or HP are
really ugly systems for soft routers. CPU frequency tuning, PCM
debugging are real nightmare on this systems.


And why is that any different on a supermicro system? Isn't it all the 
same hardware? I personally would recommend buying from Dell or HP, as 
they things like 4hr turn around times (at least in the major urban 
centers, usually it's about an hour). I don't know how good Supermicro 
purchase/procurement system is. Dell has some neat things for asset 
management, support etc. HP probably has the same.



 Please beware of they!


Supermicro is very clear and do not block useful functions of platform.




What don't they "block"? What vendors block things, and what things do 
they block?

RE: AT&T/Telia issue

[Tyler Applebaum]

Still seeing this as of 7:40AM PST. Looks isolated to AT&T and Telia in Seattle.

HOST: PC-002Loss%  Snt  LastAvg Best Wrst  StDev
  1.|-- 172.31.255.1   0.0%   10 00.803
0.9
  2.|-- 10.98.0.4  0.0%   10 11.514
1.1
  3.|-- 67.51.253.17   0.0%   10 62.826
1.2
  4.|-- 67.51.253.10.0%   10 21.412
0.5
  5.|-- 67.51.253.30.0%   10 21.312
0.5
  6.|-- v202.core1.pdx1.he.net 0.0%   10 12.014
1.2
  7.|-- 10ge12-4.core1.sea1.he.net 0.0%   10 9   10.99   13
1.0
  8.|-- sea-b1-link.telia.net 50.0%   1042   42.0   42   42
0.0
  9.|-- att-ic-153030-sea-b1.c.telia.net  50.0%   1046   44.8   43   46
1.3
 10.|-- cr84.st0wa.ip.att.net 40.0%   1071   73.8   71   76
1.8
 11.|-- cr2.st6wa.ip.att.net  40.0%   1074   73.7   72   75
1.2
 12.|-- 12.122.158.14670.0%   1074   73.7   73   74
0.6
 13.|-- 12.122.158.15750.0%   1071   71.0   71   71
0.0
 14.|-- 12.248.207.6  20.0%   1071   71.0   71   71
0.0
 15.|-- ancr-5-1-12-12.attalascom.net 30.0%   1071   71.0   71   71
0.0
 16.|-- 66-2-12-12.attalascom.net 30.0%   1085   85.3   85   86
0.5
 17.|-- KCHC-42-7-12-12.attalascom.net30.0%   1095   95.6   95   96
0.5

-Original Message-
From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Tyler Applebaum
Sent: Tuesday, May 19, 2015 4:20 PM
To: nanog@nanog.org
Subject: AT&T/Telia issue

Seeing this on AS7018 to AS1299. Anyone out there at either provider know 
anything about this?

HOST: PC-002  Loss%  Snt  LastAvg Best Wrst  StDev
  1.|-- 172.31.255.1 0.0%   10 10.7030.9
  2.|-- 10.98.0.30.0%   10 11.0110.0
  3.|-- 67.51.253.17 0.0%   10 22.5240.7
  4.|-- 67.51.253.3  0.0%   10 11.2120.4
  5.|-- v202.core1.pdx1.he.net   0.0%   10 7   10.57   121.9
  6.|-- 10ge12-4.core1.sea1.he.net   0.0%   10 55.0550.0
  7.|-- sea-b1-link.telia.net0.0%   10 55.85   122.2
  8.|-- den-b1-link.telia.net0.0%   10   108  107.3  106  1080.7
  9.|-- sjo-b21-link.telia.net  20.0%   10   137  134.9  134  1371.0
10.|-- 192.205.33.45   40.0%   10   136  136.2  135  1381.2
11.|-- cr1.sffca.ip.att.net10.0%   10   141  141.9  139  1451.9
12.|-- 12.122.2.77 20.0%   10   140  140.1  137  1422.0
13.|-- 12.122.160.149  10.0%   10   138  141.1  137  1648.6
14.|-- 12.117.131.214  30.0%   10   139  141.0  139  1451.9
15.|-- 199.103.47.230.0%   1051  128.0   51  142   34.0

HOST: PC-002  Loss%  Snt  LastAvg 
Best Wrst  StDev
  1.|-- 172.31.255.1 0.0%   20 11.1 
   030.6
  2.|-- 10.98.0.40.0%   20 11.3 
   140.7
  3.|-- 67.51.253.17 0.0%   20 34.9 
   2   48   10.2
  4.|-- 67.51.253.1  0.0%   20 21.1 
   120.3
  5.|-- 67.51.253.11 0.0%   20 11.4 
   120.5
  6.|-- v202.core1.pdx1.he.net   0.0%   20 69.1 
   1   123.2
  7.|-- 10ge12-4.core1.sea1.he.net   0.0%   20 56.5 
   5   111.7
  8.|-- sea-b1-link.telia.net0.0%   20 55.1 
   560.3
  9.|-- att-ic-153030-sea-b1.c.telia.net 0.0%   20 97.7 
   691.2
10.|-- cr83.st0wa.ip.att.net5.0%   20   118  119.7  
117  1231.5
11.|-- cr2.ptdor.ip.att.net 0.0%   20   119  120.1  
118  1221.4
12.|-- cr2.sffca.ip.att.net 0.0%   20   120  119.2  
117  1211.4
13.|-- cr2.sc1ca.ip.att.net 0.0%   20   119  121.1  
118  1496.6
14.|-- 12.122.151.129   0.0%   20   118  119.8  
117  1221.5
15.|-- ???100.0%   20 00.0  
  000.0
16.|-- 71.157.120.39   75.0%   20   119  118.6  
118  1190.5
17.|-- 108-248-29-59.lightspeed.renonv.sbcglobal.net5.0%   20   139  137.1  
135  1462.5
18.|-- 108-241-228-42.lightspeed.renonv.sbcglobal.net   5.0%   20   143  139.2  
135  1524.9
Attention: Information contained in this message a

Re: Low Cost 10G Router

[Nick Hilliard]

On 20/05/2015 15:25, Aled Morris wrote:
> Couldn't your back-end scripts running under ExaBGP also manage the FIB,
> using standard Unix tools/APIs?
> 
> Managing the FIB is basically just "route add" and "route delete" right?

Yes, you could probably do this.  No, you probably wouldn't want to do
this.  Pls see the netlink interface modules in bird and quagga to
understand why.

Nick

Re: Low Cost 10G Router

[Pavel Odintsov]

Hello!

Yes, we could run route add / route del when we got any announce from
external world with ExaBGP directly. I have implemented custom custom
Firewall (netmap-ipfw) management tool which implement in similar
manner. But I'm working with BGP flow spec. It's so complex, standard
BGP is much times simpler.

And I could share my ExaBGP configuration and hook scripts.

ExaBGP config: 
https://github.com/FastVPSEestiOu/fastnetmon/blob/master/src/scripts/exabgp_firewall.conf

Hook script which put all announces to Redis Queue:
https://github.com/FastVPSEestiOu/fastnetmon/blob/master/src/scripts/exabgp_queue_writer.py

But full BGP route table is enough big and need external processing.

But yes, with some Python code is possible to implement route server
with ExaBGP.

On Wed, May 20, 2015 at 5:25 PM, Aled Morris  wrote:
> On 20 May 2015 at 15:00, Pavel Odintsov  wrote:
>>
>> Yes, you could do filtering with Quagga. But Quagga is pretty old tool
>> without multiple dynamic features. But with ExaBGP you could do really
>> any significant route table transformations with Python in few lines
>> of code. But it's definitely add additional point of failure/bug.
>
>
> Couldn't your back-end scripts running under ExaBGP also manage the FIB,
> using standard Unix tools/APIs?
>
> Managing the FIB is basically just "route add" and "route delete" right?
>
> Aled
>



-- 
Sincerely yours, Pavel Odintsov

Re: Low Cost 10G Router

[Colton Conor]

So, from the sounds of it most are saying for low cost, the way to go would
be a software router, which I was trying to avoid. To answer the bandwidth
question, we would have three 10G ports with three different carriers and
at max push 10Gbps of total traffic to start.

I think this leaves me with hardware routers that can support full BGP
tables. So, who actually sells full bgp routers. So far on my list I have:
Juniper MX Series
Brocade MLXe or CER
Cisco ASR 9K
Huawei NE40E-X1-M4
ZTE, not sure which model?
ALU 7750

Besides the above, am I missing anyone else that makes a true carrier grade
hardware router?

On Wed, May 20, 2015 at 9:54 AM, Pavel Odintsov 
wrote:

> Hello!
>
> Yes, we could run route add / route del when we got any announce from
> external world with ExaBGP directly. I have implemented custom custom
> Firewall (netmap-ipfw) management tool which implement in similar
> manner. But I'm working with BGP flow spec. It's so complex, standard
> BGP is much times simpler.
>
> And I could share my ExaBGP configuration and hook scripts.
>
> ExaBGP config:
> https://github.com/FastVPSEestiOu/fastnetmon/blob/master/src/scripts/exabgp_firewall.conf
>
> Hook script which put all announces to Redis Queue:
>
> https://github.com/FastVPSEestiOu/fastnetmon/blob/master/src/scripts/exabgp_queue_writer.py
>
> But full BGP route table is enough big and need external processing.
>
> But yes, with some Python code is possible to implement route server
> with ExaBGP.
>
> On Wed, May 20, 2015 at 5:25 PM, Aled Morris  wrote:
> > On 20 May 2015 at 15:00, Pavel Odintsov 
> wrote:
> >>
> >> Yes, you could do filtering with Quagga. But Quagga is pretty old tool
> >> without multiple dynamic features. But with ExaBGP you could do really
> >> any significant route table transformations with Python in few lines
> >> of code. But it's definitely add additional point of failure/bug.
> >
> >
> > Couldn't your back-end scripts running under ExaBGP also manage the FIB,
> > using standard Unix tools/APIs?
> >
> > Managing the FIB is basically just "route add" and "route delete" right?
> >
> > Aled
> >
>
>
>
> --
> Sincerely yours, Pavel Odintsov
>

Re: Low Cost 10G Router

[Colton Conor]

So are the rest of the processes in Mikrotik OS multi threaded? I would
hope so to take advantage of 36 cores!

What is up with all of these network vendors not supporting more than one
core in their OS? I just don't get it.



On Tue, May 19, 2015 at 9:49 PM, Josh Baird  wrote:

> The BGP daemon on the CCR routers is not multi-threaded; it only will use
> one core.
>
> Josh
>
> On Tue, May 19, 2015 at 10:06 PM, Colton Conor 
> wrote:
>
>>  So this new $1295 Mikrotik CCR1036-8G-2S+EM  has a 36 core Tilera CPU
>> with
>> 16GB of ram. Each core is running at 1.2Ghz? I assume that Mikrotik is
>> multicore in software, so why does this box not outperform these intel
>> boxes that everyone is recommending? Is it just a limitation of ports?
>>
>>
>>
>> On Tue, May 19, 2015 at 6:03 PM, Faisal Imtiaz 
>> wrote:
>>
>> >
>> >
>> >
>> > > I've seen serious, unusual performance bottlenecks in Mikrotik CCR, in
>> > some
>> > > cases not even achieving a gigabit speeds on 10G interfaces.
>> Performance
>> > > drops more rapidly then Cisco with smaller packet sizes.
>> > >
>> > >  -mel beckman
>> >
>> >
>> > Folks often forget that Mikrotik ROS can also run on x86 machines.
>> >
>> > Size your favorite hardware (server) or network appliance with
>> appropriate
>> > ports, add MT ROS on a CF card, and you are good to go.
>> >
>> > We use i7 based network appliance with dual 10g cards (you can use a
>> quad
>> > 10g card, such as those made by hotlav).
>> >
>> > with a 2gig of ram, you can easily do multiple (4-5 or more full bgp
>> > peers), and i7 are good for approx 1.2mill pps.
>> >
>> >
>> > Best of luck.
>> >
>> >
>> > Faisal Imtiaz
>> > Snappy Internet & Telecom
>> >
>>
>
>

Re: Low Cost 10G Router

[Thomas Mangin]

Hello Pavel,

Using ExaBGP as an SDN already has been done (and in a very large 
scale). But I would agree with Nick; It is not something I would 
recommend to everyone.


Once more to echo Nick, to add/remove route/fw entries on Linux please 
do use netlink. The lastest ExaBGP master has some start of code to 
implement NetLink in python but I recently found a python module for it: 
https://github.com/svinota/pyroute2


Before ExaBGP can become a route server, I must complete a number of 
pieces (like the CLI which I am currently coding).
I have spoken with the IX community about making ExaBGP a RR/RS and the 
idea was not badly received, but no one offered to help so it is on the 
back burner.


Thomas

On 20 May 2015, at 15:54, Pavel Odintsov wrote:


Hello!

Yes, we could run route add / route del when we got any announce from
external world with ExaBGP directly. I have implemented custom custom
Firewall (netmap-ipfw) management tool which implement in similar
manner. But I'm working with BGP flow spec. It's so complex, standard
BGP is much times simpler.

And I could share my ExaBGP configuration and hook scripts.

ExaBGP config: 
https://github.com/FastVPSEestiOu/fastnetmon/blob/master/src/scripts/exabgp_firewall.conf


Hook script which put all announces to Redis Queue:
https://github.com/FastVPSEestiOu/fastnetmon/blob/master/src/scripts/exabgp_queue_writer.py

But full BGP route table is enough big and need external processing.

But yes, with some Python code is possible to implement route server
with ExaBGP.

On Wed, May 20, 2015 at 5:25 PM, Aled Morris  wrote:
On 20 May 2015 at 15:00, Pavel Odintsov  
wrote:


Yes, you could do filtering with Quagga. But Quagga is pretty old 
tool
without multiple dynamic features. But with ExaBGP you could do 
really

any significant route table transformations with Python in few lines
of code. But it's definitely add additional point of failure/bug.



Couldn't your back-end scripts running under ExaBGP also manage the 
FIB,

using standard Unix tools/APIs?

Managing the FIB is basically just "route add" and "route delete" 
right?


Aled





--
Sincerely yours, Pavel Odintsov

Re: Low Cost 10G Router

[Blake Dunlap]

good, cheap, built by someone else


pick 2

On Wed, May 20, 2015 at 9:42 AM, Colton Conor  wrote:
> So, from the sounds of it most are saying for low cost, the way to go would
> be a software router, which I was trying to avoid. To answer the bandwidth
> question, we would have three 10G ports with three different carriers and
> at max push 10Gbps of total traffic to start.
>
> I think this leaves me with hardware routers that can support full BGP
> tables. So, who actually sells full bgp routers. So far on my list I have:
> Juniper MX Series
> Brocade MLXe or CER
> Cisco ASR 9K
> Huawei NE40E-X1-M4
> ZTE, not sure which model?
> ALU 7750
>
> Besides the above, am I missing anyone else that makes a true carrier grade
> hardware router?
>
> On Wed, May 20, 2015 at 9:54 AM, Pavel Odintsov 
> wrote:
>
>> Hello!
>>
>> Yes, we could run route add / route del when we got any announce from
>> external world with ExaBGP directly. I have implemented custom custom
>> Firewall (netmap-ipfw) management tool which implement in similar
>> manner. But I'm working with BGP flow spec. It's so complex, standard
>> BGP is much times simpler.
>>
>> And I could share my ExaBGP configuration and hook scripts.
>>
>> ExaBGP config:
>> https://github.com/FastVPSEestiOu/fastnetmon/blob/master/src/scripts/exabgp_firewall.conf
>>
>> Hook script which put all announces to Redis Queue:
>>
>> https://github.com/FastVPSEestiOu/fastnetmon/blob/master/src/scripts/exabgp_queue_writer.py
>>
>> But full BGP route table is enough big and need external processing.
>>
>> But yes, with some Python code is possible to implement route server
>> with ExaBGP.
>>
>> On Wed, May 20, 2015 at 5:25 PM, Aled Morris  wrote:
>> > On 20 May 2015 at 15:00, Pavel Odintsov 
>> wrote:
>> >>
>> >> Yes, you could do filtering with Quagga. But Quagga is pretty old tool
>> >> without multiple dynamic features. But with ExaBGP you could do really
>> >> any significant route table transformations with Python in few lines
>> >> of code. But it's definitely add additional point of failure/bug.
>> >
>> >
>> > Couldn't your back-end scripts running under ExaBGP also manage the FIB,
>> > using standard Unix tools/APIs?
>> >
>> > Managing the FIB is basically just "route add" and "route delete" right?
>> >
>> > Aled
>> >
>>
>>
>>
>> --
>> Sincerely yours, Pavel Odintsov
>>

Re: Low Cost 10G Router

[Rafael Possamai]

Since you are considering multiple options, I'd build a decision matrix.
You can put down all the requirements, score each option, and then
normalize it to give each a final score. After that you can calculate some
other things such as throughput per dollar, etc.

http://asq.org/learn-about-quality/decision-making-tools/overview/decision-matrix.html

Regarding the Mikrotik, there's a difference between Multithreading and
Multiprocessing.


On Wed, May 20, 2015 at 11:44 AM, Colton Conor 
wrote:

> So are the rest of the processes in Mikrotik OS multi threaded? I would
> hope so to take advantage of 36 cores!
>
> What is up with all of these network vendors not supporting more than one
> core in their OS? I just don't get it.
>
>
>
> On Tue, May 19, 2015 at 9:49 PM, Josh Baird  wrote:
>
> > The BGP daemon on the CCR routers is not multi-threaded; it only will use
> > one core.
> >
> > Josh
> >
> > On Tue, May 19, 2015 at 10:06 PM, Colton Conor 
> > wrote:
> >
> >>  So this new $1295 Mikrotik CCR1036-8G-2S+EM  has a 36 core Tilera CPU
> >> with
> >> 16GB of ram. Each core is running at 1.2Ghz? I assume that Mikrotik is
> >> multicore in software, so why does this box not outperform these intel
> >> boxes that everyone is recommending? Is it just a limitation of ports?
> >>
> >>
> >>
> >> On Tue, May 19, 2015 at 6:03 PM, Faisal Imtiaz <
> fai...@snappytelecom.net>
> >> wrote:
> >>
> >> >
> >> >
> >> >
> >> > > I've seen serious, unusual performance bottlenecks in Mikrotik CCR,
> in
> >> > some
> >> > > cases not even achieving a gigabit speeds on 10G interfaces.
> >> Performance
> >> > > drops more rapidly then Cisco with smaller packet sizes.
> >> > >
> >> > >  -mel beckman
> >> >
> >> >
> >> > Folks often forget that Mikrotik ROS can also run on x86 machines.
> >> >
> >> > Size your favorite hardware (server) or network appliance with
> >> appropriate
> >> > ports, add MT ROS on a CF card, and you are good to go.
> >> >
> >> > We use i7 based network appliance with dual 10g cards (you can use a
> >> quad
> >> > 10g card, such as those made by hotlav).
> >> >
> >> > with a 2gig of ram, you can easily do multiple (4-5 or more full bgp
> >> > peers), and i7 are good for approx 1.2mill pps.
> >> >
> >> >
> >> > Best of luck.
> >> >
> >> >
> >> > Faisal Imtiaz
> >> > Snappy Internet & Telecom
> >> >
> >>
> >
> >
>

Re: Low Cost 10G Router

[Aled Morris]

On 20 May 2015 at 17:44, Colton Conor  wrote:

> So are the rest of the processes in Mikrotik OS multi threaded? I would
> hope so to take advantage of 36 cores!
>

The forthcoming new major software release from Mikrotik apparently will
have multi-threaded BGP - it is targetted at their (also forthcoming) 72
core 8x10GE router, the CCR1072

I would treat this as speculation until you can order it though - it's been
"promised" for 18 months now.

Aled

Re: Low Cost 10G Router

[Blake Hudson]

As mentioned by others on the list, a properly configured ASR1004 and up 
can do this.


--Blake


Colton Conor wrote on 5/20/2015 11:42 AM:

So, from the sounds of it most are saying for low cost, the way to go would
be a software router, which I was trying to avoid. To answer the bandwidth
question, we would have three 10G ports with three different carriers and
at max push 10Gbps of total traffic to start.

I think this leaves me with hardware routers that can support full BGP
tables. So, who actually sells full bgp routers. So far on my list I have:
Juniper MX Series
Brocade MLXe or CER
Cisco ASR 9K
Huawei NE40E-X1-M4
ZTE, not sure which model?
ALU 7750

Besides the above, am I missing anyone else that makes a true carrier grade
hardware router?

On Wed, May 20, 2015 at 9:54 AM, Pavel Odintsov 
wrote:


Hello!

Yes, we could run route add / route del when we got any announce from
external world with ExaBGP directly. I have implemented custom custom
Firewall (netmap-ipfw) management tool which implement in similar
manner. But I'm working with BGP flow spec. It's so complex, standard
BGP is much times simpler.

And I could share my ExaBGP configuration and hook scripts.

ExaBGP config:
https://github.com/FastVPSEestiOu/fastnetmon/blob/master/src/scripts/exabgp_firewall.conf

Hook script which put all announces to Redis Queue:

https://github.com/FastVPSEestiOu/fastnetmon/blob/master/src/scripts/exabgp_queue_writer.py

But full BGP route table is enough big and need external processing.

But yes, with some Python code is possible to implement route server
with ExaBGP.

On Wed, May 20, 2015 at 5:25 PM, Aled Morris  wrote:

On 20 May 2015 at 15:00, Pavel Odintsov 

wrote:

Yes, you could do filtering with Quagga. But Quagga is pretty old tool
without multiple dynamic features. But with ExaBGP you could do really
any significant route table transformations with Python in few lines
of code. But it's definitely add additional point of failure/bug.


Couldn't your back-end scripts running under ExaBGP also manage the FIB,
using standard Unix tools/APIs?

Managing the FIB is basically just "route add" and "route delete" right?

Aled




--
Sincerely yours, Pavel Odintsov

Re: Low Cost 10G Router

[Mike Hammett]

Well, the cores on a many-core CPU aren't going to have the "torque" that a 
Xeon would. They're also still working on the software. It has gotten a ton 
better over the life of the CCRs thus far. BGP is still atrocious on the CCRs, 
but that's because the route update process isn't multithreaded. It won't be 
multithreaded in the next major version either, but they will have done some 
programming voodoo (all programming is voodoo to me) to reign in the poor 
performance issues with full tables. 

https://youtu.be/ihZiAC-Rox8?t=37m8s 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 



Midwest Internet Exchange 
http://www.midwest-ix.com 


- Original Message -

From: "Colton Conor"  
To: "Faisal Imtiaz"  
Cc: "North American Network Operators Group"  
Sent: Tuesday, May 19, 2015 9:06:26 PM 
Subject: Re: Low Cost 10G Router 

So this new $1295 Mikrotik CCR1036-8G-2S+EM has a 36 core Tilera CPU with 
16GB of ram. Each core is running at 1.2Ghz? I assume that Mikrotik is 
multicore in software, so why does this box not outperform these intel 
boxes that everyone is recommending? Is it just a limitation of ports? 



On Tue, May 19, 2015 at 6:03 PM, Faisal Imtiaz  
wrote: 

> 
> 
> 
> > I've seen serious, unusual performance bottlenecks in Mikrotik CCR, in 
> some 
> > cases not even achieving a gigabit speeds on 10G interfaces. Performance 
> > drops more rapidly then Cisco with smaller packet sizes. 
> > 
> > -mel beckman 
> 
> 
> Folks often forget that Mikrotik ROS can also run on x86 machines. 
> 
> Size your favorite hardware (server) or network appliance with appropriate 
> ports, add MT ROS on a CF card, and you are good to go. 
> 
> We use i7 based network appliance with dual 10g cards (you can use a quad 
> 10g card, such as those made by hotlav). 
> 
> with a 2gig of ram, you can easily do multiple (4-5 or more full bgp 
> peers), and i7 are good for approx 1.2mill pps. 
> 
> 
> Best of luck. 
> 
> 
> Faisal Imtiaz 
> Snappy Internet & Telecom 
> 

Re: Low Cost 10G Router

[Mike Hammett]

There will *not* be multi-threaded BGP in RouterOS. I was going to refer you to 
the post I made last night, but due to the unique way the e-mail list is setup, 
I replied directly to Colton instead of the list. I resent it again to the 
list. 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 



Midwest Internet Exchange 
http://www.midwest-ix.com 


- Original Message -

From: "Aled Morris"  
To: "Colton Conor"  
Cc: "North American Network Operators Group"  
Sent: Wednesday, May 20, 2015 11:59:04 AM 
Subject: Re: Low Cost 10G Router 

On 20 May 2015 at 17:44, Colton Conor  wrote: 

> So are the rest of the processes in Mikrotik OS multi threaded? I would 
> hope so to take advantage of 36 cores! 
> 

The forthcoming new major software release from Mikrotik apparently will 
have multi-threaded BGP - it is targetted at their (also forthcoming) 72 
core 8x10GE router, the CCR1072 

I would treat this as speculation until you can order it though - it's been 
"promised" for 18 months now. 

Aled 

Re: Low Cost 10G Router

[Cody Grosskopf]

Add Alcatel-Lucent 7750? I have no experience but this list seems to love
them.

On Wed, May 20, 2015, 9:44 AM Colton Conor  wrote:

> So, from the sounds of it most are saying for low cost, the way to go would
> be a software router, which I was trying to avoid. To answer the bandwidth
> question, we would have three 10G ports with three different carriers and
> at max push 10Gbps of total traffic to start.
>
> I think this leaves me with hardware routers that can support full BGP
> tables. So, who actually sells full bgp routers. So far on my list I have:
> Juniper MX Series
> Brocade MLXe or CER
> Cisco ASR 9K
> Huawei NE40E-X1-M4
> ZTE, not sure which model?
> ALU 7750
>
> Besides the above, am I missing anyone else that makes a true carrier grade
> hardware router?
>
> On Wed, May 20, 2015 at 9:54 AM, Pavel Odintsov 
> wrote:
>
> > Hello!
> >
> > Yes, we could run route add / route del when we got any announce from
> > external world with ExaBGP directly. I have implemented custom custom
> > Firewall (netmap-ipfw) management tool which implement in similar
> > manner. But I'm working with BGP flow spec. It's so complex, standard
> > BGP is much times simpler.
> >
> > And I could share my ExaBGP configuration and hook scripts.
> >
> > ExaBGP config:
> >
> https://github.com/FastVPSEestiOu/fastnetmon/blob/master/src/scripts/exabgp_firewall.conf
> >
> > Hook script which put all announces to Redis Queue:
> >
> >
> https://github.com/FastVPSEestiOu/fastnetmon/blob/master/src/scripts/exabgp_queue_writer.py
> >
> > But full BGP route table is enough big and need external processing.
> >
> > But yes, with some Python code is possible to implement route server
> > with ExaBGP.
> >
> > On Wed, May 20, 2015 at 5:25 PM, Aled Morris  wrote:
> > > On 20 May 2015 at 15:00, Pavel Odintsov 
> > wrote:
> > >>
> > >> Yes, you could do filtering with Quagga. But Quagga is pretty old tool
> > >> without multiple dynamic features. But with ExaBGP you could do really
> > >> any significant route table transformations with Python in few lines
> > >> of code. But it's definitely add additional point of failure/bug.
> > >
> > >
> > > Couldn't your back-end scripts running under ExaBGP also manage the
> FIB,
> > > using standard Unix tools/APIs?
> > >
> > > Managing the FIB is basically just "route add" and "route delete"
> right?
> > >
> > > Aled
> > >
> >
> >
> >
> > --
> > Sincerely yours, Pavel Odintsov
> >
>

Re: Low Cost 10G Router

[Baldur Norddahl]

ZTE M6000-3S.

It is what we use. Works well for us. Just remember to get a memory upgrade
to 8 GB memory or you will run out of RIB space.

Regards

Baldur
 Den 20/05/2015 18.43 skrev "Colton Conor" :

> So, from the sounds of it most are saying for low cost, the way to go would
> be a software router, which I was trying to avoid. To answer the bandwidth
> question, we would have three 10G ports with three different carriers and
> at max push 10Gbps of total traffic to start.
>
> I think this leaves me with hardware routers that can support full BGP
> tables. So, who actually sells full bgp routers. So far on my list I have:
> Juniper MX Series
> Brocade MLXe or CER
> Cisco ASR 9K
> Huawei NE40E-X1-M4
> ZTE, not sure which model?
> ALU 7750
>
> Besides the above, am I missing anyone else that makes a true carrier grade
> hardware router?
>
> On Wed, May 20, 2015 at 9:54 AM, Pavel Odintsov 
> wrote:
>
> > Hello!
> >
> > Yes, we could run route add / route del when we got any announce from
> > external world with ExaBGP directly. I have implemented custom custom
> > Firewall (netmap-ipfw) management tool which implement in similar
> > manner. But I'm working with BGP flow spec. It's so complex, standard
> > BGP is much times simpler.
> >
> > And I could share my ExaBGP configuration and hook scripts.
> >
> > ExaBGP config:
> >
> https://github.com/FastVPSEestiOu/fastnetmon/blob/master/src/scripts/exabgp_firewall.conf
> >
> > Hook script which put all announces to Redis Queue:
> >
> >
> https://github.com/FastVPSEestiOu/fastnetmon/blob/master/src/scripts/exabgp_queue_writer.py
> >
> > But full BGP route table is enough big and need external processing.
> >
> > But yes, with some Python code is possible to implement route server
> > with ExaBGP.
> >
> > On Wed, May 20, 2015 at 5:25 PM, Aled Morris  wrote:
> > > On 20 May 2015 at 15:00, Pavel Odintsov 
> > wrote:
> > >>
> > >> Yes, you could do filtering with Quagga. But Quagga is pretty old tool
> > >> without multiple dynamic features. But with ExaBGP you could do really
> > >> any significant route table transformations with Python in few lines
> > >> of code. But it's definitely add additional point of failure/bug.
> > >
> > >
> > > Couldn't your back-end scripts running under ExaBGP also manage the
> FIB,
> > > using standard Unix tools/APIs?
> > >
> > > Managing the FIB is basically just "route add" and "route delete"
> right?
> > >
> > > Aled
> > >
> >
> >
> >
> > --
> > Sincerely yours, Pavel Odintsov
> >
>

Re: Low Cost 10G Router

[Colton Conor]

Yep, thats what I meant be ALU 7750 :)

On Wed, May 20, 2015 at 12:17 PM, Cody Grosskopf 
wrote:

> Add Alcatel-Lucent 7750? I have no experience but this list seems to love
> them.
>
> On Wed, May 20, 2015, 9:44 AM Colton Conor  wrote:
>
>> So, from the sounds of it most are saying for low cost, the way to go
>> would
>> be a software router, which I was trying to avoid. To answer the bandwidth
>> question, we would have three 10G ports with three different carriers and
>> at max push 10Gbps of total traffic to start.
>>
>> I think this leaves me with hardware routers that can support full BGP
>> tables. So, who actually sells full bgp routers. So far on my list I have:
>> Juniper MX Series
>> Brocade MLXe or CER
>> Cisco ASR 9K
>> Huawei NE40E-X1-M4
>> ZTE, not sure which model?
>> ALU 7750
>>
>> Besides the above, am I missing anyone else that makes a true carrier
>> grade
>> hardware router?
>>
>> On Wed, May 20, 2015 at 9:54 AM, Pavel Odintsov > >
>> wrote:
>>
>> > Hello!
>> >
>> > Yes, we could run route add / route del when we got any announce from
>> > external world with ExaBGP directly. I have implemented custom custom
>> > Firewall (netmap-ipfw) management tool which implement in similar
>> > manner. But I'm working with BGP flow spec. It's so complex, standard
>> > BGP is much times simpler.
>> >
>> > And I could share my ExaBGP configuration and hook scripts.
>> >
>> > ExaBGP config:
>> >
>> https://github.com/FastVPSEestiOu/fastnetmon/blob/master/src/scripts/exabgp_firewall.conf
>> >
>> > Hook script which put all announces to Redis Queue:
>> >
>> >
>> https://github.com/FastVPSEestiOu/fastnetmon/blob/master/src/scripts/exabgp_queue_writer.py
>> >
>> > But full BGP route table is enough big and need external processing.
>> >
>> > But yes, with some Python code is possible to implement route server
>> > with ExaBGP.
>> >
>> > On Wed, May 20, 2015 at 5:25 PM, Aled Morris  wrote:
>> > > On 20 May 2015 at 15:00, Pavel Odintsov 
>> > wrote:
>> > >>
>> > >> Yes, you could do filtering with Quagga. But Quagga is pretty old
>> tool
>> > >> without multiple dynamic features. But with ExaBGP you could do
>> really
>> > >> any significant route table transformations with Python in few lines
>> > >> of code. But it's definitely add additional point of failure/bug.
>> > >
>> > >
>> > > Couldn't your back-end scripts running under ExaBGP also manage the
>> FIB,
>> > > using standard Unix tools/APIs?
>> > >
>> > > Managing the FIB is basically just "route add" and "route delete"
>> right?
>> > >
>> > > Aled
>> > >
>> >
>> >
>> >
>> > --
>> > Sincerely yours, Pavel Odintsov
>> >
>>
>

Re: Low Cost 10G Router

[Alain Hebert]

Well, in my experience, which is limited to small iron mostly.

Juniper MX104

Do not forget to get a second RE (Routine Engine) for software
upgrade, and be prepare to accept to pay a "license" to use the 10Gbps
ports on top of buying the IO cards.
(1 license per 2 ports).

Don't forget to set aside some times to port your configuration
into it, if you are used to Cisco/Brocade style config.

And that I'm too stupid to figure out a way to make 'test
policy' do the same thing as "show ip bgp route-map XYZ"

CER2K (latest revision)

Has plenty of RAM for 6 full routing table (and maybe more) and
1.5M RIB compared to the ~524k from the first gen.
( Got burned on those )

MLX

Juniper MX104 where cheaper for about the same platform using
MLX products.

Cisco

I don't know about the licensing for the ASR but I mostly deal
with second hand devices.

They are not flashy but do the job.

Huawei, ZTE

I didn't touch those and mostly won't beside looking into some
security concern some people are having.

PS: With almost 130k prefixes polluting the routing table you could
use a software route server and feed an auto-summary of the full route
into a router/switch that can handle the RIB/FIB.  I have yet to test
Bird but I heard good things about using it for that function.
( By pollution, I mean, it was a test made on 6 peers where I found
~130k prefixes where using the same path as their larger subnet, I have
to put up more time on that bench thou )

-
Alain Hebertaheb...@pubnix.net   
PubNIX Inc.
50 boul. St-Charles
P.O. Box 26770 Beaconsfield, Quebec H9W 6G7
Tel: 514-990-5911  http://www.pubnix.netFax: 514-990-9443

On 05/20/15 12:42, Colton Conor wrote:
> So, from the sounds of it most are saying for low cost, the way to go would
> be a software router, which I was trying to avoid. To answer the bandwidth
> question, we would have three 10G ports with three different carriers and
> at max push 10Gbps of total traffic to start.
>
> I think this leaves me with hardware routers that can support full BGP
> tables. So, who actually sells full bgp routers. So far on my list I have:
> Juniper MX Series
> Brocade MLXe or CER
> Cisco ASR 9K
> Huawei NE40E-X1-M4
> ZTE, not sure which model?
> ALU 7750
>
> Besides the above, am I missing anyone else that makes a true carrier grade
> hardware router?
>
> On Wed, May 20, 2015 at 9:54 AM, Pavel Odintsov 
> wrote:
>
>> Hello!
>>
>> Yes, we could run route add / route del when we got any announce from
>> external world with ExaBGP directly. I have implemented custom custom
>> Firewall (netmap-ipfw) management tool which implement in similar
>> manner. But I'm working with BGP flow spec. It's so complex, standard
>> BGP is much times simpler.
>>
>> And I could share my ExaBGP configuration and hook scripts.
>>
>> ExaBGP config:
>> https://github.com/FastVPSEestiOu/fastnetmon/blob/master/src/scripts/exabgp_firewall.conf
>>
>> Hook script which put all announces to Redis Queue:
>>
>> https://github.com/FastVPSEestiOu/fastnetmon/blob/master/src/scripts/exabgp_queue_writer.py
>>
>> But full BGP route table is enough big and need external processing.
>>
>> But yes, with some Python code is possible to implement route server
>> with ExaBGP.
>>
>> On Wed, May 20, 2015 at 5:25 PM, Aled Morris  wrote:
>>> On 20 May 2015 at 15:00, Pavel Odintsov 
>> wrote:
 Yes, you could do filtering with Quagga. But Quagga is pretty old tool
 without multiple dynamic features. But with ExaBGP you could do really
 any significant route table transformations with Python in few lines
 of code. But it's definitely add additional point of failure/bug.
>>>
>>> Couldn't your back-end scripts running under ExaBGP also manage the FIB,
>>> using standard Unix tools/APIs?
>>>
>>> Managing the FIB is basically just "route add" and "route delete" right?
>>>
>>> Aled
>>>
>>
>>
>> --
>> Sincerely yours, Pavel Odintsov
>>
>

Re: AT&T/Telia issue

[Mel Beckman]

There is a massive fiber cut in Santa Barbara affecting coastal paths for some 
carriers. That might be a factor. 

 -mel beckman

> On May 20, 2015, at 7:42 AM, Tyler Applebaum  wrote:
> 
> Still seeing this as of 7:40AM PST. Looks isolated to AT&T and Telia in 
> Seattle.
> 
> HOST: PC-002Loss%  Snt  LastAvg Best Wrst  
> StDev
>  1.|-- 172.31.255.1   0.0%   10 00.803
> 0.9
>  2.|-- 10.98.0.4  0.0%   10 11.514
> 1.1
>  3.|-- 67.51.253.17   0.0%   10 62.826
> 1.2
>  4.|-- 67.51.253.10.0%   10 21.412
> 0.5
>  5.|-- 67.51.253.30.0%   10 21.312
> 0.5
>  6.|-- v202.core1.pdx1.he.net 0.0%   10 12.014
> 1.2
>  7.|-- 10ge12-4.core1.sea1.he.net 0.0%   10 9   10.99   13
> 1.0
>  8.|-- sea-b1-link.telia.net 50.0%   1042   42.0   42   42
> 0.0
>  9.|-- att-ic-153030-sea-b1.c.telia.net  50.0%   1046   44.8   43   46
> 1.3
> 10.|-- cr84.st0wa.ip.att.net 40.0%   1071   73.8   71   76
> 1.8
> 11.|-- cr2.st6wa.ip.att.net  40.0%   1074   73.7   72   75
> 1.2
> 12.|-- 12.122.158.14670.0%   1074   73.7   73   74
> 0.6
> 13.|-- 12.122.158.15750.0%   1071   71.0   71   71
> 0.0
> 14.|-- 12.248.207.6  20.0%   1071   71.0   71   71
> 0.0
> 15.|-- ancr-5-1-12-12.attalascom.net 30.0%   1071   71.0   71   71
> 0.0
> 16.|-- 66-2-12-12.attalascom.net 30.0%   1085   85.3   85   86
> 0.5
> 17.|-- KCHC-42-7-12-12.attalascom.net30.0%   1095   95.6   95   96
> 0.5
> 
> -Original Message-
> From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Tyler Applebaum
> Sent: Tuesday, May 19, 2015 4:20 PM
> To: nanog@nanog.org
> Subject: AT&T/Telia issue
> 
> Seeing this on AS7018 to AS1299. Anyone out there at either provider know 
> anything about this?
> 
> HOST: PC-002  Loss%  Snt  LastAvg Best Wrst  StDev
>  1.|-- 172.31.255.1 0.0%   10 10.7030.9
>  2.|-- 10.98.0.30.0%   10 11.0110.0
>  3.|-- 67.51.253.17 0.0%   10 22.5240.7
>  4.|-- 67.51.253.3  0.0%   10 11.2120.4
>  5.|-- v202.core1.pdx1.he.net   0.0%   10 7   10.57   121.9
>  6.|-- 10ge12-4.core1.sea1.he.net   0.0%   10 55.0550.0
>  7.|-- sea-b1-link.telia.net0.0%   10 55.85   122.2
>  8.|-- den-b1-link.telia.net0.0%   10   108  107.3  106  1080.7
>  9.|-- sjo-b21-link.telia.net  20.0%   10   137  134.9  134  1371.0
> 10.|-- 192.205.33.45   40.0%   10   136  136.2  135  1381.2
> 11.|-- cr1.sffca.ip.att.net10.0%   10   141  141.9  139  1451.9
> 12.|-- 12.122.2.77 20.0%   10   140  140.1  137  1422.0
> 13.|-- 12.122.160.149  10.0%   10   138  141.1  137  1648.6
> 14.|-- 12.117.131.214  30.0%   10   139  141.0  139  1451.9
> 15.|-- 199.103.47.230.0%   1051  128.0   51  142   34.0
> 
> HOST: PC-002  Loss%  Snt  LastAvg 
> Best Wrst  StDev
>  1.|-- 172.31.255.1 0.0%   20 1
> 1.1030.6
>  2.|-- 10.98.0.40.0%   20 1
> 1.3140.7
>  3.|-- 67.51.253.17 0.0%   20 3
> 4.92   48   10.2
>  4.|-- 67.51.253.1  0.0%   20 2
> 1.1120.3
>  5.|-- 67.51.253.11 0.0%   20 1
> 1.4120.5
>  6.|-- v202.core1.pdx1.he.net   0.0%   20 6
> 9.11   123.2
>  7.|-- 10ge12-4.core1.sea1.he.net   0.0%   20 5
> 6.55   111.7
>  8.|-- sea-b1-link.telia.net0.0%   20 5
> 5.1560.3
>  9.|-- att-ic-153030-sea-b1.c.telia.net 0.0%   20 9
> 7.7691.2
> 10.|-- cr83.st0wa.ip.att.net5.0%   20   118  
> 119.7  117  1231.5
> 11.|-- cr2.ptdor.ip.att.net 0.0%   20   119  
> 120.1  118  1221.4
> 12.|-- cr2.sffca.ip.att.net 0.0%   20   120  
> 119.2  117  1211.4
> 13.|-- cr2.sc1ca.ip.att.net 0.0%   20   119  
> 121.1  118  1496.6
> 14.|-- 12.122.151.129   0.0%   20   118  
> 119.8  117  1221.5
> 15.|-- ???100.0%   20 0
> 0.00  

Re: Spamhaus BGP feed experiences?

[Matthias Leisi]

At dnswl.org  we check our data against the DROP list every 
once in a while. The overlap of DROP with legitimate sources of SMTP traffic is 
very, very small: a low single-digit number, and most of them are crappy to 
start with (so we don’t publish them, but only keep them in our database for 
reference purposes). 

— Matthias

> Am 19.05.2015 um 20:38 schrieb Max Tulyev :
> 
> How much false positives (i.e. blackholing traffic users want to reach)?
> 
> On 18.05.15 21:04, Marco d'Itri wrote:
>> On May 17, Mike Lyon  wrote:
>> 
>>> Any ISPs out there (big or small) ever used the Spamhaus BGP feed to
>>> prevent against botnet, spam, etc? If so, how has your experience been? Is
>>> it worthwhile? Has it helped? On / off list responses are appreciated in
>>> advance.
>> We use Spamhaus DROP (not the BGP version: our software asks a human to 
>> review each change).
>> The benefits are not obvious since we do not have access customers, but 
>> it will blackhole some networks you obviously do not want to talk to,
>> and it has not caused any troubles either.
>> 
> 



smime.p7s
Description: S/MIME cryptographic signature

[no subject]

[Marty Strong via NANOG]

This post was from a subscriber whose From: address domain has a DMARC
policy of reject or quarantine. The NANOG mailing list has
automatically wrapped this message to prevent other subscribers mail
systems from rejecting it.--- Begin Message ---
It was resolved at around 2015-05-20 17:18 UTC

Regards,
Marty Strong
--
CloudFlare - AS13335
Network Engineer
ma...@cloudflare.com
+44 20 3514 6970 UK (Office)
+44 7584 906 055 UK (Mobile)
+1 888 993 5273 US (Office)
smartflare (Skype)

http://www.peeringdb.com/view.php?asn=13335

> On 20 May 2015, at 19:00, Mel Beckman  wrote:
> 
> There is a massive fiber cut in Santa Barbara affecting coastal paths for 
> some carriers. That might be a factor. 
> 
> -mel beckman
> 
>> On May 20, 2015, at 7:42 AM, Tyler Applebaum  wrote:
>> 
>> Still seeing this as of 7:40AM PST. Looks isolated to AT&T and Telia in 
>> Seattle.
>> 
>> HOST: PC-002Loss%  Snt  LastAvg Best Wrst  
>> StDev
>> 1.|-- 172.31.255.1   0.0%   10 00.803
>> 0.9
>> 2.|-- 10.98.0.4  0.0%   10 11.514
>> 1.1
>> 3.|-- 67.51.253.17   0.0%   10 62.826
>> 1.2
>> 4.|-- 67.51.253.10.0%   10 21.412
>> 0.5
>> 5.|-- 67.51.253.30.0%   10 21.312
>> 0.5
>> 6.|-- v202.core1.pdx1.he.net 0.0%   10 12.014
>> 1.2
>> 7.|-- 10ge12-4.core1.sea1.he.net 0.0%   10 9   10.99   13
>> 1.0
>> 8.|-- sea-b1-link.telia.net 50.0%   1042   42.0   42   42
>> 0.0
>> 9.|-- att-ic-153030-sea-b1.c.telia.net  50.0%   1046   44.8   43   46
>> 1.3
>> 10.|-- cr84.st0wa.ip.att.net 40.0%   1071   73.8   71   76   
>>  1.8
>> 11.|-- cr2.st6wa.ip.att.net  40.0%   1074   73.7   72   75   
>>  1.2
>> 12.|-- 12.122.158.14670.0%   1074   73.7   73   74   
>>  0.6
>> 13.|-- 12.122.158.15750.0%   1071   71.0   71   71   
>>  0.0
>> 14.|-- 12.248.207.6  20.0%   1071   71.0   71   71   
>>  0.0
>> 15.|-- ancr-5-1-12-12.attalascom.net 30.0%   1071   71.0   71   71   
>>  0.0
>> 16.|-- 66-2-12-12.attalascom.net 30.0%   1085   85.3   85   86   
>>  0.5
>> 17.|-- KCHC-42-7-12-12.attalascom.net30.0%   1095   95.6   95   96   
>>  0.5
>> 
>> -Original Message-
>> From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Tyler Applebaum
>> Sent: Tuesday, May 19, 2015 4:20 PM
>> To: nanog@nanog.org
>> Subject: AT&T/Telia issue
>> 
>> Seeing this on AS7018 to AS1299. Anyone out there at either provider know 
>> anything about this?
>> 
>> HOST: PC-002  Loss%  Snt  LastAvg Best Wrst  StDev
>> 1.|-- 172.31.255.1 0.0%   10 10.7030.9
>> 2.|-- 10.98.0.30.0%   10 11.0110.0
>> 3.|-- 67.51.253.17 0.0%   10 22.5240.7
>> 4.|-- 67.51.253.3  0.0%   10 11.2120.4
>> 5.|-- v202.core1.pdx1.he.net   0.0%   10 7   10.57   121.9
>> 6.|-- 10ge12-4.core1.sea1.he.net   0.0%   10 55.0550.0
>> 7.|-- sea-b1-link.telia.net0.0%   10 55.85   122.2
>> 8.|-- den-b1-link.telia.net0.0%   10   108  107.3  106  1080.7
>> 9.|-- sjo-b21-link.telia.net  20.0%   10   137  134.9  134  1371.0
>> 10.|-- 192.205.33.45   40.0%   10   136  136.2  135  1381.2
>> 11.|-- cr1.sffca.ip.att.net10.0%   10   141  141.9  139  1451.9
>> 12.|-- 12.122.2.77 20.0%   10   140  140.1  137  1422.0
>> 13.|-- 12.122.160.149  10.0%   10   138  141.1  137  1648.6
>> 14.|-- 12.117.131.214  30.0%   10   139  141.0  139  1451.9
>> 15.|-- 199.103.47.230.0%   1051  128.0   51  142   34.0
>> 
>> HOST: PC-002  Loss%  Snt  Last
>> Avg Best Wrst  StDev
>> 1.|-- 172.31.255.1 0.0%   20 1
>> 1.1030.6
>> 2.|-- 10.98.0.40.0%   20 1
>> 1.3140.7
>> 3.|-- 67.51.253.17 0.0%   20 3
>> 4.92   48   10.2
>> 4.|-- 67.51.253.1  0.0%   20 2
>> 1.1120.3
>> 5.|-- 67.51.253.11 0.0%   20 1
>> 1.4120.5
>> 6.|-- v202.core1.pdx1.he.net   0.0%   20 6
>> 9.11   123.2
>> 7.|-- 10ge12-4.core1.sea1.he.net   0.0%   20 5
>> 6.55   111.7
>> 8.|-- sea-b1-link.telia.net0.0%   20 5
>> 5.1560.3
>> 9.|-- a

Re: Low Cost 10G Router

[Eddie Tardist]

On Wed, May 20, 2015 at 2:07 PM, Mike Hammett  wrote:

> Well, the cores on a many-core CPU aren't going to have the "torque" that
> a Xeon would. They're also still working on the software. It has gotten a
> ton better over the life of the CCRs thus far. BGP is still atrocious on
> the CCRs, but that's because the route update process isn't multithreaded.
> It won't be multithreaded in the next major version either, but they will
> have done some programming voodoo (all programming is voodoo to me) to
> reign in the poor performance issues with full tables.
>
> https://youtu.be/ihZiAC-Rox8?t=37m8s
>

I honestly don't know why most people gets impressed by the number of
Tylera cores on CCR and think it's a good thing.
Your "torque" point makes much sense to me. A few cores with decent clock
and Xeon or Rangeley "torque" is just better. Adding that much weak tylera
cores with low clock only results in much more context switching, much more
CPU Affinity needs.

Multithreading the relevant grained bit of code will also lead to more
context switching, but for threads now instead of processes.

As I understand the architecture of those solutions, I don't see why a bgp
daemon mono threaded is a problem. Ok, multithreaded would give a better
full routing convergence. But once the routing table is loaded it does not
matter how many threads the bgp process will use. The dirty work on Linux
(RouterOS kernel for that matter) will be done on the forward information
table, on the packet forwarding code and specially on softirq (interrupt
requests). This is where the bottleneck seems to be, IMHO. Linux is not
good at multithreaded packet forwarding and not good specially at handling
interrupt requests on multi-queue NICs. So, RouterOS is not good as well.

Therefore that "several dozens" cheap and weak tylera cores powering CCR
boxes is absolutely not friendly for Linux core and RouterOS itself.

I'm better served off with a smaller amount of cores with better clock and
better "torque" as Mr Hammett mentioned (I liked the expression usage yes)
and that's why a Linux or a BSD box with a couple Xeon CPUs will perform
better than CCR. Sometimes as someone mentioned a couple i7 cores will
outperform a CCR box as well. More torque, yeah. Less context switching and
time sharing wasted.

However this horizontal scalar number of tylera cores on the CCR is good
for marketing. After all "you are buying a 36 CPU box" paying "a couple
hundred bucks". Impressive, hum? Well not for me.






>
>
> -
> Mike Hammett
> Intelligent Computing Solutions
> http://www.ics-il.com
>
>
>
> Midwest Internet Exchange
> http://www.midwest-ix.com
>
>
> - Original Message -
>
> From: "Colton Conor" 
> To: "Faisal Imtiaz" 
> Cc: "North American Network Operators Group" 
> Sent: Tuesday, May 19, 2015 9:06:26 PM
> Subject: Re: Low Cost 10G Router
>
> So this new $1295 Mikrotik CCR1036-8G-2S+EM has a 36 core Tilera CPU with
> 16GB of ram. Each core is running at 1.2Ghz? I assume that Mikrotik is
> multicore in software, so why does this box not outperform these intel
> boxes that everyone is recommending? Is it just a limitation of ports?
>
>
>
> On Tue, May 19, 2015 at 6:03 PM, Faisal Imtiaz 
> wrote:
>
> >
> >
> >
> > > I've seen serious, unusual performance bottlenecks in Mikrotik CCR, in
> > some
> > > cases not even achieving a gigabit speeds on 10G interfaces.
> Performance
> > > drops more rapidly then Cisco with smaller packet sizes.
> > >
> > > -mel beckman
> >
> >
> > Folks often forget that Mikrotik ROS can also run on x86 machines.
> >
> > Size your favorite hardware (server) or network appliance with
> appropriate
> > ports, add MT ROS on a CF card, and you are good to go.
> >
> > We use i7 based network appliance with dual 10g cards (you can use a quad
> > 10g card, such as those made by hotlav).
> >
> > with a 2gig of ram, you can easily do multiple (4-5 or more full bgp
> > peers), and i7 are good for approx 1.2mill pps.
> >
> >
> > Best of luck.
> >
> >
> > Faisal Imtiaz
> > Snappy Internet & Telecom
> >
>
>

Re: Low Cost 10G Router

[Faisal Imtiaz]

Well said Eddie,

It would be worth pointing out that on CCR's each port also has a core 
dedicated to it, a benefit of such a design is that each port is able to handle 
a much higher PPS rate, and if there is a DDOS attack on one port, it will not 
bring down the rest of the ports / router etc. (disclaimer, if the router is 
setup properly, without all traffic going thru the CPU etc etc).



Faisal Imtiaz
Snappy Internet & Telecom
- Original Message -
> From: "Eddie Tardist" 
> To: "North American Network Operators Group" 
> Sent: Wednesday, May 20, 2015 6:34:11 PM
> Subject: Re: Low Cost 10G Router
> 
> On Wed, May 20, 2015 at 2:07 PM, Mike Hammett  wrote:
> 
> > Well, the cores on a many-core CPU aren't going to have the "torque" that
> > a Xeon would. They're also still working on the software. It has gotten a
> > ton better over the life of the CCRs thus far. BGP is still atrocious on
> > the CCRs, but that's because the route update process isn't multithreaded.
> > It won't be multithreaded in the next major version either, but they will
> > have done some programming voodoo (all programming is voodoo to me) to
> > reign in the poor performance issues with full tables.
> >
> > https://youtu.be/ihZiAC-Rox8?t=37m8s
> >
> 
> I honestly don't know why most people gets impressed by the number of
> Tylera cores on CCR and think it's a good thing.
> Your "torque" point makes much sense to me. A few cores with decent clock
> and Xeon or Rangeley "torque" is just better. Adding that much weak tylera
> cores with low clock only results in much more context switching, much more
> CPU Affinity needs.
> 
> Multithreading the relevant grained bit of code will also lead to more
> context switching, but for threads now instead of processes.
> 
> As I understand the architecture of those solutions, I don't see why a bgp
> daemon mono threaded is a problem. Ok, multithreaded would give a better
> full routing convergence. But once the routing table is loaded it does not
> matter how many threads the bgp process will use. The dirty work on Linux
> (RouterOS kernel for that matter) will be done on the forward information
> table, on the packet forwarding code and specially on softirq (interrupt
> requests). This is where the bottleneck seems to be, IMHO. Linux is not
> good at multithreaded packet forwarding and not good specially at handling
> interrupt requests on multi-queue NICs. So, RouterOS is not good as well.
> 
> Therefore that "several dozens" cheap and weak tylera cores powering CCR
> boxes is absolutely not friendly for Linux core and RouterOS itself.
> 
> I'm better served off with a smaller amount of cores with better clock and
> better "torque" as Mr Hammett mentioned (I liked the expression usage yes)
> and that's why a Linux or a BSD box with a couple Xeon CPUs will perform
> better than CCR. Sometimes as someone mentioned a couple i7 cores will
> outperform a CCR box as well. More torque, yeah. Less context switching and
> time sharing wasted.
> 
> However this horizontal scalar number of tylera cores on the CCR is good
> for marketing. After all "you are buying a 36 CPU box" paying "a couple
> hundred bucks". Impressive, hum? Well not for me.
> 
> 
> 
> 
> 
> 
> >
> >
> > -
> > Mike Hammett
> > Intelligent Computing Solutions
> > http://www.ics-il.com
> >
> >
> >
> > Midwest Internet Exchange
> > http://www.midwest-ix.com
> >
> >
> > - Original Message -
> >
> > From: "Colton Conor" 
> > To: "Faisal Imtiaz" 
> > Cc: "North American Network Operators Group" 
> > Sent: Tuesday, May 19, 2015 9:06:26 PM
> > Subject: Re: Low Cost 10G Router
> >
> > So this new $1295 Mikrotik CCR1036-8G-2S+EM has a 36 core Tilera CPU with
> > 16GB of ram. Each core is running at 1.2Ghz? I assume that Mikrotik is
> > multicore in software, so why does this box not outperform these intel
> > boxes that everyone is recommending? Is it just a limitation of ports?
> >
> >
> >
> > On Tue, May 19, 2015 at 6:03 PM, Faisal Imtiaz 
> > wrote:
> >
> > >
> > >
> > >
> > > > I've seen serious, unusual performance bottlenecks in Mikrotik CCR, in
> > > some
> > > > cases not even achieving a gigabit speeds on 10G interfaces.
> > Performance
> > > > drops more rapidly then Cisco with smaller packet sizes.
> > > >
> > > > -mel beckman
> > >
> > >
> > > Folks often forget that Mikrotik ROS can also run on x86 machines.
> > >
> > > Size your favorite hardware (server) or network appliance with
> > appropriate
> > > ports, add MT ROS on a CF card, and you are good to go.
> > >
> > > We use i7 based network appliance with dual 10g cards (you can use a quad
> > > 10g card, such as those made by hotlav).
> > >
> > > with a 2gig of ram, you can easily do multiple (4-5 or more full bgp
> > > peers), and i7 are good for approx 1.2mill pps.
> > >
> > >
> > > Best of luck.
> > >
> > >
> > > Faisal Imtiaz
> > > Snappy Internet & Telecom
> > >
> >
> >
> 

Re: Low Cost 10G Router

[BPNoC Group]

On Wed, May 20, 2015 at 1:42 PM, Colton Conor 
wrote:

> So, from the sounds of it most are saying for low cost, the way to go would
> be a software router, which I was trying to avoid. To answer the bandwidth
> question, we would have three 10G ports with three different carriers and
> at max push 10Gbps of total traffic to start.
>
> I think this leaves me with hardware routers that can support full BGP
> tables. So, who actually sells full bgp routers. So far on my list I have:
> Juniper MX Series
> Brocade MLXe or CER
> Cisco ASR 9K
> Huawei NE40E-X1-M4
> ZTE, not sure which model?
> ALU 7750
>
> Besides the above, am I missing anyone else that makes a true carrier grade
> hardware router?
>

right now I'm pushing 11G/s 1.2Mpps, ServerU L-800 + Chelsio T580-CR, see
below
although you can ssh in, it's definitely not a software router since it's
essentially T5 ASICS hardware pushing the packets

% sudo rate -i cxgbe0 -R -b
=> Currently 11.08 Gbps/1199.50 kpps, Average: 11.08 Gbps/1199.50 kpps
=> Currently 11.13 Gbps/1206.68 kpps, Average: 11.10 Gbps/1203.08 kpps
=> Currently 11.11 Gbps/1202.70 kpps, Average: 11.10 Gbps/1202.95 kpps
=> Currently 11.13 Gbps/1206.54 kpps, Average: 11.11 Gbps/1203.85 kpps
=> Currently 11.24 Gbps/1207.24 kpps, Average: 11.12 Gbps/1204.53 kpps
=> Currently 11.12 Gbps/1208.79 kpps, Average: 11.12 Gbps/1205.24 kpps
=> Currently 11.22 Gbps/1208.03 kpps, Average: 11.12 Gbps/1205.63 kpps
=> Currently 11.12 Gbps/1207.79 kpps, Average: 11.12 Gbps/1205.90 kpps
=> Currently 11.23 Gbps/1207.76 kpps, Average: 11.12 Gbps/1206.11 kpps
=> Currently 11.24 Gbps/1207.46 kpps, Average: 11.12 Gbps/1206.24 kpps
=> Currently 11.32 Gbps/1207.82 kpps, Average: 11.12 Gbps/1206.39 kpps
=> Currently 11.03 Gbps/1207.04 kpps, Average: 11.12 Gbps/1206.44 kpps

btw this is a 40G QSFP SR4 port
it's a thousand dollar card on top of a thousand dollar router + a penny
for their x8 raiser card
you won't find anything like that below 3k USD for your 10G routing low
cost needs, I'm guessing

Re: Low Cost 10G Router

[Bryan Fields]

On 5/19/15 1:22 PM, Colton Conor wrote:
> What options are available for a small, low cost router that has at least
> four 10G ports, and can handle full BGP routes? All that I know of are the
> Juniper MX80, and the Brocade CER line. What does Cisco and others have
> that compete with these two? Any other vendors besides Juniper, Brocade,
> and Cisco to look at?

In the same price range as the MX80 there is the Alcatel SRa-4/8 router.
These will do 100g in and out, and handle full tables.  You get redundant
control modules vs. a single on the juniper.

BGP is multi-threaded on the box, does RPKI for route verification, and it's
got extensive HQoS functionality amongst other features.

-- 
Bryan Fields

727-409-1194 - Voice
727-214-2508 - Fax
http://bryanfields.net

Re: Low Cost 10G Router

[Eduardo Schoedler]

2015-05-20 20:54 GMT-03:00 BPNoC Group :
> right now I'm pushing 11G/s 1.2Mpps, ServerU L-800 + Chelsio T580-CR, see
> below
> although you can ssh in, it's definitely not a software router since it's
> essentially T5 ASICS hardware pushing the packets
>
> % sudo rate -i cxgbe0 -R -b
> => Currently 11.08 Gbps/1199.50 kpps, Average: 11.08 Gbps/1199.50 kpps
> => Currently 11.13 Gbps/1206.68 kpps, Average: 11.10 Gbps/1203.08 kpps
> => Currently 11.11 Gbps/1202.70 kpps, Average: 11.10 Gbps/1202.95 kpps
> => Currently 11.13 Gbps/1206.54 kpps, Average: 11.11 Gbps/1203.85 kpps
> => Currently 11.24 Gbps/1207.24 kpps, Average: 11.12 Gbps/1204.53 kpps
> => Currently 11.12 Gbps/1208.79 kpps, Average: 11.12 Gbps/1205.24 kpps
> => Currently 11.22 Gbps/1208.03 kpps, Average: 11.12 Gbps/1205.63 kpps
> => Currently 11.12 Gbps/1207.79 kpps, Average: 11.12 Gbps/1205.90 kpps
> => Currently 11.23 Gbps/1207.76 kpps, Average: 11.12 Gbps/1206.11 kpps
> => Currently 11.24 Gbps/1207.46 kpps, Average: 11.12 Gbps/1206.24 kpps
> => Currently 11.32 Gbps/1207.82 kpps, Average: 11.12 Gbps/1206.39 kpps
> => Currently 11.03 Gbps/1207.04 kpps, Average: 11.12 Gbps/1206.44 kpps

How much routes in the FIB?

Thanks.

-- 
Eduardo Schoedler

Re: Low Cost 10G Router

[Colton Conor]

Bryan,

Very interesting. Doesn't ALU mainly compare the new Alcatel SRa-4/8 router
vs a MX104 though?

Besides no redundancy, what limitations does the MX80 and MX104 have? I am
assume the Juniper does not have "BGP is multi-threaded on the box, does
RPKI for route verification, and it's
got extensive HQoS functionality"? I heard the MX80 was limited on QoS, but
never looked into it.

On Wed, May 20, 2015 at 7:03 PM, Bryan Fields  wrote:

> On 5/19/15 1:22 PM, Colton Conor wrote:
> > What options are available for a small, low cost router that has at least
> > four 10G ports, and can handle full BGP routes? All that I know of are
> the
> > Juniper MX80, and the Brocade CER line. What does Cisco and others have
> > that compete with these two? Any other vendors besides Juniper, Brocade,
> > and Cisco to look at?
>
> In the same price range as the MX80 there is the Alcatel SRa-4/8 router.
> These will do 100g in and out, and handle full tables.  You get redundant
> control modules vs. a single on the juniper.
>
> BGP is multi-threaded on the box, does RPKI for route verification, and
> it's
> got extensive HQoS functionality amongst other features.
>
> --
> Bryan Fields
>
> 727-409-1194 - Voice
> 727-214-2508 - Fax
> http://bryanfields.net
>

Re: Low Cost 10G Router

[BPNoC Group]

On Wed, May 20, 2015 at 9:16 PM, Eduardo Schoedler 
wrote:

> 2015-05-20 20:54 GMT-03:00 BPNoC Group :
> > right now I'm pushing 11G/s 1.2Mpps, ServerU L-800 + Chelsio T580-CR, see
> > below
> > although you can ssh in, it's definitely not a software router since it's
> > essentially T5 ASICS hardware pushing the packets
> >
> > % sudo rate -i cxgbe0 -R -b
> > => Currently 11.08 Gbps/1199.50 kpps, Average: 11.08 Gbps/1199.50 kpps
> > => Currently 11.13 Gbps/1206.68 kpps, Average: 11.10 Gbps/1203.08 kpps
> > => Currently 11.11 Gbps/1202.70 kpps, Average: 11.10 Gbps/1202.95 kpps
> > => Currently 11.13 Gbps/1206.54 kpps, Average: 11.11 Gbps/1203.85 kpps
> > => Currently 11.24 Gbps/1207.24 kpps, Average: 11.12 Gbps/1204.53 kpps
> > => Currently 11.12 Gbps/1208.79 kpps, Average: 11.12 Gbps/1205.24 kpps
> > => Currently 11.22 Gbps/1208.03 kpps, Average: 11.12 Gbps/1205.63 kpps
> > => Currently 11.12 Gbps/1207.79 kpps, Average: 11.12 Gbps/1205.90 kpps
> > => Currently 11.23 Gbps/1207.76 kpps, Average: 11.12 Gbps/1206.11 kpps
> > => Currently 11.24 Gbps/1207.46 kpps, Average: 11.12 Gbps/1206.24 kpps
> > => Currently 11.32 Gbps/1207.82 kpps, Average: 11.12 Gbps/1206.39 kpps
> > => Currently 11.03 Gbps/1207.04 kpps, Average: 11.12 Gbps/1206.44 kpps
>
> How much routes in the FIB?
>
> Thanks.
>

actually it makes no difference, the relevant route entries are stored in
the T5 chip
cxgbetool tells me I have 532447 entries right now
for fib 0 anyway, I have a similar number of entries (a couple more due to
pinned ipv6 not triggered to the card), but other than management port for
ssh, snmp, webgui and netflow, only 180kpps for a trunked copper dmz
segment is actually forwarded at fib. everything else is done on the card


>
> --
> Eduardo Schoedler
>