Jared,
This is good stuff and I'm understanding how you interpret the
data. So this confirms what we are seeing. How do we take this towards a
root cause? Mash it with the Juniper threads and see where it goes?
-Hammer-
"I was a normal American nerd"
-Jack Herer
On 11/07/2011 11:01 AM, Jared Mauch wrote:
Here's some real data for those interested. It seems a quick view seems many TATA<->
Level3 and TATA<-> GBLX sets of instability.
Combined with the overall update levels seen over that 30 minutes, we saw
~1.566M updates at route views.
Compared with the 24h prior (2011.11.06 14:15 as reference) we see ~17-20x the
updates in the same time period. Now take your control plane and make it work
17-20x as hard, and you can see why we had some even general instability.
I don't have better tools for doing a more detailed analysis at this time, but
this should get you started in talking to others about what happened. (There
were no stand-out prefixes for this time-period, they all had about the same
number of updates per prefix).
#Updates|Filename
--------+-----------------
42041 20111106.1415.out
727711 20111107.1400.out
838894 20111107.1415.out
2011.11.07 14:15 datafile - Most updates/as-path (complete as seen @ rviews)
Count AS_PATH
-----+----------------------------
4563 3549 3356
3172 3549 3356 11492
2912 8492 3356 6453 4755
2729 812 6453 4755
2695 3549 6453 14080 10620
2504 3549 3356 11492 11492 11492
2421 3549 3356 7029
2362 3356 209 721 27064
2244 3356 209 20115
2130 3356 209 22561
2044 7018 6453 14080 10620
2000 3356 209
1934 3549 3356 2907
1909 3356 15412 18101
1821 3549 6453 4755
1807 3356 6453 4755
1660 8492 3356 174
1634 3549 3356 18566
1619 3549 3356 4837 17623
1617 7018 6453 4755
1581 2497 6453 7545 7545 7545
1496 8492 3356 209 721 27064
1478 2497 6453 4755
1437 8492 8928 3356 11492 11492 11492
1427 3356 3549 3216 8402
1395 3356 701 19262
1362 8492 3356 209 20115
1357 3549 7843 7843 7843 10796
1336 3549 3356 6830 6830 6830
1259 2497 3356
1238 3356 9498
1229 812 6453 14080 10620
1229 3356 6453 14080 10620
1221 3356 9121 42926
1220 3549 3356 29314
1203 3549 3356 680 680 680 680 680
1194 2497 3356 5650 7011
1165 2497 3356 9498
1151 8001 4436 7843 10796
1150 3549 3356 15412 18101 18101 18101 18101 17803
1106 8492 3356 6762 7303
1105 3549 3356 55410 45528
1098 3549 3356 15412 18101 17803
1073 3549 7843 7843 7843
1072 3549 3356 7713 17974
1069 3549 3356 15290
1041 3356 3549
1032 3549 3356 11992
1030 3356 4323
1021 3549 6453 7843 11351
1020 8492 8928 3356 7843 11351
1016 2497 3356 11492
1014 3356 2907
1011 7018 3356 2907
2011.11.07 14:00 datafile - Most updates/as-path (complete as seen @ rviews)
Count AS_PATH
-----+----------------------------
3894 3356 6453 4755
3504 3549 3356
2930 812 6453 4755
2793 3549 6453 4755
2254 3549 3356 11492 11492 11492
2235 812 6453 14080 10620
1826 2497 6453 4755
1795 3549 3356 7029
1753 3356 3549
1689 7018 6453 14080 10620
1653 812 6453 4755 17488
1649 3549 3356 18566
1613 2497 3356
1568 3549 3356 6830 6830 6830
1496 7018 6453 4755
1419 3549 6453 14080 10620
1394 3356 701 19262
1389 3356 9121 42926
1382 3549 3356 21565
1293 3549 6453 4755 17488
1252 3549 3356 13407
1188 2497 3356 5650 7011
1156 7018 6453 4755 17488
1154 3356 6453 14080 10620
1127 3356 701 3216 8402
1106 8492 9002 3549 6762 7303
1104 3549 3356 4837 17623
1049 39756 3356 7018
1048 39756 3257 7018
1035 2497 6453 7713 17974
1008 3356 3320
