I would have liked more RAM, but with the number of hosts we have, we'll have enough RAM for the forseeable future. One new host has about as much RAM as the entire existing environment.
Well, in that hat. In another hat I'm running R900s that mostly have 256 GB of RAM. I could probably downsize those to something more modern, but I'm too busy for things that aren't critical. ----- Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP ----- Original Message ----- From: "Mike Hammett" <af...@ics-il.net> To: "AnimalFarm Microwave Users Group" <af@af.afmug.com> Sent: Thursday, December 31, 2020 6:31:56 PM Subject: Re: [AFMUG] Mikrotik 1072 Frustrations Some place out of Georgia. I wasn't the one that procured them. I believe we were right around $1k for dual sockets and 144 GB of RAM, DC power supplies, that dual 10G NIC, 10x 2.5" drive bays. ----- Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP ----- Original Message ----- From: "Josh Baird" <joshba...@gmail.com> To: "AnimalFarm Microwave Users Group" <af@af.afmug.com> Sent: Thursday, December 31, 2020 1:12:34 PM Subject: Re: [AFMUG] Mikrotik 1072 Frustrations Where did you get them? Get a good deal on them? I have my eye on a few R630/R620 right now to replace some aging vSphere hosts. On Thu, Dec 31, 2020 at 1:44 PM Mike Hammett < af...@ics-il.net > wrote: I just got some R630s, but mostly for compute and light CHR duties. I got 10 core CPUs. I got it with the 2x SFP "integrated" NIC and I still have three or four expansion slots to add more. ----- Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP From: "Mark - Myakka Technologies" < m...@mailmt.com > To: "AnimalFarm Microwave Users Group" < af@af.afmug.com > Sent: Thursday, December 31, 2020 10:32:21 AM Subject: Re: [AFMUG] Mikrotik 1072 Frustrations Mike, Been batting around maybe using a dell R630 or R620 for this Can get a R630 with 2x Xeon E5-2678 v3 2.5ghz 24-Cores total R620 with 2x Xeon E5-2670v2 2.50GHz - 20 Cores total Both of these units have an array of daughter cards for networking. -- Best regards, Mark mailto:m...@mailmt.com Myakka Technologies, Inc. www.Myakka.com ------ Thursday, December 31, 2020, 10:47:29 AM, you wrote: The Vengeance routers don't have enough horsepower for their port configurations, at least Vengeance v1. The v2 does have 50% more cores, but I'm not sure that it's enough. For moving a lot of data, you really need big CPUs. Multiple sockets of high core count CPUs. ----- Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP From: "Tushar Patel" < tpa...@ecpi.com > To: "AnimalFarm Microwave Users Group" < af@af.afmug.com > Sent: Thursday, December 31, 2020 8:18:01 AM Subject: Re: [AFMUG] Mikrotik 1072 Frustrations Has anyone tried Maxwave from Baltic networks? https://www.balticnetworks.com/manufacturers/maxxwave/maxxwave-routermaxx-vengeance-8-port-gigabit-core-i7-i7-8700k-hexa-core-router They have bunch of cards options which can be added Tushar On Dec 30, 2020, at 12:28 PM, Steven Kenney < st...@wavedirect.org > wrote: Welp replacing it with a brand new 1072 with new transceivers etc lasted about a week before it rebooted itself. Mikrotik basically said RMA it or it may be faulty hardware. Well I ruled that out with a unit that came from the factory with a pretty recent firmware. Mikrotik if you read this. Get off your ass and lab this up. If you are happy dealing with residential equipment for the rest of your lives keep doing what you are doing. If you want to move up into the enterprise level you better get on top of this. STEVEN KENNEY DIRECTOR OF GLOBAL CONNECTIVITY & CONTINUITY A: 158 Erie St. N | Leamington ON E: st...@wavedirect.org | P: 519-737-9283 W: www.wavedirect.net From: "Josh Baird" < joshba...@gmail.com > To: "af" < af@af.afmug.com > Sent: Tuesday, December 22, 2020 10:20:36 AM Subject: Re: [AFMUG] Mikrotik 1072 Frustrations Fair enough. But, I have quad port x710's in many ESXi hosts (running CHR and otherwise) - and they are completely fine. On Tue, Dec 22, 2020 at 8:58 AM Steven Kenney < st...@wavedirect.org > wrote: I've been warned away from using the quad cards and speaking from experience I myself have used quad ethernet cards with x86 mikrotiks in the past with little success. All sorts of funky issues. I understand clearly we can use vlans and such in ESXI but I'm talking the pure amount of traffic I'll need will be beyond the demands the bus on the motherboard will be able to handle. I guess I really need to look at the specs. I know PCI Express V6 can do over 100Gbps. But I'll prob use a couple off lease Dells in its own routing cluster. STEVEN KENNEY DIRECTOR OF GLOBAL CONNECTIVITY & CONTINUITY A: 158 Erie St. N | Leamington ON E: st...@wavedirect.org | P: 519-737-9283 W: www.wavedirect.net From: "Josh Baird" < joshba...@gmail.com > To: "af" < af@af.afmug.com > Sent: Monday, December 21, 2020 4:03:21 PM Subject: Re: [AFMUG] Mikrotik 1072 Frustrations The x710 has 4 SFP+ ports. You don't necessarily need dedicated physical interfaces for each router either. Our CHR boxes have a single x710 and several instances of CHR "sharing" the physical interfaces. We use ESXi with a virtual switch trunking VLANs to the CHR instances. On Mon, Dec 21, 2020 at 3:43 PM Steven Kenney < st...@wavedirect.org > wrote: Right those are 2x though for a total of 4 ports. I need more than that, especially if I want to run more than one router on these. Might even consider to see if there is any support for 40Gbps cards or whether a server's bus can handle those speeds with a card. I can aggregate different devices on a switch instead. STEVEN KENNEY DIRECTOR OF GLOBAL CONNECTIVITY & CONTINUITY A: 158 Erie St. N | Leamington ON E: st...@wavedirect.org | P: 519-737-9283 W: www.wavedirect.net From: "Josh Baird" < joshba...@gmail.com > To: "af" < af@af.afmug.com > Sent: Monday, December 21, 2020 3:34:23 PM Subject: Re: [AFMUG] Mikrotik 1072 Frustrations 2x Intel x710. On Mon, Dec 21, 2020 at 2:22 PM Steven Kenney < st...@wavedirect.org > wrote: I've considered this too as a temporary measure because I can get some off lease well maintained Dell servers for peanuts. Problem is finding the right cards when you need 8 sfp+ ports. STEVEN KENNEY DIRECTOR OF GLOBAL CONNECTIVITY & CONTINUITY A: 158 Erie St. N | Leamington ON E: st...@wavedirect.org | P: 519-737-9283 W: www.wavedirect.net From: "TJ Trout" < t...@voltbb.com > To: "af" < af@af.afmug.com > Sent: Monday, December 21, 2020 12:41:53 PM Subject: Re: [AFMUG] Mikrotik 1072 Frustrations We just bought 5x brand new 1072s and they are rebooting by watchdog. Upgraded ros and firmware, Disabled watchdog and then they hard freeze and require a reboot. We are going chr route. 1072 is junk. On Mon, Dec 21, 2020, 9:34 AM Dennis Burgess < dmburg...@linktechs.net > wrote: Hate to say it, but worked. Keep in mind that these units that we have had this issue had been running for 3+ years, so, it was out of warranty anyways. Dennis Burgess, Mikrotik Certified Trainer MTCNA, MTCRE, MTCWE, MTCTCE, MTCINE, MTCSE, HE IPv6 Sage, Cambium ePMP Certified Author of "Learn RouterOS- Second Edition” Link Technologies, Inc -- Mikrotik & WISP Support Services Office : 314-735-0270 Website: http://www.linktechs.net Create Wireless Coverage’s with www.towercoverage.com From: AF < af-boun...@af.afmug.com > On Behalf Of Josh Baird Sent: Monday, December 21, 2020 9:45 AM To: AnimalFarm Microwave Users Group < af@af.afmug.com > Subject: Re: [AFMUG] Mikrotik 1072 Frustrations Haha - "Get a new one" .. only with MT. On Mon, Dec 21, 2020 at 8:22 AM Dennis Burgess < dmburg...@linktechs.net > wrote: Get a new one. That’s what we did, replaced it and the issue went away. Its like the 1072 has some kind of bug, or maybe a leaky cap, that eventually causes them to do this. No matter of fiddling fixes it, we swapped out the unit and have not have the same issue since. We have 1072s running 15gig inbound without issues. So, while I do attest that this is a MT (rather routeros) issue, we don’t have any data to know where it is unless the manufacture can tell us. So far they are tight lipped. Dennis Burgess, Mikrotik Certified Trainer MTCNA, MTCRE, MTCWE, MTCTCE, MTCINE, MTCSE, HE IPv6 Sage, Cambium ePMP Certified Author of "Learn RouterOS- Second Edition” Link Technologies, Inc -- Mikrotik & WISP Support Services Office : 314-735-0270 Website: http://www.linktechs.net Create Wireless Coverage’s with www.towercoverage.com From: AF < af-boun...@af.afmug.com > On Behalf Of Steven Kenney Sent: Monday, December 21, 2020 12:56 AM To: af < af@af.afmug.com > Subject: Re: [AFMUG] Mikrotik 1072 Frustrations Their support is behaving the same way exactly. They told me to take the watchdog off and allow the internet to be frozen (downtime for all my customers) whenever it happens at any random point in the day/night while I rush to log into it with a serial cable. The absurdity of it. Luckily I told him from day one that I have a CONSOLE SERVER and that I log all the serial output of all my switches and routers. But even though I told them that a couple times they still requested I plug in a serial cable and wait for it to freeze. So I'm like .. hellooo there is a serial cable plugged in at all times! I told him there is no output. It just reboots. I even humored him and turned the watchdog off and we had a reboot on the weekend while the cable was plugged in and the console was logged in. I even had logging set to echo for a lot of things and nothing.. just a hard freeze. There is no BGP on this one and the processor is not high leading up to the crash at all. Updated 5 times already and its just a placebo. They need to get their shit together. STEVEN KENNEY DIRECTOR OF GLOBAL CONNECTIVITY & CONTINUITY A: 158 Erie St. N | Leamington ON E: st...@wavedirect.org | P: 519-737-9283 W: https://imsva91-ctp.trendmicro.com:443/wis/clicktime/v1/query?url=www.wavedirect.net&umid=D7E8655E-B6F3-F705-88EA-2A08303811DF&auth=079c058f437b7c6303d36c6513e5e8848d0c5ac4-a20d49e3e6a4f958e127c88b05686fb87748d1dd From: "Colin Stanners" < cstann...@gmail.com > To: "af" < af@af.afmug.com > Sent: Monday, December 21, 2020 12:59:09 AM Subject: Re: [AFMUG] Mikrotik 1072 Frustrations This last year, I've seen a MikroTik CCR1072 switch from long being rock-solid to now having occasional random reboots (from watchdog) or 100% CPU usage, which strangles the BGP process. In the latter case, tools->profile would show the firewall taking 100% of CPU, even after temporarily disabling all firewall filter and NAT rules and connection tracking. Not fun. MT tech support did not seem super helpful or interested, mostly recommending to disable watchdog (unacceptable on a production router) or to upgrade firmware (without specifying the suspected cause of the problem or nature of the fix). Tried 1 update, that didn't seem to help, have now tried another... On Sun, Dec 20, 2020, 11:38 PM Steven Kenney < st...@wavedirect.org > wrote: MIkrotik has been rock solid for me for years. Until this year and the 1072's. Random reboots set off by watchdog timer on all of my 1072's. Some more than others. Threads in the forum all discuss the same problem exactly. Its a connection tracking issue.. however I need connection tracking on one particular router. I've adjusted everything I could. Firmware and board firmware all up to date etc. Happens randomly with low levels of traffic, high levels of traffic, sometimes a couple times a day, sometimes weeks. No DDOS evidence at all from upstream routers. Configs checked and rechecked by third party experts. I graph everything about the Mikrotik and there are no clues or anything abnormal happening before the crash. Plenty of memory, disk space, CPU etc. Replaces all the trannies, power cables and such. Not running BGP only OSPF on the one that is giving me the most trouble. Even have a serial console cable plugged into them to my opengear and set it to log pretty much everything to console including the kernel and nothing. A hard freeze. Then there is Mikrotik support... I've never needed their support before until now. So I put a ticket in and the shitty attitude I'm getting from them seems like they KNOW there is something wrong with the hardware and they are intentionally not being helpful. It is pretty clear to see with all the people reporting this issue that there IS an issue. If this is any indication of how things are going to go with Mikrotik on the newer hardware going forware I think its time to jump to an enterprise level system. Juniper most likely. Shame because they are just about keeping up with the demands with their hardware. Getting closer to 100Gbps etc and ROS7 ... but at their current pace I think we've outgrew them. All the threads discussing this issue has been absolutely quiet when it comes to Mikrotik jumping in to mention or try to help troubleshoot. I think they know they had bad hardware out there and do not want to honor warranties. I've heard rumors of bad batches of 1072's. Anyone else encounter this? STEVEN KENNEY DIRECTOR OF GLOBAL CONNECTIVITY & CONTINUITY A: 158 Erie St. N | Leamington ON E: st...@wavedirect.org | P: 519-737-9283 W: www.wavedirect.net -- AF mailing list AF@af.afmug.com http://af.afmug.com/mailman/listinfo/af_af.afmug.com -- AF mailing list AF@af.afmug.com http://af.afmug.com/mailman/listinfo/af_af.afmug.com -- AF mailing list AF@af.afmug.com http://af.afmug.com/mailman/listinfo/af_af.afmug.com -- AF mailing list AF@af.afmug.com http://af.afmug.com/mailman/listinfo/af_af.afmug.com -- AF mailing list AF@af.afmug.com http://af.afmug.com/mailman/listinfo/af_af.afmug.com -- AF mailing list AF@af.afmug.com http://af.afmug.com/mailman/listinfo/af_af.afmug.com -- AF mailing list AF@af.afmug.com http://af.afmug.com/mailman/listinfo/af_af.afmug.com -- AF mailing list AF@af.afmug.com http://af.afmug.com/mailman/listinfo/af_af.afmug.com -- AF mailing list AF@af.afmug.com http://af.afmug.com/mailman/listinfo/af_af.afmug.com -- AF mailing list AF@af.afmug.com http://af.afmug.com/mailman/listinfo/af_af.afmug.com -- AF mailing list AF@af.afmug.com http://af.afmug.com/mailman/listinfo/af_af.afmug.com -- AF mailing list AF@af.afmug.com http://af.afmug.com/mailman/listinfo/af_af.afmug.com -- AF mailing list AF@af.afmug.com http://af.afmug.com/mailman/listinfo/af_af.afmug.com -- AF mailing list AF@af.afmug.com http://af.afmug.com/mailman/listinfo/af_af.afmug.com -- AF mailing list AF@af.afmug.com http://af.afmug.com/mailman/listinfo/af_af.afmug.com -- AF mailing list AF@af.afmug.com http://af.afmug.com/mailman/listinfo/af_af.afmug.com -- AF mailing list AF@af.afmug.com http://af.afmug.com/mailman/listinfo/af_af.afmug.com
-- AF mailing list AF@af.afmug.com http://af.afmug.com/mailman/listinfo/af_af.afmug.com
