Looks like 6.48 was a big update and had some fixes that could potentially improve the issue. We shall see. Support still isn't helping but I hope I make enough of a stink about it they look deeply into the issue. You don't see people getting random reboots from 1036's and other units.
Check out the threads about this.. Back to 2017!! [ https://forum.mikrotik.com/viewtopic.php?p=790406 | https://forum.mikrotik.com/viewtopic.php?p=790406 ] [ https://forum.mikrotik.com/viewtopic.php?f=3&t=122525&start=100 | https://forum.mikrotik.com/viewtopic.php?f=3&t=122525&start=100 ] There are a few newer ones too. Just keeps falling on deaf ears. [ https://www.wavedirect.net/ | ] [ https://www.facebook.com/ruralhighspeed ] [ https://www.instagram.com/wave.direct/ ] [ https://www.linkedin.com/company/wavedirect-telecommunication/ ] [ https://twitter.com/wavedirect1 ] [ https://www.youtube.com/user/WaveDirect ] STEVEN KENNEY DIRECTOR OF GLOBAL CONNECTIVITY & CONTINUITY A: 158 Erie St. N | Leamington ON E: st...@wavedirect.org | P: 519-737-9283 W: www.wavedirect.net From: "Josh Luthman" <j...@imaginenetworksllc.com> To: "af" <af@af.afmug.com> Sent: Friday, January 1, 2021 6:00:29 PM Subject: Re: [AFMUG] Mikrotik 1072 Frustrations Been a while but I think 5 years is probably a stretch now that I think about it... Josh Luthman 24/7 Help Desk: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373 On Thu, Dec 31, 2020 at 12:09 PM Mark - Myakka Technologies < [ mailto:m...@mailmt.com | m...@mailmt.com ] > wrote: Josh, 5 years ago, I don't think CHR existed -- Best regards, Mark [ mailto:m...@mailmt.com | mailto:m...@mailmt.com ] Myakka Technologies, Inc. [ http://www.myakka.com/ | www.Myakka.com ] ------ Thursday, December 31, 2020, 12:02:54 PM, you wrote: Oh so you went legit x86 instead of CHR? Josh Luthman 24/7 Help Desk: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373 On Thu, Dec 31, 2020 at 11:58 AM Mark - Myakka Technologies < [ mailto:m...@mailmt.com | m...@mailmt.com ] > wrote: Josh, Did some tweaking to the IRQ stuff. These are BGP routers so majority of traffic is limited to two ports. Did some manual assignments of CPU instead of auto. Also went to multi-queue-ethernet-default as interface queue type. I don't think fastpath or fasttrack are supported on x86 -- Best regards, Mark [ mailto:m...@mailmt.com | mailto:m...@mailmt.com ] Myakka Technologies, Inc. [ http://www.myakka.com/ | www.Myakka.com ] ------ Thursday, December 31, 2020, 11:44:49 AM, you wrote: What kind of tweaking for greater speeds? Did you just disable services/packages? Or did you enable fastpath? Josh Luthman 24/7 Help Desk: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373 On Thu, Dec 31, 2020 at 9:53 AM Mark - Myakka Technologies < [ mailto:m...@mailmt.com | m...@mailmt.com ] > wrote: Tushar, I have two that I bought back in 2015. Have the 4 port SFP+ card in it. Using them as BGP routers taking full routes. Both have a 10G uplink to my providers. Took a bit of tweak with MT config, but I can move 8-9 Gig of live traffic on them if needed. Most of the time they are doing anywhere between 2 - 5 Gig. Going to need another one in the near future. Debating if I want to roll my own or just buy one from Baltic. With past performance, I'm leading on just buying one. -- Best regards, Mark [ mailto:m...@mailmt.com | mailto:m...@mailmt.com ] Myakka Technologies, Inc. [ http://www.myakka.com/ | www.Myakka.com ] ------ Thursday, December 31, 2020, 9:24:25 AM, you wrote: Allows two, four ports SFP+ modules Tushar On Dec 31, 2020, at 8:19 AM, Tushar Patel < [ mailto:tpa...@ecpi.com | tpa...@ecpi.com ] > wrote: Has anyone tried Maxwave from Baltic networks? [ https://www.balticnetworks.com/manufacturers/maxxwave/maxxwave-routermaxx-vengeance-8-port-gigabit-core-i7-i7-8700k-hexa-core-router | https://www.balticnetworks.com/manufacturers/maxxwave/maxxwave-routermaxx-vengeance-8-port-gigabit-core-i7-i7-8700k-hexa-core-router ] They have bunch of cards options which can be added Tushar On Dec 30, 2020, at 12:28 PM, Steven Kenney < [ mailto:st...@wavedirect.org | st...@wavedirect.org ] > wrote: Welp replacing it with a brand new 1072 with new transceivers etc lasted about a week before it rebooted itself. Mikrotik basically said RMA it or it may be faulty hardware. Well I ruled that out with a unit that came from the factory with a pretty recent firmware. Mikrotik if you read this. Get off your ass and lab this up. If you are happy dealing with residential equipment for the rest of your lives keep doing what you are doing. If you want to move up into the enterprise level you better get on top of this. STEVEN KENNEY DIRECTOR OF GLOBAL CONNECTIVITY & CONTINUITY A: 158 Erie St. N | Leamington ON E: [ mailto:st...@wavedirect.org | st...@wavedirect.org ] | P: 519-737-9283 W: [ http://www.wavedirect.net/ | www.wavedirect.net ] From: "Josh Baird" < [ mailto:joshba...@gmail.com | joshba...@gmail.com ] > To: "af" < [ mailto:af@af.afmug.com | af@af.afmug.com ] > Sent: Tuesday, December 22, 2020 10:20:36 AM Subject: Re: [AFMUG] Mikrotik 1072 Frustrations Fair enough. But, I have quad port x710's in many ESXi hosts (running CHR and otherwise) - and they are completely fine. On Tue, Dec 22, 2020 at 8:58 AM Steven Kenney < [ mailto:st...@wavedirect.org | st...@wavedirect.org ] > wrote: I've been warned away from using the quad cards and speaking from experience I myself have used quad ethernet cards with x86 mikrotiks in the past with little success. All sorts of funky issues. I understand clearly we can use vlans and such in ESXI but I'm talking the pure amount of traffic I'll need will be beyond the demands the bus on the motherboard will be able to handle. I guess I really need to look at the specs. I know PCI Express V6 can do over 100Gbps. But I'll prob use a couple off lease Dells in its own routing cluster. STEVEN KENNEY DIRECTOR OF GLOBAL CONNECTIVITY & CONTINUITY A: 158 Erie St. N | Leamington ON E: [ mailto:st...@wavedirect.org | st...@wavedirect.org ] | P: 519-737-9283 W: [ http://www.wavedirect.net/ | www.wavedirect.net ] From: "Josh Baird" < [ mailto:joshba...@gmail.com | joshba...@gmail.com ] > To: "af" < [ mailto:af@af.afmug.com | af@af.afmug.com ] > Sent: Monday, December 21, 2020 4:03:21 PM Subject: Re: [AFMUG] Mikrotik 1072 Frustrations The x710 has 4 SFP+ ports. You don't necessarily need dedicated physical interfaces for each router either. Our CHR boxes have a single x710 and several instances of CHR "sharing" the physical interfaces. We use ESXi with a virtual switch trunking VLANs to the CHR instances. On Mon, Dec 21, 2020 at 3:43 PM Steven Kenney < [ mailto:st...@wavedirect.org | st...@wavedirect.org ] > wrote: Right those are 2x though for a total of 4 ports. I need more than that, especially if I want to run more than one router on these. Might even consider to see if there is any support for 40Gbps cards or whether a server's bus can handle those speeds with a card. I can aggregate different devices on a switch instead. STEVEN KENNEY DIRECTOR OF GLOBAL CONNECTIVITY & CONTINUITY A: 158 Erie St. N | Leamington ON E: [ mailto:st...@wavedirect.org | st...@wavedirect.org ] | P: 519-737-9283 W: [ http://www.wavedirect.net/ | www.wavedirect.net ] From: "Josh Baird" < [ mailto:joshba...@gmail.com | joshba...@gmail.com ] > To: "af" < [ mailto:af@af.afmug.com | af@af.afmug.com ] > Sent: Monday, December 21, 2020 3:34:23 PM Subject: Re: [AFMUG] Mikrotik 1072 Frustrations 2x Intel x710. On Mon, Dec 21, 2020 at 2:22 PM Steven Kenney < [ mailto:st...@wavedirect.org | st...@wavedirect.org ] > wrote: I've considered this too as a temporary measure because I can get some off lease well maintained Dell servers for peanuts. Problem is finding the right cards when you need 8 sfp+ ports. STEVEN KENNEY DIRECTOR OF GLOBAL CONNECTIVITY & CONTINUITY A: 158 Erie St. N | Leamington ON E: [ mailto:st...@wavedirect.org | st...@wavedirect.org ] | P: 519-737-9283 W: [ http://www.wavedirect.net/ | www.wavedirect.net ] From: "TJ Trout" < [ mailto:t...@voltbb.com | t...@voltbb.com ] > To: "af" < [ mailto:af@af.afmug.com | af@af.afmug.com ] > Sent: Monday, December 21, 2020 12:41:53 PM Subject: Re: [AFMUG] Mikrotik 1072 Frustrations We just bought 5x brand new 1072s and they are rebooting by watchdog. Upgraded ros and firmware, Disabled watchdog and then they hard freeze and require a reboot. We are going chr route. 1072 is junk. On Mon, Dec 21, 2020, 9:34 AM Dennis Burgess < [ mailto:dmburg...@linktechs.net | dmburg...@linktechs.net ] > wrote: Hate to say it, but worked. Keep in mind that these units that we have had this issue had been running for 3+ years, so, it was out of warranty anyways. Dennis Burgess, Mikrotik Certified Trainer MTCNA, MTCRE, MTCWE, MTCTCE, MTCINE, MTCSE, HE IPv6 Sage, Cambium ePMP Certified Author of "Learn RouterOS- Second Edition” Link Technologies, Inc -- Mikrotik & WISP Support Services Office : 314-735-0270 Website: [ http://www.linktechs.net/ | http://www.linktechs.net ] Create Wireless Coverage’s with [ http://www.towercoverage.com/ | www.towercoverage.com ] From: AF < [ mailto:af-boun...@af.afmug.com | af-boun...@af.afmug.com ] > On Behalf Of Josh Baird Sent: Monday, December 21, 2020 9:45 AM To: AnimalFarm Microwave Users Group < [ mailto:af@af.afmug.com | af@af.afmug.com ] > Subject: Re: [AFMUG] Mikrotik 1072 Frustrations Haha - "Get a new one" .. only with MT. On Mon, Dec 21, 2020 at 8:22 AM Dennis Burgess < [ mailto:dmburg...@linktechs.net | dmburg...@linktechs.net ] > wrote: Get a new one. That’s what we did, replaced it and the issue went away. Its like the 1072 has some kind of bug, or maybe a leaky cap, that eventually causes them to do this. No matter of fiddling fixes it, we swapped out the unit and have not have the same issue since. We have 1072s running 15gig inbound without issues. So, while I do attest that this is a MT (rather routeros) issue, we don’t have any data to know where it is unless the manufacture can tell us. So far they are tight lipped. Dennis Burgess, Mikrotik Certified Trainer MTCNA, MTCRE, MTCWE, MTCTCE, MTCINE, MTCSE, HE IPv6 Sage, Cambium ePMP Certified Author of "Learn RouterOS- Second Edition” Link Technologies, Inc -- Mikrotik & WISP Support Services Office : 314-735-0270 Website: [ http://www.linktechs.net/ | http://www.linktechs.net ] Create Wireless Coverage’s with [ http://www.towercoverage.com/ | www.towercoverage.com ] From: AF < [ mailto:af-boun...@af.afmug.com | af-boun...@af.afmug.com ] > On Behalf Of Steven Kenney Sent: Monday, December 21, 2020 12:56 AM To: af < [ mailto:af@af.afmug.com | af@af.afmug.com ] > Subject: Re: [AFMUG] Mikrotik 1072 Frustrations Their support is behaving the same way exactly. They told me to take the watchdog off and allow the internet to be frozen (downtime for all my customers) whenever it happens at any random point in the day/night while I rush to log into it with a serial cable. The absurdity of it. Luckily I told him from day one that I have a CONSOLE SERVER and that I log all the serial output of all my switches and routers. But even though I told them that a couple times they still requested I plug in a serial cable and wait for it to freeze. So I'm like .. hellooo there is a serial cable plugged in at all times! I told him there is no output. It just reboots. I even humored him and turned the watchdog off and we had a reboot on the weekend while the cable was plugged in and the console was logged in. I even had logging set to echo for a lot of things and nothing.. just a hard freeze. There is no BGP on this one and the processor is not high leading up to the crash at all. Updated 5 times already and its just a placebo. They need to get their shit together. STEVEN KENNEY DIRECTOR OF GLOBAL CONNECTIVITY & CONTINUITY A: 158 Erie St. N | Leamington ON E: [ mailto:st...@wavedirect.org | st...@wavedirect.org ] | P: 519-737-9283 W: [ https://imsva91-ctp.trendmicro.com/wis/clicktime/v1/query?url=www.wavedirect.net&umid=D7E8655E-B6F3-F705-88EA-2A08303811DF&auth=079c058f437b7c6303d36c6513e5e8848d0c5ac4-a20d49e3e6a4f958e127c88b05686fb87748d1dd | https://imsva91-ctp.trendmicro.com:443/wis/clicktime/v1/query?url=www.wavedirect.net&umid=D7E8655E-B6F3-F705-88EA-2A08303811DF&auth=079c058f437b7c6303d36c6513e5e8848d0c5ac4-a20d49e3e6a4f958e127c88b05686fb87748d1dd ] From: "Colin Stanners" < [ mailto:cstann...@gmail.com | cstann...@gmail.com ] > To: "af" < [ mailto:af@af.afmug.com | af@af.afmug.com ] > Sent: Monday, December 21, 2020 12:59:09 AM Subject: Re: [AFMUG] Mikrotik 1072 Frustrations This last year, I've seen a MikroTik CCR1072 switch from long being rock-solid to now having occasional random reboots (from watchdog) or 100% CPU usage, which strangles the BGP process. In the latter case, tools->profile would show the firewall taking 100% of CPU, even after temporarily disabling all firewall filter and NAT rules and connection tracking. Not fun. MT tech support did not seem super helpful or interested, mostly recommending to disable watchdog (unacceptable on a production router) or to upgrade firmware (without specifying the suspected cause of the problem or nature of the fix). Tried 1 update, that didn't seem to help, have now tried another... On Sun, Dec 20, 2020, 11:38 PM Steven Kenney < [ mailto:st...@wavedirect.org | st...@wavedirect.org ] > wrote: MIkrotik has been rock solid for me for years. Until this year and the 1072's. Random reboots set off by watchdog timer on all of my 1072's. Some more than others. Threads in the forum all discuss the same problem exactly. Its a connection tracking issue.. however I need connection tracking on one particular router. I've adjusted everything I could. Firmware and board firmware all up to date etc. Happens randomly with low levels of traffic, high levels of traffic, sometimes a couple times a day, sometimes weeks. No DDOS evidence at all from upstream routers. Configs checked and rechecked by third party experts. I graph everything about the Mikrotik and there are no clues or anything abnormal happening before the crash. Plenty of memory, disk space, CPU etc. Replaces all the trannies, power cables and such. Not running BGP only OSPF on the one that is giving me the most trouble. Even have a serial console cable plugged into them to my opengear and set it to log pretty much everything to console including the kernel and nothing. A hard freeze. Then there is Mikrotik support... I've never needed their support before until now. So I put a ticket in and the shitty attitude I'm getting from them seems like they KNOW there is something wrong with the hardware and they are intentionally not being helpful. It is pretty clear to see with all the people reporting this issue that there IS an issue. If this is any indication of how things are going to go with Mikrotik on the newer hardware going forware I think its time to jump to an enterprise level system. Juniper most likely. Shame because they are just about keeping up with the demands with their hardware. Getting closer to 100Gbps etc and ROS7 ... but at their current pace I think we've outgrew them. All the threads discussing this issue has been absolutely quiet when it comes to Mikrotik jumping in to mention or try to help troubleshoot. I think they know they had bad hardware out there and do not want to honor warranties. I've heard rumors of bad batches of 1072's. Anyone else encounter this? STEVEN KENNEY DIRECTOR OF GLOBAL CONNECTIVITY & CONTINUITY A: 158 Erie St. N | Leamington ON E: [ mailto:st...@wavedirect.org | st...@wavedirect.org ] | P: 519-737-9283 W: [ https://imsva91-ctp.trendmicro.com/wis/clicktime/v1/query?url=http%3a%2f%2fwww.wavedirect.net&umid=D7E8655E-B6F3-F705-88EA-2A08303811DF&auth=079c058f437b7c6303d36c6513e5e8848d0c5ac4-3ca9c8d0ac0d6566b9daf27839af27539533a38d | www.wavedirect.net ] -- AF mailing list [ mailto:AF@af.afmug.com | AF@af.afmug.com ] [ http://af.afmug.com/mailman/listinfo/af_af.afmug.com | http://af.afmug.com/mailman/listinfo/af_af.afmug.com ] -- AF mailing list [ mailto:AF@af.afmug.com | AF@af.afmug.com ] [ http://af.afmug.com/mailman/listinfo/af_af.afmug.com | http://af.afmug.com/mailman/listinfo/af_af.afmug.com ] -- AF mailing list [ mailto:AF@af.afmug.com | AF@af.afmug.com ] [ http://af.afmug.com/mailman/listinfo/af_af.afmug.com | http://af.afmug.com/mailman/listinfo/af_af.afmug.com ] -- AF mailing list [ mailto:AF@af.afmug.com | AF@af.afmug.com ] [ http://af.afmug.com/mailman/listinfo/af_af.afmug.com | http://af.afmug.com/mailman/listinfo/af_af.afmug.com ] -- AF mailing list [ mailto:AF@af.afmug.com | AF@af.afmug.com ] [ http://af.afmug.com/mailman/listinfo/af_af.afmug.com | http://af.afmug.com/mailman/listinfo/af_af.afmug.com ] -- AF mailing list [ mailto:AF@af.afmug.com | AF@af.afmug.com ] [ http://af.afmug.com/mailman/listinfo/af_af.afmug.com | http://af.afmug.com/mailman/listinfo/af_af.afmug.com ] -- AF mailing list [ mailto:AF@af.afmug.com | AF@af.afmug.com ] [ http://af.afmug.com/mailman/listinfo/af_af.afmug.com | http://af.afmug.com/mailman/listinfo/af_af.afmug.com ] -- AF mailing list [ mailto:AF@af.afmug.com | AF@af.afmug.com ] [ http://af.afmug.com/mailman/listinfo/af_af.afmug.com | http://af.afmug.com/mailman/listinfo/af_af.afmug.com ] -- AF mailing list [ mailto:AF@af.afmug.com | AF@af.afmug.com ] [ http://af.afmug.com/mailman/listinfo/af_af.afmug.com | http://af.afmug.com/mailman/listinfo/af_af.afmug.com ] -- AF mailing list [ mailto:AF@af.afmug.com | AF@af.afmug.com ] [ http://af.afmug.com/mailman/listinfo/af_af.afmug.com | http://af.afmug.com/mailman/listinfo/af_af.afmug.com ] -- AF mailing list [ mailto:AF@af.afmug.com | AF@af.afmug.com ] [ http://af.afmug.com/mailman/listinfo/af_af.afmug.com | http://af.afmug.com/mailman/listinfo/af_af.afmug.com ] -- AF mailing list [ mailto:AF@af.afmug.com | AF@af.afmug.com ] [ http://af.afmug.com/mailman/listinfo/af_af.afmug.com | http://af.afmug.com/mailman/listinfo/af_af.afmug.com ] -- AF mailing list [ mailto:AF@af.afmug.com | AF@af.afmug.com ] [ http://af.afmug.com/mailman/listinfo/af_af.afmug.com | http://af.afmug.com/mailman/listinfo/af_af.afmug.com ] -- AF mailing list [ mailto:AF@af.afmug.com | AF@af.afmug.com ] [ http://af.afmug.com/mailman/listinfo/af_af.afmug.com | http://af.afmug.com/mailman/listinfo/af_af.afmug.com ] -- AF mailing list [ mailto:AF@af.afmug.com | AF@af.afmug.com ] [ http://af.afmug.com/mailman/listinfo/af_af.afmug.com | http://af.afmug.com/mailman/listinfo/af_af.afmug.com ] -- AF mailing list AF@af.afmug.com http://af.afmug.com/mailman/listinfo/af_af.afmug.com
-- AF mailing list AF@af.afmug.com http://af.afmug.com/mailman/listinfo/af_af.afmug.com