Edgerouter 4/6P/12/12P vs MikroTik RB3011/RB4011/CCR1009-7G-1C-1S+PC (Bell Fibe FTTH)
Posted: Sat Aug 08, 2020 8:16 am
In the last 2 months I signed up for Bell Canada’s Fibe FTTH service and using their “HomeHub3000” which everyone is trying to bypass in the DSL Reports thread “https://www.dslreports.com/forum/r31118 ... meHub-3000” with various solutions ranging from various routers to Linux servers using specific 2.5Gbps kernel patches to PF Sense/OPNsense to using Edgeswitch ES-16-XG or UniFi US-16-XG which can sync to 2.5Gbps and I guess putting a routing device that talks PPPoE on the same VLAN that the SFP GPON Inserted in the switch, etc but this is overkill for me
Anyway I’m on the 1Gbps plan and I don’t see the need to go to higher 1.5Gbps plan especially when there isn’t any cost effective devices that let you deliver it to your desktops and I don’t really want rackmount form factor devices nor want to spend $800+ for a switch and then similarly or more expensive for a router type device (e.g. $1200-2000 for a PFSense or high end Edgerouter and even a self built box wouldn’t be cheap) I’m not really up to upgrading to a 10G switch at home yet as I don’t need it and my devices are too old anyway at this point
I’m open to a bit of future proofing however so if the device has 1G and 10G SFP ports then I can use with current GPON and when Bell upgrades to 10G GPON but that won’t really help if the router only has 1G ether copper ports or perhaps aggregate 2 ports to a specific device so perhaps no point of going with a single SFP+ 10G port if you can’t get it off the router unless you do VLANs on the SFP+ port to a 10G switch like ES-16-XG where you would put a GPON in the switch and pass into the router over a VLAN and back to another VLAN internally if this doesn’t pose security concerns
i am currently using a Mikrotik hEX RB750GR3 but I don’t think it can do HW offloading with bridging, VLANs, FW/NAT and PPPoE and I recently came across /tools profile that shows CPU activity with respect to thing like bridging, etc and I see the CPU spikes to 80-95% when doing Speedtest so I guess the router can’t keep up and I get about 500-650/600 when the router is running PPPoE bypass behind the Bell HH3000 as the hEX doesn’t have SFP port and I was getting up 725 down max I think when in advanced DMZ setup but I was loosing connectivity but not getting close to 940/940 that people report using a media converter and their router but if I need to buy a media converter I figure may as well buy a router with SFP port and put the $40 towards the router
I started to look at RB4011 as it looks it is much faster CPU then the RB750GR3 (I read in one forum 7x faster, RB3011 is 2x faster) as it has SFP+ port but I don’t think either device can do HW offloading in my particular situation using bridging, VLAN, FW/NAT and PPPoE. I don’t think that pure switching even if VLANs were supported by the RB4011 would help since I don’t think I can say eth0 is my trunk to switch1 (TL-SG1024DE) and eth1 is my trunk to switch2 (TL-SG108E) without bridging the two switches together to make one L2 network
Initially I was dumbfounded when I found the RB4011 didn’t support passive DAC and GPON modules but they recently removed GPON from the list and a few users have confirmed (https://www.dslreports.com/forum/r32467 ... N~start=30) with 6.47.1 that the Bell Huawei SFP ONT GPON works in the RB4011 and they were getting speeds similar to EdgeRouter X SFP (which can only do half duplex gigabit and hence I would be looking at 4/6P/12/12P)
The RB4011 would probably be the simplest path for me to upgrade since I already have a working config and would only have to change my WAN from eth1 to sfpplus1 but I’m concerned that pretty much nothing is hardware offloaded in most people situation where they are using bridging, VLAN, FW/NAT, PPPoE, etc and not using switching between ports or switching between individual VLANs As I said I use the eth2/3 on the hEX currently for connecting two trunks for two switches Does the RB4011 have more than enough CPU horsepower that Mikrotik / SoC developer didn’t bother with HW offloading or only did IPSEC?? I also understand that RouterOS doesn’t currently support OpenVPN UDP nor Wireguard but apparently Mikrotik is working on UDP support and wireguard is in latest 5.x kernels so may be in ROS 7.x
When I look at the Edgerouter 4/6P neither one of these have a switch chip unlike the RB4011 but the RB4011 can’t do HW acceleration using bridging or VLANs anyway I see the Edgerouter 12 has a switch chip like the X/X SFP but then I see threads (https://community.ui.com/questions/Is-E ... 139016b266) where it is questioned as to whether it is actually being utilized and not sure if this is more the development/beta branch as I’ve come to understand it but as I alluded above with respect to Mikrotik I’m not sure if it would be beneficial to me considering I would be using bridging with VLANs with FW/NAT and PPPoE (something typical use) I see in this page (https://help.ui.com/hc/en-us/articles/1 ... Offloading) that Cavium based devices can do most HW offloading Including NAT, VLAN, PPPoE but not bridging (but for some reason Realtek device can do bridging) but not sure if a lot of this would get dropped to CPU because of combination with bridging requiring CPU
I also see in the new development branch 2.0.x that there is discussion about performance issues with HW offload especially with VLANs being turned on and discussion about HW offload in this branch and that Ubiquiti has been left out to dry by the SoC developer for a year and this issue resurfaced between the driver in the 1.x and 2.x branches and UBNT will try to fix it on their own once they get a stable release of 2.0.9.5 but this is a bit concerning and not sure if it will ever be released and if Ubiquiti will just have to choose a different supplier fo next Edgerouter and wonder about sustainability of updating these Edgerouter firmware beyond maintenance releases to get features/functionality
So as you can see there issues on both sides. I think I’m ok with the RB4011 as long as it has more than adequate CPU and can deal without HW offloading and hopefully they add OpenVPN UDP and Wireguard support (neither which I’m currently using but may in near future) Since they just released 6.47.1 to enable GPON support I’m a bit weary and not sure when that will be fully incorporated into ROS 7.x releases
Also I’m quite familiar with RouterOS having used over the last 4 years for my limited scope in a home network (but I’ve done OSPF/BGP/IsIs on ciscos before) and configuration seems more explicit and complete with ability to put comments in things and I have all my static DHCP assignments and not sure how easily it will be to translate this to EdgeOS via CLI with the programming like syntax of the config (using curly braces, etc) as it is a hybrid of C and Cisco IOS syntax, etc I like the elegance of the ROS /export with everything there
I’m not opposed to learning EdgeOS (as long as it is fully documented syntax out there) at all but wonder feature for feature if it will be limited compared to RouterOS From what I can see it looks like EdgeOS doesn’t support commenting except perhaps C style /* */ on the sam line instead of # or ;;;;; style comments and doesn’t save in online router config only when parsing it into router As I said I’ve configured and supported Cisco IOS and CatOS devices before with BGP/OSPF, etc config but now primarily Linux server stuff.
It looks like EdgeOS may have more HW offloading support from looking at the pages but not sure how much of a factor that is as it looks like CPU wise the Edgerouter 4/6P/12 and RB4011 are in same ballpark and I’ve seen people mention they were doing 940/940 without HW offload on the Edge as well.
Looking for simple, elegant solution as I don’t want server racks, etc at home as I don’t have space or time for this and do enough Linux builds during my day job and don’t want to be doing rackmount servers or setting up huge homelab as my basement and house is too small
I don’t really see our household going over the 1Gbps barrier anytime soon as we just wanted faster than our cable 30/5 package primarily for upload reasons (especially with our household in COVID lockdown/hideout) and the third party internet package to go to 60/10 (replaced with 100/10) or 250/20 was priced out of the market by FTTH from Bell at this point but if our regulator opens up TPIA providers to FTTH then I will switch in a heartbeat
The RB3011 is rackmount which is too big for where I’m thinking and it does have fully supported switch chip according to the matrix (https://wiki.mikrotik.com/wiki/Manual:S ... p_Features) but again everything except bridging so not sure if it would turn off all hw offloading because of that
The Mikrotik CCR1019 does come with option for SFP/eth0 combo port along with option for SFP+ but again how to get 10G in/out of the device unless linking internally only to a switch and the price is at least 2x more than the RB4011 and the RB4011 can blow away the CCR1019 in a lot of specs as the CPU is older technology, etc
I was eager to possibly buy the Mikrotik RB4011 today but the local distributor looks to have office closed today possibly due to COVID so that was a wasted trip but started to have second thoughts especially around the HW offloading and CPU breathing room and that GPON support was just added in the last month withv6.47.1 for MikroTik RB4011. .
So I thought I would post for input and if other suggestions of hardware solution Ideally like to future proof as much as possible
Im ok with Edgerouter as well and CLI syntax would be a breeze Having used Cisco but wonder about future EdgeOS development limitations due to driver/kernel / SoC integration issues that I noticed in forums but wondering how easy it will be to translate my elegant hEX RB750GR3 config including my static DHCP leases With comments, etc over and what functionality I might miss
If there is empirical data that shows the RB4011 will blow away my hEX RB750GR3 and run circles around it running a 1-10Gbps internet connection with bridging, VLAN trunk to two switches, FW/NAT, PPPoE and can possibly do OpenVPN UDP and Wireguard in future them in likely in
Open to other solutions but would prefer appliance like router solution in small form factor device
i was also considering UDM Pro after it was suggested in forums but it seems extremely buggy, not polished, doesn’t have PoE ports, and doesn’t have flexibility of EdgeOS or Mikrotik RouterOS to configure things and also doesn’t support 2.5Gbps sync (or 5Gbps) along with 1/10GBPs (yet) and all the Ethernet ports are on same 1Gbps bus so can’t justify higher price, etc and would still require a ES-16-XG or US-16-XG to sync at 2.5Gbps if I was going to spend that kind of money or wait for Bell to release 10Gbps SFP+ GPONs and rate shape/ limit on their concentrator to 1.25, 1.5, 2.5, 5, 10Gbps so we can throw the HH4000 away too
I also have 3 UniFi UAP-AC Pro connected to my switches (TP Link TL-SG1024DE and TL-SG108E).
ok enough rambling on my side and thanks for any input
Anyway I’m on the 1Gbps plan and I don’t see the need to go to higher 1.5Gbps plan especially when there isn’t any cost effective devices that let you deliver it to your desktops and I don’t really want rackmount form factor devices nor want to spend $800+ for a switch and then similarly or more expensive for a router type device (e.g. $1200-2000 for a PFSense or high end Edgerouter and even a self built box wouldn’t be cheap) I’m not really up to upgrading to a 10G switch at home yet as I don’t need it and my devices are too old anyway at this point
I’m open to a bit of future proofing however so if the device has 1G and 10G SFP ports then I can use with current GPON and when Bell upgrades to 10G GPON but that won’t really help if the router only has 1G ether copper ports or perhaps aggregate 2 ports to a specific device so perhaps no point of going with a single SFP+ 10G port if you can’t get it off the router unless you do VLANs on the SFP+ port to a 10G switch like ES-16-XG where you would put a GPON in the switch and pass into the router over a VLAN and back to another VLAN internally if this doesn’t pose security concerns
i am currently using a Mikrotik hEX RB750GR3 but I don’t think it can do HW offloading with bridging, VLANs, FW/NAT and PPPoE and I recently came across /tools profile that shows CPU activity with respect to thing like bridging, etc and I see the CPU spikes to 80-95% when doing Speedtest so I guess the router can’t keep up and I get about 500-650/600 when the router is running PPPoE bypass behind the Bell HH3000 as the hEX doesn’t have SFP port and I was getting up 725 down max I think when in advanced DMZ setup but I was loosing connectivity but not getting close to 940/940 that people report using a media converter and their router but if I need to buy a media converter I figure may as well buy a router with SFP port and put the $40 towards the router
I started to look at RB4011 as it looks it is much faster CPU then the RB750GR3 (I read in one forum 7x faster, RB3011 is 2x faster) as it has SFP+ port but I don’t think either device can do HW offloading in my particular situation using bridging, VLAN, FW/NAT and PPPoE. I don’t think that pure switching even if VLANs were supported by the RB4011 would help since I don’t think I can say eth0 is my trunk to switch1 (TL-SG1024DE) and eth1 is my trunk to switch2 (TL-SG108E) without bridging the two switches together to make one L2 network
Initially I was dumbfounded when I found the RB4011 didn’t support passive DAC and GPON modules but they recently removed GPON from the list and a few users have confirmed (https://www.dslreports.com/forum/r32467 ... N~start=30) with 6.47.1 that the Bell Huawei SFP ONT GPON works in the RB4011 and they were getting speeds similar to EdgeRouter X SFP (which can only do half duplex gigabit and hence I would be looking at 4/6P/12/12P)
The RB4011 would probably be the simplest path for me to upgrade since I already have a working config and would only have to change my WAN from eth1 to sfpplus1 but I’m concerned that pretty much nothing is hardware offloaded in most people situation where they are using bridging, VLAN, FW/NAT, PPPoE, etc and not using switching between ports or switching between individual VLANs As I said I use the eth2/3 on the hEX currently for connecting two trunks for two switches Does the RB4011 have more than enough CPU horsepower that Mikrotik / SoC developer didn’t bother with HW offloading or only did IPSEC?? I also understand that RouterOS doesn’t currently support OpenVPN UDP nor Wireguard but apparently Mikrotik is working on UDP support and wireguard is in latest 5.x kernels so may be in ROS 7.x
When I look at the Edgerouter 4/6P neither one of these have a switch chip unlike the RB4011 but the RB4011 can’t do HW acceleration using bridging or VLANs anyway I see the Edgerouter 12 has a switch chip like the X/X SFP but then I see threads (https://community.ui.com/questions/Is-E ... 139016b266) where it is questioned as to whether it is actually being utilized and not sure if this is more the development/beta branch as I’ve come to understand it but as I alluded above with respect to Mikrotik I’m not sure if it would be beneficial to me considering I would be using bridging with VLANs with FW/NAT and PPPoE (something typical use) I see in this page (https://help.ui.com/hc/en-us/articles/1 ... Offloading) that Cavium based devices can do most HW offloading Including NAT, VLAN, PPPoE but not bridging (but for some reason Realtek device can do bridging) but not sure if a lot of this would get dropped to CPU because of combination with bridging requiring CPU
I also see in the new development branch 2.0.x that there is discussion about performance issues with HW offload especially with VLANs being turned on and discussion about HW offload in this branch and that Ubiquiti has been left out to dry by the SoC developer for a year and this issue resurfaced between the driver in the 1.x and 2.x branches and UBNT will try to fix it on their own once they get a stable release of 2.0.9.5 but this is a bit concerning and not sure if it will ever be released and if Ubiquiti will just have to choose a different supplier fo next Edgerouter and wonder about sustainability of updating these Edgerouter firmware beyond maintenance releases to get features/functionality
So as you can see there issues on both sides. I think I’m ok with the RB4011 as long as it has more than adequate CPU and can deal without HW offloading and hopefully they add OpenVPN UDP and Wireguard support (neither which I’m currently using but may in near future) Since they just released 6.47.1 to enable GPON support I’m a bit weary and not sure when that will be fully incorporated into ROS 7.x releases
Also I’m quite familiar with RouterOS having used over the last 4 years for my limited scope in a home network (but I’ve done OSPF/BGP/IsIs on ciscos before) and configuration seems more explicit and complete with ability to put comments in things and I have all my static DHCP assignments and not sure how easily it will be to translate this to EdgeOS via CLI with the programming like syntax of the config (using curly braces, etc) as it is a hybrid of C and Cisco IOS syntax, etc I like the elegance of the ROS /export with everything there
I’m not opposed to learning EdgeOS (as long as it is fully documented syntax out there) at all but wonder feature for feature if it will be limited compared to RouterOS From what I can see it looks like EdgeOS doesn’t support commenting except perhaps C style /* */ on the sam line instead of # or ;;;;; style comments and doesn’t save in online router config only when parsing it into router As I said I’ve configured and supported Cisco IOS and CatOS devices before with BGP/OSPF, etc config but now primarily Linux server stuff.
It looks like EdgeOS may have more HW offloading support from looking at the pages but not sure how much of a factor that is as it looks like CPU wise the Edgerouter 4/6P/12 and RB4011 are in same ballpark and I’ve seen people mention they were doing 940/940 without HW offload on the Edge as well.
Looking for simple, elegant solution as I don’t want server racks, etc at home as I don’t have space or time for this and do enough Linux builds during my day job and don’t want to be doing rackmount servers or setting up huge homelab as my basement and house is too small
I don’t really see our household going over the 1Gbps barrier anytime soon as we just wanted faster than our cable 30/5 package primarily for upload reasons (especially with our household in COVID lockdown/hideout) and the third party internet package to go to 60/10 (replaced with 100/10) or 250/20 was priced out of the market by FTTH from Bell at this point but if our regulator opens up TPIA providers to FTTH then I will switch in a heartbeat
The RB3011 is rackmount which is too big for where I’m thinking and it does have fully supported switch chip according to the matrix (https://wiki.mikrotik.com/wiki/Manual:S ... p_Features) but again everything except bridging so not sure if it would turn off all hw offloading because of that
The Mikrotik CCR1019 does come with option for SFP/eth0 combo port along with option for SFP+ but again how to get 10G in/out of the device unless linking internally only to a switch and the price is at least 2x more than the RB4011 and the RB4011 can blow away the CCR1019 in a lot of specs as the CPU is older technology, etc
I was eager to possibly buy the Mikrotik RB4011 today but the local distributor looks to have office closed today possibly due to COVID so that was a wasted trip but started to have second thoughts especially around the HW offloading and CPU breathing room and that GPON support was just added in the last month withv6.47.1 for MikroTik RB4011. .
So I thought I would post for input and if other suggestions of hardware solution Ideally like to future proof as much as possible
Im ok with Edgerouter as well and CLI syntax would be a breeze Having used Cisco but wonder about future EdgeOS development limitations due to driver/kernel / SoC integration issues that I noticed in forums but wondering how easy it will be to translate my elegant hEX RB750GR3 config including my static DHCP leases With comments, etc over and what functionality I might miss
If there is empirical data that shows the RB4011 will blow away my hEX RB750GR3 and run circles around it running a 1-10Gbps internet connection with bridging, VLAN trunk to two switches, FW/NAT, PPPoE and can possibly do OpenVPN UDP and Wireguard in future them in likely in
Open to other solutions but would prefer appliance like router solution in small form factor device
i was also considering UDM Pro after it was suggested in forums but it seems extremely buggy, not polished, doesn’t have PoE ports, and doesn’t have flexibility of EdgeOS or Mikrotik RouterOS to configure things and also doesn’t support 2.5Gbps sync (or 5Gbps) along with 1/10GBPs (yet) and all the Ethernet ports are on same 1Gbps bus so can’t justify higher price, etc and would still require a ES-16-XG or US-16-XG to sync at 2.5Gbps if I was going to spend that kind of money or wait for Bell to release 10Gbps SFP+ GPONs and rate shape/ limit on their concentrator to 1.25, 1.5, 2.5, 5, 10Gbps so we can throw the HH4000 away too
I also have 3 UniFi UAP-AC Pro connected to my switches (TP Link TL-SG1024DE and TL-SG108E).
ok enough rambling on my side and thanks for any input