XenServer 6.5, RB44Ge High CPU utilization on Ethernet

gerakon · Fri Jul 24, 2015 5:45 am

I've installed Xenserver 6.5 on a Gigabyte GA-MA770-UD3 Rev 1 mainboard, AMD Phenom 1035 6 Core, 5Gb RAM. I created a VM with 1 processor, 512Mb RAM, added the first 3 interfaces from an RB44Ge and converted the config from my RB2011 to match the ethernet ports on the VM. I was surprised that XenServer recognized the RB44Ge as I had read that it didn't have drivers for it, but that must be new with 6.5. I'm seeing high CPU utiliztion on "Ethernet" as shown in the screenshot below. Most of the traffic in the picture is from 2 Hikvision IP Cameras coming across an IPSEC tunnel recording to a Synology Diskstation on the LAN. This is somewhat of a test as I would like to put a RB44Ge in a dual six core Xeon Xen Host at the other end of the IPSEC tunnel assuming I can get this to work well. I was expecting lower CPU utilization.

Am I seeing this because of the lack of support for XenTools in RouterOS? Would using PCI passthrough for the RB44Ge card thus using the native RouterOS drivers for the RB44Ge fix this? Unfortunately this mainboard doesn't support IOMMU required for PCI passthrough though the Dual Xeon system I'm pretty sure does.

XenMTHighCPU.jpg



/interface ethernet
set [ find default-name=ether1 ] comment=WAN mac-address=C4:64:13:00:00:00
set [ find default-name=ether2 ] comment=LAN
set [ find default-name=ether3 ] comment=Wireless

/ip address
add address=172.16.10.1/24 interface=ether2 network=172.16.10.0
add address=172.16.12.1/24 interface=ether3 network=172.16.12.0

/ip ipsec proposal
set [ find default=yes ] enc-algorithms=3des
add name=S2SProposal
add enc-algorithms=3des name=S2S-2Proposal

/ip pool
add name=dhcp_poolTrusted ranges=172.16.10.100-172.16.10.150
add name=dhcp_poolUntrusted ranges=172.16.12.100-172.16.12.200

/ip dhcp-server
add address-pool=dhcp_poolTrusted disabled=no interface=ether2 lease-time=12h name=dhcpTrusted
add address-pool=dhcp_poolUntrusted disabled=no interface=ether3 lease-time=8h name=dhcpUntrusted


/ppp profile
set [ find name=default ] name=default
set [ find name=default-encryption ] name=default-encryption

/ip dhcp-client
add comment="default configuration" dhcp-options=hostname,clientid disabled=no interface=ether1

/ip dhcp-server lease
add address=172.16.10.18 client-id=1:8c:a9:82:a3:8e:4e mac-address=8C:A9:82:A3:8E:4E
add address=172.16.10.23 client-id=1:0:30:1b:bc:38:7e comment=HTPC mac-address=00:30:1B:BC:38:7E
add address=172.16.12.203 always-broadcast=yes client-id=1:8:fd:e:98:e1:ad comment=Cell2 mac-address=08:FD:0E:98:E1:AD server=dhcpUntrusted
add address=172.16.10.50 client-id=00:15:26:06:7C:8B comment=ProControl mac-address=00:15:26:06:7C:8B
add address=172.16.12.202 client-id=1:40:fc:89:d:28:68 comment=Cell1 mac-address=40:FC:89:0D:28:68 server=dhcpUntrusted
add address=172.16.12.200 comment="Sanford Scale Bridge" mac-address=64:5D:D7:01:E2:A4 server=dhcpUntrusted
add address=172.16.10.22 client-id=1:bc:ee:7b:73:6d:58 comment=Eo mac-address=BC:EE:7B:73:6D:58 server=dhcpTrusted
add address=172.16.10.53 client-id=1:0:80:91:ab:79:84 comment="Oki MC770" mac-address=00:80:91:AB:79:84 server=dhcpTrusted
add address=172.16.10.54 client-id=1:0:30:1b:bc:38:7e comment=HTPC mac-address=00:30:1B:BC:38:7E server=dhcpTrusted
add address=172.16.10.21 comment=Main7 mac-address=74:D4:35:16:E4:54 server=dhcpTrusted
add address=172.16.12.201 always-broadcast=yes client-id=1:d0:22:be:3:b8:c6 comment="Mom Cell" mac-address=D0:22:BE:03:B8:C6 server=dhcpUntrusted

/ip dhcp-server network
add address=172.16.10.0/24 dns-server=172.16.10.1 gateway=172.16.10.1
add address=172.16.12.0/24 dns-server=172.16.12.1 gateway=172.16.12.1

/ip dns
set allow-remote-requests=yes servers=3.3.4.4,3.3.4.5

/ip dns static
add address=172.16.10.1 name=router


/ip firewall address-list
add address=172.16.10.20 list=management-servers
add address=172.16.10.21 list=management-servers
add address=10.2.2.96 list=management-servers
add address=10.2.2.97 list=management-servers
add address=1.1.1.1 list=management-servers
add address=1.1.1.1-2.2.2.2 list=camerausers

add address=172.16.10.18 list=management-servers
add address=172.16.10.21 list=serveraccess
add address=172.16.10.22 list=serveraccess
add address=172.16.10.23 list=serveraccess
add address=172.16.11.20 list=Server

/ip firewall filter
add action=drop chain=forward dst-address=0.0.0.0/0 log=yes log-prefix=ServerTryingToGetOut src-address=172.16.10.20
add chain=input comment="Allow ICMP" protocol=icmp
add chain=input comment="Allow Established Connections Input" connection-state=established
add chain=input comment="Allow Related Connections Input" connection-state=related
add chain=forward comment="Allow Established Connections Forward" connection-state=established
add chain=forward comment="Allow Related Connections Forward" connection-state=related
add chain=input comment="Allow Management to connect via 21,22,23,80,443,8291" dst-port=21,22,23,80,443,8291 protocol=tcp src-address-list=management-servers
add action=drop chain=input comment="default configuration" in-interface=ether1
add action=drop chain=input comment="DROP ALL WINBOX REQUEST By MAC Address" dst-port=20561 protocol=udp
add action=drop chain=input comment="block mikrotik discovery" dst-port=5678 protocol=udp
add action=drop chain=forward comment="default configuration" connection-state=invalid
add action=drop chain=input dst-port=21,22,23,80,443,8291 protocol=tcp
add action=drop chain=input dst-port=53 in-interface=ether1 protocol=udp
add action=drop chain=input dst-port=53 in-interface=ether1 protocol=tcp
add action=drop chain=forward in-interface=ether3 log=yes log-prefix="Untrusted attempt - " out-interface=!ether1

/ip firewall nat
add chain=srcnat dst-address=10.3.3.0/24 src-address=172.16.10.0/24
add chain=srcnat dst-address=10.2.2.0/24 src-address=172.16.10.0/24
add chain=srcnat dst-address=10.2.4.0/24 src-address=172.16.10.0/24

add action=dst-nat chain=dstnat comment=Minecraft dst-address=3.3.3.3 dst-port=25252 protocol=tcp to-addresses=172.16.10.21 to-ports=25565
add action=dst-nat chain=dstnat comment=Steam27015UDP dst-address=3.3.3.3 dst-port=27015 protocol=udp to-addresses=172.16.10.21 to-ports=27015
add action=dst-nat chain=dstnat comment=Steam27015TCP dst-address=3.3.3.3 dst-port=27015 protocol=tcp to-addresses=172.16.10.21 to-ports=27015
add action=dst-nat chain=dstnat comment=UT7777UDP dst-address=3.3.3.3 dst-port=7777 protocol=udp to-addresses=172.16.10.21 to-ports=7777
add action=dst-nat chain=dstnat comment=UT7778UDP dst-address=3.3.3.3 dst-port=7778 protocol=udp to-addresses=172.16.10.21 to-ports=7778
add action=dst-nat chain=dstnat comment=UT7777TCP dst-address=3.3.3.3 dst-port=7777 protocol=tcp to-addresses=172.16.10.21 to-ports=7777
add action=dst-nat chain=dstnat comment=UT7778TCP dst-address=3.3.3.3 dst-port=7778 protocol=tcp to-addresses=172.16.10.21 to-ports=7778
add action=dst-nat chain=dstnat comment=UT27900UDP dst-address=3.3.3.3 dst-port=27900 protocol=udp to-addresses=172.16.10.21 to-ports=27900
add action=dst-nat chain=dstnat comment=UT27900TCP dst-address=3.3.3.3 dst-port=27900 protocol=tcp to-addresses=172.16.10.21 to-ports=27900
add action=dst-nat chain=dstnat comment=SMTP25TCP dst-address=3.3.3.3 dst-port=25 protocol=tcp to-addresses=172.16.10.3 to-ports=25
add action=dst-nat chain=dstnat comment=IMAPSSL993 dst-address=3.3.3.3 dst-port=993 protocol=tcp to-addresses=172.16.10.3 to-ports=993
add action=dst-nat chain=dstnat comment=SMTPSSL465 dst-address=3.3.3.3 dst-port=465 protocol=tcp to-addresses=172.16.10.3 to-ports=465
add action=dst-nat chain=dstnat comment=SSL443TCP dst-address=3.3.3.3 dst-port=443 protocol=tcp to-addresses=172.16.10.3 to-ports=443
add action=dst-nat chain=dstnat comment=WWW80 dst-address=3.3.3.3 dst-port=80 protocol=tcp to-addresses=172.16.10.3 to-ports=80
add action=dst-nat chain=dstnat comment="UTRedirected Maps 3128" dst-address=3.3.3.3 dst-port=3128 protocol=tcp to-addresses=172.16.10.3 to-ports=80
add action=masquerade chain=srcnat out-interface=ether1
add action=dst-nat chain=dstnat comment="GameServer - Teamspeak UDP9987" dst-address=3.3.3.3 dst-port=9987 protocol=udp to-addresses=172.16.10.21 to-ports=9987

/ip ipsec peer
add address=5.5.5.5/32 enc-algorithm=aes-128 secret=000000000000000000
add address=4.4.4.4/32 secret=0000000000000000

/ip ipsec policy
set 0 dst-address=0.0.0.0/0 src-address=0.0.0.0/0
add dst-address=10.2.2.0/24 proposal=S2SProposal sa-dst-address=5.5.5.5 sa-src-address=3.3.3.3 src-address=172.16.10.0/24 tunnel=yes
add dst-address=10.3.3.0/24 proposal=S2S-2Proposal sa-dst-address=4.4.4.4 sa-src-address=3.3.3.3 src-address=172.16.10.0/24 tunnel=yes
add dst-address=10.2.4.0/24 proposal=S2SProposal sa-dst-address=5.5.5.5 sa-src-address=3.3.3.3 src-address=172.16.10.0/24 tunnel=yes


/system clock
set time-zone-autodetect=no time-zone-name=America/Chicago

/system clock manual
set dst-delta=+01:00 dst-end="nov/02/2014 00:02:00" dst-start="mar/09/2014 00:02:00" time-zone=-06:00

/system identity
set name=x86

/system ntp client
set enabled=yes primary-ntp=82.165.36.179 secondary-ntp=199.241.31.224

/tool graphing interface
add allow-address=172.16.10.0/24 interface=ether1
add allow-address=172.16.10.0/24 interface=ether2
add allow-address=172.16.10.0/24 interface=ether3

/tool graphing resource
add allow-address=172.16.10.0/24

/tool mac-server
set [ find default=yes ] disabled=yes

add interface=ether2
add interface=ether3


/tool mac-server mac-winbox
set [ find default=yes ] disabled=yes
add interface=ether2
add interface=ether3

/tool mac-server ping
set enabled=no
/tool romon
set secrets=""
/tool romon port
add disabled=no

gerakon · Wed Jul 29, 2015 12:41 am

I threw another hard drive in the server and installed RouterOS directly instead of virtualizing through XenServer and my ethernet utilization problem went away. Now I see that Mikrotik has announced/released a test version of Cloud Hosted Router. I wonder if this will run on XenServer? I'll have to look into it. Just thought I would post this in case anyone else runs into this problem.

XenMTLowCPU.jpg

mdenadal · Tue Aug 04, 2015 11:32 am

Hi Gerakon,
in general RouterOS is not so happy under XS, but it works indeed.

The high CPU issue can be solved using VirtIO ethernet drivers, follow this guide: http://www.netservers.co.uk/articles/op ... 00_gigabit and switch "e1000" with "virtio".
I easily made a script to select nic emulation using a VM custom attribute (editable through XenCenter).

You can enable Xenmotion too: https://mperedim.wordpress.com/2013/03/ ... 10-guests/
but be aware: it works but it's really a dirty hack.

Cheers,
Massimo

gerakon · Tue Aug 04, 2015 8:13 pm

Hi Massimo,
Thanks for the info. I'll keep it in mind, but I think I'll wait and see how this CHR thing works. If I get brave, I'll throw the RB44Ge in XENServer at the office (SuperMicro mainboard) and see if I can get the IOMMU/PCI passthrough to work. I would think it should use the native RouterOS drivers then.
Thanks,
Gerakon

mdenadal · Tue Aug 04, 2015 8:32 pm

PCI passthrough is an option for sure, but without any motion capabilities and without SR-IOV (the RB44Ge is not SR-IOV capable) the nics are sticky with the RouterOS vm.
Not a good option to me: you need 2 nics and a switch to let a vm talk to the ros vm.

Believe me, give virtio a try.

Cheers
Massimo

XenServer 6.5, RB44Ge High CPU utilization on Ethernet

XenServer 6.5, RB44Ge High CPU utilization on Ethernet

Re: XenServer 6.5, RB44Ge High CPU utilization on Ethernet

Re: XenServer 6.5, RB44Ge High CPU utilization on Ethernet

Re: XenServer 6.5, RB44Ge High CPU utilization on Ethernet

Re: XenServer 6.5, RB44Ge High CPU utilization on Ethernet

Who is online