CHR at XCP-NG hypervisor for BGP

lucianotres · Fri Mar 06, 2020 7:14 am

Hi all.

Like everyone I'm facing issues to take a full BGP routing table with my peers on CCR. I've a CCR1036 with 20 peers filtered to default route only, on that configuration everything runs fine. The things changes when I start to take down the filters to get full routing table. Average of 15~20% CPU, but always 1 core at 100%. I know, that it's a well know issue.

Well, lot a discussions in the community and always point to ROS 7. Sure, that should be a solution soon, I expect. The fact it's that I can't wait too long, so I'm looking for another solutions. Buying the CCR1072 isn't solve anything, is not a core number issue. So I saw the Kevin Mayrs's presentation at MUM that opened my mind to CHR. I've in my datacenter a HP server with 10GB port and some resource to give a try. It's not a VMware, KVM or Hyper-V but a XCP-ng hypervisor (XenServer based). Nothing strange at the moment, Installed and configured to receive one full routing table. As expected the table goes up quick and no CPU stuck.

Now the part that I don't get. I've trouble with some bandwidth tests, sending data to VM router it's working fine as can see below, but getting out isn't well. I can't get more than 1.1Gbps on TX. I tried to change the GRO on/off, but doesn't change. I'm testing with another 4 CCR1036 with 10GB into a switch to produce a maximum btest possible. I'm new with CHR, am I do someting wrong or missing something?

Here is my setup and the results as well:

/system resource pr
                   uptime: 1h48m3s
                  version: 6.46.4 (stable)
               build-time: Feb/21/2020 11:26:37
              free-memory: 3161.7MiB
             total-memory: 3680.0MiB
                      cpu: Intel(R)
                cpu-count: 16
            cpu-frequency: 2593MHz
                 cpu-load: 1%
           free-hdd-space: 987.4MiB
          total-hdd-space: 1020.1MiB
  write-sect-since-reboot: 9256
         write-sect-total: 9257
        architecture-name: x86_64
               board-name: CHR
                 platform: MikroTik
                 
/ip route print count-only 
761793

/interface ethernet pr detail 
Flags: X - disabled, R - running, S - slave 
 0 R  name="ether1" default-name="ether1" mtu=1500 mac-address=16:DB:9C:1B:F7:65 orig-mac-address=16:DB:9C:1B:F7:65 
      arp=enabled arp-timeout=auto loop-protect=default loop-protect-status=off loop-protect-send-interval=5s 
      loop-protect-disable-time=5m disable-running-check=no auto-negotiation=yes 
      advertise=10M-half,10M-full,100M-half,100M-full,1000M-half,1000M-full,2500M-full,5000M-full,10000M-full 
      full-duplex=yes cable-settings=default speed=10Gbps 

 1 R  name="ether2" default-name="ether2" mtu=1500 mac-address=FE:1E:70:43:81:1E orig-mac-address=FE:1E:70:43:81:1E 
      arp=enabled arp-timeout=auto loop-protect=default loop-protect-status=off loop-protect-send-interval=5s 
      loop-protect-disable-time=5m disable-running-check=no auto-negotiation=yes 
      advertise=10M-half,10M-full,100M-half,100M-full,1000M-half,1000M-full,2500M-full,5000M-full,10000M-full 
      full-duplex=yes cable-settings=default speed=100Mbps

And here RX btest result and below the TX:

/interface monitor-traffic ether1 
                         name:    ether1
        rx-packets-per-second:   673 869
           rx-bits-per-second:   8.1Gbps
     fp-rx-packets-per-second:         0
        fp-rx-bits-per-second:      0bps
          rx-drops-per-second:         0
         rx-errors-per-second:         0
        tx-packets-per-second:        31
           tx-bits-per-second:  84.0kbps
     fp-tx-packets-per-second:         0
        fp-tx-bits-per-second:      0bps
          tx-drops-per-second:         0
    tx-queue-drops-per-second:         0
         tx-errors-per-second:         0
-- [Q quit|D dump|C-z pause]

/interface monitor-traffic ether1 
                         name:     ether1
        rx-packets-per-second:         64
           rx-bits-per-second:   50.4kbps
     fp-rx-packets-per-second:          0
        fp-rx-bits-per-second:       0bps
          rx-drops-per-second:          0
         rx-errors-per-second:          0
        tx-packets-per-second:     90 257
           tx-bits-per-second:  1095.9...
     fp-tx-packets-per-second:          0
        fp-tx-bits-per-second:       0bps
          tx-drops-per-second:          0
    tx-queue-drops-per-second:     31 721
         tx-errors-per-second:          0
-- [Q quit|D dump|C-z pause]

Here the screenshots of NIC and CPU load at RX and TX tests (in that sequence)

Any help will be welcome.
Thanks,
Luciano Tres

lucianotres · Mon Mar 09, 2020 6:53 am

Well, after a deep dive into the issue, I found out what was.
Before just a little tip for who is gonna try Xen or XCP. This wiki it's very very helpfull. https://wiki.xen.org/wiki/Network_Throu ... ance_Guide
So, I was focused on the hypervisor, but after try a bunch of tweaks I came with the mostly simple solution. Why not start all from the beginning?
I created a new VM, installed same CHR image and did the same configurations. But this time the results are much better. TX and RX are getting 9.7Gbps as expected (limited by interface). I did a btest on 127.0.0.1 to stress out the CPU, results below:

[luciano@INFOMAC-BGP2] > tool bandwidth-test protocol=udp direction=transmit address=127.0.0.1 
                    ;;; results can be limited by cpu, note that traffic generation/termination 
                        performance might not be representative of forwarding performance
                status: running
              duration: 56s
            tx-current: 40.0Gbps
  tx-10-second-average: 40.1Gbps
      tx-total-average: 38.8Gbps
           random-data: no
             direction: transmit
               tx-size: 1500
      connection-count: 20
        local-cpu-load: 86%
       remote-cpu-load: 86%

But why everything are working now? The only thing I did differently was the license. At the first time I made a mistake, put the license to P1, to correct I generated a new ID and defined to P10. Looks like the system didn't take well the new license. In the second VM I went right to p-unlimited.

Now with everything working I can put some real traffic into the VM, and it's exactly what I'm doing. So far, so good, after some hours 1.4Gbps of throughput and avarege CPU at 13~15%. One of sixteen cores goes to 40% once in a while (as expected). Two IPv4 and two IPv6 BGP peers receiving full routing table with around 1.5 million routes. At least I can contribute with that numbers to the community after I messed up with the licenses.

Thanks,
Luciano Tres

gulzoo · Mon Mar 09, 2020 9:49 am

Hi,

As you stated the problem was resolved with licensing. With P1 license you are limited 1Gbps upload per interface. P10 you have a limit of 10Gbps and with P-unlimited you have no limitations.

You can check the CHR licenses on the following link: https://wiki.mikrotik.com/wiki/Manual:CHR#Paid_licenses

You can also generate a 60 day trial license with no limitations.

Best regards,
Andrej Turk

lucianotres · Mon Mar 09, 2020 3:37 pm

Hi gulzzo,

Yeah, I understood that. Soon as I saw the mistake, I did the following:

[luciano@INFOMAC-BGP2] > /system license generate-new-id 
Your license will become invalid. Continue, yes? [y/N]: 

[luciano@INFOMAC-BGP2] > /system license renew level=p10 account=xxxx password=xxxx

I assumed that was changed successfully, but don't.

But now everything is fine, thanks by reply
Luciano Tres

CHR at XCP-NG hypervisor for BGP

CHR at XCP-NG hypervisor for BGP

Re: CHR at XCP-NG hypervisor for BGP

Re: CHR at XCP-NG hypervisor for BGP

Re: CHR at XCP-NG hypervisor for BGP

Who is online