Looking for Docker container ideas for RouterOS

Tue Oct 26, 2021 10:42 am

So when 7.1rc3 with Docker support came out, i instantly jumped to obvious things to use it to run stand alone DNS server - feature that is missing in RouterOS itself.
But i must admit i strugle to find any usage of this feature on a router, most of the things i would run in container i would run on x64 server, that has much more resources available.

so what would be the thing you will use Docker support for on a router and why?

ech1965 · Tue Oct 26, 2021 11:52 am

reverse proxy ( ssl terminaison ) with letsencrypt ( haproxy or traefik )

mkx · Tue Oct 26, 2021 1:54 pm

The list of services that might be run in containers is endless. Just compile list of services that people mentioned in numerous wish-list posts.

The problem is that most (if not all) RB devices are not really fit for running (full-blown) containers either due to RAM shortage or due to storage shortage (or both) and that inclusion of those services in ROS itself would make better use of scarce resources. Specially so as containers run on router don't really get the integration some people would like to see (e.g. using same L2/L3 interfaces as core ROS does) ...
As already mentioned it would make much greater sense that people would be running a general-purpose gadget (can be a humble rPI or a decent server) running all the wanted services ... either natively (e.g. in bare-metal linux OS) or in containers. But some people outright reject such solution (it's beyond my comprehension as to why), wanting to reduce number of gadgets run in their network at all costs.

Tue Oct 26, 2021 2:16 pm

Agree with mkx.

Most RouterBoards are not suited for this purpose because of RAM, storage (some boards don't even allow external storage), processing power.
Which does not mean it can not be done.
But it's not because something can be done, it might not be better done using something else.

Looking at it from another angle:
if all you have is a hammer, you tend to see every problem as a nail.

Routerboards CAN be used as Docker environments, yes.
But a lot of other (and even cheaper) devices are far more better equipped to serve this purpose.
Even a lot of existing NAS devices can nowadays be used for this goal so it's not that you HAVE to add another gadget in the mix (which on itself still should not be a problem).

ConnyMercier · Tue Oct 26, 2021 4:56 pm

I don't think the Docker-Feature will be widely used in the "Entreprise" enviroment.
In other words, no Entreprise with On-Site virtualization capability, will move there DNS-Server to a Mikrotik-Device!

It just doesn't make any sense =)

having said that,
I use Mikrotik-Devices all the time to solve problem no other Router can !
So the Docker-Feature is a welcome Tool for my Toolbox

Possible use cases...

1. Replace All-in-One Router
I find it difficult sometimes to implement a Mikrotik-Routers in a SOHO or SME's environment.
There is always a small Feature or Service from the All-in-One Router that Mikrotik can't provide.
Dockers may help to solve this problem.

2. Raspberry Pi replacement
I know a lot of IT-Enthusiast wo use RPi's at home as some sort of low performance Server.
like ioT-Server, Web-Server, Data and Logging, Authentication (freeradius) , DNS (pihole) etc...

I for exemple need at home very small MariaDB-Server for a software I use (Devolutions RDM).
My handfull of devices usually make each 1 SQL-Request a day . Since i don't have a Server or NAS,
Installing Dockers on my Router could be a very nice solution.

3. miscellaneous Idees...

ISP could have a small Web-Server installed on Client devices.
Client could access the Website to
- purchase more High-Speed volume
- Change basic setting like WLAN-Password, SSID , etc...
- See some Usage and Statistic

School could have a small Web-Server installed on the Classroom-Router or switch.
Teacher connected to the Classroom-Network opens the webbrowser and log-in to the server.
The Small Web-Application can,
- Activate Wifi for the Students
- Activate Internet for all or some computers
- Automatically disable Internet and Wifi at the end of the Class or at the end of the day.

Hominidae · Tue Oct 26, 2021 5:57 pm

2. Raspberry Pi replacement
I know a lot of IT-Enthusiast wo use RPi's at home as some sort of low performance Server.
like ioT-Server, Web-Server, Data and Logging, Authentication (freeradius) , DNS (pihole) etc...

+1
I'd like to add mqtt-broker/mosquitto and unbound to the list

semaja2 · Wed Jun 29, 2022 3:07 am

Big one for us is a light weight Zabbix proxy for remote sites without needing to add another device onsite

We also use a tik for our Out of Band devices, and being able to spin up a basic linux docker could be useful to run extra troubleshooting software onsite (eg. could have a container with nmap etc)

Another use case would be running something like netinstall in a container, running a container on each port of router/switch, allowing for quick bulk netinstall operations etc

Its a shame they cheaped out on the storage for things like the RB5009, surely 1GB of storage wouldn't add too much to the BOM, or they could add an "extra storage" model

Wed Jun 29, 2022 3:33 am

small voip pbx

woland · Wed Jun 29, 2022 10:15 am

Hi!
I will only need a single container with Openwrt. That gives all the possibilites and not much overhead,
via a huge repository of (mostly) lightweight packages: PBX, proxy, DNS (Bind, unbound, blacklisting...) ,many kinds onf VPNs and a lot more.
OpenWRT is geared towards home routers, so it comes with much less overhead as a Debian or Alpine container would come with.
Btw. it has a nice and more lightweight solution instead of PiHole. PiHole is better run on a Raspi as that needs a lot of RAM & CPU.
Regards

W

groner · Fri Jul 01, 2022 3:26 pm

I'd like to spawn a container to push DNS updates to route53.

r00t · Fri Jul 01, 2022 3:45 pm

+1 for OpenWRT minimal container. It's very lightweight and there is already package manager and other good features available for all architectures ROS runs on.

Also maybe make minimal containers with just busybox for different architectures. Something that can be easily used for doing scripting or port mapping jobs that ROS can't, with little impact on memory/flash.

cfikes · Fri Jul 01, 2022 3:46 pm

small voip pbx

This is what I am planning on using container support for.

woland · Fri Jul 01, 2022 4:34 pm

small voip pbx
This is what I am planning on using container support for.

With an Openwrt install, you could choose from: Asterisk, Fresswitch, Kamaillo, Siproxd, Yate and a few more... (I hope they all are available on Arm64)
W

StubArea51 · Fri Jul 01, 2022 4:42 pm

TACACS+ support

Free Range Routing to get IS-IS and SR-MPLS support (if possible - not sure how well it will work)

th0massin0 · Fri Jul 01, 2022 4:51 pm

Ubiquiti controller

Fri Jul 01, 2022 4:58 pm

Enlighten me please...
Why would one run Openwrt in Docker on Routeros when it can be run natively on lots of Tik HW ??
Natively performance should be better ?

No plans myself, just wondering why.

fragtion · Fri Jul 01, 2022 5:16 pm

Enlighten me please...
Why would one run Openwrt in Docker on Routeros when it can be run natively on lots of Tik HW ??
Natively performance should be better ?

No plans myself, just wondering why.

Best of both worlds... Can winbox talk to, and be used to configure OpenWrt ?

Fri Jul 01, 2022 5:17 pm

Webbrowser access ?

Fri Jul 01, 2022 5:22 pm

While I do agree that Docker is of limited use on RouterOS, there are broad cases where it'll be worth putting up with the many problems you buy in using the feature:

Anything that has to happen at the border of a network, where port-forwarding and such won't work. Examples: mDNS proxying, RTSP gateways…
To replace a weak service on the router with a better one. Examples: authoritative DNS server, DNS proxy that can integrate static + caching + DoH, etc.

Many of the problems on the list in my article linked above can be solved by switching from piggy services written in scripting languages, with many external dependencies to single statically-linked binaries. Since a lot of the cloud is moving toward such things already (e.g. microservices, serverless architectures, etc.) I think we'll find alternatives to a lot of the worst offenders, ones that will fit within the current stringent limits.

I expect MikroTik to start shipping devices with these limitations removed, but even after they do, we'll still benefit from compact, efficient containers, if only so we can run more of them on a single box.

Znevna · Fri Jul 01, 2022 5:43 pm

You keep talking about some limitations but even in your blogpost and in here you didn't mention ONE limitation that MikroTik should remove(?).

Fri Jul 01, 2022 5:56 pm

You keep talking about some limitations but even in your blogpost and in here you didn't mention ONE limitation that MikroTik should remove(?).

They're obvious inverses of the problems as listed. In the article's order:

more ARM devices
n/a (not RouterOS's fault)
more built-in flash storage space, to give room for a container or two plus space for a RouterOS upgrade left over
more devices with USB ports, microSD card slots and/or m.2 storage slots so we can risk running services likely to burn out flash storage, requiring only an aftermarket part replacement rather than resoldering on-board flash
n/a (it's an on-purpose limitation of Docker)

Also, you seem to have overlooked this quote: "If MikroTik ever releases my dream device — an ARM-based multi-core hEX S+, including a microSD slot and an SFP+ port…" That addresses points 1 and 4 on the list above, and by being newer than the current hEX line, it should also address point 3.

Fri Jul 01, 2022 5:57 pm

Ubiquiti controller

unifi ?

guipoletto · Fri Jul 01, 2022 6:01 pm

Ubiquiti controller
unifi ?

Herectics!

Fri Jul 01, 2022 6:02 pm

unifi ?
Herectics!

Znevna · Fri Jul 01, 2022 6:12 pm

Also, you seem to have overlooked this quote: "If MikroTik ever releases my dream device — an ARM-based multi-core hEX S+, including a microSD slot and an SFP+ port…" That addresses points 1 and 4 on the list above, and by being newer than the current hEX line, it should also address point 3.

So.. what's wrong with RB5009 ?

Fri Jul 01, 2022 6:39 pm

So.. what's wrong with RB5009 ?

Apart from the fact it is nowadays more rare then Kryptonite ?

Znevna · Fri Jul 01, 2022 6:40 pm

Except that, I pet mine everyday. ^^

Fri Jul 01, 2022 10:11 pm

what's wrong with RB5009 ?

Cost, size, and complexity.

Cost: The 5009 is roughly 3x the cost of a hEX S. I don't expect something for nothing, but that's a big delta to swing.

Size: I don't often need 9-11 ports of that series of devices. The 5-6 of the hEX line will do for most of my use cases. I like that the 5009 runs cooler than the 4011, but cutting it in half would improve that even more.

Complexity: The 4x rackability feature is cute, but I'll never use it. The 2.5G port also bugs me. I think by the time I find a use for that port, I'll be on 10G everything. It's a temporary stop-gap that will quickly find its way into the dustbin of history.

Mind you, I do like a lot of what they did with the 5009 relative to the 4011, but what I really want is a half-4011 with an option for external storage. That's "half" across the board: half the cost, half the power consumption, and half the size. I realize there are fixed overheads and inelastic costs involved, so I'll bend on the "half price" point; $149? I'll tolerate a reduction to 2 cores to get it.

Another way to approach the solution would be to invert the CRS305 and add some CPU grunt. Instead of one 1G port and four 10G ports, gimme one SFP+ 10G port and four 1G RJ45s. Double the cores, and I'll be reasonably happy, even at 800 MHz.

To bring us back on topic, look on this proposal as a badass Raspberry Pi replacement with strong networking, to replace all those headless Pi boards sitting around doing networky stuff despite the crappy networking subsystem they're saddled with. (Thus Docker.)

If you wonder why I want that 10G port, it's so each 1G port effectively gets a dedicated lane back to the 10G core switch. Although that means the device needs only enough CPU grunt to fill 4-5G with routing and queuing and such, not a full 10G, with Docker, there's on-device traffic to account for as well. I'd expect the box to be able to fill a 10G pipe, at least under limited circumstances.

There should also be a low-cost option closer to the current hEX S in price, while still moving to ARM. Swapping the 10G SFP+ port for a 1G SFP and dropping the CPU power accordingly may swing the deal. I'd expect that to be even more popular than my dream box. People talk a lot about the Pi price of $35, but add a case, power supply, heat sinks, and all the other little doodads, and you're nearing $69. A Docker-capable Improved hEX S (RB760iGSr2?) would be a wee killer of a machine.

Fri Jul 01, 2022 10:59 pm

maybe a redo of hEX S with the same CPU of hAP ac² with 256mb of ram

or

maybe a redo of RB450Gx4 with 1 x SFP and USB port for external storage (maybe removing console or/and one rj-45 eth to make room) reducing to 512mb of ram and only 128mb of nand

i think this ideas can let to what can be seen as RB750Gr4 of maybe hEX Sr2

theosoft · Fri Jul 01, 2022 11:04 pm

Stateful DHCPv6 Server? KEA based ?

regards

cklee234 · Sat Jul 02, 2022 12:32 pm

unifi ?
Herectics!

I have successfully launched unifi controller

4 name="04277978-29d8-4d06-be1c-1899207bbc54" tag="jacobalberty/unifi:latest"
os="linux" arch="amd64" interface=veth3 root-dir=disk1/unifi
mounts=unifi_data,unfi_log dns="" hostname="Unifi" workdir="/unifi"
logging=yes status=running

But not sure how to transfer the control from the existing one to the new docker

bpwl · Sat Jul 02, 2022 7:35 pm

Half 4011 + USB = hAP ac3 ???? (ARM OK, storage OK, 1/2 price OK, disable WLAN). No SFP. No L5 license

RB5009 Docker = "ODROID N2+" ????

Sat Jul 02, 2022 7:54 pm

Half 4011 + USB = hAP ac3 ???? (ARM OK, storage OK, 1/2 price OK, disable WLAN). No SFP. No L5 license

Close. Take those ugly ears off, then use the cost savings to double the flash and gimme my SFP+. If bumping the CPU speed to feed the SFP+ runs the cost up some, I'll take it. I said $149 above, and I meant it, but $129 would make me happier.

I'd still like it to be half the width, while I'm wishing. I think they could get it into a hEX case, but if not, and if reusing the ac³ case design saves enough NRE cost, I'll put up with it.

npeca75 · Sun Jul 03, 2022 12:05 am

so what would be the thing you will use Docker support for on a router and why?

Docker container, which run linux, which run VirtualBox, which run ESXI and on top of this setup, PhotonOS and docker host ...
as proof of concept how to waste your time ...

my personal opinion is:
router for routing,
switch for L2
and good server for lab/production, virtualization, etc ...

devinganger · Sun Jul 03, 2022 12:27 am

To bring us back on topic, look on this proposal as a badass Raspberry Pi replacement with strong networking, to replace all those headless Pi boards sitting around doing networky stuff despite the crappy networking subsystem they're saddled with. (Thus Docker.)

Not to mention how dramatically unstable Pi's storage is. I gave up using pihole on Pi devices after I had multiple Pis corrupt every SD card and USB thumb drive I attempted to use. Even though my RB2011UiAS-2HnD-IN was still working just fine for my home network, I bought a RB3011UiAS-RM because it had the combination of ARM architecture + USB 3.0 port to use with an affordable SSD. My RB2011 will now be moving upstairs as I start adding Ethernet wiring to the rest of the house and reduce my Wifi load.

Mon Jul 04, 2022 3:48 am

maybe a redo of hEX S with the same CPU of hAP ac² with 256mb of ram

or

maybe a redo of RB450Gx4 with 1 x SFP and USB port for external storage (maybe removing console or/and one rj-45 eth to make room) reducing to 512mb of ram and only 128mb of nand

i think this ideas can let to what can be seen as RB750Gr4 of maybe hEX Sr2

The Marvell SOC from the nRay would be much better suited to the RB750Gr4, or even better one of the newer 64bit Qualcomm IPQ's

Tue Jul 05, 2022 1:19 am

maybe a redo of hEX S with the same CPU of hAP ac² with 256mb of ram

or

maybe a redo of RB450Gx4 with 1 x SFP and USB port for external storage (maybe removing console or/and one rj-45 eth to make room) reducing to 512mb of ram and only 128mb of nand

i think this ideas can let to what can be seen as RB750Gr4 of maybe hEX Sr2
The Marvell SOC from the nRay would be much better suited to the RB750Gr4, or even better one of the newer 64bit Qualcomm IPQ's

nray cpu is dual ARM cortex a53 at 1ghz in-order execution light core, i think it does not provide too much advantage over ipq-4018-4019 which is quad ARM cortex a7 at 712mhz (works ok at 896mhz)

cortex a7 and a53 are very similar both in-order execution, the a53 is like a 64bit version of a7, performance of a7 is around 1.9 Dmips/Mhz, a53 is around 2.24 Dmips/Mhz

nray 88F3720 cpu can be around 4.480 Dmips (at 1.000mhz) 2 cores
ipq418 cpu around 5.411Dmips (at 712mhz) 4 cores

very far from for example rb4011 A15 cores which are aprox at 4.0 Dmips/Mhz and out-of-order execution reaching around 22.400 Dmips, amost 4x than small in-order cpus

cfikes · Wed Jul 06, 2022 8:34 pm

With the release of the new RB5009UPr+S+IN using containers becomes the perfect SOHO solution for MSP's.

Power in from all directions, PoE out for phones and AP's, Wireguard/ZeroTier VPN for multisite connectivity, and containers for extended applications like VoIP, authentication and the like it AMAZING!

OlofL · Thu Jul 07, 2022 1:04 pm

Stateful DHCPv6 Server? KEA based ?

Please tell if you build this.
Would be nice to have dhcpv6 server sync leases with another router.

kobuki · Sun Jul 10, 2022 8:26 pm

As soon as the container feature is stable, an OpenVPN container, as the ROS implementation is a fraction of the upstream. I could finally stop depending on another device for my OVPN needs.

strex · Tue Jul 12, 2022 6:36 pm

Using it to install bird2 and replacing the not great mikrotik ospf and bgp daemon.

Wolfraider · Wed Jul 13, 2022 9:12 pm

SBC for VOIP. I posted the idea on the 3CX forums and my post was instantly deleted

gotsprings · Wed Jul 13, 2022 9:15 pm

MPTCPRouter?

Or something similar that let us use Multiple ISPs and Bond as a VPS elsewhere!

jbl42 · Wed Jul 13, 2022 9:32 pm

https://hub.docker.com/r/andrius/asterisk
should run on RB5009/RB4011 and similar arm/arm64 MT devices, but did not try it yet.

cfikes · Thu Jul 14, 2022 12:29 am

https://hub.docker.com/r/andrius/asterisk
should run on RB5009/RB4011 and similar arm/arm64 MT devices, but did not try it yet.

I'll have to give this one a shot. Could not get the 3CX images started on here. I should probably just build a new FusionPBX image.

cklee234 · Thu Jul 14, 2022 1:36 am

I am able to build my debian based asterisk and migrate the existing SIP setting to the docker. It works without issues.

cfikes · Thu Jul 14, 2022 3:23 am

I am able to build my debian based asterisk and migrate the existing SIP setting to the docker. It works without issues.

Thats awesome!

theosoft · Thu Jul 14, 2022 9:05 am

A public PBX docker release with howto would be fine

. Is there a GUI included? *upss*

foureight84 · Tue Sep 06, 2022 2:44 am

+1 for OpenWRT minimal container. It's very lightweight and there is already package manager and other good features available for all architectures ROS runs on.

Also maybe make minimal containers with just busybox for different architectures. Something that can be easily used for doing scripting or port mapping jobs that ROS can't, with little impact on memory/flash.

Interesting. I was looking into documentation and I didn't see how one would do port mapping with Containers in ROS 7.

Tue Sep 06, 2022 7:12 am

I was looking into documentation and I didn't see how one would do port mapping with Containers in ROS 7.

How did you miss this? It's precisely the same thing as "docker create --publish 80:80".

kraal · Tue Sep 06, 2022 11:36 am

Hi,

I'm reading this and other similar threads for some times now about using containers on mikrotik devices and, sorry to say that, I still don't get why this would be a good idea nor can find what I would name "a valid use case" for having containers deployed on a mikrotik device.

Most arguments that I read are "something is missing/not good enough on the mikrotik device and I want to add/replace it". It's like willing to stockpile old plastic in your living room because your cellar is to small, and doing it because your living room is big enough... Fine, but it both sounds like a pad on a wooden leg to me and may not be worth potential involved risks. The cost argument is not valid either as cheap and power efficient devices can be found on the market.

So can you please enlighten me ? Why are people that enthusiastic about what sound like a gadget to me and which probably lowers Mikrotik bandwidth for fixing real issues (bgp for instance, pim-sm) and adding core features which are in my opinion really missing and cannot (not to say shouldn't) be handled through containers anyway (nut for instance).

Apart from the "it would be cool to be able to", I really see no "mindbreaking" argument so far.

Thanks in advance.

Tue Sep 06, 2022 12:08 pm

Most arguments that I read are "something is missing/not good enough on the mikrotik device and I want to add/replace it".

What's wrong with that argument? It's a perfectly valid use case.

The cost argument is not valid either as cheap and power efficient devices can be found on the market.

Some of us have RouterOS devices with enough free RAM, flash, and CPU to do this for "free." Buying another device to run the container is not free.

If you do not possess a device yet that will do this in a sensible fashion, there's a fair chance your next RouterOS device will. What will you do with your spare slice of free compute power?

Even when cost is no object, the ability to deliver a complete, integrated, custom solution may be compelling.

lowers Mikrotik bandwidth for fixing real issues (bgp for instance, pim-sm)

Development talent isn't fungible. The people working on containers likely weren't pulled off your pet projects, nor could they be reassigned to them and be immediately productive.

Furthermore, most of this container stuff is out there, off the shelf, ready to repurpose. A good bit of it lives in the kernel they had to update to produce v7 already. Initial development on what would become the modern container infrastructure goes back to kernel 3.10.

I really see no "mindbreaking" argument so far.

Perhaps it is not for you, then.

That doesn't mean it isn't for anyone else, though.

pe1chl · Tue Sep 06, 2022 12:20 pm

Some of us have RouterOS devices with enough free RAM, flash, and CPU to do this for "free." Buying another device to run the container is not free.

If you do not possess a device yet that will do this in a sensible fashion, there's a fair chance your next RouterOS device will. What will you do with your spare slice of free compute power?

It is a bit sad that RouterOS does not allow to use the resources... e.g. I have a RB4011 that has 1GB RAM which is sitting 90% unused.
The flash is "only" 512MB and I have partitioned it so only 256MB available. And no USB or SD interface to extend it.

With a ramdisk I could at least run a volatile container stored in /ramdisk, but MikroTik refuse to enable it (the code is there, it only requires some trivial "if"s at startup).

kobuki · Tue Sep 06, 2022 12:27 pm

"something is missing/not good enough on the mikrotik device and I want to add/replace it"

I think that's all to it. They they seem to keep adding extras to ROS to allure to more potential customers. Until it doesn't compromise base functionality or security, I don't thing there's harm in doing that.

foureight84 · Tue Sep 06, 2022 4:03 pm

I was looking into documentation and I didn't see how one would do port mapping with Containers in ROS 7.

How did you miss this? It's precisely the same thing as "docker create --publish 80:80".

Then what do you do with two containers both using port 80? They're both on the same VETH.

kraal · Tue Sep 06, 2022 4:40 pm

Hi,

Most arguments that I read are "something is missing/not good enough on the mikrotik device and I want to add/replace it".

What's wrong with that argument? It's a perfectly valid use case.

Well if you consider that as a customer you should "help yourself" that's inded fine. For me the "plain wrong part" is that instead of fixing / adding what's missing (but was existing prior to ROS7) and should be part of a working solution, Mikrotik works on containers.

The cost argument is not valid either as cheap and power efficient devices can be found on the market.

Some of us have RouterOS devices with enough free RAM, flash, and CPU to do this for "free." Buying another device to run the container is not free.

Ok, but how much free RAM/CPU to do what exactly ? For instance I don't see a single usefull service in my 20+ VMs and containers deployed on my home network which could be a candidate for deployment on a mikrotik device. Security, performance, CPU usage, increased heat, separation of concerns, etc. Every time I evaluate a potential service I run into one of these cases: the Mikrotik device is not the place where to deploy it or the service is not lightweight enough. That's why I'm asking: am I the only one in this situation ?

The cost argument is not valid either as cheap and power efficient devices can be found on the market.

If you do not possess a device yet that will do this in a sensible fashion, there's a fair chance your next RouterOS device will. What will you do with your spare slice of free compute power?

Even when cost is no object, the ability to deliver a complete, integrated, custom solution may be compelling.

Well I don't think so. I won't deploy DMZ services on an network infrastructure appliance, nor will deploy security related services on it, nor will deploy potentially high RAM/CPU/I/O consuming services, nor critical services. That's why I'm asking what do you really deploy on containers and for how many users ?

Even when cost is no object, the ability to deliver a complete, integrated, custom solution may be compelling.

This is an argument which may be of interest. But again what services are you talking about and for how many users ? And "integrated" with external storage ? hmm...

lowers Mikrotik bandwidth for fixing real issues (bgp for instance, pim-sm)

Development talent isn't fungible. The people working on containers likely weren't pulled off your pet projects, nor could they be reassigned to them and be immediately productive.

It's part of my job to work on skills development so I know that you don't "turn a pig into a dog just by telling him to bark". The problem here is that these people were hired in the first place, they didn't appear out of nowhere... There was a decision taken at a given point time to hire people to work on what appears to me as "wrong priorities" (away from what the voice of customers would ask for).

Furthermore, most of this container stuff is out there, off the shelf, ready to repurpose. A good bit of it lives in the kernel they had to update to produce v7 already. Initial development on what would become the modern container infrastructure goes back to kernel 3.10.

Yep as many other features in RouterOS as it is built on top of F/OSS.

I really see no "mindbreaking" argument so far.

Perhaps it is not for you, then.

That doesn't mean it isn't for anyone else, though.

Indeed, but apart from your "integrated solution" which is not really precise either. I still didn't read what makes it a "must have" for ROS users. If I only knew, maybe it would be as well for me don't you think ?

Regards

Edit: typo

pe1chl · Tue Sep 06, 2022 4:49 pm

It is not very interesting (read: of no interest at all) what you would or would not install on your router or elsewhere.
Everyone is free to decide what service they want to deploy where, and what method they want to use for it.
Given the fact that a MikroTik router does not allow a generic shell login, making it impossible to copy some random program to the device and run it as a normal Linux process, the possibility of containers is useful in many use cases, even if you cannot imagine one of them.

Of course I also would prefer if MikroTik work on feature parity in v7 (relative to v6) instead of on new features, but it is likely that the person(s) that are able to work on container are different from the one(s) that could fix and finish the BGP routing.
So we have to be patient and hope that the focus does not shift too much (with the risk that BGP will be labeled "not of interest for the RouterOS future")...

jvanhambelgium · Tue Sep 06, 2022 5:06 pm

How did you miss this? It's precisely the same thing as "docker create --publish 80:80".
Then what do you do with two containers both using port 80? They're both on the same VETH.

1 vETH for EACH container!
The IP's of the containers can be in the same subnet or use different subnets.

vETH1 = 172.17.0.1
vETH2 = 172.17.0.2
...is perfectly possible.

foureight84 · Tue Sep 06, 2022 5:10 pm

That makes sense. I tried implementing this at the time of the original post but I am on a level 1 license that restricts to just 1 VETH interface. I was also thrown off with the documentation's example of using 172.17.0.2/16 which made me think the large IP block is for multiple containers using the same interface.

jvanhambelgium · Tue Sep 06, 2022 5:54 pm

Yeah I made the same wrong assumption some time ago.
And with the different vETH's , you have full flexibility with things like DNAT etc if you want to expose to the outside world etc.

kraal · Tue Sep 06, 2022 6:01 pm

It is not very interesting (read: of no interest at all) what you would or would not install on your router or elsewhere.

This is not what my post was about. It was about why *other people* find it useful to figure out if it could be useful *for me*.

Everyone is free to decide what service they want to deploy where, and what method they want to use for it.

Sure. Asking why they do it this way helps going out of ignorance.

The possibility of containers is useful in many use cases, even if you cannot imagine one of them.

That's my point, please share them.
What are the real world usecases which make you people consider it a "must have", worth the effort and consider that it is the right solution (versus a workaround, and if it is a workaround why this one instead of one which is external to the router) ?

By the way, I'm not against having Mikrotik work on "new features", far from it. Command line return codes would greatly help provisioning and automating the configuration of Mikrotik devices; ability to declare a mikrotik device as a NUT client would help rationalize UPS provisionning. Firewall Rules "tagging" would help build scripts which would not have to rely on "comments" to turn on/off rules, a set of ansible tasks ready to use would help automation (but without return codes, kind of difficult to achieve), etc. These are features which are not available today and which if available would help customers cut huge costs, which in turn could be invested back in Mikrotik devices.

pe1chl · Tue Sep 06, 2022 6:48 pm

The possibility of containers is useful in many use cases, even if you cannot imagine one of them.
That's my point, please share them.

Well, I think several of them have already been mentioned above. Anything that is network-related and that you could run on a router.
Full-featured DNS server/resolver, small WEB server for a personal page, VPN implementation for todays-hot-VPN-protocol-not-supported-in-RouterOS, etc.
Of course, when you have a VM host already running, you would not consider most of these use cases useful.
But when you have a personal network without always-on computers, or when you have a head-office/branch-office network where you have no servers in each branch but want some services to be available, it can be an option.

There has been a lot of demand (also from me) for "user processes on the router". I was in favour of a lighter version, where we could just upload executables in a folder and configure them to be run from a chroot environment (that folder) and as non-root user. The Docker container is heavier than what I had in mind, but probably a good idea for standardization and access to stuff not available to a normal user (like creation of network tunnels).
Not only for my own usage, but also to silence the many "please implement my favorite VPN" etc topics (and other feature-creep) with the 1000s of +1 replies.
There already are too many useless services in RouterOS (like proxy, SMB) that could be offloaded into containers to make room for useful router functions.

ech1965 · Tue Sep 06, 2022 7:07 pm

It is not very interesting (read: of no interest at all) what you would or would not install on your router or elsewhere.

This is not what my post was about. It was about why *other people* find it useful to figure out if it could be useful *for me*.

Everyone is free to decide what service they want to deploy where, and what method they want to use for it.

Sure. Asking why they do it this way helps going out of ignorance.

The possibility of containers is useful in many use cases, even if you cannot imagine one of them.

That's my point, please share them.

Adguardhome
bind
traefiek as reverse proxy with letsencrypt and real server in dmz
....

Tue Sep 06, 2022 7:37 pm

if you consider that as a customer you should "help yourself" that's inded fine.

There's tremendous value in choosing solutions that give you the freedom to build your own solutions atop the platform. If the method avoids lock-in, so much the better.

Later today, I'll be working with a different vendor that will not give you programming docs, API references, tools, etc. without a signed NDA, a faxed copy of your business license, a recommendation from a priest, and a sample of your blood. Then on top of that, the resulting programs won't run on any other company's hardware, because their platform is proprietary. Guess which company's products we're least likely to do custom development work atop.

MikroTik, on the other hand, is leveraging the single most popular container platform — contrast BSD jails, Solaris zones, OpenVZ… — and they're giving the feature away for free, for anyone to do with what they want, within the platform's considerable limitations. Even with all the problems, this does not completely suck.

fixing / adding what's missing

I am quite certain that most of MikroTik's customers would look at you strangely if you started talking to them about PIM-SM. That's a very niche feature. I happen to be one of the few that does need it, but at the scale of LAN I generally deal with, it needs to exist in only one place, the big core router that separates all the VLANs, which isn't likely to be MikroTik anyway. How many customers are we talking about, vs all the ones who couldn't expand the acronym at all, much less justify using the technology it refers to?

I couldn't say for certain whether BGP is more or less popular than inter-VLAN multicast — they feel about equally niche to me — but I can confidently predict that most RouterOS device owners don't own or manage an AS themselves. I expect most MikroTik routers end up as WISP CPEs and such, not big corporate edge routers.

Spare utility computing, though, that's something you can use everywhere, if you have sufficient imagination.

Ok, but how much free RAM/CPU to do what exactly ? For instance I don't see a single usefull service in my 20+ VMs and containers deployed on my home network which could be a candidate for deployment on a mikrotik device.

It's usually a mistake to think of a Docker-style container as a lightweight VM. There's no kernel, and even on big iron hardware, you have an incentive to pare the container down as far as is practical. RouterOS multiplies that incentive by 4-10x.

In the big-boy container world, they talk about microservice fabrics and swarms of cooperating services, but we can't afford that at the RouterOS scale. What we're going to do instead is more of a return to the early days of containers, where it was one service per, tightly-scoped.

This is nothing like a VM, except in the most myopic manner. It's closer to an embedded system: a single service that boots and immediately begins work, not stopping until the device loses power.

Security

Containers are generally quite secure. "Root" on the container only has a handful of the hundreds of kernel capabilities real root has out on the host, for one.

For another, you will notice that the current implementation requires NAT, not allowing direct access to the host's bridge. That's a sensible default, though I hope MikroTik eventually lifts it, as there are services you can only provide when bound to real hardware.

As an example, I want to see a netinstall container. Put that on a hEX-sized device, and now you have a utility box a tech can carry around and plug into ailing RouterOS boxes to bring them back online without going through the fourteen documented steps needed to reconfigure a Windows laptop so it'll serve the same end, then reverse those steps to get their laptop back into a useful state again.

No, in my dream world, you run a cable from the rescue box to the target box, reset the latter, wait, reboot, done. With PoE, you won't even need to carry a power brick.

Or, do the same thing to a disused RB3011, running a separate netinstall container behind each port, one for each RouterOS NPK type you have in use, and you've got a benchtop version of the same facility. Someone brings in a flatlined router, you pull over a labeled cable matching the CPU type of the victim, call "clear!" and zap the patient back to life.

The only thing is, I suspect netinstall won't work through NAT, so even if they do produce ARM builds of the program for us, it still won't work. Still, it's a nice dream, and it's within reach.

performance

Not everything requires a Xeon. You have only to look at all the Raspberry Pi boards pressed into service as network utility boxes. It's why you're seeing so many of the people in these threads talking about PiHole and AdGuard.

increased heat

ARM is about 3x more efficient on a MIPS per watt basis than Intel, so unless you're telling me you've got ARM servers over there, I'm gonna call bull on that one. The service had to run somewhere, so if you have a choice between ARM and Intel, and the ARM processor is sufficiently free to carry the load, it's a better place to put it.

separation of concerns

What do you think a container is but a way to separate concerns?

If your point is that containers let you run multiple services on a single box, then why are you telling me about VMs and big-boy servers? Your argument is incoherent and falls in on itself.

am I the only one in this situation ?

Have you even tried to read through the various threads on this? Ideas for how to use containers abound. Many are absurd, and I've done my share of puncturing the worst ideas, but they're not all bad.

I won't deploy DMZ services on an network infrastructure appliance, nor will deploy security related services on it, nor will deploy potentially high RAM/CPU/I/O consuming services, nor critical services.

That sounds like a tailor-made list of criteria against running a VPN on RouterOS.

So here's the situation: RouterOS offers a plethora of VPN options, yet someone always wants one more. Let's pick on Tailscale today. MikroTik almost have that one, between WireGuard and ZeroTier, but they've chosen not to give it to us.

And yet, they have: we have containers, on which we may hope to get Tailscale working on our own.

I don't believe this is possible today due to restrictions in the current RouterOS container platform, but which would you rather have MikroTik working on: yet another VPN protocol, or the generic platform that lets you deploy whatever damn VPN you like?

Even when cost is no object, the ability to deliver a complete, integrated, custom solution may be compelling.

This is an argument which may be of interest. But again what services are you talking about

I had PBX service in mind, mentioned in post #8 in this thread. (Thus why I challenged you on whether you'd even tried to understand this before railing on the idea.)

If you think that's risible, I wonder what you'd tell the WISP operator what he's supposed to do instead when selling against all the triple-play packages from the cable and fiber providers, given that MikroTik shows no interest in putting an RJ11 POTS plug on their devices.

With containers, there's hope that you can put Asterisk or similar on the WiFi CPE you provide, then sell the customer a set of SIP phones to go with it. That's called turning a problem into profit where I come from.

And "integrated" with external storage ? hmm...

For the USB case, there are low-profile memory sticks, the size of those Logitech unifying receivers.

For the microSD case, they're nearly zero-profile already.

And for the m.2 case, they're inside the enclosure.

these people were hired in the first place, they didn't appear out of nowhere...

MikroTik only hires BGP engineers, and they've never put out a new hiring offer since v7 was forked from v6?

I'm quite certain not every software developer on MikroTik's staff is equally capable of going after your pet problems. They're certain to have different strengths, different training, and different predilections, as in any other workforce. The one working on containers probably doesn't even know the BGP protocols at a deep enough level to help out.

ability to declare a mikrotik device as a NUT client would help rationalize UPS provisionning

There you are, then. This container is already built for ARM, it's configurable for the shutdown command (e.g. "ssh admin@hostip /system/shutdown") and it comes to 7-8 MiB. If you run it from flash on a 16 MiB device, you can't upgrade ROS while it's running, but containers are all about automated recreation. So, stop all the NUT containers, remove them, do the upgrade, and redeploy.

This sounds like a fine use of containers on RouterOS.

And best of all, from your perspective, it didn't require that MikroTik divert any development effort away from your pet problems for it, specifically. Instead, they spent the time on a generic platform so they don't have to answer every single random request. It probably nets out an improvement for them, freeing people from chasing random customer requests.

Larsa · Tue Sep 06, 2022 8:01 pm

Great sum-up!

foureight84 · Tue Sep 06, 2022 8:44 pm

Yeah I made the same wrong assumption some time ago.
And with the different vETH's , you have full flexibility with things like DNAT etc if you want to expose to the outside world etc.

Curious, is there more detailed documentation other than this page? I've searched around but I haven't found any details on expected behavior from MT's implementation.

Tue Sep 06, 2022 8:50 pm

No, you're right: the current RouterOS docs on containers positively suck compared to what you get for other container platforms.

Your next-best option is SSHing into a box running the containers.npk package, typing "/container", and then pressing the F1 and Tab keys a lot. Between that, the Docker docs, and a general understanding of the facilities of RouterOS, you can often piece together how it must work and how you can make it do what you want.

But if you want cookie-cutter guides, it's way too early to be expecting that.

foureight84 · Tue Sep 06, 2022 9:03 pm

No, you're right: the current RouterOS docs on containers positively suck compared to what you get for other container platforms.

Your next-best option is SSHing into a box running the containers.npk package, typing "/container", and then pressing the F1 and Tab keys a lot. Between that, the Docker docs, and a general understanding of the facilities of RouterOS, you can often piece together how it must work and how you can make it do what you want.

But if you want cookie-cutter guides, it's way too early to be expecting that.

I'm not expecting cookie cutter guides. I had an inclination that's how it works after testing a few methods. This still falls in the assumption category. I am also new to using Router OS and not familiar with the design paradigm to quickly piece it together. I have to ask experienced users in order to confirm and develop a base of understanding.

I am also testing Ruter OS 7 with a level 1 license trying to decide if it's the right fit before buying a level 4 license. So there are certain limits to what I can test. And in hindsight, it seems that it would have been a better choice to stay on the 24hr trial and reload the OS on my machine or try this on a VM. I went into it not knowing the current state. All I knew is that it has a stable release and there's some documentation.

Also, I did not make any assertion that the documentation "sucks." I simply asked if there's something I am missing to try and fill the gap.

Tue Sep 06, 2022 9:28 pm

I am also testing Ruter OS 7 with a level 1 license

That sounds like CHR, in which case containers are kind of silly. It's far better to either run another VM on the same host, or if this is running on a Type 2 Hypervisor (e.g. VirtualBox, Hyper-V) then start Docker Engine out on the host, alongside the hypervisor if the goal of the project is to test Containers.™

The Containers feature of RouterOS is best used when you have actual MikroTik hardware, and it has enough spare resources to do something interesting, so you spin it up there.

I did not make any assertion that the documentation "sucks."

That was my claim. And I'm right.

own3r1138 · Tue Sep 06, 2022 9:48 pm

It's far better to either run another VM on the same host.

For a small user like me, paying for an extra VM is an overhead I don't need.

Tue Sep 06, 2022 9:57 pm

It's far better to either run another VM on the same host.
For a small user like me, paying for an extra VM is an overhead I don't need.

Docker Engine is far lighter than a single VM, and it's far more capable than RouterOS's Containers feature is ever likely to be. If you want even lighter-weight solutions, you've got Podman and containerd+nerdctl. I would not run containers on CHR, at all, ever, except for producing a proof-of-concept meant to roll out to actual hardware.

own3r1138 · Tue Sep 06, 2022 10:44 pm

Docker Engine is far lighter than a single VM

I understand the argument you made. Furthermore, I agree with you. However, where should one install the Docker Engine? doesn't it require another Linux/Win VM, or perhaps I misunderstand you? Recently I played a lot with docker to build my own image still a work in progress.

2022-09-07_00-06-45.jpg

Wed Sep 07, 2022 12:07 am

where should one install the Docker Engine? doesn't it require another Linux/Win VM…?

You've got two basic options:

You're running a type-1 hypervisor (e.g. ESXi) so you have no choice but to spin up a VM running the container runtime environment alongside CHR and whatever else you're running. There are lightweight distros for the purpose like Flatcar, or you can press something general-purpose like CentOS into service here. However you go, it'll be more functional than container-in-CHR, and it may even be more performant by being a more mature platform.
You're running VMs in a type-2 hypervisor (e.g. KVM, VirtualBox) atop a full-power host OS on bare metal. Rather than run the engine in a VM, you run it out on the host, parallel with the VM hypervisor. This is likely to give you better performance and lower overhead.

There have been efforts toward bare-metal container runtimes along the lines of ESXi (e.g. Joyent's unsuccessful SmartOS and Triton products) but I don't know of any that are viable, battle-tested, and ready to run.

own3r1138 · Wed Sep 07, 2022 1:02 am

@tangent
Thank you for the comments.

Znevna · Thu Sep 15, 2022 3:09 pm

For another, you will notice that the current implementation requires NAT, not allowing direct access to the host's bridge. That's a sensible default, though I hope MikroTik eventually lifts it, as there are services you can only provide when bound to real hardware.

My AdGuardHome runs fine with an IP from the subnet sitting on the main bridge, and veth added to that bridge. You're not forced to use NAT, probably you should take another look into it

Thu Sep 15, 2022 6:04 pm

My AdGuardHome runs fine with an IP from the subnet sitting on the main bridge

Good to know; thanks.

This brings us back to the skimpy state of the docs, of course.

Znevna · Thu Sep 15, 2022 6:30 pm

They only recommend keeping containers on another subnet, you know, for .. containerization purposes.

Shahid · Fri Sep 23, 2022 3:02 am

1+ for Openwrt as a container Guest in Mikrotik, with lot of possibilities regarding routing, very light weight & best for OpenVPN Configurations.
I think Mikrotik doesn't like "opensource of things" much because of complicated implementation of OVPN in Mikrotik.

gotsprings · Fri Sep 23, 2022 4:00 am

1+ for Openwrt as a container Guest in Mikrotik, with lot of possibilities regarding routing, very light weight & best for OpenVPN Configurations.
I think Mikrotik doesn't like "opensource of things" much because of complicated implementation of OVPN in Mikrotik.

Now that Wireguard is supported... Why bother with OpenVPN?

own3r1138 · Fri Sep 23, 2022 5:03 am

@gotsprings
I live in Iran. I don't know how much you are familiar with our current government. Due to the latest movements, 2/3 of The internet is down including WG protocol, but OVPN*** is working.

gotsprings · Fri Sep 23, 2022 5:09 am

@gotsprings
I live in Iran. I don't know how much you are familiar with our current government. Due to the latest movements, 2/3 of The internet is down including WG protocol, but OPVN is working.

Not familiar at all.

But openVPN uses UDP 1194 and LZO compression.

Wireguard is also UDP and assignable on some random port. It's lighter weight...

So I am curious why OVPN would still be working and Wireguard wasn't.

own3r1138 · Fri Sep 23, 2022 5:36 am

Well, the default OVPN port was filtered long before WG existed. I have both services on 443/UDP. I don't know how they did it too.

semaja2 · Mon Sep 26, 2022 2:33 pm

Or, do the same thing to a disused RB3011, running a separate netinstall container behind each port, one for each RouterOS NPK type you have in use, and you've got a benchtop version of the same facility. Someone brings in a flatlined router, you pull over a labeled cable matching the CPU type of the victim, call "clear!" and zap the patient back to life.

The only thing is, I suspect netinstall won't work through NAT, so even if they do produce ARM builds of the program for us, it still won't work. Still, it's a nice dream, and it's within reach.

I actually just built an image to run netinstall on ARM/ARM64/x86 tiks for this exact purpose

viewtopic.php?t=189485

In all honesty, if your not already running linux, this is a bloody quick and easy way to do netinstalls

PS. I used to run a dedicated Windows VM I would pass through a USB NIC, just to do netinstalls

Shahid · Thu Sep 29, 2022 12:50 am

1+ for Openwrt as a container Guest in Mikrotik, with lot of possibilities regarding routing, very light weight & best for OpenVPN Configurations.
I think Mikrotik doesn't like "opensource of things" much because of complicated implementation of OVPN in Mikrotik.
Now that Wireguard is supported... Why bother with OpenVPN?

Because i am using openvpn server in ubuntu & a lot of clients on different devices including Android Windows & IOS. setting up wireguard for each individual using public keys etc is a hectic task. openvpn uses pre configured files just import & bump...
does wireguard have something like that?? so i can share a file instead of keys & import it in client app on different architectures as mentioned before??

kamurdoch · Fri Oct 07, 2022 1:22 am

Home Assistant. If anyone has got it right, let me know,
To reply to the 'why would we' people, one good reason is that this is how we learn.
But my reason is to have fewer devices running off an inverter to prolong battery life during loadshedding in South Africa.
What's loadshedding? - Eskomplicated.

semaja2 · Fri Oct 07, 2022 2:09 am

Home Assistant. If anyone has got it right, let me know,
To reply to the 'why would we' people, one good reason is that this is how we learn.
But my reason is to have fewer devices running off an inverter to prolong battery life during loadshedding in South Africa.
What's loadshedding? - Eskomplicated.

I tried this on a hAP AC3 with ext usb drive but it failed every time, it is a really big container (over 1GB) so suspect you will need a powerful tik to handle it, if I could get access to an RB5009 or something more powerful I could test further

rextended · Fri Oct 07, 2022 3:06 am

A container to run RouterOS v6 on v7 for all missing features.

pe1chl · Fri Oct 07, 2022 12:12 pm

A container to run RouterOS v6 on v7 for all missing features.

Well, if it is possible to have a container (or even a package) to have the v6 BGP+BFD code on v7, it would certainly be useful here!

semaja2 · Mon Oct 10, 2022 1:46 am

I feel like a lot of these suggestions are either entirely a joke, or lack the understanding of what a container is

Nothing stops you wrapping a BGP service into a container, but your RouterOS would still need to establish a BGP session to it

Eg. Run FRRouting in a container, run it as a route reflector or just a big digestor of all the IX/Transit sessions etc, then establish a BGP session with the host router

However all this would unlikely improve the situation, as it would increase the potential for faults and likely be slower

Znevna · Mon Oct 10, 2022 9:32 am

I have a dream with a small dnsmasq container to replace the MikroTik DNS server and DHCPv4 server.

semaja2 · Mon Oct 10, 2022 1:32 pm

That’s quite easy, given the way the containers are implemented you can just load a basic alpine image

Set the start command to sleep forever, console in, install all your bits you need, change the start command to run dnsmasq

Or if just build a image that has it all of course

One missing feature is there is no auto restart if the container stops for any reason, which could be pretty bad for said service

fragtion · Mon Oct 10, 2022 1:50 pm

But my reason is to have fewer devices running off an inverter to prolong battery life during loadshedding in South Africa.
What's loadshedding? - Eskomplicated.

Awe bru, the struggle is real !

One missing feature is there is no auto restart if the container stops for any reason, which could be pretty bad for said service

Simple scheduler script could solve this. But a watchdog option alongside the start-on-boot functionality is a good suggestion.

jvanhambelgium · Wed Oct 19, 2022 11:49 pm

I've updated my RB5009 to 7.6 and running 1 "pihole" container in production.
As some reported earlier, I'm not really convinced about good "memory management" here (read : looks like a memory leak?)
I'm going to evaluate for the coming days...luckily an RB5009 has 1Gbytes so there is some headroom ... but stil....

Screenshot from 2022-10-19 22-47-57.png

sirbryan · Tue Oct 25, 2022 10:22 pm

Here's my CCR2116 running 7.6 with a brand new pi-hole container added the other day. I haven't tried it on a 5009 or 2004 yet.

image.gif

jbl42 · Tue Oct 25, 2022 11:06 pm

I'm going to evaluate for the coming days...luckily an RB5009 has 1Gbytes so there is some headroom ... but stil....

If I read your chart right, the memory consumption increased by about 4MB in about 6h and seems to stabilize towards the end of the available data.
PiHole is caching things like resolved hosts, compiled filter lists and similar in memory. 4MB is no alarming amount, as long as it does not keep counting.

PS
As a safety measure, Docker allows to limit the max memory/CPU a running image can use. Hitting the memory limit will most likely crash the running image, but it avoids to take the host system down by exhausting all available memory.
But as far as I can see, resource limits for docker are not available (yet?) in ROS.

sirbryan · Tue Oct 25, 2022 11:52 pm

I looked at my ESXi server and the VM's it's running, and I'm considering moving what I can over to my CCR2116.

pi-hole - Already moved

Asterisk/FreePBX

Beta Unifi/UISP servers

And with 2-4TB NVMe SSD, I could do

OwnTone (DAAPd) to replace macOS 12 running iTunes 24/7

ownCloud/NextCloud

NAS (NFS, SMB, or worst case scenario WebDAV)

For NFS, this user-space implementation looks promising (needs to be containerized): https://github.com/unfs3/unfs3

semaja2 · Wed Oct 26, 2022 5:34 am

For anyone concerned about memory leaks... can you run just a base alpine image with nothing on it?

There is little to gain mentioning memory leaks for a 3rd party container as its not RouterOS problem

That being said ROS container system does let you set memory limits to prevent a bad container consuming all the memory of the router

RAM usage can be limited by using:

[*]/container/config/set ram-high=200M

this will soft limit RAM usage - if a RAM usage goes over the high boundary, the processes of the cgroup are throttled and put under heavy reclaim pressure.

pe1chl · Wed Oct 26, 2022 12:08 pm

Before you consider it "a leak", note that some software will just check how much memory there is and use it, e.g. for "cache" functionality.
When the amount of memory is limited, code may become active that expires old cache entries, for example.

elbob2002 · Wed Oct 26, 2022 2:33 pm

Here's a screenshot of one of my CHRs. Two containers. Both running Alpine. First is running Caddy as a Reverse Proxy, Second is running Openspeed Test.

You can clearly see where I fired up the second container on Monday. Memory usage is very stable.

Screenshot 2022-10-26 122914.png

elico · Thu Nov 17, 2022 2:49 am

I looked at my ESXi server and the VM's it's running, and I'm considering moving what I can over to my CCR2116.

pi-hole - Already moved

Asterisk/FreePBX

Beta Unifi/UISP servers
And with 2-4TB NVMe SSD, I could do
OwnTone (DAAPd) to replace macOS 12 running iTunes 24/7

ownCloud/NextCloud

NAS (NFS, SMB, or worst case scenario WebDAV)
For NFS, this user-space implementation looks promising (needs to be containerized): https://github.com/unfs3/unfs3

There are limitations on containers limiting them for NFS and if you would use a user-space implementation you would loose a lot of things in terms of performance.
Unless you really need I would not recommend you to use it.

About SMB, there is an issue with the SMB implementation of RouterOS and I have seen couple SAMBA containers based on Alpine but you would need to create a script the populate or copy a share and create the users and password etc...
It's not impossible but I believe that you should only use shares on external devices and only if you have a static "known" setup.
For example a public shared folder or something small like 5-10 users top.

whitefxdesign · Mon Nov 21, 2022 3:44 am

[/quote]
Adguardhome
bind
traefiek as reverse proxy with letsencrypt and real server in dmz
....
[/quote]

I have an RB5009 and would like to move Adguard and my reverse proxy from my NAS and have it all run from the tik with a NVME SSD plugged in. I've never used Traefik but have considered giving it a try.

Has anyone put a reverse proxy into production lately on one a tik device with RouterOS that supports containers like the RB5009? I don't plan on running a lot of containers this way but hoping to keep any network-related services off of the NAS as I take it down from time to time and sometimes forget to update my DNS servers and seem to notice it when someone else in the house no longer can connect.

Whitehawk29FR · Wed Nov 23, 2022 1:54 pm

NMAP container ?
It's something missing on routerOS to easily scan client LAN (check if a port is open on a device or not).
Or if someone have a easy way for this (ssh tunnel ?)

elico · Sun Dec 04, 2022 10:47 pm

NMAP container ?
It's something missing on routerOS to easily scan client LAN (check if a port is open on a device or not).
Or if someone have a easy way for this (ssh tunnel ?)

A very simple ssh container can do that.
I have created one based on alpine linux 3.17 but not sure if I have published it.
I will look later on and see about this.

boldsalt2800 · Tue Dec 06, 2022 1:16 am

For another, you will notice that the current implementation requires NAT, not allowing direct access to the host's bridge. That's a sensible default, though I hope MikroTik eventually lifts it, as there are services you can only provide when bound to real hardware.

My AdGuardHome runs fine with an IP from the subnet sitting on the main bridge, and veth added to that bridge. You're not forced to use NAT, probably you should take another look into it

They only recommend keeping containers on another subnet, you know, for .. containerization purposes.

Does the network setup of the containers (bridge, VLAN, subnet, etc) have any special considerations compared to just a baremetal server running that same container's services but plugged into an ether port?

Apart from the normal considerations with hosts sharing a broadcast domain, are there any container-implementation-specific downsides with putting the containers on a separate VLAN/subnet but on the same bridge as all my other subnets/VLANs?
Or with putting the containers on the same bridge/VLAN/subnet as other trusted network services hosts?

jvanhambelgium · Tue Dec 06, 2022 10:00 am

For another, you will notice that the current implementation requires NAT, not allowing direct access to the host's bridge. That's a sensible default, though I hope MikroTik eventually lifts it, as there are services you can only provide when bound to real hardware.

My AdGuardHome runs fine with an IP from the subnet sitting on the main bridge, and veth added to that bridge. You're not forced to use NAT, probably you should take another look into it

They only recommend keeping containers on another subnet, you know, for .. containerization purposes.

Does the network setup of the containers (bridge, VLAN, subnet, etc) have any special considerations compared to just a baremetal server running that same container's services but plugged into an ether port?

Apart from the normal considerations with hosts sharing a broadcast domain, are there any container-implementation-specific downsides with putting the containers on a separate VLAN/subnet but on the same bridge as all my other subnets/VLANs?
Or with putting the containers on the same bridge/VLAN/subnet as other trusted network services hosts?

I have a separate bridge and use a separate IP-block with various vETH's in that range.
I create a separate interface-list/zone "containers" to have a clear view on these entities. They are for sure not part of the "LAN" environment in my setup.
And specific rules manage traffic from/to

sirbryan · Sat Dec 17, 2022 4:41 pm

I got a simple SMB file server container to work on a CCR2116 and 7.6.

And WOW, I can saturate the 10Gbps network with file transfers with no problem to and from the NVMe storage. Performance is so much better than the built-in SMB server.

I based it off this one: https://github.com/fschuindt/docker-smb.

I wasn't successful at building that container on my Mac and uploading it, so instead...

- I manually created a container based on Alpine linux:

## Use your preferred veth settings here; I chose this subnet because 172.17.x.x is in use elsewhere
/interface veth
add address=172.16.0.2/24 gateway=172.16.0.1 name=veth1
/container mounts
add dst=/etc/samba/ name=samba-etc src=/disk1/samba-etc
add dst=/Shared/ name=samba-share src=/disk1/samba-share
/container envs
add key=TZ name=samba value=America/Denver
/container config
set registry-url=https://registry-1.docker.io tmpdir=disk1/docker-pull
/container
add cmd="/bin/sh -c \"while ((true)); do sleep 1; done\"" envlist=samba interface=veth1 logging=yes mounts=samba-etc,samba-share root-dir=disk1/samba-root start-on-boot=yes remote-image=library/alpine:latest
## The cmd above keeps the container running so I can shell into it.  In production, it would start the SMB daemon:
## smbd --foreground --no-process-group

- Start and connect to the shell (in this case it's the first container; note: typing it all on one line causes the MikroTik CLI to lock up and you have to ssh back in):

/container
start 0
shell 0

- Manually installed the packages listed in the Dockerfile (https://github.com/fschuindt/docker-smb ... Dockerfile):

 apk add --no-cache --update samba-common-tools samba-client samba-server

- Edited the smb.conf file manually inside the container (this is a quick and dirty publicly available share, good enough to test it out):

[global]
        map to guest = Bad User
        log file = /var/log/samba/%m
        log level = 10

[guest]
        path = /Shared/
        read only = no
        guest ok = yes

- Started the SMB server manually. Once you decide to go into production, replace the "cmd" argument for the container with this.

 
 # Run this inside the container
 smbd --foreground --no-process-group --debug-stdio --debug-level <whatever level you want for testing, or don't use this flag>

- Boom! Super fast NAS.

I'll be testing this on 2004's, 5009's, and hAP's with external USB drives.

Since this is on my router, all hosts can directly connect to the container's private IP to mount the share.

Further customizations and Samba security are left as an exercise to the reader.

This obviously isn't The Right Way™ to build a reproducible container, but it's a good place to start. Maybe @elico can get it working faster than I can for everybody else.

Sat Dec 17, 2022 10:55 pm

I wasn't successful at building that container on my Mac and uploading it, so instead...

It builds and exports just fine here:

$ git clone https://github.com/fschuindt/docker-smb
$ cd docker-smb
$ docker build -t smb --platform=linux/arm/v7 .     # for 32-bit ARM
$ docker build -t smb --platform=linux/arm64 .      # for 64-bit ARM
$ docker create --publish 139:139 --publish 445:445 --name smb smb
$ docker export smb > smb.tar    
$ scp smb.tar myrouter:

sirbryan · Sat Dec 17, 2022 11:49 pm

Those instructions aren't the same as what I tried. I used Anton's instructions on MikroTik's documentation pages...

I'll give yours a try and report.

In the meantime, I got it working on a CCR2004 (the 16-port version with USB port) and an RB5009, both with identical Samsung USB SSD's I got from Walmart. The test consisted of copying a number of video files ranging in size from 400MB to 5GB. I got up to 3Gbps transfer on the 2004 and 1.7Gbps transfer on the 5009, and 3.8Gbps to the NVMe drive on my CCR2116 (that test could have been limited by some CAT6 cabling issues I had during testing).

Sun Dec 18, 2022 12:05 am

The "buildx build" vs "build" distinction doesn't matter. If the old image builder works, so will the new one. If you need the new one (buildx) to get some feature not available in the old one, then use the new one.

The platform name differences aren't important. The container build system is pretty flexible about what it will take, having multiple aliases for each platform.

As for "export" vs "save", I see that the docs are right, in retrospect. The two types of tarball aren't equivalent, and I winged the one I tried here. I didn't try actually running it. (I'd want the container much more badly than this to be running random code off GitHub.)

That in turn tells us that the "docker create" command isn't helpful. It isn't wrong, it just gets you a container useful only on the build host. If you don't intend to try it there first, you'd put the "docker save" command before that and continue on the RouterOS box. The "--publish" flags turn into RouterOS NAT directives.

All of this is kind of a side issue: my method is mistargeted, but it still has the benefit of showing it builds on macOS, with Docker Desktop 4.15.0. (The current version.) Post what output you get instead.

sirbryan · Sun Dec 18, 2022 12:13 am

I was able to build for the arm64 platform just fine (because I'm on an M1 Mac). But your "winged" instructions couldn't find/create the arm/v7 version, which I realized a few minutes ago. I got buildx to properly build it, and came back to look over the publishing/exporting piece and saw your response.

Both are built now and initial tests on the CCR2004 and RB5009 are successful. Thanks for pointing me in the right direction.

I will post an updated "tutorial" as it were for those interested in a more succinct and replicable deployment process.

sirbryan · Sun Dec 18, 2022 5:47 am

Here's my updated method for building a Samba container that runs on arm/arm64 (tested on CCR2116, CCR2004, RB5009, which are all arm64).

Below is the Dockerfile from the perviously linked repo (as of this writing). Obviously check the copy you clone first to verify you're comfortable with the RUN, COPY, and CMD statements.

FROM alpine:3.12.0

RUN apk add --no-cache --update \
    samba-common-tools \
    samba-client \
    samba-server

COPY smb.conf /etc/samba/smb.conf

EXPOSE 139/tcp 445/tcp

CMD ["smbd", "--foreground", "--log-stdout", "--no-process-group"]

In essence, it loads the stock Alpine Linux image, instructs it to install the Samba packages, copies a stripped-down smb.conf file, and executes the smb daemon when the container starts.

Certainly you could modify the Dockerfile if you want your image to have more to it, such as a text editor (other than 'vi') for manipulating the config. I changed the Alpine linux image label to "latest", and learned the newer version of Samba that is loaded doesn't supports the CMD line flag "--log-stdout", so you'll have to remove that flag if you make similar changes.

My container instructions maps /etc/samba to disk, so that any changes I make to smb.conf persist across container deployments, as opposed to using the simple one included in this guy's repo.

Run this on your container-creating computer:

$ git clone https://github.com/fschuindt/docker-smb
$ cd docker-smb

# Make any desired changes to Dockerfile and smb.conf

$ docker buildx build -t samba-arm --platform=linux/arm/v7 .     # for 32-bit ARM targets, built on any PC/Mac hosts
$ docker save samba-arm > samba-arm.tar

$ docker buildx build -t samba-arm64 --platform=linux/arm64 .      # for 64-bit ARM targets, built on non-M1/M2 hosts
$ docker build -t samba-arm64 --platform=linux/arm64 .      # for 64-bit ARM targets built on M1/M2 (the buildx example above works too, honestly)
$ docker save samba-arm64 > samba-arm64.tar

# Copy to your ARM router, like a hAP AC2/AC3
$ scp samba-arm hAPac3:/path/to/disk/

# Copy to your ARM64 router, like CCR2216, CCR2116, CCR2004, RB5009
$ scp samba-arm64 ccrARM64:/path/to/disk/

I inadvertently copied the samba-arm.tar (arm/v7) file to my CCR2004 and RB5009 (both arm64). While they worked just fine, I realized my mistake and copied the arm64 versions over and replaced the container. I saw slight performance improvements in both throughput and CPU, enough to make it worth the effort of matching the platform more accurately.

Container setup on router (revised version):

# Configure your veth1 however you like; mine is successfully bridged to the router's main bridge, putting the container in the same LAN as the other hosts.
# This alleviates the need for NAT to the container
#
/container mounts
add dst=/etc/samba/ name=samba-etc src=/disk1/samba-etc
add dst=/Shared/ name=samba-share src=/disk1/samba-share
/container config
set registry-url=https://registry-1.docker.io tmpdir=disk1/docker-pull
/container envs
add key=TZ name=samba value=America/Denver
/container
# My USB/NVMe disks are labeled "disk1"
add  envlist=samba interface=veth1 logging=yes mounts=samba-etc,samba-share root-dir=disk1/samba-root start-on-boot=yes file=disk1/samba-arm64.tar

# Assuming this is the first container
start 0

Thank you @tangent and others for your help. Now I've got a decent sandbox to play with.

spippan · Sun Jan 08, 2023 10:13 pm

Eg. Run FRRouting in a container, run it as a route reflector or just a big digestor of all the IX/Transit sessions etc, then establish a BGP session with the host router

well ... as somehow "hackey" and seemingly silly that sounds, but in fact that could be (a quite dirty, yes....but) a workaround until MIKROTIK finally addresses the the importance of a router being able to properly do BGP (especially a device named "CLOUD CORE ROUTER").

but how would one expose the containter to bgp partners?

sirbryan · Tue Jan 10, 2023 1:45 am

Eg. Run FRRouting in a container, run it as a route reflector or just a big digestor of all the IX/Transit sessions etc, then establish a BGP session with the host router

well ... as somehow "hackey" and seemingly silly that sounds, but in fact that could be (a quite dirty, yes....but) a workaround until MIKROTIK finally addresses the the importance of a router being able to properly do BGP (especially a device named "CLOUD CORE ROUTER").

but how would one expose the containter to bgp partners?

VETH can be bridged to any of the other ports. The containers don't have to have their own bridge.
Even if they did, you can make the subnet anything you like. Then you use OSPF to announce the container subnet and loopbacks.

I now have a project for the afternoon...

spippan · Tue Jan 10, 2023 1:51 pm

VETH can be bridged to any of the other ports. The containers don't have to have their own bridge.
Even if they did, you can make the subnet anything you like. Then you use OSPF to announce the container subnet and loopbacks.

I now have a project for the afternoon...

thx, i that i know. i have 2 container running (adG and frrouting) on veth1 and veth2 which are bridged to "br0" (main bridge) and PVID 10
works without any troubles.

but the concept with FRR to expose/connect with other "ext." routers/peers and propagate the routes to the "host" mikrotik is a bit of a work of planing yet to do. currently i am testing an iBGP between the host and the frr-container which works with some precautions to take (e.g. in/out filters to not redistribute the connected route back to the host.)
IMHO, i guess there is no need to redistribute "connected" from the FRR to the host in the iBGP (only the learnt BGP, OSPF, IS-IS or EIGRP routes)
further lab work is ahead

need to quickly setup a cisco router to test how far i will come with EIGRP (which i think, is one of the best IGPs)

mjezierski · Wed Jan 11, 2023 7:04 pm

Big one for us is a light weight Zabbix proxy for remote sites without needing to add another device onsite

This would be a great purpose if Zabbix proxy would just query and forward the SNMP data, however there's a DB in there taking up a bit of storage space.

elico · Wed Jan 11, 2023 9:46 pm

exabgp would be great.
I have somewhere an exabgp setup which defines the next-hop for a bgp advertisment and it's good for anycast dns advertisment in ISP networks.

leonardogyn · Fri Feb 03, 2023 9:05 pm

This would be a great purpose if Zabbix proxy would just query and forward the SNMP data, however there's a DB in there taking up a bit of storage space.

.
That would be a PERFECT scenario for a tmpfs filesystem, to store the zabbix DB, an sqlite3 file for example. I fully agree that having that on the Mikrotik NAND storage is not a good idea. But having that on a tmpfs, why not? You would lose all data on a reboot, but if you can cope with that, zabbix-proxy+DB on tmpfs would be awesome!

brotherdust · Sat Feb 18, 2023 1:04 am

So when 7.1rc3 with Docker support came out, i instantly jumped to obvious things to use it to run stand alone DNS server - feature that is missing in RouterOS itself.
But i must admit i strugle to find any usage of this feature on a router, most of the things i would run in container i would run on x64 server, that has much more resources available.

so what would be the thing you will use Docker support for on a router and why?

As it stands, the containers seem to be running in some kind of unprivileged mode, which limits what one can do. For my own part, I'd like to be able to control the kernel capabilities that a container can use to enable more use-cases. For my own part, I'd like access to the underlying host's network configuration (interfaces, route tables, netns, etc) through something like Zebra. This would give me the ability to enable advanced use-cases that recent upstream Linux kernels have native support for, namely iOAM6 (which was upstreamed in v5.16). Additional use-cases would be something like enabling SRMPLS SRv6, ISIS, EVPN.

The current kernel in RouterOS v7 is v5.6. So, not only would Mikrotik have to allow for capability control, they'd have to upgrade the kernel. My understanding is that they've made upgrading the kernel a relatively painless process, so who knows?

I can keep dreaming right? =)

marsbeetle · Sun Feb 19, 2023 10:25 am

I’m using AdGuard, iperf3 and mrtg(bandwidth/resources graphing) containers on hAPax3 / usb3 sandisk ultra 128gb. I run additional containers like home assistant etc on a nas. My nas is not up 24/7 like router and router has backup with mini ups (gizzu 100w lifepo4) - necessary for our load shedding which lasts 4+ hours.

cdhtlr · Mon Feb 20, 2023 11:42 pm

I've created an ookla speedtest container (viewtopic.php?t=193053) for speedtest based failover purposes.
Not long ago I also updated it with parallel connection capabilities to minimize downtime.

soonwai · Fri Apr 14, 2023 2:40 pm

I'm currently running the following containers on my hAP ax3 with Sandisk Ultra 64GB thumbdrive. As it is, the ax3 is underused as just an AP so may as well let it do a bit more work.

- adguard/adguardhome
- louislam/uptime-kuma
- foorschtbar/routeros-letsencrypt (will be scheduling this to come up every 60 days or so to renew certs)

With all 3 running, ax3 free mem is about 200MiB.

So far, just 2 weeks, everything is working pretty well. Not sure what else I can use but always looking for more container ideas.

bpwl · Fri Apr 14, 2023 3:14 pm

more ideas? ... cheating on your neighbours : "Home Assitant" has an interesting list of add-ons and "community add-ons", not sure that all are docker images.
https://www.home-assistant.io/addons/
https://community.home-assistant.io/tag ... repository
https://github.com/hassio-addons/repository

nosovk · Sat Jan 18, 2025 5:04 pm

It would be nice to spin up WINS server in a container. It will reduce amount of problems in SMB segment. (old request: viewtopic.php?p=1119903#p1119903)

Larsa · Sat Jan 18, 2025 5:37 pm

WINS is a 31-year-old, obsolete Microsoft legacy service and an implementation of the NetBIOS Name Service (NBNS) that was needed for MS Windows 95 and earlier versions. Starting with Windows XP in 2001, newer versions switched to using DNS, so WINS isn’t really necessary anymore.

Since it’s no longer supported, Microsoft recommends decommissioning WINS: https://learn.microsoft.com/en-us/windo ... s/wins-top

Amm0 · Sat Jan 18, 2025 7:32 pm

WINS is a 31-year-old, obsolete Microsoft legacy service

Agree. But still really curious on what drives any use case for WINS...

But just to answer the question... you can install samba in an Alpine container, and enable WINS in it's config (and add LMHOSTS, have RouterOS DHCP set container as WINS server, etc.).

Larsa · Sat Jan 18, 2025 10:12 pm

FWIW, one should at least be aware that WINS has several security vulnerabilities that Microsoft hasn’t and won’t patch. If possible, I’d avoid WINS altogether.

That said, it’s possible to run WINS in parallel while implementing IP-based name resolution using, for example, Samba as an Active Directory Domain Controller or Microsoft Active Directory Domain Services (AD DS)