v7.1rc3 adds container support

Fri Jun 17, 2022 10:20 pm

Any idea what command I should actually use not to brick the vm again?

The one given in the docs: “/system/device-mode/update container=yes”.

Home mode is very wrong for this.

elbob2002 · Sat Jun 18, 2022 11:24 am

Thank you! I don't know how I missed that in the docs!

bma · Sat Jun 18, 2022 1:09 pm

I did the following and no longer see the "no space to extract layer" error when adding a container:
1) disabled SMB
2) deleted the SMB share referencing /disk1
3) reformatted the drive as ext4 without partition table. changed name to disk1
4) re-added the container

Maybe there is a conflict with SMB and containers both accessing the file system?

I'm still see the "sudo: effective uid is not 0, is /usr/bin/sudo on a file system with the 'nosuid' option set or an NFS file system without root privileges?" error when modifying the configuration (which doesn't update) on the pihole website. As noted in the previous posts this is a known issue. I can ftp in, modify the container config files, and restart the container to update the configuration.

mistercovert · Sun Jun 19, 2022 6:37 pm

I am successfully running Pihole in a container of my 5009 on 7.4RC using a Sandisk USB stick set to to disk2, I can pass traffic through the Pihole and DNS is pointing at the PiHole. When I log into the PiHole GUI, I cannot import additional adlists or add blacklist entries, when I run the Gravity update nothing is updated. Is anyone else having these problems?

KexyBiscuit · Sun Jun 19, 2022 9:34 pm

Is it possible to allow access to /dev/qce in the container, for faster crypto operations on Qualcomm platform routers?

pi0 · Sun Jun 19, 2022 11:07 pm

Congratulations to the Mikrotik team! Super excited containers are back again!

I have made a tutorial and image to run generic Debian/Alpine SSH server using containers: viewtopic.php?p=940768

Please let me know what you think and I hope this helps to try container features and possibilities and find new ideas.

DeviceLocksmith · Tue Jun 21, 2022 8:09 pm

Great work MikroTik! Now if we could specify the working directory for container pull operation, that would be great!

Now I am running Ubiquiti Unifi controller on MikroTik router!

Tue Jun 21, 2022 9:13 pm

specifying directory will be possible in next release

ghostinthenet · Tue Jun 21, 2022 9:23 pm

Now that we're running containers under RouterOS' Linux kernel, it would be good to know what features are compiled in. Does MikroTik publish (or plan to publish) the kernel config file for each RouterOS release? If we're planning to run containers that need kernel features that aren't present, I'd like to know before I do the work to create the container.

chrisfr · Wed Jun 22, 2022 10:48 am

Is there some plan for device without usb and with a small flash storage (16MB) ? Maybe you could implement ramdisk ? or NFS client ?

Is there a way to use this kind of container ( https://hub.docker.com/r/d3fk/nfs-client/ ) to have space for other container ?

thanks

pe1chl · Wed Jun 22, 2022 11:17 am

Devices with small flash storage already have a ramdisk. What we need is a ramdisk option on ALL devices (e.g. /ramdisk directory in the file space).

bma · Wed Jun 22, 2022 9:52 pm

If at reboot I start the container directly from the scheduler at Start Time 'startup' with On-Event '/container/start 0' it silently fails. A fixed :delay is one workaround, but I prefer to block on some state so I wrote the following script. It waits on the dhcp client to be in a bound state. I'm not a programmer and am new to MT so I'm sure it can be improved. Perhaps it's useful to others who have a similar problem.

# container name comparison string.  ".*" to run all containers
:local containerName ".*";
#:local containerName "0110745e-6724-4c41-ab87-e21c054ebf3d|1306f802-a3d8-4436-ae41-4d991709cef0";

# timeout in seconds for up state
:local timeout 60;

# $up return - nonzero when containers can execute
:local up do={ :return [/ip/dhcp-client/print count-only where status=bound]; }

# wait for $up to be nonzero or timeout to expire
:local count 0;
:while (($count < $timeout) and ([$up] = 0)) do={
  :delay 1s
  :set count ($count+1);
}

if ([$up] = 0) do={
  :log warning "container $containerName timeout after $count seconds";
} else={
  :local min 0;
  :local max ([/container/print count-only]-1);

  :log info "container $containerName ready to start after $count seconds";
  :local startCount 0;
  for i from=$min to=$max do={
    if (([/container/print as-value]->$i->"name") ~ $containerName) do={
      :local name ([/container/print as-value]->$i->"name");
      :local tag ([/container/print as-value]->$i->"tag");

      if (([/container/print as-value]->$i->"status") = "stopped") do={
        :log info "container [$i] name=$name tag=$tag starting";
        /container/start $i;
        :set startCount ($startCount+1);
      } else={
        :log info "container [$i] name=$name tag=$tag not in stopped state";
      }
    }
  }
  :log info "container $startCount containers started";
}

FisHlaBsoMAN · Thu Jun 23, 2022 1:26 am

Hi! Could it be possible to use tun/tap interfaces? it is very necessary for tinc mesh network. Now tun module is loaded after using Zerotier interface, but no rights to create /dev/no/tun device. However, the bridge interface can be created, which is already very pleasing.
I have now created a container for myself with alpine, set up openssh and am very happy with the opportunity to deploy though custom dns and http services. If the possibility of tinc and generally access to tun/tap services appears, then it will be super cool!

haedertowfeq · Thu Jun 23, 2022 8:55 am

Hi
Anyone run adguard/adgurdhome with container
I get : can't graet /opt/adgurdhome/work/ada, permission denied

fragtion · Thu Jun 23, 2022 1:48 pm

Hi! Could it be possible to use tun/tap interfaces? it is very necessary for tinc mesh network. Now tun module is loaded after using Zerotier interface, but no rights to create /dev/no/tun device. However, the bridge interface can be created, which is already very pleasing.
I have now created a container for myself with alpine, set up openssh and am very happy with the opportunity to deploy though custom dns and http services. If the possibility of tinc and generally access to tun/tap services appears, then it will be super cool!

+1

pe1chl · Thu Jun 23, 2022 2:03 pm

That kind of features would always be difficult. How do you want the traffic to be routed? For the static veth interfaces you define that at the RouterOS side during setup of your container, but when the container would be allowed to setup dynamic interfaces (like tunnels), what would they be connected to?

johnsonX · Fri Jun 24, 2022 6:00 pm

CCR2004-16G using 7.4 Beta4 docker find the bug, add the new contanier has a chance to cover other contanier‘s config，if reboot the device，the root-dir file and config will null！

Znevna · Fri Jun 24, 2022 10:22 pm

If at reboot I start the container directly from the scheduler at Start Time 'startup' with On-Event '/container/start 0' it silently fails.

This is a scripting thing, "0" only means something only after a print was issued, otherwise it doesn't point to anything, that's why it fails, it's mentioned somewhere in the docs and/or in the forums for sure.
I've mentioned a simple way to deal with it, here: viewtopic.php?p=941801#p941801

ayufan · Sat Jun 25, 2022 1:17 pm

That kind of features would always be difficult. How do you want the traffic to be routed? For the static veth interfaces you define that at the RouterOS side during setup of your container, but when the container would be allowed to setup dynamic interfaces (like tunnels), what would they be connected to?

It is not. Containers (docker/lxc) do usually have its own namespace for PID, network, etc. It does mean that you fully control network interfaces, ip routes, iptables, and this is separate from the host. It means that you can configure VPN that will only be available in a container, and the host (mikrotik) controls how the traffic from veth should be handled.

The Mikrotik's container support (as usual for MikroTik) is somehow limited as it only allows a single `veth` as this moment, and it also appears that this `veth` cannot be bridged with local network.

And it does not allow `docker exec` into a running container, something that is very handy for debugging.

pe1chl · Sat Jun 25, 2022 1:31 pm

Yes I understand that, but I expected that people wanted their container to be a node in some VPN and that they want to make connections both to other nodes and to local networks connected to the router.
I do not see how that could be dynamically set up from inside the container. Maybe in the limited case of allowing a single local network and a single connection to the outside world you could get it working via some static configuration.

ayufan · Sat Jun 25, 2022 1:37 pm

There's a problem with filesystems mounted with `nosuid` preventing usage of `sudo` (and possibly PTY access). This is also present on all mounted disk.

# RouterOS
/interface veth
add address=172.17.0.3/16 gateway=172.17.0.1 name=ssh-veth
/container config
set registry-url=https://registry-1.docker.io
/container envs
add list=ssh name=SUDO_ACCESS value=true
add list=ssh name=PASSWORD_ACCESS value=true
add list=ssh name=USER_NAME value=user
add list=ssh name=USER_PASSWORD value=password
add list=ssh name=TZ value=Europe/Warsaw
/container
add envlist=ssh interface=ssh-veth logging=yes remote-image=linuxserver/openssh-server:latest

# Terminal
$ ssh -p 2222 user@172.17.0.3
PTY allocation request failed
$ ssh -p 2222 user@172.17.0.3 mount
/dev/sda2 on / type ext3 (rw,nosuid,nodev,noatime)
tmpfs on /dev type tmpfs (rw,nosuid,relatime,size=65536k,mode=755,uid=32768,gid=32768)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,mode=600,ptmxmode=000)

daaf · Sat Jun 25, 2022 9:23 pm

I had the same problem, install starwarsfan/alpine-sshd:latest and got good result, use the following environment variables:
ROOT_LOGIN_UNLOCKED="true"
ROOT_PASSWORD="123456"

Tuve el mismo problema, instale starwarsfan/alpine-sshd:latest y tuve buen resultado, use las siguientes variables de entorno:
ROOT_LOGIN_UNLOCKED="true"
ROOT_PASSWORD="123456"

There's a problem with filesystems mounted with `nosuid` preventing usage of `sudo` (and possibly PTY access). This is also present on all mounted disk.

# RouterOS
/interface veth
add address=172.17.0.3/16 gateway=172.17.0.1 name=ssh-veth
/container config
set registry-url=https://registry-1.docker.io
/container envs
add list=ssh name=SUDO_ACCESS value=true
add list=ssh name=PASSWORD_ACCESS value=true
add list=ssh name=USER_NAME value=user
add list=ssh name=USER_PASSWORD value=password
add list=ssh name=TZ value=Europe/Warsaw
/container
add envlist=ssh interface=ssh-veth logging=yes remote-image=linuxserver/openssh-server:latest

# Terminal
$ ssh -p 2222 user@172.17.0.3
PTY allocation request failed
$ ssh -p 2222 user@172.17.0.3 mount
/dev/sda2 on / type ext3 (rw,nosuid,nodev,noatime)
tmpfs on /dev type tmpfs (rw,nosuid,relatime,size=65536k,mode=755,uid=32768,gid=32768)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,mode=600,ptmxmode=000)

daaf · Sat Jun 25, 2022 11:26 pm

Here is a list of containers that I have successfully tested.

container01.jpg

grafana.jpg

Rhoos · Sun Jun 26, 2022 7:32 pm

Hello people, I downloaded and installed the latest firmware "7.4beta4 Testing" in order to activate the containers, however when activating the container mode,

"/system/device-mode/update container=yes"

and then rebooting my router several times, including turning off the router completely and turning it on a few seconds later and the container mode is not activated,

([admin@RB3011RRML] > system/device-mode/print
mode: enterprise

My hardware is RB3011UiAS(arm) and I'm using an external memory stick.

What am I doing wrong?
Thanks.

Larsa · Sun Jun 26, 2022 8:52 pm

Just power off/on or use the reset button (ie no soft reboot). Don't forget to install the Container package.

luma · Sun Jun 26, 2022 9:07 pm

Does anyone have this working with PiHole and with the ability to make changes? The PiHole container will run for me, but any attempt to make changes through the UI and save fails, so I cannot actually configure the thing. System is configured per the docs with an external SSD mounted as `disk1`.

[admin@router01] > /disk print
Flags: M, r - RAID-MEMBER; p - PARTITION
Columns: SLOT, MODEL, SERIAL, INTERFACE, NAME, FS, LABEL, FREE, SIZE
#    SLOT        MODEL                    SERIAL        INTERFACE         NAME   FS    LABEL         FREE      SIZE    
0    usb1        Samsung Portable SSD T3  0123456789AB	USB 2.10 480Mbps                                       232.9GiB
1 Mp usb1-part1                                                           disk1  ext4  router01-usb  230.7GiB  232.9GiB
[admin@router01] > /container export
# jun/26/2022 14:04:10 by RouterOS 7.4beta4
# software id = SX2N-XPCP
#
# model = RB5009UG+S+
# serial number = 0123456789AB
/container mounts
add dst=/etc/pihole name=etc_pihole src=/disk1/etc
add dst=/etc/dnsmasq.d name=dnsmasq_pihole src=/disk1/etc-dnsmasq.d
/container
add envlist=pihole_envs interface=veth1 mounts=dnsmasq_pihole,etc_pihole root-dir=disk1/pihole
/container config
set registry-url=https://registry-1.docker.io
/container envs
add list=pihole_envs name=TZ value=America/Detroit
add list=pihole_envs name=WEBPASSWORD value=<password>
add list=pihole_envs name=DNSMASQ_USER value=root

Rhoos · Sun Jun 26, 2022 11:57 pm

Just power off/on or use the reset button (ie no soft reboot). Don't forget to install the Container package.

Thanks a lot!! I tried with the physical reset button and this time it worked!

[admin@RB3011RRML] > system/device-mode/print
mode: enterprise
container: yes

kilodelta · Mon Jun 27, 2022 3:07 am

Does anyone have this working with PiHole and with the ability to make changes? The PiHole container will run for me, but any attempt to make changes through the UI and save fails, so I cannot actually configure the thing. System is configured per the docs with an external SSD mounted as `disk1`.

I had the same issue. In my case, I found it was best to do configuration by setting environment variables to change all the settings I wanted as defined in the PiHole Docker documentation.

An example from my router:

[admin@KDTA-1806Grace] /container/config> /container/envs/print
 0 list="pihole" name="TZ" value="UTC" 

 1 list="pihole" name="WEBPASSWORD" value="transrightsarehumanrights" 

 2 list="pihole" name="DNSMASQ_USER" value="root" 

 3 list="pihole" name="FTLCONF_REPLY_ADDR4" value="172.17.0.2" 

 4 list="pihole" name="ADMIN_EMAIL" value="nope@youwish.example" 

 5 list="pihole" name="PIHOLE_DOMAIN" value="lan" 

 6 list="pihole" name="TEMPERATUREUNIT" value="f" 

 7 list="pihole" name="DNSMASQ_LISTENING" value="all" 

 8 list="pihole" name="REV_SERVER" value="true" 

 9 list="pihole" name="REV_SERVER_DOMAIN" value="lan" 

10 list="pihole" name="REV_SERVER_TARGET" value="192.168.0.1" 

11 list="pihole" name="REV_SERVER_CIDR" value="192.168.0.0/24" 

12 list="pihole" name="PIHOLE_DNS_" value="1.1.1.2;1.0.0.2"

DaSnipe · Mon Jun 27, 2022 3:44 am

Don't want to be another +1 but Pihole won't save any data, would be nice to resolve my local domains and CNAME them.

Also curious to see if something big like Portainer would run, that would be something

tpedko · Mon Jun 27, 2022 9:31 am

Does anyone have this working with PiHole and with the ability to make changes? The PiHole container will run for me, but any attempt to make changes through the UI and save fails, so I cannot actually configure the thing. System is configured per the docs with an external SSD mounted as `disk1`.

This is a bug, waiting for a fix.
post support viewtopic.php?t=178342#p940376

daaf · Mon Jun 27, 2022 5:06 pm

After upgrading from 7.4beta4 -> 7.4beta5, my configuration under /container/mounts, /container/envs on a hAP ac3 disappeared.

Znevna · Mon Jun 27, 2022 5:35 pm

Yup, it's broken viewtopic.php?t=186583#p942105
Short version: stay away from 7.4beta5.

haedertowfeq · Mon Jun 27, 2022 5:35 pm

adguardhome don't run
log:
[fatal] Cannot create DNS data dir at /opt/adguardhome/work/data: mkdir /opt/adguardhome/work/data: permission denied

tpedko · Mon Jun 27, 2022 5:47 pm

correct envs
/container envs
add key=TZ name=pihole_envs value=Europe/Moscow
add key=ServerIP name=pihole_envs value=172.17.0.2
add key=WEBPASSWORD name=pihole_envs value=password
add key=DNSMASQ_USER name=pihole_envs value=root
/container
add envlist=pihole_envs hostname=pihole interface=veth1 logging=yes mounts=etc_pihole,dnsmasq_pihole

Znevna · Mon Jun 27, 2022 5:51 pm

Yes, "list" was replaced with "key" but it doesn't work.

daaf · Mon Jun 27, 2022 5:54 pm

"list" replaced with "name", "name" replaced with "key"

Yes, "list" was replaced with "key" but it doesn't work.

Znevna · Mon Jun 27, 2022 5:58 pm

That I didn't try. And it works?

daaf · Mon Jun 27, 2022 6:00 pm

I had to remove the containers and start over. and it worked for me.

That I didn't try. And it works?

Znevna · Mon Jun 27, 2022 6:30 pm

Nice, is the sudo bug fixed? viewtopic.php?t=178342#p940376

Rhoos · Mon Jun 27, 2022 6:54 pm

I had to remove the containers and start over. and it worked for me.
That I didn't try. And it works?

Would you be so kind as to explain to me how you remove the container, I am an apprentice in the matter?
Thanks.

daaf · Mon Jun 27, 2022 8:28 pm

It will remove containers, environments and mount points.

/container/remove [find];
/container/envs/remove [find];
/container/mounts/remove [find];

I had to remove the containers and start over. and it worked for me.

Would you be so kind as to explain to me how you remove the container, I am an apprentice in the matter?
Thanks.

tpedko · Mon Jun 27, 2022 11:17 pm

Nice, is the sudo bug fixed? viewtopic.php?t=178342#p940376

fixed!

Rhoos · Mon Jun 27, 2022 11:59 pm

Muchas gracias!

It will remove containers, environments and mount points.
Code: Select all
/container/remove [find];
/container/envs/remove [find];
/container/mounts/remove [find];
Would you be so kind as to explain to me how you remove the container, I am an apprentice in the matter?
Thanks.

Znevna · Tue Jun 28, 2022 12:02 am

Nice, is the sudo bug fixed? viewtopic.php?t=178342#p940376
fixed!

Nice! I've done a quick test with pihole and it doesn't seem to throw that error anymore, indeed.
I'll test other stuff tomorrow.
Cheers!

daaf · Tue Jun 28, 2022 8:57 am

After upgrading to version 7.4beta5 on my hAP ac3, my setup no longer worked, so I removed the containers, environments and mounts and started over.

/container/remove [find];
/container/envs/remove [find];
/container/mounts/remove [find];

Mount the syslog-ng, mariadb, grafana containers; those containers had previously left the data on the mount points on my storage drive; When mounting pihole, it had no data saved, I installed it and it worked perfectly, however, notice that it does not create the folders for the mount points on the storage unit.

gsmphoenix · Tue Jun 28, 2022 2:18 pm

Hi
I am currently running the pihole on my RB3011 and its working the only problem i have is when we power outages and I am formed to do manually startup on the container.
is the a way to do a script that can automatically start the container after every reboot even intentional reboot.

Hi,

if someone is interested, I built containers for OLSRd V1 and OLSRd V2.
I posted it on the FunkFeuer forum (german).
All containers as also experienced information about it with used commands and scripts to properly start the containers after reboot is also mentioned.

Thanks for this awesome feature in RouterOS!

infabo · Tue Jun 28, 2022 2:25 pm

scheduler system startup script

semaja2 · Wed Jun 29, 2022 3:38 am

Can we get an option to just print the log from a container, and run commands or open a cli whilst its running

tpedko · Wed Jun 29, 2022 10:17 am

someone was able to run adguardhome?
share your settings

joedoelv · Wed Jun 29, 2022 1:54 pm

In 7.4beta5 I have an issue with packets forwarding between container and host when destination is internal.

My scenario:
I'm running pihole in container but using conditional forwarding to resolve local domain hostnames/IPs by DNS server on router itself.
According to pihole logs all queries towards router were N/A.
I've tried to forward queries to bridge "dockers" IP, router LAN interface IP even to loopback bridge IP but without success.

P.S. Workaround was to have another DNS server on LAN segment to bounce queries back to router.

bma · Wed Jun 29, 2022 9:26 pm

Anyone else seeing increasing memory usage over time with running containers even when they are idle?

I noticed unexpectedly high (700MB+) memory usage in winbox on a RB5009 7.4beta5. Seems like memory usage grows over time when a container is running independent of type. It's hard to see because of the scale in the picture, but usage is growing even for the last reboot after the first container is started. The only time it doesn't increase over time is when all containers are stopped. I was expecting overall higher memory usage when running containers, but not the constant, perhaps unbounded, increase.

I can wait until it gets to 700+ again and check what each container thinks its using. Pretty sure that plus the ~230MB base will be around 500MB.

Logging is set to "no" for both containers.

rb5009_containers_memory_usage.png

Znevna · Thu Jun 30, 2022 12:12 am

someone was able to run adguardhome?
share your settings

Yeah, I'm no expert here but it seems the mounts don't get created for some reason and something else restricts mkdir from functioning properly.
I only managed to get it working by using as mounts some directories from inside the container >.> not ideal.

/container mounts
add dst=/opt/adguardhome/work name=agh_workdir src=/zdisk/containers/adguardhome/tmp
add dst=/opt/adguardhome/conf name=agh_conf src=/zdisk/containers/adguardhome/home

/container
add remote-image=adguard/adguardhome:latest interface=veth2 logging=yes mounts=agh_workdir,agh_conf root-dir=zdisk/containers/adguardhome

Rhoos · Thu Jun 30, 2022 2:45 am

someone was able to run adguardhome?
share your settings

I am a user of AdGuard Home and PiHole, the former I use in the primary DNS and the latter in the secondary, and I am very interested in putting them to work as a container inside my RB3011, but I have not been successful in configuring AdGuard Home as a container well .

Rhoos · Thu Jun 30, 2022 2:50 am

someone was able to run adguardhome?
share your settings
Yeah, I'm no expert here but it seems the mounts don't get created for some reason and something else restricts mkdir from functioning properly.
I only managed to get it working by using as mounts some directories from inside the container >.> not ideal.
Code: Select all
/container mounts
add dst=/opt/adguardhome/work name=agh_workdir src=/zdisk/containers/adguardhome/tmp
add dst=/opt/adguardhome/conf name=agh_conf src=/zdisk/containers/adguardhome/home
Znevna, you use AdGuard Home image for "arm" if it is correct please where did you get it?
Thanks.
Code: Select all
/container
add remote-image=adguard/adguardhome:latest interface=veth2 logging=yes mounts=agh_workdir,agh_conf root-dir=zdisk/containers/adguardhome

daaf · Thu Jun 30, 2022 6:36 am

It happened to me too, I had to create the mount folders by hand.

viewtopic.php?p=942234#p942234

someone was able to run adguardhome?
share your settings
Yeah, I'm no expert here but it seems the mounts don't get created for some reason and something else restricts mkdir from functioning properly.
I only managed to get it working by using as mounts some directories from inside the container >.> not ideal.
Code: Select all
/container mounts
add dst=/opt/adguardhome/work name=agh_workdir src=/zdisk/containers/adguardhome/tmp
add dst=/opt/adguardhome/conf name=agh_conf src=/zdisk/containers/adguardhome/home
Code: Select all
/container
add remote-image=adguard/adguardhome:latest interface=veth2 logging=yes mounts=agh_workdir,agh_conf root-dir=zdisk/containers/adguardhome

Znevna · Thu Jun 30, 2022 8:46 am

I've tried creating them manually but it can't write in them.

daaf · Thu Jun 30, 2022 9:06 am

I installed the "starwarsfan/alpine-sshd" container, put a mount drive pointing to my usb drive (disk3), shut down the other containers, started the "starwarsfan/alpine-sshd" container, connected via SSH and from there create the directories to mount with "mkdir".

I've tried creating them manually but it can't write in them.

Znevna · Thu Jun 30, 2022 9:11 am

I thought that doing something similar might help, but it was late last night, thank you!

Znevna · Thu Jun 30, 2022 11:25 am

Anyone else seeing increasing memory usage over time with running containers even when they are idle?

I have two containers currently added, you can see in the first part of the graph AdGuardHome running over night with no clients, just some lists loaded.
In the 2nd part of the graph (much shorter since it ran only for 2 hours) you can see Pi-hole sitting doing nothing (AdGuardHome was stopped before starting Pi-hole), but memory increase is visible over time.

memusage adguardhome vs pihole.PNG

Since Pi-hole is made up of a lot of stuff underneath you can't know what part of it is leaking memory, or even if it's something that got broken in the recent releases.
Meh, something to consider if someone wants to run Pi-hole

Thu Jun 30, 2022 11:30 am

I doubt it is pihole on itself leaking memory.
I run it on a ... Pi ( duh !) but have never experienced reboots or crashes. It already runs some months ( which reminds me I need to check the updates on that little bugger).

Znevna · Thu Jun 30, 2022 11:32 am

I have it running on a Raspberry too, but it didn't see any updates in the last year

probably that's why it didn't crash yet, lol (if it's not broken I'm not fixing it).
Like I've said, Pi-hole is made up of a modified dnsmasq, lighthttpd and other stuff, any of them could've caught a memory leak in recent versions.
AdGuardHome is just one binary.

bma · Thu Jun 30, 2022 1:14 pm

Ubuntu with just sshd running is also seeing increasing usage so it's not just pi-hole. Also, if I stop both containers after 500MB is used then I only free up ~80MB for a total of 420MB used. If the containers are never started after a reboot it's only at 230MB.

I'm hoping someone from MT could make sure there isn't some (debug?) data structure in the containers implementation growing without cleanup or leaking.

Znevna · Thu Jun 30, 2022 1:16 pm

Then my AdGuardHome would've seen increased memory usage, yet it didn't increase.
And you can always set how much memory a container can use

bma · Thu Jun 30, 2022 1:18 pm

Not necessarily. It could be tied to something the container is doing in the background. Adguard maybe completely idle vs ubuntu/sshd and pi-hole still do that thing that triggers In any case, I welcome a response from MT directly.

Znevna · Thu Jun 30, 2022 1:38 pm

I wouldn't point fingers without more data.
You say you have the same problem with debian, pi-hole is also debian based, adguardhome is alpine based.
I don't have time today for tests, we'll see.

johnsonX · Fri Jul 01, 2022 4:17 am

Is it possible to support the function of upgrading docker images?

fragtion · Fri Jul 01, 2022 4:34 pm

Doesn't look like it's possible to do a "forced restart" (or poweroff) with some popular cloud providers, like amazon AWS... making it tricky to enable container device-mode on CHR/X86 in those cases. What's the recommended approach for such scenarios?

devinganger · Sun Jul 03, 2022 12:17 am

Doesn't look like it's possible to do a "forced restart" (or poweroff) with some popular cloud providers, like amazon AWS... making it tricky to enable container device-mode on CHR/X86 in those cases. What's the recommended approach for such scenarios?

Use the cloud provider's native Docker solution, perhaps?

devinganger · Mon Jul 04, 2022 6:58 am

Is it fair to assume that the MT containers do not support the equivalent of Docker bridge mode networking?

And if that is correct, is future support for bridged mode on the roadmap?

fragtion · Tue Jul 05, 2022 2:45 am

documentation says about mounts:

src= points to RouterOS location (could also be src=disk1/etc_pihole if, for example, You decide to put configuration files on external USB media), dst= points to defined location (consult containers manual/wiki/github for information on where to point). If src directory does not exist on first time use then it will be populated with whatever container have in dst location.

However, after adding a mount point to a container and starting it up, the "src" directory (valid path) was not created automatically. Anyone else had this problem with mounts?

I could create the folder manually with SMB which then seems to mount successfully, but is then read-only to the container when it starts up

Also, even if I don't use mounts, it seems any changes made in the root dir from within the container, are persistent anyway. Is this by design?

Edit: I see I'm not the only one having problems with mounts, hopefully this gets fixed in next version

lorenzo95 · Tue Jul 05, 2022 8:39 pm

Figured I give container support a try since I didn't the last time it was released.
Turns out you can run tailscale as a userspace container on it without a lot of issues. This is using wireguard-go under the hood I believe.
I am now accessing winbox via tailscale. Not a bad deal.
They already provide a container that is meant for kubernetes but it just needs a startup script to run on mikrotik and voila.

It's a quick and dirty test but maybe someone else wants to play with it.

All I did was make a dockerfile:

FROM ghcr.io/tailscale/tailscale:unstable
COPY run.sh /run.sh
CMD "/run.sh"

and the run.sh looks like this to get it working:

echo "Starting..."
if [[ -e /tmp/tailscaled.sock ]]; then
   echo "Deleting previous sock file"
   rm -rf /tmp/tailscaled.sock
fi

echo "Starting tailscaled"
tailscaled --tun=userspace-networking --socket=/tmp/tailscaled.sock &> /dev/null &
echo "Done, PID=$!"
PID=$!

echo "Waiting for sock file"
COUNTER=0
while [[ ! -e /tmp/tailscaled.sock ]]; do
    echo The counter is $COUNTER
    let COUNTER=COUNTER+1 
    ls -l /tmp/tailscaled.sock
    sleep 1
done
echo "Sock file found at:"
ls -l /tmp/tailscaled.sock

echo "Starting tailscale"
tailscale --socket=/tmp/tailscaled.sock up --accept-dns=false --accept-routes=false --advertise-exit-node --advertise-routes="172.17.0.0/16" --authkey=tskey-...
echo "Done, tailscale started"

wait ${PID}
echo "Done"

Obviously this can be done with env variables and loops to be more reliable and functional. I just wanted to see if it would work at all and it does.

Cheers

daaf · Sat Jul 09, 2022 10:23 am

Excellent progress, tomorrow I will be reporting some things about version 7.4rc2

Shell containers.png

Znevna · Sat Jul 09, 2022 10:45 am

Nice! I'm still on 7.4beta5 as I'm currently in the other side of the country.
But a nice detailed changelog would've been nice, again.
Did they fix the mount folder creation?

Mon Jul 11, 2022 10:12 am

mount issues are still not fully fixed - but we are working on this issue.
is there any more functionality we should consider adding before container goes to stable release?

Mon Jul 11, 2022 10:15 am

Is it fair to assume that the MT containers do not support the equivalent of Docker bridge mode networking?

And if that is correct, is future support for bridged mode on the roadmap?

how is "bridge mode" any different from current implementation?

tpedko · Mon Jul 11, 2022 2:12 pm

home adguard not working after upgrade from rc1 to rc2
log
[info] AdGuard Home, version v0.107.7
[info] Initializing auth module: /opt/adguardhome/work/data/sessions.db
[error] auth: open DB: /opt/adguardhome/work/data/sessions.db: open /opt/adguardhome/work/data/sessions.db: permission denied
[fatal] Couldn't initialize Auth module

if you delete and recreate, it works, but you need to reconfigure adguad

daaf · Mon Jul 11, 2022 9:03 pm

1-A better management of the file system.
For example, initially I want to create this structure to work with containers:

disk1
 |
 +-- docker
	  |
	  +-- containers
	  |
	  +-- images
	  |
	  +-- mount
	  |
	  +-- tmp

Currently (ROS7.4rc2) it can only be done via FTP, however this leaves the created folders without sufficient permissions, which could be solved with point 3 below.

2-Facility to copy the units mounted from one router to another or to copy a backup previously made.
Currently (ROS7.4rc2) it can only be done via FTP or File through Winbox, however this leaves the mount folders without sufficient permissions, which could be solved with point 3 below.

3-Being able to set sufficient permissions to a folder so that the Docker process can use it.

mount issues are still not fully fixed - but we are working on this issue.
is there any more functionality we should consider adding before container goes to stable release?

pe1chl · Mon Jul 11, 2022 9:20 pm

is there any more functionality we should consider adding before container goes to stable release?

Adding a ramdisk to all router models also those that do not have 16MB flash. To preserve backward compatibility, create a /ramdisk directory in the file space for it.
Allow containers (images) to be setup in ramdisk (i.e. handle the situation that they are completely lost on reboot and will be re-uploaded by the user and started again).

r00t · Mon Jul 11, 2022 9:38 pm

Yeah, proper file/dir operations in winbox are badly needed: create directories, change permissions, usual basic stuff that's still missing...
Also it would be nice to have button to download directory as .tar.gz and to upload (and unpack) .tar.gz archive. Because on windows you can't easily copy and preserve linux fs attributes.
Downloading/uploading files/dirs as tar would make that possible in easy and widely compatible way.

Tue Jul 12, 2022 12:41 am

download directory as .tar.gz and to upload (and unpack) .tar.gz archive.

If the facilities for adding that aren't available on-device already, I'm not sure it'd be a good use of code space. Instead, it'd be easy enough to write a wrapper around this NPK library to convert to/from tarballs, zip archives, etc.

kovalski · Wed Jul 13, 2022 4:38 pm

Doesn't look like it's possible to do a "forced restart" (or poweroff) with some popular cloud providers, like amazon AWS... making it tricky to enable container device-mode on CHR/X86 in those cases. What's the recommended approach for such scenarios?

I'm having the same problem on AWS.

ayufan · Wed Jul 13, 2022 9:35 pm

download directory as .tar.gz and to upload (and unpack) .tar.gz archive.
The layers being pulled from registry of container images are `.tar.gz` and MikroTik unpacks them.

kobuki · Thu Jul 14, 2022 9:27 pm

Will it be possible to add capabilities to containers? For instance, cap NET_ADMIN is required to create/use TUN/TAP virtual network devices for various functions. OpenVPN or other VPN solutions, like OpenConnect requires this functionality.

Fri Jul 15, 2022 10:06 am

Mount issues should be fixed in next release.

Will it be possible to add capabilities to containers? For instance, cap NET_ADMIN is required to create/use TUN/TAP virtual network devices for various functions. OpenVPN or other VPN solutions, like OpenConnect requires this functionality.

currently this wouldn't be possible.

pe1chl · Fri Jul 15, 2022 10:33 am

Will it be possible to add capabilities to containers? For instance, cap NET_ADMIN is required to create/use TUN/TAP virtual network devices for various functions. OpenVPN or other VPN solutions, like OpenConnect requires this functionality.
currently this wouldn't be possible.

Maybe you can add such network config capability to the RouterOS container config/setup menus? It could mean certain software (after adaptation) could use the network devices previously setup, without the container having permission to do it by itself.

kobuki · Fri Jul 15, 2022 11:34 am

Maybe you can add such network config capability to the RouterOS container config/setup menus? It could mean certain software (after adaptation) could use the network devices previously setup, without the container having permission to do it by itself.

For one, OpenVPN requires this cap after startup. So if this is not possible, then the possibilities are limited to other appliances that are not network-related. It would be a disappointment, really. Proxies, DNS and other things using only sockets or listening ports would work, but adding a new VPN solution is a no go as things stand. Too bad, it had my hopes up.

pe1chl · Fri Jul 15, 2022 11:48 am

Maybe you can add such network config capability to the RouterOS container config/setup menus? It could mean certain software (after adaptation) could use the network devices previously setup, without the container having permission to do it by itself.
For one, OpenVPN requires this cap after startup.

OpenVPN is open source so it can be modified to adapt to a different situation.

kobuki · Fri Jul 15, 2022 12:05 pm

OpenVPN is open source so it can be modified to adapt to a different situation.

Are you being sarcastic? But anyway, no, it can't. Using the tuntap kernel module, tun/tap devices, network admin functions, etc. is in its very foundation. And anyway, who would have the capability to make such changes? Even Mikrotik seem to have failed at it.

pe1chl · Fri Jul 15, 2022 2:24 pm

OpenVPN is open source so it can be modified to adapt to a different situation.
Are you being sarcastic? But anyway, no, it can't.

Sure it can! I have written software that uses TUN/TAP myself. It requires privileges to create a TUN or TAP interface but once you have done that and opened it you can pass it along as an fd that can be used by less-privileged code.
So it would be possible to arrange that a container config can create a tun/tap interface and pass it to the container e.g. as fd 3, and the OpenVPN server can be modified to not create the interface itself but assume it is already open on fd 3 (via some parameter).
That would be a simple modification and when you can get it accepted by the upstream it would be a one-time effort (vs a patch you need to apply whenever you get a new version and compile it).

kobuki · Fri Jul 15, 2022 3:05 pm

Sure it can! I have written software that uses TUN/TAP myself. It requires privileges to create a TUN or TAP interface but once you have done that and opened it you can pass it along as an fd that can be used by less-privileged code.
So it would be possible to arrange that a container config can create a tun/tap interface and pass it to the container e.g. as fd 3, and the OpenVPN server can be modified to not create the interface itself but assume it is already open on fd 3 (via some parameter).
That would be a simple modification and when you can get it accepted by the upstream it would be a one-time effort (vs a patch you need to apply whenever you get a new version and compile it).

If it's indeed the case, then there's some faint hope. But it requires changes in ROS and also requires changes in OVPN. And OVPN is just one application that uses TUN/TAP and/or capabilities. All the rest would need a similar change in upstream specifically for supporting containers in ROS, which, let's be honest, is a niche user base. And that in itself is pretty discouraging WRT container support in ROS. It would be much more efficient and simple if some specific caps would be allowed to be changed in ROS. No other changes would be needed. Naturally, allowing caps in a router for random apps in CTs poses a security risk (containers in themselves do as well), but I think it should be left to the users to deal with it by using or not using the feature.

pe1chl · Fri Jul 15, 2022 4:30 pm

There will always be some changes or requirement for configuration. Creating a TUN or TAP is not enough, you also need to configure how it is to be connected to the remainder of the network.
Even when you run OpenVPN natively on a Linux box you will need to do that. So it is nothing special.

kobuki · Fri Jul 15, 2022 4:40 pm

There will always be some changes or requirement for configuration. Creating a TUN or TAP is not enough, you also need to configure how it is to be connected to the remainder of the network.
Even when you run OpenVPN natively on a Linux box you will need to do that. So it is nothing special.

We're not talking about the other configuration requirements. If it's not possible to start a CT requiring tun/tap, there's nothing to talk about. The rest comes after. I'm not worried about that. So it is indeed special if we need quirks and workarounds. But let's wait and see what the final versions bring.

Tue Jul 26, 2022 10:02 am

tun/tap support will be added in next release. Anything else to consider adding before stable release?

mafiosa · Tue Jul 26, 2022 10:14 am

tun/tap support will be added in next release. Anything else to consider adding before stable release?

Yes a working guide for an MDNS repeater container

Tue Jul 26, 2022 11:03 am

Yes a working guide for an MDNS repeater container

If you have one of MikroTik's higher-end switches, this might work:

/interface/ethernet/switch/rule
add switch=switch1 mirror=yes ports=ether4,ether5 \
mac-protocol=ip dst-address=224.0.0.251/32 \
protocol=udp dst-port=5353 

add switch=switch1 mirror=yes ports=ether4,ether5 \
mac-protocol=ipv6 dst-address6=ff02::fb/128 \
protocol=udp dst-port=5353

I haven't tried it, since I haven't been able to justify use of VLANs here, but what I think this says is, "Copy packets sent to the mDNS multicast addresses on the mDNS UDP port to ether4 and ether5." Obviously you need to change "ether4,ether5" to the set of ports that need to participate in this mDNS conversation.

There's a chance it can be made to work on an RB5009 as well, by using its unique mirror-ports feature.

None of the other switch chip families look powerful enough to do this, so in that case, maybe you do need an Avahi container. This Dockerfile builds a container that's about as small as you can hope for, 5.25 MiB, a suitable base to build on. Beware that you if you put that on a RouterOS box with 16 MiB of storage, it won't have enough room left to upgrade itself. I suppose you could remove and reinstall the container each time, but…ick.

It's possible to squeeze some fat out of that by starting with "FROM scratch" instead of "FROM alpine", but then you need to have cross-compilers for your router's CPU, with which you build up Avahi and all of its dependencies. I suspect you can get it under 1 MiB that way, but it'll be a lot of work.

Thus my preference for the switch chip rule, if that'll work.

Someone please try it and let me know.

kobuki · Tue Jul 26, 2022 11:34 am

tun/tap support will be added in next release. Anything else to consider adding before stable release?

That sounds good! How will it be supported? Will we need to create them on the router (host), or will it be possible to apply NET_ADMIN to a specific container so it can add it for itself? I think many software would need to be modified if it can use, but not create the tun/tap device inside the container. That would render the feature unusable as far as I understand.

kobuki · Tue Jul 26, 2022 12:50 pm

tun/tap support will be added in next release. Anything else to consider adding before stable release?

Well, just another idea. As I plan to run Suricata on an x86 host, I read this page and found: "This container will attempt to run Suricata as a non-root user provided the containers has the capabilities to do so. In order to monitor a network interface, and drop root privileges the container must have the sys_nice, net_admin, and net_raw capabilities. If the container detects that it does not have these capabilities, Suricata will be run as root."

The possibility of adding those caps (for security, it's always better to run unprivileged and/or as a normal user) and the ability of running something like Suricata (or other similar IPS/IDS appliances) on a more powerful MT router would be pretty nice. It would open up a whole lot of new possibilities.

cklee234 · Wed Jul 27, 2022 3:40 am

tun/tap support will be added in next release. Anything else to consider adding before stable release?

Could you allow systemd to work inside container?

kobuki · Wed Jul 27, 2022 10:04 am

tun/tap support will be added in next release. Anything else to consider adding before stable release?
Could you allow systemd to work inside container?

It's practically never needed (but not impossible) for a container. A Docker container is not a VM nor an LXC container that provides full OS virtualization. If you need control over your process(es) in a CT, you could use something like the s6 overlay. If you need systemd, you're probably better off with OS virtualization.

Wed Jul 27, 2022 10:34 am

Could you allow systemd to work inside container?

NO!

That sounds good! How will it be supported?

/dev/net/tun device now is available for container use

kobuki · Wed Jul 27, 2022 11:37 am

Could you allow systemd to work inside container?
NO!

That sounds good! How will it be supported?
/dev/net/tun device now is available for container use

I'll try as soon as I can after work. Is net_admin capability available along with tun?

elbob2002 · Wed Jul 27, 2022 11:41 am

Add your tun/taps as variables for your container. I tested with Zerotier in an Alpine Linux image and it works perfectly:

ZTTUNTAP.png

kobuki · Wed Jul 27, 2022 11:43 am

Add your tun/taps as variables for your container. I tested with Zerotier in an Alpine Linux image and it works perfectly:

ZTTUNTAP.png

Nice!

CTassisF · Wed Jul 27, 2022 5:06 pm

I'm running RouterOS 7.5beta4 on my hAP ac³ and:
1- mounts still doesn't work for me;
2- I noticed that a default route is not being added to my container, even though gateway is defined in interface/veth.

[cesar@MikroTik] > /interface/veth/export 
# jul/27/2022 13:49:44 by RouterOS 7.5beta4
/interface veth
add address=172.31.0.1 gateway=172.31.0.254 name=ripe-atlas

[cesar@MikroTik] > /container/export 
# jul/27/2022 13:57:35 by RouterOS 7.5beta4
/container mounts
add dst=/var/atlas-probe/etc name=ripe-atlas-etc src=/disk1/ripe-atlas-etc
add dst=/var/atlas-probe/status name=ripe-atlas-status src=/disk1/ripe-atlas-status
/container
add dns=172.31.0.254 hostname=ripe-atlas interface=ripe-atlas logging=yes mounts=ripe-atlas-etc,ripe-atlas-status root-dir=disk1/ripe-atlas workdir=/var/atlas-probe
/container config
set ram-high=128.0MiB registry-url=https://registry-1.docker.io tmpdir=disk1/container-tmp

[cesar@MikroTik] > /file/print 
Columns: NAME, TYPE, SIZE, CREATION-TIME
 #  NAME                                    TYPE             SIZE    CREATION-TIME       
...
26  disk1                                   disk                     jul/27/2022 13:36:31
27  disk1/lost+found                        directory                jul/14/2022 15:18:28
28  disk1/container-tmp                     directory                jul/27/2022 13:36:41
29  disk1/ripe-atlas                        container store          jul/27/2022 13:36:36

[cesar@MikroTik] > /container/shell number=0
/ # route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
/ # ip route show
/ # ifconfig 
eth0      Link encap:Ethernet  HWaddr 32:14:89:BF:50:B9  
          inet addr:172.31.0.1  Bcast:0.0.0.0  Mask:0.0.0.0
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

/ # 
done

[cesar@MikroTik] >

The container I'm trying to run is this one: https://hub.docker.com/r/ctassisf/ripe-atlas-alpine

These configs were created from scratch after upgrading to 7.5beta4, nothing was reused from before the upgrade.

Wed Jul 27, 2022 5:25 pm

I'm running RouterOS 7.5beta4 on my hAP ac³ and:
1- mounts still doesn't work for me;
2- I noticed that a default route is not being added to my container, even though gateway is defined in interface/veth.

1. mounts needs to be removed and created again if upgraded from version with broken mounts.
2. ip route adds succesfully for me:

[admin@MikroTik] > container/shell 2
root@MikroTik:/# ip route show
default via 172.17.0.1 dev eth0 
172.17.0.0/24 dev eth0 proto kernel scope link src 172.17.0.4

Is veth interface running (R flag)?

/interface veth print

if not restart router - if wrong values have been provided to veth interface after creation it will lose R-status, and there is no way to re-initialize it at the moment.

edit:
i just tried locally - your veth configuration is invalid - you are missing bitmask to address, set address to 172.31.0.1/24

CTassisF · Wed Jul 27, 2022 6:25 pm

i just tried locally - your veth configuration is invalid - you are missing bitmask to address, set address to 172.31.0.1/24

Thanks! Default route is working now that veth is configured correctly, but mounts are still broken.

[cesar@MikroTik] > /interface/veth/print 
Flags: X - disabled; R - running 
 0  R name="ripe-atlas" address=172.31.0.1/24 gateway=172.31.0.254 

[cesar@MikroTik] > /container/export 
# jul/27/2022 15:19:10 by RouterOS 7.5beta4
/container config
set ram-high=128.0MiB registry-url=https://registry-1.docker.io tmpdir=disk1/container-tmp

[cesar@MikroTik] > /file/print 
Columns: NAME, TYPE, SIZE, CREATION-TIME
 #  NAME                                    TYPE        SIZE    CREATION-TIME       
...
26  disk1                                   disk                jul/27/2022 15:18:58
27  disk1/lost+found                        directory           jul/14/2022 15:18:28

[cesar@MikroTik] > /container/mounts/add dst=/var/atlas-probe/etc name=ripe-atlas-etc src=disk1/ripe-atlas-etc

[cesar@MikroTik] > /container/mounts/add dst=/var/atlas-probe/status name=ripe-atlas-status src=disk1/ripe-atlas-status

[cesar@MikroTik] > /container/add dns=172.31.0.254 hostname=ripe-atlas interface=ripe-atlas mounts=ripe-atlas-etc,ripe-atlas-status remote-image=ctassisf/ripe-atlas-alpine:arm32v7 root-dir=disk1/ripe-atlas logging=yes 

[cesar@MikroTik] > /log/print 
...
 15:19:24 container,info,debug importing remote image: ctassisf/ripe-atlas-alpine, tag: arm32v7
 15:19:24 system,info item added by cesar
 15:19:26 container,info,debug getting layer sha256:6366ba92f08e2418e90171f1e34bd86ecd50fdc95953b3f33b8943c143518eca
 15:19:30 container,info,debug layer sha256:6366ba92f08e2418e90171f1e34bd86ecd50fdc95953b3f33b8943c143518eca downloaded
 15:19:30 container,info,debug getting layer sha256:7b664ffcb01348f101ae9482b4449efa814ca80e18f5868b5dab7338f5eb4781
 15:19:32 container,info,debug layer sha256:7b664ffcb01348f101ae9482b4449efa814ca80e18f5868b5dab7338f5eb4781 downloaded
 15:19:32 container,info,debug getting layer sha256:88339ef1f59938ee0a99f3ffc2e42eedb2f7873706e2fc7f8827213dc00ba6fe
 15:19:33 container,info,debug layer sha256:88339ef1f59938ee0a99f3ffc2e42eedb2f7873706e2fc7f8827213dc00ba6fe downloaded
 15:19:33 container,info,debug getting layer sha256:fc720c19a07d46fe07832f7da19961fb22103a44bf269b14c779cfeb3c98dcc2
 15:19:35 container,info,debug layer sha256:fc720c19a07d46fe07832f7da19961fb22103a44bf269b14c779cfeb3c98dcc2 downloaded
 15:19:36 container,info,debug getting layer sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1
 15:19:37 container,info,debug layer sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1 downloaded
 15:19:37 container,info,debug import successful, container 26844012-e16b-4b21-a097-db3f6a22bcb1

[cesar@MikroTik] > /container/print 
 0 name="26844012-e16b-4b21-a097-db3f6a22bcb1" tag="ctassisf/ripe-atlas-alpine:arm32v7" os="linux" arch="arm" interface=ripe-atlas root-dir=disk1/ripe-atlas mounts=ripe-atlas-etc,ripe-atlas-status dns=172.31.0.254 hostname="ripe-atlas" 
   workdir="/var/atlas-probe" logging=yes status=stopped 

[cesar@MikroTik] > /file/print 
Columns: NAME, TYPE, SIZE, CREATION-TIME
 #  NAME                                    TYPE             SIZE    CREATION-TIME       
...
26  disk1                                   disk                     jul/27/2022 15:19:24
27  disk1/lost+found                        directory                jul/14/2022 15:18:28
28  disk1/container-tmp                     directory                jul/27/2022 15:19:37
29  disk1/ripe-atlas                        container store          jul/27/2022 15:19:30

[cesar@MikroTik] > /container/start 0

[cesar@MikroTik] > /container/print 
 0 name="26844012-e16b-4b21-a097-db3f6a22bcb1" tag="ctassisf/ripe-atlas-alpine:arm32v7" os="linux" arch="arm" interface=ripe-atlas root-dir=disk1/ripe-atlas mounts=ripe-atlas-etc,ripe-atlas-status dns=172.31.0.254 hostname="ripe-atlas" 
   workdir="/var/atlas-probe" logging=yes status=running 

[cesar@MikroTik] > /file/print 
Columns: NAME, TYPE, SIZE, CREATION-TIME
 #  NAME                                    TYPE             SIZE    CREATION-TIME       
...
26  disk1                                   disk                     jul/27/2022 15:19:24
27  disk1/lost+found                        directory                jul/14/2022 15:18:28
28  disk1/container-tmp                     directory                jul/27/2022 15:19:37
29  disk1/ripe-atlas                        container store          jul/27/2022 15:19:30

[cesar@MikroTik] > /container/shell number=0
/ # route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         172.31.0.254    0.0.0.0         UG    0      0        0 eth0
172.31.0.0      0.0.0.0         255.255.255.0   U     0      0        0 eth0
/ # ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: seq=0 ttl=117 time=3.544 ms
64 bytes from 8.8.8.8: seq=1 ttl=117 time=3.656 ms
^C
--- 8.8.8.8 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 3.544/3.600/3.656 ms
/ # 
done

[cesar@MikroTik] >

Wed Jul 27, 2022 6:42 pm

i think i got where it goes wrong. As a workaround currently you can after adding container start it, stop it and start again. Mounts should appear. we will try to fix this in next releases

CTassisF · Wed Jul 27, 2022 11:32 pm

i think i got where it goes wrong. As a workaround currently you can after adding container start it, stop it and start again. Mounts should appear. we will try to fix this in next releases

Thanks for the workaround, seems to be working as expected.

kobuki · Thu Jul 28, 2022 12:42 am

I got around testing the container functionality with OpenVPN 2.5.7, Alpine Linux, x86 VM and 7.15beta. It's working, with a few oddities.

- The CT can't write to the mounted dirs - is that normal? Is there a way to write to them or to any place that's visible on the host? I need the container to write logs and status files.
- I'm not sure that adding caps works as shown in this post - the /dev/net/tun node already exists in the container per config, so there's no need to create it via entrypoint, for instance. OVPN can add an IP, but can't remove the added IP at CT shutdown.

Many times a change can be only effective if I copy the container config into a new one and remove the old one, like updating the image .tar file or changing the address of the veth interface. I had to do this with the docker bridge and veth interface as well. But I guess this is beta so it will get fixed, eventually.

noradtux · Thu Jul 28, 2022 10:30 am

tun/tap support will be added in next release. Anything else to consider adding before stable release?

I'd like to see IPv6 support. Or is there a way to get v6 communication with a container? Most of my VPNs are v6-only nowadays, so I can't really use containers as is.

Thu Jul 28, 2022 1:36 pm

I'd like to see IPv6 support.

thanks, noted.

- The CT can't write to the mounted dirs - is that normal? Is there a way to write to them or to any place that's visible on the host? I need the container to write logs and status files.

mount folders are not browsable from RouterOS, but you can access file through ftp,sftp, etc.

- I'm not sure that adding caps works as shown in this post - the /dev/net/tun node already exists in the container per config, so there's no need to create it via entrypoint, for instance. OVPN can add an IP, but can't remove the added IP at CT shutdown.

/dev/net/tun should be available without creating new ENV variables.

kobuki · Thu Jul 28, 2022 3:10 pm

- The CT can't write to the mounted dirs - is that normal? Is there a way to write to them or to any place that's visible on the host? I need the container to write logs and status files.
mount folders are not browsable from RouterOS, but you can access file through ftp,sftp, etc.

I see the mounts, but I want to write to them, which is not possible (permission error).

Thu Jul 28, 2022 6:00 pm

I see the mounts, but I want to write to them, which is not possible (permission error).

I can't reproduce this. Please write to support with exact steps how are you getting this error and reference to this forum thread.

kobuki · Thu Jul 28, 2022 7:41 pm

I see the mounts, but I want to write to them, which is not possible (permission error).
I can't reproduce this. Please write to support with exact steps how are you getting this error and reference to this forum thread.

I used a test x86 KVM VM (fresh install from CD) with a single drive, can this be a problem? Should I attach a secondary disk and retry?

kobuki · Thu Jul 28, 2022 9:00 pm

I see the mounts, but I want to write to them, which is not possible (permission error).
I can't reproduce this. Please write to support with exact steps how are you getting this error and reference to this forum thread.

Alright, it works, but only in a certain way - as your workaround suggests, starting/stopping the CT makes the mount directory appear and I can access its contents via SFTP. It seems to copy the contents of the pre-existing directory from inside the CT with original permissions. Weird, but not a serious problem.

But - if I pre-create the mount directory via SFTP on the host, it appears to have a permission of 0700 and it cannot be changed. That can cause issues.

Other than these, it works as expected, so far. A little bit unusual compared to standard docker tools, but that's not an issue.

CTassisF · Fri Jul 29, 2022 2:49 am

I'm seeing an error when trying to add a container using this remote image: zabbix-proxy-sqlite3:alpine-6.0-latest

Logs:

23:25:41 container,info,debug importing remote image: zabbix-proxy-sqlite3, tag: alpine-6.0-latest 
23:25:41 system,info item added by cesar 
23:25:43 container,info,debug error response getting manifests: 401 
23:25:43 container,info,debug was unable to import, container 83158f44-edaf-45e5-a46f-df86a1039830

Should I open a support ticket?

elpeh · Fri Jul 29, 2022 12:10 pm

In SUP-85158 asking for RAM disk support on more devices I got the reply three weeks ago:
"as a workaround there is a possibility that we will include some network storage mount options, that will allow mounting HW storage devices from other MikroTik devices over network."
Even with this beeing only a possibility, please allow use as container mounts, too, if implemented.

kobuki · Fri Jul 29, 2022 1:30 pm

I'm seeing an error when trying to add a container using this remote image: zabbix-proxy-sqlite3:alpine-6.0-latest

Logs:
Code: Select all
23:25:41 container,info,debug importing remote image: zabbix-proxy-sqlite3, tag: alpine-6.0-latest 
23:25:41 system,info item added by cesar 
23:25:43 container,info,debug error response getting manifests: 401 
23:25:43 container,info,debug was unable to import, container 83158f44-edaf-45e5-a46f-df86a1039830 
Should I open a support ticket?

Are you sure you provided the URL for the router's architecture?

CTassisF · Fri Jul 29, 2022 4:57 pm

Are you sure you provided the URL for the router's architecture?

Yes.

I tried on a RB5009 and this zabbix/zabbix-proxy-sqlite3:alpine-6.0-latest tag has linux/arm64 in it: https://hub.docker.com/layers/zabbix-pr ... xt=explore

pihole/pihole:latest, which is used as an example in the documentation, worked fine for me and it also has linux/arm64 in it: https://hub.docker.com/layers/pihole/pi ... xt=explore

Mon Aug 01, 2022 8:45 am

I tried on a RB5009 and this zabbix/zabbix-proxy-sqlite3:alpine-6.0-latest tag has linux/arm64 in it: https://hub.docker.com/layers/zabbix-pr ... xt=explore

remote-image=zabbix/zabbix-proxy-sqlite3:alpine-6.0-latest imported on mine 5009 without any problems.
Why it returns 401 for you, there should be no authentication.

CTassisF · Wed Aug 03, 2022 9:58 pm

remote-image=zabbix/zabbix-proxy-sqlite3:alpine-6.0-latest imported on mine 5009 without any problems.

Sorry, my mistake. I was trying to pull remote-image=zabbix-proxy-sqlite3:alpine-6.0-latest instead of the correct remote-image=zabbix/zabbix-proxy-sqlite3:alpine-6.0-latest. That is why it was returning error 401.

bma · Thu Aug 04, 2022 1:38 am

I was having problems on a RB5009 using partitioning to copy-to a backup which was related to containers. Here are the errors and workarounds in case other people are getting them:

----------
(1) Stopping all running containers appears to free up the RAM drive which prevents this overflow error on copy-to. I'm waiting to hear back if support confirms this is a bug or if I have to continue using the workaround. This may be a more general problem, and not container specific, with the RAM drive getting copied along with NAND.

[admin@MikroTik] > /partitions/copy-to part1
status: ERROR: /ram/partd/dst/rw/container/97741a79-b830-4241-9d8c-b3af117328b3/root/usr/lib/aarch64-linux-gnu/perl-base
/unicore/lib/Age/V11.pl: not enough space
----------
(2) Support was able to reproduce and debug this. You'll need to remove (not disable) all VETH interfaces to prevent the error. You can then recreate them and restart the containers.

[admin@MikroTik] > /partitions/copy-to part1
status: ERROR: /flash/rw/run/netns/11-user: Invalid argument

cklee234 · Mon Aug 08, 2022 5:45 am

Would like to ask if there is working example to use guacamole in routeros container.

I am not sure how to set up the pgsql container image to work with other guacamole images

rowant · Wed Aug 10, 2022 3:09 am

CCR2004-16G-2S+ - I followed the instructions to install pihole, with a 1Tb SSD. Initially I was getting "unable to extract" error until I manually installed a different image manually (just grabbed Ubuntu image), then pihole was able to extract, with no config change.

But both containers have been running incredibly slow. I'm talking 45 minutes for pihole to start up (see the timestamp against the log below). Ubuntu started quicker but I logged in via SSH and every command took 10's of seconds to run, even simple ls, df, top.

CPU usage is constantly 0%, this is an unused CCR2004 with basically zero other config on it.

Anyone else experienced this?

Znevna · Wed Aug 10, 2022 5:03 pm

Show the output of:

/container/config/export

cklee234 · Thu Aug 11, 2022 9:08 am

Is it possible to have more help on how to use CMD in container and how to interact with other containers?

Thu Aug 11, 2022 10:41 am

Is it possible to have more help on how to use CMD in container

By "CMD" do you mean the Linux command shells? That information is widely available, and it goes deep. There is zero reason for MikroTik to bother trying to add to that vast ocean of training.

how to interact with other containers?

The best reference is the Docker docs. The part you want is likely either the networking section or the storage section.

cklee234 · Thu Aug 11, 2022 12:38 pm

for example a shell command to interact the docker image after it strarts

Thu Aug 11, 2022 2:54 pm

for example a shell command to interact the docker image after it strarts

Yes, the subcommands of "/container" are underdocumented on the help site. Until their doc people get around to filling that out, use the CLI to discover the commands by poking around with F1 and TAB keystrokes. From there, the Docker docs will help you piece things together.

In this case, you probably want something like "/container shell 0".

cklee234 · Fri Aug 12, 2022 2:55 am

for example a shell command to interact the docker image after it strarts

Yes, the subcommands of "/container" are underdocumented on the help site. Until their doc people get around to filling that out, use the CLI to discover the commands by poking around with F1 and TAB keystrokes. From there, the Docker docs will help you piece things together.

In this case, you probably want something like "/container shell 0".

Thank you. I didn't notice this trick
will try

jvanhambelgium · Sat Aug 13, 2022 3:48 pm

Trying to get pihole working on my lab 3011 running 7.5 BETA8 but it seems the container does not start at all ? Also no logging is produced even if I as it to.
"disk3" is a fresh formatted USB-stick (ext4)
So container is pulled correctly, extracted etc but cannot start it ... does just nothing. (tried both "Winbox" and CLI)

Any tips ?

[user@gateway] /container> print
0 name="bf53d808-ed71-40fb-8444-194b2e320a3f" tag="pihole/pihole:latest" os="linux" arch="arm" interface=veth1 envlist="pihole_envs" root-dir=disk3/pihole
mounts=etc_pihole,dnsmasq_pihole dns="" workdir="disk3" logging=yes status=stopped
[user@gateway] /container>
config envs mounts add comment edit export find print remove reset set shell start stop
[user@gateway] /container> start number=0
[user@gateway] /container

[user@gateway] /container> export verbose
# aug/13/2022 14:46:23 by RouterOS 7.5beta8
# software id = XXXXXX
#
# model = RouterBOARD 3011UiAS
# serial number = XXXXXXXX
/container mounts
add dst=/etc/pihole name=etc_pihole src=/disk3/etc
add dst=/etc/dnsmasq.d name=dnsmasq_pihole src=/disk3/etc-dnsmasq.d
/container
add dns="" envlist=pihole_envs interface=veth1 logging=yes mounts=etc_pihole,dnsmasq_pihole root-dir=disk3/pihole workdir=disk3
/container config
set ram-high=512 registry-url=https://registry-1.docker.io tmpdir=disk3/pull
/container envs
add key=TZ name=pihole_envs value=Europe/Brussels
add key=WEBPASSWORD name=pihole_envs value=testpassword
add key=DNSMASQ_USER name=pihole_envs value=root
add key=FTLCONF_REPLY_ADDR4 name=pihole_envs value=172.17.0.2
[user@gateway] /container>

Znevna · Sat Aug 13, 2022 4:04 pm

ram-high without a b,k,m or g suffix means the value is in bytes, and I don't think Pi-hole runs under 512 bytes of ram that you've set as max for all the containers.

jvanhambelgium · Sat Aug 13, 2022 4:42 pm

Thanks for the tip, but it did not help.

[user@gateway] /container> export verbose
# aug/13/2022 15:39:57 by RouterOS 7.5beta8
# software id = XXXXXXXXX
#
# model = RouterBOARD 3011UiAS
# serial number = XXXXXXXXXX
/container mounts
add dst=/etc/pihole name=etc_pihole src=/disk3/etc
add dst=/etc/dnsmasq.d name=dnsmasq_pihole src=/disk3/etc-dnsmasq.d
/container
add dns="" envlist=pihole_envs interface=veth1 logging=yes mounts=etc_pihole,dnsmasq_pihole root-dir=disk3/pihole workdir=disk3
/container config
set ram-high=488.3MiB registry-url=https://registry-1.docker.io tmpdir=disk3/pull
/container envs
add key=TZ name=pihole_envs value=Europe/Brussels
add key=WEBPASSWORD name=pihole_envs value=password
add key=DNSMASQ_USER name=pihole_envs value=root
add key=FTLCONF_REPLY_ADDR4 name=pihole_envs value=172.17.0.2
[user@gateway] /container> start number=0
[user@gateway] /container>

Hmm, perhaps this BETA8 simply is broken in the container-departement....
Also the fact that I'm getting no logging is strange. While creating the container/pull/extract phase I did see some stuff in the log, but not of an actual container-start attempt...
I've added under the system > logging following

add prefix=container topics=container

Znevna · Sat Aug 13, 2022 10:26 pm

I don't know why you wrongly defined the workdir, you don't have to write anything there.
And I think that you have to start it without mounts for now, as there's currently a problem with mounts.

jvanhambelgium · Sat Aug 13, 2022 11:20 pm

I don't know why you wrongly defined the workdir, you don't have to write anything there.
And I think that you have to start it without mounts for now, as there's currently a problem with mounts.

I've removed all mounts / workdir references and cleaned up to the bare essentials. Nothing ;-( (and no logging)
This should not be happening I guess.

I'll try with another release...

UPDATE : Completely erased the container and re-deployed without mounts etc in the first place. Now the container seems to start! in stays "running" so that's something.
Now let's see if can reach the GUI...looking good...
And after some fiddling with FW-rules etc, my lab now uses the container-pihole for resolving on the RB3011.
Let's keep it running for some days...

Sun Aug 14, 2022 12:50 am

Completely erased the container and re-deployed without mounts etc in the first place. Now the container seems to start!

Some settings you make on a container when deploying it the first time from an image get baked into the container permanently. Mounts are one. Port mappings are another. You can't change either without redeploying the container. This isn't specific to RouterOS's implementation: you can cause the same problem with Docker Desktop.

Don't think of containers merely as "lightweight VMs." There's more to it, a kind of Zen of Docker that you have to get your head around before it begins to make sense. Part of it is that "long-running Docker" is nearly an oxymoron. Dockers are made to be created cookie-cutter, run until they're no longer useful, and then destroyed or recreated.

One way this is often put is the cattle vs. pets analogy.

gittubaba · Sun Aug 14, 2022 3:09 am

I have an interesting observation. Anyone able to run any "FROM scratch" images in mikrotik? As if the container only have a static binary and in last step it was build "FROM scratch" in dockerfile. It runs in my computer / vm etc... But doesn't in routeros. Extraction is successful. But container doesn't start. Always in "stopped" state. There is no error shown in log. Yes logging=yes is set. This is the package I was trying: https://github.com/sarim/stubby-docker/ ... Dockerfile. You can also reproduce with "hello-world" package.

Does the implementation of mikrotik expects certain standard directory structure / files inside the container? Without any log Its hard to debug why container didn't start.

Sun Aug 14, 2022 3:31 am

Anyone able to run any "FROM scratch" images in mikrotik?

Yes. I got this one running a few days ago.

This is the package I was trying

How did you build it? It should be a command like:

$ docker buildx build -t stubby --platform linux/arm/v7 --load .

…from the directory containing the Dockerfile.

I'd try it for you, but I'm not running random code from GitHub on my home router, thankyouverymuch.

I did skim the Dockerfile, and it looks like it's doing much the same as my Fossil container, only more complex.

Does the implementation of mikrotik expects certain standard directory structure / files inside the container?

It shouldn't. It runs CMD or ENTRY_POINT, and from then on, the container's internal binaries take over.

If you have a full-featured Docker build system (e.g. Docker Desktop) it may have QEMU set up to run ARM containers under emulation. You say the container runs there, but I'm not talking about running it as an x86_64 container, but as armv7, to eliminate unnecessary differences between the test and the deployment platforms.

gittubaba · Sun Aug 14, 2022 6:50 am

Yes. I got this one running a few days ago.

Thanks. I'll keep it as reference. It looks like its doing a bit more stuff, that RUN [ "/bin/busybox", "--install", "/bin" ], then some dev null etc.. Those may be the difference.
I tried From alpine, then removing every package using apk del musl apk-tools etc.... . If I do that then the container runs successfully. So I'm guessing its due to some directory structure.

How did you build it? It should be a command like:

Yes I tried both docker buildx, podman build etc.. The problem isn't in build, as I said if I add a bit of alpine base It works.

It shouldn't. It runs CMD or ENTRY_POINT, and from then on, the container's internal binaries take over.

Yes, thats what I knew too. But I guess there is something else at play here, mikrotik is doing something before calling the ENTRYPOINT/CMD.

If you have a full-featured Docker build system (e.g. Docker Desktop) it may have QEMU set up to run ARM containers under emulation. You say the container runs there, but I'm not talking about running it as an x86_64 container, but as armv7, to eliminate unnecessary differences between the test and the deployment platforms.

Thanks, it runs in x86_64 linux via qemu emulation. Then to be sure I started arm64 alpine. Then installed podman inside that container, then run the stubby container there, it still works there.

The problem is mikrotik's container runtime is neither docker, neither podman, neither oci etc... There is no source code or spec doc. So kinda have to shoot in the blind. Maybe there is some error logged in supout, but from my quick glance I didn't find it. There is error about can't mount container directory, and error about tap device. Not sure if those are related to container I'm trying to run, as I didn't add any mount to this container, neither doing anything tap related. I'm running rb5009 with latest 7.5beta8. So according to changelog tun/tap device should be fixed, not sure why its error appear in supout. But thats probably not the current issue I'm facing.

Thanks so much for the feedback. I'll take pointer from the fossil Dockerfile you linked and hopefully figure it out

If any mikrotik dev can provide info about standard directory structure then I'd be great help :3

rowant · Sun Aug 14, 2022 7:26 am

But both containers have been running incredibly slow. I'm talking 45 minutes for pihole to start up (see the timestamp against the log below). Ubuntu started quicker but I logged in via SSH and every command took 10's of seconds to run, even simple ls, df, top.

CPU usage is constantly 0%, this is an unused CCR2004 with basically zero other config on it.

I made the same mistake as jvanhambelgium above and had simply set ram-high to 2048 bytes. Works perfectly now after changing it to 512M.

Sun Aug 14, 2022 8:13 am

RUN [ "/bin/busybox", "--install", "/bin" ]

That tells BusyBox to install all its sub-command links. (Details) I can only speculate about whether doing this in your container will help. The only solid case I could think of is that it might be calling system(3), thus needing a /bin/sh to exist, but that hypothesis isn't borne out by a quick grep over the source code involved. Still, there may be calls out to other common Unix utilities that Busybox provides.

then some dev null

Fossil needs /dev/null and /dev/urandom to exist inside the runtime environment. Although they do exist in the "FROM scractch" container without this help, Fossil runs in a chroot() jail, so we need copies of those dev nodes inside the jail. (Details)

I tried From alpine, then removing every package using apk del musl apk-tools etc.... . If I do that then the container runs successfully. So I'm guessing its due to some directory structure.

Running "strace -e open /opt/stubby/bin/stubby" inside the Alpine-based container will get you a list of files the program accesses, which may prove enlightening.

marlab · Sun Aug 14, 2022 4:16 pm

Hi, gurus

.
Since ROS still does not support tls-auth for OpenVPN (so many years...

), I've been trying to find a workaround with OpenVPN client (giggio/openvpn-arm) running within the ROS container. However, no success. The client within container successfully connects to VPN server, I can reach the local tunnel endpoint, but for some reason packets are not being forwarded through the OpenVPN tunnel. Has anyone been successful with it?

kobuki · Sun Aug 14, 2022 5:44 pm

Hi, gurus .
Since ROS still does not support tls-auth for OpenVPN (so many years... ), I've been trying to find a workaround with OpenVPN client (giggio/openvpn-arm) running within the ROS container. However, no success. The client within container successfully connects to VPN server, I can reach the local tunnel endpoint, but for some reason packets are not being forwarded through the OpenVPN tunnel. Has anyone been successful with it?

Same issue here using a self-built Alpine based image. VPN connection is stablished, I can ping the VPN endpoint just fine on the server side of the tunnel from a windows OVPN client, but packets are not forwarded out, even if I push a static route. Packets are dropped in the OpenVPN server on the router. Hopefully it's just a bug in the beta and not an intentional limitation.

gittubaba · Sun Aug 14, 2022 10:27 pm

RUN [ "/bin/busybox", "--install", "/bin" ]

That tells BusyBox to install all its sub-command links. (Details) I can only speculate about whether doing this in your container will help. The only solid case I could think of is that it might be calling system(3), thus needing a /bin/sh to exist, but that hypothesis isn't borne out by a quick grep over the source code involved. Still, there may be calls out to other common Unix utilities that Busybox provides.

Yeah, it turns out that was the answer. Adding busybox solves the issue. It seems like mikrotik needs more than just /bin/sh. I tried providing only /bin/sh, but that doesn't work. Surely shouldn't need all applets of busybox. Next time looking for time to waste I'll turn them off one by one to find out exactly which ones are needed. Then need to compile busybox with those applets disabled.

Good thing is it works. I set 172.17.0.2 as dns in routeros and its working good so far. Though it looks like network namespace is shared by all containers. All gets same ip of 172.17.0.2. Thats kinda limiting -_-
Feature Suggestion, integrate it with /ip/pool and /ip/dhcp/server. Let containers get ip from routeros's dhcp.

Sun Aug 14, 2022 10:59 pm

It seems like mikrotik needs more than just /bin/sh.

I can't see why, other than to support "/container shell".

Next time looking for time to waste I'll turn them off one by one

Cut your wasted time to log₂(n) by bisecting it. First "rm /bin/[a-m]*" after installation, which tells you which half of the set contains the key binary: if it breaks, it's in the a-m set, else it's in the n-z set. Bisect the implicated set — e.g. "rm /bin/a-h" to narrow it to that subset or i-m — to cut the problem size into 1/4. With 305 hard links, you can decide the matter in 9 tries. (log₂(305) = 8.25.)

jvanhambelgium · Sun Aug 14, 2022 11:45 pm

Hi,
Is my understanding correct that I could create additional "vethX" interfaces and attach containers to it ? I mean, there is nothing magical on these virtual interfaces right?
Strangely enough I can't make it work creating a "veth2" for example

veth1 = 172.17.0.2/16 with gateway 172.17.0.1
veth2 = 172.17.0.3/16 with gateway 172.17.0.1

Attaching "Adguard" to this veth2 on creating does not work. Sure the logging tells me to connect to http://172.17.0.3:3000 but that does not work.
Destroy + create again using "veth1" (=172.17.0.2) DOES work and I can run the installation-wizzard of Adguard.

I could also not "ping" the container, which I could do if I created it with "veth1" on 172.17.0.2 . It responded normally.
I have a forward FW-rules allowing all communications between "Bridge" (= main LAN) and "Dockers"

Sun Aug 14, 2022 11:57 pm

I can't make it work creating a "veth2"

By putting veth2 at 172.17.0.3/16, it's inside the subnet of veth1. That means you need to attach it to the "dockers" bridge so it's visible to all the other 172.17.0.0/16 hosts, it participates in the established NAT rule, etc.

Alternately, you can switch to a different subnet like 172.18.0.0/16, then do all of the other network setup steps for that second container subnet. This means you'll end up with a "dockers2" bridge, among other things. This method is far more complex, but it can allow for isolation between containers by imposing a routing layer, where you can jam in some RouterOS magic.

gittubaba · Mon Aug 15, 2022 12:36 am

With 305 hard links, you can decide the matter in 9 tries. (log₂(305) = 8.25.)

Hmm, very interesting, but what if mikrotik needs 10 times, spread across all over a-z

Maybe I'll just write a script to automate it

build, save, curl upload, ssh container add, start. rinse and repeat in loop

Mon Aug 15, 2022 12:57 am

what if mikrotik needs 10 times, spread across all over a-z

You'd better hope not, because drawing 10 items from a pool of 305 creates 1.6×10¹⁸ possible combinations. (Math)

Maybe I'll just write a script to automate it build, save, curl upload, ssh container add, start. rinse and repeat in loop

In principle, you should have that for any container, since easy redeployment is half the point.

However, a quintillion container redeployments is likely to burn through several routers' flash write cycle limits, not to mention several geek lifecycles to monitor it.

The more you know…

bah, dum, ba, daaah!

jvanhambelgium · Mon Aug 15, 2022 8:22 am

I can't make it work creating a "veth2"
By putting veth2 at 172.17.0.3/16, it's inside the subnet of veth1. That means you need to attach it to the "dockers" bridge so it's visible to all the other 172.17.0.0/16 hosts, it participates in the established NAT rule, etc.

Alternately, you can switch to a different subnet like 172.18.0.0/16, then do all of the other network setup steps for that second container subnet. This means you'll end up with a "dockers2" bridge, among other things. This method is far more complex, but it can allow for isolation between containers by imposing a routing layer, where you can jam in some RouterOS magic.

Indeed although I added a /ip/interface "veth2" for some reason it was not hooked on my container bridge!
Thanks for pointing that out again! Works fine now.

tpedko · Mon Aug 15, 2022 3:53 pm

after upgrade beta5 to beta8 container AdGuard not start
log
15:50:08 container,info,debug 2022/08/15 12:50:08.460390 [info] AdGuard Home, version v0.107.8
15:50:08 container,info,debug 2022/08/15 12:50:08.482883 [info] Initializing auth module: /opt/adguardhome/work/data/sessions.db
15:50:08 container,info,debug 2022/08/15 12:50:08.483123 [error] auth: open DB: /opt/adguardhome/work/data/sessions.db: open /opt/adguardhome/work/data/sessions.db: permission denied
15:50:08 container,info,debug 2022/08/15 12:50:08.483258 [fatal] Couldn't initialize Auth module
please fix it

Znevna · Mon Aug 15, 2022 5:33 pm

It didn't work right in beta5 either.

jvanhambelgium · Mon Aug 15, 2022 5:46 pm

after upgrade beta5 to beta8 container AdGuard not start
log
15:50:08 container,info,debug 2022/08/15 12:50:08.460390 [info] AdGuard Home, version v0.107.8
15:50:08 container,info,debug 2022/08/15 12:50:08.482883 [info] Initializing auth module: /opt/adguardhome/work/data/sessions.db
15:50:08 container,info,debug 2022/08/15 12:50:08.483123 [error] auth: open DB: /opt/adguardhome/work/data/sessions.db: open /opt/adguardhome/work/data/sessions.db: permission denied
15:50:08 container,info,debug 2022/08/15 12:50:08.483258 [fatal] Couldn't initialize Auth module
please fix it

The whole file-structure/permissions/mount-points thing for sure needs some work/fixing in future versions.
I also had to set manual some permissions on the directory structure because I got also permission-errors while deploying the container.
But after fixing the permissions on the "data" and "config" folders Adguard runs fine ... until the next update where I need to redo this work no doubt.

Znevna · Thu Aug 18, 2022 11:37 pm

@MikroTik can we fix the mounts? I'm guessing it's not that hard to find out why mounts are created with 32766:32766 instead of 32768:32768

/container mounts
add dst=/opt/adguardhome/conf name=agh_conf src=/zdisk/container_mounts/agh/conf
add dst=/opt/adguardhome/work name=agh_work src=/zdisk/container_mounts/agh/work

->>

 17:39:40 container,info,debug 2022/08/18 14:39:40.171926 [info] AdGuard Home, version v0.107.10
 17:39:40 container,info,debug 2022/08/18 14:39:40.172404 [info] This is the first time AdGuard Home is launched
 17:39:40 container,info,debug 2022/08/18 14:39:40.172564 [info] Checking if AdGuard Home has necessary permissions
 17:39:40 container,info,debug 2022/08/18 14:39:40.173658 [info] AdGuard Home can bind to port 53
 17:39:40 container,info,debug 2022/08/18 14:39:40.179003 [fatal] Cannot create DNS data dir at /opt/adguardhome/work/data: mkdir /opt/adguardhome/work/data: permission denied

Taking the drive out and mounting it under some ubuntu:
(yes I know it can be done using some other container without taking the drive out, I just don't know how and lack the time and/or interest to investigate).

ubuntu@ubuntu:/media/ubuntu/containers/container_mounts/agh$ ls -la
total 16
drwxr-xr-x 4 32768 32768 4096 aug 18 17:39 .
drwxr-xr-x 3 32768 32768 4096 aug 18 17:39 ..
drwxr-xr-x 2 32766 32766 4096 aug 18 17:39 conf
drwxr-xr-x 2 32766 32766 4096 aug 18 17:39 work
ubuntu@ubuntu:/media/ubuntu/containers/container_mounts/agh$ sudo chown 32768:32768 conf
ubuntu@ubuntu:/media/ubuntu/containers/container_mounts/agh$ sudo chown 32768:32768 work
ubuntu@ubuntu:/media/ubuntu/containers/container_mounts/agh$ ls -la
total 16
drwxr-xr-x 4 32768 32768 4096 aug 18 17:39 .
drwxr-xr-x 3 32768 32768 4096 aug 18 17:39 ..
drwxr-xr-x 2 32768 32768 4096 aug 18 17:39 conf
drwxr-xr-x 2 32768 32768 4096 aug 18 17:39 work

And everything works when the drive is back in RouterOS! ->>

 23:25:01 container,info,debug 2022/08/18 20:25:01.556901 [info] AdGuard Home, version v0.107.10
 23:25:01 container,info,debug 2022/08/18 20:25:01.557206 [info] This is the first time AdGuard Home is launched
 23:25:01 container,info,debug 2022/08/18 20:25:01.557325 [info] Checking if AdGuard Home has necessary permissions
 23:25:01 container,info,debug 2022/08/18 20:25:01.558153 [info] AdGuard Home can bind to port 53
 23:25:01 container,info,debug 2022/08/18 20:25:01.568534 [info] Initializing auth module: /opt/adguardhome/work/data/sessions.db
 23:25:01 container,info,debug 2022/08/18 20:25:01.580749 [info] auth: initialized.  users:0  sessions:0
 23:25:01 container,info,debug 2022/08/18 20:25:01.581431 [info] Initialize web module
 23:25:01 container,info,debug 2022/08/18 20:25:01.582222 [info] This is the first launch of AdGuard Home, redirecting everything to /install.html 
 23:25:01 container,info,debug 2022/08/18 20:25:01.582710 [info] AdGuard Home is available at the following addresses:
 23:25:01 container,info,debug 2022/08/18 20:25:01.586599 [info] Go to http://127.0.0.1:3000
 23:25:01 container,info,debug 2022/08/18 20:25:01.586796 [info] Go to http://[::1]:3000
 23:25:01 container,info,debug 2022/08/18 20:25:01.586976 [info] Go to http://192.168.70.202:3000
 23:25:01 container,info,debug 2022/08/18 20:25:01.587243 [info] Go to http://[2a02:edited]:3000

Thanks.

Thu Aug 18, 2022 11:55 pm

@MikroTik can we fix the mounts?

File ownership between the inside and outside of the container is always tricky with containers, because they don't share user and group databases.

I'm guessing it's not that hard to find out why mounts are created with 32766:32766 instead of 32768:32768

Those are user and group IDs. The question is, why those particular values?

I dug up the Dockerfile for AdGuard Home and found this line: "chown -R nobody: /opt/adguardhome". If you open a stock Alpine container, which this one is based on, you find that user "nobody" has UID and GID 65534.

To me, this suggests someone's clamping user and group IDs to 15 bits for some reason, leaving you with 2¹⁵-2 instead of 2¹⁶-2.

As for what the 32768 value is doing there, I have no idea.

Regardless, you shouldn't be using raw UIDs and GIDs. Use symbolic names. If "chown nobody /my/mount/path" doesn't do what you want, then you have a bug to complain about.

Znevna · Fri Aug 19, 2022 12:19 am

Eh?
I doubt that

chown nobody /my/mount/path

will do anything good here.
As "nobody" is usually

nobody:x:65534:65534:nobody:/nonexistent:/usr/sbin/nologin

So no, I'm pretty sure it won't work.
I'm guessing 32768 is the uid/gid that the container stuff runs under ROS, the question isn't why 32768 is used, but why are the mounts created with 32766 while all the other container stuff are created with 32768, except the container tmpdir which is created as root (0).
Even the directory that holds the mounts is created properly, but the mount directories themselves are not, like shown in the previous post.

containers$ ls -lan
total 36
drwxr-xr-x  6     0     0  4096 aug 18 17:39 .
drwxr-x---+ 3     0     0  4096 aug 19 00:01 ..
drwxr-xr-x  3 32768 32768  4096 aug 18 17:39 container_mounts
drwxr-xr-x  3 32768 32768  4096 aug 18 21:34 containers
drwxr-xr-x  3     0     0  4096 aug 18 17:38 docker
drwx------  2     0     0 16384 iun 24 14:26 lost+found

containers/containers$ ls -lan
total 12
drwxr-xr-x  3 32768 32768 4096 aug 18 21:34 .
drwxr-xr-x  6     0     0 4096 aug 18 17:39 ..
drwxr-xr-x 19 32768 32768 4096 aug 18 17:38 adguardhome

Fri Aug 19, 2022 12:24 am

While I agree that this "-2" offset is odd, what I'm saying is, the AGH container expects 65534, the UID/GID of the Alpine user "nobody." Since you're using external mounts, these values should persist across container rebuilds. If not, then that's a bug.

The exact values don't matter. Whatever they were when the container was last stopped, they should be once more when the container is recreated and restarted. If not, the filesystem isn't properly persistent. That's the real bug, not curious details about specific ID values.

Znevna · Fri Aug 19, 2022 12:33 am

Well, that command is run inside the container, not on the outside mount, isn't it?
I took the drive out again, took a peek inside the container in /opt

containers/containers/adguardhome/opt$ ls -lan
total 12
drwxr-xr-x  3 32768 32768 4096 aug 17 18:48 .
drwxr-xr-x 19 32768 32768 4096 aug 18 17:38 ..
drwxr-xr-x  4 32766 32766 4096 aug 17 18:48 adguardhome

containers/adguardhome/opt/adguardhome$ ls -lan
total 34384
drwxr-xr-x 4 32766 32766     4096 aug 17 18:48 .
drwxr-xr-x 3 32768 32768     4096 aug 17 18:48 ..
-rwxr-xr-x 1 32766 32765 35192832 aug 17 18:48 AdGuardHome
drwxr-xr-x 2 32766 32766     4096 aug 17 18:48 conf
drwxr-xr-x 2 32766 32766     4096 aug 17 18:48 work

I have no ideea where 32765 came from. MikroTik stop decrementing IDs! lol.

Wed Aug 24, 2022 5:24 pm

Thanks for detailed reports, we will look into it.

zjzxj · Sat Aug 27, 2022 9:22 am

Only directories can be mount, not files，In some environments, mount files is required

Znevna · Sat Aug 27, 2022 1:04 pm

The official docker supports volumes, not files.
LE: my bad, it can be done using bind mounts.
https://docs.docker.com/storage/bind-mounts/
Not currently implemented under RouterOS.

socs · Sun Aug 28, 2022 1:49 pm

Kudos for adding container support. I have a couple of questions related to container storage

I’m not sure how to run container rootfs on ram like in v7.1rc4. Is it possible or planned in v7.5?
I’m unable to add gzipped container images, it fails to find docker manifest. The docs seem indicate to this being supported.

Both of these options would be very nice to have when internal/external storage is limited.

Znevna · Tue Aug 30, 2022 10:12 pm

Can anyone check what's wrong with the mounts/volumes permissions in 7.5rc2? I'm not near any devices currently and I'm not trying anything fancy from a distance.
Is it the same uid/gid problem?
Thank you!

jvanhambelgium · Tue Aug 30, 2022 11:22 pm

Same story no ?
Upgrade my RB3011 to 7.5RC2 (including firmware)
Deleted existing Adguard and re-created from scratch. Then unmounted USB-stick and checked on my machine.
Strange that there is still August 19 on these items ? I've re-created container, it pulled it fresh from Github so why isn't everything August 30 then ?
(I will remove forefully and re-create again)

Screenshot from 2022-08-30 22-19-00.png

jvanhambelgium · Tue Aug 30, 2022 11:32 pm

No difference after I delete everything, including mountpoints-folder and recreate everything.
Adguard will not start and throws the well known permission errors.

Screenshot from 2022-08-30 22-31-12.png

Znevna · Wed Aug 31, 2022 8:52 am

The timestamps are weird indeed.
Thanks for confirming that the issue is still the same.

dynek · Wed Aug 31, 2022 2:22 pm

Asking for the sake of asking, but I'm pretty sure it's going to be no

Any chance to see /container available on RB1100AHx2/PPC architecture at some point ?

rextended · Wed Aug 31, 2022 2:23 pm

Znevna · Wed Aug 31, 2022 2:26 pm

Probably not, you can make a feature request.

bma · Wed Aug 31, 2022 6:21 pm

Would it be possible to add permissions for mounting cifs (smb) inside a container?

alpine:~# mount -t cifs -o rw,username=xx //yy/share /media/share
Password for xx@/yy/share:
mount error(1): Operation not permitted
Refer to the mount.cifs(8) manual page (e.g. man mount.cifs) and kernel log messages (dmesg)
mount: permission denied (are you root?)

alpine:~# dmesg
dmesg: klogctl: Operation not permitted

alpine:~# whoami
root

Wed Aug 31, 2022 6:31 pm

Would it be possible to add permissions for mounting

As far as I can tell, that requires CAP_SYS_ADMIN, the shameful secret capability that grants a user a whole raft of abilities. I wouldn't be surprised if MikroTik said, "Nah, are you nuts? We're not giving you that one! It'd let you root the whole OS."

In the normal container world, you'd mount that on the host, then map it into the container. The proper feature, then, would be an SMB client for RouterOS, which has been requested many times before.

pe1chl · Wed Aug 31, 2022 7:19 pm

The proper feature, then, would be an SMB client for RouterOS, which has been requested many times before.

Why not NFS instead of SMB? That would be more likely to get accepted and the permissions would work better.

Wed Aug 31, 2022 8:06 pm

Why not NFS

Sure, fine.

While we're dreaming, let's ask for iSCSI, too.

That would be more likely to get accepted

I dunno. Both facilities are equally available in the kernel. Most NASes speak SMB by default. Obviously Windows does, too, but what might surprise you is that Apple went over to SMB from AFP years ago. I hesitate to call SMB the lingua franca of LAN file sharing protocols, but it kinda is.

the permissions would work better

Not out of the box. NFS requires a helper like rpc.idmapd or <shudder> LDAP to get that ability. SMB actually does better on this score, because it can map IDs by name, rather than require that the UIDs and GIDs match on both ends.

ech1965 · Mon Sep 26, 2022 3:48 pm

Is it possible to "mount" usb devices in a container ?
I'd like to deploy a nut server in a container connected to my UPS

Routeros is too cumbersome for propagating ups events to the rest of my infra. ( NAS, Servers...)

Mon Sep 26, 2022 4:45 pm

Is it possible to "mount" usb devices in a container ?

Did you even try to read the docs? Thin as they currently are, they do cover this.

ech1965 · Mon Sep 26, 2022 5:07 pm

Is it possible to "mount" usb devices in a container ?

Did you even try to read the docs? Thin as they currently are, they do cover this.

Did you ever TRY to read my message ? I'm searching for HID devices like UPS or even usb/serial devices, not STORAGE devices.

Mon Sep 26, 2022 5:38 pm

Since you used the term "mount"

Mount = file system = storage.
So your wording might have been a bit "off" if that was not what you meant.

Mon Sep 26, 2022 5:42 pm

When you say USB and mount together, you imply block devices and file systems. HID devices are neither. You can’t “mount” a keyboard in Linux.

As to your actual question, you might be able to do an mknod(8) call and map the dev node in that way. It depends in part on whether the SYS_MKNOD capability is granted at container runtime and whether you can reliably work out what dev node ID the UPS appears under on the host side.

Tricky, but not impossible in principle.

ech1965 · Mon Sep 26, 2022 7:34 pm

When you say USB and mount together, you imply block devices and file systems. HID devices are neither. You can’t “mount” a keyboard in Linux.

As to your actual question, you might be able to do an mknod(8) call and map the dev node in that way. It depends in part on whether the SYS_MKNOD capability is granted at container runtime and whether you can reliably work out what dev node ID the UPS appears under on the host side.

Tricky, but not impossible in principle.

Routeros is a picky community.

But my bad, i was wrongly remembering docker command uses the the same parameter for mounting folders and "devices".
But it's not: I'm simply looking for the equivalent of the --device command line parameter of the docker command in linux.

Will try to mount

/dev in a container and see where it goes.

Tue Sep 27, 2022 12:30 am

Routeros is a picky community.

Computers are picky things. Those of us who've learned to become facile with them learn not to use the wrong terminology and to jump on instances of it where we see it elsewhere as a sign of either sloppy or outright incorrect thinking. When you're asking for help, you do not want to be giving off the impression of either, else you're likely to get the right answer to the wrong question.

i was wrongly remembering docker command uses the the same parameter for mounting folders and "devices".

There does exist "docker create --mount", but you normally do not use it with /dev, because the container creation process normally does that for you. It's hidden down in the guts of Docker and tricky to get to, but you can see an example of it exposed here, down at the end. Notice the several /dev mounts.

However, although /dev is automatically mounted, you will notice that, if you fire up a random container (e.g. alpine:latest) you won't get things like /dev/sda. This is on purpose, else root inside the container could scribble on your hard disk.

I'm not being pointlessly pedantic here. I'm trying to show why I believe you're off on the wrong track.

I'm simply looking for the equivalent of the --device command line parameter of the docker command in linux.

RouterOS's implementation of containers is closer to the way bare-bones OCI bundle runners like runc and crun work than to full-fat Docker Engine. You'll find yourself missing a lot of what you take to be common affordances.

I doubt the problem is as simple as all this anyway. I did some digging into NUT's USB HID driver, and it looks like it uses libusb to go digging through the device tree. I believe it needs "/sys/bus/usb/devices" to be populated for that to work, which it certainly won't be in a stock container.

If true, even my suggestion to hard-code a /dev node directly into the container won't work.

I also came across this in my investigation, which I found discouraging.

ech1965 · Tue Sep 27, 2022 1:26 pm

Routeros is a picky community.

Computers are picky things. Those of us who've learned to become facile with them learn not to use the wrong terminology and to jump on instances of it where we see it elsewhere as a sign of either sloppy or outright incorrect thinking. When you're asking for help, you do not want to be giving off the impression of either, else you're likely to get the right answer to the wrong question.

i was wrongly remembering docker command uses the the same parameter for mounting folders and "devices".

There does exist "docker create --mount", but you normally do not use it with /dev, because the container creation process normally does that for you. It's hidden down in the guts of Docker and tricky to get to, but you can see an example of it exposed here, down at the end. Notice the several /dev mounts.

However, although /dev is automatically mounted, you will notice that, if you fire up a random container (e.g. alpine:latest) you won't get things like /dev/sda. This is on purpose, else root inside the container could scribble on your hard disk.

I'm not being pointlessly pedantic here. I'm trying to show why I believe you're off on the wrong track.

I'm simply looking for the equivalent of the --device command line parameter of the docker command in linux.

RouterOS's implementation of containers is closer to the way bare-bones OCI bundle runners like runc and crun work than to full-fat Docker Engine. You'll find yourself missing a lot of what you take to be common affordances.

I doubt the problem is as simple as all this anyway. I did some digging into NUT's USB HID driver, and it looks like it uses libusb to go digging through the device tree. I believe it needs "/sys/bus/usb/devices" to be populated for that to work, which it certainly won't be in a stock container.

If true, even my suggestion to hard-code a /dev node directly into the container won't work.

I also came across this in my investigation, which I found discouraging.

Thanks you very much for your detailed answer,
Sorry for having mismatched devices and mounts
Implying I didn't read the docs "hurted" because I did and couldn't find anytying avout hid devices.

I'll keep trying playing with the container features of routeros
I might keep pursuing my plan of attaching a raspi on my ups and have a "native" nut implementation...

Znevna · Tue Sep 27, 2022 1:56 pm

The same way tun/tap access was enabled in containers under RouterOS maybe access to usb devices can be enabled too, with a proper feature request.

Tue Sep 27, 2022 6:51 pm

The same way tun/tap access was enabled in containers under RouterOS maybe access to usb devices can be enabled too, with a proper feature request.

this is in our to do list.

Znevna · Tue Sep 27, 2022 7:05 pm

Sweet, thank you!
PS: I've managed to trash the container config somehow, but I can't reproduce it, even though I've tried to redo all the steps before it "broke".
I had AdGuardHome running on it's own root-dir and veth1, I've added a new veth for testing this: viewtopic.php?p=958966#p958961
I've prepared the mount dir, added the container in a new root-dir obviously and with veth2.
Obviously it failed because it didn't have support for arm64 so I went on to try it on hAP ac3.
When I went back to my RB5009, container print was listing the last container added but twice with different status (I repeat, same "name"), one with error, one with some other status.
AdGuardHome was not showing in container list, but it was still running (checked). I've tried deleting them but here something broke:
Terminal history was only going up two commands up to /container and it crashed there, no other input was possible, only in a new terminal window only to crash again at the same history line (same thing via SSH).
A reboot fixed it, I readded the AdGuardHome container to be sure of no problems.
Weird.

Pl07R3K · Sun Nov 27, 2022 6:06 pm

I'm going to deploy flungo/avahi in a container and I'm wondering how to connect it with VLANs.
Can anyone help me?

Pl07R3K · Sun Nov 27, 2022 6:20 pm

Avahi (mdns Reflector)

Hap AC^2 (256M)

It took a while to get working, but I was able to build an Avahi container. Using the cross build instructions at

https://hub.docker.com/r/taoyou/iperf3-alpine

And the avahi container at

https://github.com/flungo-docker/avahi

It came out a bit over 8M in size, so quite big.

It uses a couple of vlans (Vlan setup currently hard coded in the tar file) on it's veth to connect to the local networks.

It is configured as a mdns reflector.

While working out how to make it work, I used netcat from openwrt to provide a couple of cli sessions. The cli is very basic but works, (You need 2 for when you break one) Probably should learn how to setup a dropbear.

I used 7zip on my desktop to open the tar file, and edit config files, and insert the executables (netcat binary and runcat2.sh script)

It seems to work ok.
Not sure I really want it now though

Hi @rplan,
I am also going to deploy flungo/avahi in container therefore I am looking for information on how to connect it with VLANs. One bridge for every VLAN?
Can you please guide me?

Sun Nov 27, 2022 9:42 pm

Like this.

Pl07R3K · Mon Nov 28, 2022 12:30 am

Like this.

Yes. Thank you very much!

shahidi · Wed Dec 07, 2022 11:25 am

Dear Friends , I have tried many dockers from hub.dockers.com most of them was related to VPN and Proxies and packet ofuscators, and I faced mostly with two problems , first problem , there is no "--cap-add NET_ADMIN" so you can not add dynamic tun interface inside the docker or you can not add special iptable rules inside the docker , and second huge problem when I start second container and set dst nat rules on same range docker bridge and interface or even on second docker bridge and interface , traffic for the first running container becomes lost , i have checked everything and many times , I did all settings with blank config and test environments but the problem exist , please guide me if there are solutions for those two problems. regards

elbob2002 · Sat Dec 10, 2022 2:55 pm

You need to add the variables for your container before you pull it down:

/container/envs/add key=cap-add name=MYCONTAINER value=NET_ADMIN
/container/envs/add key=device name=MYCONTAINER value=/dev/net/tun

Where MYCONTAINER is the name of your container.

Pl07R3K · Sat Dec 10, 2022 10:27 pm

Dear Friends , I have tried many dockers from hub.dockers.com most of them was related to VPN and Proxies and packet ofuscators, and I faced mostly with two problems , first problem , there is no "--cap-add NET_ADMIN" so you can not add dynamic tun interface inside the docker or you can not add special iptable rules inside the docker , and second huge problem when I start second container and set dst nat rules on same range docker bridge and interface or even on second docker bridge and interface , traffic for the first running container becomes lost , i have checked everything and many times , I did all settings with blank config and test environments but the problem exist , please guide me if there are solutions for those two problems. regards

Indeed, advanced container network configuration (e.g. with VLAN support) is missing or not described in the documentation.
The docker engine provides a virtual network between host and containers and automatically configures it in containers, in ROS you have to do it manually (via veth with additional bridge and masquerade) it resembles the bridge network which is the primary/default in docker.
BTW I was able to add veth to VLANs on the bridge and subinterfaces to eth0 in the container with alpine linux and get addresses on them from DHCP. For this to work at all it was necessary to assign IP address and gateway in veth.

Mon Dec 12, 2022 10:23 am

You need to add the variables for your container before you pull it down:
Code: Select all
/container/envs/add key=cap-add name=MYCONTAINER value=NET_ADMIN
/container/envs/add key=device name=MYCONTAINER value=/dev/net/tun
Where MYCONTAINER is the name of your container.

/dev/net/tun access was added in 7.5beta4 and is also included in 7.6. Your provided commands will do nothing.

MayestroPW · Mon Dec 12, 2022 3:58 pm

It would be great to have option to nest docker in a docker, or at least have an option to mount /var/run/docker.sock

Mon Dec 12, 2022 6:35 pm

It would be great to have option to nest docker in a docker, or at least have an option to mount /var/run/docker.sock

I don’t believe RouterOS is running full-fat Docker Engine. By all the signs, it’s a barebones OCI runtime, closer to crun or systemd-nspawn.

There is no API socket to be had.