Container crashes randomly

Niki28 · Fri Oct 21, 2022 9:10 am

Hi forum,

I currently have a problem with crashing Containers. My original goal was to run a Websockify container on my RB5009, but when I try to start it, it crashes after after a few seconds.
I have come across this problem multiple times when testing Containers like Alpine, Debian and Websockify. I can always start the Container and sometimes even connect to its shell but if i try running anything remotly demanding, the Container quits with the message

done

(sometimes even when just idling).
I also tried Containers on my hAP ac2, and there, the containers where a bit more stable but also crashed after I ran htop for a few seconds or pinged my host.
The problem doesnt seem to have always the same pattern, because the time a container is running is always diffrent and doesnt seem to be RAM related either (hAP ac2 128mb ram, RB5009 1GB RAM, same problem).
I did set the "RAM High" in Container config to 500, 900, 100 etc. but it doesnt change anything, execpt maybe running the container a few seconds longer(or its just the placebo effect).

A reboot to the Router does seem to fix the problem for a few Minutes, but then the container crashes randomly again.
Im currently on RouterOS 7.6 stable.

[admin@MikroTik] > container start 0

[admin@MikroTik] > container shell 0
open: No such file or directory
done
[admin@MikroTik] >

[admin@MikroTik] > container shell 1
/ # help
Built-in commands:
------------------
        . : [ [[ alias bg break cd chdir command continue echo eval exec
        exit export false fg getopts hash help history jobs kill let
        local printf pwd read readonly return set shift source test times
        trap true type ulimit umask unalias unset wait
/ # echo hello
hello
/ # ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
^C
--- 8.8.8.8 ping statistics ---
3 packets transmitted, 0 packets received, 100% packet loss
done
[admin@MikroTik] >

Container configs:

[admin@MikroTik] > container print
 0 name="83158f44-edaf-45e5-a46f-df86a1039830" tag="" os="linux" arch="arm64" interface=veth1 envlist="linux_envs" root-dir=docker/linux_root 
   mounts=linux_data dns="" hostname="Websockify" workdir="/opt/websockify" logging=yes status=stopped 

 1 name="6ce28b36-ee5e-4ed1-971d-22250d41d42a" tag="" os="linux" arch="arm64" interface=veth1 envlist="linux_envs" root-dir=disk1/docker/linux_root 
   mounts=linux_data dns="" hostname="alpine" status=stopped 

 2 name="f8aedd73-bb14-4a92-97df-7bf2bb2449e0" tag="" os="linux" arch="arm64" interface=veth1 envlist="linux_envs" root-dir=docker/linux_root 
   mounts=linux_data dns="" hostname="alpine" status=stopped 
[admin@MikroTik] >

Can someone maybe loacte the source of this problem(because there is nothing in the log either), or maybe suggest a fix? Or is this a bug in RouterOS, because Containers are still pretty new on ROS.

Sat Oct 22, 2022 9:40 pm

I currently have a problem with crashing Containers.

Why do you characterize a result of "done" as "crashing?" I'd take it literally: it's done with what you asked it to do.

Indeed, I can't see that you've told these containers to do anything, so it comes back and says, "I've done all of the NOTHING you told me to do, boss. What's next?"

That's partly speculation because…

My original goal was to run a Websockify container

…you haven't said which one. There are 58 containers registered on Docker Hub that match a search for "websockify", two of which are precompiled for ARM64. It could be any of them or none of them, as when you build a container from source.

I'm not willing to install and diagnose all of them for you and list all of my findings. Tell us which one you're trying to use, and give any necessary instructions for building it.

I have come across this problem multiple times when testing Containers like Alpine, Debian

Prime guess: you're assuming containers are "lightweight VMs" and are expecting them to boot up and offer SSH or some other remote login mechanism. While some containers do exist like this, it isn't inherent in the way containers operate.

Instead, most containers are built around a single process that — if you don't give them a service request to start with — finish without having done anything, having been told to do nothing.

This is the case with the base Alpine container, at least. Unless you give it something like 'cmd="sleep 3600"' it will start, have no command to run, and stop.

A more useful command to start a bare Alpine container with, if you want it to operate like a VM, is "/bin/login -f root". The only reason I have that command at hand is because I have a test container for use in building other containers that are themselves based on Alpine. I use it when I need to test something and don't want to rebuild the container-under-construction merely to test an idea. That container lives on my desktop PC; it isn't something I run on MikroTik routers.

Niki28 · Mon Oct 24, 2022 12:14 pm

Hello, sorry for the late answer and thanks for the detailed reply. Indeed Im still kind of new to containers and had indeed assumend that they are some kind of lightweight VM, thanks for the clarification.
But this still does not explain the "randomness" of the Container quiting. I just dont see any pattern here, so I assumend it to be a RAM issue, but this isnt the case either because I tested it on both the hAP ac2 and the RB5009.
Regarding Websockify:

I just downloaded the latest Websockify release on github https://github.com/novnc/websockify and build the Docker container using

docker buildx build --compress --platform=linux/arm64/v8 websockify-docker --load

then I saved it to a tar archive, moved it to the Router and added a Container with the file there. Everything works fine, except it quits after some time.

Mon Oct 24, 2022 8:40 pm

But this still does not explain the "randomness" of the Container quiting.

I expect you're simply seeing the normal variation in starting and stopping delays. Keep in mind, this isn't a full-power desktop computer, there's a virtualization and containerization layer to deal with atop that, and the host router has other things to do besides. You can't expect a container to start and stop in the same amount of time atop such a platform every single time. These delays will be random even when you've got it working properly.

the latest Websockify release on github https://github.com/novnc/websockify

Okay, now we can get concrete. Take a look at the tail end of the Dockerfile, where you see it passing the "--help" argument to the entrypoint script. If you check your container logs, the canned help output is doubtless what you'll find there.

This is what I meant when I told you that you have to tell the container what you want it to do: the "--help" default is meant to clue you into the fact that you haven't given it explicit instructions. The same arguments the project documents for starting it standalone have to be passed with the "cmd" args when creating the container from the image on the RouterOS side, overriding this "--help" default.

This is one of the two common patterns with containers: instead of the full command, as in my Alpine example, this container wants the "cmd" value to be the arguments to the entrypoint script. As you can see, everything you pass as "cmd" are passed as arguments to websockify. ("$@")

From here, you're on your own. I don't use websockify, so I can't tell you what arguments to pass. Even if I did have a working use case here, you haven't told us what your use case is, so I can't even guess at the proper args for you.

Niki28 · Wed Oct 26, 2022 2:55 pm

To clarify my project:

I want to allow a Website to talk with a device, that offers socket connection only and my plan was to connect them through a Websockify Container on the Mikotik Router. In our situation the socket device "EtherCAN" provides signals, that I want to visualize in a web browser with updates every 100ms.

Here is the schematic on how it should work:

Router Setup Script:

add dont-require-permissions=no name=add-Websockify owner=admin policy=\
    ftp,reboot,read,write,policy,test,password,sniff,sensitive,romon source="\
    \r\
    \n/container/add file=---tarFile--- interface=veth1 root-dir=disk2/docker/websockify_root mounts=websockify_data envlist=websockify_envs logging=yes hostname=Websockify;"

JS:

<script>
        const ws = new WebSocket("ws:/172.19.50.238:8082");
        
        ws.onopen = () => {
          ws.send('ping')
        }

        ws.onmessage = (e) => {
            console.log(e)
        }
        //setTimeout() => {  console.log("waiting"); }, 100);
        //ws.send("Hello ich sende eine Nachricht!");


    </script>

The Website starts the websocket connection to the docker container. The docker accepts the connection and opens the connection to the socket connection. The Data can flow directly from the gateway to the EtherCAN(Socket device).

I know there are other ways of achieving this, like building an native android app or using a linux gateway, but for us, it would be the most practical to use the container solution on the Router, as we have the setup repeated in many diffrent applications.

In a linux environement websockify is started like this

websockify [options] [source_addr:]source_port [target_addr:target_port]

eg.:

websockify 172.19.50.238:8082  172.19.48.1:8083

In a linux docker environment Websockify works fine.
Now I want to do that, in a Container on the mikrotik router. The challange is to get this working.

Thanks very much for you help regarding this!

Wed Oct 26, 2022 10:02 pm

I've already told you at least twice that you need to pass those same options to the container on the RouterOS side in the "cmd" parameter.

/container/add cmd="172.19.50.238:8082 172.19.48.1:8083" ...

anav · Wed Oct 26, 2022 11:38 pm

Lets see if third time is a charm eh?

In the meantime just be thankful that the crashes are not predictable and reproducible at will. What would be the fun in that?

HiltonT · Mon Jan 16, 2023 6:40 am

I am having a similar, but *slightly* worse issue.

I have installed the adguard/adguardhome:latest image from https://registry-1.docker.io and it extracts and runs fine, I can set it to auto-start which also works, but after running for a few minutes, the entire hAP ac2 simply reboots. Not all that friendly... And I'm running RouterOS v7.7.

In case it was an issue with a particular Kingston USB key I used a completely different brand/model (this time a Jetflash) and had the exact same unfriendly result.

The only error message I can see that looks related is, wait for it...
memory system, error, critical kernel failure in previous boot

The system is running at <50% CPU load when it crashes, generally <40%.

Mon Jan 16, 2023 6:47 am

I don’t know what’s up with your container, but I do know enough to be confident that you’re thread-jacking. What you’ve got going on has nothing to do with the OP’s problems in this thread.

elico · Tue Jan 17, 2023 1:34 am

With what OS have you tried to build the Websockify container?
It seems that both Debian and Alpine has a package so you'd better stick to it.
I can try to create a container for that and resolve the issue and I wish there was funding for the time on such a project.

To clarify my project:

I want to allow a Website to talk with a device, that offers socket connection only and my plan was to connect them through a Websockify Container on the Mikotik Router. In our situation the socket device "EtherCAN" provides signals, that I want to visualize in a web browser with updates every 100ms.

Here is the schematic on how it should work:

Router Setup Script:
Code: Select all
add dont-require-permissions=no name=add-Websockify owner=admin policy=\
    ftp,reboot,read,write,policy,test,password,sniff,sensitive,romon source="\
    \r\
    \n/container/add file=---tarFile--- interface=veth1 root-dir=disk2/docker/websockify_root mounts=websockify_data envlist=websockify_envs logging=yes hostname=Websockify;"
JS:
Code: Select all
<script>
        const ws = new WebSocket("ws:/172.19.50.238:8082");
        
        ws.onopen = () => {
          ws.send('ping')
        }

        ws.onmessage = (e) => {
            console.log(e)
        }
        //setTimeout() => {  console.log("waiting"); }, 100);
        //ws.send("Hello ich sende eine Nachricht!");


    </script>
    
The Website starts the websocket connection to the docker container. The docker accepts the connection and opens the connection to the socket connection. The Data can flow directly from the gateway to the EtherCAN(Socket device).

I know there are other ways of achieving this, like building an native android app or using a linux gateway, but for us, it would be the most practical to use the container solution on the Router, as we have the setup repeated in many diffrent applications.

In a linux environement websockify is started like this
Code: Select all
websockify [options] [source_addr:]source_port [target_addr:target_port]
eg.:
Code: Select all
websockify 172.19.50.238:8082  172.19.48.1:8083
In a linux docker environment Websockify works fine.
Now I want to do that, in a Container on the mikrotik router. The challange is to get this working.

Thanks very much for you help regarding this!

ljb · Sun Jul 09, 2023 3:51 pm

@ Niki28

Hi there...

I have picked up that many lightweight alpine images do not have any active daemons running which is why they are almost instantly halted. Have you tried a Command ''tail -f /dev/null" to keep them running? This has worked for me.

Greetings

LJB

Container crashes randomly

Container crashes randomly

Re: Container crashes randomly

Re: Container crashes randomly

Re: Container crashes randomly

Re: Container crashes randomly

Re: Container crashes randomly

Re: Container crashes randomly

Re: Container crashes randomly

Re: Container crashes randomly

Re: Container crashes randomly

Re: Container crashes randomly

Who is online