Parsing problems with 'print terse' and 'print as-value'

rarylson · Thu Feb 15, 2018 8:22 pm

Hi,

As I know, there are four good ways of processing complex print outputs from Mikrotik: using the Mikrotik API, or using the 'print detailed', 'print terse' and 'print as-value' commands.

I use Fabric (http://www.fabfile.org/) to manage Mikrotik hosts. It uses SSH as backend: so, I rely on 'print' commands over SSH to get more complex info from my Mikrotik hosts. /Iso I want a good way of process the outputs of print commands.

Some points:

1) Print parsing library???

I checked Pypy (Python package repository) for a lib that parses the Mikrotik 'print' commands. I only find libraries (some of them seems to be very good) that parse command outputs via the Mikrotik API. I did not find any library that can parse print commands.

Do you know any library for this? Or any implementation in other programing language?

2) 'print' command parse problems

As I did not find a library for finding print commands via SSH, I'm implementing one.

However, I am having some problems with the print commands.

2.1) 'print terse' do not escape spaces

This observation is also found in a post from @alcohol: viewtopic.php?f=9&t=122122

I also discovered that 'print detail' can not escape spaces in some cases.

2.2) 'print terse' do not escape the equal '=' char

The problem 2.1 can be solved if we consider that spaces can have different meanings in a 'print terse' output: key-value separators or spaces inside a value. Also, we need also to consider that empty values have special representation in 'print terse' ('""').

For example:

key1=value1 key2=my value 2 key3=""

Means (in a Python notation):

[('key1', 'value1'), ('key2', 'my value 2'), ('key3', '')]

Or (another notation):

key1="value1" key2="my value 2" key3=""

But I still have problems with the '=' char. For example, if I create the interface:

/interface vlan add name="teste mtu=1400" vlan-id=10 interface=ether1

The print terse command will return:

[admin@router] > /interface print terse where type=vlan
 0  R  name=teste mtu=1400 type=vlan mtu=1500 actual-mtu=1500 l2mtu=1522 mac-address=00:0C:42:74:99:F1 fast-path=no last-link-up-time=feb/15/2018 06:20:59 link-downs=0

So my parser cannot parse mtu (`mtu=1400` or `mtu=1500`???).

2.3) Problems with 'print as-value' and 'print detail'

A similar problem occurs in 'print as-value' when using the ';' char:

[admin@router] > /interface vlan add name="teste mtu=1400" vlan-id=10 interface=ether1
[admin@router] > :put [/interface print as-value where type=vlan]
.id=*15;actual-mtu=1500;comment=;l2mtu=1522;mac-address=00:0C:42:74:99:F1;name=teste;mtu=1400;type=vlan

Again, problem while parsing mtu: the parser thinks 'mtu=1400' while it's 1500.

A similar problem can occur when using 'print detail':

[admin@router] > /interface vlan add name="teste\" mtu=\"1400" vlan-id=10 interface=ether1
[admin@router] > /interface print detail where type=vlan
Flags: D - dynamic, X - disabled, R - running, S - slave
 0  R  name="teste" mtu="1400" type="vlan" mtu=1500 actual-mtu=1500 l2mtu=1522 mac-address=00:0C:42:74:99:F1 fast-path=no
       last-link-up-time=feb/15/2018 06:20:59 link-downs=0

Again, I can't parse mtu since it seems to be 1400 and 1500 at the same time.

How should I parse 'print terse', 'print as-value' and 'print detail' commands in these cases? It is a bug in these commands?

Thu Feb 15, 2018 11:48 pm

If you use "print as-value", you then have an array of associative arrays, that you can then manually output into whatever string format you want. You do have to make a custom formatter, yes, but at least you can get an API-like level of reliability that way.

Here's one possible API-ish output format:

:local list [print as-value];
:foreach item in=$list do={
    :put "3|!re";
    :foreach k,v in=$item do={
        :put ((1 + [:len $k] + 1 + [:len $v]) . "|=" . $k . "=" . $v)
    }
    :put "0|";
};
:put "5|!done"
:put "0|"

On the client side, you'd decode this similarly to the API - read up until the first "|", treat the number in all bytes before as additional bytes to read up to, then interpret those next bytes the same way as you'd interpret equivalent API words. Then just ignore the newline (which each :put call makes), and parse the next word the same way.

Side note: You can also get an associative array (which you can then print in a similar fashion) for a particular item by calling "get" without a property name to get. Those property lists often include details not present in print. And to get all IDs in a menu (to then feed to "get"), you can loop over "find", e.g.

:local list [find];
:foreach id in=$list do={
    :local item [get $id];
    :put "3|!re";
    :foreach k,v in=$item do={
        :put ((1 + [:len $k] + 1 + [:len $v]) . "|=" . $k . "=" . $v)
    }
    :put "0|";
};
:put "5|!done"
:put "0|"

rarylson · Sun Feb 18, 2018 12:25 am

Just adding other outputs that will be wrongly parsed when using print terse:

/interface bridge add name="teste mtu=1500"

That produces:

 2  R name=teste mtu=1500 mtu=auto actual-mtu=1420 l2mtu=1522 arp=enabled arp-timeout=auto mac-address=00:0C:42:74:99:F3 protocol-mode=rstp fast-forward=yes priority=0x8000 auto-mac=yes admin-mac=00:00:00:00:00:00 max-message-age=20s forward-delay=15s transmit-hold-count=6 ageing-time=5m

And:

/interface bridge add name="\"\""

That produces:

 0  R name="" mtu=auto actual-mtu=1500 l2mtu=65535 arp=enabled arp-timeout=auto mac-address=00:00:00:00:00:00 protocol-mode=rstp fast-forward=yes priority=0x8000 auto-mac=yes admin-mac=00:00:00:00:00:00 max-message-age=20s forward-delay=15s transmit-hold-count=6 ageing-time=5m

The last one makes impossible to distinguish between an empty value and the `""` string.

rarylson · Sun Feb 18, 2018 12:38 am

I think the solution given by boen_robot (completely stop using 'print terse' and 'print default' and so use 'print as-value' to create a custom but parseable output). I also like the idea to print an API compatible output (since there are libraries that can parse API outputs for us).

Sun Feb 18, 2018 5:10 pm

To be clear, my format above is not a 1:1 the API, due to the way the length is encoded/decoded, so you can't just give that portion to an API client with no alterations to the API client itself.

After some experimentation, I see :put can output control characters as well with no escapes (which btw, also means you can f.e. color your output, among other things), which means you could theoretically output the length in the actual API way, which in turn means you can THEN pass that to an API client with no alterations to the API client... as long as said client can operate on any given stream, not just its own internally created one.

Actually making such a formatter isn't as trivial or as efficient as the above though, as RouterOS doesn't have a function to convert a character to its ASCII number. The only way to make one is to have an array where all bytes are written out, then :pick based on the number... At least there are bitwise operators, so while less efficient, lengths above 127 can also be encoded with little additional hassle. Even if you do make such a formatter, you'd really only be able to debug it with API debugging tools, whereas this one is also readable from a terminal window.

PortalNET · Sat Sep 25, 2021 4:18 pm

Hi

i know its an old thread but i have found somewhere on google i think o DaniC website a ssh mikrotik implementation class, perhaps this could be usefull for some ssh php project..

for php7 needs some changes on the code.. as split and other functions have been deprecated.

<?php
error_reporting(E_ALL);
ini_set('display_errors', '1');
// The mikrotik username must at least have read,write permissions enabled, and ip/services/ssh enabled also


class _ssh
 {
   private $connection = NULL;
   private $port = 22;    //change port according to your ssh custom port

   private $methods = array("kex" => "diffie-hellman-group1-sha1");

   private $callbacks = array("disconnect" => "disconnect");


   function connect($host,$username,$password)
    {
      $this->connection = ssh2_connect($host,$this->port,$this->methods,array($this,$this->callbacks));

      if(!$this->connection)
       {
         echo "FAIL: unable to establish connection<br />\n";
       }
      else
       {
         if(!ssh2_auth_password($this->connection,$username,$password))
          {
            echo "FAIL: unable to authenticate<br />\n";
          }
         else
          {
            // echo "OK: logged in...<br />\n";
          }
        }

      return $this->connection;
    }

   function disconnect($reason,$message,$language)
    {
      $this->connection = NULL;
      printf("Server disconnected with reason code [%d] and message: %s<br />\n",$reason,$message);
    }

   function fingerprint()
    {
      return ssh2_fingerprint($this->connection,SSH2_FINGERPRINT_MD5 | SSH2_FINGERPRINT_HEX);
    }



   function command($command)
    {
      if(!($stream = ssh2_exec($this->connection,$command,null,null,80,25)))
       {
         echo "FAIL: unable to execute command<br />\n";
       }
      else
       {
         stream_set_blocking($stream,true);
         $data = "";
         while($buf = fread($stream,4096)) $data .= $buf;
         fclose($stream);
       }

      return $data;
    }
 }


$ssh = new _ssh;

?>


<?php

// Mikrotik config field
$CONFIG = array();
// block/section separator
$header = "-- %s --";


function ParseBasic($string)
 {
   global $CONFIG,$header;
   
   
   $lines = explode("\n",$string);
   $regexp_header = "/^".sprintf($header,"(.*)")."$/i";
   $block = "";

   foreach ($lines as $line)
    {
      $line = trim($line);
      if($line == "") continue;  // we ignore the empty lines

      if(preg_match($regexp_header,$line,$matches))
       {
         // we have a section header
         $block = $matches[1];
       }
      else
        switch ($block)
         {
           case "IDENT":
           case "CLOCK":
           case "NTP":
           case "LICENSE":
           case "ROUTERBOARD":
           case "RESOURCES":
           case "DNS":         
             //ParseVars(strtolower($block),$line);
           break;

           case "IP ADDRESS": ;
           case "HOTSPOT USERS":
           case "HOTSPOT PROFILES":
           case "WALLED GARDEN": ;
           case "DNS":
             //ParseDetailOutput(strtolower($block),$line);
           break;

           case "LOG":
             //ParseLog(strtolower($block),$line);
           break;
         }
    }
 }


$getdata = array("IDENT" => "system identity print",
                 "CLOCK" => "system clock print",
                 "NTP" => "system ntp client print",
                 "LICENSE" => "system license print",
                 "ROUTERBOARD" => "system routerboard print",
                 "RESOURCES" => "system resource print",
                 "IP ADDRESS" => "ip address print detail without-paging",
                 "ROUTES" => "ip route print detail without-paging",
                 "HOTSPOT USERS" => "ip hotspot user print detail without-paging",
                 "HOTSPOT PROFILES" => "ip hotspot user profile print detail without-paging",                 
                 "WALLED GARDEN" => "ip hotspot walled-garden print without-paging",
                 "DNS" => "ip dns print",
                 "LOG" => "log print without-paging");


$command = "";

foreach ($getdata as $key => $value)
 {
   $command .= ':put "'.sprintf($header,$key).'";';
   $command .= sprintf("%s;",$value);
 }


$ssh->connect("mikrotik-ip-address","username","password");

$result = $ssh->command($command);
ParseBasic($result);


echo "<pre>";
var_dump($CONFIG);
echo "</pre>";

?>

Parsing problems with 'print terse' and 'print as-value'

Parsing problems with 'print terse' and 'print as-value'

Re: Parsing problems with 'print terse' and 'print as-value'

Re: Parsing problems with 'print terse' and 'print as-value'

Re: Parsing problems with 'print terse' and 'print as-value'

Re: Parsing problems with 'print terse' and 'print as-value'

Re: Parsing problems with 'print terse' and 'print as-value'