MikroTik Devices Controller

Mon May 30, 2022 1:53 pm

Hello,

MikroTik is planning to develop and build a controller app for MikroTik Devices. Currently we are researching possibilities and options, what should be there and how it could be done and implemented. At the moment we do not want to stick to a specific implementation or standard, but build our own that will help to manage, develop and deploy different scale networks running MikroTik devices.
Any suggestions about features and options are very welcome.

elbob2002 · Mon May 30, 2022 1:57 pm

Centralised updates and configuration management!

parham · Mon May 30, 2022 2:10 pm

Hello,

MikroTik is planning to develop and build a controller app for MikroTik Devices. Currently we are researching possibilities and options, what should be there and how it could be done and implemented. At the moment we do not want to stick to a specific implementation or standard, but build our own that will help to manage, develop and deploy different scale networks running MikroTik devices.
Any suggestions about features and options are very welcome.

This is fantastic Idea, whoever what you guys think about something like what Unifi or Meraki do, a nice controller that can be hosted, and adopt all Mikrotik device with potentially dude integrated to it for nice network diagram and more... would be killing feature for us deploy and managed Mikrotik

blingblouw · Mon May 30, 2022 2:16 pm

This is extremely exciting!

Definitely configuration templates but please allow webhooks early on. For example, you may want to create a template with some variable information that can be retrieved from some sort of restful api, fully automated

parham · Mon May 30, 2022 2:19 pm

1- firmware update.
2- configuration backup and compare.
3- Wireguard VPN generator for client side (a file that can be import to fireguard software).
4- site to site ipse vpn
5- network and wifi settings to any mikrotik in the same site
6- firewall rules and NAT
7- IPS IDS
8- sd-wan
9- geo IP location for block and allow list.
10- WAN performance check: speeds, ping, jitter.

fragtion · Mon May 30, 2022 2:21 pm

Exciting news !
- web based, so the controller can be accessed from nearly any device or platform and eventually phase out winbox. ideally the controller should not be based on java but static binaries deployment or open source ?
- better graphing and analytics, time to move away from mrtg. rrdtool is good, what about grafana ?
- integrated "tailscale-like" controller to easily set up wireguard links between managed endpoints and automatically handle endpoint ports, NAT hole punching etc. "one click wireguard vpn" could be a great marketing tool
- devices overview with status page, device model image, and a satellite map overview to plot wireless links and do line-of-site calculations similar to ubnt's UISP
I'll update my list if I can think of anything else :D

winap · Mon May 30, 2022 2:30 pm

Centralised updates and configuration management!

Yes definitely! Just make update and upgrade routerboard in one click, just mark clients on AP.. In which hour and if will include AP, wait to download clinets FW and than upgrade AP on last.
Most users still don't know, how to upgrade FW..(not only install and reboot). But this function must be enabled, some users don't want newer firmware, because some functions are better on spec. FW.
Or example: ISP add new AP because other is overloaded, so some clients migrate to new AP and it want new IP adress. It will be better to mark some new user and give them new IPs so old will be rewritten.
Thank you!

BrateloSlava · Mon May 30, 2022 2:30 pm

Dude has the ability to specify a "parent" for the current device. With centralized management, it is necessary to check the entire chain of "parents" before rebooting during an update. So, that there is no situation, when the "parent" has already downloaded the update and started rebooting, and the "child" has not yet had time to download the update.

dakobg · Mon May 30, 2022 2:33 pm

Single app to control all mk devices ?

Winbox with menu -> list devices -> select mk device -> done :)
Winbox place with all devices overview
Winbox templates for auto-configuring ?!

and many many

Other option -> dedicated app (new dude build from scratch ?! ) with UI from 2022 with nice monitoring and mgment functionality ;)

pe1chl · Mon May 30, 2022 2:40 pm

Hello,

MikroTik is planning to develop and build a controller app for MikroTik Devices.

How do you define that? Something like TikApp? I suppose not, that is already there.
Or should we read "for MikroTik Devices" as "a centralized solution for management of a whole lot of devices" as some others above appear to infer?
Would that then be only "via an app" or would it be a solution that can also be used from a generic computer?

woland · Mon May 30, 2022 2:45 pm

Hi!
Great idea!
Features which I missed elswhere:
-revision control
-having everything in cleartext as well
-grouping (labeling) the devices
-starting scripts/actions on many devices (on a group) at once
-sending/receiving files to from devices
-central log collector
-SW upgrade/rollback
-device status in detail

Improve The Dude instead?
:)

W

dvreshta · Mon May 30, 2022 2:56 pm

Great idea.
Initially it will be great to focus on small scale configuration i.e. SME that want to run a ccr as gateway, use the firewall to protect their network and devices, extend the network with some crs or css switches and manage capsman.

That's will be more than enough for over 70% of installations.

elel · Mon May 30, 2022 3:59 pm

Cloud based even with subscription for any ROS device. Generate temporary links to access the Device from the cloud. Make a pool of credits for a given account and let the user of the account use the credits based on time of the generated links for its devices. Implement a strong protocol for communication of devices with the cloud. Make the account with only two factor authentication.

From there, if one can connect to a core router then he can use a VPN of choice to control the whole network with tools such as romon.

marcelbohmer · Mon May 30, 2022 4:07 pm

Have an agent package for all hardware types, like prtg agents or zabix agents?

Panbambaryla · Mon May 30, 2022 4:15 pm

Let's call it by it's name: Ubiquiti solution (UISP) is a good example how it should work together with the technology they use (docker).

volga629 · Mon May 30, 2022 4:40 pm

Unifi controller not good, too many pictures no functionality.

I would suggest that Mikrotik will do it based decentralized model, similar to cisco ACI.

This way controllers never impact infrastructure.

Also do layered functionality deployment. Meaning set functionality based on categories and priority.
Example:
management access highest priority.

Also important to have configuration files enforcing on controller side. Like diff ( version control ), backup, security audit.

Mon May 30, 2022 5:09 pm

I have seen several mentions of config files, config compare ...
Do you suggest for the controller to operate as a configuration export uploader?

parham · Mon May 30, 2022 5:20 pm

I have seen several mentions of config files, config compare ...
Do you suggest for the controller to operate as a configuration export uploader?

yes and No, would be good if controller can do the backup of the correct config, can do compare between the versions of the config and it would be great to push the config too.

but what is the Mikrotik plan for the controller: are you going to develop something like Meraki or Unifi, if yes, then that's a fantastic news, but if that a simple mobile app or similar then please even don't try.

RouterOS is fantastic and if you can develop a controller and integrated with ROS7 that would be a killer features, and if the controller can be a appliance or hosted then you can add IDS, IPS and sd-wan and to integrate with ROS.

parham · Mon May 30, 2022 5:21 pm

And also please make this topic HOT as this is the one of the best feature you all come up with.

Mon May 30, 2022 5:24 pm

The initial goal is to develop a protocol to apply and monitor config, hence the question about needed features that this protocol should be able to do.

parham · Mon May 30, 2022 5:29 pm

The initial goal is to develop a protocol to apply and monitor config, hence the question about needed features that this protocol should be able to do.

I see, are you going to developing something like Meraki or Unifi as a controller? is Mikrotik planing to have some king of NGFW feature in future produce?

pe1chl · Mon May 30, 2022 5:47 pm

The initial goal is to develop a protocol to apply and monitor config, hence the question about needed features that this protocol should be able to do.

- connect to some cloud server and establish trust
- send existing config to cloud server
- receive full config or deltas from cloud server
- upgrade RouterOS to version specified by cloud server
- user interface to cloud server allows flexible config templates and groups to distribute standard config to a group of devices, including macros for variable parts
- files stored by cloud server are in human-readable form (like export)
- cloud server software is available for installation on own hardware

Larsa · Mon May 30, 2022 6:04 pm

@sergejs wrote: MikroTik is planning to develop and build a controller app for MikroTik Devices. Currently we are researching possibilities and options, what should be there and how it could be done and implemented. At the moment we do not want to stick to a specific implementation or standard, but build our own that will help to manage, develop and deploy different scale networks running MikroTik devices. Any suggestions about features and options are very welcome.

@mrz wrote: The initial goal is to develop a protocol to apply and monitor config, hence the question about needed features that this protocol should be able to do.

Hello @sergejs and @mrz, this is my two cents to start with:

The phrase "controller app" as well "a protocol" is pretty vague. Please supply some more details of the needs and please feel free to give examples of similar solutions on the market.

Q1: Did you mean any of theses:

A) a "smartphone app"?
B) a true network monitoring and management solution (which it was hopefully intended)?

Q2: What is the overall objective and main purpose, as well as what is the target audience, eg a consumer, a professional network technician or something else?

Depending on the answer regarding Q1:

In case of A) a new smartphone app it might be useful for the consumer market but is not my area of expertise and is in general uninteresting for me.
In case of B) regarding a new monitoring and management solution, I strongly advice not to build anything from scratch but rather use some OSS or license a third party solution to build on.

--

However, as I explained above please begin to describe the intention by supplying some more details. Then I'm sure you will get plenty of more useful suggestions.

Thanks in advance!

EDIT:
This comment is not meant to be mean in any way but maybe some manager might be willing to develop their communication skills a bit and how to formulate a project description. I'm pretty sure it would make life easier for all involved parties.

woland · Mon May 30, 2022 6:19 pm

Hi,
as I read the responses, I see a lot of mentioning of "cloud". While I don´t think cloud is something inherently bad, one of the selling points of MT for me is that no features _require_ binding to any cloud (except the cloud backup obviously).

Since mrz wrote we are only talking about a protocol:
- make the protocol able to push/pull and exec config changes and scripts
- support versioning configs/track changes
- mass deployment support to groups (labeled devices)
- deployment of configs on some kind of condition (if mac adress equals, if SW version >7.x, if ethernet1 is down, etc)

W

felixka · Mon May 30, 2022 6:24 pm

If there will be a frontend for it other than Winbox or Webfig I think it should retain that Windows 95 look though.

Larsa · Mon May 30, 2022 6:37 pm

as I read the responses, I see a lot of mentioning of "cloud". While I don´t think cloud is something inherently bad, one of the selling points of MT for me is that no features _require_ binding to any cloud (except the cloud backup obviously).

A cloud only solution using US services might be sensitive in the EU depending on the CLOUD ACT, thus regardless of which solution the devs come up with you need to be able to install this on premise IMO.

woland · Mon May 30, 2022 6:51 pm

A cloud only solution using US services might be sensitive in the EU depending on the CLOUD ACT, thus regardless of which solution the devs come up with you need to be able to install this on premise IMO.

If using cloud services inside the EU, this is perfectly fine on the legal side.
This was not my point, but a cloud only solution has lots of other implications, like privacy, like the need for the Internet availability, like the total dependency on the cloud ressource maintainer and the cloud provider itself....

Larsa · Mon May 30, 2022 6:53 pm

This was not my point, but a cloud only solution has lots of other implications, like privacy, like the need for the Internet availability, like the total dependency on the cloud ressource maintainer and the cloud provider itself....

Concur!

Amm0 · Mon May 30, 2022 7:12 pm

I really don't think y'all are that far from this, a "modernized Dude" look a lot like a "Mikrotik Devices Controller" – it already has pretty good schema/protocol/storage. Basically I think it's architecture is pretty good – it's just the "client" that needs a new look. If it didn't look like a 1990s windows MFC app, and picked off some off the Dude feature requests, you'd have something sooner than starting from scratch...

The initial goal is to develop a protocol to apply and monitor config, hence the question about needed features that this protocol should be able to do.

I think issue today is the export/import isn't symmetrical, backup/restore is monolithic+binary, . What's missing is idempotent configuration file. And that's the first step to being able to monitor/apply a configuration in "controller software".

If the current configuration scheme supported a "update" operation that took an ".id" field that I think go a long way to solving this problem. While the .id is generated by the "add" today and isn't settable today, y'all can change that ;). e.g. there was "export idempotent" that didn't use "add"/"set" operations in config file, but something like:

/ip address update .id=*1 interface=bridge1 address=192.168.88.1/24

A smart "import" would accept the "update" and either add new or change existing, likely some options to either only update ones in the file, or use it replace everything. And, likely some syntax checker that only allow a "valid configuration" to be applied (e.g. not a "half applied" import because an error, that can happen today ;)).

Mon May 30, 2022 7:22 pm

very good and needed initiative

very important the ability to follow, track and deploy config changes massively and selectively by groups, templates, sites etc

Mon May 30, 2022 7:23 pm

It is intended as a true network management controller, of course probably in the future there could be an option to connect and manage the controller by a smartphone app or any other app or web GUI or whatever.
Think of it like a capsman, but not just for wireless.

Amm0 · Mon May 30, 2022 7:31 pm

Think of it like a capsman, but not just for wireless.

Do mean the "tree/hierarchical configuration style" of capsman, or forwarding of network traffic to a central router part (e.g. DTLS tunnels/local-forwarding=no). Or, both?

Mon May 30, 2022 7:38 pm

I am talking about the concept, there is a controller that should be able to push configuration, and there are devices that should be able to use that configuration.
We are talking about what the controller features that it should be able to do in terms of communication: controller <---------> controlled_device

We are not talking about any controller configuration styles or what management app should be used to connect to the controller or what the configuration app should look like, aka controller <--> user

rextended · Mon May 30, 2022 7:54 pm

For start, start just with:
Daily backup on text format, not binary on any point, and full configuration, included ssh keys, certificates, user-manager and dude database.
Some instruments to compare backup among various days for see the changes.
Some instrument for push configuration (like change NTP servers on all devices, or only on a group of devices, or only to selected devices.
Possibility to select/search devices by Group / Hardware / Branding/Platform, ROS version, BIOS/RouterBOOT version, installed packages.
Possibility to send .npk / .dpk / single file and select on what folder put it remotely based on internal memory type "root" or "root/flash".

ksteink · Mon May 30, 2022 8:03 pm

It was about time!!!! The whole market is moving to SDNx technologies and Mikrotik shouldn't be the exception !!!.

We should start with baby steps and basic features! such as:

- It should run on a multi-platform (Windows, Linux, MacOS, docker containers) and run on-premises (self hosted on x86/64 and ARM processors) and/or Cloud (AWS, Azure, etc) with web interface. I prefer a external controller than a package as part or RouterOS (or make a dedicated small / cost effective appliance).
- It should be lightweight! and simple to install and configure.
- it should have a companion mobile App (iOS and Android)
- KISS Method (Keep it Super Simple) with an easy to use interface (and eye catching as well)

What should be the minimum capabilities that I am looking to have in this controller:

- Zero Touch Provisioning (ZTP!). That means I ship an unconfigured Mikrotik device and the device should find the controller (on-premise or on cloud) so we can take control to push the configuration (or access via jump service (?) to the device so we can load the desired configuration via SSH / APIs or even Winbox).

- Monitoring & Analytics, to collect data on all the metrics of the controlled devices! like Bandwidth utilization, top talkers, applications, etc. (Netflow built-in?). This could be a replacement of the Dude also aside of other features.

- Automated configuration backup & restore.

- Move the CAPsMAN function of the Mikrotik device(s) and centrally managed by the controller

- Central Configuration management to push all the low level configurations massively to all or some devices. That means that the interface of the controller provides options to define "policies" as an abstraction layer that will be translated to low level configuration management commands that will be pushed to each of the target devices (via SSH, APIs, etc.). Not sure if you guys plans to use Ansible underneath or a new in-house protocol / automation solution here.

If Mikrotik does this right it with the features above (to start) will be a home run!!!. Fortinet has a similar model to deploy a central controller or alternatively keep local control of the devices like RouterOS does today without a dependence of a central controller.

Larsa · Mon May 30, 2022 8:38 pm

I am talking about the concept, there is a controller that should be able to push configuration, and there are devices that should be able to use that configuration. We are talking about what the controller features that it should be able to do in terms of communication: controller <-> controlled_device

Please make the controller expandable.

Besides the ordinary stuff that's normally included in regular configuration management this might be the beginning of a smart SDN management controller. In addition, if there is an API for user developed add-ons, there might be endless of possibilities to expand the controller with new functionality like for example network service templates for various SDN targets, backup/restore solutions, configuration templates for MT devices, etc.

And once again, please don't develop everything from scratch but use OSS or license a solution from a third party. IMHO, MT is like most network tech companies to small and does often lack the skill set to develop something on the application level thus if doing it completely from scratch it will most likely end in disaster ie DOA.

cwade · Mon May 30, 2022 8:44 pm

My number one suggestion and highest priority is to build in strong security from the start, not as an afterthought. MikroTik could show the rest of the network equipment industry how to establish best practices for securely maintaining network devices, and that should be the goal!

Some suggested approaches:

Incorporate a robust management facility for establishing and maintaining administrator and user accounts. Ideally, this should also support "machine" accounts that could be used for automated status queries and management of devices. The controller itself should use a "machine" account for its purposes, and this account must be customizable by the customers. Integration with enterprise systems like Microsoft's Active Directory would likely be desirable for some customers.

Build in a Certificate Authority (CA) for issuing certs to network devices. The new controller should incorporate hardware protection for private keys (especially CA's own private keys), along with the ability to securely clone the CA's private keys to a backup controller using multi-party controls. Network devices should be able to request renewal of certs from the CA using automated methods. There should also be automated tools for installing certs in network devices. Certificate revocation must be supported using a dynamic protocol like OCSP with ability to push out revocations immediately (e.g., via CRL update). An optional approach could be to support integration with a third-party Certificate Issuing/Management system, but these days the tools to implement the subset of services required for network devices are readily available from multiple Open Source projects, including OpenSSL itself.

Use CA in new controller to also issue client-side certs for network administrators. Client side certs would be used with mutual authentication to handle logins to Winbox and other device-specific services, including an option for providing SSH keys via certs. Automated client cert renewal should be supported, and it must be possible to revoke client certs with immediate pushing of revocation notices to devices along with dynamic cert checking. (Aside: WinBox might directly support requesting administrator certs from new controller's CA.)

CA should also issue user certs for Wireless access, PPP/VPN remote access, HotSpot services, etc. This implies that there should be specialized access to the controller from users to handle cert requests or update their account details, such as email addresses, phone numbers, workstation details, mobile device details, etc. In an enterprise environment, it might be possible to pull this sort of information from a central service.

Fully support the latest cryptographic algorithms and measures, including the widely accepted elliptic curve algorithms (e.g., ED25519). Provide policy controls to limit/restrict use of cryptographic suites in the network devices from a network-wide perspective.

Provide a complete implementation of a robust RADIUS service for legacy devices and services. For extra credit, support RADIUS integration with Microsoft AD NPS/RADIUS facilities.

Implement an SSH key management system that would support pushing administrators' SSH public keys to network devices and rolling keys as appropriate. Immediate removal or disabling of SSH public keys for administrator login is also necessary. One possibility would be to use SSH and SSH key management to handle securely pushing updates to devices, along with invoking of scripts and automated retrieval of device information, including device configurations.

Provide an encrypted storage system for maintaining sensitive information at rest, especially for device configurations and any other sensitive information.

Build in a software repository for redistributing RouterOS (and possibly other software/firmware packages) to network devices in a controlled manner without requiring that individual network devices have access to the Internet. This could be an adjunct to the requests from others on this Post for RouterOS bulk updates. Ideally, this system should support two or more storage partitions on devices that support this option to make it easier and safer to rollback an update. For devices that are not equipped (or configured) with multiple partitions, a rollback facility would still be a valuable capability.

Implement support for redundant device deployments, including for the new controllers. For example, support measures to independently update RouterOS in each member of a redundant device pair thereby allowing the other member to maintain services during the upgrade. This capability could also allow staging of firmware in redundant systems to confirm stability before completely updating all devices. Similar capabilities would also be necessary for redundant controllers. Resilience is an often-overlooked essential security requirement.

Support RANCID or an equivalent service for maintaining network device configurations in a source control system (e.g., Git). This could be an add-on package for users dealing with larger networks or complex support requirements. (Aside, my own experience using RANCID with a complex network involving devices from multiple vendors illustrated that this is an invaluable tool for not only tracking configuration changes, but also monitoring changes made by multiple administrators, which in turn provides further security controls with the added ability to recover from unapproved or ill-conceived changes.)

Support management of security credentials for SNMPv3, including the ability to update credentials periodically in a controlled and automated manner. Provide methods for pushing SNMPv3 credentials to network management systems (e.g., via secure upload of an exported dictionary of credentials).

Provide tools for automated responses to DDoS attacks using parameter-driven approaches for invoking mitigation measures.

Implement a comprehensive system logging facility. This could be optimized for MikroTik devices to leverage enhanced features. The system logging should support TCP logging, as well as optional support for logging via encrypted links (SSH, IPsec or other VPN). It should be feasible for customers to implement redundant syslog servers for resilience as well as protecting logs from being modified by attackers. The logging system should be capable of relaying log records to more advanced enterprise-oriented logging systems (e.g., Elastic Search).

Since DNS is one of the most essential services and also one of the most sensitive from a security perspective, centralized management of DNS services in network devices would be a valuable service. This could include the ability to maintain static DNS caches across some or all network devices to improve availability of essential DNS resolution during periods of degraded operations, such as network outages or partitioning.

Provide robust NTP services, ideally supporting authenticated access. The new controller would ideally provide an option for GPS time sync so that it could operate as a Tier 1 NTP server. This would also be an underlying security facility for supporting certificate management and use of time-based authentication services.

Yes, this is a lot. However, everything listed above is readily available and supported in the Open Source realm. What is important is to build these capabilities into the product plan, and build other controller features and capabilities on top of a secure base. Not everything needs to be in version 1, but everything (and more) needs to be in the product plan and resulting design. Security is just too important an issue these days to not be the primary objective for anything that purports to control network devices and maintain a network system.

dakobg · Mon May 30, 2022 8:45 pm

Please do not make the mistake as hUi company to have centralised control and "stupid" devices !!!

Larsa · Mon May 30, 2022 8:59 pm

What is a "hUi" company and what do you mean by "stupid" devices? All devices sold by MT include RoS or SwOS thus they are "smart" right?

Amm0 · Mon May 30, 2022 9:06 pm

My number one suggestion and highest priority is to build in strong security from the start, not as an afterthought. MikroTik could show the rest of the network equipment industry how to establish best practices for securely maintaining network devices, and that should be the goal!
[...]
Yes, this is a lot. Not everything needs to be in version 1, but everything (and more) needs to be in the product plan and resulting design. Security is just too important an issue these days to not be the primary objective for anything that purports to control network devices and maintain a network system.

So basically "Make Certificates Great Again", which go a long way as base for AAA in this concept. Now that includes dealing certificates better in RouterOS export/backup as a first step (see @rextended comments above re this topic)...

mada3k · Mon May 30, 2022 9:08 pm

- As a VM appliance
- Web based. Not phone apps and such non-sense.
- Manage software updates, pushing out updates in a controlled manner
- Configuration templates, backup and change-logs
- "Global rules" such as firewall, access-lists and so on
- REST-API for integrations with other systems
- Network maps, with and without Google/OpenStreetmap etc.
- Graphs of interfaces and so on
- Devices overview with status page, total inventory
- Configure alarms of things (BGP peers, OSPF adjacency)
- Configureable scripted "Actions", like "setup tunnel", "add subnet to interface", "reboot" and so on, tied to configuration templates.

dakobg · Mon May 30, 2022 9:29 pm

What is a "hUi" company and what do you mean by "stupid" devices? All devices sold by MT include RoS or SwOS thus they are "smart" right?

ui.com, "stupid" - device without config and havey really on central control manager

Larsa · Mon May 30, 2022 9:42 pm

Got it! Well, as long as MT continues to produce devices using RoS or SwOS I don't think we'll end up there.

carl0s · Mon May 30, 2022 9:52 pm

I still love the fact that CAPsMAN can run on an any existing equipment - no extra controller hardware required. It can run on a router, or you can run it on one of your two or three APs.
If possible, please keep that functionality, hopefully with the new wifi drivers.

benkreuter · Mon May 30, 2022 9:52 pm

I would urge you to follow standards instead of creating your own proprietary protocols/APIs/formats wherever possible, or to at least allow some minimal compatibility with a standard. If that is not possible or desirable, then my biggest request would be to document your protocols/APIs so that we can write our own tools/scripts/whatever for situations where your app does not meet some particular need.

npeca75 · Mon May 30, 2022 10:48 pm

my only hope is that MT will hire some new developer for this .. APP ? whatever, and leave alone existing DEVs to finish v7 this year so we could start 2023 with stable v7

rushlife · Mon May 30, 2022 11:00 pm

This is great idea and I am pleased by it.
Thx Mikrotik.

btw. viewtopic.php?p=907977#p907977
😁

Guscht · Tue May 31, 2022 12:01 am

I like the idea, but I use Ansible for such stuff already.

And a note to MT:
Why no solving unfinished things, like Queueing >4,3GBit is still not possible (beacuase thats a limit for 32Bit). Why is PIM-Routing still broken up to this day in your "stable" V7? Why is the ROSv7 documentation aka "help" in wide parts non existing...

Please solve your open topics, before you bind development capacities for a new app.

cfikes · Tue May 31, 2022 12:32 am

I know there will be a ton of wants from everyone, but if I feel with basic firewall, vpn, vlan configuration on switches, intervlan filter rules, and good wireless configuration with hotspot and easy 1x auth, you got like 90% of the market needs. How cool would it be to be able to just install a package on a supported device to manage it, throw in a flash drive for logging.

Tue May 31, 2022 10:04 am

Like stated above, there is no talk about a smartphone app or web app. The question is about concept of how a mass configuration protocol should operate.

pe1chl · Tue May 31, 2022 10:47 am

Then maybe it would be a good idea to not mention the word app in the first sentence in the opening post.
It should not be too difficult to design a protocol in-house and without input from us. You know best how the configuration objects work in RouterOS
and how it could be possible to have a router connected to some remote server that adjusts its configuration (much like winbox in reverse).
I think the protocol should operate at a low level and be independent of what you actually want to configure. That would be the next layer, where
there are some capabilities for templates, groups, mass deployment, etc. But that would not affect the low-level protocol that actually sends the config.

Larsa · Tue May 31, 2022 10:53 am

Like stated above, there is no talk about a smartphone app or web app. The question is about concept of how a mass configuration protocol should operate.

Looks like someone needs to practice their communication skills and talk to each other internally before starting advertise things. Here we go agan:

- is it a network communication protocol, hight level application protocol, an intelligent controller, all together or none?
- is the intention a pure configuration manager or something broader?
- what's the overall objective and main purpose?
- what's the target group/target audience?

If you're unable to answer these questions, it's probably a good idea to sit down and specify a project plan with a clear motive and objectives that you may use to communicate with others.

Tue May 31, 2022 11:37 am

before starting advertise things

huh?

sszbv · Tue May 31, 2022 12:07 pm

My ideas about a controller would be something like this, I would start using it right away!

- Controller server would be a separate package installed on a router
- Controller client would be part of standard routeros package

On the server I would want to create profiles, like 'accesspoint', 'client router wired', 'client router wireless' etc
each profile has a config script that would configure the entire device. In the profile you can also state on which map in dude the device should be added. Also the required RouterOS version. Maybe even some more usefull things that come up later while developing.

On the server I create provision rules, much like in capsman, based on the serial number and/or source IP and/or model of the devices.
Like, serial xyz uses profile abc.
Or * uses default profile.
Or 'hAP ac' from network 10.12.2.0/24 uses profile Locationx-clientrouter.

The server listens to mac broadcasts, and/or a TCP port. The IP address is published on the dhcpserver in an option. This way, the devices can either find the controller via mac protocol, or find it via IP address they get in the dhcp option. This would make it possible to use one central controller, published by all the dhcp servers on the entire network.

On a device, I would want to enter 'managed mode' in a way like entering 'capsman mode'. With the reset button.
In managed mode, a device uses mac protocol to find the controller, but also activates a dhcp-client on ether1 (or maybe each interface gets dhcpclient? bridge would be unwise because it can create loops) to see if there is a controller dhcp option.
The device connects to either the controller found via mac protocol, or preferred, via the IP address.

Server provisions the device according to the rules and sends a config file to the device, which is executed.
Optional, first there could be a version check and upgrade of software and firmware. Maybe even a list of files/directory structure that have to be pushed to the device.

Some useful functionality would be:

- push and execute a script file to multiple devices
- centralized backup
- remote shutdown
- connect winbox to the controller to get a list of devices (like romon)
- compare configs of devices to see differences
- auto add devices to dude

And please:

- make dude more scriptable from terminal
- include capsman into wireless snmp stats

- a webinterface would be nice, but I prefer using winbox
- there should at least be a list of provisioned devices with their details

I hope this is useful info for you.
Looking forward to the controller :)

Oh BTW, I would be really really really happy with an iOS APP that would include all winbox functionality!!! Multi window etc, just like winbox.
And while at it, make it universal so it runs on macos too :)

bratislav · Tue May 31, 2022 1:00 pm

At the moment we do not want to stick to a specific implementation or standard, but build our own that will help to manage, develop and deploy different scale networks running MikroTik devices.

Maybe it would be wiser to drop a bit of vanity and instead of inventing everything from the scratch base this new tool on some established standard such as RESTCONF for example which should not only cut developing time and effort but allow for MikroTik devices management to be easier integrated into an existing enterprise management systems ...

cfikes · Tue May 31, 2022 1:59 pm

- Controller server would be a separate package installed on a router
- Controller client would be part of standard routeros package.

On a device, I would want to enter 'managed mode' in a way like entering 'capsman mode'. With the reset button.

Can't quote your whole thing, but exactly how I want it to work. I always thought capsman could be extended to support switching and other features. Just mark profiles for a supported feature levels and ignore the rest. Capsman works just fine ( from my experience ) and seems to have all the framing needed to start quickly with this new controller concept. Could even call it MADman Mikrotok Access Device manager.

npeca75 · Tue May 31, 2022 2:04 pm

maybe the best idea will be to MT bring up own cloud for Controller APP and devices simply connect to personal account ?
this way, updates on Controller program could be in "in place", no more downloading new app, stopping,starting web,container,syncing versions on all computers, etc ...
similar to cloud backup

eddieb · Tue May 31, 2022 2:41 pm

please no cloud stuff, no fancy phone app.
create this as a package that can be run in High-Availiability on MT routers.
integrate some capsman, userman and dude stuff.
central configuration and pushing to devices
central upgrade management
central backup management

npeca75 · Tue May 31, 2022 2:51 pm

create this as a package that can be run in High-Availiability on MT routers.

you want another brick ?
my MT vX.xx work with Contr vY.yy, but not with vZ.zz
and yeah, my other MT will crash with this NPK, but if i first downgrade, then skip 2 version, then go one down, then it will work
and no, only on ARM works as expected, MIPS ... sorry
or ... you could continue this "OH NOOOO" chain forever :)

leemans · Tue May 31, 2022 3:03 pm

1- firmware update.
2- configuration backup and compare.
3- Wireguard VPN generator for client side (a file that can be import to fireguard software).
4- site to site ipse vpn
5- network and wifi settings to any mikrotik in the same site
6- firewall rules and NAT
7- IPS IDS
8- sd-wan
9- geo IP location for block and allow list.
10- WAN performance check: speeds, ping, jitter.
11- Web based
12- Division of the devices per customer
x Client Y has x devices
13- Visualization per Customer (like you can do in Dude)
14- Automatically create VPN connections (all possible types) by dragging the connection from the starting device to the remote device with needed setup parameters and created routes.
15- New devices - Automatically connect to the Remote Management Device Controller via secure Tunnel (ex. IPSec), by drag & drop or by pushing a butting 'Connect to Central Management'.

cfikes · Tue May 31, 2022 3:11 pm

create this as a package that can be run in High-Availiability on MT routers.
only on ARM works as expected

I do feel anything new needs to be only ARM. Heck MIPS only designs ARM cores now.

Larsa · Tue May 31, 2022 4:20 pm

huh?

Haha, bless you for at least reading the post even although the level of the comment is not what I expected. :-D However, I'm pretty sure that deep down you know what I mean thus feel free to answer the actual questions.

Anyhow, as you probably have noticed by now there are plenty of comments regarding the vagueness of the requirements and consequently a wide range of suggestions regarding how and on what platform this should be implemented on, all the way from a cloud platform, on premise to a regular MT-device. My suggestion is that someone at least try to narrow it down a bit by establishing some basic parameters for the runtime environment.

Or maybe you're just out fishing because you have no clue at all yourselves... ;-)

Amm0 · Tue May 31, 2022 5:00 pm

huh?

Or maybe you're just out fishing because you have no clue at all yourselves... ;-)

Hard to know, but they have a pretty list now, from "Layer 0" (fix bugs first) to "Layer 8" (maybe start with requirements).

Tue May 31, 2022 5:01 pm

Yes we are obviously fishing, did you not read the first post? Nothing has been made, we are asking for ideas how such a system should work in all of your opinion

olivier2831 · Tue May 31, 2022 5:21 pm

I think issue today is the export/import isn't symmetrical, backup/restore is monolithic+binary, . What's missing is idempotent configuration file. And that's the first step to being able to monitor/apply a configuration in "controller software".

+1

Tue May 31, 2022 5:22 pm

The central configuration system should send configuration to devices and should be able to retreive current/running configuration from devices.
The meaning of the word "should" is described in RFCs

Please, no Ubiquity style where configuration is stored in a local database and you are bounded to particular computer to reconfigure your network.

Tue May 31, 2022 5:24 pm

I think issue today is the export/import isn't symmetrical, backup/restore is monolithic+binary, . What's missing is idempotent configuration file. And that's the first step to being able to monitor/apply a configuration in "controller software".
+1

RouterOS does not store configuration in one plain text configuration file.

olivier2831 · Tue May 31, 2022 5:26 pm

For start, start just with:
Daily backup on text format, not binary on any point, and full configuration, included ssh keys, certificates, user-manager and dude database.
Some instruments to compare backup among various days for see the changes.
Some instrument for push configuration (like change NTP servers on all devices, or only on a group of devices, or only to selected devices.
Possibility to select/search devices by Group / Hardware / Branding/Platform, ROS version, BIOS/RouterBOOT version, installed packages.
Possibility to send .npk / .dpk / single file and select on what folder put it remotely based on internal memory type "root" or "root/flash".

+1

pe1chl · Tue May 31, 2022 5:55 pm

RouterOS does not store configuration in one plain text configuration file.

Yeah, fix that first. /export has to be a complete text dump of the configuration.

Tue May 31, 2022 6:12 pm

Export already is the complete text dump of the configuration but that text dump is not the way the config is stored. You cannot upload that text file to the router and expect it to replace the configuration on the router.

cfikes · Tue May 31, 2022 6:18 pm

Yes we are obviously fishing, did you not read the first post? Nothing has been made, we are asking for ideas how such a system should work in all of your opinion

Fishing is fun right!

Y'all have amazing api's, is there a need for a new protocol?

Please don't force cloud hosted. I feel Mikrotik is one of the last vendors I can use to host everything locally.

r00t · Tue May 31, 2022 6:21 pm

RouterOS does not store configuration in one plain text configuration file.

But it stores configuration in binary structure that can be backed up/recovered. So provide the tools to convert this binary into text configuration and back.
it's like registry on windows, where currently there is no way to export/import it from/to .reg file.
it would be perfectly fine if some parts were exported as binary blobs (hex dump, etc.), but doing export/import should result in 100% identical router configuration/settings.
It's perfectly fine to store settings in binary structure in ROS, it makes sense and it's more efficient than text config files... but please provide tools to losslessly convert it to readable form and back. This would be a good start...

metricmoose · Tue May 31, 2022 6:51 pm

Very exciting! This has been a bit of a missing component for Mikrotik's products and it could really help make our deployments easier to use.

Some things I would like to see:
-It should definitely have a mode where the router reaches out to the controller, like how cnMaestro and UISP work. It allows devices behind NAT to be monitored and maintained without punching holes in a firewall.

-Use the controller as a RADIUS server so device logins can be authenticated against it

-Push out software updates

-Push out config changes, using some kind of templating to make config changes repeatable. cnMaestro has this.
Example: You would have a "Change PPPoE client credentials" script that would look something like "/int pppoe-client set user={{USERNAME}} password={{PASSWORD}} numbers=0" and the user interface would allow you to pick that "change PPPoE client credentials" script and then prompt for the USERNAME and PASSWORD variables.

-Graph / monitor device stats. Preferably with different selectable modes/views tailored to the role of the device. For a home router, you might want latency between the controller and device, port bandwidth usage, number of connected WiFi devices, average signal strength of WiFi devices and uptime. For a tower site router, you would want a different view that shows things like temperature, input voltage, number of connected VPN/PPPoE tunnels, DHCP lease count, number of OSPF neighbors, ect that might not be useful to see on a home router.

-Alerting based on device stats. Simple high/low or reachability alerts would be great, like alerting when voltage or connection counts drop below a certain level or a device is unreachable. These could be setup in such a way that alerts could be logged, listed in an "active alarms" type list on a dashboard, or configured to send email / HTTP POST messages.

-Centralized logging

-Ability to group devices to separately apply updates, config changes or monitoring/alert settings. Have the ability to restrict visibility to certain groups of devices based on the user logged into the controller (Example: CSRs can see home routers, but not infrastructure devices)

-REST API access would be very helpful for integration.

-If it's a web-based tool, I would appreciate a way of pointing directly to a device by URL using the MAC address or serial number (Example, serial number: "https://mikrotikcontroller.yourisp.com/ ... 34567890AB"). This would make it easier for CRMs and billing platforms to link directly to a device in the controller from an inventory screen without having to involve background API calls.

-Self hostable. It would be nice if this could run on RouterOS, but a Linux software package or VM appliance would also be fine. If it has to run on an actual server, it would be nice to have a "proxy" or "remote sensor" for RouterOS that could be used for pings or device access from within that segment of the network.

-Make it very easy to onboard new devices to this controller. Right now it's a pretty tedious process to prepare new home routers with our custom config. If we could have fresh Mikrotik routers take DHCP specific DHCP options out of the box, or a config from a USB stick, that would make things go pretty smoothly. The "best case scenario" that I can think of would be having a Mikrotik PoE switch running a special DHCP server config, I unbox a bunch of hap ac2 routers, connect he PoE IN/ether1 port to the switch, they somehow show up in the controller, and get our "default" config without any intervention, or at least a few clicks to onboard the entire batch of routers.

Amm0 · Tue May 31, 2022 6:59 pm

Like stated above, there is no talk about a smartphone app or web app. The question is about concept of how a mass configuration protocol should operate.

If you're looking to be more radical, have you looked at ZeroTier's LF protocol (which is separate from ZT itself)?

The GitHub project describe it as "LF (pronounced "aleph") is a fully decentralized fully replicated key/value store" and licensed under MPL:
https://github.com/zerotier/lf

A controlled device could query the key/value store to find it's configuration by some tags specific for the device, with the controller maintaining the device/configuration in the distributed store.

kraal · Tue May 31, 2022 7:03 pm

we are asking for ideas how such a system should work in all of your opinion

Well, the first message is rather "vague". What is a controller app ? Is it a mobile application ? A thin client ? A thick client ? Public Cloud ? Private Cloud ? Local network ? Give us more information of what you're aiming at. The best would be to first define your own list of ideas, submit them to the community / to your list of registered clients as a survey and ask people to vote for features, then let them add more ideas or check a checkbox "you're on a wrong path".

Now if you want the "wishlist to Santa Claus" answer from "requirement engineers working for free" : Same as latest TP-Link's Omada SDN Controller but with the following additional features:

with a staging feature (i.e. ability to configure / provision on a test (flat) network a bunch of new devices, push the configuration onto the devices, export the controller configuration, import the configuration on the production controller and deploy the devices in production and "voila" it works seemingly without having to deal with network complexity for provisionning)
ability to one-click turn on/off SSIDs
ability to filter graphs/data to have a focussed view of network usage by a specific network client device (to answer questions such as "which device was eating the bandwidth at 3am ?")
with 2fa fido auth support
open source
using up-to date libraries (nobody wants to have to dig for 4 years old libraries to be installed)
using only non proprietary libraries libraries (nobody wants to have to download N additional proprietary libraries from N sites to install a single tool)
with an API / plugin framework to enable the build of "connectors" to allow control of devices other than Mikrotik ones
with a more complete map/WiFi simulation i.e. with more wall, room, furniture, doors, standard material types
with "AP placement recommendations" based on facilities maps and areas where signal should be strong
with the ability to display "custom" sensors placement and measured values on the maps, etc
and on the devices to be discovered side: do not spam the network every 10s with "hello" announcements (who adds network devices every 10s ? who can't wait for 5 min to discover a newly added device ? why the hell an already discovered device continues to yell "hello" every 10s after having been registered ?)

Oh and please make sure that your tool is:

fully tested
using a standard release cycle (i.e. LTS == experience shows that it works and we commit to making sure it will still work in the future, but we won't backport new features, only security/bugfixes != "it should work", "crossing fingers", "sounds cool")
respecting opensource licenses of used libraries (for instance having to share and licence your own code under GPL if you use GPL code, listing all open source libraries i.e. with no exception, making the code available to all third party while "[not being] allowed to charge more money than the cost of copying the media and shipping it" ;-p )
using an accessible bug tracker

That being said, please first focus on existing issues and make sure that RouterOS7 provides PIM-SM, that hAP AC2 is able to use all cores, etc...

Thank you ! ;-)

Sob · Tue May 31, 2022 7:04 pm

Export already is the complete text dump of the configuration ...

I beg to differ, I still didn't have any luck finding e.g. users or certificates in mine. So it's not exactly what I'd call complete.

gmsmstr · Tue May 31, 2022 7:40 pm

There are services out there that do parts of what is requests. We have https://cloud.linktechs.net that does this for many customers. Just a FYI.

Larsa · Tue May 31, 2022 8:47 pm

Yes we are obviously fishing, did you not read the first post? Nothing has been made, we are asking for ideas how such a system should work in all of your opinion

Since you mentioned that fishing was the main purpose of this thread then it might be easier to just ask what people need in general instead of focusing on side issues like "an app", "a protocol", "a configuration protocol" "a controller", which are just implementations details. As you probably have noticed most suggestions covers all parts of FCAPS ie montering, configuration, performance, provisioning, accounting and fault management, thus limiting the discussion to just configuration seems a bit odd in this case since the intention of the thread was just about "fishing" for a general feeling of what's needed.

Hopefully MT is to conducting a somewhat broader market research by other means than just asking the forum.

When it comes to standard management protocols there are plenty of them like for example CMIS/CMIP, TL1, SNMP, etc. ITIL for Network Service Delivery is a good entrance for getting a grip of best practice and also get some insight how things works in regards of FCAPS. .

And just a friendly reminder, doing everything from scratch as well as adopting the doctrine "not invented here" when it comes to for example protocols and tech-stacks, is an excellent way to deliver a solution DOA.

pe1chl · Tue May 31, 2022 8:49 pm

Export already is the complete text dump of the configuration but that text dump is not the way the config is stored. You cannot upload that text file to the router and expect it to replace the configuration on the router.

Yeah, fix that too!
I have discussed it many times. RouterOS needs a way to migrate config from one device to the next. As restore of a backup is not possible in this case, it has to be done via export/import. But import is far too finicky to use it for this purpose.
There should be some mechanism to tell a router to forget its current config and import a new config that is from a different but feature wise similar router.
E.g. from a 2011 to a 3011 or 4011.
When the new router encounters config that it cannot apply, e.g. for LED or LCD, it should just ignore it.
Also the longstanding bug with "reset defaults and run script at startup" (introduced somewhere in 6.3x) where the script starts before the initialization of the router is finished and thus interface configuration commands result in error and termination of the import should finally be fixed.

Larsa · Tue May 31, 2022 9:11 pm

As @pe1chl and @sindy pointed out, the whole backup/restore/export/import thingy needs to be sorted out and would probably need some adjustments in ros as well since I believe it's not enough with just a "smart controller" for this purpose.

miroslaw · Tue May 31, 2022 9:20 pm

Whatever you are doing, hope it'll work on linux (not like wine+winbox)

Larsa · Tue May 31, 2022 9:31 pm

Whatever you are doing, hope it'll work on linux (not like wine+winbox)

If they are smart they implement the solution as a "container appliance" that's able to run as a cloud service, on premise or perhaps even on MT devices like RB5009 or CCR2004 if they meet the requirements for RAM and storage.

cfikes · Tue May 31, 2022 9:44 pm

Whatever you are doing, hope it'll work on linux (not like wine+winbox)

If they are smart they implement the solution as a "container appliance" that's able to run as a cloud service, on premise or perhaps even on MT devices like RB5009 or CCR2004 if they meet the requirements for RAM and storage.

I whole heartly agree. This is a 100% perfect use case for OCI containers on ROS.

vanikcz · Tue May 31, 2022 11:49 pm

Great idea!
I'm suggesting to create some high level configuration that generates a rsc script, that will be downloaded to RB in provisioning process. As user I would like to add some lines ro generated code...

killersoft · Wed Jun 01, 2022 5:45 am

Its a great idea.

I manage approximately 97 mikrotik devices from my desk. Of which I have about 12 different models of MT hardware incl a couple of VM's
Dude only gets me so far with f/w updates with the hardware side.
Ideally I want a platform that :
1. Keeps an eye on configs across all devices and alerts me to manual changes of configs
2. Back's up / Restores configs to a local dB and or local drive->nas etc( can get to config file if things go bad! ).
3. Capsman on the controller ???
4. Some basic dude ( is it up/down ).
5. Push/Pull out common configs( e.g set time/date, SNMP, log etc ) to ALL devices, so we can ensure those items are the same, and a COMPARISON option to visualize( table of config info ?)

There are plenty of good suggestions in this list already,

kikikaka · Wed Jun 01, 2022 9:33 am

Pls keep the winbox or similar thing which I think it is a very convenient tool for configuring the device, especially for general user which usually have only a single device for home router purpose. Although I am also using UBNT AP and running a controller for that.....

OlofL · Wed Jun 01, 2022 10:16 am

Selective configuration sync between two or more "clustered" devices (firewall/mangle etc)
Some kind of HA...

Container support!

Smart firewall address list support (geoip, Adblock, bad IPs)

corp9592 · Wed Jun 01, 2022 11:17 am

Fantastic idea!!

Centralized FW upgrade and version control. To be able to see what version the devices are running and update them in 1 click.

Web based

Better visuals

Metrics and nice graphics

Client topology and client information (good for home networks, to identify devices)

VPN wizard and Wireguard client config generator

Cloud backup and restore (similar to how is it now)

SiB · Wed Jun 01, 2022 2:47 pm

Set Distribution Profiles per feature, like we now can replace a one file and give differ GUI in WebFig.

"LTE Global Profile " who will set on all my devices my genal settings like the same packages. ntp, dhcp server, etc....
"LTE APN ATT, EM160-G" can deploy my .rsc dedicated to that modem with bunch of dedicated scripts only to that modem. Set APN to that ISP, Set other stuff.

For me that Deploy Tool should replace a name.auto.rsc over ftp.
The Dude is NMS but it not have a way to Deploy configuration and this can be added now.

I use TheDude server, old version installed at Windows with own extrnal scripts who send over ftp my file to second IP in Dude who is internal VPN IP... but I cannot select many Devices and do that as one queue.

Just add a discribution of auto.rsc over The Dude and it will be perfect. In my case of course.

raffav · Wed Jun 01, 2022 5:26 pm

Not over complicated stuff..
For a 1st version it will be great "caps man" for every device...
Also I think this to work perfectly I will start making the ip cloud not just a simple ddns but also a reverse proxy kind of...
So we can reach out the device even when device is behind "cgnat"
And use this address as a provisioning way
Similar to caps man provision but instead of using radio mac address use the serial of the device.

And a 2nd stage I really think you can use the dude a all in 1 solution..
Monitor and manager all mk devices from a single point...

Larsa · Wed Jun 01, 2022 5:48 pm

Not over complicated stuff..

I beg to differ. In my opinion, it depends entirely on the use case. As you’ve probably noticed, there are many different views on what's right or not.

hecatae · Wed Jun 01, 2022 6:10 pm

Could it be an extension of the mobile app?

TP-Link offer TP-Link Tether, which connects into various devices using api username and password.

Amm0 · Wed Jun 01, 2022 8:17 pm

Set Distribution Profiles per feature, like we now can replace a one file and give differ GUI in WebFig.

[…]
I use TheDude server

[…]

Just add a discribution of auto.rsc over The Dude and it will be perfect. In my case of course.

100% agree here. On practical level, this what a “controller” do that be exactly what I’m looking for my actual use cases.

Quickset already has “profile” - they just are so finicky and unflexiable. But that’s fixable.

And if there was Dude2, perhaps addingcentral conntrack (like V7 sync tracking VRRP) and capsman-like tunnels that used WG automatically.

Taking “CAPsMAN but for all Interface” as concept. If you could use it to apply SiB’s distribution profile - which in my mind a combo of a quickset profile to use but that take device info from Dude DB instead of end-user doing it.

If quickset actually work well (which under MT control to fix…), it solve the webfig GUI. Since the configuration for quickset be controlled by a Dude server in this example, the dude need be found from discovery via MDP/CDP/RoMON/DHCP a la capsman but again for an interface. Now IMO discovery work via mDNS locally, falling back to SRV records in global DNS.

Obviously the option to apply an entire configure (beyond just quickset) in the “distribution profile” should be included. As would customize the quickset profiles via branding kit for OEMs be a nice-to-have in this concept.

Anyway more fodder for y’all.

Edit:
With security via certificates (eg config pushes are signed similar to MDM configure on smartphones)

nithinkumar2000 · Wed Jun 01, 2022 9:43 pm

Features:
1. Centralized Monitoring of Routers (Like Tempertaure, Interface Down Alarms, CPU/Memory Usages and Fan Status)
2. Provision to monitor the SFP TX and RX in a central console
3. Option to save System Logs from Routers to a Central server for troubleshooting Purpose
4. Opiton to Backup both configuration and bkp file from a central point.

scampbell · Wed Jun 01, 2022 11:30 pm

Exciting news !
- web based, so the controller can be accessed from nearly any device or platform and eventually phase out winbox. ideally the controller should not be based on java but static binaries deployment or open source ?
- better graphing and analytics, time to move away from mrtg. rrdtool is good, what about grafana ?
- integrated "tailscale-like" controller to easily set up wireguard links between managed endpoints and automatically handle endpoint ports, NAT hole punching etc. "one click wireguard vpn" could be a great marketing tool
- devices overview with status page, device model image, and a satellite map overview to plot wireless links and do line-of-site calculations similar to ubnt's UISP
I'll update my list if I can think of anything else :D

I'd not like to see Winbox go - its Layer2 discovery and control capabilities are invaluable

ck230885 · Thu Jun 02, 2022 1:18 am

Please feel free to look at Mikloud which is UK based we supply hardware also with a free cloud controller for all Mikrotik devices
we are happy to give demos and will answer any questions you may have
www.mikloud.co.uk - 00441507862718 chris.kent@tutelanetworks.co.uk

npeca75 · Thu Jun 02, 2022 12:24 pm

If MT ever make this Controller , maybe the best route will be a (hated) cloud

why ?
there is still many device with low amount of flash/ram
so MT mantra that every device must run every version of ROS is breaking this way
then, messing with NPK versions
then, exporting hidden database from device, backing up, etc

from my point of view, at start, there is a strong reasons why Cloud
embedded linux DEVs could concentrate on "ring 0" aka low level driver on MT device
and "fancy" java/css/php programmers could develop CloudController as they want

only setting on low level driver need to be:
1. use Controller on/off
2. Controller Address default/custom

this way door for self hosted docker controller containers will be opened from start
but at first run, MT could focus on their own programming, and not the zillion of user who tried container under zillion of condition on zillion OS with dozen of CPU arch/MT hardware

after few year, when remote controller protocol stabilize, and become safe, MT could publish containers, and from this point, every one could start using them on closed network

but until then, dealing with MT hardware, dealing with protocol, dealing with docker images, and putting pressure on ticketing/help center ... no way it could work

after all, PRO users who need closed network will delaying this controller anyway so it is safe to assume that home/small office user will start to use this technology. And they are all connected to internet anyway so ...
if we NEED to use this, and as i see, MT will push this thing because other vendors have similar Controller, less problematic will be in MT controlled Cloud for start

infabo · Thu Jun 02, 2022 12:58 pm

It depends how they are going to implement the controller.

They can go the Unifi way and create a standalone platform-independent controller application. The regular user would just install the controller on any non-mikrotik device. But "powerful" ARM mikrotik devices could run the controller inside a docker container on any network-device too. Max. flexibility.

They would limit themselves too much, when developing a controller NPK package for ROS. Hardly any now existing ROS-hardware-device would have enough free disk space nor enough RAM.

linas · Thu Jun 02, 2022 1:30 pm

I think, the dude is good enough software and mikrotik just needs to expand its functionality,because maps of the dude and visuality is unique. Plus functionality by scripts and user needs its very important. You folks just make it more flexible plus more already integrated functions like mass password change, configuration download ( for now I made script in services, but it has some limits) and more and more. I dont think, that another system will make something better, when you have this, just make it workable on linux, not just ROS.

pe1chl · Thu Jun 02, 2022 2:07 pm

I think, the dude is good enough software and mikrotik just needs to expand its functionality,because maps of the dude and visuality is unique.

It would be a nice option to have a controller that is connected by the router, instead of the other way around. That enables management of routers that are behind NAT.
Of course one implementation could be to have some kind of VPN setup by the router (L2TP/IPsec, SSTP, etc) which then is used for the controller to connect to the router in the existing way (API, winbox).
A problem with Dude is that it is so much work (on a larger network) to detect, configure and map everything.
That should not be a mandatory activity. Lots of users are well served with only "table" presentations of the data and no fancy map.

linas · Thu Jun 02, 2022 2:13 pm

It would be a nice option to have a controller that is connected by the router, instead of the other way around. That enables management of routers that are behind NAT.
Of course one implementation could be to have some kind of VPN setup by the router (L2TP/IPsec, SSTP, etc) which then is used for the controller to connect to the router in the existing way (API, winbox).
A problem with Dude is that it is so much work (on a larger network) to detect, configure and map everything.
That should not be a mandatory activity. Lots of users are well served with only "table" presentations of the data and no fancy map.

to access router behind nat no need to use vpn, just use NAT mikrotik as dude agent, and all other router will be visible in your centrall dude

odge · Thu Jun 02, 2022 2:59 pm

Ensure your configuration happens on the existing API. Ensure the controller speaks "whatever protocols it can" but always uses API as entry/exit point for config. You can short hand that API on certain lightweight protocols, but please, do not now add another entry point to your device. Fix what you got!

Support multiple protocols on top of that. Support the control plane to use this like automatic gateway detection (but only the control plane). Support an eco system on top of your controller, including self-hosted for higher security.

Allow your global controllers to point devices to custom hosted controllers (if you support out of the box config). Once owned, require release if reset to defaults for global controller to point it again somewhere else. Never allow global controller to reset device owned elsewhere.

turnip · Fri Jun 03, 2022 8:08 am

I've written some Lambda functions and other AWS services to connect Mikrotik devices to a Wireguard VPN, assign an IP, collect Netflow and logs at a central location and store them in S3. Nowhere near finished, and I also plan on taking advantage of the REST API to push out configuration changes.
Most of what I've read here sounds good. If the controller can be managed with a REST API, we can tie it in with other systems (eg SDN).
It'd have to be self-hosted or able to host in our own cloud tenant, not SaaS.
Zero Touch Provisioning would be great (especially once my wholesale ISP moves away from PPPoE).
Mikrotik or third-parties could also offer plugin services, eg a cloud based realtime threat analysis, bandwidth monitoring etc.

Fri Jun 03, 2022 3:50 pm

The primary idea was the option to control RouterOS devices. But when it comes to that we will look for a possibility to control the SwOS devices too.

SiB · Fri Jun 03, 2022 6:38 pm

mrz write:

The primary idea was the option to control RouterOS devices. But when it comes to that we will look for a possibility to control the SwOS devices too.

Then please add a disctribution feature to mass deploy spript to The Dude and we are happy with that. Come Back us TheDude as server at Windows (add contener or linux) and we will build our "clouds" online distribution platform's.
Even now in TheDude one device have few IP Address field what I use and can targer PublicIPs and InternalIPs by them BUT it's lack of MultiSelect devices, group them etc..
I don't belive that you create a next tool who will do a VLANs on 3 ways depend of detected RB and differ for SwitchOS when you cannot finish a migration from ros6->ros7 on one go.

Be more focus at RouterOS 7, on Wifi 6E, and take a wifi alliance cert and UserManager and more HowTo !. Those are more important stuff then some distribution tool on ros who change CLI/API from version to version.

PS. Thank you MikroTik for new RB LHGGM LTE18 !

Amm0 · Fri Jun 03, 2022 7:35 pm

Agree with @SiB...

@mrz keeps describing a push-based mikrotik device manager... Yet, it's like Mikrotik wants to forget The Dude even exists. I just can't but think a few "bug fixes" to the dude go a long – pick any of the suggestions here. Since a "new device controller" should "monitor" the controlled_devices, they'd need to re-build basically same things that Dude already does. Even modern IoT things do same as Dude long ago: write time-series data to a sqlite, just with MQTT [which ROS supports now] instead of SNMP. Why throw that away?

What I know is there are at least 837 Mikroitk customers that petitioned the U.S. government:

The Dude is a extremely powerfull application developed by Mikrotik to manage and monitor network devices running SNMP protocol. For years its development is stopped and mikrotik keep it for it self. This petition is at the same time a tribute and a ultimate request for Mikrotik to release the source code and let the opensource community develop the ultimate NMS System for us all.

(from https://www.change.org/p/mikrotik-relea ... ource-code)

I'm not advocating releasing the source – just saying clearly there is demand for a Dude2 & that sounds a lot like "Device Controller [Software]" y'all started with. If the Dude could manage keys/etc for WG, boy that be nifty bonus [and get you secure tunnels to protect "lightly-secured" winbox/api protocol the dude uses today, largely by storing the needed WG info in Dude device DB].

mducharme · Sat Jun 04, 2022 1:45 am

The NETCONF protocol is designed for this sort of thing. Other router vendors are using it for this exactly.

https://en.wikipedia.org/wiki/NETCONF

Probably makes sense for MikroTik to do the same instead of developing a new protocol?

npeca75 · Sat Jun 04, 2022 7:21 am

whatever they do, client need to be really tiny footprint. 100-200k
if they want to fit in 15.2 MB space and be available on every (client) device
how secure & bullet proof will be client code with such a tiny space ?

pe1chl · Sat Jun 04, 2022 10:48 am

Yes, space requirements are probably a concern. That is part of the reason why I suggested to make a "configure anything" protocol that is basically the same as API but in reverse (the router connects to the controller, trust is established between the two, then the controller issues API calls over that connection).
That could likely share most of the code with API and "we" would not have to list beforehand what we want the protocol to be able to do, because it can just do "anything" and it is the application that defines what is possible, not the protocol.
But I have apparently misunderstood the question, as there was no real reaction to that and the discussion continues to wander in many directions.
Maybe MikroTik really did not ask for the protocol. Which would surprise me anyway, as it is them who have the expertise on the internal workings of their closed-source system, how can we suggest what the protocol would look like.

Unless indeed it would be an already standardized protocol like NETCONF or SNMP or some generic protocol like REST.
But in that case, there likely would have to be a "conversion layer" and when it cannot be made completely independent from the actual transactions (i.e. there would be an implied 1:1 translation between protocol verbs and API options), it will likely be very bulky and support only a limited number of settings.
That would make it more like TR-069. We already have that.

Bruzxce · Sat Jun 04, 2022 10:49 am

Most of the things that MT is missing is that they don't have centralized monitoring, dude is way left behind and they discontinued to upgrade the DUDE which is more helpful to all the users.

For there concept of creating a controller for MT its quite amazing if they can unified all the hardware devices that they have and also to increase some of there hardware specs which still they have low end specs. Now technology is getting huge in the Infra so they need also to do drastic upgrade on there hardware specs on the devices.

Paternot · Sat Jun 04, 2022 2:57 pm

Please, no Ubiquity style where configuration is stored in a local database and you are bounded to particular computer to reconfigure your network.

Oh, yes. The day Mikrotik goes this route would be the day I would need another vendor.

EDIT
Clarification:
Something centralized, to config/monitor all Mikrotik devices, would be great (why not expand the Dude?)
But it MUST be optional - just like it is today.
/EDIT

DjM · Sun Jun 05, 2022 11:16 am

Would be great to have docker version of the new app, which can run on Raspberry Pi 3 and newer.
And also have version which is fully cloud, or other 3rd party independend.

troffasky · Sun Jun 05, 2022 12:28 pm

-It should definitely have a mode where the router reaches out to the controller, like how cnMaestro and UISP work. It allows devices behind NAT to be monitored and maintained without punching holes in a firewall.

Yes please, make sure endpoints work from behind CGN.

mwisniewski · Sun Jun 05, 2022 12:34 pm

Please do not go this way. I had nasty security accident (for my defense - I discovered in on first day of my job) with Unifi Controller itself. Remote management is always best way to introduce new security holes - unless of course you do that in a secirity oriented paranoic style.

pe1chl · Sun Jun 05, 2022 1:15 pm

You can (unless the manufacturer makes that impossible) always choose to run the controller within your own local network or within a VPN overlay.
Don't know if that is still possible with (all) Unifi devices, they were tending to move towards "cloud only" when I last checked.
But of course MikroTik do not need to make that same mistake. A Unifi controller within a separate management VLAN within a company (possibly linked using VPN) is not that much of a security risk, isn't it?

Phillip · Sun Jun 05, 2022 3:35 pm

Since Capsman would require a total rewrite to work with WiFi5/6 interfaces and the Dude server basically being obsolete, and Winbox only working on windows, I can see why they are looking at going this route.

I would stick to the ARM (Maybe Risk-V) processor platforms for the external unit controller and also offer routers that have it as well for the smaller business and home users. They may also want to take a look at Nagios Core and NEMS Linux to get some ideals from, or maybe implement instead of reinventing the wheel.

Nagios Core https://www.nagios.org/
NEMS Linux https://nemslinux.com/

For the remote unit, I would allow it to be VPN'd into, have two Ethernet ports (One OOB Management) and two USB ports that can be used for logging with a stick or drive. Also, having the capability of internal storage would be nice. It should control and setup access points, routers/firewalls and switches (POE As well), and give detail maps and log files. It should allow you to upload and push config files and have a better interface for firewall rule generation through the GUI.

For a router that will have incorporated, I would not allow VPN control capabilities, but instead have a OOB Management port that can be tied to a remote PC for control.

Amm0 · Sun Jun 05, 2022 11:06 pm

The NETCONF protocol is designed for this sort of thing. Other router vendors are using it for this exactly.

I'd prefer to live in an XML-free world – while text, it's just not that easy for humans to write or read.

But if you need more suggestions: no one has mentioned a BGP-like protocol for configuration.

At high level, BGP message could contain config instead of prefixes. Since BGP embodies the notion of "communities"/groups and "advertisements"/discovery, those are needed for any controller protocol. Even BGP states strike me as similar to config mgmt.: "open", "update", "keepalive", "notify". Not saying it be a good idea, but another one, OR that it actually be BGP protocol just a similar architecture.

mkx · Mon Jun 06, 2022 8:09 am

Mentioning other, already existing management protocols: what's wrong with TR-069? It's widely adopted and seems it's intended to do remote provisioning and management.

mada3k · Mon Jun 06, 2022 10:07 pm

TR-069 is horrible and basically just good for customer CPEs.

There should be two ways for handling management. API from devices to the management server (for devices behind NAT) - and - direct management/monitoring (as in the management server is reaching directly to a device via ssh or some api)

SiB · Tue Jun 07, 2022 12:17 pm

For me it's transparent that will be used own ros protocol, tr-069, ftp, ssh because we need allow to communication from our end-device to this "DistributionTik" app and this solve all CGNAT etc.
This app must give us way to
* group units, create bigger group from other group's like we can use those groups: "RB95x" + "RB_LTE6" give us our "RB_Branch" who + "RB_CCRs" = "All_Customer1"
* Multi Select as group/unit that I can say to push config to that grups and that units...
* Re-Connection and Re-Send configuration (like push update.auto.rsc who do reboot ... and resend update.auto.rsc who do latest but upgrade firmware and reboot... resend who confirm latest ros and firmware and remove itself and the end of task, report that all steps done)
* Use all IP to unit who works as RoundRobin, like first will be internal IP over VPN, ... Public IP ... and RoMON connection too!.

In my scenario, TheDude can do that and for easier it should have import connection from WinBox . In WinBox I have at least 2 IP to the same unit (Public and Internal PPP and internal LAN of branch)

Summary, give us way to send our scripts.auto.rsc over ftp over TheDude who can be installed on any VPS/Container and we build or differ distribution centers. TheDuce can do provisioning of configuration and it's all.

raffav · Tue Jun 07, 2022 1:26 pm

For me it's transparent that will be used own ros protocol, tr-069, ftp, ssh because we need allow to communication from our end-device to this "DistributionTik" app and this solve all CGNAT etc.
This app must give us way to
* group units, create bigger group from other group's like we can use those groups: "RB95x" + "RB_LTE6" give us our "RB_Branch" who + "RB_CCRs" = "All_Customer1"
* Multi Select as group/unit that I can say to push config to that grups and that units...
* Re-Connection and Re-Send configuration (like push update.auto.rsc who do reboot ... and resend update.auto.rsc who do latest but upgrade firmware and reboot... resend who confirm latest ros and firmware and remove itself and the end of task, report that all steps done)
* Use all IP to unit who works as RoundRobin, like first will be internal IP over VPN, ... Public IP ... and RoMON connection too!.

In my scenario, TheDude can do that and for easier it should have import connection from WinBox . In WinBox I have at least 2 IP to the same unit (Public and Internal PPP and internal LAN of branch)

Summary, give us way to send our scripts.auto.rsc over ftp over TheDude who can be installed on any VPS/Container and we build or differ distribution centers. TheDuce can do provisioning of configuration and it's all.

I think the same.
I think Mt can reuse dude for that as well be all in one solution.
I think that since most rb have small storage..
This controller can't be inside ros..
But the can have some kind of proxy to the dude then can comunicate aslo when device is behind cgnat..

Amm0 · Wed Jun 08, 2022 3:42 am

Improved Dude + @S[io][bB] config's suggestions should be nexthop...
But running with [/controller/fantasy]...

At the moment we do not want to stick to a specific implementation or standard, but build our own that will help to manage, develop and deploy different scale networks running MikroTik devices.

Another concept be the controlled device is just a

git

repo with config (instead of "source") – basically the git "push" and "pull" be controlled by some TBD Mikrotik scheme (e.g. RouterOS use git hooks that config/upgrade/etc based on the repo, and run by Mikrotik-coded hooks). So the controller be similar to "enterprise" GitHub (self-hosted, but same actions, push requests to auth, orgs for groups). A Git-like approach certainly make "diff" RSC easy. And if done right users could configuring the push/pull scheme, or even "forks"/"branches".

Marvinjul · Wed Jun 08, 2022 3:00 pm

Ability to monitor the speed of the Internet, and get a notification in case of reduction.

woland · Wed Jun 08, 2022 3:12 pm

Ability to monitor the speed of the Internet, and get a notification in case of reduction.

Like having a constant stream of prioritized data saturating your uplinks. (Not so) Great idea! :)
But actually there are devices which you can have configured with alerting if bandwidth drops below some defined level on an interface.

W

prawira · Thu Jun 09, 2022 10:02 am

like ACS ?

woland · Thu Jun 09, 2022 10:55 am

like ACS ?

Hi
I don´t know about such functionality on ACS (if you are writing about the AccessControl Server from the big C) , but I have seen this on load balancers and on firewalls.
On the LB it works like the following: if there is suspiciously low traffic incoming, it marks the interface dead. (Of course you can configure ping checks and other protocol checks as well, but ping check is also there on MT.)
On the firewall:
-you can have the functionality like on the LB with monitoring traffc
-you can actually have true throughput measurement between two firewalls, but that is connected with a proprietary VPN technology and used if you have at least 2 uplinks
-you put 2 VPN tunnels over 2 uplinks between the 2 Firewalls, the firewalls observe the traffic between them and send out short bursts of traffic periodically to determine how to load balance the traffic between the two VPN tunnels over the two uplinks
-you can do this between many firewalls over many uplinks
-that is probably not something MT would implement in 7.5...
-I don´t want to advertise this product, but here is a link as some inspiration for devs :
https://campus.barracuda.com/product/cl ... th-sd-wan/

BR
W

pe1chl · Thu Jun 09, 2022 11:07 am

Of course you can do all of that, or most of it, on RouterOS as well...
The problem is how to trigger an alert. Sure, when you have an internet connection that is saturated 100% of the time (like those "wireless ISP" that share a single 20Mbps line with 100 customers) you can do something like "when my input rate drops below 10Mbps send an alert".
But in the general case, where the internet is usually lightly loaded (running below 10% capacity most of the time, maybe even idle during night), it is much more difficult.
Of course you could try to make "background traffic" that is handled at lower priority in queues, to fill the line, but that requires complete confidence in the priority handling at all places in the network (you often cannot influence how the ISP does their queueing), and also you could be limited by max data quota etc.

prawira · Thu Jun 09, 2022 11:08 am

there all,

there are couple options of ACS that can be use with MikroTik but not the one for SDN, so perhaps MikroTik can develop the SDN software as it should be able to control the routers with the flows planning

dear woland, you can see the following post to know about ACS:
viewtopic.php?t=172399
and the are some presentation regarding this topics (ACS) as well, the last one on MUM ID 2021.

BR

woland · Thu Jun 09, 2022 11:44 am

Of course you can do all of that, or most of it, on RouterOS as well...

yes you can probably do some of that on ROS by utilizing scripts and routing and NAT and QoS , but it´s very impractical.

The problem is how to trigger an alert. Sure, when you have an internet connection that is saturated 100% of the time (like those "wireless ISP" that share a single 20Mbps line with 100 customers) you can do something like "when my input rate drops below 10Mbps send an alert".
But in the general case, where the internet is usually lightly loaded (running below 10% capacity most of the time, maybe even idle during night), it is much more difficult.
Of course you could try to make "background traffic" that is handled at lower priority in queues, to fill the line, but that requires complete confidence in the priority handling at all places in the network (you often cannot influence how the ISP does their queueing), and also you could be limited by max data quota etc.

This type of uplink monitoring and traffic steering is implemented by many vendors and works so well, that most of big industries and enterprises are abandonig a lot of MPLS links in favor of the much cheaper Internet links.
In a normal enterprise scenario, you have many sites and each have 2-3 Internet or MPLS uplinks. You also have a few central sites with fat internet and MPLS links.
In between there are ISPs with much bigger pipes than your remote sites have. There is no QoS on the Internet, but that´s not an issue in real life. The probes are managed dinamically and they take the production traffic into consideration. The algorithm of the probes is proprietary and not fully disclosed, but there is no magic in there, it must work good enough and not perfectly.
If you want perfection, you buy a dark fiber or at least a wavelength, but even then the big excavator may find you, so you better also buy some backup link. (few millions to few thousands of $$$ p.a.)

@prawira: I was not aware of that ACS, but thanks for the links !
W

pe1chl · Thu Jun 09, 2022 12:12 pm

I am doing that on MikroTik with just a couple of tunnels and BGP to autoroute/failover between them.
No need to watch throughput as the lines are normally lightly loaded. We normally tunnel over IPv4, when that fails we try IPv6 (yes, it has happened that IPv4 routing was down at the ISP but IPv6 still worked) and when both fail we use LTE.
No fancy mysterious secret stuff, just plain routing with MikroTik.
But inter-office links are becoming a thing of the past anyway.

woland · Thu Jun 09, 2022 3:08 pm

I am doing that on MikroTik with just a couple of tunnels and BGP to autoroute/failover between them.
No need to watch throughput as the lines are normally lightly loaded. We normally tunnel over IPv4, when that fails we try IPv6 (yes, it has happened that IPv4 routing was down at the ISP but IPv6 still worked) and when both fail we use LTE.
No fancy mysterious secret stuff, just plain routing with MikroTik.
But inter-office links are becoming a thing of the past anyway.

Hi pe1chl
You are missing some of the features. Yes VPNs and dynamic routing over them is what you do. There is at most ECMP and mangle rules to load balance.
What you probably don´t do is: dynamically load balancing over links. Also you are in trouble if you must make a local breakout for some applications like Office 365+Youtube+Slaesforce+Zoom+Webex+Google at all sites, but the rest of the traffic has to go over some central box.
You might manage to do all that, but it´s a lot of work...

Inter office links are getting used a bit less, but they are not gone. There are mostly applications in data centers, which should be accessed via the WAN. Not everything is in the cloud, ZeroTrust and co. are not there yet.

W

pe1chl · Thu Jun 09, 2022 4:29 pm

We don't need to load balance. Our main office has two fiber connections (different ISP) with more bandwidth than the branch offices have on their main internet connection (fiber or VDSL), so there is nothing to balance. We only need to cover line failures, and we do that as described above.

mrigi · Thu Jun 09, 2022 6:42 pm

You could make the use of MQTT (or any other broker) for this. So monitoring enabled devices efficiently send info into the broker and if "controller" is up it can display received data the way it wants and without actual connections to the every device. This decoupling will be more flexible and might scale better. You can even run multiple controllers without any troubles.

Amm0 · Thu Jun 09, 2022 6:56 pm

You could make the use of MQTT (or any other broker) for this. So monitoring enabled devices efficiently send info into the broker and if "controller" is up it can display received data the way it wants and without actual connections to the every device. This decoupling will be more flexible and might scale better. You can even run multiple controllers without any troubles.

Well that approach works for AWS IoT Core. Basically this mythical controller, using MQTT, could borrow their "device shadows": https://docs.aws.amazon.com/iot/latest/ ... adows.html

MrBarakat · Thu Jun 09, 2022 9:45 pm

Something like UISP of ubnt

rextended · Thu Jun 09, 2022 9:47 pm

noooooooooo NOT THAT...

devrand · Thu Jun 09, 2022 10:50 pm

It kinda depends on what delivers the most value to the customer and adds the most value to the Mikrotik devices

A management tool for the devices (like extended winbox) or a central controller like the unifi controller?

status overview of devices / health overview / device discovery
Centrally manage software updates of devices
Centrally configure devices
Backup and restore config
UI accessible from mobile as well as desktop devices
Not require internet access for operation
Be able to manage the device via WebUI as well as the new controller at the same time

kiler129 · Fri Jun 10, 2022 5:51 am

Many great things has been said here and I sign under them. I will have one suggestion:

Please start small and ascetic.

We don’t want a perfect solution in 10 years. An incremental small thing which grows will be much better. v7 is amazing but I think it suffered from a waterfall. The new controller doesn’t need a fancy react-based dashboard with live updates and AR app to see ports… it just needs to work. Look at the Ruckus Unleashed interface: it’s simple, blazing fast, slightly ugly, yet it’s feature-rich and web based ;) I’m saying this as e.g. UniFi dashboard can easily spin a fan on my i7… which is ridiculous.

robertpenz · Fri Jun 10, 2022 8:41 am

It should be Web-based and the Server should run also on Linux - we don't have Windows Servers.

pe1chl · Fri Jun 10, 2022 10:58 am

I’m saying this as e.g. UniFi dashboard can easily spin a fan on my i7… which is ridiculous.

UniFi is written in Java. At first that seems attractive as it achieves portability, on the other hand it is a resource hog and lately it has gotten a bad reputation because clueless developers implement attractive modules used by many, which cause nasty security issues. A bit like PHP.
(e.g. they do not understand the difference between system configuration data where you might want to parse ${expression} constructs, and user data where you definitely do not want to do that)

Amm0 · Fri Jun 10, 2022 3:31 pm

Well that's about the best one here:

Many great things has been said here and I sign under them. I will have one suggestion:

Please start small and ascetic.

We don’t want a perfect solution in 10 years.

And clearly a UBNT clone is what no one is looking for.

andrewe02000 · Fri Jun 10, 2022 10:14 pm

Would be nice if it had a tool to migrate configs to different models when exact replacement model is not available. I have found myself doing this quite a bit. Especially on core router upgrades to new core routers. Other than that everything that Ubiquiti's UISP has but not grandmafied. :) Way out there stuff would be things like freq coordination/planning, recommended fixes for common config problems, hell maybe even a integrated openflow controller. Besides all that crazy stuff the basics are best. :) Thanks for doing this.

Larsa · Sat Jun 11, 2022 12:04 am

(e.g. they do not understand the difference between system configuration data where you might want to parse ${expression} constructs, and user data where you definitely do not want to do that)

Yeah, that's why network folks are in general terrible application developers. ;-)

This is because people who engage in low-level network programming often have a completely different mindset and are therefore usually unsuitable for that kind of job. Bottom line, never ever put a network developer in charge of a large application development project.

Sat Jun 11, 2022 10:54 am

Dudes ... let the Dude join the party again just after a liitle bit of funcional workout in a programming gym.

pe1chl · Sat Jun 11, 2022 11:28 am

(e.g. they do not understand the difference between system configuration data where you might want to parse ${expression} constructs, and user data where you definitely do not want to do that)

Yeah, that's why network folks are in general terrible application developers. ;-)

This is because people who engage in low-level network programming often have a completely different mindset and are therefore usually unsuitable for that kind of job. Bottom line, never ever put a network developer in charge of a large application development project.

This is not limited to network folks. The Atlassian system used by MikroTik for the help system and issue tracker were down due to such an issue (waiting for a fixed version).
The system could be hacked using ${expression} constructs. Most likely not even because of a coding error by the Atlassian programmers, but in some useful module they used. The "log4j" vulnerability was another example of this.
I sincerely doubt the sanity of programmers that write stuff like this, and they have to be kept far away from anything exposed to the internet. Which such a Devices Controller likely will be.

PackElend · Sat Jun 11, 2022 3:01 pm

The initial goal is to develop a protocol to apply and monitor config, hence the question about needed features that this protocol should be able to do.

Why that?
Why not simply using the REST API?

mducharme · Sun Jun 12, 2022 12:38 am

noooooooooo NOT THAT...

The nice thing about the UISP design is how the devices "phone home" to the controller instead of the controller needing to reach them, which works great for devices behind some kind of NAT where the controller does not have direct access as well. TR069 can do this too but it is not suitable for uses outside of residential gateway management, it would be very strange to use TR069 to manage a BGP router at the core.

mada3k · Sun Jun 12, 2022 12:35 pm

I think we are talking about two things.
A "device manager" that acts as a hub for CPEs and stuff.
And a conventional NMS that actually monitors and manages a network.

doush · Mon Jun 13, 2022 4:12 pm

I think the best overall controller that I have used till now is AirControl2 by UBNT (shame that they have EOLed it for no reason)
From mass config to scheduled operations, AC2 was a beast.
You can take AC2 as a reference for functionality.

marcperea · Tue Jun 14, 2022 4:30 pm

@mikrotik - I think the idea of a CAPSMAN like protocol to manage lots of Tiks from a single location would be incredibly useful.

I've also noticed several people asking for a web UI that can control, manage and provide remote access to Mikrotiks while also providing backups and config diffs and firmware management, RADIUS user management, historical graphs and charts, as well as bulk configuration.

You should check out https://remotewinbox.com

Disclaimer: I'm part of the RWB team

Tue Jun 14, 2022 7:30 pm

Hello,

MikroTik is planning to develop and build a controller app for MikroTik Devices. Currently we are researching possibilities and options, what should be there and how it could be done and implemented. At the moment we do not want to stick to a specific implementation or standard, but build our own that will help to manage, develop and deploy different scale networks running MikroTik devices.
Any suggestions about features and options are very welcome.

Make a DUDE app. May be modularize Dude so a small installation could use only the device list part. Add configuration management.

WizGirl · Tue Jun 14, 2022 10:55 pm

I read through some of this discussion, maybe someone has mentioned this already, but I would like to throw my idea into the hat here: It would be cool to have the ability to "Stack" Switches or Routers with this utility, eg: Keep configuration files between the stack or maybe HA group in sync (Think firewall rules etc). I feel like this would bring a serious edge to Mikrotik hardware in a business environment.

ferilagi · Fri Jun 17, 2022 2:32 pm

Grouping, etc router device, switch device, AP device, LTE device.

Traffic / device flow like the dude.

madman22 · Sat Jun 18, 2022 8:07 am

I built an app that can provision multiple devices at once, it can do 24 devices in 10 minutes, it takes longer to unbox and plug in the mikrotiks then it does to provision them all. Provisioning includes setting a base config, updating to a selected version, applying the final configuration, and adding it to the "inventory". There are multiple reboots in the process to verify everything. At first I used a custom net-stack that used neighbor discovery without arp so it could connect to multiple devices with the same default 192.168.88.1 address at the same time. Today, I use multiple containers with the default linux net-stack on its own vlan.

Now that 7.4 supports containers again, i'm working on getting my app to work on a RB5009 so a dedicated server is not required.

Here are some things my app does that I would like to see from the mikrotik controller:
provision multiple devices at once
api to get/set settings from a billing system (like queues, port forwards, etc)
update all devices to a given configuration
api/integration to a network monitor similar to The Dude

millenium7 · Mon Jun 20, 2022 5:41 am

Lot of this probably been mentioned already but i'll throw my 2c in

- Cloud based, absolutely. Something that can just reach a known public server out-of-the-box as long as it gets an internet connection. Make it something that runs over port 443 to get through firewalls and not need ANY config deployed on the device. Integrate this natively into the default config of all devices
- Onboarding done solely from the cloud controller. So you can enter multiple variables and adopt the device as soon as its online. Eth1 MAC address, serial number, build date etc. Enter these into the cloud portal and as soon as the device gets internet it will adopt it to your organization
- White labelled and multi tiered so you can have sub domains under a parent and manage other organizations/customers equipment, but let them also manage it themselves without seeing parent domain
- Config backups and diff'ing with alerts
- Device logs grabbed and stored, without needing to setup via syslog
- Log parser to take action or send alerts
- Alerts should include native SLACK support, as well as email, SMS
- Pushing configs, but with scripting/variable support from the controller as well
--- i.e. you can enter /system identity set name="{GROUPNAME} - {DEVICE NAME} ({DEVICE MODEL})"
--- And then when this gets pushed to the device it will rename itself i.e. "TowerRadios - LocationABC East (LHG 60G)"
- Better script handling so it doesn't just abruptly stop if there's an error. Have options on all scripts to specify the action to take, ignore/abort/revert. Report back all errors to the controller
- Baseline compliance templates, as above with scripting and ability to ignore certain parts of the config. I.e. all radio's in GroupABC must be set to country=Australia frequency=auto. Ignore this rule if identity is prefixed with 'LABTEST'
- Per-device Interface compliance states, i.e. ether3 should be 1gb but suddenly dropped to 100mbit. Ether4 should not be running, but now is. Alerts should be triggered. Default state would be ignore for all
- Reporting on the above, not just instantaneously but something that can be scheduled to run hourly/daily/weekly/monthly etc.
- Make a lot of these options drop-down menu's, without requiring scripting knowledge. Simple step-by-step 'if [condition] [equals/greater/less/not/etc] then [take action] (optionally AND/OR to add another statement)'
- LDAP/AD/RADIUS login integration to allow staff access, with permission restrictions
- More granular control than what we have in WinBox. I.e. GroupB can read/write IP addresses but can't view anything related to PPP/Firewall/Routing/etc
- Assignable groups to folders/groups of devices for alerting only to staff responsible for those devices
- Speed test, able to pick devices internally, or do a speed test to cloud controller to test internet speed

bpwl · Wed Jun 22, 2022 11:34 am

Interesting idea. Actually the current experience is already quite good.

Managing nearly 100 MT routers at a very remote location, can be done.

What is possible today ...

-using a hEX with DUDE, the following is done.
- the hEX connects VPN to the access gate @ control site for remote access
- hEX monitors all MT (syslog + DUDE)
- DUDE as distribution point for upgrade npk
- recover/reconnect of all MT through MAC Telnet, MAC SSH (even after somebody doing factory reset)
- RoMon

What is missing
- simple multi MT remote command (e.g. repeated application of Telnet/SSH string to different MT routers)
- Netinstall (TFTP/PXE-boot server) on hEX/Dude
- "Reboot trigger" command
- Wake-on-LAN on MT devices (now faked with PoE)

Xhat would be nice. : CLI command to execute some CLI command on another ROS device. (e.g. RADIUS based wifi login, with limits, creates the queue's on internet facing ROS router. The rate limits are mostly intended for the internet connection, not the local LAN rate)

millenium7 · Thu Jun 23, 2022 12:55 am

Interesting idea. Actually the current experience is already quite good.

Managing nearly 100 MT routers at a very remote location, can be done.

Can be done yes, but could be done immensely better with a central cloud controller
It's not just about number of devices either

Relatively simple things like config compliance open up a whole new world that enormously simplifies things (if implemented properly) - such as having address lists for remote management and then a change occurs - you need to add or remove an entry
Whilst yes you can log in to all of them and change it, it's not a great way to go about it. Takes a lot of time and is prone to human error. You can use a SSH pusher but its error prone in its operation, and importantly it doesn't tell you if the existing config is correct, you just have to assume that it is or double check yourself manually
Config change templates allow you to know for certain what is configured correctly and what isn't, and then take action to push a mass change to all devices. And again very importantly - to verify it actually did change properly

Any time a process is manual it gets exponentially more likely you will forget about something along the way (maybe you get half way through and then interrupted for the rest of the day). Having a cloud controller is not just about managing more devices, its about managing them correctly with far better reliability and less mistakes

bpwl · Thu Jun 23, 2022 9:36 am

it doesn't tell you if the existing config is correct,

That would be interesting for sure. But might be difficult to implement. What is a correct config?
Today one is not sure ROS will act as expected. "Toruble" shooting can take some time, as there are so many settings, and so many things that impact the behaviour in an unexpected way.

Story ... use a ROS device as WAN connection with load balancing, one of the WAN connections has a reduced MTU size due to several encapsulations in the uplink (VPN, IPsec NAT traversal, sattellite link, etc etc.). To optimise this, reducing the ethernet MTU size to 1400 improves the data flow.
The satellite gets removed, and the now free ethernet port is added to the bridge, to be used as LAN port.

And some time later the smartphones and PC's on wifi claim "persistent server problem", for the ISP's mail server, synchronised with imap protocol. The flow is not changed and not over that mentioned ethernet port. Browsing is OK, no problem. But no imap mail sync, and no identification/authentication delegation with https://www.itsme-id.com/ .
The 'root' cause was the old reduced MTU size, that propagated to the bridge (after reboot ???).

Being able to synchronise (parts of) the config between some ROS devices would be very wellcome. Start of a HA solution? Cold standby, hot standby, even needs more than this.

millenium7 · Thu Jun 23, 2022 12:14 pm

it doesn't tell you if the existing config is correct,
That would be interesting for sure. But might be difficult to implement. What is a correct config?
Today one is not sure ROS will act as expected. "Toruble" shooting can take some time, as there are so many settings, and so many things that impact the behaviour in an unexpected way.

We use Solarwinds NCM which is good as a general all-rounder. But MikroTik could make it enormously better/easier/simpler if it were tailored specifically for MikroTik
I use RegEx expressions and it simply looks for matching lines in the config. So it's easy to see if every requested line exists in an address list, and no additional ones. And if it doesn't match then it'll flag it in a report and it can be rectified (can also push a config change automatically that will simply redo the address list)

At the moment in NCM it's fairly simple but it isn't implemented quite as logically laid out as it should be. MikroTik could massively improve the usability and give you a step by step system with drop-down menu's catered for config sections, not just using RegEx expressions and SSH scripts

NCM there is a lot more care required to make sure - as you say - you don't screw up some config if you push a script to fix a problem. Because it's going to push whatever script you type in (and it also might fail part-way through or disconnect)
I'd expect a MikroTik system to let you simply tell it how something should be in laymans terms, and it'll automatically handle the config change properly

mekmek · Wed Jun 29, 2022 10:06 pm

- no vendor lockout
- topology view (incl. STP)
- examples and guides
- configuration for home, hotel, school, events, company network, home office,…
- configuration for standard and special settings like: homelab, tor, BGP, VPN Providers, …
- DHCP options for other device
- name convention with official city codes like: https://service.unece.org/trade/locode/lv.htm
- best practice / security analyzer
- remote checks
- multisite compatible
- may multitenant
- global and custom variables for network objects like $VoipNetwork{All}Offices, $PBX, $DNS1, DNS2,$NTP
- dynamic / rule based variables like: $VoipNetwork{ThisLocal}{Office}
- central firewall rules,
- central blocklist (selfhosted)
- blocklist shared with other user or closed groups
- modules/ plugins/ examples from mikrotik, community, vendors,…
- autoconfig for devices
- dedicated config port, device connected to this port get preset local config
- configuration over the internet
- beeper alarm, when device is disconnected or fees not payed
- working time tracking with wifi device
- pooping time tracking needs an expensive special ap and subscription …
- labeling templates
- optional DIN-Rail cases
- direct access to (external | public) support/ sales/ seller contact adresses
- offer service for other user
- temporary access to a deputy
- deputy solution for one man it
- external Identidy Provider
- 2FA
- rating feature
- traffic monitoring
- detection of suspection traffic
- campare to similar setups
- automatic dokumentation
- fancy reports for the managers
- automatic replace reminder for old devices
- issue sharing/ warning

TomosRider · Fri Jul 01, 2022 10:44 am

Good news!
We have been using MT devices across our network for a long time and planning to do so in the future.
Implementing one solution that will help to update and backup configurations would be great.
Also, integrated netinstall feature(if possible) would solve so many unnecessary on site travels...:D

ubikrotik · Sat Jul 02, 2022 6:48 am

Rock on TIK!

we need:

- a list of router, one click access to open winbox
- graphs
- mass update
- automatic backup schedule
- local host in a VM
- mass configuration option. I.E : changing the value of a string for selected routers. Let say you want to change check IPSOCKS on 380 routers, just select routers in a list, and then send the script.
-

mducharme · Tue Jul 05, 2022 3:13 am

I think the general idea of TR069 is a good one, where the device "phones home" to be told what to do. This works great with devices behind NAT etc as only the server needs to be publicly accessible. Ideally the device would be configured with a URL for the server that it could get all of its config from with just port 443 open, so that you can easily put a device at any customer premises even behind existing firewalls and still have management of the device. This is the way Ubiquiti UISP works. I don't like much about UISP, but I do like the decision to design it in this way.

TR069 itself however is ill-suited to managing anything other than CPE devices. The ACS systems are really only intended for CPEs as that is what the protocol is meant for, so it would be very strange to manage something like a BGP edge router for a big ISP with TR069. I am already managing our CPE devices through GenieACS but could use something like this for everything that is not a CPE device.

shiju22 · Thu Jul 07, 2022 4:33 am

its been ages is there anything going
some thing which have everything hat dude have and also the flexibility to install any on-premise system and can at-lest show snmp data from other network equipment too
wich can give total network usage of any interface and performance graph of cpu, ram, flash, interfaces......etc
and most important have a web GUI which can be accessed anywhere the one think wich dude use to have and then removed in later vertion

mjezierski · Thu Jul 07, 2022 6:47 pm

Everything in Winbox + TR-069 in one on-prem solution would be killer in my use.

eduplant · Thu Jul 14, 2022 6:35 am

I would vastly prefer development effort be put towards making RouterOS 7 more automation-friendly rather than towards making another single-pane-of-glass management solution. I certainly understand the urge to do so, especially for single-vendor networks that want something that "just works".

Nonetheless, I think Mikrotik's strengths are in the form of innovative and cost effective hardware platforms, a fairly well-designed configuration paradigm, and that the configuration language is a domain-specific language that lends itself to scripting. If there is a desire to sell a management solution, Mikrotik should lean in to those strengths and do most of the work on the router side rather than the controller side.

One big example I can think of: give RouterOS 7 proper configuration transactions. The RouterOS configuration paradigm is already very database-like and, in my opinion, is well-designed compared to some of the big players like Cisco. A huge thing missing for automating Mikrotiks is the lack of native transaction support: the ability to stage a series of changes, commit them in their entirety, and if necessary roll them back. Work spent improving this would make the platform more attractive for all users, no matter whether they manually configure their devices with Winbox/web/CLI or whether they integrate them with an automation tool like Ansible.

Another big thing is configuration replacements. Since the strategy of most automation frameworks is to declaratively state the running state of the device now and then apply changes to get the device to a new state, the native ability to make these kinds of transformations would make RouterOS a lot more friendly to automate. For example, if I want the entire /ip/firewall tree to be different, either the script I'm writing or the automation platform I'm using has to compute the individual changes that are required to, say, add two address-lists, remove three other address-lists, add 4 new chains, modify the existing chain by a certain name, etc. etc. etc. and issue the commands to just change those pieces. All of this additional headache can be alleviated if RouterOS can be presented with a replacement hierarchy that should simply replace that whole subtree of the configuration and make it match after the transaction is committed.

Pretty much all of the network vendors (except JunOS frankly) have one or more of these exact problems and it is certainly a non-trivial effort to make big changes to the core configuration model. However, if you make these types of improvements on the RouterOS side, if you still want to implement an automation/controller/management kind of solution your own engineers won't have to invent something with duct tape and glue on the controller side to keep things in sync or try and generate the API diffs under the hood. Instead, the resulting application can be a pretty basic web app that leverages a simple database or, say, Ansible and git under the hood. You could even partner with an existing automation platform and just make some good RouterOS integrations for it.

Thanks for reaching out for input and I hope you consider my case for where best to apply your development time and money in service of RouterOS mass management.

eduplant · Thu Jul 14, 2022 6:40 am

1) Doing a controller without the ability to have idempotent commit is a fool's errand, and will only end in tears, both for the developers and users. Fix that first, the rest becomes MUCH easier.

+100 on this and the rest of your suggestions.

killersoft · Sat Jul 23, 2022 3:31 am

In regards to the 'Cloud' solution.
Not everything I have in now >100 devices touches the public internet.
I would prefer a solution I can spin up on a Virtual Machine in a closed environment.

I understand that other people could benefit from a cloud controller, but not in my current use case.

nkourtzis · Tue Jul 26, 2022 8:03 pm

I just saw the thread and it is GREAT news! What I would like to see:

Controller accessible via both modern, sleek web UI AND familiar winbox, with different menus than those of regular devices

Self-hosted (with support for all major hypervisors and docker) AND cloud-based, as-a-service, provided by Mikrotik at a reasonable cost

Support for controller redundancy for high-availability

Multisite AND multitenant

Versatile device grouping by locations, tags, types, organizational units or any other attribute

Rule-based and manual device tagging/ group assignment

Automatic device discovery, adoption and provisioning

Fully customizable lists with nice and informative visual elements

Configuration via both GUI AND programmatically, with support for predefined AND user-definced variables/data structures

Advanced configuration management functions (versions, comparison, visual block-based editing with ability to copy specific sections and sub-sections by dragging and dropping blocks from one config file to another etc.)

Configuration templates with manual and automatic (based on triggers and criteria) provisioning on devices

Manual and automatic device firmware upgrades, on groups or individually

Support for devices hierarchy and dependencies (e.g. upgrade downstream device first, then upstream)

Manual and automated backups with selectable retention

Comprehensive builder for statistics and reports, exposed via the API for use in other platforms (e.g. PowerBI, Graphana, etc.)

Topology mapping capability with automatic detection where possible (using wireless link clients/ L2 discovery etc.)

Controller support for users, user roles and user groups

Controller support for external authentication providers and 2FA

Support for automatic IP blocklist import, for use on the controller and the managed devices (via firewall rules)

Full LDAP and radius functionality

Funtions fully accessible via a RESTful API

Open architecture with active encouragment of plugin development

Ability to use a Mikrotik router as an appliance to run the controller

Specific appliance for the controller (like Mikrotik Cloudkey, with battery backup, info screen etc.)

It is a long but comprehensive list. Thank you for the great work!

kevinds · Wed Jul 27, 2022 10:59 am

The ability for controller and RouterOS to connect when they are both behind (different) NAT networks.

The ability to work on "air-gapped" networks.

See changes made over time.

CoyoGross · Tue Aug 02, 2022 4:12 am

All Dudes vibes, and better! 😉

ChristianRiesen · Wed Aug 03, 2022 3:06 pm

* Web based UI, focused on desktop/tablet, but still be able to get useful info out of it in phone view
* Run it in a docker container, easiest way for people to just grab and run with it. Make sure to use some sensible port for it (not port 80) to not clash right off the bat. Ability to add a certificate to run it with https enabled.

ffries · Sun Aug 07, 2022 11:17 pm

European EID / OpenSC support.for.authentication.
Two factor authentication
I don’t want all my eggs in the same basket if this is not secure.
Open source and open standards eunning in containers and GNU/Linux
Peer review and certification of code by public agencies

volga629 · Mon Aug 08, 2022 6:02 am

I have seen several mentions of config files, config compare ...
Do you suggest for the controller to operate as a configuration export uploader?

That you have representation of running and start up configuration. Where are you can see the differences.

pe1chl · Mon Aug 08, 2022 11:07 am

Yes that would be nice to have for "the general public".
I have this function for ages as I export my configs to a locally hosted git versioning system with the additional "gitweb" program that shows output similar to the above via a web interface (only accessible locally).
Very useful to be able to look back at what you changed before. Also useful when having more than one admin.

slackR · Thu Aug 11, 2022 3:21 am

Here are some suggestions.

-Configuration management, device policy groups, etc.

-Change management, authorized or approved changes, scheduled changes and a diff change log.

-Event based notifications channel. Like MQTT to trigger alerts and collect device metrics.

-SDWAN auto-deployment, failover.

-Deploy containers

-Schedule firmware upgrades with auto rollback on failure

-Configure single-sign-on authentication for VPNs and wireless using RADIUS or LDAP in a wizard.

I'm assuming this controller would have a web portal with reporting and live metrics.

Cha0s · Thu Aug 11, 2022 8:13 pm

Yes that would be nice to have for "the general public".
I have this function for ages as I export my configs to a locally hosted git versioning system with the additional "gitweb" program that shows output similar to the above via a web interface (only accessible locally).
Very useful to be able to look back at what you changed before. Also useful when having more than one admin.

For anyone interested in a solution like this, there's Oxidized.
https://github.com/ytti/oxidized

rviteri · Mon Aug 15, 2022 2:04 am

A Script manager so an SD-WAN solution can be implemented. Web/Winbox interface.

rextended · Wed Aug 17, 2022 4:29 pm

demo?

AHAHAHAHAHAAHAHHAH!!!!!!!!!!!

Sorry....

pe1chl · Wed Aug 17, 2022 7:07 pm

It will probably be released right after the scripting function library :-) :-)

infabo · Thu Aug 18, 2022 7:41 pm

This thread is for brainstorming. Throw in some buzzwords that will end in a tin.

pe1chl · Thu Aug 18, 2022 8:56 pm

Check this topic and what came of it: viewtopic.php?t=131692
I mean, brainstorming is alright but when you do it as a company you may get some expectations...

infabo · Thu Aug 18, 2022 10:18 pm

Can you summarize what was added to the function library from the long topic?

pe1chl · Thu Aug 18, 2022 10:35 pm

Nothing at all. The function library never came into existance, 4 years of discussion going to waste.

rextended · Fri Aug 19, 2022 1:07 am

Is not all correct, on 7 is added the random number generator...

infabo · Fri Aug 19, 2022 10:03 am

Nothing at all. The function library never came into existance, 4 years of discussion going to waste.

Basically what I said.

pe1chl · Fri Aug 19, 2022 10:21 am

Is not all correct, on 7 is added the random number generator... :| :| :|

The way I understood "function library" there would be an optional package that you could install, or some script file you could download to the router and call from your scripts, containing a collection of utility functions as a layer on top of (i.e. written in) the scripting language.
Basically anyone can write that in the latter form, and some people indeed did so.
But there never was an official MikroTik supported version, not even an endorsed one.
Of course in many cases it would be much more efficient (or it would even be the only practical way...) to have new functions available as built-in functions in the scripting language itself. I would not call that a library, but still it would be welcome.

senseivita · Fri Aug 19, 2022 11:29 am

I think I might be just Mikrotik's target audience, and I think I'd thread carefully in its place.

Something like the UniFi Controlller is pretty to look at but it ain't very useful. It's slow, it's got so many problems with from adoption, to disconnections, to being unable to handle consecutive (not "too many-", just ·"consecutives") updates, all of which Mikrotik doesn't have plus accessibility issues for installers (that thing when you come down the white bright rooftop into an air conditioned relatively dark server room and you try to read fonts too thin) which are a staple for this mgmt utils. I'd document better instead. The documentation is written for the CLI, but the CLI isn't what's encouraged to use, there's this myriad of admin UIs but only documentation for the one without any graphics, often with graphics/screenshots/suggestions for the other ones. It's also in a needlessly technical language most of the time but it's not technical enough where it matters so there are no ambiguities--there are way too many of those. Sure it's hard to contemplate all possibilities in computer networking being nearly endless, but it can be done because it's been done, there's another vendor who's managed.

I'm referring to pfSense. If I had to improve Mikrotik's <insert>, I'd take a look at what used to be called The pfSense Book (now just its documentation) for guidance. It does an outstanding job of explaining why things work as they do and even why decisions of the UI were taken in some instances addressing straight on their shortcomings.

You don't need to dumb things down the way Apple, Ubiquiti, Cloudflare, (..) and others do, it's kind of infuriating when yet another vendor discovers minimalism and your settings are gone. This is how we end up with these annoying certificate warnings in every browser on non-routable addresses, if we were taught, we probably wouldn't be needing to be treated like children, I'm sugar coating there, and end up with lack of options, and expensive, limited and cumbersome controller device or alternatively "management as a service", what UniFi turned into. UniFi still can't multi-WAN properly, basics like DHCP reservations are kind of there, it's over a decade old. Also there is telemetry; every device, even outdoor wireless radios are consistently trying to contact external web, STUN servers, IP addresses in China, were Ubiquiti hosts some of its UNMS or whatever it's called lately. The forum, is mostly complaints now and not a link easy to find anymore, maybe it's related.

Ubiquiti for years have been trying to get rid of support for "legacy" (Wi-Fi 4) devices and has deleted the needed apps to access the old self-hosted NVRs, another form of controller. People got angry because you could only access your cameras if you maintained a Play Store, Apple ID account in which you got the apps earlier before or trust the company won't have another change of heart and upgrade. As for the APs, they'd become unmanageable just because they didn't feel like maintaining that support, which wouldn't be required if the APs had a built-in admin UI like Mikotik's do. It seems they reversed that somewhat, you can still manage the APs in the latest controller but not group them to newer ones. It was too late though, a lot customers called it quits.

Focus on documentation, build-it right in Winbox or one of these great UIs you already have which are as powerful as you know how to make them do things, make documentation offline, not links to a wiki which aren't helpful when you're setting up a device--when you need it the most. Finding the default IP management address shouldn't take half an hour. Things like Mikrotik "Home" or whatever dumb things down way too hard and don't provide a learning/evolving path to follow. For those of us coming with advanced, already set up networks on platforms a little more straightforward, reaching the level of knowledge necessary to deploy the same infrastructure can be cost-prohibitive in terms of time/downtime. The first time I tried a Mikrotik router, I ended up returning it because it was going to take way too much time to set up. The last time I tried Mikrotik (it's been like 4 or 5) I couldn't find how to set up full cone NAT on a dynamic IP interface without first learning pretty high level scripting.

suhl · Fri Aug 19, 2022 5:00 pm

Do you think it would be possible to get something similar to Cisco High Availability?
https://www.cisco.com/c/en/us/td/docs/r ... ility.html
There's this other Mikrotik community project I found here:
https://github.com/svlsResearch/ha-mikrotik
Basic idea is to be able to change the configuration in one place and all the routers/switches in the HA group will be affected by that change. Versioning would help a lot as well.
Good luck. I love MikroTik from the early beginnings in the late '90. You are destined for a big success with your approach.

realmark · Tue Aug 30, 2022 3:46 pm

Native support for push metrics / streaming telemetry!

Support for pushing data to influxdb or similar. We've moved away from "network monitoring" tools towards grafana dashboards for all server monitoring, firewalls, and are attempting to do the same on routers/switches. No SNMP inbound, proxies, agents, etc. Just a clean feed of desired details streamed off to a target of choice.

viewtopic.php?p=948888&hilit=influxdb#p948888

spippan · Fri Sep 02, 2022 2:47 am

My number one suggestion and highest priority is to build in strong security from the start, not as an afterthought. MikroTik could show the rest of the network equipment industry how to establish best practices for securely maintaining network devices, and that should be the goal!

Some suggested approaches:

Incorporate a robust management facility for establishing and maintaining administrator and user accounts. Ideally, this should also support "machine" accounts that could be used for automated status queries and management of devices. The controller itself should use a "machine" account for its purposes, and this account must be customizable by the customers. Integration with enterprise systems like Microsoft's Active Directory would likely be desirable for some customers.

Build in a Certificate Authority (CA) for issuing certs to network devices. The new controller should incorporate hardware protection for private keys (especially CA's own private keys), along with the ability to securely clone the CA's private keys to a backup controller using multi-party controls. Network devices should be able to request renewal of certs from the CA using automated methods. There should also be automated tools for installing certs in network devices. Certificate revocation must be supported using a dynamic protocol like OCSP with ability to push out revocations immediately (e.g., via CRL update). An optional approach could be to support integration with a third-party Certificate Issuing/Management system, but these days the tools to implement the subset of services required for network devices are readily available from multiple Open Source projects, including OpenSSL itself.

Use CA in new controller to also issue client-side certs for network administrators. Client side certs would be used with mutual authentication to handle logins to Winbox and other device-specific services, including an option for providing SSH keys via certs. Automated client cert renewal should be supported, and it must be possible to revoke client certs with immediate pushing of revocation notices to devices along with dynamic cert checking. (Aside: WinBox might directly support requesting administrator certs from new controller's CA.)

CA should also issue user certs for Wireless access, PPP/VPN remote access, HotSpot services, etc. This implies that there should be specialized access to the controller from users to handle cert requests or update their account details, such as email addresses, phone numbers, workstation details, mobile device details, etc. In an enterprise environment, it might be possible to pull this sort of information from a central service.

Fully support the latest cryptographic algorithms and measures, including the widely accepted elliptic curve algorithms (e.g., ED25519). Provide policy controls to limit/restrict use of cryptographic suites in the network devices from a network-wide perspective.

Provide a complete implementation of a robust RADIUS service for legacy devices and services. For extra credit, support RADIUS integration with Microsoft AD NPS/RADIUS facilities.

Implement an SSH key management system that would support pushing administrators' SSH public keys to network devices and rolling keys as appropriate. Immediate removal or disabling of SSH public keys for administrator login is also necessary. One possibility would be to use SSH and SSH key management to handle securely pushing updates to devices, along with invoking of scripts and automated retrieval of device information, including device configurations.

Provide an encrypted storage system for maintaining sensitive information at rest, especially for device configurations and any other sensitive information.

Build in a software repository for redistributing RouterOS (and possibly other software/firmware packages) to network devices in a controlled manner without requiring that individual network devices have access to the Internet. This could be an adjunct to the requests from others on this Post for RouterOS bulk updates. Ideally, this system should support two or more storage partitions on devices that support this option to make it easier and safer to rollback an update. For devices that are not equipped (or configured) with multiple partitions, a rollback facility would still be a valuable capability.

Implement support for redundant device deployments, including for the new controllers. For example, support measures to independently update RouterOS in each member of a redundant device pair thereby allowing the other member to maintain services during the upgrade. This capability could also allow staging of firmware in redundant systems to confirm stability before completely updating all devices. Similar capabilities would also be necessary for redundant controllers. Resilience is an often-overlooked essential security requirement.

Support RANCID or an equivalent service for maintaining network device configurations in a source control system (e.g., Git). This could be an add-on package for users dealing with larger networks or complex support requirements. (Aside, my own experience using RANCID with a complex network involving devices from multiple vendors illustrated that this is an invaluable tool for not only tracking configuration changes, but also monitoring changes made by multiple administrators, which in turn provides further security controls with the added ability to recover from unapproved or ill-conceived changes.)

Support management of security credentials for SNMPv3, including the ability to update credentials periodically in a controlled and automated manner. Provide methods for pushing SNMPv3 credentials to network management systems (e.g., via secure upload of an exported dictionary of credentials).

Provide tools for automated responses to DDoS attacks using parameter-driven approaches for invoking mitigation measures.

Implement a comprehensive system logging facility. This could be optimized for MikroTik devices to leverage enhanced features. The system logging should support TCP logging, as well as optional support for logging via encrypted links (SSH, IPsec or other VPN). It should be feasible for customers to implement redundant syslog servers for resilience as well as protecting logs from being modified by attackers. The logging system should be capable of relaying log records to more advanced enterprise-oriented logging systems (e.g., Elastic Search).

Since DNS is one of the most essential services and also one of the most sensitive from a security perspective, centralized management of DNS services in network devices would be a valuable service. This could include the ability to maintain static DNS caches across some or all network devices to improve availability of essential DNS resolution during periods of degraded operations, such as network outages or partitioning.

Provide robust NTP services, ideally supporting authenticated access. The new controller would ideally provide an option for GPS time sync so that it could operate as a Tier 1 NTP server. This would also be an underlying security facility for supporting certificate management and use of time-based authentication services.

Yes, this is a lot. However, everything listed above is readily available and supported in the Open Source realm. What is important is to build these capabilities into the product plan, and build other controller features and capabilities on top of a secure base. Not everything needs to be in version 1, but everything (and more) needs to be in the product plan and resulting design. Security is just too important an issue these days to not be the primary objective for anything that purports to control network devices and maintain a network system.

perfect.

spippan · Sun Sep 04, 2022 10:20 pm

Yes that would be nice to have for "the general public".
I have this function for ages as I export my configs to a locally hosted git versioning system with the additional "gitweb" program that shows output similar to the above via a web interface (only accessible locally).
Very useful to be able to look back at what you changed before. Also useful when having more than one admin.
For anyone interested in a solution like this, there's Oxidized.
https://github.com/ytti/oxidized

...can also be coupled with, e.g. LibreNMS
had 2 of those running with oxidized (which pushed config versions to a local git repo)

lcohen999 · Fri Sep 09, 2022 7:01 pm

This would be fantastic

It would be nice to :

get a Tik online, register with my account, then I can push any configurations I like (IPSEC/LAN, etc) or "copy from another mikrotik"

Have it run in Mikrotik's cloud, or make it open source so I can spin up my own linux server and use that instead.

Past that, the usual stuff, remove management behind NAT, push packages, etc.

We are a small business with 150 units. I would love to one click push a ROS and Firmware update. Push wifiwave2 as needed, whatever.

Really though, at minimum, remote management behind NAT is the biggest PITA at the moment

spippan · Fri Sep 09, 2022 8:47 pm

Really though, at minimum, remote management behind NAT is the biggest PITA at the moment

you could setup a public CHR to which your to-be-managed tiks would connect via ovpn/sstp/wireguard (with a /30 subnet for instance or framed /32), establish a routing-protocol for automatic route exchange with route filters in place so every device has its own Lo0 address and you also connect to that CHR and route your management subnet (in which the Lo0 address reside) to that CHR

so NAT is no problem any more in fact. and with sstp/ovpn on tcp port 443 it even would be possible to maybe deploy tiks in china xD ;)

realmark · Fri Sep 09, 2022 11:46 pm

Following up, the devices should reach out to the controller, not the other way around. Push metrics to the controller is a good start. Controller keeps a git (or other version control database) for configs, and endpoint devices pull the latest config.

Yes, you could still potentially compromise the controller and use it to deliver compromised configs, but the controller has no inherent path to the sites. And in metrics-only mode (config pull disabled / manual on the device), there's zero path in. We're using this sort of strategy with a number of clients for whom data security is paramount. We provide remote monitoring of systems, but have provably zero access to the data within. A breach of our systems cannot get back to the customer, but we still provide a lot of value from metrics monitoring and analysis.

I know this sorta gets into the user facing end of things, but Aruba is doing a nice job of the config part with NetEdit. Other guys keep mentioning RANCID. This is the opposite flow. Config builder / git repo at the head end, and network devices pull their configs down.

lcohen999 · Mon Sep 12, 2022 5:40 pm

Really though, at minimum, remote management behind NAT is the biggest PITA at the moment
you could setup a public CHR to which your to-be-managed tiks would connect via ovpn/sstp/wireguard (with a /30 subnet for instance or framed /32), establish a routing-protocol for automatic route exchange with route filters in place so every device has its own Lo0 address and you also connect to that CHR and route your management subnet (in which the Lo0 address reside) to that CHR

so NAT is no problem any more in fact. and with sstp/ovpn on tcp port 443 it even would be possible to maybe deploy tiks in china xD ;)

I can also zerotier each router with my own hosted ZT and access it that way. Point being...it is an extra step which would be nice to avoid

robins · Sun Sep 18, 2022 12:32 pm

We are a large 802.11 WISP with a managed BYOD wireless client service. We are in a phase of transitioning heavily to MikroTik products for our backhaul. We are interested in your CAP access points to install in homes and businesses, but your CAPsMAN controller currently has an issue which is fatal to our use case:

If the controlled AP loses contact with the CAPsMAN controller, it kills the radio and stops broadcasting entirely. This will suffice for local CAPsMAN management, but we require central management of thousands of separate sites (multi-tenancy). In order to do that with your controller with uninterrupted failover, we would need to run VRRP and script an automated sync. All our last-mile wifi services would be at unacceptable risk of total catastrophic failure. Our current vendors' APs cache their config and continue broadcasting with its last known settings if it loses access to a central controller.

Is there an effort to add a feature like this for multi-tenancy and non-distributed CAPsMAN control?

mjezierski · Mon Sep 19, 2022 6:00 pm

In regards to the 'Cloud' solution.
Not everything I have in now >100 devices touches the public internet.
I would prefer a solution I can spin up on a Virtual Machine in a closed environment.

I understand that other people could benefit from a cloud controller, but not in my current use case.

^^^^^^^^^^^^^^^^^^^^^^^
THIS!!!!!

nkourtzis · Thu Sep 22, 2022 6:55 pm

In regards to the 'Cloud' solution.
Not everything I have in now >100 devices touches the public internet.
I would prefer a solution I can spin up on a Virtual Machine in a closed environment.

I understand that other people could benefit from a cloud controller, but not in my current use case.
^^^^^^^^^^^^^^^^^^^^^^^
THIS!!!!!

...Or the best of both worlds: a cloud-based service, with the option for a local "proxy controller", located on the LAN edge.

Cha0s · Thu Sep 22, 2022 8:41 pm

...Or the best of both worlds: a cloud-based service, with the option for a local "proxy controller", located on the LAN edge.

That's worse. You need an extra point of failure ("proxy controller" service) plus the need to rely on 3rd party cloud services (be it MikroTik, the cloud provider they choose, and everyone in between).
Just terrible for operators that will rely heavily on the controller. For the average Joe that has 2-3 APs and 1-2 routers/switches on their home or their SOHO business, the cloud solution might be good enough.

The best approach is what Ubiquiti does. Give the option to either use their cloud service or allow end users/integrators to install it on-premises and be 100% air-gapped.
Not every network has direct or indirect (ie proxy) access to the Internet.