Feature request: /tool fetch HTTP-POST can send a file

vanikcz · Wed Jan 05, 2022 9:40 am

Dear Mikrotik forums,
it would be great if command

/tool/fetch http-method=post url="https://blabla.server/fileservice" mode=https

can specify the file on mikrotik that will be send as data of POST request.

Better without that annoying 4kB limitation!

May I believe in that it will be implemented?

hkusulja · Mon Jan 24, 2022 10:16 pm

I also hope this feature will be implemented.
So, I can write a script which will upload my backup files and /export to remote https server...

msatter · Mon Jan 24, 2022 10:25 pm

Better without that annoying 4kB 63KB limitation!

Also:

:set $data "up to 63KB..."
/tool fetch http-method=post http-content-type="application/json" http-data=$data url="https://testserver.lv/index.php"

https://help.mikrotik.com/docs/display/ROS/Fetch

Then, for downloading "chunking" can be used when the server supports it. I don't know if "chunking" is also supported for posting.

viewtopic.php?f=9&t=177530

hkusulja · Tue Jan 25, 2022 1:22 am

Can you please help how to read file content (.backup and .rsc) from local mikrotik storage to be uploaded using $data to https remote server?
I do not see how can you use chunks for upload / http PUT method.
Can you please reference documentation stating that $data is limited do 63 KB in RouterOS 7 ?

msatter · Tue Jan 25, 2022 6:58 pm

Mikrotik did not implement a way to read files larger than 63KB. Writing bigger files is possible with the execute command.

I don't think it is documented but the 63KB for variables is reality in ROS 6+7.

A few months ago no one knew that chunking was possible in combination with variable in ROS and now it used widely.

vanikcz · Mon Jan 31, 2022 12:49 am

Dear msatter,
please canyou specify how can I load a 60kB file in variable? I cant find any example of that...

A few months ago no one knew that chunking was possible in combination with variable in ROS and now it used widely.

Thank you

hkusulja · Thu Feb 10, 2022 8:19 am

@msatter I am also interested and did not find a documented way for reading the file into variable, on RouterOS 7 when file is bigger then 10 KB but smaller then 60 KB.

hkusulja · Fri Feb 25, 2022 10:19 am

I have made a test:

:global $data "test";
:global $url "https://prod-51.westeurope.logic.azure.com:443/workflows/blabla/triggers/manual/paths/invoke....";
/tool fetch mode=https http-method=put http-data=$data url=$url

and this is recieved as, which seems kind of fine

{
    "headers": {
        "User-Agent": "Mikrotik/7.x,Fetch",
        "Content-Type": "application/x-www-form-urlencoded"
    },
    "body": {
        "$content-type": "application/x-www-form-urlencoded",
        "$content": "dGVzdA==",
        "$formdata": [
            {
                "key": "test",
                "value": ""
            }
        ]
    }
}

However, if I try to load file content into variable (file is about 7 KB)

/export file=export.rsc
:global data [/file get [/file find name=export.rsc] contents];

and after http fetch , there is no body data received on http server.

How to send a file content? also, for export and for backup files using HTTP ?

msatter · Fri Feb 25, 2022 2:04 pm

It is the first code example in this tread: viewtopic.php?f=9&t=177530

hkusulja · Fri Feb 25, 2022 2:32 pm

It is the first code example in this tread: viewtopic.php?f=9&t=177530

this is example for "file download" and using chunks.
My request is to "upload file", which is not bigger then 64 KB.

Amm0 · Thu Mar 10, 2022 7:22 pm

The key detail is what on the other end accepts. I'll presume your referring to `curl` that supports a "-F" for forms and supports a "@<filename>" to include a file. This depends on HTML5's <... input=file> within some <form enctype=...>, and enctype can be "multipart/form-data" or "x-www-urlencoded" ( curl defaults to one, depending on version and content-type). So those are TWO types of file upload, and other ways too.

But unlike `curl`,

/tool/fetch

does nothing to help with the data= part. AFAIK there is not a script I've seen that build the needed MIME nor encodes a /file. And, if it's binary file, even a script may not be able to access the contents of the file to send nor encode it without more scripting. For text files, it might be possible to build the needed data= for a file upload, but not easy.

Basically RouterOS doesn't have any primitives to build or parse any of the HTTP content-types. It will happily send a content-type and whatever data you build into a script "str" type. But ain't going to help with the data at all today. Currently, you can't even encode or decode JSON to RouterOS array, which is even bigger limitation IMO. This is why you need scripts for JSON, or rextended's "chunk download", to workaround to the lack of support dealing with the HTTP data.

Maybe someone else know of a workaround to form data and encoding... but you may want to look at other ways of transfer files than "HTTP upload". You could turn this around and have the REST API or SFTP on the webserver and pull the file based on an HTTP request.

hkusulja · Fri Mar 11, 2022 9:44 am

Doing SFTP or other things is not possible due to other reasons in my case. I am aware of this options.
Please, lets focus here on how to anyhow send any file from mikrotik to remote HTTP server. (not vice versa, etc.)

The other end is HTTP web server, working with standard HTTP stuff. I do have control on that side and I can program whatever is needed. For now I have setup HTTP REST APi endpoint accepting anything / anyhow, just to get somehow the file from mikrotik.
Regarding content type, it can be setup manually via /tool fetch as I can see, but I can also fix that on server side.

I do not need manipulation with JSON on routeros side anyhow.

So, what exactly is RouterOS limitation for /tool fetch working with http post for sending file? I think is just documentation lacking but, could be worse / not implemented features, which exactly ones?

Amm0 · Fri Mar 11, 2022 2:10 pm

Please, lets focus here on how to anyhow send any file from mikrotik to remote HTTP server. (not vice versa, etc.)

You can't.

Mikrotik will need to add support for it.

Since they have a backlog of bugs and other features, you may be waiting a while. Essentially /tool/fetch only lets you send a string (but the string can be what ever you want). RouterOS doesn't have any helpers to encode a form submission (of which a file is a type).

And, RouterOS script can only read files of 4K or less into a string, so this prevent more cleaver solution in scripting. Basically, this would be need to work, as a first step of many, to be able to send any data from a file:

:global filedata [/file get myfile contents]
:put $filedata

hkusulja · Wed Apr 06, 2022 11:08 am

@Amm0
https://help.mikrotik.com/docs/display/ ... -Variables
> Note: Variable value size is limited to 4096bytes

https://help.mikrotik.com/docs/display/ROS/Fetch

/export file=export.rsc
:global data [/file get [/file find name=export.rsc] contents];
:global $url "https://prod-51.westeurope.logic.azure. ... ths/invoke....";
/tool fetch mode=https http-method=put http-data=$data url=$url

Amm0 · Wed Apr 06, 2022 4:54 pm

@Amm0
https://help.mikrotik.com/docs/display/ ... -Variables
> Note: Variable value size is limited to 4096bytes

https://help.mikrotik.com/docs/display/ROS/Fetch

/export file=export.rsc
:global data [/file get [/file find name=export.rsc] contents];
:global $url "https://prod-51.westeurope.logic.azure. ... ths/invoke....";
/tool fetch mode=https http-method=put http-data=$data url=$url

Look I ain't arguing you ROS shouldn't support some "file upload". All I was saying "file upload" can happen in a variety of ways (form-data, WebDAV, etc.) – so it's not as simple as a missing "files=" min fetch.

What you're asking "There has to be some workaround!"... Here the problem, as you note, is variable sizes are limited, so you can't encode anything beyond 4K in ROS script. So yes you might have more workarounds possible if variables WERE bigger than >4K. But this one is likely tied to ROS variable aren't much different than environment variable and the Linux kernel has some limit to size of them. Since scripts have to run the same on a CHR as a mAP, the limit can't be TOO high since it break scripts on low memory devices. And limitless variables are attach surfaces for buffer overflows.

Containers (e.g. Docker) will likely be here first, so you'd be able to spin up an container to run whatever `curl` or other method is needed.

Should /tool/fetch has more of of `curl` "encoding" options? You'd have my vote.

hkusulja · Wed Apr 06, 2022 5:34 pm

I understand your point about option to send file is not always the same.
In my case, i test with some test.txt about 1 KB, and is not working, received data does not contain data from file anyhow

vittore · Wed Apr 20, 2022 12:15 pm

Dear Mikrotik forums,
it would be great if command
Code: Select all
/tool/fetch http-method=post url="https://blabla.server/fileservice" mode=https 
can specify the file on mikrotik that will be send as data of POST request.

Better without that annoying 4kB limitation!

May I believe in that it will be implemented?

+1 for this feature

Deeleave · Thu Apr 21, 2022 8:08 am

Dear Mikrotik forums,
it would be great if command
Code: Select all
/tool/fetch http-method=post url="https://blabla.server/fileservice" mode=https 
can specify the file on mikrotik that will be send as data of POST request.

Better without that annoying 4kB limitation!

May I believe in that it will be implemented?

+1 for this feature

Shikiko · Thu Apr 21, 2022 11:27 am

Dear Mikrotik forums,
it would be great if command
Code: Select all
/tool/fetch http-method=post url="https://blabla.server/fileservice" mode=https 
can specify the file on mikrotik that will be send as data of POST request.

Better without that annoying 4kB limitation!

May I believe in that it will be implemented?

+1 for this feature

колбаскин · Sun Mar 10, 2024 1:23 pm

*) fetch - added "head" option for "http-method";
*) fetch - added "patch" option for "http-method";
*) fetch - allow specifying link-local address in FTP mode;
*) fetch - allow to use certificate and check-certificate parameters only in HTTPS mode;
*) fetch - do not require "content-length" for HTTP (introduced in v7.13);
*) fetch - fixed DNS resolving when domain has only AAAA entries (introduced in v7.13);
*) fetch - fixed fetch execution when unexpected data is received in HTTP payload;
*) fetch - fixed fetch when using "src-path" with HTTP/HTTPS modes (introduced in v7.13);
*) fetch - fixed fetch when using "src-path" with SFTP mode (introduced in v7.13);
*) fetch - fixed incorrect "src-path" error message when "upload=yes";
*) fetch - fixed IPv4 address logging (introduced in v7.13);
*) fetch - improved fetch stability in SFTP mode;
*) fetch - improved file download stability with HTTP/HTTPS modes;
*) fetch - less verbose logging;
*) fetch - print all "Set-Cookies" headers in response;
*) fetch - treat any 2xx HTTP return code as success (introduced in v7.13);

What about multipart/form-data ?

Amm0 · Sun Mar 10, 2024 4:52 pm

They added other things to scripting that make this, slightly, easier "manually". No direct "multipart/form-data files" AFAIK in /tool/fetch however.

For example, with some of the new scripting this, you MIGHT, depending on file size/type, using v7.13+ be able to do this in script. But you can see that's it's not just "files="that multipart/form-data needs. There is the file's MIME type, field name from "form", and the desired filename.

I'll provide some sample code, but don't vouch it will work & there are still limits of scripting that still may NOT have a workaround in scripting. Mikrotik added a ":convert to=base64" operation which is what's needed, in some case, to encode the file data.

:global sendFormDataFile do={
    :local urlf $url
    :local formFieldName $field
    :local fileName $file
    :local mimeSeperator "--$[:rndstr]" 
    :local contentType "Content-Type: multipart/form-data; boundary=$mimeSeperator"
    :local fileMimeType "application/octet-stream"

    :local formdataMime "\
    $mimeSeperator\n\
    Content-Disposition: form-data; name=\"$formFieldName\"; filename=\"$formName\"\n\
    Content-Type: $fileMimeType\n\
    Content-Transfer-Encoding: base64\n\n\
    $[:convert from=raw to=base64 [/file/get $fileName content]]\n\n\
    $mimeSeperator--\n"

    /tool/fetch http-method=post url=$urlf http-header-field=$contentType http-data=$formdataMime output=user
}

$sendFormDataFile url="http://192.168.88.1:80/upload" file=myrouterfile field=formfieldname

More for ideas, than suggesting this is a solution. form-data is tricky, since the other end needs to accept it and it may be looking for some very specific things to accept it. e.g. form-data is really a browser thing, so it kinda depends on the original HTML like what's contained on the <form> etc tags, on top of just the file data.

But if you can look at the HTTP transfer in a browser or sniffer, you'd have a better idea of how to build it RouterOS script. The above gives example of how to deal with the MIME headers which are critical for form-data. But what field name, encoding etc. etc. is needed depends on the server-end. It may need other form-data to in ANOTHER MIME block to even accept the file, too! But a sniffer or browser inspect of the HTTP headers/data from a browser give you a BIG clue how to build the http-data= needed in /tool/fetch.

колбаскин · Sun Mar 10, 2024 5:50 pm

Need send to

https://api.telegram.org/bot5955340922:AAE4Ljtk4Gm8anV8QoXhgck11CdNLCUaz1M/sendDocument\?chat_id=340234876

RB4011-backup--10202-0.backup backup file from router

https://core.telegram.org/bots/api#inputfile telegram bot api
Post the file using multipart/form-data in the usual way that files are uploaded via the browser. 10 MB max size for photos, 50 MB for other files.

Amm0 · Sun Mar 10, 2024 6:59 pm

Ah, Telegram docs suggest this:

In sendDocument, sending by URL will currently only work for GIF, PDF and ZIP files.

And backup files are normally bigger than 64kB (and even bigger when base64-encoded for form-data).
So not going to work for backups I think...

But using /tool/e-mail is actually easiest/most reliable way to get files out of RouterOS. Unfortunately, Telegram has not appear to have some built-in email gateway to Telegram.

Perhaps, if it's a new ARM Mikrotik, you can add a container that has `curl`. That let you do this since it supports the @filename syntax – but that's a bit of lot of work.

Why you're generally stuck with /tool/e-mail to get files off the box from a scheduled script.

diamuxin · Sun Mar 10, 2024 8:06 pm

I send the backup files directly to my FTP server located on my NAS.

e.g.

/tool fetch address=$ftpserver mode=ftp user=$username password=$password \
    src-path=$backupfilename dst-path=$backupfilename upload=yes port=$port;

The data is declared in local variables at the beginning of the script.

By Telegram I have not succeeded so far.

Feature request: /tool fetch HTTP-POST can send a file

Feature request: /tool fetch HTTP-POST can send a file

Re: Feature request: /tool fetch HTTP-POST can send a file

Re: Feature request: /tool fetch HTTP-POST can send a file

Re: Feature request: /tool fetch HTTP-POST can send a file

Re: Feature request: /tool fetch HTTP-POST can send a file

Re: Feature request: /tool fetch HTTP-POST can send a file

Re: Feature request: /tool fetch HTTP-POST can send a file

Re: Feature request: /tool fetch HTTP-POST can send a file

Re: Feature request: /tool fetch HTTP-POST can send a file

Re: Feature request: /tool fetch HTTP-POST can send a file

Re: Feature request: /tool fetch HTTP-POST can send a file

Re: Feature request: /tool fetch HTTP-POST can send a file

Re: Feature request: /tool fetch HTTP-POST can send a file

Re: Feature request: /tool fetch HTTP-POST can send a file

Re: Feature request: /tool fetch HTTP-POST can send a file

Re: Feature request: /tool fetch HTTP-POST can send a file

Re: Feature request: /tool fetch HTTP-POST can send a file

Re: Feature request: /tool fetch HTTP-POST can send a file

Re: Feature request: /tool fetch HTTP-POST can send a file

Re: Feature request: /tool fetch HTTP-POST can send a file

Re: Feature request: /tool fetch HTTP-POST can send a file

Re: Feature request: /tool fetch HTTP-POST can send a file

Re: Feature request: /tool fetch HTTP-POST can send a file

Re: Feature request: /tool fetch HTTP-POST can send a file

Who is online