Page 1 of 1
Feature request: /tool fetch HTTP-POST can send a file
Posted: Wed Jan 05, 2022 9:40 am
by vanikcz
Dear Mikrotik forums,
it would be great if command
/tool/fetch http-method=post url="https://blabla.server/fileservice" mode=https
can specify the file on mikrotik that will be send as data of POST request.
Better without that annoying 4kB limitation!
May I believe in that it will be implemented?
Re: Feature request: /tool fetch HTTP-POST can send a file
Posted: Mon Jan 24, 2022 10:16 pm
by hkusulja
I also hope this feature will be implemented.
So, I can write a script which will upload my backup files and /export to remote https server...
Re: Feature request: /tool fetch HTTP-POST can send a file
Posted: Mon Jan 24, 2022 10:25 pm
by msatter
Better without that annoying
4kB 63KB limitation!
Also:
:set $data "up to 63KB..."
/tool fetch http-method=post http-content-type="application/json" http-data=$data url="https://testserver.lv/index.php"
Then, for downloading "chunking" can be used when the server supports it. I don't know if "chunking" is also supported for posting.
viewtopic.php?f=9&t=177530
Re: Feature request: /tool fetch HTTP-POST can send a file
Posted: Tue Jan 25, 2022 1:22 am
by hkusulja
Can you please help how to read file content (.backup and .rsc) from local mikrotik storage to be uploaded using $data to https remote server?
I do not see how can you use chunks for upload / http PUT method.
Can you please reference documentation stating that $data is limited do 63 KB in RouterOS 7 ?
Re: Feature request: /tool fetch HTTP-POST can send a file
Posted: Tue Jan 25, 2022 6:58 pm
by msatter
Mikrotik did not implement a way to read files larger than 63KB. Writing bigger files is possible with the execute command.
I don't think it is documented but the 63KB for variables is reality in ROS 6+7.
A few months ago no one knew that chunking was possible in combination with variable in ROS and now it used widely.
Re: Feature request: /tool fetch HTTP-POST can send a file
Posted: Mon Jan 31, 2022 12:49 am
by vanikcz
Dear msatter,
please canyou specify how can I load a 60kB file in variable? I cant find any example of that...
A few months ago no one knew that chunking was possible in combination with variable in ROS and now it used widely.
Thank you
Re: Feature request: /tool fetch HTTP-POST can send a file
Posted: Thu Feb 10, 2022 8:19 am
by hkusulja
@msatter I am also interested and did not find a documented way for reading the file into variable, on RouterOS 7 when file is bigger then 10 KB but smaller then 60 KB.
Re: Feature request: /tool fetch HTTP-POST can send a file
Posted: Fri Feb 25, 2022 10:19 am
by hkusulja
I have made a test:
:global $data "test";
:global $url "https://prod-51.westeurope.logic.azure.com:443/workflows/blabla/triggers/manual/paths/invoke....";
/tool fetch mode=https http-method=put http-data=$data url=$url
and this is recieved as, which seems kind of fine
{
"headers": {
"User-Agent": "Mikrotik/7.x,Fetch",
"Content-Type": "application/x-www-form-urlencoded"
},
"body": {
"$content-type": "application/x-www-form-urlencoded",
"$content": "dGVzdA==",
"$formdata": [
{
"key": "test",
"value": ""
}
]
}
}
However, if I try to load file content into variable (file is about 7 KB)
/export file=export.rsc
:global data [/file get [/file find name=export.rsc] contents];
and after http fetch , there is no body data received on http server.
How to send a file content? also, for export and for backup files using HTTP ?
Re: Feature request: /tool fetch HTTP-POST can send a file
Posted: Fri Feb 25, 2022 2:04 pm
by msatter
It is the first code example in this tread:
viewtopic.php?f=9&t=177530
Re: Feature request: /tool fetch HTTP-POST can send a file
Posted: Fri Feb 25, 2022 2:32 pm
by hkusulja
this is example for "file download" and using chunks.
My request is to "upload file", which is not bigger then 64 KB.
Re: Feature request: /tool fetch HTTP-POST can send a file
Posted: Thu Mar 10, 2022 7:22 pm
by Amm0
The key detail is what on the other end accepts. I'll presume your referring to `curl` that supports a "-F" for forms and supports a "@<filename>" to include a file. This depends on HTML5's <... input=file> within some <form enctype=...>, and enctype can be "multipart/form-data" or "x-www-urlencoded" ( curl defaults to one, depending on version and content-type). So those are TWO types of file upload, and other ways too.
But unlike `curl`,
does nothing to help with the data= part. AFAIK there is not a script I've seen that build the needed MIME nor encodes a /file. And, if it's binary file, even a script may not be able to access the contents of the file to send nor encode it without more scripting. For text files, it
might be possible to build the needed data= for a file upload, but not easy.
Basically RouterOS doesn't have any primitives to build or parse any of the HTTP content-types. It will happily send a content-type and whatever data you build into a script "str" type. But ain't going to help with the data at all today. Currently, you can't even encode or decode JSON to RouterOS array, which is even bigger limitation IMO. This is why you need scripts for JSON, or rextended's "chunk download", to workaround to the lack of support dealing with the HTTP
data.
Maybe someone else know of a workaround to form data and encoding... but you may want to look at other ways of transfer files than "HTTP upload". You could turn this around and have the REST API or SFTP
on the webserver and pull the file based on an HTTP request.
Re: Feature request: /tool fetch HTTP-POST can send a file
Posted: Fri Mar 11, 2022 9:44 am
by hkusulja
Doing SFTP or other things is not possible due to other reasons in my case. I am aware of this options.
Please, lets focus here on how to anyhow send any file from mikrotik to remote HTTP server. (not vice versa, etc.)
The other end is HTTP web server, working with standard HTTP stuff. I do have control on that side and I can program whatever is needed. For now I have setup HTTP REST APi endpoint accepting anything / anyhow, just to get somehow the file from mikrotik.
Regarding content type, it can be setup manually via /tool fetch as I can see, but I can also fix that on server side.
I do not need manipulation with JSON on routeros side anyhow.
So, what exactly is RouterOS limitation for /tool fetch working with http post for sending file? I think is just documentation lacking but, could be worse / not implemented features, which exactly ones?
Re: Feature request: /tool fetch HTTP-POST can send a file
Posted: Fri Mar 11, 2022 2:10 pm
by Amm0
Please, lets focus here on how to anyhow send any file from mikrotik to remote HTTP server. (not vice versa, etc.)
You can't.
Mikrotik will need to add support for it.
Since they have a backlog of bugs and other features, you may be waiting a while. Essentially /tool/fetch only lets you send a string (but the string can be what ever you want). RouterOS doesn't have any helpers to encode a form submission (of which a file is a type).
And, RouterOS script can only read files of 4K or less into a string, so this prevent more cleaver solution in scripting. Basically, this would be need to work, as a first step of many, to be able to send any data from a file:
:global filedata [/file get myfile contents]
:put $filedata
Re: Feature request: /tool fetch HTTP-POST can send a file
Posted: Wed Apr 06, 2022 11:08 am
by hkusulja
@Amm0
https://help.mikrotik.com/docs/display/ ... -Variables
> Note: Variable value size is limited to 4096bytes
https://help.mikrotik.com/docs/display/ROS/Fetch
/export file=export.rsc
:global data [/file get [/file find name=export.rsc] contents];
:global $url "
https://prod-51.westeurope.logic.azure. ... ths/invoke....";
/tool fetch mode=https http-method=put http-data=$data url=$url
Re: Feature request: /tool fetch HTTP-POST can send a file
Posted: Wed Apr 06, 2022 4:54 pm
by Amm0
Look I ain't arguing you ROS shouldn't support some "file upload". All I was saying "file upload" can happen in a variety of ways (form-data, WebDAV, etc.) – so it's not as simple as a missing "files=" min fetch.
What you're asking "There has to be some workaround!"... Here the problem, as you note, is variable sizes are limited, so you can't encode anything beyond 4K in ROS script. So yes you might have more workarounds possible if variables WERE bigger than >4K. But this one is likely tied to ROS variable aren't much different than environment variable and the Linux kernel has some limit to size of them. Since scripts have to run the same on a CHR as a mAP, the limit can't be TOO high since it break scripts on low memory devices. And limitless variables are attach surfaces for buffer overflows.
Containers (e.g. Docker) will likely be here first, so you'd be able to spin up an container to run whatever `curl` or other method is needed.
Should /tool/fetch has more of of `curl` "encoding" options? You'd have my vote.
Re: Feature request: /tool fetch HTTP-POST can send a file
Posted: Wed Apr 06, 2022 5:34 pm
by hkusulja
I understand your point about option to send file is not always the same.
In my case, i test with some test.txt about 1 KB, and is not working, received data does not contain data from file anyhow
Re: Feature request: /tool fetch HTTP-POST can send a file
Posted: Wed Apr 20, 2022 12:15 pm
by vittore
Dear Mikrotik forums,
it would be great if command
/tool/fetch http-method=post url="https://blabla.server/fileservice" mode=https
can specify the file on mikrotik that will be send as data of POST request.
Better without that annoying 4kB limitation!
May I believe in that it will be implemented?
+1 for this feature
Re: Feature request: /tool fetch HTTP-POST can send a file
Posted: Thu Apr 21, 2022 8:08 am
by Deeleave
Dear Mikrotik forums,
it would be great if command
/tool/fetch http-method=post url="https://blabla.server/fileservice" mode=https
can specify the file on mikrotik that will be send as data of POST request.
Better without that annoying 4kB limitation!
May I believe in that it will be implemented?
+1 for this feature
Re: Feature request: /tool fetch HTTP-POST can send a file
Posted: Thu Apr 21, 2022 11:27 am
by Shikiko
Dear Mikrotik forums,
it would be great if command
/tool/fetch http-method=post url="https://blabla.server/fileservice" mode=https
can specify the file on mikrotik that will be send as data of POST request.
Better without that annoying 4kB limitation!
May I believe in that it will be implemented?
+1 for this feature
Re: Feature request: /tool fetch HTTP-POST can send a file
Posted: Sun Mar 10, 2024 1:23 pm
by колбаскин
*) fetch - added "head" option for "http-method";
*) fetch - added "patch" option for "http-method";
*) fetch - allow specifying link-local address in FTP mode;
*) fetch - allow to use certificate and check-certificate parameters only in HTTPS mode;
*) fetch - do not require "content-length" for HTTP (introduced in v7.13);
*) fetch - fixed DNS resolving when domain has only AAAA entries (introduced in v7.13);
*) fetch - fixed fetch execution when unexpected data is received in HTTP payload;
*) fetch - fixed fetch when using "src-path" with HTTP/HTTPS modes (introduced in v7.13);
*) fetch - fixed fetch when using "src-path" with SFTP mode (introduced in v7.13);
*) fetch - fixed incorrect "src-path" error message when "upload=yes";
*) fetch - fixed IPv4 address logging (introduced in v7.13);
*) fetch - improved fetch stability in SFTP mode;
*) fetch - improved file download stability with HTTP/HTTPS modes;
*) fetch - less verbose logging;
*) fetch - print all "Set-Cookies" headers in response;
*) fetch - treat any 2xx HTTP return code as success (introduced in v7.13);
What about multipart/form-data ?
Re: Feature request: /tool fetch HTTP-POST can send a file
Posted: Sun Mar 10, 2024 4:52 pm
by Amm0
They added other things to scripting that make this, slightly, easier "manually". No direct "multipart/form-data files" AFAIK in /tool/fetch however.
For example, with some of the new scripting this, you MIGHT, depending on file size/type, using v7.13+ be able to do this in script. But you can see that's it's not just "files="that multipart/form-data needs. There is the file's MIME type, field name from "form", and the desired filename.
I'll provide some sample code, but don't vouch it will work & there are still limits of scripting that still may NOT have a workaround in scripting. Mikrotik added a ":convert to=base64" operation which is what's needed, in some case, to encode the file data.
:global sendFormDataFile do={
:local urlf $url
:local formFieldName $field
:local fileName $file
:local mimeSeperator "--$[:rndstr]"
:local contentType "Content-Type: multipart/form-data; boundary=$mimeSeperator"
:local fileMimeType "application/octet-stream"
:local formdataMime "\
$mimeSeperator\n\
Content-Disposition: form-data; name=\"$formFieldName\"; filename=\"$formName\"\n\
Content-Type: $fileMimeType\n\
Content-Transfer-Encoding: base64\n\n\
$[:convert from=raw to=base64 [/file/get $fileName content]]\n\n\
$mimeSeperator--\n"
/tool/fetch http-method=post url=$urlf http-header-field=$contentType http-data=$formdataMime output=user
}
$sendFormDataFile url="http://192.168.88.1:80/upload" file=myrouterfile field=formfieldname
More for ideas, than suggesting this is a solution. form-data is tricky, since the other end needs to accept it and it may be looking for some very specific things to accept it. e.g. form-data is really a browser thing, so it kinda depends on the original HTML like what's contained on the <form> etc tags, on top of just the file data.
But if you can look at the HTTP transfer in a browser or sniffer, you'd have a better idea of how to build it RouterOS script. The above gives example of how to deal with the MIME headers which are critical for form-data. But what field name, encoding etc. etc. is needed depends on the server-end. It may need other form-data to in ANOTHER MIME block to even accept the file, too! But a sniffer or browser inspect of the HTTP headers/data from a browser give you a BIG clue how to build the http-data= needed in /tool/fetch.
Re: Feature request: /tool fetch HTTP-POST can send a file
Posted: Sun Mar 10, 2024 5:50 pm
by колбаскин
Need send to
https://api.telegram.org/bot5955340922:AAE4Ljtk4Gm8anV8QoXhgck11CdNLCUaz1M/sendDocument\?chat_id=340234876
RB4011-backup--10202-0.backup backup file from router
https://core.telegram.org/bots/api#inputfile telegram bot api
Post the file using multipart/form-data in the usual way that files are uploaded via the browser. 10 MB max size for photos, 50 MB for other files.
Re: Feature request: /tool fetch HTTP-POST can send a file
Posted: Sun Mar 10, 2024 6:59 pm
by Amm0
Ah, Telegram docs suggest this:
In sendDocument, sending by URL will currently only work for GIF, PDF and ZIP files.
And backup files are normally bigger than 64kB (and even bigger when base64-encoded for form-data).
So not going to work for backups I think...
But using /tool/e-mail is actually easiest/most reliable way to get files out of RouterOS. Unfortunately, Telegram has not appear to have some built-in email gateway to Telegram.
Perhaps, if it's a new ARM Mikrotik, you can add a container that has `curl`. That let you do this since it supports the @filename syntax – but that's a bit of lot of work.
Why you're generally stuck with /tool/e-mail to get files off the box from a scheduled script.
Re: Feature request: /tool fetch HTTP-POST can send a file
Posted: Sun Mar 10, 2024 8:06 pm
by diamuxin
I send the backup files directly to my FTP server located on my NAS.
e.g.
/tool fetch address=$ftpserver mode=ftp user=$username password=$password \
src-path=$backupfilename dst-path=$backupfilename upload=yes port=$port;
The data is declared in local variables at the beginning of the script.
By Telegram I have not succeeded so far.