Page 1 of 1

uploading files from "bash" to 7.18's /ip/cloud/file-share feature using `curl`

Posted: Fri Feb 07, 2025 6:00 pm
by Amm0
I've giving the new "file-share" a try in the 7.18 beta. It's working pretty, outside minor kinks, and /ip/cloud/file-share likely have a lot of useful applications.

It should be noted file-share does supports uploading files too, which you can do from the web page at the "File Sharing URL" for a specific /ip/cloud/file-share. I check the "Allow Uploads" box and it works.

It was not clear how to do this with curl. Inspecting the browser, it's using standard POST form encoding using multipart/form-data. So in curl from a desktop you can copy a file name "example.txt" using the following command:
curl -v https://0fak3serial123.routingthecloud.net/s/sEcRetKEYfromRtROS/ -F "file=@example.txt"
The only real trick is a CURL thing, but the URL must have a trailing / on the end of it. Otherwise it may not work with how processes the -F. If you cut-and-paste from RouterOS, you'll need to add a / at the end of the "File Direct Url" from winbox.

If you want to copy a set of files, I wrote a quick-and-dirty bourne shell (ash, bash, zsh) function, copy2router(). It will take 1 or files to copy from shell prompt and use "curl" with a /ip/cloud/file-share. Importantly the shell will expect the wildcard like *.txt or *.* or *, all those should work. It use an environment variable ICRS_URL="https://...." which must be set to use the function, and the URL should be without any trailing slash, to match shown in RouterOS (the function below will ADD the / for you - so it's still needed want envvar to match what RouterOS showed).

Now I'm not sure what happen when files are "big",/slow connections, etc... but the curl command to copy MULTIPLE FILES to try is here:
ICFS_URL="https://0fak3serial123.routingthecloud.net/s/sEcRetKEYfromRtROS"
copy2router() {
    local files=("$@")
    if [[ -z "$ICFS_URL" ]]; then
	echo "ICFS_URL must be to the 'File Share URL' from a RouterOS /ip/cloud/file-share"
	echo "    The environment variable must be in you shell $0"
	echo "         export ICFS_URL=\"https://sn12345677.routingthecloud.new/s/key123456\"" 
    fi
    local form_data=()
    for file in "${files[@]}"; do
        form_data+=("-F" "file=@$file")
    done
    curl "$ICFS_URL/" "${form_data[@]}"
}
Some of this may change since 7.18 is in beta, but at least I'm ahead of some future "how to use file-share with curl" questions ;)

Re: uploading files from "bash" to 7.18's /ip/cloud/file-share feature using `curl`

Posted: Sat Feb 08, 2025 8:36 pm
by optio
Next step - creating MT cloud mountable FS with FUSE, something like https://github.com/fangfufu/httpdirfs :)

Re: uploading files from "bash" to 7.18's /ip/cloud/file-share feature using `curl`

Posted: Sat Feb 08, 2025 10:19 pm
by Amm0
FWIW, sometimes I show `bash` code since folk like to suggest "give me a real shell for RouterOS"... I think forget it's actually more tedious than RouterOS scripting. ;).

Personally, I'd prefer WebDAV to newer custom protocols to mounting files, since WebDAV is an RFC & Apple supports it still AFAIK, which is how I end up enabling SMB on RouterOS for script dev.

Now, I've always like the concept of "virtual file systems". So don't challenge me here... I actually had the thought before to use FUSE to mount RouterOS config as virtual file system, with each attribute being a file & use FUSE + REST API as UNIX file-based config scheme. But I have enough Mikrotik side-projects. But, as technical matter, I think it be possible to implement a FUSE provider, over /ip/cloud/file-share — other than delete which isn't support AFAIK. Peeked at the HTML, there is ?list – which be central to any 3rd part thing – that get JSON with list of files and directories, but since whole thing is framework-less SPA that uses fetch()/XMLHTTPRequest calls... I don't see why it won't work (other than time/effort).

Re: uploading files from "bash" to 7.18's /ip/cloud/file-share feature using `curl`

Posted: Sun Feb 09, 2025 2:24 am
by Amm0
Next step - creating MT cloud mountable FS with FUSE, something like https://github.com/fangfufu/httpdirfs :)
We'll here is a write up of my notes on how the HTTP works inside file-share. Maybe ChatGPT can whip you up something ;)

?list - get the JSON you'd need
To get a list of directories, use a query parameter on URL of ?list, here is example CURL:
curl 'https://00serial00.routingthecloud.net/s/11key11?list' -X 'GET'

With the curl returning JSON structure, with the “files:” array contain an object with “path:” (i.e. files[x].path )
{
"board": "RB1100Dx4",
"dir": true,
"edit": true,
"files": [
{
"path": "readme.md",
"size": 12192,
"time": 1738943210,
"type": "file"
},
{
"path": "folder1",
"size": 1,
"time": 1739043458,
"type": "dir"
},
{
"path": "folder2",
"size": 2,
"time": 1739043712,
"type": "dir"
}
],
"identity": "bigdude",
"name": ""
}

Note that subfolder show shown in the JSON, to the files[x].type will be either “dir” for directory/subfolders or “file”


Subfolder are just appended to the URL, and similarly use same ?list, to get the same JSON format

?dl - get a single file, or ZIP of directory

?dl avoid the web interface to directly download a file, or directory. If the filename is last part URL last part, with any ?dl after the file name. So, both should be the same, for a file:
curl -X GET https://00serial00.routingthecloud.net/ ... est.txt?dl
or
curl -X GET https://00serial00.routingthecloud.net/ ... est.txt?dl


If it is a directory, it has a "special property" to ZIP the direct.
curl -X GET "https://00serial00.routingthecloud.net/ ... older2/?dl"
would generate a ZIP, if there are multiple files.

If you use same syntax on a folder, but it only has one file, it will directly download the one file for you. i.e. No ZIP is produced, if one is not needed.

POST uploads files - either replacing or creating a new, doesn't matter...
Basically using curl's -F with name of "file". Now any tool should work as long as content-type is "multipart/form-data" which will get the right MIME HTTP body. file-share does NOT support application/x-www-form-urlencoded, so that not going to work. See https://curl.se/docs/manpage.html#-F for the `curl -F` syntax for files. But there should be some "form-data" option in most automation tools.

To create a new file, or update an existing that use POST, again use the same relative file paths, same as the orginal post (curl defaults to POST with -F):
curl -v https://0fak3serial123.routingthecloud. ... fromRtROS/ -F "file=@example.txt"


DELETE is not implemented
There is no HTTP DELETE (reporting "Not implemented") operation, nor any obvious `?rm` or `?rmdir` or `?remove` or '?delete` operations on the GET or POST.

Re: uploading files from "bash" to 7.18's /ip/cloud/file-share feature using `curl`

Posted: Sun Feb 09, 2025 12:36 pm
by optio
Personally, I'd prefer WebDAV to newer custom protocols to mounting files, since WebDAV is an RFC
Yes, WebDAV would be much better for usage compatibility if is implemented, but as I see from your HTTP requests analysis in later post, MT cloud file share functionality is lacking basic things - create directory and delete file/dir, needed for creating functional virtual FS regardless if is WebDAV or custom made. Let's hope MT will improve it, or even better switch to WebDAV.

Now, I've always like the concept of "virtual file systems".
Me too, I'm using FUSE made FS's more-less actively since Gmail introduced GB inbox storage using GmailFS - a cloud drive like solution using email attachments for FS storage on Gmail, but it was slow :). These days on Mac I'm using FUSE made FS's for mounting real FS, like r-w NTFS and r-o ext2,3,4.