Convert cyrillic to PDU modem format for sending SMS

Sertik · Fri Jul 07, 2023 12:47 pm

Our Rextended has already written the wonderful conversion functions $gsm7topdu and $pdutogsm7 to convert code page 1252, if I understand correctly. And how could the text CP1251 (Cyrillic) be converted into PDU format for sending SMS in Russian? For good, we need a function for converting mixed text (CP1252 and CP1251 / Latin and Cyrillic) so that the output could receive data to be sent via SMS LTE at-chat

rextended · Fri Jul 07, 2023 4:22 pm

Mixed text do not exist, or is CP1252 or is CP1251 (or another)
UTF-8 have all symbols.

The PDU can be encoded GSM-7 (or ascii-8) or UTF-2, not any CPx.
If Cyrillic is present on SMS, is sended by UTF-2, not GSM-7 (or CP1251)

Function to convert UTF-8 to UCS-2
viewtopic.php?p=985041#p985041

Function to convert UTF-8 to UCS-2 string for create the mesage part on SMS PDU encoded with UCS-2.
viewtopic.php?p=985041#p985041

Function to convert UCS-2 to UTF-8
viewtopic.php?p=983695#p983695

Sertik · Fri Jul 07, 2023 8:45 pm

Oh Rex, I'm sorry I didn't understand anything... Could you please explain to the teapot

what should I do to send a text to CP1251 via SMS?

rextended · Fri Jul 07, 2023 8:46 pm

Again, short for translator:
1) You can NOT send CP1251 by SMS
2) The text MUST be first converted from CP1251 to UTF-8
(and since it's not an alphabet that I know, I'm not going to make a function for the conversion)
3) The UTF-8 MUST be converted to UCS-2
alternative to 2) and 3) The text MUST be first converted directly from CP1251 to UCS-2
4) When you have UCS-2, must be encoded on PDU user data part.

*** please if someone can write on better english.... ***

Sertik · Fri Jul 07, 2023 8:53 pm

So good.
1. I have a working function to convert CP1251 to UTF-8.
2. You wrote a function to convert UTF-8 to UCS-2.
3. And then how?

rextended · Fri Jul 07, 2023 8:57 pm

You already have UTF8toUCS2hexstring linked on my previous post.

At that point the entire SMS PDU needs to be created, but I've never done that.
At most I decoded it, I didn't create functions to recode everything,
but meanwhile with that function you have the "user part" of the PDU in UCS-2 format.

Sertik · Fri Jul 07, 2023 9:21 pm

I have a link to an example of encoding and sending SMS via PDU in Russian.

https://wireless-e.ru/development/pdu/

Maybe this will help you?

... entire SMS PDU needs to be created ...

I won't be able to write this...

rextended · Sat Jul 08, 2023 1:02 am

I'm currently too busy due to the high season to bother with it, maybe in the future...

However if you reverse (rewrite) the algorithms that decode... they encode...

optio · Sat Jul 08, 2023 4:33 pm

Tested with implementation that I have, viewtopic.php?t=196458#p1004704, cyrillic characters are correctly decoded and sent to email as is.

Edit: Sry I misread, you need for sending..., then you need function that does reversed, utf8 to ucs-2 as rextended wrote.

Amm0 · Sat Jul 08, 2023 6:28 pm

Yeah the encoding is likely easier than decoding – still a lot of work.

But what I worry about is how to send the PDU to the modem via script (e.g. /interface/lte/at-chat), since the GSM AT commands enter a weird "PDU entry mode". e.g. the AT command for sending SMS PDUs (AT+CMGS=<length>) creates a separate prompt. The linked Russian article above actually describes this unusual sequence with the > prompt.

There is a website that let you encode a PDU via a web site, so I'd test using pre-generated PDUs – if that works, THEN move on to creating the PDU.
PDU encode - https://www.smsdeliverer.com/online-sms ... coder.aspx
PDU decode - https://www.smsdeliverer.com/online-sms ... coder.aspx
These site will take the cyrillic as UTF-8 actually (since that what webpages use) and generate the entire PDU in UCS-2.

And if you look at a PDU output, you'll notice the bytes appear "flipped", but that because they use a different byte-order. So the final step, after getting UCS-2, is to convert the byte string from little endian to big endian in the Mikrotik script. Now RouterOS script can do this – but I just worry that the "at-chat" may NOT accept any PDU & you'd be stuck....

optio · Sat Jul 08, 2023 6:32 pm

It would be much easier if MT improves SMS tool, viewtopic.php?t=191963

Sertik · Mon Jul 10, 2023 9:07 am

It would be much easier if MT improves SMS tool

It is unlikely that we will wait for this, it's better to wait until Rex becomes freer.

rextended · Mon Jul 10, 2023 10:17 am

It would be much easier if MT improves SMS tool
It is unlikely that we will wait for this, it's better to wait until Rex becomes freer.

You are probably right

rextended · Fri Jul 14, 2023 9:11 pm

The new function I write ASCIItoCP1251toUNICODE can be used to convert Cyrillic CP1251 to UNICODE / UTF-2 for PDU, just remove 0x and the last space on
:local unicode "0x$($CP1251toUNICODE->[:find $ascii [:pick $string $pos ($pos + 1)] -1]) "
for obtain
тест = \F2\E5\F1\F2 = 0442043504410442
viewtopic.php?t=177551#p1012963

Sertik · Wed Jul 19, 2023 1:42 pm

How now to transfer this to the modem using lte-at chat ? Just a line?

For PDU, after all, it is necessary to read and transmit the length of the SMS, the recipient's number and the SMS itself ...

rextended · Wed Jul 19, 2023 4:15 pm

For Cyrillic is better UTF-2, and I have already done a converter from CP1251 to UCS-2, and from UTF-8 to UCS-2

Create all complete PDU for send the SMS is another question...

The function
viewtopic.php?t=177551#p1013154
help if you are forced to use GSM-7.
You can use directly the RouterOS function to send the message.
And can be easily adapted for own language, if transliteration is not good enough.

I apologize in advance for any mistranslation, it is not my intention to offend anyone.

For example with

example code

/tool sms send lte1 phone-number=3939 message="\CC\E8 \E2\F1\B3 \E1\F0\E0\F2\E8, \F5\EE\F7\E5\F8 \ED\E5 \F5\EE\F7\E5\F8."

(where \CC\E8 \E2\F1\B3 \E1\F0\E0\F2\E8, \F5\EE\F7\E5\F8 \ED\E5 \F5\EE\F7\E5\F8. is Ми всі брати, хочеш не хочеш. on CP1251)
I do not receive anything (or better only " , .") , but with

working example code

/tool sms send lte1 phone-number=3939 message=[$CP1251to7BITbyGHOST ("\CC\E8 \E2\F1\B3 \E1\F0\E0\F2\E8, \F5\EE\F7\E5\F8 \ED\E5 \F5\EE\F7\E5\F8.")]

I receive
Mi vsi brati, xochesh ne xochesh
(Probably is more precise "My vsi braty, khochesh ne khochesh." and is easy to fix it changing on function и -> y and х -> kh for adapt own language....)

Sertik · Thu Jul 20, 2023 2:18 pm

Mi vsi brati, xochesh ne xochesh

That's right.

That is, we do not communicate with the PDU, but simply transliterate the SMS and send it using the usual /tool sms send tool.
This, of course, is not what I wanted, but also a way out.

But in the link I already gave, it says that sending Russian-language messages through the modem's PDU mode is possible ...
https://wireless-e.ru/development/pdu/

Sertik · Thu Jul 20, 2023 2:46 pm

sometime in 2020 I wrote the function of transliterating Russian messages for Telegram
There is not a complete CP table, but all the characters needed for transliteration.
https://habr.com/ru/articles/519406/

# Function Translite of Russian characters for sending in Telegram
# by Sertik 16/09/2020
# usage [$FuncTransliteToTele "Строка String .,!+"]
:global FuncTransliteToTele do={

:global string; :set $string $1;

#  table of the codes of Russian letters Translite
:local rsimv [:toarray {"А"="A"; "Б"="B"; "В"="V"; "Г"="G"; "Д"="D"; "Е"="E"; "Ж"="ZH"; "З"="Z"; "И"="I"; "Й"="J"; "К"="K"; "Л"="L"; "М"="M"; "Н"="N"; "О"="O"; "П"="P"; "Р"="R"; "С"="S"; "Т"="T"; "У"="U"; "Ф"="F"; "Х"="KH"; "Ц"="C"; "Ч"="CH"; "Ш"="SH"; "Щ"="SCH"; "Ъ"="``"; "Ы"="Y`"; "Ь"="`"; "Э"="E`"; "Ю"="JU"; "Я"="YA"; "а"="a"; "б"="b"; "в"="v"; "г"="g"; "д"="d"; "е"="e"; "ж"="zh"; "з"="z"; "и"="i"; "й"="j"; "к"="k"; "л"="l"; "м"="m"; "н"="n"; "о"="o"; "п"="p"; "р"="r"; "с"="s"; "т"="t"; "у"="u"; "ф"="f"; "х"="kh"; "ц"="c"; "ч"="ch"; "ш"="sh"; "щ"="sch"; "ъ"="``"; "ы"="y`"; "ь"="`"; "э"="e`"; "ю"="ju"; "я"="ya"; "Ё"="Yo"; "ё"="yo"; "№"="#"}]

# encoding of the symbols and аssembly line
:local StrTele ""; :local code "";
:for i from=0 to=([:len $string]-1) do={:local keys [:pick $string $i (1+$i)];

:local key ($rsimv->$keys); if ([:len $key]!=0) do={:set $code ($rsimv->$keys);} else={:set $code $keys};
:if (($keys="Ь")  and ([:pick $string ($i+1) (2+$i)]="Е")) do={:set $code "I"; :set $i ($i+1)}
:if (($keys="ь")  and ([:pick $string ($i+1) (2+$i)]="е")) do={:set $code "i"; :set $i ($i+1)}
:if (($keys="Ь")  and ([:pick $string ($i+1) (2+$i)]="е")) do={:set $code "I"; :set $i ($i+1)}
:if (($keys="ь")  and ([:pick $string ($i+1) (2+$i)]="Е")) do={:set $code "i"; :set $i ($i+1)}
:if (($keys="Ы")  and ([:pick $string ($i+1) (2+$i)]="Й")) do={:set $code "I"; :set $i ($i+1)}
:if (($keys="ы")  and ([:pick $string ($i+1) (2+$i)]="й")) do={:set $code "i"; :set $i ($i+1)}
:if (($keys="ы")  and ([:pick $string ($i+1) (2+$i)]="Й")) do={:set $code "i"; :set $i ($i+1)}
:if (($keys="Ы")  and ([:pick $string ($i+1) (2+$i)]="й")) do={:set $code "I"; :set $i ($i+1)}
 :set $StrTele ("$StrTele"."$code")}

:return $StrTele
}

It will also work when transliterating SMS. But that's not it, I would like to send real Russian SMS...

rextended · Thu Jul 20, 2023 3:17 pm

You must use UCS-2.

If you have the UTF-8, you can convert it to UCS-2 by
viewtopic.php?p=985041#p985041

Or if you have CP1251 you can convert directly to UCS-2 ( IS = TO UNICODE ENTRY POINT)
viewtopic.php?t=177551#p1012963
(simply remove 0x and the end space for have the UCS-2 text for create the User Data for the PDU)

Sertik · Thu Jul 20, 2023 10:39 pm

Yes, but the received data can no longer be sent using the /tool sms send method. It will be necessary to send through the modem PDU mode, right? But we don’t have the function of creating a ready-made PDU ... Do I understand correctly?

rextended · Fri Jul 21, 2023 12:26 am

Exactly.

Maybe this winter when I have more time...

Sertik · Fri Jul 21, 2023 11:42 am

Sure. We will wait for you, Master.

Fri Jul 21, 2023 12:02 pm

Just use ASCII latin characters. Why is that an issue? Translit is a thing.

Sertik · Fri Jul 21, 2023 12:38 pm

This is not a problem, but a desire.

Amm0 · Fri Jul 21, 2023 3:32 pm

You run into this issue even if you have a valid UCS-2 PDU:
viewtopic.php?p=174760&hilit=pdu#p174760

Even a perfectly PDU encoding isn't going to be helpful in "/tool sms" to SEND, since before the "message" in PDU there is a flag with the encoding. And Mikrotik is always declaring the message as ASCII (GSM7) in the PDU its building internally, so encoding the message= is likely not going to work. Maybe if MBIM is used, the message can be UCS-2, but doubt it.

And, if you use AT command to send, the PDU entry mode creates a "command prompt" needing a specific sequence that cannot be done with /interface lte at-chat [find] input="..." – AT+CMGF=1 needs a specific sequence since there is a "> " prompt that appears.

The encoding is one step, but has to get into the modem somehow...

Sertik · Fri Jul 21, 2023 3:55 pm

It's clear. So this is not possible with the help of the script. This can only be implemented by the developers of the Router OS, but they will not do this, since it is necessary to support not only the Russian language, but also all other national languages. Although they could make it possible to install one national language in Ros in addition to English.
And then each user could additionally set their own national language.

rextended · Fri Jul 21, 2023 4:02 pm

Send

AT+CMGS=12

[12 is the length of the PDU 1 Byte (2 octects) less]
and wait for "> "
then send

0021000181F1001104F4F29C0E\0x1a

(\0x1a = message end)
done.

But.... but... but...

Sertik · Fri Jul 21, 2023 4:16 pm

Yes, but sends that from the CLI probably. And from a script?

rextended · Fri Jul 21, 2023 4:20 pm

Yes, but sends that from the CLI probably. And from a script?

That's the problem...

rextended · Fri Jul 21, 2023 4:34 pm

Until someone find a way to send a test one,
it is completely useless to recode/create the PDU...

Amm0 · Fri Jul 21, 2023 4:43 pm

It's the "wait for" part that isn't possible... the at-chat add \n (maybe \r\n) and waits for OK. You can say not wait (wait=no), but still won't help since you do need to wait for the "SMS text mode" and there is a specific control codes that prevent splitting into two commands. And SMS PDU is build before you get to message= in /tool/sms.

I'm still not sure why some encoding=ucs2 option be difficult in /tool/sms. Now multi-lingual RouterOS, that I'm not expecting. Setting one byte (actually less, half-byte/semi-octet) in one command seems doable & allowing a message like "\00\80\00\80" to passthrough. Not saying it should be top priority, but they are pitching IoT devices & sell to many non-English countries.

And, BTW, it's also poor "emoji people"

that are effected

— which you can't send either, since they're unicode (and require similar UCS2 treatment).

Sertik · Mon Jul 24, 2023 10:34 am

Now multi-lingual RouterOS

no, a multilingual OS router is not needed, you only need the ability to install support packages for national languages to send SMS