Special characters in message text not working (anymore)

[PastCoder]

I am sending message via s script and signal-cli.
Since some days I am receiving on my mobile phone only � instead of special characters (German Umlaut) and emojis.

I think I did not change anything other related to sending the messages. So I assume that some recent update of signal-cli caused the problem.

[AsamK]

Please check the LANG environment variable, it should use a UTF-8 locale, like fr_FR.UTF-8

[PastCoder]

This looks quite good:
echo $LANG
gives
de_DE.UTF-8

@AsamK Any additional idea what I could check?

I am sending the message with
sudo -u ... /usr/bin/signal-cli -u +49... send -m "<message with äöüö" +49...

sudo -u ... echo $LANG
gives as well de_DE.UTF-8

[PastCoder]

Does anybody have an idea what I could check in addition to find the reason?

[PastCoder]

I tried further to find the reason but did not succeed. I did so just as a user (not debugging).
Log files did not help.
When I receive a message the problem is the same: Normal test is received but all special characters are only "?".
Am I the only one with problems sending and receiving special characters (since 2-3 weeks)?

I am using signal-cli 0.11.5.1 on Ubuntu 22.04.1 LTS

[Wax-Replica]

I am getting a similar problem when sending or receiving messages with emoji and such in windows 10, signal-cli 0.11.5.1.
when i send a message from signal-cli with an emoji in it, it appears on the receiving device as "??"
when signal-cli receives a message with emoji in it, the emoji are replaced with "?"
sending a 'ä' (lowercase a with umlaut) works but 'א' (hebrew letter aleph) does not

my LANG environment variable is en_US.UTF-8

in other issue threads AsamK brings up java -XshowSettings:properties -version 2>&1.

i get

java -XshowSettings:properties -version 2>&1 | findstr encoding
file.encoding = Cp1252
native.encoding = Cp1252
sun.io.unicode.encoding = UnicodeLittle
sun.jnu.encoding = Cp1252

i understand they should be set to UTF-8 not Cp1252, and that this is probably the cause of the problem, but i can't seem to find how to change them anywhere.

[PastCoder]

The mentioned parameter shown in java -XshowSettings:properties -version are on my system:
file.encoding = UTF-8
native.encoding --> not listed
sun.io.unicode.encoding = UnicodeLittle
sun.jnu.encoding = UTF-8

[PastCoder]

Does anybody know if the problem may be caused in https://github.com/signalapp/libsignal and we need to create an issue ticket there!?

[almet]

I confirm I'm having the same issue. My locale is set to french and it doesn't fix the problem.

The issue happened with a recent upgrade of signal-cli-native and signal-cli-service from 0.11.4 to 0.11.5.1 (using this Debian repository).

I checked the changes between 0.11.4 and 0.11.5.1 and haven't found anything noticeable regarding how messages are send appart from the handling of attachments but I'm not sure it's relevant.

@AsamK don't hesitate to ping me back if you have troubles reproducing. I'll continue investigating a bit.

[PastCoder]

I am still trying to figure out where the problems come from (but not able to debug).
I saw that I have different versions installed for service and native. Is that correct? Could that cause problems?

Are there people able to send special characters and emojis?

[morph027]

@PastCoder different versions do not matter, as the service is basically just the systemd unit file.

btw. sending works for me (tested using jsonrpc via http)

[PastCoder]

btw. sending works for me (tested using jsonrpc via http)
@morph027 Could yoou please send a script or anything (for me as rpc unexperienced) so that I can test with the same way? Till now I am just using the command line (which worked before but do have that problem).
Does anybody have an idea how I could try to locate the problem cause?

[morph027]

Sure. I've just started signal-cli like this signal-cli daemon --http=localhost:3000

Then using curl:

curl -H "content-type: application/json" -d'{"jsonrpc":"2.0", "method": "send", "id": "42", "params": {"account": "+<your-signal-account>", "recipients": ["+<recipient>"], "message": "äüö"}} http://127.0.0.1:3000/api/v1/rpc

[PastCoder]

Hi @morph027 ,
thanks for the command. I just executed them.
With this the äöü are correctly transferred.
Could you test, if it works for you, if you use the command line?
signal-cli -u +49... send -m "<message with äöüö" +49...

If it works, which version do you use?

[morph027]

Jip, right, using command line i do get ������ too .....

[PastCoder]

As a workaround I changed now my implementation to daemon and curl.
But one problem I was not able to solve:
I am creating the curl command-line in a Javascript script. If there is a new line (\n) in the message, it does not work.
Does anybody know how to solve that? this was working with the command line :-)

[morph027]

The daemon and json-rpc approach has the advantage of being faster as the daemon is running all the time and the jvm does not need to be started each call.

If so, you should probably just use java script builtin http requests (https://livecodestream.dev/post/5-ways-to-make-http-requests-in-javascript/).

In the meantime, i've just added a new build pipeline for v0.11.6-4 which re-adds the buildArgs and a quick test shows it works. Should be releases soon.

But i'd still would recommend the daemon and native http requests ;) or probably even json rpc client exists alreay.

[PastCoder]

Since the update to 0.11.6-4 it works again with the command line.

[disaster123]

i've the same problem since upgrading from signal-cli-0.11.5.1 to signal-cli 0.12.7 - all locales and LANG settings are set to utf-8. Command line öäü now results in ? in received messages on my iphone.

[morph027]

Hm....still works on my machine

[disaster123]

not sure how a screenshot helps. But here is mine ;-)

[morph027]

Cayn you try with JAVA_OPTS='-Dfile.encoding=UTF-8'

[disaster123]

Do you mean like this?

linuxbox ~ # JAVA_OPTS='-Dfile.encoding=UTF-8' ./signal-cli send $NUMBER -m "Test öäü"
1702974924494

if yes - same result

[disaster123]

It works fine with:

signal-cli-0.11.5.1-Linux-native.tar.g

not working with:

signal-cli0127_ubuntu2004_amd64.gz

[m-ueberall]

not working with:

signal-cli0127_ubuntu2004_amd64.gz

I can reproduce the above; however, in order to reopen this issue, you'd need a reproducer using the upstream native binary/Java archive.

[morph027]

If it happens w/ upstream native build too, it might help to add this patch ... will see if i can try to reproduce.

Edit: tested latest upstream binary and it works

[disaster123]

not working with:

signal-cli0127_ubuntu2004_amd64.gz

I can reproduce the above; however, in order to reopen this issue, you'd need a reproducer using the upstream native binary/Java archive.

I truly understand but I‘ve no newer system than bullseye available.

[morph027]

not working with:

signal-cli0127_ubuntu2004_amd64.gz

I can reproduce the above; however, in order to reopen this issue, you'd need a reproducer using the upstream native binary/Java archive.

I truly understand but I‘ve no newer system than bullseye available.

You could still try my buster based build:

curl -LO https://gitlab.com/packaging/signal-cli/-/jobs/5770244895/artifacts/raw/signal-cli-native-amd64-tag/signal-cli-native_0.12.7+2_amd64.deb
mkdir signal-cli-native_0.12.7+2
dpkg-deb -R signal-cli-native_0.12.7+2_amd64.deb signal-cli-native_0.12.7+2/
signal-cli-native_0.12.7+2/usr/bin/signal-cli-native send $NUMBER -m "Test öäü"

[m-ueberall]

[…] it might help to add this patch ... will see if i can try to reproduce.

I've added the two lines and that fixes the problem! They're worth a pull request! 😁
signal-cli0127r2_ubuntu2004_{amd64,arm64}.gz will be published shortly.
EDIT: done.

[morph027]

I've added the two lines and that fixes the problem! They're worth a pull request! 😁 signal-cli0127r2_ubuntu2004_{amd64,arm64}.gz will be published shortly. EDIT: done.

Great, created a PR

[disaster123]

i can confirm that this solves the issue! Thanks a lot!