New JSON-RPC Api

[AsamK]

I've implemented a JSON-RPC API using stdin/stdout, based on @technillogue's idea to reuse the existing commands.
It supports almost all current commands, except some list* commands that don't have json output yet.
There's no documentation yet, but the request syntax can be derived from the cli documentation.

signal-cli -u +XXX --output=json jsonRpc

Some example requests:

--> {"jsonrpc":"2.0","method":"listGroups","id":"5"}
<-- {"jsonrpc":"2.0","result":[...],"id":"5"}

--> {"jsonrpc":"2.0","method":"send","params":{"recipient":["+YYY"],"message":"MESSAGE"},"id":4}
<-- {"jsonrpc":"2.0","result":{"timestamp":999},"id":4}

--> {"jsonrpc":"2.0","method":"updateGroup","params":{"groupId":"GROUP_ID=","name":"new group name","members":["+ZZZ"],"link":"enabledWithApproval","setPermissionEditDetails":"onlyAdmins"},"id":"someId"}
<-- {"jsonrpc":"2.0","result":{"timestamp":9999},"id":"someId"}

--> {"jsonrpc":"2.0","method":"sendSyncRequest","id":9}
<-- {"jsonrpc":"2.0","result":{},"id":9}

--> {"jsonrpc":"2.0"}
<-- {"jsonrpc":"2.0","error":{"code":-32600,"message":"method field must be set","data":null},"id":null}

Incoming messages are sent as JSON-RPC notifications:

<-- {"jsonrpc":"2.0","method":"receive","params":{"envelope":{...}}}

I'd be happy for feedback, the API might still change before the next release.

[exquo]

This is really great!
Most people would probably prefer to communicate with the signal-cli daemon through stdin/out rather than over dbus. (And in some setups dbus is not even an option)

An FYI for anyone interested, stdI/O can be easily redirected to a socket. See e.g. this question on superuser.

One question offhand: shouldn't jsonRpc already imply --output=json?

[technillogue]

I did #668 and #396 assuming that I needed to keep plain-text as an option, but I'm not sure if there are really any actual use cases for it. If there are, maybe wrapping output in {"plain-text": ...} would make sense?

[exquo]

Thanks for your work on this!
The plain-text output (i.e. signal-cli w/o --output=…) is required at least for the backwards compatibility. I'm sure some folks have already written parsers for that format (maybe before there was a JSON output option).

[AsamK]

Thanks for the feedback!
Indeed jsonRpc should default to json output, I'll change that.

plain-text output will remain.
Just for the jsonRpc command I think having the plain text output from commands wrapped in a json response is not that useful.

[AsamK]

Now json output is the default for the jsonRpc command (ef2a013)

[technillogue]

I've read through the changes. This seems pretty reasonable, though I'm a little confused about the implementation being split across files like that.

Can you explain getRequestType and why have both JsonRpcCommand and JsonRpcLocalCommand?

The documentation needs to clearly indicate that dashed separators are converted to camelcase.

It might be helpful to merge the namespace of the outer command with that of the inner command so that the username etc don't have to be specified each time.

I'll try updating my code against this next week.

[AsamK]

With getRequestType the commands can specify the data type of the request params, currently it's just a Map for all commands.
I've added the JsonRpcCommand interface to have the possibility for commands, that are only available via the json API or have a more specialized interface for the json API.

The username doesn't have to be specified each time in the json request payload. All jsonRpc commands use the Signal account, that was specified on the jsonRpc command line.

[technillogue]

{"jsonrpc":"2.0", "method":"version"}
{"jsonrpc":"2.0", "method":"version", "id":"a"}
{"jsonrpc":"2.0","result":{"version":"0.8.5"},"id":"a"}

This matches spec, maybe we should warn users (perhaps on stderr?) that they haven't given an id for a method that has output.

[AsamK]

Maybe in verbose logging, it's a totally valid request, if the user is not interested in the result.

[AsamK]

I've added a debug log for this case (19f7b5d)

[johnfreed]

I added a page to the wiki:

https://github.com/AsamK/signal-cli/wiki/Json-service

[pseudocoder]

For your server the nc listener stops when the first connection ends, but it will continue if you put it in a loop like this:

(while true; do nc -l -p 9182 <sigsocket; done) | signal-cli --output=json -u (your number) jsonRpc >sigsocket &

[bbernhard]

Many thanks for that feature! I just gave it a quick try and it works very well. Awesome work!

Just one small thing: Would it be possible to expose the receive command via the json rpc interface too and make it possible to disable the auto-receive?

[exquo]

Maybe caching the messages in virtual memory (tmpfs / ramdisk / dev/shm) rather than on disk would provide the required security?

[johnfreed]

All attachments are already stored to disk at ~/.local/share/signal-cli/attachments, so it's not clear to me how that is any more or less of a security issue.

[andreasbrett]

I also don't see a huge security issue. The signal-cli REST API doesn't offer authentication nor authorization so an attacker could basically just retrieve the message(s) via the REST API remotely rather than having to gain access to the local storage of the messages. I agree though that it's better to have this storage in virtual memory as @exquo suggested.

[pseudocoder]

Bear in mind that high security and acceptable message loss is one end of the spectrum, at the other end you have lower security but a desire to keep messages

Since the location of the storage is clearly defined, I would leave it up to the operator to mount that in whatever location they feel is most appropriate, and not try to build security into signal-cli.

Ditto API security - a proxy can provide almost any security option you need, with in depth testing and frequent patches which simply isn't going to happen in signal-cli. Although it may be useful in signal-cli to have a basic login (or use of the PIN?) for security between individual numbers.

[bbernhard]

My main concern are deleted messages or messages with a disappearing timer set.

e.g

• User writes a message
• The message is received and stored locally
• User deletes the message again
• User asks the REST API via GET request for all the received messages. This would return the message, even it was deleted later.

I haven't tried it yet, but I think signal-cli probably also sends a notification when a message gets deleted. So I guess I could then also delete the message from the local storage. But I think finding the correct message to delete might not be as straight forward.

My "solution" for that is, that the receive endpoint is not exposed as HTTP GET request anymore in json-rpc mode. Instead it will be using websockets.

[pseudocoder]

Doesn’t home assistant support REST for inbound events? So you just need to add that instead of echoing to a file.

For example I have another script which does this for signal-cli, watching a folder for new files and passing them as a JSON-RPC send message. (It’s easy to mount the folder in my containers and just echo into a file, the the actual sending mechanism is independent).

[bbernhard]

Doesn’t home assistant support REST for inbound events? So you just need to add that instead of echoing to a file.

For example I have another script which does this for signal-cli, watching a folder for new files and passing them as a JSON-RPC send message. (It’s easy to mount the folder in my containers and just echo into a file, the the actual sending mechanism is independent).

I did a quick search before, but I haven't found anything. But I'll definitely do some more research tomorrow. 👍

[andreasbrett]

Unfortunately Home Assistant does not have inbound websocket but rather an outbound websocket API. The aforementioned REST API is documented here.

I'd suggest the Home Assistant webhook feature though. I like to use HA webhooks to pass whole JSON objects to HA and then parse them to one or multiple sensors using HA's templating feature. This is also great for NodeRED users as the webhooks work the same there and JSON can be easily used in NodeRED (also: some HA users like me don't use HA automations but rather automate directly via NodeRED).

[bbernhard]

Unfortunately Home Assistant does not have inbound websocket but rather an outbound websocket API. The aforementioned REST API is documented here.

I'd suggest the Home Assistant webhook feature though. I like to use HA webhooks to pass whole JSON objects to HA and then parse them to one or multiple sensors using HA's templating feature. This is also great for NodeRED users as the webhooks work the same there and JSON can be easily used in NodeRED (also: some HA users like me don't use HA automations but rather automate directly via NodeRED).

Thanks a lot for the info! :)

[bbernhard]

Short question regarding the JSON-RPC API: I think the register/verify is currently not supported, right? (At least I am always getting the error User +xxx is not registered. when trying to start signal-cli in JSON-RPC mode.

Is that something that will be supported at some point or does the user first need to register the number in "standalone mode" before enabling JSON-RPC?

[AsamK]

Yes, for now register/verify and linking to an existing account isn't supported by the JSON-RPC mode.
However it might be added in a later version.

[bbernhard]

Perfect, thanks! :)

[piramiday]

hey all, I have been reading up on the JSON-RPC functionality.

dumb question though, how would a user go about and implement this?
what I mean is, the simplest version of a signal bot with classic commands is as follows:

# think of this as a cronjob that periodically
# (e.g. every 10 mins) fetches new messages

# different TMP file for each run
TMPF="messages-$(date +%s).log"

# receive first, e.g.
signal-cli receive > "$TMPF"

# parse and do stuff
./my-signal-parser "$TMPF"

# send result, e.g.
./my-script | signal-cli send +1234567890

# etc

what would the simplest version of the JSON-RPC bot be?
I understand that I can easily send messages, as long as I pipe those JSON RPC commands in the right socket.
how would receiving work?
I do not want an endless program to continuously check the piped output of signal-cli, because I often make changes to such program and I would not be able to get on-the-fly deployment of such new versions.

what was posted in #679 (reply in thread) , that is, saving to a file, seems a relatively clunky solution to me, because then you still have to design a way in which your "message parser" postprocessor reads the message file, without causing issues to the signal-cli process which is writing to it, without double-parsing messages and without dropping messages!

any suggestions are welcome. thanks!

[piramiday]

yes, I understand that, but how would you go and design the process that ingests the data?
forget about networking. you have a piped output into a socket with all received messages, the sent status of sent messages with your ids, and so on.
how do you design the process that parses such data?

[bbernhard]

Not sure if it's the best approach, but in my case I am doing it like this:

• the JSON-RPC writer/reader lives in a dedicated thread
• when I want to send something, I pass the command and the payload to the JSON-RPC worker thread (I am using go channels to communicate between the worker and the main application) which then writes the data to the socket.
• inside the thread I am continuously trying to read data from the socket (I am reading line by line here until I have a JSON message that I can parse). When a new message is received, then the JSON-RPC worker thread hands the received data over to the main thread (I am again using go channels here).
• the main thread now checks whether the received data is a response to a request or some asynchronously received data.

[piramiday]

what happens if your thread hangs, or crashes?
then signal-cli will output data that nobody will fetch and read, and it will be lost since it's not stored.

what happens when you want to update your software?
if you don't stop signal-cli, then you still might lose data.
if you do stop signal-cli, then all the rest of the software that might think it's sending a message by piping to the socket will pipe nowhere and fail at sending.

am I missing something?
thanks for the back and forth, eventually I'll understand how to proceed.

[bbernhard]

what happens if your thread hangs, or crashes?

If my thread hangs I would consider that as a bug then, which needs to be fixed. If signal-cli for some reason crashes, supervisord, automatically restarts it. If my application for some reason panics, the go runtime tries to recover. If that doesn't work and the application crashes, the docker daemon restarts the container.

what happens when you want to update your software?

As everything is containerized, everything that needs to be done is updating the docker image and restarting the container. So it should be pretty fast. But of course, there's a small time span where the application is not available. But in that case it would be noticeable, as the application isn't running and accepting HTTP connections. So, the caller would notice it and could retry a bit later.

[piramiday]

thanks for the replies, but unfortunately this does not answer my questions.
I still don't get how I could design a simple application to make sure I'm not losing messages.

if anyone else has any other ideas, I'd be happy to hear them.

[cuu508]

--> {"jsonrpc":"2.0","method":"send","params":{"recipient":["+YYY"],"message":"MESSAGE"},"id":4}
<-- {"jsonrpc":"2.0","result":{"timestamp":999},"id":4}

It would be neat if there was an option to expose this interface over HTTP (listening on 127.0.0.1 by default) as well.

I haven't yet tried to use JSON-RPC over sockets yet, but I have a couple reservations:

• IIUC the caller program must start signal-cli as a subprocess. signal-cli runs with the same privileges as the caller program
• if signal-cli crashes, the caller must be able to restart it
• multiple long-lived signal-cli processes cannot run at the same time, so I also cannot have multiple callers
• working with sockets is not as common as working with HTTP services

If there was a HTTP interface:

• signal-cli could be run as a different OS user, or from a Docker container, or it could even be on a remote host
• if signal-cli crashes, a process manager (systemd, supervisor etc.) can restart it
• multiple clients can access the HTTP interface concurrently

Thoughts?

[bbernhard]

--> {"jsonrpc":"2.0","method":"send","params":{"recipient":["+YYY"],"message":"MESSAGE"},"id":4}
<-- {"jsonrpc":"2.0","result":{"timestamp":999},"id":4}

It would be neat if there was an option to expose this interface over HTTP (listening on 127.0.0.1 by default) as well.

I haven't yet tried to use JSON-RPC over sockets yet, but I have a couple reservations:

* IIUC the caller program must start signal-cli as a subprocess. signal-cli runs with the same privileges as the caller program

* if signal-cli crashes, the caller must be able to restart it

* multiple long-lived signal-cli processes cannot run at the same time, so I also cannot have multiple callers

* working with sockets is not as common as working with HTTP services

If there was a HTTP interface:

* signal-cli could be run as a different OS user, or from a Docker container, or it could even be on a remote host

* if signal-cli crashes, a process manager (systemd, supervisor etc.) can restart it

* multiple clients can access the HTTP interface concurrently

Thoughts?

I am not arguing against a HTTP interface here, but aren't the above points all solvable with the TCP socket interface too?

[cuu508]

Ah, sorry, I should have experimented with this myself before commenting. I assumed the parent process would spawn signal-cli and then write to its stdin and read from its stdout.

[AsamK]

I've extend the daemon command with a socket JSON-RPC interface, see this discussion for details: #799

[Rosch]

JSON RPC is great extension! Thank you.

I yet could not find an example on how to send to a group via JSON RPC
Thanks for a hint :-)

[AsamK]

{"jsonrpc":"2.0","method":"send","params":{"groupId":"base64-id","message":"MESSAGE"},"id":4}

[Rosch]

Thank you very much. This works!

[KevinRoebert]

Is there a way to upload attachments via the JSON-RPC API (e.g. images)?

[AsamK]

Currently the only way to add attachments is by specifying a file path that the signal-cli daemon can access.
I've thought about allowing base64 encoded data as well, but that's not implemented yet.

[KevinRoebert]

Ah, I see. I am currently sitting on an implementation.

[KevinRoebert]

@AsamK Created a PR that added base64 attachment support: #966