cli_wallet crashes when connection to witness node is lost #177
Comments
|
From @ByronAP on April 30, 2016 14:6 on windows the cli wallet doesn't crash but rpc requests fail with invalid state and the wallet needs to be restarted to reconnect. This has been an issue for some time and should be priority. I agree with the round robin idea. |
|
I was after this issue in the last few days, i know is old but the problem described is still relevant. At first, i didn't agree with the round robin idea. I don't think we need to disconnect every X seconds and connect to a new node if the main one is working. My first approach to this was to add a command line option "--backup-servers" expecting a vector of nodes that the client may use in case the main one is failing. So the idea was: If main node is disconnected -> use the first backup server. I started to code this in the cli wallet client, had some problems i have to admit but that actually made me think on an alternative way to accomplish the same thing. The less code we introduce to the core the less bugs we may have and this is also true for the cli_wallet. The cli_wallet is actually acting a server for other apps connect to it so we want to keep it bug free. The new code added to bitshares need to be reviewed , tested and finally merged. This takes time but it is the correct way to go, with rigorous developer community analysis of new introduced code. Unless there is no alternative way, we only then introduce new code. If it can be resolved by alternative ways then we expose them and use that instead. This is a kind of no written law ;) This is no laziness from my side here, in fact, find an alternative method to go over this particular issue actually toke the same or more time than adding the command line option to the core itself. With the introduction said, lets go over the alternative way to go over this disconnection issue. Problem is a cli_wallet connected to a witness node loosing connection to this witness. Cli_wallet will crash and applications depending on it too. The rich linux environment should have a solution for this networking problem. Yes, i think is a networking issue, not a cli_wallet problem. Tried some python subprocess libs but i had no luck, tried some other alternatives before finding what i think it can be the best. Meet expect(http://expect.sourceforge.net/), a tool designed to interact with interactive programs. The cli_wallet is an interactive process, where user send commands and program send a response to the console. Expect can be simple installed(if it is not already there) in our favorite debian based box by simply:
My approach to expect was because when the node stops the cli wallets terminate always with msg:
My idea was to catch(expect) this msg and spawn new cli_wallet when the disconnection message is shown. Expect is scriptable, so a file with commands will be now in charge of launching the cli_wallet. For simplicity we are going to just use 2 node servers: X.X.X.X:8090 -> main node server The same concept can be extended to more than 2 severs, actually with any number of backup nodes a user will want. Expect is not easy to use, it was not easy to get what you need done. There is plenty of information , questions and answers in the internet about it but to do complex stuff the most valuable resources are the man page of expect and a book named "Exploring Expect" by Don Libes published by O`Reilly. Lets suppose that our cli_wallet command is something like:
This start the cli wallet while it expose an HTTP RPC port at 127.0.0.1:6666 This is used for example as:
As server X.X.X.X:8090 gets down port 6666 will be closed and commands will not work anymore. Check the following expect script exp3.exp: Make the file executable:
Run simply as:
Start with debug info by:
As we are interested on the cli just to serve in port 6666 we can't interact as a user with the cli_wallet. This can be changed by the use of "interact" keyword in our script. Interact will allow to actually use the cli the expect script launched but i left that outside as my intention was not to interact but just to serve at port 6666. The script will run the first command until eof(end of program in this case) is found. Cli_wallet(and probably all programs) will send eof when they end(by disconnection in this case). We don't need to be after specific "Disconnected" string as i tough in the begging. At the moment of EOF of the first wallet, script will spawn the second command. When the second cli_wallet command is executed port 6666 will be available again for curl but this time we will be pulling data from Y.Y.Y.Y:8060 instead of X.X.X.X:8090 because main node is down. Please note that exposing a web socket instead of an http one(this is with option -r instead of -H in the cli starting command) can be more tricky. Tools as wscat will detect the disconnection and kick the user off even if the port 6666 is available again some seconds later. This setup will need some more work but it is still possible with expect itself to regain websocket connection. I hope this helps not only for this issue in particular but to also for similar ones that can be resolved with expect. A tool that i learned to use while writing this and that i am sure it will be part of my nodemaster arsenal, side by side with screen, wscat, upticks and many others. |
|
in telegram we discussed this issue with @abitmore and @pmconrad today. This is true, but it can be adapted, consider the following expect script: This will allow to interact with the first node and when disconnected it will start a new one. Please check the following script output: As you can see above as soon as we disconnect from the first node we open a new process to the backup one. In regards to the history, we are kind of dead there with this alternative. The history is not dumped to file but lives fully in process memory. As initial process is closed we don't have access to that memory anymore in the second. In order to keep history across sessions we may need to dump it to a local file but that i suppose will be a subject of another issue. |
|
Same can be done by wscat with the following expect script, maybe someone find it useful: When disconnected script opens new websocket to alternative server, command history is lost just as i n the cli_wallet previous sample, it is a new process: |
|
i am closing this issue as there are workarounds for this with tools outside the project to handle the situation and there is no further complains about the subject, issue is also very old. |
|
just an advice.
then I told to my self you need this fantastic websocket command line client called websocat here is an article that teached me how to script it without expect or any external utility (using only bash). it works super : https://stackoverflow.com/questions/48912184/wscat-commands-from-script-how-to-pass here i past the script maybe the article get deleted :
#!/usr/bin/env bash
runscript() {
commands=( "first command" "second command" "third command" )
for command in "${commands[@]}"; do
echo "Writing command to server" >&2
echo "$command"
echo "Reading response from server (assuming exactly one line)" >&2
read -r line
echo "Received response: $line" >&2
done
# kill websocat, even if the websocket doesn't get closed
kill "$PPID"
}
export -f runscript
websocat ws://echo.websocket.org sh-c:'exec bash -c runscript'and websocat have a simple mode and an advanced mode both can be used in scripts. $ echo test | websocat ws://echo.websocket.org
test sorry for my english :/ |
From @valzav on February 18, 2016 15:15
It's a big issue for the faucet - when the main witness node (out of 4) crashes it makes cli_wallet to crash and this stops faucet from registering new accounts.
It would be nice if cli_wallet could try to reconnect to several witness_node ws sockets in round robin manner.
It can read a list of available witness_node from command line arguments.
Copied from original issue: cryptonomex/graphene#591
The text was updated successfully, but these errors were encountered: