Blockchain pruning with JoinMarket

[chris-belcher]

The release notes say that

With 0.12 it is possible to use wallet functionality in pruned mode. This can reduce the disk usage from currently around 60 GB to around 2 GB.
However, rescans as well as the RPCs importwallet, importaddress, importprivkey are disabled.

JoinMarket uses importaddress with the rescan flag false,, which could work with pruning as long as you know for sure the addresses are empty (and JoinMarket guarantees they are.) So in principle that RPC call doesn't have to be disabled entirely.

domob uses a pruned node here. On careful reading you see that even then importaddress didn't work.

(By the way, I'm actually using a pruned node with JoinMarket on a VPS right now. It works fine in normal operation, but I do have nodes with full history ready that I can use for importing more addresses when necessary.)

[chris-belcher]

According to this, we're fine: https://np.reddit.com/r/Bitcoin/comments/46f67s/bitcoin_v0120_has_been_tagged_for_release/d04o6n6?context=1

After someone tries this I can close the issue.

[dreboli]

Sendpayment and yield-generator-basic works with a pruned node on a Raspberry Pi B+
Sendpayment.py successfully sent some coins.
Yield-generator-basic.py was a maker in a finished tx.
Tumble.py wasn't able to finish but it made 1 successful tx, so I think the problem is that the raspi can't take the load.

$ uname -a
Linux patent 4.1.17+ #838 Tue Feb 9 12:57:10 GMT 2016 armv6l GNU/Linux

$ bitcoind -version
Bitcoin Core Daemon version v0.12.99.0-a08c41d

$ bitcoind -prune=550 -daemon -proxy=127.0.0.1:9050 -datadir=/home/pi/.bitcoin/ -dbcache=1 -checkblocks=1

$ cat /home/pi/.bitcoin/bitcoin.conf
server=1
listen=0
rpcuser=xxx
rpcpassword=xxx

#joinmarket
walletnotify=wget -q --spider --timeout=5 --tries=1 http://127.0.0.1:62602/walletnotify?%s
alertnotify=wget -q --spider --timeout=5 --tries=1 http://127.0.0.1:62602/alertnotify?%s

$ tor --version
Tor version 0.2.5.12 (git-3731dd5c3071dcba).

Joinmarket-0.1.3

[chris-belcher]

Thanks @dreboli
Did you ever use importaddress ? For example when creating a new empty wallet.

[dreboli]

When importing some new wallet, joinmarket instructs to run bitcoin core with -rescan option

$ python wallet-tool.py somenewwallet.json
2016-02-27 16:03:45,665 [MainThread  ] [DEBUG]  hello joinmarket
Enter wallet decryption passphrase: 
2016-02-27 16:03:51,312 [MainThread  ] [DEBUG]  requesting wallet history
2016-02-27 16:03:56,579 [MainThread  ] [DEBUG]  rpc: getaddressesbyaccount ['joinmarket-wallet-xxx']
2016-02-27 16:03:56,588 [MainThread  ] [DEBUG]  importing 200 addresses into account joinmarket-wallet-xxx
restart Bitcoin Core with -rescan if you're recovering an existing wallet from backup seed otherwise just restart this joinmarket script

But bitcoin-qt -prune=550 -rescan throws an error:

$ bitcoin-qt -prune=550 -rescan
"Rescans are not possible in pruned mode. You will need to use -reindex which will download the whole blockchain again."

I didn't want to download the whole blockchain again just to prune it yet again. This way the imported wallet shows 0 balance even though it's not empty. Only new transactions (that happened after the wallet was imported) show up. (not sure what happens when the gap limit has to be raised though)

Luckily I had a normal, unpruned blockchain on my desktop PC, so I made a copy of the whole ~/.bitcoin directory that also held a wallet file that already had every joinmarket wallet imported.

Then pruned the copied directory on my desktop PC

bitcoin-qt -prune=550 -datadir=~/.bitcoin-pruned

The size of the directory decreased to about 2GB.

Then copied the directory containing the now pruned blockchain to my pi.
This way the raspi did not have to bother with pruning and syncing the whole blockchain, the whole thing could be done on my faster desktop PC much faster.

I'm not sure what happens when the joinmarket-wallet-xxx account uses up all the already imported addresses so the gap limit has to be raised in order to have new addresses. Will this lead to problems?

[raedah]

Id be curious to hear about testing with -blocksonly mode as well.

[chris-belcher]

I'm not sure what happens when the joinmarket-wallet-xxx account uses up all the already imported addresses so the gap limit has to be raised in order to have new addresses. Will this lead to problems?

When the wallet creates more addresses than were imported, it will import new addresses. Since they're guaranteed to be empty rescan is not needed. https://github.com/JoinMarket-Org/joinmarket/blob/master/joinmarket/wallet.py#L229-L236 Rescan is only needed when recovering a wallet from backup.

It looks like the importaddress RPC call with rescan=false works perfectly fine in pruned mode, which is what I was interested in, which means JoinMarket can indeed run in pruned mode (As long as you don't recover a wallet from backup) (so this issue can be closed after some time)

For blocksonly, I just opened #436 which I think it necessary for it to work.

[chris-belcher]

My own node now runs with pruning enabled. Here are some notes since it wasn't absolutely trivial to do this.

• I used Bitcoin Core 0.11. Recently my hard disk partition ran out of space, seems like a good time to update to 0.12.
• Installed 0.12, ran with pruning enabled.
• It synchronized but had dead unconfirmed transactions from November 2015 (the mallebaility attack?) which messed up JoinMarket by making it believe it had more bitcoins than it really did.
• I read about -zapwallettxes but it cant be run with pruning enabled. Also I don't have the disk space to sync without pruning.
• For some reason I decide against -zapwallettxes and try to sync from scratch (to see if it can be done) So I delete wallet.dat
• Rename .bitcoin/blocks/ to .bitcoin/block2/ so the node will have to sync from the start, run it with -reindex -prune=0 -connect=0.0.0.0 so sync'ing will be paused. Then run JoinMarket's wallet-tool to import all of my watch-only addresses.
• Restart the node, running with the normal -prune=N, .bitcoin/blocks2/ renamed back to .bitcoin/blocks/ and let it sync normally. After it's finished, my JoinMarket wallet is working correctly.
• Some time could be saved by using -zapwallettxes to save yourself having to import all the JoinMarket wallet addresses.

[dreboli]

It synchronized but had dead unconfirmed transactions from November 2015 (the mallebaility attack?) which messed up JoinMarket by making it believe it had more bitcoins than it really did.

Experienced similar thing after updating from 0.11.2 to 0.12, the wallet showed some extra bitcoins under Overview - Balances - Pending in bitcoin-qt. These extra coins were on old used joinmarket addresses. Did a -rescan (or -reindex, don't remember) to solve it.