Samourai-Wallet · kenshin-samourai · Feb 17, 2020 · Nov 7, 2019 · Nov 7, 2019 · Nov 7, 2019
diff --git a/README.md b/README.md
@@ -17,7 +17,8 @@ It provides in a single command the setup of a full Samourai backend composed of
 * the backend database,
 * the backend modules with an API accessible as a static Tor hidden service,
 * a maintenance tool accessible through a Tor web browser,
-* a block explorer ([BTC RPC Explorer](https://github.com/janoside/btc-rpc-explorer)) accessible through a Tor web browser.
+* a block explorer ([BTC RPC Explorer](https://github.com/janoside/btc-rpc-explorer)) accessible through a Tor web browser,
+* an optional indexer of Bitcoin addresses ([addrindexrs](https://github.com/Samourai-Wallet/addrindexrs)) providing fast and private rescans of HD accounts and loose addresses.
 
 See [the documentation](./doc/DOCKER_setup.md) for detailed setup instructions.
 
@@ -61,13 +62,16 @@ Authentication is enforced by an API key and Json Web Tokens.
 
 **Import of HD Accounts and data sources**
 
-* First import of an unknown HD account relies on a data source (local bitcoind or OXT). After that, the tracker will keep everything current.
+* First import of an unknown HD account relies on a data source (local bitcoind, local indexer or OXT). After that, the tracker will keep everything current.
 
-* Default option relies on the local bitcoind and makes you 100% independent of Samourai Wallet's infrastructure. This option is recommended for better privacy.
+* Using the local bitcoind (default option) or the local indexer makes you 100% independent of Samourai Wallet's infrastructure and is recommended for better privacy.
 
 * Activation of bitcoind as the data source:
   * Edit /keys/index.js and set "indexer.active" to "local_bitcoind". OXT API will be ignored.
 
+* Activation of the local indexer as the data source:
+  * Edit /keys/index.js and set "indexer.active" to "local_indexer". OXT API will be ignored.
+
 * Activation of OXT as the data source (through socks5):
   * Edit /keys/index.js and set "indexer.active" to "third_party_explorer".
 
@@ -77,5 +81,8 @@ Authentication is enforced by an API key and Json Web Tokens.
   * It's slightly slower than using the option relying on the OXT API.
   * It may fail to correctly import an existing wallet if this wallet had a large activity.
   * If you use bitcoind and if the import seems to return an invalid balance, you can use the "XPUB rescan" function provided by the maintenance tool. This function allows you to force the minimum number of addresses to be derived and the start index for the derivation.
-  * As a rule of thumb, we recommend to use bitcoind as the source of imports and to setup your Dojo with a new clean wallet. It increases your privacy and it removes all potential issues with the import of a large wallet.
-
+
+* Main drawbacks of using your local indexer for these imports:
+  * It requires 120GB of additional disk space during its initialization.
+
+As a rule of thumb, we recommend to use the local indexer as the source of imports and to setup your Dojo with a new clean wallet. It increases your privacy and it removes all potential issues with the import of a large wallet.
diff --git a/doc/DOCKER_advanced_setups.md b/doc/DOCKER_advanced_setups.md
@@ -3,17 +3,114 @@
 
 The configuration files of Dojo provide a few advanced options allowing to tune your setup.
 
-A word of caution, though, the default values of these options try to maximize your privacy at a network level. Most of the advanced setups described in this document may damage your privacy. Use at your own risk!
+A word of caution, though, the default values of these options try to maximize your privacy at a network level. Some of the advanced setups described in this document may damage your privacy. Use at your own risk!
 
 
 ## Table of Content ##
+- [Local indexer of Bitcoin addresses](#local_indexer)
+- [Local Electrum server used as data source for imports/rescans](#local_electrum)
 - [External Bitcoin full node](#external_bitcoind)
 - [bitcoind RPC API ans ZMQ notifications exposed to external apps](#exposed_rpc_zmq)
 - [Static onion address for bitcoind hidden service](#static_onion)
 - [Configure Tor Bridges](#tor_bridges)
 - [Support of testnet](#testnet)
 
 
+<a name="local_indexer"/>
+
+## Local indexer of Bitcoin addresses ##
+
+By default, Dojo uses the local full node as its data source for imports and rescans of HD accounts and addresses. While private, this default option has many limitations. MyDojo allows to install a local indexer ([addrindexrs](https://github.com/Samourai-Wallet/addrindexrs)) providing the best of both worlds (no request sent to a third party, fast and real time rescans, complete transactional history is retrieved).
+
+
+### Requirements ###
+
+To date, the initial installation of the indexer requires 120GB of additionnal disk space.
+
+
+### Main benefits ###
+
+- Fast, private and exhaustive real time rescans,
+- Allows the block explorer to display the detailed activity of Bitcoin addresses
+
+
+### Known drawbacks ###
+
+* Additionnal disk space consumed by the index,
+* Increased duration of upgrades (from 5 to 20 minutes depending on the machine hosting your Dojo),
+* Slight increase of startup duration,
+* First indexation will require a few hours.
+
+
+### Procedure ###
+
+```
+# Edit the indexer config template file
+nano ./conf/docker-indexer.conf.tpl
+
+#
+# Set the value of INDEXER_INSTALL to "on"
+# Save and exit nano
+#
+
+# Edit the nodejs config file (or the corresponding template file if it's your first installation of Dojo)
+nano ./conf/docker-node.conf
+
+#
+# Set the value of NODE_ACTIVE_INDEXER to "local_indexer"
+# Save and exit nano
+#
+
+#
+# Launch the installation or the upgrade of your Dojo
+# with the commands `dojo.sh install` or `dojo.sh upgrade`
+#
+
+#
+# Be patient!
+# First indexation of all Bitcoin addresses will require a few hours.
+# Let the indexer complete all these operations before trying to use it for an import or a rescan.
+# You can follow the progress made by the indexer with the commands:
+#   `dojo.sh logs`
+# or
+#   `dojo.sh logs indexer`
+#
+```
+
+
+<a name="local_electrum"/>
+
+## Local Electrum server used as data source for imports/rescans ##
+
+If you're running an instance of ElectrumX or Electrs on your local network, Dojo allows you to define this instance as the data source used for imports and rescans. This setup is an alternative to the local indexer provided by MyDojo.
+
+Important: Do not use an Electrum server operated by a third party or hosted on a different local network.
+
+
+### Procedure ###
+
+```
+# Edit the indexer config template file
+nano ./conf/docker-indexer.conf.tpl
+
+#
+# Set the value of INDEXER_INSTALL to "off"
+# Set the value of INDEXER_IP with the IP address of your Electrum server
+# Set the value of INDEXER_RPC_PORT with the port used by the RPC API of your Electrum server (default= 50001)
+# Set the value of INDEXER_BATCH_SUPPORT to "active" if your Electrum server is ElectrumX, otherwise set the value to "inactive"
+# Save and exit nano
+#
+
+# Edit the nodejs config file (or the corresponding template file if it's your first installation of Dojo)
+nano ./conf/docker-node.conf
+
+#
+# Set the value of NODE_ACTIVE_INDEXER to "local_indexer"
+# Save and exit nano
+#
+```
+
+
 <a name="external_bitcoind"/>
 
 ## External Bitcoin full node ##

diff --git a/doc/DOCKER_setup.md b/doc/DOCKER_setup.md
@@ -6,6 +6,7 @@ MyDojo is a set of Docker containers providing a full Samourai backend composed
 * backend modules with an API accessible as a static Tor hidden service,
 * a maintenance tool accessible through a Tor web browser,
 * a block explorer ([BTC RPC Explorer](https://github.com/janoside/btc-rpc-explorer)) accessible as a static Tor hidden service.
+* an optional indexer of Bitcoin addresses ([addrindexrs](https://github.com/Samourai-Wallet/addrindexrs)) providing fast and private rescans of HD accounts and loose addresses.
 
 
 ## Table of Content ##
@@ -40,33 +41,39 @@ MyDojo is a set of Docker containers providing a full Samourai backend composed
                   Host machine                  | (Tor hidden services)
                  ______________________________ | _____________________________
                 |                               |                              |
-                |                      -------------------           dmznet    |
-                |                     |   Tor Container   |                    |
-                |                      -------------------                     |
-                |                             |        |                       |
-                |             -------------------      |                       |
-                |            |  Nginx Container  |     |                       |
-                |             -------------------      |                       |
-                |- - - - - - - - - - - | - - -|- - - - | - - - - - - - - - - - |
-                |     --------------------    |     --------------------       |
-                |    |  Nodejs Container  | ------ | Bitcoind Container |      |
-                |     --------------------    |     --------------------       |
-                |               |             |               |                |
-                |     --------------------    |     --------------------       |
-                |    |  MySQL Container   |   ---- |  BTC RPC Explorer  |      |
-                |     --------------------          --------------------       |
-                |                                                              |
-                |                                                  dojonet     |
+                |                           ---------                  dmznet  |
+                |                  --------|   Tor   |------------             |
+                |                 |         ---------             |            |
+                |                 |                               |            |
+                |             ---------                           |            |
+                |          --|  Nginx  |--------                  |            |
+                |         |   ---------         |                 |            |
+                |- - - - -|- - - - - - - - - - -|- - - - - - - - -|- - - - - - |
+                |         |                     |                 |            |
+                |     ----------            ----------        ----------       |
+                |    |  Nodejs  |----------| Explorer |------| Bitcoind |      |
+                |     ----------            ----------        ----------       |
+                |         |   |                 |                 |            |
+                |         |    -------          |                 |            |
+                |         |           |         |                 |            |
+                |     ----------      |     ----------            |            |
+                |    |  MySQL   |      ----|  Indexer |-----------             |
+                |     ----------            ----------                         |
+                |                                                      dojonet |
                 |______________________________________________________________|
 
 
+
+
+
+
 <a name="requirements"/>
 
 ## Requirements ##
 
 * A dedicated computer (host machine) connected 24/7 to internet
 * OS: Linux is recommended
-* Disk: 500GB (minimal) / 1TB (recommended) - SSD is recommended
+* Disk: 600GB (minimal) / 1TB (recommended) - SSD is recommended
 * RAM: 4GB (minimal)
 * Docker and Docker Compose installed on the host machine (be sure to run a recent version supporting v3.2 of docker-compose files, i.e. Docker Engine v17.04.0+)
 * Check that the clock of your computer is properly set (required for Tor)
@@ -77,10 +84,11 @@ MyDojo is a set of Docker containers providing a full Samourai backend composed
 
 ## Configuration files ##
 
-Each new release of Dojo is packaged with 6 template files stored in the `<dojo_dir>/docker/my-dojo/conf` directory:
+Each new release of Dojo is packaged with 7 template files stored in the `<dojo_dir>/docker/my-dojo/conf` directory:
 - docker-common.conf.tpl
 - docker-bitcoin.conf.tpl
 - docker-explorer.conf.tpl
+- docker-indexer.conf.tpl
 - docker-mysql.conf.tpl
 - docker-node.conf.tpl
 - docker-tor.conf.tpl
@@ -106,6 +114,9 @@ For MacOS, see this detailed [installation guide](./DOCKER_mac_setup.MD).
 
 For Synology, see this detailed [installation guide](./DOCKER_synology_setup.md).
 
+For Raspberry Pi4 and Odroid N2, see the [Ronin Dojo Project](https://github.com/RoninDojo/RoninDojo)
+
+
 This procedure allows to install a new Dojo from scratch.
 
 * Install [Docker and Docker Compose](https://docs.docker.com/compose/install/) on the host machine and check that your installation is working.
@@ -146,7 +157,9 @@ This procedure allows to install a new Dojo from scratch.
       * If you want to deactivate the block explorer, set the value of `EXPLORER_INSTALL` to `off`.
     See this [section](#explorer) for more details about the block explorer.
 
-* Dojo provides a few additional settings for advanced setups: 
+* Dojo provides a few additional settings for advanced setups:
+  * installation of an address indexer used for fast imports and rescans,
+  * support of an external electrum server (ElectrumX or electrs) used for fast imports and rescans,
   * static onion address for your full node,
   * bitcoind RPC API exposed to external apps,
   * use of an external full node,
@@ -239,6 +252,7 @@ Available commands:
                                   dojo.sh logs bitcoind       : display the logs of bitcoind
                                   dojo.sh logs db             : display the logs of the MySQL database
                                   dojo.sh logs tor            : display the logs of tor
+                                  dojo.sh logs indexer        : display the logs of the internal indexer
                                   dojo.sh logs api            : display the logs of the REST API (nodejs)
                                   dojo.sh logs tracker        : display the logs of the Tracker (nodejs)
                                   dojo.sh logs pushtx         : display the logs of the pushTx API (nodejs)

diff --git a/docker/my-dojo/.env b/docker/my-dojo/.env
@@ -16,7 +16,8 @@ DOJO_BITCOIND_VERSION_TAG=1.4.1
 DOJO_NODEJS_VERSION_TAG=1.4.0
 DOJO_NGINX_VERSION_TAG=1.4.0
 DOJO_TOR_VERSION_TAG=1.3.0
-DOJO_EXPLORER_VERSION_TAG=1.0.0
+DOJO_EXPLORER_VERSION_TAG=1.1.0
+DOJO_INDEXER_VERSION_TAG=1.0.0
 
 
 #########################################
@@ -58,3 +59,10 @@ NODE_PREFIX_STATUS_PUSHTX=status
 
 NODE_TRACKER_MEMPOOL_PERIOD=10000
 NODE_TRACKER_UNCONF_TXS_PERIOD=300000
+
+
+#########################################
+# INDEXER
+#########################################
+
+INDEXER_BATCH_SIZE=10
diff --git a/docker/my-dojo/conf/docker-indexer.conf.tpl b/docker/my-dojo/conf/docker-indexer.conf.tpl
@@ -0,0 +1,42 @@
+#########################################
+# CONFIGURATION OF A LOCAL INDEXER
+#########################################
+
+# Install and run a local indexer inside Docker
+# Set this option to 'off' for using an indexer hosted outside of Docker
+# or when using a different data source (local bitcoind, OXT)
+# Value: on | off
+INDEXER_INSTALL=off
+
+# IP address of the local indexer used by Dojo
+# Set value to 172.28.1.6 if INDEXER_INSTALL is set to 'on'
+# Type: string
+INDEXER_IP=172.28.1.6
+
+# Port of the RPC API
+# Set value to 50001 if INDEXER_INSTALL is set to 'on'
+# Type: integer
+INDEXER_RPC_PORT=50001
+
+# Support of batch requests by the local indexer
+# Set value to inactive if INDEXER_INSTALL is set to 'on'
+# Value: active | inactive
+INDEXER_BATCH_SUPPORT=inactive
+
+
+#
+# EXPERT SETTINGS
+# (ACTIVE IF INDEXER_INSTALL IS SET TO ON)
+#
+
+# Number of blocks to get in one JSONRPC request from bitcoind
+# Type: integer
+INDEXER_BATCH_SIZE=10
+
+# Total size of block txids to cache (in MB)
+# Type: integer
+INDEXER_BLK_TXIDS_CACHE_SIZE_MB=10
+
+# Number of transactions to lookup before returning an error
+# Type: integer
+INDEXER_TXID_LIMIT=501
diff --git a/docker/my-dojo/conf/docker-node.conf.tpl b/docker/my-dojo/conf/docker-node.conf.tpl
@@ -21,7 +21,7 @@ NODE_ADMIN_KEY=myAdminKey
 NODE_JWT_SECRET=myJwtSecret
 
 # Indexer or third-party service used for imports and rescans of addresses
-# Values: local_bitcoind | third_party_explorer
+# Values: local_bitcoind | local_indexer | third_party_explorer
 NODE_ACTIVE_INDEXER=local_bitcoind
 
 # FEE TYPE USED FOR FEES ESTIMATIONS BY BITCOIND

diff --git a/docker/my-dojo/docker-compose.yaml b/docker/my-dojo/docker-compose.yaml
@@ -31,6 +31,7 @@ services:
       - ./conf/docker-mysql.conf
       - ./conf/docker-bitcoind.conf
       - ./conf/docker-explorer.conf
+      - ./conf/docker-indexer.conf
       - ./conf/docker-node.conf
     restart: always
     command: "/home/node/app/wait-for-it.sh db:3306 --timeout=720 --strict -- /home/node/app/restart.sh"