Dev (#292)

* Add oracles cache (#238) * Add oracles cache * Review fix * Use separate tasks for main functions (#230) * Use separate tasks for main functions Signed-off-by: cyc60 <avsysoev60@gmail.com> * Refactored func names Signed-off-by: cyc60 <avsysoev60@gmail.com> * Run HarvestTask only if needed Signed-off-by: cyc60 <avsysoev60@gmail.com> * Review fix Signed-off-by: cyc60 <avsysoev60@gmail.com> * Move interrupt_handler processing to tasks Signed-off-by: cyc60 <avsysoev60@gmail.com> --------- Signed-off-by: cyc60 <avsysoev60@gmail.com> * Move ipfs retries to client (#252) * Move ipfs retries to client * Fix test * Add settings for genesis validators ipfs client * Increase default GENESIS_VALIDATORS_IPFS_RETRY_TIMEOUT * Refactored keystores (#249) * Refactored keystores Signed-off-by: cyc60 <avsysoev60@gmail.com> * Fix tests Signed-off-by: cyc60 <avsysoev60@gmail.com> * Remove unused comment Signed-off-by: cyc60 <avsysoev60@gmail.com> * Remove double log Signed-off-by: cyc60 <avsysoev60@gmail.com> * Update packages in prod build (#256) * Update packages in prod build * Fix * Remove goerli network (#257) Signed-off-by: cyc60 <avsysoev60@gmail.com> --------- Signed-off-by: cyc60 <avsysoev60@gmail.com> Co-authored-by: antares-sw <23400824+antares-sw@users.noreply.github.com> * Log format (#260) * Add json log formatter Signed-off-by: cyc60 <avsysoev60@gmail.com> * Poetry update Signed-off-by: cyc60 <avsysoev60@gmail.com> * Use record.created for json logs Signed-off-by: cyc60 <avsysoev60@gmail.com> * Review fixes Signed-off-by: cyc60 <avsysoev60@gmail.com> * Review fix Signed-off-by: cyc60 <avsysoev60@gmail.com> * Remove int conversion Signed-off-by: cyc60 <avsysoev60@gmail.com> --------- Signed-off-by: cyc60 <avsysoev60@gmail.com> * Add log envs to example (#262) Signed-off-by: cyc60 <avsysoev60@gmail.com> * Fixes after merge * Rework signature rotation parallel (#273) * Add flake8-datetimez plugin (#274) * Add flake8-datetimez plugin * Fix poetry.lock after merge * Update execution.py (#284) Signed-off-by: Dmitri Tsumak <tsumak.dmitri@gmail.com> * Add setup_logging (#285) * Add execution_transaction_timeout setting (#272) Signed-off-by: cyc60 <avsysoev60@gmail.com> * Update README.md (#275) Signed-off-by: joethechicken <143952437+joethechicken@users.noreply.github.com> * Add latest on tag (#276) * Add latest on tag * Empty * Fix * Fix lint * Release v1.0.8 (#277) Signed-off-by: cyc60 <avsysoev60@gmail.com> * Add permissions to token (#279) * Add setup_logging (#283) * Fix after merge * Fix pip-audit --------- Signed-off-by: cyc60 <avsysoev60@gmail.com> Signed-off-by: joethechicken <143952437+joethechicken@users.noreply.github.com> Co-authored-by: Alexander Sysoev <avsysoev60@gmail.com> Co-authored-by: joethechicken <143952437+joethechicken@users.noreply.github.com> Co-authored-by: antares-sw <23400824+antares-sw@users.noreply.github.com> * Rework remote signer (#288) * Add signature shares logic * Del remote signer config * Add --log-level arg * Review fixes, fix timeout * Adapt remote_db * Adapt tests * Review fixes 2 * Review fixes 2 * Fix tests * Add interruptible sleep (#290) * Add --pool-size arg to start command (#291) * Add --pool-size arg to start command * Del annotations * Del pip-audit ignore (#294) * Rework remote db (#293) * Del parent-public-key in remote-db * Rename test_tasks -> test_commands * Fixes * Add web3_log_level setting (#295) * Add api for obol (#280) * Add api for obol * Uvicorn logging and signals * Fix log message * Fix lint * Add test_exit_signature_shards_without_keystore * Add run_check_deposit_data_root * Fix pydantic version * Fix naming, add docstring * Move endpoints.py to api/ * Del skip_validator_registration_tx * Use secrets * Warm up oracles cache * Del todo * Add register_and_remove_pending_validators * Fixes after merge * Fix pip-audit starlette * Add start-api command * Review fixes * Move signatures encryption out of keystore * Fix grammar * Fix tests * Moved get_exit_signature_shards out of keystore * Revert "Fix tests" This reverts commit 97154d7. * Adapt tests * Del enable_api * Rename tests * Moved hashi vault tests * Rename test_signing.py -> test_common.py * Review fixes * Rename approvals -> registration requests * Fix slicing * goerli to holesky rename (#297) Signed-off-by: exe.cute <98374381+execute237@users.noreply.github.com> * Fix tx confirm (#300) * Fix _fetch_outdated_indexes (#302) * Fix pip-audit (#303) * Fix validators check (#304) * Add priority fee (#301) * Fix tx confirm * Add priority fee setting * Del custom priority fee from harvest tx * Fix custom_priority_fee * Add max-priority-fee to network config * Revert "Add max-priority-fee to network config" This reverts commit 141d349. * Del max_priority_fee_per_gas_gwei setting * Del _calculate_median_priority_fee * Review fixes * Add fee settings * Add rounding * Bump version v1.1.0 (#305) * Bump version v1.1.0 * Upd version everywhere * Docker Hub (#296) (#306) Co-authored-by: antares-sw <23400824+antares-sw@users.noreply.github.com> --------- Signed-off-by: cyc60 <avsysoev60@gmail.com> Signed-off-by: Dmitri Tsumak <tsumak.dmitri@gmail.com> Signed-off-by: joethechicken <143952437+joethechicken@users.noreply.github.com> Signed-off-by: exe.cute <98374381+execute237@users.noreply.github.com> Co-authored-by: evgeny-stakewise <123374581+evgeny-stakewise@users.noreply.github.com> Co-authored-by: antares-sw <23400824+antares-sw@users.noreply.github.com> Co-authored-by: Evgeny Gusarov <evgeny@stakewise.io> Co-authored-by: Dmitri Tsumak <tsumak.dmitri@gmail.com> Co-authored-by: joethechicken <143952437+joethechicken@users.noreply.github.com> Co-authored-by: exe.cute <98374381+execute237@users.noreply.github.com>
stakewise · Mar 19, 2024 · 2d8f3c0 · 2d8f3c0
1 parent 642c2b5
commit 2d8f3c0
Show file tree

Hide file tree

Showing 63 changed files with 2,695 additions and 1,985 deletions.
diff --git a/.env.example b/.env.example
@@ -21,7 +21,7 @@ CONSENSUS_ENDPOINTS=http://localhost:3500
 # METRICS_HOST=127.0.0.1
 # METRICS_PORT=9100
 
-# The network of the Vault. Choices are: mainnet, gnosis, goerli
+# The network of the Vault. Choices are: mainnet, gnosis, holesky
 # Default value will be taken from vault config
 # NETWORK=mainnet
 
@@ -68,3 +68,8 @@ CONSENSUS_ENDPOINTS=http://localhost:3500
 # HOT_WALLET_FILE=/home/user/.stakewise/${VAULT_CONTRACT_ADDRESS}/wallet/wallet.json
 # Default is ${DATA_DIR}/${VAULT_CONTRACT_ADDRESS}/wallet/password.txt
 # HOT_WALLET_PASSWORD_FILE=/home/user/.stakewise/${VAULT_CONTRACT_ADDRESS}/wallet/password.txt
+
+# Log level
+# LOG_LEVEL=INFO
+# Log record format. Can be "plain" or "json". Default is "plain"
+# LOG_FORMAT=plain
diff --git a/.github/workflows/ci.yaml b/.github/workflows/ci.yaml
@@ -122,4 +122,3 @@ jobs:
       - uses: pypa/gh-action-pip-audit@v1.0.7
         with:
           inputs: audit_requirements.txt
-          ignore-vulns: PYSEC-2022-43059
diff --git a/README.md b/README.md
@@ -1,3 +1,4 @@
+
 # StakeWise V3 Operator
 
 1. [What is V3 Operator?](#what-is-v3-operator)
@@ -42,12 +43,13 @@ The validator registration process consists of the following steps:
 1. Check whether Vault has accumulated enough assets to register a validator (e.g., 32 ETH for Ethereum)
 2. Get the next free validator public key from the deposit data file attached to the operator. The validators are
    registered in the same order as specified in the deposit data file.
-3. Share the exit signature of the validator with StakeWise Oracles:
-    1. Using [Shamir's secret sharing](https://en.wikipedia.org/wiki/Shamir%27s_secret_sharing), generate shares for the
-       validator's BLS private key. The number of shares is equal to the number of oracles.
-    2. Sign the exit message with every private key share and encrypt exit signatures with oracles' public keys.
+3. Obtain BLS signature for exit message using local keystores or remote signer.
+4. Share the exit signature of the validator with StakeWise Oracles:
+    1. Using [Shamir's secret sharing](https://en.wikipedia.org/wiki/Shamir%27s_secret_sharing), split
+       validator's BLS signature. The number of shares is equal to the number of oracles.
+    2. Encrypt exit signatures with oracles' public keys.
     3. Send encrypted exit signatures to all the oracles and receive registration signatures from them.
-4. Send transaction to Vault contract to register the validator.
+5. Send transaction to Vault contract to register the validator.
 
 ### Exit signatures rotation
 
@@ -147,22 +149,22 @@ Head to [Usage](#usage) to launch your operator service.
 Pull the latest docker operator docker image:
 
 ```bash
-docker pull europe-west4-docker.pkg.dev/stakewiselabs/public/v3-operator:v1.0.8
+docker pull europe-west4-docker.pkg.dev/stakewiselabs/public/v3-operator:v1.1.0
 ```
 
 You can also build the docker image from source by cloning this repo and executing the following command from within
 the `v3-operator` folder:
 
 ```bash
-docker build --pull -t europe-west4-docker.pkg.dev/stakewiselabs/public/v3-operator:v1.0.8 .
+docker build --pull -t europe-west4-docker.pkg.dev/stakewiselabs/public/v3-operator:v1.1.0 .
 ```
 
 You will execute Operator Service commands using the format below (note the use of flags are optional):
 
 ```bash
 docker run --rm -ti \
 -v ~/.stakewise/:/data \
-europe-west4-docker.pkg.dev/stakewiselabs/public/v3-operator:v1.0.8 \
+europe-west4-docker.pkg.dev/stakewiselabs/public/v3-operator:v1.1.0 \
 src/main.py COMMAND \
 --flagA=123 \
 --flagB=xyz
@@ -384,7 +386,7 @@ below:
 ```bash
 docker run --restart on-failure:10 \
 -v ~/.stakewise/:/data \
-europe-west4-docker.pkg.dev/stakewiselabs/public/v3-operator:v1.0.8 \
+europe-west4-docker.pkg.dev/stakewiselabs/public/v3-operator:v1.1.0 \
 src/main.py start \
 --vault=0x3320ad928c20187602a2b2c04eeaa813fa899468 \
 --data-dir=/data \

diff --git a/docker-compose.yml b/docker-compose.yml
@@ -4,7 +4,7 @@ services:
   operator:
     build:
       context: .
-      dockerfile: europe-west4-docker.pkg.dev/stakewiselabs/public/v3-operator:v1.0.8
+      dockerfile: europe-west4-docker.pkg.dev/stakewiselabs/public/v3-operator:v1.1.0
     image: v3-operator
     container_name: v3-operator
     command: ["src/main.py", "start"]

diff --git a/poetry.lock b/poetry.lock
diff --git a/pyproject.toml b/pyproject.toml
@@ -1,6 +1,6 @@
 [tool.poetry]
 name = "v3-operator"
-version = "v1.0.8"
+version = "v1.1.0"
 description = "StakeWise operator service for registering vault validators"
 authors = ["StakeWise Labs <info@stakewise.io>"]
 
@@ -9,8 +9,8 @@ python = ">=3.10,<3.11"
 python-decouple = "==3.8"
 sentry-sdk = "==1.32.0"
 py-ecc = "==6.0.0"
-multiproof = { git = "https://github.com/stakewise/multiproof.git", rev = "v0.1.6" }
-sw-utils = { git = "https://github.com/stakewise/sw-utils.git", rev = "v0.3.31" }
+multiproof = { git = "https://github.com/stakewise/multiproof.git", rev = "v0.1.7" }
+sw-utils = { git = "https://github.com/stakewise/sw-utils.git", rev = "v0.6.1" }
 staking-deposit = { git = "https://github.com/ethereum/staking-deposit-cli.git", rev = "v2.4.0" }
 pycryptodomex = "3.19.1"
 milagro-bls-binding = "==1.9.0"
@@ -21,6 +21,10 @@ prometheus-client = "==0.17.1"
 psycopg2 = "==2.9.9"
 pyyaml = "==6.0.1"
 aiohttp = "==3.9.3"
+python-json-logger = "==2.0.7"
+starlette = "==0.36.2"
+uvicorn = "==0.27.0"
+pydantic = "==2.5.3"
 
 [tool.poetry.group.dev.dependencies]
 pylint = "==3.0.1"
@@ -34,8 +38,8 @@ bandit = { version = "==1.7.5", extras = ["toml"] }
 black = { version = "==23.10.0", extras = ["d"] }
 pyinstaller = "==5.13.2"
 faker = "==19.11.0"
-flake8-datetime-utcnow-plugin = "==0.1.2"
 flake8-print = "==5.0.0"
+flake8-datetimez = "==20.10.0"
 types-pyyaml = "==6.0.12.12"
 coverage = "==7.3.2"
 aioresponses = "^0.7.4"

diff --git a/scripts/install.sh b/scripts/install.sh
@@ -281,7 +281,7 @@ http_copy() {
 github_release() {
   owner_repo=$1
   version=$2
-  test -z "$version" && version="v1.0.8"
+  test -z "$version" && version="v1.1.0"
   giturl="https://github.com/${owner_repo}/releases/${version}"
   json=$(http_copy "$giturl" "Accept:application/json")
   test -z "$json" && return 1

diff --git a/src/api.py b/src/api.py
@@ -0,0 +1,11 @@
+from starlette.applications import Starlette
+from starlette.routing import Route
+
+from src.validators.api.endpoints import get_validators, submit_validators
+
+app = Starlette(
+    routes=[
+        Route('/validators', get_validators, methods=['GET']),
+        Route('/validators', submit_validators, methods=['POST']),
+    ]
+)
diff --git a/src/commands/create_keys.py b/src/commands/create_keys.py
@@ -49,7 +49,7 @@
 @click.option(
     '--pool-size',
     help='Number of processes in a pool.',
-    # do not prompt
+    envvar='POOL_SIZE',
     type=int,
 )
 @click.command(help='Creates the validator keys from the mnemonic.')

diff --git a/src/commands/recover.py b/src/commands/recover.py
@@ -10,7 +10,7 @@
 from src.common.contracts import v2_pool_contract, vault_contract
 from src.common.credentials import CredentialManager
 from src.common.execution import SECONDS_PER_MONTH
-from src.common.logging import setup_logging
+from src.common.logging import LOG_LEVELS, setup_logging
 from src.common.password import generate_password, get_or_create_password_file
 from src.common.utils import greenify, log_verbose
 from src.common.validators import validate_eth_address, validate_mnemonic
@@ -76,6 +76,16 @@
         case_sensitive=False,
     ),
 )
+@click.option(
+    '--log-level',
+    type=click.Choice(
+        LOG_LEVELS,
+        case_sensitive=False,
+    ),
+    default='INFO',
+    envvar='LOG_LEVEL',
+    help='The log level.',
+)
 # pylint: disable-next=too-many-arguments
 def recover(
     data_dir: str,
@@ -86,6 +96,7 @@ def recover(
     execution_endpoints: str,
     per_keystore_password: bool,
     no_confirm: bool,
+    log_level: str,
 ) -> None:
     # pylint: disable=duplicate-code
     config = VaultConfig(
@@ -106,6 +117,7 @@ def recover(
         vault=vault,
         network=network,
         vault_dir=config.vault_dir,
+        log_level=log_level,
     )
 
     try: