Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Offline RPC #3866

Merged
merged 40 commits into from
Jan 30, 2024
Merged
Show file tree
Hide file tree
Changes from 16 commits
Commits
Show all changes
40 commits
Select commit Hold shift + click to select a range
892e9a5
Impl node less rpc
sudo-shashank Jan 11, 2024
a257bef
fix
sudo-shashank Jan 16, 2024
8b745c9
Set SyncStage to idle and Cleanup
sudo-shashank Jan 16, 2024
8e1beaa
fmt
sudo-shashank Jan 16, 2024
447303d
Merge branch 'main' into shashank/node-less-rpc
sudo-shashank Jan 16, 2024
ed75192
fix and add new methods
sudo-shashank Jan 16, 2024
731e154
add new rpc
sudo-shashank Jan 17, 2024
03de8c8
lint-fix
sudo-shashank Jan 17, 2024
200c84d
Merge branch 'main' into shashank/node-less-rpc
sudo-shashank Jan 17, 2024
60256eb
add ci test
sudo-shashank Jan 18, 2024
f49c717
fix check rpc script
sudo-shashank Jan 18, 2024
8764c90
updated changelog
sudo-shashank Jan 18, 2024
a47ec6f
lint fix
sudo-shashank Jan 18, 2024
fd47b3b
fix
sudo-shashank Jan 18, 2024
d0c68fa
Merge branch 'main' into shashank/node-less-rpc
sudo-shashank Jan 18, 2024
fcb1ad3
Merge branch 'main' into shashank/node-less-rpc
sudo-shashank Jan 18, 2024
8de82d7
refactored api serve subcommand
sudo-shashank Jan 23, 2024
4d8b7f4
use logger
sudo-shashank Jan 23, 2024
2a6adb0
fix logger
sudo-shashank Jan 23, 2024
6e0e8a0
Merge branch 'main' into shashank/node-less-rpc
sudo-shashank Jan 23, 2024
dda6e33
remove use of tmp dir
sudo-shashank Jan 23, 2024
9d17dfd
Updated offline rpc test script
sudo-shashank Jan 23, 2024
a88cb56
lint fix
sudo-shashank Jan 23, 2024
0540c17
fix
sudo-shashank Jan 24, 2024
854325e
fix typo
sudo-shashank Jan 24, 2024
2cad4ba
limit tipset in compare
sudo-shashank Jan 24, 2024
2e4c090
cleanup
sudo-shashank Jan 24, 2024
99627e0
Merge branch 'main' into shashank/node-less-rpc
sudo-shashank Jan 24, 2024
366eff8
Merge branch 'main' into shashank/node-less-rpc
sudo-shashank Jan 29, 2024
0b05045
Merge branch 'main' into shashank/node-less-rpc
sudo-shashank Jan 29, 2024
0f82ed6
Merge branch 'main' into shashank/node-less-rpc
sudo-shashank Jan 29, 2024
0ea032e
treat timeout as success
sudo-shashank Jan 29, 2024
4595a4e
fix changelog
sudo-shashank Jan 29, 2024
59a55d2
fmt
sudo-shashank Jan 29, 2024
7f21806
fmt
sudo-shashank Jan 29, 2024
0bf99d0
add auto snapshot download option
sudo-shashank Jan 30, 2024
1b4e0c8
Merge branch 'main' into shashank/node-less-rpc
sudo-shashank Jan 30, 2024
79002c4
fix peek method call
sudo-shashank Jan 30, 2024
2e03771
Merge branch 'main' into shashank/node-less-rpc
sudo-shashank Jan 30, 2024
e16498e
Merge branch 'main' into shashank/node-less-rpc
sudo-shashank Jan 30, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 27 additions & 0 deletions .github/workflows/forest.yml
Original file line number Diff line number Diff line change
Expand Up @@ -381,3 +381,30 @@ jobs:
timeout-minutes: 15
- name: Dump docker logs
uses: jwalton/gh-docker-logs@v2

# node-less rpc check on calibnet
node-less-rpc-check:
needs:
- build-ubuntu
name: Node-Less RPC check
runs-on: ubuntu-latest
steps:
# To help investigate transient test failures
- run: lscpu
- name: Checkout Sources
uses: actions/checkout@v4
- uses: actions/download-artifact@v4
with:
name: forest-${{ runner.os }}
path: ~/.cargo/bin
- uses: actions/download-artifact@v4
with:
name: forest-${{ runner.os }}
path: ~/.cargo/bin
# Permissions are lost during artifact-upload
# https://github.com/actions/upload-artifact#permission-loss
- name: Set permissions
run: |
chmod +x ~/.cargo/bin/forest*
- name: RPC check
run: ./scripts/tests/calibnet_rpc_check.sh
hanabi1224 marked this conversation as resolved.
Show resolved Hide resolved
2 changes: 2 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,8 @@
flag to `forest-tool api compare`.
- [#3806](https://github.com/ChainSafe/forest/pull/3806) Implement the
`Filecoin.EthGasPrice` lotus-compatible RPC API.
- [#3866](https://github.com/ChainSafe/forest/pull/3866) Implement Node-Less RPC
API.

### Changed

Expand Down
53 changes: 53 additions & 0 deletions scripts/tests/calibnet_rpc_check.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
#!/usr/bin/env bash
# This script is used to test the Node-Less API server against itself.
# It's run in CI, and uses forest-tool api compare subcommand to test RPC endpoints.

set -euxo pipefail

FOREST_TOOL_PATH="forest-tool"
PORTS=(8080 8081)

# Function to get the number of files in the present working directory
num_files_here() {
find . -type f | wc --lines
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: this overcounts if I make a file with a newline in the name

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed, now directly using the path returned from snapshot fetch

}

# Function to stop services on specified ports
stop_services() {
for port in "${PORTS[@]}"; do
fuser -k "$port/tcp" || true
done
}

TEMP_DIR=$(mktemp --directory)
pushd "$TEMP_DIR"
# Fetch latest calibnet snapshot
"$FOREST_TOOL_PATH" snapshot fetch --chain calibnet
test "$(num_files_here)" -eq 1
snapshot=$(find . -type f | head -1)

# Start Node-Less RPC servers on ports 8080 and 8081
for port in "${PORTS[@]}"; do
"$FOREST_TOOL_PATH" api serve "$snapshot" --chain calibnet --port "$port" &
done

# Check if services on ports 8080 and 8081 have started
while ! (nc -z localhost 8080 && nc -z localhost 8081); do
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: this could be an until loop
also hardcoding the ports here defeats the purpose of using the PORTS array

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

right, fixed

sleep 30
done

# Compare
result="$($FOREST_TOOL_PATH api compare "$snapshot" --forest /ip4/127.0.0.1/tcp/8080/http --lotus /ip4/127.0.0.1/tcp/8081/http)"

# Check the result
if echo "$result" | tail -n +3 | grep -E -v "\| *(Valid|Timeout) *\| *(Valid|Timeout) *\|"; then
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Parsing the output is an odd choice - can we not modify forest-tool api compare to give us useful exit codes?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed

stop_services
exit 1
fi
popd

# Stop services on ports 8080 and 8081
stop_services

# Cleanup temporary directory
rm -rf "$TEMP_DIR"
2 changes: 1 addition & 1 deletion src/daemon/db_util.rs
Original file line number Diff line number Diff line change
Expand Up @@ -94,7 +94,7 @@ pub async fn import_chain_as_forest_car(
Ok((forest_car_db_path, ts))
}

async fn download_to(url: &Url, destination: &Path) -> anyhow::Result<()> {
pub async fn download_to(url: &Url, destination: &Path) -> anyhow::Result<()> {
snapshot::download_file_with_retry(
url,
destination.parent().with_context(|| {
Expand Down
2 changes: 1 addition & 1 deletion src/daemon/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
// SPDX-License-Identifier: Apache-2.0, MIT

pub mod bundle;
mod db_util;
pub mod db_util;
pub mod main;

use crate::auth::{create_token, generate_priv_key, ADMIN, JWT_IDENTIFIER};
Expand Down
4 changes: 2 additions & 2 deletions src/rpc/auth_api.rs
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ use fvm_ipld_blockstore::Blockstore;
use jsonrpc_v2::{Data, Error as JsonRpcError, Params};

/// RPC call to create a new JWT Token
pub(in crate::rpc) async fn auth_new<DB: Blockstore>(
pub async fn auth_new<DB: Blockstore>(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like it

data: Data<RPCState<DB>>,
Params(params): Params<AuthNewParams>,
) -> Result<LotusJson<Vec<u8>>, JsonRpcError> {
Expand All @@ -20,7 +20,7 @@ pub(in crate::rpc) async fn auth_new<DB: Blockstore>(
}

/// RPC call to verify JWT Token and return the token's permissions
pub(in crate::rpc) async fn auth_verify<DB>(
pub async fn auth_verify<DB>(
data: Data<RPCState<DB>>,
Params((header_raw,)): Params<(String,)>,
) -> Result<Vec<String>, JsonRpcError>
Expand Down
2 changes: 1 addition & 1 deletion src/rpc/beacon_api.rs
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ use jsonrpc_v2::{Data, Error as JsonRpcError, Params};
/// `BeaconGetEntry` returns the beacon entry for the given Filecoin epoch. If
/// the entry has not yet been produced, the call will block until the entry
/// becomes available
pub(in crate::rpc) async fn beacon_get_entry<DB: Blockstore>(
pub async fn beacon_get_entry<DB: Blockstore>(
data: Data<RPCState<DB>>,
Params((first,)): Params<(ChainEpoch,)>,
) -> Result<LotusJson<BeaconEntry>, JsonRpcError> {
Expand Down
26 changes: 13 additions & 13 deletions src/rpc/chain_api.rs
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ use sha2::Sha256;
use std::sync::Arc;
use tokio::sync::Mutex;

pub(in crate::rpc) async fn chain_get_message<DB: Blockstore>(
pub async fn chain_get_message<DB: Blockstore>(
data: Data<RPCState<DB>>,
Params(LotusJson((msg_cid,))): Params<LotusJson<(Cid,)>>,
) -> Result<LotusJson<Message>, JsonRpcError> {
Expand All @@ -42,7 +42,7 @@ pub(in crate::rpc) async fn chain_get_message<DB: Blockstore>(
}))
}

pub(in crate::rpc) async fn chain_get_parent_message<DB: Blockstore>(
pub async fn chain_get_parent_message<DB: Blockstore>(
data: Data<RPCState<DB>>,
Params(LotusJson((block_cid,))): Params<LotusJson<(Cid,)>>,
) -> Result<LotusJson<Vec<ApiMessage>>, JsonRpcError> {
Expand All @@ -59,7 +59,7 @@ pub(in crate::rpc) async fn chain_get_parent_message<DB: Blockstore>(
}
}

pub(in crate::rpc) async fn chain_get_parent_receipts<DB: Blockstore + Send + Sync + 'static>(
pub async fn chain_get_parent_receipts<DB: Blockstore + Send + Sync + 'static>(
data: Data<RPCState<DB>>,
Params(LotusJson((block_cid,))): Params<LotusJson<(Cid,)>>,
) -> Result<LotusJson<Vec<ApiReceipt>>, JsonRpcError> {
Expand Down Expand Up @@ -105,7 +105,7 @@ pub(crate) async fn chain_get_messages_in_tipset<DB: Blockstore>(
Ok(LotusJson(messages))
}

pub(in crate::rpc) async fn chain_export<DB>(
pub async fn chain_export<DB>(
data: Data<RPCState<DB>>,
Params(ChainExportParams {
epoch,
Expand Down Expand Up @@ -169,7 +169,7 @@ where
}
}

pub(in crate::rpc) async fn chain_read_obj<DB: Blockstore>(
pub async fn chain_read_obj<DB: Blockstore>(
data: Data<RPCState<DB>>,
Params(LotusJson((obj_cid,))): Params<LotusJson<(Cid,)>>,
) -> Result<LotusJson<Vec<u8>>, JsonRpcError> {
Expand All @@ -181,14 +181,14 @@ pub(in crate::rpc) async fn chain_read_obj<DB: Blockstore>(
Ok(LotusJson(bytes))
}

pub(in crate::rpc) async fn chain_has_obj<DB: Blockstore>(
pub async fn chain_has_obj<DB: Blockstore>(
data: Data<RPCState<DB>>,
Params(LotusJson((obj_cid,))): Params<LotusJson<(Cid,)>>,
) -> Result<bool, JsonRpcError> {
Ok(data.state_manager.blockstore().get(&obj_cid)?.is_some())
}

pub(in crate::rpc) async fn chain_get_block_messages<DB: Blockstore>(
pub async fn chain_get_block_messages<DB: Blockstore>(
data: Data<RPCState<DB>>,
Params(LotusJson((blk_cid,))): Params<LotusJson<(Cid,)>>,
) -> Result<BlockMessages, JsonRpcError> {
Expand Down Expand Up @@ -218,7 +218,7 @@ pub(in crate::rpc) async fn chain_get_block_messages<DB: Blockstore>(
Ok(ret)
}

pub(in crate::rpc) async fn chain_get_tipset_by_height<DB: Blockstore>(
pub async fn chain_get_tipset_by_height<DB: Blockstore>(
data: Data<RPCState<DB>>,
Params(LotusJson((height, tsk))): Params<LotusJson<(ChainEpoch, TipsetKey)>>,
) -> Result<LotusJson<Tipset>, JsonRpcError> {
Expand All @@ -234,21 +234,21 @@ pub(in crate::rpc) async fn chain_get_tipset_by_height<DB: Blockstore>(
Ok((*tss).clone().into())
}

pub(in crate::rpc) async fn chain_get_genesis<DB: Blockstore>(
pub async fn chain_get_genesis<DB: Blockstore>(
data: Data<RPCState<DB>>,
) -> Result<Option<LotusJson<Tipset>>, JsonRpcError> {
let genesis = data.state_manager.chain_store().genesis_block_header();
Ok(Some(Tipset::from(genesis).into()))
}

pub(in crate::rpc) async fn chain_head<DB: Blockstore>(
pub async fn chain_head<DB: Blockstore>(
data: Data<RPCState<DB>>,
) -> Result<LotusJson<Tipset>, JsonRpcError> {
let heaviest = data.state_manager.chain_store().heaviest_tipset();
Ok((*heaviest).clone().into())
}

pub(in crate::rpc) async fn chain_get_block<DB: Blockstore>(
pub async fn chain_get_block<DB: Blockstore>(
data: Data<RPCState<DB>>,
Params(LotusJson((blk_cid,))): Params<LotusJson<(Cid,)>>,
) -> Result<LotusJson<CachingBlockHeader>, JsonRpcError> {
Expand All @@ -260,7 +260,7 @@ pub(in crate::rpc) async fn chain_get_block<DB: Blockstore>(
Ok(blk.into())
}

pub(in crate::rpc) async fn chain_get_tipset<DB: Blockstore>(
pub async fn chain_get_tipset<DB: Blockstore>(
data: Data<RPCState<DB>>,
Params(LotusJson((tsk,))): Params<LotusJson<(TipsetKey,)>>,
) -> Result<LotusJson<Tipset>, JsonRpcError> {
Expand All @@ -273,7 +273,7 @@ pub(in crate::rpc) async fn chain_get_tipset<DB: Blockstore>(

// This is basically a port of the reference implementation at
// https://github.com/filecoin-project/lotus/blob/v1.23.0/node/impl/full/chain.go#L321
pub(in crate::rpc) async fn chain_set_head<DB: Blockstore>(
pub async fn chain_set_head<DB: Blockstore>(
data: Data<RPCState<DB>>,
Params(LotusJson((tsk,))): Params<LotusJson<(TipsetKey,)>>,
) -> Result<(), JsonRpcError> {
Expand Down
8 changes: 4 additions & 4 deletions src/rpc/common_api.rs
Original file line number Diff line number Diff line change
Expand Up @@ -14,11 +14,11 @@ use uuid::Uuid;
static SESSION_UUID: Lazy<Uuid> = Lazy::new(Uuid::new_v4);

/// The session UUID uniquely identifies the API node.
pub(in crate::rpc) async fn session() -> Result<String, JsonRpcError> {
pub async fn session() -> Result<String, JsonRpcError> {
Ok(SESSION_UUID.to_string())
}

pub(in crate::rpc) async fn version(
pub async fn version(
block_delay: u64,
forest_version: &'static str,
) -> Result<APIVersion, JsonRpcError> {
Expand All @@ -30,7 +30,7 @@ pub(in crate::rpc) async fn version(
})
}

pub(in crate::rpc) async fn shutdown(shutdown_send: Sender<()>) -> Result<(), JsonRpcError> {
pub async fn shutdown(shutdown_send: Sender<()>) -> Result<(), JsonRpcError> {
// Trigger graceful shutdown
if let Err(err) = shutdown_send.send(()).await {
return Err(JsonRpcError::from(err));
Expand All @@ -39,7 +39,7 @@ pub(in crate::rpc) async fn shutdown(shutdown_send: Sender<()>) -> Result<(), Js
}

/// gets start time from network
pub(in crate::rpc) async fn start_time<DB: Blockstore>(
pub async fn start_time<DB: Blockstore>(
data: Data<RPCState<DB>>,
) -> Result<chrono::DateTime<chrono::Utc>, JsonRpcError> {
Ok(data.start_time)
Expand Down
10 changes: 5 additions & 5 deletions src/rpc/eth_api.rs
Original file line number Diff line number Diff line change
Expand Up @@ -18,12 +18,12 @@ use jsonrpc_v2::{Data, Error as JsonRpcError, Params};
use num_bigint::BigInt;
use num_traits::Zero as _;

pub(in crate::rpc) async fn eth_accounts() -> Result<Vec<String>, JsonRpcError> {
pub async fn eth_accounts() -> Result<Vec<String>, JsonRpcError> {
// EthAccounts will always return [] since we don't expect Forest to manage private keys
Ok(vec![])
}

pub(in crate::rpc) async fn eth_block_number<DB: Blockstore>(
pub async fn eth_block_number<DB: Blockstore>(
data: Data<RPCState<DB>>,
) -> Result<String, JsonRpcError> {
// `eth_block_number` needs to return the height of the latest committed tipset.
Expand Down Expand Up @@ -51,7 +51,7 @@ pub(in crate::rpc) async fn eth_block_number<DB: Blockstore>(
}
}

pub(in crate::rpc) async fn eth_chain_id<DB: Blockstore>(
pub async fn eth_chain_id<DB: Blockstore>(
data: Data<RPCState<DB>>,
) -> Result<String, JsonRpcError> {
Ok(format!(
Expand All @@ -60,7 +60,7 @@ pub(in crate::rpc) async fn eth_chain_id<DB: Blockstore>(
))
}

pub(in crate::rpc) async fn eth_gas_price<DB: Blockstore>(
pub async fn eth_gas_price<DB: Blockstore>(
data: Data<RPCState<DB>>,
) -> Result<GasPriceResult, JsonRpcError> {
let ts = data.state_manager.chain_store().heaviest_tipset();
Expand All @@ -74,7 +74,7 @@ pub(in crate::rpc) async fn eth_gas_price<DB: Blockstore>(
}
}

pub(in crate::rpc) async fn eth_get_balance<DB: Blockstore>(
pub async fn eth_get_balance<DB: Blockstore>(
data: Data<RPCState<DB>>,
Params(LotusJson((address, block_param))): Params<LotusJson<(Address, BlockNumberOrHash)>>,
) -> Result<EthBigInt, JsonRpcError> {
Expand Down
10 changes: 5 additions & 5 deletions src/rpc/gas_api.rs
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ use rand_distr::{Distribution, Normal};
const MIN_GAS_PREMIUM: f64 = 100000.0;

/// Estimate the fee cap
pub(in crate::rpc) async fn gas_estimate_fee_cap<DB: Blockstore>(
pub async fn gas_estimate_fee_cap<DB: Blockstore>(
data: Data<RPCState<DB>>,
Params(params): Params<LotusJson<(Message, i64, TipsetKey)>>,
) -> Result<String, JsonRpcError> {
Expand Down Expand Up @@ -49,7 +49,7 @@ fn estimate_fee_cap<DB: Blockstore>(
}

/// Estimate the fee cap
pub(in crate::rpc) async fn gas_estimate_gas_premium<DB: Blockstore>(
pub async fn gas_estimate_gas_premium<DB: Blockstore>(
data: Data<RPCState<DB>>,
Params(params): Params<LotusJson<(u64, Address, i64, TipsetKey)>>,
) -> Result<String, JsonRpcError> {
Expand Down Expand Up @@ -141,7 +141,7 @@ pub async fn estimate_gas_premium<DB: Blockstore>(
}

/// Estimate the gas limit
pub(in crate::rpc) async fn gas_estimate_gas_limit<DB>(
pub async fn gas_estimate_gas_limit<DB>(
data: Data<RPCState<DB>>,
Params(LotusJson((msg, tsk))): Params<LotusJson<(Message, TipsetKey)>>,
) -> Result<i64, JsonRpcError>
Expand Down Expand Up @@ -195,7 +195,7 @@ where
}

/// Estimates the gas parameters for a given message
pub(in crate::rpc) async fn gas_estimate_message_gas<DB>(
pub async fn gas_estimate_message_gas<DB>(
data: Data<RPCState<DB>>,
Params(LotusJson((msg, spec, tsk))): Params<
LotusJson<(Message, Option<MessageSendSpec>, TipsetKey)>,
Expand All @@ -209,7 +209,7 @@ where
.map(Into::into)
}

pub(in crate::rpc) async fn estimate_message_gas<DB>(
pub async fn estimate_message_gas<DB>(
data: &Data<RPCState<DB>>,
msg: Message,
_spec: Option<MessageSendSpec>,
Expand Down
Loading
Loading