failed while querying src chain for latest height

[rootulp]

Context

I'm trying to set up IBC clients between two tesnets:

1. https://github.com/cosmos/chain-registry/blob/master/testnets/cosmoshubtestnet/chain.json
2. https://github.com/cosmos/chain-registry/blob/master/testnets/celestiatestnet3/chain.json

Problem

I was able to create a client on mocha-4

$ hermes create client --host-chain mocha-4 --reference-chain theta-testnet-001
2023-09-15T15:15:54.602346Z  INFO ThreadId(01) using default configuration from '/Users/rootulp/.hermes/config.toml'
2023-09-15T15:15:54.602777Z  INFO ThreadId(01) running Hermes v1.6.0+1c1cf02
2023-09-15T15:15:55.655123Z  WARN ThreadId(01) Unsupported tendermint version, will use v0.37 compatibility mode but relaying might not work as desired: unsupported Tendermint version reported by the node: 1.0.0-rc14
SUCCESS CreateClient(
    CreateClient(
        Attributes {
            client_id: ClientId(
                "07-tendermint-0",
            ),
            client_type: Tendermint,
            consensus_height: Height {
                revision: 0,
                height: 1,
            },
        },
    ),

However, I can't create the other client

$ hermes create client --host-chain theta-testnet-001 --reference-chain mocha-4
2023-09-15T15:14:01.495020Z  INFO ThreadId(01) using default configuration from '/Users/rootulp/.hermes/config.toml'
2023-09-15T15:14:01.495421Z  INFO ThreadId(01) running Hermes v1.6.0+1c1cf02
2023-09-15T15:14:02.273418Z  WARN ThreadId(01) Unsupported tendermint version, will use v0.37 compatibility mode but relaying might not work as desired: unsupported Tendermint version reported by the node: 1.0.0-rc14
ERROR foreign client error: error raised while creating client for chain mocha-4: failed while querying src chain for latest height: RPC error to endpoint https://rpc-celestia-mocha.architectnodes.com/: response error: Method not found (code: -32601)

However, as far as I can tell, the RPC URL of mocha-4 is a superset of the methods available on Cosmos hub testnet.

1. https://rpc.sentry-01.theta-testnet.polypore.xyz/
2. https://rpc-celestia-mocha.architectnodes.com/

Note: I have tried a few other RPC URLs for mocha-4 from RPC endpoints: https://docs.celestia.org/nodes/mocha-testnet/#rpc-endpoints

Proposal

Include more information in the error. For example which method was not found?

[rootulp]

ERROR foreign client error: error raised while creating client for chain mocha-4: failed while querying src chain for latest height: RPC error to endpoint https://rpc-celestia-mocha.architectnodes.com/: response error: Method not found (code: -32601)

This error message is confusing b/c it claims that the source chain is mocha-4 but mocha-4 is the reference chain in the command:

$ hermes create client --host-chain theta-testnet-001 --reference-chain mocha-4

Update: Disregard. In the code here: src chain == reference chain and dst chain == host chain.

[rootulp]

It looks like it is calling

hermes/crates/relayer/src/chain/handle.rs

Line 434 in 070f60d

Ok(self.query_application_status()?.height)

so I would expect https://rpc-celestia-mocha.architectnodes.com/status? to suffice

[rootulp]

My config.toml: https://gist.github.com/rootulp/fb3573c8fb64378c53014cc0a8807cff

[rootulp]

There were a lot of layers of indirection but I found

hermes/crates/relayer/src/chain/cosmos.rs

Lines 1115 to 1145 in 1c1cf02

	/// Query the application status
	fn query_application_status(&self) -> Result<ChainStatus, Error> {
	crate::time!(
	"query_application_status",
	{
	"src_chain": self.config().id.to_string(),
	}
	);
	crate::telemetry!(query, self.id(), "query_application_status");
	// We cannot rely on `/status` endpoint to provide details about the latest block.
	// Instead, we need to pull block height via `/abci_info` and then fetch block
	// metadata at the given height via `/blockchain` endpoint.
	let abci_info = self
	.block_on(self.rpc_client.abci_info())
	.map_err(|e| Error::rpc(self.config.rpc_addr.clone(), e))?;
	// Query `/header` endpoint to pull the latest block that the application committed.
	let response = self
	.block_on(self.rpc_client.header(abci_info.last_block_height))
	.map_err(|e| Error::rpc(self.config.rpc_addr.clone(), e))?;
	let height = ICSHeight::new(
	ChainId::chain_version(response.header.chain_id.as_str()),
	u64::from(abci_info.last_block_height),
	)
	.map_err(|_| Error::invalid_height_no_source())?;
	let timestamp = response.header.time.into();
	Ok(ChainStatus { height, timestamp })
	}

https://rpc-celestia-mocha.architectnodes.com/abci_info? has

{
  "jsonrpc": "2.0",
  "id": -1,
  "result": {
    "response": {
      "data": "celestia-app",
      "version": "1.0.0-rc14",
      "app_version": "1",
      "last_block_height": "77966",
      "last_block_app_hash": "fers/+RRSRIgPx9DqxZaONvg4ARIxkEdsaT+Jyx3Oq0="
    }
  }
}

so last_block_height = 77966 but I don't see a /header endpoint on

• https://rpc-celestia-mocha.architectnodes.com/
• https://rpc.sentry-01.theta-testnet.polypore.xyz/

both RPC providers contain a /block endpoint but the Celestia codepath doesn't use that endpoint because the warning before:

2023-09-15T15:15:55.655123Z  WARN ThreadId(01) Unsupported tendermint version, will use v0.37 compatibility mode but relaying might not work as desired: unsupported Tendermint version reported by the node: 1.0.0-rc14

Celestia hits this warning because celestiaorg/celestia-app#2516

[rootulp]

I was able to resolve by falling back to Tendermit v0.34 instead of v0.37. In other words:

$ git diff
diff --git a/crates/relayer-cli/src/commands/listen.rs b/crates/relayer-cli/src/commands/listen.rs
index c7b779529..34b3dd2fe 100644
--- a/crates/relayer-cli/src/commands/listen.rs
+++ b/crates/relayer-cli/src/commands/listen.rs
@@ -177,8 +177,8 @@ fn detect_compatibility_mode(
     let client = HttpClient::new(config.rpc_addr.clone())?;
     let status = rt.block_on(client.status())?;
     let compat_mode = CompatMode::from_version(status.node_info.version).unwrap_or_else(|e| {
-        warn!("Unsupported tendermint version, will use v0.37 compatibility mode but relaying might not work as desired: {e}");
-        CompatMode::V0_37
+        warn!("Unsupported tendermint version, will use v0.34 compatibility mode but relaying might not work as desired: {e}");
+        CompatMode::V0_34
     });
     Ok(compat_mode)
 }
diff --git a/crates/relayer/src/chain/cosmos.rs b/crates/relayer/src/chain/cosmos.rs
index 29c89cc7b..2f73ea848 100644
--- a/crates/relayer/src/chain/cosmos.rs
+++ b/crates/relayer/src/chain/cosmos.rs
@@ -875,8 +875,8 @@ impl ChainEndpoint for CosmosSdkChain {
         let node_info = rt.block_on(fetch_node_info(&rpc_client, &config))?;

         let compat_mode = CompatMode::from_version(node_info.version).unwrap_or_else(|e| {
-            warn!("Unsupported tendermint version, will use v0.37 compatibility mode but relaying might not work as desired: {e}");
-            CompatMode::V0_37
+            warn!("Unsupported tendermint version, will use v0.34 compatibility mode but relaying might not work as desired: {e}");
+            CompatMode::V0_34
         });
         rpc_client.set_compat_mode(compat_mode);

Can close this issue for now b/c a temporary fix has been identified. Will open a new issue for a long term fix (i.e. ability to specify in hermes Config file for a chain which Tendermint version to use if abci_info doesn't return expected response).

[seanchen1991]

Hi @rootulp, just wanted to say thanks for circling back and updating this issue after you found a working temporary fix, as well as for opening the follow up issue 🙂