Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Epic: Add scraper support for Cosmos #3329

Closed
6 tasks done
tkporter opened this issue Feb 29, 2024 · 1 comment
Closed
6 tasks done

Epic: Add scraper support for Cosmos #3329

tkporter opened this issue Feb 29, 2024 · 1 comment

Comments

@tkporter
Copy link
Collaborator

tkporter commented Feb 29, 2024

The scraper makes a few EVM-centric assumptions, but nothing excessive. You can think of the work as roughly: "change the scraper to not be EVM-centric, implement some Cosmos-specific code, make sure we can test this, and deploy"

In chronological order & each containing day estimates:

Note: sequencing-wise, it may make sense to coordinate with #4271 for many things including shipping this

Cosmos scraping

Preview Give feedback
  1. agent alt-VM
    ameten
  2. agent cosmos scraper
  3. agent scraper
    ameten
  4. ameten
  5. CI cosmos scraper testing
    ameten
  6. cosmos scraper
    ameten

based on research completed in https://github.com/hyperlane-xyz/issues/issues/903

Related: hyperlane-xyz/hyperlane-explorer#165

@tkporter
Copy link
Collaborator Author

tkporter commented Feb 29, 2024

A "scratch pad" for my thoughts as a dove in:

  • Why does it need to use the forward message sync cursor instead of the forward backward? Feels like forward backward is better
  • Why do we have so many rows in cursor?
    • seems like we insert a new one each time we update the watermark, why? why not just update?
select count(*) from cursor;
  count
---------
 9688274
(1 row)

Changes to schema required:

  1. Our address type isn't 32 bytes. Should it be?
/// Addresses are to be stored as binary.
#[allow(non_upper_case_globals)]
pub const Address: ColumnType = ColumnType::Binary(BlobSize::Tiny);

Maybe this isn't so bad? But this feels awkward - would probably be easier to keep everything as 32 bytes. Worth considering the explorer situation.

We use address_to_bytes in various places to convert H256s like senders / recipients / mailbox addresses to a 20 bye representation, which is very EVM specific.

This is used:
a. message table: sender, recipient, and origin mailbox
b. delivered_message table: destination_mailbox

  1. unsigned doesn't do what we want - we need a larger data type Postgres doesn't support unsigned integers, but scraper DB schema assumes it does #3121

  2. transaction is very ethereum-specific:

 id  |        time_created        |                                hash                                | block_id | gas_limit | max_priority_fee_per_gas | max_fee_per_gas | gas_price  | effective_gas_price | nonce |                   sender                   |                 recipient                  | gas_used | cumulative_gas_used
-----+----------------------------+--------------------------------------------------------------------+----------+-----------+--------------------------+-----------------+------------+---------------------+-------+--------------------------------------------+--------------------------------------------+----------+---------------------
 299 | 2023-10-20 21:35:29.902644 | \xff8c550fd9d8311983af61ece61ba0983312e191e6fa0ea076540cc0749ea386 |      293 |    305335 |               1500000000 |      1500000100 | 1500000050 |          1500000050 |    76 | \xfad1c94469700833717fa8a3017278bc1ca8031c | \x086e902d2f99bcceaa28b31747ec6dc5fd43b1be |   250733 |              297586

We don't need cumulative_gas_used. The rest are probably ok to keep tbh

@avious00 avious00 changed the title Cosmos scraper epic Epic: Cosmos scraper Mar 1, 2024
@avious00 avious00 moved this to Backlog in Hyperlane Tasks Mar 1, 2024
@avious00 avious00 changed the title Epic: Cosmos scraper Epic: Add scraper support for Cosmos Mar 1, 2024
@avious00 avious00 moved this from Backlog to Next Sprint in Hyperlane Tasks Mar 2, 2024
@tkporter tkporter moved this from Next Sprint to Backlog in Hyperlane Tasks Mar 13, 2024
@cmcewen cmcewen closed this as completed Dec 19, 2024
@github-project-automation github-project-automation bot moved this from Backlog to Done in Hyperlane Tasks Dec 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: Done
Development

No branches or pull requests

3 participants