Gh#4968 inline actions

[heifner]

Resolves #4968

• Action traces now stored instead of actions
  -- New action_traces collection replaces actions collection
  -- action_traces include all actions including inline, implicit, and scheduled
• irreversible attribute on transactions and blocks is now only added with a value of true when irreversible to prevent a race condition of an update of transaction overwriting the irreversible true with a false. This is now needed since upsert is used for insert and strict ordering is not guaranteed.
• Simplified transaction serialization and now store all eosio abi as abi_def instead of bytes.
• Added filter-on and filter-out similar to history_plugin.
  -- Thanks to Scott Sallinen (TeamGreymass) for implementation.
• Added options to not store blocks, block-states, transactions, transaction-traces, and action-traces.

[heifner]

brief of changes:

The optional eosio::mongo_db_plugin provides archiving of blockchain data into a MongoDB. It is recommended that the plugin be added to a non-producing node as it is designed to shut down on any failed insert into the MongoDB and is resource intensive. For best results dedicate a nodeos instance to running this one plugin. The rationale behind this shutdown on error is so that any issues with the mongo database or connectivity can be fixed and the nodeos can be restarted without having to resync or replay.

The mongo_db_plugin does slow down replay/resync as the conversion of block data to JSON and insertion into MongoDB is resource intensive. The plugin does use a worker thread for processing the block data, but this does not help much when replaying/resyncing.

It is recommended that a large --abi-serializer-max-time-ms value be passed into the nodeos running the mongo_db_plugin as the default abi serializer time limit is not large enough to serialize large blocks.

It is also recommended that read-only mode is used to avoid speculative execution. See read-only db mode documentation. Forked data is still recorded (data that never becomes irreversible) but speculative transaction processing and signaling is avoided minimizing the transaction_traces/action_traces stored.

action data is stored on chain as raw bytes. This plugin attempts to use associated ABI on accounts to deserialize the raw bytes into explanded abi_def form for storage into mongo. Note that invalid or missing abi on a contract will result in the action data being stored as raw bytes. For example the eosio system contract does not provide abi for the onblock action so it is stored as raw bytes.

mongo_db_plugin options:

Config Options for eosio::mongo_db_plugin:
  -q [ --mongodb-queue-size ] arg (=1024)
                                        The target queue size between nodeos 
                                        and MongoDB plugin thread.
  --mongodb-abi-cache-size arg (=2048)  The maximum size of the abi cache for 
                                        serializing data.
  --mongodb-wipe                        Required with --replay-blockchain, 
                                        --hard-replay-blockchain, or 
                                        --delete-all-blocks to wipe mongo 
                                        db.This option required to prevent 
                                        accidental wipe of mongo db.
  --mongodb-block-start arg (=0)        If specified then only abi data pushed 
                                        to mongodb until specified block is 
                                        reached.
  -m [ --mongodb-uri ] arg              MongoDB URI connection string, see: 
                                        https://docs.mongodb.com/master/referen
                                        ce/connection-string/. If not specified
                                        then plugin is disabled. Default 
                                        database 'EOS' is used if not specified
                                        in URI. Example: mongodb://127.0.0.1:27
                                        017/EOS
  --mongodb-store-blocks (=1)           Enables storing blocks in mongodb.
  --mongodb-store-block-states (=1)     Enables storing block state in mongodb.
  --mongodb-store-transactions (=1)     Enables storing transactions in 
                                        mongodb.
  --mongodb-store-transaction-traces (=1) 
                                        Enables storing transaction traces in 
                                        mongodb.
  --mongodb-store-action-traces (=1)    Enables storing action traces in 
                                        mongodb.
  --mongodb-filter-on arg               Mongodb: Track actions which match 
                                        receiver:action:actor. Actor may be 
                                        blank to include all. Receiver and 
                                        Action may not be blank. Default is * 
                                        include everything.
  --mongodb-filter-out arg              Mongodb: Do not track actions which 
                                        match receiver:action:actor. Action and
                                        Actor both blank excludes all from 
                                        reciever. Actor blank excludes all from
                                        reciever:action. Receiver may not be 
                                        blank.

--mongodb-store-* options all default to true.
--mongodb-filter-* options currently only applies to the action_traces collection.

Example filter options:

mongodb-filter-out = eosio:onblock:
mongodb-filter-out = gu2tembqgage::
mongodb-filter-out = blocktwitter:: 

The following mongo collections are created:

• accounts - created on applied transaction. Always updated even if mongodb-store-action-traces=false.

  • Currently limited to just name and ABI if contract abi on account
  • Mostly for internal use as the stored ABI is used to convert action data into JSON for storage as associated actions on contract are processed.
  • Invalid ABI on account will prevent conversion of action data into JSON for storage resulting in just the action data being stored as hex. For example, the original eosio.system contract did not provide ABI for the onblock action and therefore all 'onblockaction data is stored as hex until the timeonblock` ABI is added to the eosio.system contract.

• action_traces - created on applied transaction

  • receipt - action_trace action_receipt - see eosio::chain::action_receipt
  • trx_id - transaction id
  • act - action - see eosio::chain::action
  • elapsed - time in microseconds to execute action
  • console - console output of action. Always empty unless contracts-console = true option specified.

• block_states - created on accepted block

  • block_num
  • block_id
  • block_header_state - see eosio::chain::block_header_state
  • validated
  • in_current_chain

• blocks - created on accepted block

  • block_num
  • block_id
  • block - signed block - see eosio::chain::signed_block
  • validated - added on irreversible block
  • in_current_chain - added on irreversible block
  • irreversible=true - added on irreversible block

• transaction_traces - created on applied transaction

  • see chain::eosio::transaction_trace

• transactions - created on accepted transaction - does not include inline actions

  • see eosio::chain::signed_transaction. In addition to signed_transaction data the following are also stored.
  • trx_id - transaction id
  • irreversible=true - added on irreversible block
  • block_id - added on irreversble block
  • block_num - added on irreversible block
  • signing_keys
  • accepted
  • implicit
  • scheduled

• account_controls - created on applied transaction. Always updated even if mongodb-store-action-traces=false.

  • controlled_account
  • controlling_permission
  • controlling_account

The equivalent of /v1/history/get_controlled_acounts with mongo: db.account_controls.find({"controlling_account":"hellozhangyi"}).pretty()

• pub_keys - created on applied transaction. Always updated even if mongodb-store-action-traces=false.
  • account
  • premission
  • public_key

The equivalent of /v1/history/get_key_accounts with mongo:
db.pub_keys.find({"public_key":"EOS7EarnUhcyYqmdnPon8rm7mBCTnBoot6o7fE2WzjvEX2TdggbL3"}).pretty()

[DenisCarriere]

Looking forward to see this merged in the future releases 😄

[heifner]

@DenisCarriere Sorry about the actions->action_traces and changes to transactions. Should be more stable going forward. Hopefully, it will not cost you too much rework on: https://github.com/EOS-BP-Developers/eosio-mongodb-queries . On the plus side it is much faster and now includes scheduled transactions and inline actions.

[brianjohnson5972]

Is there a reason not to optimize this with if (!include && ...

[brianjohnson5972]

Wrap in if (!include) {...}

[brianjohnson5972]

add break to optimize

[brianjohnson5972]

Wondering if using the linux command line convention would make this clearer:
receiver:action:[actor]
??

[brianjohnson5972]

if we do above then this would be receiver:[action]:[actor]

[DenisCarriere]

@heifner No worries, I'll adjust accordingly to these changes.

I'll rework MongoDB Queries once this is released 👍

[DenisCarriere]

🎉 Woot

[DenisCarriere]

I'm trying to only use mongodb-filter-on however it still returns all action_traces as if it would be mongodb-filter-on=* (the default).

config.ini example

mongodb-filter-on = eosio:delegatebw:
mongodb-filter-on = eosio:undelegatebw:

At the moment, this config still returns eosio:onblock & all others

Maybe I'm missing something?

CC: @heifner & @ScottSallinen

[heifner]

Looks like you are correct. I'll get a fix put in.

[jafri]

Perhaps a timestamp on both transaction_trace and action_trace would be useful