You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The Extract-Transform-Load component, ETL, aims to supply historical data for further data analysis. The data is obtained primarily from a software system referred to as the Blockchain System, and then loaded into a relational database referred to as the DB. The structure of the data does not undergo any changes. Some data attributes need value transformation in order to be easily consumed afterward.
Architecture
The ETL component runs as a standalone service under the supervision of a system services manager, for example, systemctl. The service connects to the Blockchain System to receive events and query entities, a Market Data Provider to obtain aggregated market data, and the DB to push the data by inserting records. No updates, nor deletes of DB records.
Global configuration
an endpoint to a Nolus node
addresses of the smart contract instances of interest, see this example as a reference for the structure
market data provider, type and endpoint, for example Coinmarketcap or Coingecko
DB connection parameters
supported currencies - a set of (Nolus internal symbol, currency symbol, decimal digits) tuples providing the mapping between the currency symbols used internally by the system and their external equivalent. Currency symbols are unique.
stable currency - a symbol of the currency the market prices are into. The stable may or may not be amongst the supported currencies.
aggregation interval - number of hours between each aggregation. The interval is global so the records created at the end of an interval bear same timestamp.
Data Model
The data model consists of entities and aggregated state.
The entity instances usually represent transactions, whereas the state accumulates the effect of those transactions.
Data Extraction Mechanism
Entities
Except MP_Asset all other entities are received with Blockchain Events. The specification below defines the events and their attributes. Obtaining MP_Asset data is performed at regular intervals by a mechanism specific to the Market Data Provider API.
State
Except LP_Pool all other state data is prepared by the System and is retrieved querying it at aggregation intervals. LP_Pool is populated initially as specified below.
The specification of all Data Types along with their attributes, types and source follows. By naming convention, a State Data Type ends in "_State".
The precision of all date time values is up to a second.
MP_Asset - Historical Data [Primary key = MP_asset_symbol + MP_asset_timestamp]
The price data is obtained from the market data provider once a preconfigured interval for all supported currencies. The prices are against the stable currency.
The lender's total amount of pool's native asset in stable
LP_Lender_asset
Unsigned Int(128)
int(lpp.balance.balance * lpp.price.price)
The lender's total amount of pool's native asset
LP_Lender_receipts
Unsigned Int(128)
lpp.balance.balance
The lender's total amount of pool's receipts
LP_Pool [Primary Key = LP_Pool_id]
Populated initially with an instance per an Lpp smart contract instance configured with smart contract instances. Consult the Lpp query API on details of how to obtain Lpp currency symbol. It is one of the supported currencies internal currency symbols.
Property Name
Type
Origin
Description
LP_Pool_id
Alphanumeric(64)
Smart Contract ID
Liquidity Providers' Pool, LPP, Smart Contract Instance ID
The total value locked amount of pool's native asset in LP_Pool.LP_symbol = LPN
LP_Pool_total_issued_receipts
Unsigned Int(128)
lpp.lpp_balance.balance_nlpn
The total amount of pool's receipts issued to the lenders on each deposit
LP_Pool_total_borrowed_stable
Unsigned Int(128)
in_stable(lpp.lpp_balance.total_principal_due)
The total amount borrowed from pool's native asset in stable
LP_Pool_total_borrowed_asset
Unsigned Int(128)
lpp.lpp_balance.total_principal_due
The total amount borrowed from pool's native asset in LP_Pool.LP_symbol = LPN
LP_Pool_total_yield_stable
Unsigned Int(128)
0
The total amount in yield in stable
LP_Pool_total_yield_asset
Unsigned Int(128)
0
The total amount in yield in LP_Pool.LP_symbol = LPN
MP_Yield - Historical Data [Primary key = MP_yield_symbol + MP_yield_timestamp]
Property Name
Type
Query API
Description
MP_yield_symbol
Alphanumeric(20)
TBD
Unique name of the yield source
MP_yield_timestamp
Timestamp
TBD
Date time at which the information is relevant
MP_apy_permilles
Int(16)
TBD
Annual Percentage Yield
TR_Profit - Transfers from bought-back NLS into Treasury [Primary key = TR_Profit_height + TR_Profit_idx]
Property Name
Type
Event, Attribute
Description
TR_Profit_height
Int(64)
wasm-tr-profit.height
Height of the block this transaction is in
TR_Profit_idx
Int(32)
wasm-tr-profit.idx
Index in the block this transaction is at
TR_Profit_timestamp
Timestamp
wasm-tr-profit.at
Date time of the profit transfer
TR_Profit_amnt_stable
Unsigned Int(128)
in_stable(wasm-tr-profit.profit-amount)
The amount transferred in stable. The currency symbol is carried with wasm-tr-profit.profit-symbol
TR_Profit_amnt_nls
Unsigned Int(128)
wasm-tr-profit.profit-amount
The amount transferred in NLS. The currency symbol is carried with wasm-tr-profit.profit-symbol
Tx_Hash
Alphanumeric(64)
tx hash
Transaction hash
TR_Rewards_Distribution - Transfers from Treasury to the Liquidity Pools [Primary key = TR_Rewards_height + TR_Rewards_idx + TR_Rewards_Pool_id]
Property Name
Type
Event, Attribute
Description
TR_Rewards_height
Int(64)
wasm-tr-rewards.height
Height of the block this transaction is in
TR_Rewards_idx
Int(32)
wasm-tr-rewards.idx
Index in the block this transaction is at
TR_Rewards_Pool_id
Alphanumeric(64)
wasm-tr-rewards.to
Liquidity Providers' Pool, LPP, Smart Contract ID
TR_Rewards_timestamp
Timestamp
wasm-tr-rewards.at
Date time of the reward distribution
TR_Rewards_amnt_stable
Unsigned Int(128)
in_stable(wasm-tr-rewards.rewards-amount)
The amount transferred in stable. The currency symbol is carried with wasm-tr-rewards.rewards-symbol
TR_Rewards_amnt_nls
Unsigned Int(128)
wasm-tr-rewards.rewards-amount
The amount transferred in NLS. The currency symbol is carried with wasm-tr-rewards.rewards-symbol
Tx_Hash
Alphanumeric(64)
tx hash
Transaction hash
TR_State [Primary Key = TR_timestamp]
The balance of the Treasury is obtained by querying the system with a Bank Balances message. The result is a list of coins. ETL must report an error if there is more than one coin.
Property Name
Type
Smart Contract, Query API
Description
TR_timestamp
Timestamp
The end of <aggregation interval>
Date time at which the information is relevant
TR_amnt_stable
Unsigned Int(128)
in_stable(bank.balances[0]::0)
The total amount in yield in stable
TR_amnt_nls
Unsigned Int(128)
bank.balances[0]::0
The total amount in yield in NLS
PL_State [Primary key = PL_timestamp]
Aggragation is done over all records pertaining to the same aggregation interval. For example, PL_pools_TVL_stable is equal to the sum of LP_Pool_State::LP_Pool_total_value_locked_stable for all LP_Pool_id up to the same interval.
Property Name
Type
Smart Contract, Query API
Description
PL_timestamp
Timestamp
The end of <aggregation interval>
Date time at which the information is relevant
PL_pools_TVL_stable
Unsigned Int(140)
sum(LP_Pool_State::LP_Pool_total_value_locked_stable) for all records with LP_Pool_timestamp == PL_timestamp
Total locked Funds in all pool instances
PL_pools_borrowed_stable
Unsigned Int(140)
sum(LP_Pool_State::LP_Pool_total_borrowed_stable) for all records with LP_Pool_timestamp == PL_timestamp
Total borrowed Funds in all pool instances
PL_pools_yield_stable
Unsigned Int(140)
sum(LP_Pool_State::LP_Pool_total_yield_stable) for all records with LP_Pool_timestamp == PL_timestamp
Total Funds in yield from all pool instances
PL_LS_count_open
Unsigned Int(64)
count(LS_State) all records with LS_Timestamp == PL_timestamp
Total Lease Accounts in open state, i.e. with pending loans or unclaimed
PL_LS_count_closed
Unsigned Int(64)
count(LS_Closing) all records with LS_Timestamp within the interval
Total Lease Accounts closed since the last Platform snapshot
PL_LS_count_opened
Unsigned Int(64)
count(LS_Opening) all records with LS_Timestamp within the interval
Total Lease Accounts opened since the last Platform snapshot
PL_IN_LS_cltr_amnt_opened_stable
Unsigned Int(140)
sum(LS_Opening::LS_cltr_amnt_stable) all records with LS_Timestamp within the interval
Total collateral received on opening of Lease Accounts since the last Platform snapshot
PL_LP_count_open
Unsigned Int(64)
count(LP_Lender_State) all records with LP_Timestamp == PL_timestamp
Total Deposit Accounts in open state, i.e. with a positive balance
PL_LP_count_closed
Unsigned Int(64)
count(LP_Withdraw) all records with LP_Timestamp within the interval and LP_deposit_close == True
Total Deposit Accounts closed since the last Platform snapshot, i.e. withdrawn the whole amount
PL_LP_count_opened
Unsigned Int(64)
count(LP_Deposit) all records with LP_Timestamp within the interval
Total Deposit Accounts opened since the last Platform snapshot, i.e. new deposits by new lenders
PL_OUT_LS_loan_amnt_stable
Unsigned Int(140)
sum(LS_Opening::LS_loan_amnt_stable) all records with LS_Timestamp within the interval
Total amount of issued loans in stable since the last Platform snapshot
PL_IN_LS_rep_amnt_stable
Unsigned Int(140)
sum(next four PLIN_LS_rep*)
Total amount of repayments in stable since the last Platform snapshot
PL_IN_LS_rep_prev_margin_stable
Unsigned Int(140)
sum(LS_Repayment::LS_prev_margin_stable) all records with LS_Timestamp within the interval
Total amount of margin amounts for the previous period since the last Platform snapshot
PL_IN_LS_rep_prev_interest_stable
Unsigned Int(140)
sum(LS_Repayment::LS_prev_interest_stable) all records with LS_Timestamp within the interval
Total amount of interest amounts for the previous period since the last Platform snapshot
PL_IN_LS_rep_current_margin_stable
Unsigned Int(140)
sum(LS_Repayment::LS_current_margin_stable) all records with LS_Timestamp within the interval
Total amount of margin amounts for the current period since the last Platform snapshot
PL_IN_LS_rep_current_interest_stable
Unsigned Int(140)
sum(LS_Repayment::LS_current_interest_stable) all records with LS_Timestamp within the interval
Total amount of interest amounts for the current period since the last Platform snapshot
PL_IN_LS_rep_principal_stable
Unsigned Int(140)
sum(LS_Repayment::LS_principal_stable) all records with LS_Timestamp within the interval
Total amount of paid principal amounts since the last Platform snapshot
PL_OUT_LS_cltr_amnt_stable
Unsigned Int(140)
sum(LS_Opening::LS_cltr_amnt_stable) for all records LS_Opening::LS_contract_id == LS_Closing::LS_contract_id and LS_Closing::LS_Timestamp within the interval
Total amount of released collateral amount in stable since the last Platform snapshot
PL_OUT_LS_amnt_stable
Unsigned Int(140)
sum(LS_Opening::LS_loan_amnt_stable + LS_Opening::LS_cltr_amnt_stable) for all records LS_Opening::LS_contract_id == LS_Closing::LS_contract_id and LS_Closing::LS_Timestamp within the interval
Total amount of released lease amounts in stable since the last Platform snapshot
PL_native_amnt_stable
Unsigned Int(140)
0
Total NLS amount in stable used to get discount since the last Platform snapshot
PL_native_amnt_nolus
Unsigned Int(140)
0
Total NLS amount used to get discount since the last Platform snapshot
PL_IN_LP_amnt_stable
Unsigned Int(140)
sum(LP_Deposit::LP_amnt_stable) for all records with LP_Timestamp within the interval
Total deposited amount in stable since the last Platform snapshot
PL_OUT_LP_amnt_stable
Unsigned Int(140)
sum(LP_Withdraw::LP_amnt_stable) for all records with LP_Timestamp within the interval
Total withdrawn amount in stable since the last Platform snapshot
PL_TR_profit_amnt_stable
Unsigned Int(140)
sum(TR_Profit::TR_Profit_amnt_stable) for all records with TR_Profit_timestamp within the interval
Total amount bought-back in stable since the last Platform snapshot
PL_TR_profit_amnt_nls
Unsigned Int(140)
sum(TR_Profit::TR_Profit_amnt_nls) for all records with TR_Profit_timestamp within the interval
Total amount bought-back in NLS since the last Platform snapshot
PL_TR_tax_amnt_stable
Unsigned Int(128)
TR_State::TR_amnt_stable + PL_OUT_TR_rewards_amnt_stable - PL_TR_profit_amnt_stable - TR_State::TR_amnt_stable at the end of the previous period
Total amount of the additional tax charged on each transaction in stable since the last Platform snapshot, state_old + tax + profit - rewards = state_now => tax = state_now + rewards - profit - state_old
PL_TR_tax_amnt_nls
Unsigned Int(128)
TR_State::TR_amnt_nls + PL_OUT_TR_rewards_amnt_nls - PL_TR_profit_amnt_nls - TR_State::TR_amnt_nls at the end of the previous period
Total amount of the additional tax charged on each transaction in NLS since the last Platform snapshot
PL_OUT_TR_rewards_amnt_stable
Unsigned Int(140)
sum(TR_Rewards_Distribution::TR_Rewards_amnt_stable) for all records with TR_Rewards_Distribution::TR_Rewards_timestamp within the interval
Total amount transferred in stable since the last Platform snapshot
PL_OUT_TR_rewards_amnt_nls
Unsigned Int(140)
sum(TR_Rewards_Distribution::TR_Rewards_amnt_nls) for all records with TR_Rewards_Distribution::TR_Rewards_timestamp within the interval
Total amount transferred in NLS since the last Platform snapshot
Due to the lack of unsigned integer types in the databases with more that 64 bits we define the mapping of Unsigned Int(128) as Decimal(log(2 ** 128, 10), 0), i.e. Decimal(39, 0). Simmilarly, the fields that accumulate such values we map into Decimal(log(2 ** 140, 10), 0), i.e. Decimal(42, 0).
Logging
Any significant event that may further be used for troubleshooting or user information should be logged out from the system. The destination of the logs is the system log.