20221222 Merge Last sprint progress (#24)

* 74 index loanentity data types (#86) * feat: create loanService * refactor: improved service getters * feat: increase outstandingDebt on borrowings * feat: basics for handle loan prices * feat: save priced loans * feat: activate loan when priced * feat: handle writeoffs * feat: handle loan closed and write offs * fix: loan.repay() should increase totalRepaid * fix: invest execution decimals and avoid empty transactions (#87) * 82 add prices to all investortransaction types (#88) * chore: update chain data * feat: extend InvestorTransaction decoration and refactor * feat: use rpc prices from trancheTokenPrices() when investor transactions are processed (#89) * feat: improved loans and borrower transactions (#90) * 63 convert outflows to dai (#91) * feat: add currency value for redeem fields * fix: currency conversion should support null prices * 98 failing poolservicegettranchetokenprices causes period to be skipped for remaining pools (#99) * ci: fix deployment version * fix: handle missing price updates from RPC * fix: throw errors on missing prices * feat: aggregate total investments and repayments for epoches (#100) * feat: index accounts and tranche balances (#101) * docs: add deployment status badge * Update README.md docs: fix badge link to deployment workflow * feat: tracking loans history with state snapshot (#102) * 56 handle updates to pool information poolsupdated event (#106) * chore: upgraded packages * feat: track pool updates * 103 totalinvested and totalredeemed are missing in poolsnapshot (#107) * fix: add missing properties to snapshot * deploy: update node and query versions * fix: pool investment and redemptions totals * feat: indexing missing values (#108) * ci: updated deployment * ci: upgraded packages * feat: paginated getters for snapshots and models (#109) * chore: upgrade packages * 104 track currency balances for accounts (#111) * feat: add data model for currency balances * feat: track endowments withdrawals and deposits of currency * feat: get initial balance on currency balance init * ci: revert subql/node version * 112 develop suitable unit testing approach (#117) * test: fundamental approach * test: poolHandlers testing * test: run pools and tranche tests in CI * test: include generation of entities * ci: execute testing in correct order * docs: updated data model diagram (#118) * 115 extend kpis as from spreadsheet (#119) * feat: aggregate loan writeoffs and total overdue * feat: index additional data * chore: remove unnecessary imports * feat: parse and index loanSpecs * fix: indexing of overdue amount * 114-Integration and fixes for new centrifuge-chain release (#133) * 113 enhance standasdization and reusability of services (#126) * refactor: leverage inheritance of entities * fix: private methods * test: extend testing for new inheritance * test: state snapshotter (#127) * unit testing for investortransactions and loans (#128) * test: investor transaction tests * test: investor transaction amounts tests * test: loans test suite * fix: chain release * fix: tests * ci: upgrade deployment versions * ci: updated versions and parameters * ci: use deployment defaults * ci: disable historical data * 132 implement naming changes [BREAKING] (#134) * refactor: rework indexer naming * feat!: new suffix for field resetting on new pwriod * ci: disable flags for stati history tracking * ci: include empty dict dor deployment command * ci: disable historical data * ci: update subql cli * ci: configure project.yaml for different chains * ci: omit historical data * fix: reset property naming * ci: omit historical data flag in deploy * fix: update project.yaml * fix: precision of currency to token amount conversion * ci: escaping deploy.yaml * ci: deploy.yaml escaping * ci: more escaping * ci: more escaping * ci: escaping * ci: escaping with single quotes * ci: escaping hell * ci: more escaping * ci: no escaping * ci: syntax change * ci: new approach * ci: test yq * ci: test yq * ci: updated approach * ci: deploy version disabling history * ci: use frozen lockfile * chore: packages versions upgrade * ci: use matrix deployments * ci: fix yaml bug * ci: refactor deployment yaml * ci: reusable job * ci: convert import to workflow * ci: fix deployment type * ci: rename to inputs * ci: fix inputs * ci: use npx for subQL publish version CID * ci: use null dictionary * ci: use empty dict * ci: use npx overall * ci: omit dictionary * ci: null dict * ci: use defaults * ci: derive name and description from manifest * ci: use CAPS env * ci: indexer and query versions defined in imported workflow * ci: add chain yamls Co-authored-by: Timon <timon@embrio.tech>
embrio-tech · Dec 22, 2022 · d3088bc · d3088bc
1 parent 514d398
commit d3088bc
Show file tree

Hide file tree

Showing 43 changed files with 6,295 additions and 4,311 deletions.
diff --git a/.github/workflows/deploy.yml b/.github/workflows/deploy.yml
@@ -6,62 +6,57 @@ on:
   push:
     branches:
       - main
+      - release-*
   workflow_dispatch:
 
 permissions:
   contents: read
   issues: write
 
 jobs:
-  subql_deploy:
-    name: Deploy to SubQL
-    runs-on: ubuntu-latest
-    env:
-      SUBQL_ACCESS_TOKEN: ${{ secrets.SUBQL_ACCESS_TOKEN }}
-      SUBQL_PROJ_ORG: embrio-tech
-      SUBQL_PROJ_NAME: centrifuge-subql
-      SUBQL_PROJ_DESCRIPTION: 'SubQuery API powered by EMBRIO.tech to query Centrifuge chain data for analytics insights on how Centrifuge is unlocking economic opportunity for all by connecting people to borrow and lend money transparently and cost-effectively.'
-      SUBQL_PROJ_IMAGE: 'https://explorer.subquery.network/uploads/1660216119964.png'
-      SUBQL_ENDPOINT: wss://fullnode.development.cntrfg.com
-    steps:
-      - name: Check out repo's default branch
-        uses: actions/checkout@v3
-      - name: Setup Node
-        uses: actions/setup-node@v3
-        with:
-          node-version: 16
-          cache: npm
-      - name: Setup Yarn
-        run: yarn
-      - name: Codegen
-        run: yarn codegen
-      - name: Version
-        run: npx subql --version
-      - name: Delete Project
-        run: |
-          npx subql project:delete-project \
-          --org="$SUBQL_PROJ_ORG" \
-          --projectName="$SUBQL_PROJ_NAME"
-        continue-on-error: true
-      - name: Create Project
-        run: |
-          npx subql project:create-project \
-          --apiVersion=2 \
-          --description="$SUBQL_PROJ_DESCRIPTION" \
-          --gitRepo="https://github.com/$GITHUB_REPOSITORY" \
-          --org="$SUBQL_PROJ_ORG" \
-          --projectName="$SUBQL_PROJ_NAME" \
-          --logoURL="$SUBQL_PROJ_IMAGE"
-      - name: Deploy Version
-        run: |
-          IPFSCID=$(npx subql publish -o -f .)
-          sleep 5
-          npx subql deployment:deploy \
-          --org="$SUBQL_PROJ_ORG" \
-          --endpoint="$SUBQL_ENDPOINT" \
-          --dict=" " \
-          --projectName="$SUBQL_PROJ_NAME" \
-          --ipfsCID="$IPFSCID" \
-          --type=primary \
-          --indexerVersion="v1.9.2" \
-          --queryVersion="v1.6.0"
+  subql_deploy_embrio_staging:
+    if: ${{ github.repository == 'embrio-tech/centrifuge-subql'}}
+    name: Deploy to SubQL (EMBRIO staging)
+    strategy:
+      matrix:
+        chainId: [ centrifuge-devel-embrio ] # ADD ARRAY OF CHAINS TO DEPLOY
+    uses: ./.github/workflows/subql_deploy_workflow.yaml
+    with:
+      chainId: ${{ matrix.chainId }}
+      projOrg: embrio-tech
+      projImage: 'https://explorer.subquery.network/uploads/1660216119964.png'
+      deploymentType: primary
+      resetProject: true
+    secrets:
+      accessToken: ${{ secrets.SUBQL_ACCESS_TOKEN }}
+
+  subql_deploy_centrifuge_staging:
+    if: ${{ github.repository == 'centrifuge/pools-subql' && github.ref_name == 'main' }}
+    name: Deploy to SubQL (CENTRIFUGE staging)
+    strategy:
+      matrix:
+        chainId: [ centrifuge-devel, altair,  ] # ADD ARRAY OF CHAINS TO DEPLOY
+    uses: ./.github/workflows/subql_deploy_workflow.yaml
+    with:
+      chainId: ${{ matrix.chainId }}
+      projOrg: centrifuge
+      projImage: 'https://explorer.subquery.network/uploads/1660216119964.png'
+      deploymentType: primary
+      resetProject: true
+    secrets:
+      accessToken: ${{ secrets.SUBQL_ACCESS_TOKEN }}
+
+  subql_deploy_centrifuge_production:
+    if: ${{ github.repository == 'centrifuge/pools-subql' && startsWith(github.ref_name, 'release-') }}
+    name: Deploy to SubQL (CENTRIFUGE mainnet)
+    strategy:
+      matrix:
+        chainId: [ altair ] # ADD ARRAY OF CHAINS TO DEPLOY
+    uses: ./.github/workflows/subql_deploy_workflow.yaml
+    with:
+      chainId: ${{ matrix.chainId }}
+      projOrg: centrifuge
+      deploymentType: stage
+      resetProject: false
+    secrets:
+      accessToken: ${{ secrets.SUBQL_ACCESS_TOKEN }}
diff --git a/.github/workflows/subql_deploy_workflow.yaml b/.github/workflows/subql_deploy_workflow.yaml
@@ -0,0 +1,91 @@
+on:
+  workflow_call:
+    inputs:
+      chainId:
+        required: true
+        type: string
+      projOrg:
+        required: true
+        type: string
+      projImage:
+        required: false
+        type: string
+      deploymentType:
+        description: stage or primary
+        required: false
+        type: string
+        default: stage
+      resetProject:
+        default: false
+        type: boolean
+    secrets:
+      accessToken:
+        required: true
+
+
+jobs:
+  subql_deploy_workflow:
+    runs-on: ubuntu-latest
+    env:
+      SUBQL_INDEXER_VERSION: v1.15.1
+      SUBQL_QUERY_VERSION: v1.9.1
+      CHAIN_ID: ${{ inputs.chainId }}
+      SUBQL_ACCESS_TOKEN: ${{ secrets.accessToken }}
+      SUBQL_PROJ_ORG: ${{ inputs.projOrg }}
+      SUBQL_PROJ_IMAGE: ${{ inputs.projImage }}
+      SUBQL_DEPLOYMENT_TYPE: ${{ inputs.deploymentType }}
+    steps:
+      - name: Check out repo's default branch
+        uses: actions/checkout@v3
+      - name: Generate new project.yaml
+        run: yq ". *= load(\"chains/$CHAIN_ID.yaml\")" chains/base.yaml > project.yaml
+      - name: Extract Chain Parameters to ENV
+        run: |
+          echo "SUBQL_PROJ_NAME=$(yq '.name' project.yaml)" >> $GITHUB_ENV
+          echo "SUBQL_PROJ_DECRIPTION=$(yq '.description' project.yaml)" >> $GITHUB_ENV
+          echo "SUBQL_ENDPOINT=$(yq '.network.endpoint' project.yaml)" >> $GITHUB_ENV
+      - name: Setup Node
+        uses: actions/setup-node@v3
+        with:
+          node-version: 18
+          cache: npm
+      - name: Setup Yarn
+        run: yarn install --frozen-lockfile
+      - name: Codegen
+        run: yarn codegen
+      - name: SubQL CLI Version
+        run: yarn subql --version
+      - name: Publish SubQL version
+        run: |
+          IPFSCID=$(npx subql publish -o -f .)
+          echo "IPFSCID=$IPFSCID" >> $GITHUB_ENV
+          echo "CID: $IPFSCID"
+      - name: Delete Project
+        if: ${{ inputs.resetProject }}
+        run: |
+          npx subql project:delete-project \
+          --org="$SUBQL_PROJ_ORG" \
+          --projectName="$SUBQL_PROJ_NAME"
+        continue-on-error: true
+      - name: Create Project
+        if: ${{ inputs.resetProject }}
+        run: |
+          npx subql project:create-project \
+          --apiVersion=2 \
+          --description="$SUBQL_PROJ_DESCRIPTION" \
+          --gitRepo="https://github.com/$GITHUB_REPOSITORY" \
+          --org="$SUBQL_PROJ_ORG" \
+          --projectName="$SUBQL_PROJ_NAME" \
+          --logoURL="$SUBQL_PROJ_IMAGE"
+      - name: Deploy Version
+        run: |
+          npx subql deployment:deploy \
+          -d \
+          --org="$SUBQL_PROJ_ORG" \
+          --endpoint="$SUBQL_ENDPOINT" \
+          --projectName="$SUBQL_PROJ_NAME" \
+          --ipfsCID="$IPFSCID" \
+          --type=$SUBQL_DEPLOYMENT_TYPE \
+          --disableHistorical \
+          --indexerVersion="$SUBQL_INDEXER_VERSION" \
+          --queryVersion="$SUBQL_QUERY_VERSION"
diff --git a/.nvmrc b/.nvmrc
@@ -1 +1 @@
-16
+18
diff --git a/.vscode/launch.json b/.vscode/launch.json
@@ -13,7 +13,16 @@
       "name": "Debug SubQL Node",
       "type": "node",
       "request": "launch",
-      "runtimeArgs": ["--inspect-brk", "${workspaceRoot}/node_modules/.bin/subql-node", "-f", "${workspaceRoot}"],
+      "runtimeArgs": [
+        "--inspect-brk",
+        "${workspaceRoot}/node_modules/.bin/subql-node",
+        "-f",
+        "${workspaceRoot}/",
+        "--db-schema",
+        "app",
+        "--disable-historical",
+        "true"
+      ],
       "console": "integratedTerminal",
       "internalConsoleOptions": "neverOpen"
     }

diff --git a/chains/altair.yaml b/chains/altair.yaml
@@ -0,0 +1,6 @@
+name: altair-pools #CHANGE BASED ON SUBQL PROJECT NAME
+network:
+  endpoint: wss://fullnode.altair.centrifuge.io
+  genesisHash: '0xaa3876c1dc8a1afcc2e9a685a49ff7704cfd36ad8c90bf2702b9d1b00cc40011'
+  chaintypes:
+    file: ./dist/chaintypes.js
diff --git a/chains/antares.yaml b/chains/antares.yaml
@@ -0,0 +1,6 @@
+name: antares-pools #CHANGE BASED ON SUBQL PROJECT NAME
+network:
+  endpoint: wss://fullnode-collator.antares.centrifuge.io
+  genesisHash: '???' # CHANGE ME
+  chaintypes:
+    file: ./dist/chaintypes.js
diff --git a/chains/base.yaml b/chains/base.yaml
@@ -0,0 +1,121 @@
+specVersion: 1.0.0
+name: centrifuge-pools
+version: 1.0.0
+runner:
+  node:
+    name: '@subql/node'
+    version: '1'
+  query:
+    name: '@subql/query'
+    version: '1'
+description: 'SubQuery API powered by EMBRIO.tech to query Centrifuge chain data for analytics insights on how Centrifuge is unlocking economic opportunity for all by connecting people to borrow and lend money transparently and cost-effectively.'
+repository: https://github.com/centrifuge/pools-subql
+schema:
+  file: ./schema.graphql
+dataSources:
+  - kind: substrate/Runtime
+    startBlock: 1
+    mapping:
+      file: ./dist/index.js
+      handlers:
+        - handler: handleBlock
+          kind: substrate/BlockHandler
+          filter:
+            timestamp: "1 0,12 * * *"
+        - handler: logEvents
+          kind: substrate/EventHandler
+        - handler: handlePoolCreated
+          kind: substrate/EventHandler
+          filter:
+            module: poolSystem
+            method: PoolCreated
+        - handler: handleEpochClosed
+          kind: substrate/EventHandler
+          filter:
+            module: poolSystem
+            method: EpochClosed
+        - handler: handleEpochExecuted
+          kind: substrate/EventHandler
+          filter:
+            module: poolSystem
+            method: EpochExecuted
+        - handler: handleInvestOrderUpdated
+          kind: substrate/EventHandler
+          filter:
+            module: investments
+            method: InvestOrderUpdated
+        - handler: handleRedeemOrderUpdated
+          kind: substrate/EventHandler
+          filter:
+            module: investments
+            method: RedeemOrderUpdated
+        - handler: handleInvestOrdersCollected
+          kind: substrate/EventHandler
+          filter:
+            module: investments
+            method: InvestOrdersCollected
+        - handler: handleRedeemOrdersCollected
+          kind: substrate/EventHandler
+          filter:
+            module: investments
+            method: RedeemOrdersCollected
+        - handler: handleProxyAdded
+          kind: substrate/EventHandler
+          filter:
+            module: proxy
+            method: ProxyAdded
+        - handler: handleProxyRemoved
+          kind: substrate/EventHandler
+          filter:
+            module: proxy
+            method: ProxyRemoved
+        - handler: handleProxyAnonymousCreated
+          kind: substrate/EventHandler
+          filter:
+            module: proxy
+            method: AnonymousCreated
+        - handler: handleLoanCreated
+          kind: substrate/EventHandler
+          filter:
+            module: loans
+            method: Created
+        - handler: handleLoanBorrowed
+          kind: substrate/EventHandler
+          filter:
+            module: loans
+            method: Borrowed
+        - handler: handleLoanPriced
+          kind: substrate/EventHandler
+          filter:
+            module: loans
+            method: Priced
+        - handler: handleLoanRepaid
+          kind: substrate/EventHandler
+          filter:
+            module: loans
+            method: Repaid
+        - handler: handleLoanClosed
+          kind: substrate/EventHandler
+          filter:
+            module: loans
+            method: Closed
+        - handler: handleTokenTransfer
+          kind: substrate/EventHandler
+          filter:
+            module: ormlTokens
+            method: Transfer
+        - handler: handleTokenEndowed
+          kind: substrate/EventHandler
+          filter:
+            module: ormlTokens
+            method: Endowed
+        - handler: handleTokenDeposited
+          kind: substrate/EventHandler
+          filter:
+            module: ormlTokens
+            method: Deposited
+        - handler: handleTokenWithdrawn
+          kind: substrate/EventHandler
+          filter:
+            module: ormlTokens
+            method: Withdrawn
diff --git a/chains/catalyst.yaml b/chains/catalyst.yaml
@@ -0,0 +1,6 @@
+name: catalyst-pools #CHANGE BASED ON SUBQL PROJECT NAME
+network:
+  endpoint: wss://fullnode.catalyst.cntrfg.com
+  genesisHash: '0x6cb9c6d7d3c9150a1f3cd75784834c53a12d5e0c2d4ed9bf694a901be564eac9'
+  chaintypes:
+    file: ./dist/chaintypes.js
diff --git a/chains/centrifuge-devel-embrio.yaml b/chains/centrifuge-devel-embrio.yaml
@@ -0,0 +1,7 @@
+name: 'centrifuge-subql'
+repository: 'https://github.com/embrio-tech/centrifuge-subql'
+network:
+  endpoint: wss://fullnode.development.cntrfg.com
+  genesisHash: '0xb59f31491c2dd2760c7a5aa3d3d3694678bbfffce99ab487228b980921a027d3'
+  chaintypes:
+    file: ./dist/chaintypes.js
diff --git a/chains/centrifuge-devel.yaml b/chains/centrifuge-devel.yaml
@@ -0,0 +1,6 @@
+name: '' #CHANGE ME BASED ON SUBQL DEPLOYMENT
+network:
+  endpoint: wss://fullnode.demo.cntrfg.com
+  genesisHash: '0xb59f31491c2dd2760c7a5aa3d3d3694678bbfffce99ab487228b980921a027d3'
+  chaintypes:
+    file: ./dist/chaintypes.js