feat: chainSpec based controller config; Types from apps-config #351

emostov · 2020-11-24T00:19:23Z

Closes #334

Overview

This PR introduces the foundations of allowing Sidecar to work smoothly with FRAME based chains. It accomplishes this by

including type definitions from @polkadot/apps-config which are injected into the API
Creating a way to specify which controllers to mount based on a chains specName and hard coded config that aligns with the specName.

These changes will allow most FRAME chain builders to use Sidecar by just specifying end points their chain supports in the controllers config and keeping there chain types up to date in @polkadot/apps-config.

In this PR I also include controller configuration for Kulupu since it was requested in an issue (#350) and makes sense to have a live example to try out the setup.

Would be nice to get this through in the near future, since it will be a blocker for meaningful work in this repo.

Changes

Chain types

Include chains types from @polkadot/apps-config and use the in ApiPromise

Controller config

Controller configuration object that allows to specify the controllers to use (see ControllerConfig in src/types/chains-config)
Mandala controller config (read down for more about controller configs)
controller config for Kulupu to close /blocks/head return error #350
Create default controller config (primarily for Polkadot and Kusama)
Dynamically instantiate controllers base on specified controller config (see getControllersForSpec and getControllersFromConfig from src/chains-config/index.ts)
src/controllers/index.ts now exports a concrete controllers object so we can have better type safety when dynamically creating the controllers with getControllersFromConfig in src/chains-config/index.ts
finalizes filed that is used to decide if the getFinalizedHead should be used in the blocks controller

substrate-api-sidecar/src/controllers/blocks/BlocksController.ts

Line 94 in 0ac5573

finalized === 'false' || !this.finalizes

Endpoints (services/ controllers)

Creation of KulupuBlocksController which has new /kulupuBlocks/{head, :number} endpoint. I had to do this because there is no notion of finalized blocks since it is a simple PoW consensus (someone please correct if I am wrong). In the new controller only a couple lines here taken out to remove the ?finalized=boolean query param, but I created an entire other controller with a new endpoint name to keep things simple; However I am certainly open to other ideas, I just don't want the controllers to scale with a huge amount of conditionals as we add more chains.
accounts/{accountId}/balance-info
- now supports a token query param that allows the user to specify tokens from ORML tokens pallet
- tokenSymbol field in the response to reduce ambiguity on the token of the balances represented
- Openapi docs updated to reflect changes
BlocksService is updated so it does not break when it encounters chains that do not have all the components to support CalcFee.from_params

Other changes

Bump all deps to latest
Factor out start up prompts from main to declutter.
renamed Config => SidecarConfig to reduce ambiguity against the new type ControllerConfig
Docs for chain builders and end users on how to incorporate their chain into Sidecar (closes Create docs walking through how a chain builder can integrate with Sidecar #352)

How to try out the PR

Follow the instructions in the README: https://github.com/paritytech/substrate-api-sidecar#source-code-installation-and-usage and then start the service against Kulupu or Mandala by running SAS_SUBSTRATE_WS_URL=wss://rpc.kulupu.corepaper.org/ws yarn start. You can then discover the available endpoints by visiting GET http://127.0.0.1:8080/

Follow up work

Test to see if the user configurable CUSTOM_TYPES still works and see if there is a better solution
More testing against Kulupu and Mandala (e.g. submitting transactions etc.)
Token identifier verification middleware (Token identifier verification #360)

Clean up

dvdplm

I'll have to tinker with this a bit before I leave a proper review.

src/chains-config/defaultControllers.ts

src/chains-config/index.ts

src/main.ts

src/services/accounts/AccountsBalanceInfoService.ts

src/services/blocks/BlocksService.ts

Co-authored-by: David <dvdplm@gmail.com>

…te-api-sidecar into types-package

emostov · 2020-12-03T03:11:32Z

Added a CHAIN_INTEGRATION.md guide that @joepetrowski may be interested in reviewing

CHAIN_INTEGRATION.md

joepetrowski · 2020-12-04T11:03:44Z

CHAIN_INTEGRATION.md

+
+##### Basic balance transfer support
+
+In order to support traditional balance transfers the chain's Sidecar endpoints should support account balance lookup, transaction submission, transaction material retrieval, and block queries.


Nice. Even if we haven't decided the exact token standard, good to start somewhere.

Could be good to link to examples of how each of these look like.

@dvdplm - what exactly do you have in mind for examples? My idea here was essentially to say "make sure to add the relevant controllers for these endpoints".

Yeah maybe it's dumb but my thinking was to link to example implementations of the existing Polkadot/Kusama endpoints; a dev adding sidecar for their chain might want to peek at how you did those endpoints as theirs will likely be fairly similar?

CHAIN_INTEGRATION.md

Co-authored-by: joe petrowski <25483142+joepetrowski@users.noreply.github.com>

CHAIN_INTEGRATION.md

dvdplm · 2020-12-08T14:45:10Z

CHAIN_INTEGRATION.md

+
+##### Basic balance transfer support
+
+In order to support traditional balance transfers the chain's Sidecar endpoints should support account balance lookup, transaction submission, transaction material retrieval, and block queries.


Could be good to link to examples of how each of these look like.

CHAIN_INTEGRATION.md

dvdplm · 2020-12-08T14:50:52Z

src/chains-config/index.ts

-			acc.push(
-				new controllers[controllerName](api, config.options.finalizes)
-			);
+			acc.push(new controllers[controllerName](api, config.options));


maciejhirsz

Minor grumbles and comments

src/main.ts

src/services/blocks/BlocksService.ts

maciejhirsz · 2020-12-08T15:02:11Z

src/chains-config/index.ts

+		boolean
+	][];
+
+	return controllersToInclude.reduce((acc, [controllerName, shouldMount]) => {


One day JavaScript will get filterMap on arrays, one day... :)

src/services/blocks/BlocksService.ts

Co-authored-by: David <dvdplm@gmail.com>

…te-api-sidecar into types-package

emostov · 2020-12-09T02:50:33Z

Responded to all comments. I think it is is looking pretty good.

maciejhirsz

LGTM

dvdplm

One tiny nit left!

CHAIN_INTEGRATION.md

dvdplm · 2020-12-09T19:45:59Z

src/services/accounts/AccountsBalanceInfoService.ts

-Skip to content
-Search or jump to…
-
-Pull requests
-Issues
-Marketplace
-Explore
-
-@emostov
-paritytech
-	/
-	substrate - api - sidecar
-12
-56
-29
-Code
-Issues
-20
-Pull requests
-1
-Actions
-Security
-4
-Insights
-Settings
-substrate - api - sidecar / src / services / accounts / AccountsBalanceInfoService.ts /
-@emostov
-emostov Initial reply david review
-Latest commit be66056 6 days ago
-History
-2 contributors
-@emostov @joepetrowski
-We found potential security vulnerabilities in your dependencies.
-You can see this message because you have been granted access to Dependabot alerts for this repository.
-
-95 lines(87 sloc)  2.38 KB


Interesting glitch in the matrix here. :D

Co-authored-by: David <dvdplm@gmail.com>

feat: chainSpec based controller config; Types from apps-config

ae1de13

Clean up

emostov requested review from dvdplm and maciejhirsz November 24, 2020 00:19

Update mock api to include derive getBlock - specs work

5d254dd

emostov requested a review from sorpaas November 24, 2020 00:38

emostov mentioned this pull request Nov 24, 2020

Create docs walking through how a chain builder can integrate with Sidecar #352

Closed

emostov requested a review from joepetrowski November 24, 2020 00:49

emostov added 2 commits November 23, 2020 18:31

Update to reflect TS 4.1

7072fdc

Clean up comments

c682665

emostov mentioned this pull request Dec 1, 2020

config: Enhance CUSTOM_TYPES to take Object too. #357

Merged

emostov added 4 commits December 1, 2020 16:22

Save

4ab1b2d

Merge origin master

cc9a962

feat: Token query param for non-native token balance-info

ede1787

Revert package.json

39e1a1a

emostov requested review from ewang8 and removed request for ewang8 December 2, 2020 04:56

dvdplm reviewed Dec 2, 2020

View reviewed changes

emostov and others added 5 commits December 2, 2020 13:51

Initial reply david review

be66056

Update src/main.ts

e465092

Co-authored-by: David <dvdplm@gmail.com>

Remove kulupu controller

19a66e7

Merge branch 'types-package' of https://github.com/paritytech/substra…

0ac5573

…te-api-sidecar into types-package

Add chain builder integration guide

c5e9284

Patch CHAIN_INTEGRATION.md

0c57ecd

joepetrowski reviewed Dec 4, 2020

View reviewed changes

emostov and others added 2 commits December 4, 2020 10:02

Apply suggestions from code review

6692e50

Co-authored-by: joe petrowski <25483142+joepetrowski@users.noreply.github.com>

Update CHAIN_INTEGRATION.md

4b4d9a2

emostov mentioned this pull request Dec 8, 2020

Token identifier verification #360

Open

emostov added 2 commits December 7, 2020 18:58

Bump deps

b81c0d1

Merge remote-tracking branch 'origin' into types-package

46f067d

Use whole options object for controller creation

92a8856

dvdplm reviewed Dec 8, 2020

View reviewed changes

maciejhirsz suggested changes Dec 8, 2020

View reviewed changes

emostov and others added 7 commits December 8, 2020 18:21

Apply suggestions from code review

8451309

Co-authored-by: David <dvdplm@gmail.com>

Respond to maciej feedback

87572a9

Merge branch 'types-package' of https://github.com/paritytech/substra…

c38c105

…te-api-sidecar into types-package

Update chain integration guide

a1a988a

Remove excessive nullish colescing operators

79f9eb8

Merge origin master

cf84d63

Fix merge issue

b34cca2

maciejhirsz approved these changes Dec 9, 2020

View reviewed changes

dvdplm approved these changes Dec 9, 2020

View reviewed changes

Update CHAIN_INTEGRATION.md

a16235a

Co-authored-by: David <dvdplm@gmail.com>

emostov merged commit 5936a1c into master Dec 9, 2020

emostov deleted the types-package branch December 9, 2020 20:29

emostov mentioned this pull request Jan 15, 2021

fix: Use correct registry when creating calls; Remove usage of derive.chain.getBlock #391

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: chainSpec based controller config; Types from apps-config #351

feat: chainSpec based controller config; Types from apps-config #351

emostov commented Nov 24, 2020 •

edited

Loading

dvdplm left a comment

emostov commented Dec 3, 2020

joepetrowski Dec 4, 2020

dvdplm Dec 8, 2020

emostov Dec 9, 2020

dvdplm Dec 9, 2020

dvdplm Dec 8, 2020

dvdplm Dec 8, 2020

maciejhirsz left a comment

maciejhirsz Dec 8, 2020

emostov commented Dec 9, 2020

maciejhirsz left a comment

dvdplm left a comment

dvdplm Dec 9, 2020


		##### Basic balance transfer support

		In order to support traditional balance transfers the chain's Sidecar endpoints should support account balance lookup, transaction submission, transaction material retrieval, and block queries.

feat: chainSpec based controller config; Types from apps-config #351

feat: chainSpec based controller config; Types from apps-config #351

Conversation

emostov commented Nov 24, 2020 • edited Loading

Overview

Changes

Chain types

Controller config

Endpoints (services/ controllers)

Other changes

How to try out the PR

Follow up work

dvdplm left a comment

Choose a reason for hiding this comment

emostov commented Dec 3, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

maciejhirsz left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

emostov commented Dec 9, 2020

maciejhirsz left a comment

Choose a reason for hiding this comment

dvdplm left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

emostov commented Nov 24, 2020 •

edited

Loading