Proposal to add E2E and Integration testing on 1.x branch

[cgewecke]

(This proposal is continuation of #2682, a companion to #3070, and a tracking issue for all integration testing related topics for the 1.x branch)

E2E

Web3 1.x's stability might benefit from following ethereum/solidity's example and E2E testing its latest state against large projects that already use the package. Solidity does this nightly in CI - they build a new solc, swap it into Zeppelin, Gnosis and Colony and run those projects' own tests using the new version.

Some targets that seem promising for this purpose here are:

• @truffle/contract
• gnosis/safe-react

Technical

Web3 is a Lerna monorepo and so are two of the potential targets listed above. Because its repo structure and published formats are quite different it won't just npm install from a github url.

However, there's a nice tool called verdaccio that lets you set up a local private npm registry in CI. Web3 could "publish" a candidate branch to this ephemeral registry, clone an E2E target (ex: embark) and install the new web3 via npm proxy. Then run the target's tests.

It's also possible to chain publications together. For example, openzeppelin-solidity uses truffle so you could:

• publish a new Web3,
• install it in Truffle,
• publish a new Truffle,
• swap the new Truffle into Zeppelin and run Zeppelin's tests.

Additionally:

• could be used to verify that built assets are located correctly and usable.
• ethereum-js may migrate some things to monorepo in future. Any solution developed here might pay dividends there as well.

Integration

Clients

To recap #2682 - Web3 1.x's unit tests rely primarily on mocked server responses. They're super fast but some things are hard to validate without testing against a node. There's also variance in behavior between the three most widely used clients. It would be good to create a small suite of tests for stuff like:

• simple send and call
• events
• subscriptions
• errors
• transaction signing

..and make sure everything executes as expected across this matrix:

Client	instamine	automine 2s
geth --dev	x	x
parity --dev	x	x
ganache-cli	x

Client Helpers

To simplify things in CI and make this work portable, Web3 could also publish small Node cli wrappers for the dockerized Parity and Geth dev clients.

These would:

• pull down any client version from docker hub
• abstract away most of command config flags,
• open all APIs,
• have many well-funded accounts
• make it easy for people to generate reproduction cases for client specific problems.

Something that launches like this:

$ web3-geth-dev --period 2 --tag latest

See also: #2688 and #2693: I think access to these clients can be provided in a flexible way with JS wrappers alone - e.g without building any containers or using docker compose.

• nomiclabs/builder-docker provides a nice API for the general case of managing docker published resources from JS.
• A basic geth dev wrapper I've used in CI for other projects. Not perfect at all but works.

• Parity dev Node cli wrapper
• Geth dev Node cli wrapper

Network Emulators

• Dropped connections are frequently mentioned in the issues. Emulate with comcast?

Installation

• Windows: see Can't install web3 version 1.2 on Windows 10 #2971, npm install succeeds on latest version of Node on Windows but does show errors #3033
• Mac OSX

The EF has an account with CircleCI. (EthereumJS uses it). They let you specify which OS a job is run with pretty easily.

cf: #3034 (Azure Pipelines)

Other stuff 🐘

There is definitely other stuff to test. Would be grateful for any and all community suggestions about this topic.

• generic create-react-app
• headless browser testing for chromium, firefox, safari (PR Add karma & chrome/firefox headless browser fixtures #3146)

[holgerd77]

Can't completely judge any measure from this proposal but I think that such strong and practical integrational testing might really be a game changer for the future stability of the library, my general feeling for the situation of the web3 library is still limited though.

@cgewecke do you have got a rough estimate about the scope of work a stripped-down barebone version of this would eventually take?

[cgewecke]

@holgerd77 I'm not sure. Days? It's kind of hard to tell. Some things like test suites might take a bit longer to develop a good set of cases.

[holgerd77]

@cgewecke Ok, thanks!

[nivida]

@cgewecke I think we should now first focus on adding all e2e test cases. We can after extend the e2e tests with the virtual registry and the headless browser tests. Do you agree with that?

[cgewecke]

@nivida Yes, that makes sense to me! Will work in that direction.

[cgewecke]

This is mostly done..