Benchmarks

[dignifiedquire]

In order to know how fast things are, and in which area we can improve we want to add benchmarks to js-ipfs and its modules.

For the first round I suggest adding the following benchmarks

js-ipfs

• {offline|online} file add {file|folder}
  • files: count {1|5|10} - size {1KB|10KB|100KB|1MB}
  • folders: count {1|5|10}
• {offline|online} file get {file|folder} 
  • files: count {1|5|10} - size {1KB|10KB|100KB|1MB}
  • folders: count {1|5|10}
• {offline|online} object put
  • count {1|5|10} - size {1KB|10KB|100KB|1MB}
• {offline|online} object get
  • count {1|5|10} - size {1KB|10KB|100KB|1MB}
• block distribution between {2|5|10} nodes (these should be different processes)
  • block count {10|100} - size {1KB|10KB}

js-libp2p-secio

• throughput on a connection over {pull-pair|tcp|websockets}
  • messsage: size {100Bytes|10KB|100KB} - count {100|1000}

[daviddias]

Let's make sure these benchmarks present valid measures by adding the quantity of operations and size of files they will have to perform.

[Kubuxu]

Take a look at https://github.com/whyrusleeping/ipfs-whatever which is very simple and naive benchmarks we have used for go-ipfs over HTTP API.

It would be best if we could reuse some/most benchmarks for both js-ipfs and go-ipfs not to double up the efforts.

[dignifiedquire]

Let's make sure these benchmarks present valid measures by adding the quantity of operations and size of files they will have to perform.

Added some numbers, feel free to suggest better ones, these are wild guesses currently.

[dignifiedquire]

It would be best if we could reuse some/most benchmarks for both js-ipfs and go-ipfs not to double up the efforts.

I would love if I can just run this against js-ipfs. That would be a great start and save us some time.

[dignifiedquire]

I just tested to run ipfs-whatever against js-ipfs. If we have add and cat at the same place as go we can actually use it! Which I very much prefer over having a separate test suite for the first round.

[dignifiedquire]

First results for things that worked out of the box. Before is js-ipfs (pull branch), After is go-ipfs (master)

Results          Before  After   % Change
PatchOpsPerSec   208.54  681.61  226.85%
DirAddOpsPerSec  133.70  758.74  467.51%
DirAddOpsStdev   7.64    81.62   968.80%

[daviddias]

It would also be cool to have the benchmarks over ipfs-api, both against go-ipfs or js-ipfs :D

[daviddias]

bump the order of magnitude by one or two on the benchmark tests

[dignifiedquire]

bump the order of magnitude by one or two on the benchmark tests

Not sure that would be good, those will already be running very long.

[daviddias]

Adding 10 files of 1MB is taking a lot of time?

[dignifiedquire]

Adding 10 files of 1MB is taking a lot of time?

No, but if you calculate the above matrix and add the fact that each thing should be run at least 10 times to get some statistical relevance this will explode very quickly.

[daviddias]

Still, we should bump it (or have a way to bump it, like interface-stream-muxer and mega test)

[dignifiedquire]

Just had a chat with @diasdavid, for the first round we will

• await the PR from @victorbjelkholm to fix and add add and cat on the http api
• expose add and cat at the top level as aliases
• add a benchmark.sh script which go gets https://github.com/whyrusleeping/ipfs-whatever and runs the tests against a js-ipfs daemon

[victorb]

@dignifiedquire

await the PR from @victorbjelkholm to fix and add add and cat on the http api

This has been merged.

expose add and cat at the top level as aliases

This is in a PR now: #493

[daviddias]

💯 yes, let's get the ipfs-whatever benchmarks in :)

[dignifiedquire]

@diasdavid PR already exists here : #488

[daviddias]

closing this one in favor of the PR :)