Investigate overconsumption of CPU with latest master

[tiero]

You can see the change in the CPU consumption just switching from commit b3965b316a5ed37fe5ff53781dc9db6a254e34b3 to 0ab84f87237a1023258401d8e0d168f4ef59bcc9

I tried to "drop" only that commit (ie. git revert --strategy resolve 0ab84f87237a1023258401d8e0d168f4ef59bcc9), but every subsequent commit is too tangled to that, so would be better to inverstigate how to make it less expensive.

golang/go#27707
https://stackoverflow.com/questions/55367231/golang-for-select-loop-consumes-100-of-cpu

I suspect time.Sleep introdcued in the for loop is not the best strategy here

[tiero]

instead of "sleeping" before inserting an observable, would be better to attach specific options to each observable that will dictate the behavior ie. when to trigger. In #276 we introduce this topic better and I think that would be the definitive solution.

[tiero]

Latest master

Daemon "empty" just started

Daemon after funding and opening a market

[tiero]

Commit hash b3965b316a5ed37fe5ff53781dc9db6a254e34b3

empty

funded market

[tiero]

Don't ask me why, following this comment

Adding this microsecond of sleep in listenToEventChannel in blockchain_listener visibly lowered the "noise" my computer was doing. I don't have much experience on how to "demonstrate" this with pprof (cc/ @sekulicd) but pretty sure this is the problem in there.

[tiero]

On a amazon ec2 the v0.2.0-rc.2 was eating 100% of container CPU all the time

Now with the latest v0.2.0-rc.3 we have a variable consumption that goes from 0.74% to max 34%. I suspect that those spikes are the ticker calling the explorer, so this makes sense.

[sekulicd]

@tiero u can check CPU profile by doing as follows:

1. go tool pprof http://localhost:8024/debug/pprof/profile\?seconds\=30 -> this will profile CPU for 30 sec, do this after some time when u hear the noise.
2. after profiling finishes type 'web' and u should see image with most intensive method calls

Another way:

1. go tool pprof http://localhost:8024/debug/pprof/profile\?seconds\=30
2. in ~/pprof it should be generated profile file
3. Then u run go tool pprof -http :8080 {cpu file} -> in my case name was pprof.samples.cpu.001.pb.gz

Please find attachment of CPU profile that i did if u dont want to bother.

In picture u can see that unblindingUtxo method is 'killing' us, more precisely confidential.UnblindOutputWithKey.
Before when we had one ticker, which by default was set to 5000(5sec) we would crawl all addresses async and do unblinding and then CPU would relax, now we have ticker for each address and we constantly fetch and unblind which makes CPU work all the time. RateLImiter helps in a way that is limits to 10 calls in second(as per current configuration).

[tiero]

Closing, altough we may need to lower the depenedency on many cgo calls @altafan