Datasaver save speed

[astafan8]

Recently, two users have reported that it takes too much time to save data to the database during the experiment. In particular, saving 10,000 points for 5 parameters (2 dependent, and 3 independent) is reported to take 5-10s.

After investigating the DataSaver add_results method using cProfile profiler, I could improve the saving speed by a factor of ~2. Hence, this pull-request.

The fix is in the insert_many_values method. The data is split into chunks in order to pass as many values as possible within a single INSERT call. The problem was that each of these INSERT calls had a commit which was decreasing the performance. As it is suggested everywhere on the web, in this situation, all the INSERT calls should happen within one transaction, meaning there should be a single COMMIT for all of them. This way the performance is improved.

Changes in this PR:

• one COMMIT per all INSERTs
• remove unnecessary commit after the insert_many_values call, because commit is done within that function
• use itertools instead of list comprehension for flattening a list of lists (advised on StackOverflow)
• add convenient profile decorator function

Note that the users would like to have this amount of data saved within ~0.1s which raises a question whether sqlite is the right backend. But let's keep this discussion aside of this pull-request.

[codecov]

Codecov Report

Merging #1187 into master will decrease coverage by 0.08%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master    #1187      +/-   ##
==========================================
- Coverage   79.76%   79.67%   -0.09%     
==========================================
  Files          48       47       -1     
  Lines        6662     6668       +6     
==========================================
- Hits         5314     5313       -1     
- Misses       1348     1355       +7

[Dominik-Vogel]

I additionally tested:

• increasing cache -> no improvement
• using isolation_level='EXCLUSIVE'-> no improvement
• using a pure in memory database: -> no significant improvement (from ~3 down to 2.19 s ± 224 ms)
  So I suggest to test storing the data as binary blobs until we come up with a better solution.

[astafan8]

@Dominik-Vogel Shall we at least merge the 2x fix from this PR?

[jenshnielsen]

Only one comment

[jenshnielsen]

So this is technically an api change for anyone that uses add_results directly. Do we need to worry about that? Should we rather add a new function that does not commit and make this an atomic around that?

[astafan8]

insert_many_values already commits, so this line is just redundant. There is not API change at all, neither for add_results, nor for insert_many_values

[astafan8]

as discussed with @WilliamHPNielsen , I will remove the benchmark code from this PR, and make a new PR with the benchmark code that uses asv (like it is used in numpy).