Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Setup benchmarking and add simple dataset benchmark #1202

Merged
merged 20 commits into from
Jul 25, 2018
Merged

Setup benchmarking and add simple dataset benchmark #1202

merged 20 commits into from
Jul 25, 2018

Conversation

astafan8
Copy link
Contributor

@astafan8 astafan8 commented Jul 24, 2018 •
edited
Loading

This PR enables benchmarking of QCoDeS features using AirSpeed Velocity benchmarking package (asv). In principle, this PR replaces #1050 and #959.

Changes:

  • A new directory in the repository is created for benchmark code
  • README is added to explain how to run benchmarks
  • travis CI now runs the benchmarks in a special "development" mode just to make sure they do not throw exceptions (results are also displayed, but in the command line)
  • extra rules for .gitignore
  • benchmark for the data saving speed is added (the code is similar to the one used for Datasaver save speed #1187 PR)

Accumulating benchmarking results, and making them available online is left out of scope of this PR. Let's discuss it separately.

Let me know if something is missing.

astafan8 added 20 commits July 23, 2018 10:50
@astafan8
Initialize benchmarking infrastructure based on asv package
297f413 
- add benchmarking directory
- add asv.conf.json configuration for asv package to run correctly
- add a basic README about benchmarking QCoDeS
- exclude asv.conf.json in gitignore
- exclude in gitignore directories generated by asv during benchmarking
@astafan8
Add dirty benchmarking code for adding data to dataset
54bf4b9
@astafan8
Reformat Travis test commands
9ccbf5a
@astafan8
Add benchmarking to Travis
6decfda
@astafan8
Add a comment (trigger travis)
7c1640f
@astafan8
do not limit building branches to master
2b69497
@astafan8
build all branches
c65084c
@astafan8
enable only master, install pip asv, reformat a big again
ed909a1
@astafan8
try with machine equals travis
5024077
@astafan8
Add more info on how to run benchmarking (for commit, etc.)
3063357
@astafan8
Improve data saving speed benchmark
105521b 
- make sure data is flushed during benchmarking
- parameterize it
- add comments
@astafan8
let travis run 'asv dev' to check correctness of benchmarks
1085a5f
@astafan8
Add asv to test requirements.txt
fd73e65
@astafan8
Merge branch 'master' into setup-benchmarking-and-add-simple-dataset-…
7ba7714 
…benchmark
@astafan8
Fix indentation
f83ae91
@astafan8
Remove asv dummy benchmarks
5b65bd0
@astafan8
Add docstring for benchmarks module
7fa7185
@astafan8
Fix some formatting in readme for benchmarking
899a90d
@astafan8
Use --machine travis when running asv
70dcb2f
@astafan8
Fix asv preview description
f97c5c0
@astafan8 astafan8 self-assigned this Jul 24, 2018
This was referenced Jul 24, 2018
Closed
Closed
jenshnielsen
jenshnielsen approved these changes Jul 25, 2018
View reviewed changes
Copy link
Collaborator

@jenshnielsen jenshnielsen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks like a good start. We can always extend it as we go

@jenshnielsen jenshnielsen merged commit f7b173f into microsoft:master Jul 25, 2018
@astafan8 astafan8 deleted the setup-benchmarking-and-add-simple-dataset-benchmark branch July 25, 2018 08:18
giulioungaretti pushed a commit that referenced this pull request Jul 25, 2018
Generated gh-pages for commit f7b173f
b917bae 
Merge: ded5447 f97c5c0
Author: Jens Hedegaard Nielsen <jenshnielsen@gmail.com>

    Merge pull request #1202 from astafan8/setup-benchmarking-and-add-simple-dataset-benchmark
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jenshnielsen jenshnielsen jenshnielsen approved these changes

@sohailc sohailc Awaiting requested review from sohailc

@WilliamHPNielsen WilliamHPNielsen Awaiting requested review from WilliamHPNielsen

@Dominik-Vogel Dominik-Vogel Awaiting requested review from Dominik-Vogel

Assignees

@astafan8 astafan8

Labels
enhancement
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@astafan8 @jenshnielsen