Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support different metrics #102

Open
farquet opened this issue Sep 2, 2019 · 2 comments
Open

Support different metrics #102

farquet opened this issue Sep 2, 2019 · 2 comments

Comments

@farquet
Copy link

farquet commented Sep 2, 2019

While the tool doesn't really care about what the data is about, it actually cares a lot about improvement/regression, but for this, it assumes that a lower score is better.
This assumption is wrong if we feed data like throughput; so it would be a great addition if the tool wouldn't report improvements as regressions and vice versa.
Maybe a command line flag setting a boolean higher-is-better would be a good starting point.

@vext01
Copy link
Member

vext01 commented Sep 2, 2019

This is a good point.

Until now, we've only used this tool with wall-clock times where lower is better.

@ltratt
Copy link
Member

ltratt commented Sep 2, 2019

I think that we have quite a few places where the assumption is that lower is better. A horrible temporary hack is for users to invert their y-axis scales when inputting data. However, we have one additional feature which would defeat even that: because we currently assume that times are in seconds, we have the "times less than 0.001s are irrelevant" tweak, which is clearly going to do unexpected things if the units aren't in seconds.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants