Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add CI check that tool is deterministic #171

Open
siddharth-krishna opened this issue Feb 1, 2024 · 0 comments
Open

Add CI check that tool is deterministic #171

siddharth-krishna opened this issue Feb 1, 2024 · 0 comments

Comments

@siddharth-krishna
Copy link
Collaborator

siddharth-krishna commented Feb 1, 2024

Ah, this is because iterating through a set results in a non-deterministic order (see also #50 and #67). This makes it very difficult to debug regressions, because I rely on the --verbose flag and a diff tool to find out which transformation caused the regression. But with nondeterministic behaviour, there are too many changes in the diff..

I'm considering adding a check to CI that runs the tool on a small benchmark ~5-10 times and ensures that all intermediate tables are identical across runs.

Originally posted by @siddharth-krishna in #166 (comment)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant