Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: check existing great expectation suite before adding a new one #1458

Merged
merged 7 commits into from
Aug 1, 2024

Conversation

linglp
Copy link
Contributor

@linglp linglp commented Jul 11, 2024

Related to: https://sagebionetworks.jira.com/browse/FDS-2155

Problem

Sometimes our GH workflows failed with great expectation suite error:

(1424 durations < 0.005s hidden.  Use -vv to show these durations.)
=========================== short test summary info ============================
FAILED tests/test_validation.py::TestManifestValidation::test_rule_combinations[int-IsNA] - great_expectations.exceptions.exceptions.ExpectationSuiteError: An ExpectationSuite named Manifest_test_suite already exists.
====== 1 failed, 715 passed, 691 warnings, 4 rerun in 1449.84s (0:24:09) =======

I think this problem occurred because the code adds an expectation suite without checking if it exists or not. See the original code below:

self.suite = self.context.add_expectation_suite(
            expectation_suite_name=self.expectation_suite_name,
        )

Link of failed GH action: https://github.com/Sage-Bionetworks/schematic/actions/runs/9893404399/attempts/1

Solution

Check if an expectation suite already exists before adding a new expectation suite. Based on the documentation here, you could use function list_expectation_suite_name to check if an expectation suite has been created. If it does, then we could retrieve the current suite (see documentation here)

@linglp linglp marked this pull request as ready for review July 22, 2024 13:12
@linglp
Copy link
Contributor Author

linglp commented Jul 22, 2024

This is the same issue as github action workflow run failure here: https://github.com/Sage-Bionetworks/schematic/actions/runs/10014376516/job/27683959671

@linglp linglp requested a review from GiaJordan July 22, 2024 15:10
Copy link
Contributor

@GiaJordan GiaJordan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just one small comment to address before merge but otherwise looks good!
As an aside at some point we should probably switch to using the GX abbreviation to match how GreatExpectations refers to themselves but that's not pressing at all

Copy link

@linglp linglp merged commit 6db0860 into develop Aug 1, 2024
6 checks passed
@linglp linglp deleted the develop-fix-ge branch August 1, 2024 18:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants