-
Notifications
You must be signed in to change notification settings - Fork 4.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[iOS][libraries] Skip failing iOS tests with ActiveIssues and proj level skips #51491
Conversation
I couldn't figure out the best area label to add to this PR. If you have write-permissions please help me learn by adding exactly one area label. |
521b003
to
bfa8f8a
Compare
Which are they? |
@marek-safar So far the PNSE failure test suites being completely skipped are There are some test facts that fail with PNSE that are skipped individually, I'll need to ensure that they're being tracked/marked with some note of PNSE. |
Out of these 4 I think only |
c144b51
to
05a9946
Compare
On my local machine |
@marek-safar Of all the previous CI runs of iOS and tvOS, the following suites have the following number of methods that fail with PNSE. Should
|
Preferably not |
2387efb
to
0da7e61
Compare
…m.Net.NameResolution.Unit test suite skips
This reverts commit 163ff12da16e602e52830c79f267e14c420284c0.
…ng on CI but passes locally
…on CI but passes locally
…Algorithms test exclusions
@@ -80,6 +80,7 @@ public void KeySizeProp() | |||
} | |||
|
|||
[Theory, MemberData(nameof(TestNewCurves))] | |||
[ActiveIssue("https://github.com/dotnet/runtime/issues/51332", TestPlatforms.iOS | TestPlatforms.tvOS | TestPlatforms.MacCatalyst)] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yup sounds good, but we'll still go ahead and merge this PR for now to get to a green state in runtime-staging :)
In effort to assess the state of the iOS library tests, a number of tests will be skipped
Test suites that crash, hang, or have a significant number of failures (mostly 20+ failing unique test methods) are skipped on at the
tests.proj
level. Otherwise,[ActiveIssue]
attributes have been attached to either the individual test methods that fail or the entire class if the whole test class fails. Any pre-existingActiveIssue
attribute associated with iOS had been reassessed to either remain or be closed.A few test suites have been skipped because they fail with some form of
PlatformNotSupportedException
.[SkipOnPlatform]
attributes are attached to the failing test methods.All of the Issues can be tracked at https://github.com/dotnet/runtime/projects/48#column-9236436
The ActiveIssues and
tests.proj
exclusions have been tested on #49917 as well.This PR makes the following changes in the following files:
src/libraries/System.IO.Ports/tests/System.IO.Ports.Tests.csproj
- Removed$(NetCoreAppCurrent)-iOS
from target frameworks as it is stated to currently only supported on Windows.src/libraries/tests.proj
- Update the test suites that need to be skipped at the project level, reducing the number and categorizing as a skip due to PNSE, a skip due to numerous failures, and a skip due to the suite crashing.The remaining files changed are either:
[ActiveIssue("<issue-url>", TestPlatforms.iOS | TestPlatforms.tvOS | TestPlatforms.MacCatalyst)]
[SkipOnPlatform(TestPlatforms.iOS | TestPlatforms.tvOS, "Not supported on iOS or tvOS.")]