E2E testing for Fabric

[chiaramooney]

E2E Testing for Fabric Architecture

(last updated December 2023)
This is the meta issue capturing the effort of adding E2E testing for the new Fabric architecture.

Current Testing Status

Number of Automated Tests for Fabric: 280

Current Test Coverage Progress

Here are links to the issues we have created to track our progress of adding tests. To see the components, props, and app scenarios that currently have automated tests, click on the links below. Visual Tests describe the set of tests which capture how components are being rendered on the screen. Functional Tests describe the set of tests which capture component/app behavior such as press events, focus changes, etc. Accessibility Tests describe the set of tests which capture accessibility behavior such as narrator and keyboard navigation as well as validate a component's UIA tree.
Visuals Test Coverage: #11343
Functional Test Coverage: #12031
Accessibility Test Coverage: need to add link

Test Progress by Component:

Priority	Component	Tests Available	Test Coverage (includes Visual, Functional, and Accessibility)	Links
0	Button	✅	60% (12/20)	Functional Test Coverage: #12459 Visual Test Coverage: #11198
0	FlatList	🟥	0% (0/85)	Functional Test Coverage: #12473 Visual Test Coverage: #11277
0	Pressable	✅	51% (48/94)	Functional Test Coverage: #12461 Visual Test Coverage: #11295
0	View	✅	51% (37/73)	Functional Test Coverage: #12468 Visual Test Coverage: #11313
0	Text	✅	42% (42/100)	Functional Test Coverage: #12464 Visual Test Coverage: #11299
0	TouchableHighlight	✅	12% (10/86)	Functional Test Coverage: #12465 Visual Test Coverage: #11310
0	TouchableOpacity	✅	15% (12/80)	Functional Test Coverage: #12466 Visual Test Coverage: #11311
0	TouchableWithoutFeedback	✅	8% (8/80)	Functional Test Coverage: #12467 Visual Test Coverage: #11312
0	Image	✅	42% (33/79)	Visual Test Coverage: #11281
0	TextInput	✅	40% (41/97)	Functional Test Coverage: #12458 Visual Test Coverage: #11309
1	ScrollView	✅	8% (10/126)	Functional Test Coverage: #12462 Visual Test Coverage: #11297
1	Modal	🟥	0% (0/0)
1	Switch	✅	31% (14/45)	Functional Test Coverage: #12463, Visual Test Coverage: #11298
2	ActivityIndicator	✅	19% (13/70)	Visual Test Coverage #12914
2	RefreshControl	🟥	0% (0/0)

Supported Types of Testing

Here is the list of types of testing that we plan to support in the E2E test app on Fabric. Test types marked with the green checkmark (✅) currently have infrastructure support and samples available in main.

For more information on how to add tests to E2E test app for Fabric, take a look at our documentation (insert documentation once created)

Priority	Test Type	Infrastructure Available
0	Dump Composition tree for a control. (Current properties tracked include size, offset, opacity, testid, and brush color)	✅
0	Dump UIA (Accessibility) tree for a control. (Current properties tracked include AutomationId, ControlType, HelpText, IsEnabled, IsKeyboardFocusable, LocalizedControlType, and Name)	✅
0	Collect Jest Snapshot tests for a component or page.	✅
0	Functional tests which use WebDriver IO's setValue/getValue().	✅
0	Functional tests which use WebDriver IO's click().	✅
1	Functional tests which run fast refresh scenarios.	🟥
1	Functional tests which run onHover scenarios.	🟥
1	Functional tests which test 'pressed' state for controls.	🟥
1	Screenshot comparison tests.	🟥

Testing Milestones

Below are descriptions for the testing milestones that we have identified for the new architecture. We are currently working on the TM3 milestone.

TM0 - "Base Setup" ✅

• E2E test app running on Fabric exists in the repo with RNTester as its JS contents.
• App should have Jest configuration and should run a control test when yarn e2etest is run.
• Jest Snapshot tests should be added and run for all RNTester pages (*can only run pages which currently successfully build on Fabric).

TM1 - "Bast Testing Infrastructure" ✅

• E2E test app should be able use types/functionality of the @react-native-windows/automationclient package.
• E2E test app should be able to find elements by their test id and run commands such as click, setValue, etc. Commands list should come from WebDriverIO commands supported in the @react-native-windows/automationclient package.
• Test sample created for navigating to a particular component page.
• Test sample created for visiting all RNTester pages.
• E2E test script should run in the React Native Windows CI and PR pipelines.

TM2 - "Test Infrastructure Parity" ✅

• E2E test app testing infrastructure should be able to dump the composition tree for a component. Test sample created for composition tree dumps.
• E2E test app testing infrastructure should be able to dump the accessibility tree for a component. Test sample created for composition tree dumps.

TM3 - "Bonus Infrastructure" (Current Milestone In Progress)

• Test samples created for functional testing of components. Test should cover general cases of programmatic text/coloring updates, fast refresh text/coloring updates, and updates from component state changes (i.e. hover, pressed, disabled).
• E2E test app infrastructure should be able to support accessibility behavior testing. Test sample created for key accessibility scenarios.

TM4 - "Testing Parity"

• Testing parity with Paper.
• At this milestone, the Fabric architecture will have enough validation that Fabric could be deemed "ready for production".

TM5 - "100% Coverage"

• Functional testing for all behavior scenarios.
• Visual component testing for all component scenarios.
• Accessibility testing for all accessibility scenarios.

Testing Tasklist

Milstone	Category	Description	Status
TM0	E2E Testing	Determine project structure for E2E testing for Fabric (i.e. separate app or reuse existing, directory structure).	Completed
TM0	E2E Testing	Create RNW app using Fabric architecture to be used as E2E test app for Fabric.	#11407
TM0	E2E Testing	Fill contents of E2E test app with RNTester.	#11407
TM0	E2E Testing	Add Jest configuration to E2E test app. Ensure control test can run successfully.	#11944
TM0	E2E Testing	Add Jest Snapshot Testing for RNTester Sample Pages	#11944
TM0	Linting	Ensure Fabric code is passing through linting checks.
TM1	E2ETesting	Add support for enabling the AutomationClient.	#12037
TM1	E2E Testing	Implement findElementbyId logic. Test correctness by toggling between Component and API tabs.	#12052
TM1	E2E Testing	Implement search for a particular component page logic. Test correctness by navigating to the Button page.	#12052
TM1	E2E Testing	Implement which visits all pages.	#12052
TM1	Native Build Testing	Add native build tests to CI.	#12037
TM2	E2E Testing	Implement Composition tree dump logic. Test correctness using app home page.	#12322
TM2	CLI Testing	Add CLI tests to CI.	#12183
TM2	E2E Testing	Implement test which dumps contents of each component page and matches page to stored snapshot.	#11944
TM2	E2E Testing	Implement mouse click tests.
TM2	E2E Testing	Implement Legacy tests.	#12845
TM3	E2E Testing	Implement functional testing of components (More granular tasks to come). #12031
TM3	Template Testing	Add template tests to CI.	#12376
TM4	E2E Testing	Implement accessibility testing of components (More granular tasks to come). #12031
TM4	Community Module Testing	Create a build of React Native Gallery on Fabric. Decide if both copies should exist and be published separately, or if migrating from one to the other is preferred.

Tasks

Preview Give feedback

1. Determine project structure for E2E testing for Fabric (i.e. separate app or reuse existing, directory structure).
2. Add E2E Test App for Fabric #11407
   Area: Fabric New Architecture +2
   
3. [Fabric] Add Snapshot Testing for RNTester Sample Pages #11944
   +0
4. Ensure Fabric code is passing through linting checks.
5. Enable AutomationClient for E2E Testing on Fabric #12037
   +0
6. Add VisitAllPages Test to E2E Test App on Fabric #12052
   +0
7. Implement Composition tree dump logic. Test correctness using app home page.
8. Add CLI tests to CI
9. Implement test which dumps contents of each component page and matches page to stored snapshot.
10. Implement mouse click tests.
11. [Fabric] Build Suite of Functional JS Tests for Fabric #12031
    0 of 12
    Area: Fabric Area: Test Infrastructure New Architecture Workstream: Test Coverage enhancement +5
12. Add template tests to CI.
13. Implement accessibility testing of components (More granular tasks to come).
14. Create a build of React Native Gallery on Fabric. Decide if both copies should exist and be published separately, or if migrating from one to the other is preferred.
15. Add UI Tests Validating Live Reload/Fast Refresh #8609
    Area: Tests Engineering Improvement Candidate Needs: Dev Design Scenario: Reload Workstream: Test Coverage enhancement +6
16. Add UI Tests Validating onHover Behavior #13235
    Area: Fabric Area: Pressable Area: Tests Workstream: Test Coverage enhancement +5
17. Add UI Tests Validating Pressed State #13236
    Area: Fabric Area: Pressable Area: Tests New Architecture Workstream: Test Coverage enhancement +6
18. Add Text Data to Composition Tree Dump Contents #13237
    Area: Fabric Area: Tests Area: Text New Architecture Workstream: Test Coverage enhancement +6
19. Implement accessibility testing of components #13238
    Area: Accessibility Area: Fabric Area: Tests New Architecture Workstream: Accessibility enhancement +6
20. Add Tests for Flyout and Popup Launch #11102
    Area: Flyout Area: Paper Area: Popup Area: Tests Old Architecture Workstream: Test Coverage enhancement +7