[BUG]: Differences between Chrome and Firefox

[tombrunet]

Project

accessibility-checker-engine

Browser

Firefox

Operating system

MacOS

Description

When building the Java tool, I tried with the Firefox Selenium engine. These testcases don't match the results. It might be because Firefox is auto-fixing some things, but we should at least verify:

"aria_role_redundant_ruleunit", "Fail.html"
"style_focus_visible_ruleunit", "CSS-used.html"
"object_text_exists_ruleunit", "act_fail_3.html"
"aria_attribute_conflict_ruleunit", "aria-hidden.html"
"aria_attribute_valid_ruleunit", "ValidRoleSpecified.html"
"aria_attribute_valid_ruleunit", "ValidAttribute.html"
"aria_attribute_valid_ruleunit", "ValidRoleSpecifiedValidAttribute.html"
"aria_attribute_valid_ruleunit", "InvalidAttribute.html"
"aria_attribute_valid_ruleunit", "InValidRoleInvalidAttribute.html"
"aria_attribute_valid_ruleunit", "InValidRoleSpecified.html"
"aria_attribute_valid_ruleunit", "ValidRoleSpecifiedInvalidAttribute.html"
"aria_attribute_valid_ruleunit", "ValidRoleSpecifiedMultiple.html"
"aria_attribute_valid_ruleunit", "InValidRoleSpecifiedMultiple.html"
"aria_attribute_valid_ruleunit", "area_element_test.html"
"aria_attribute_valid_ruleunit", "elementsWithSupportingAttributes.html"
"aria_role_valid_ruleunit", "area_no_href.html"
"element_scrollable_tabbable_ruleunit", "act_fail2.html"
"element_scrollable_tabbable_ruleunit", "textarea_pass2.html"
"element_scrollable_tabbable_ruleunit", "act_fail1.html"
"fieldset_label_valid_ruleunit", "FieldSet-hasarialabel.html"
"fieldset_label_valid_ruleunit", "FieldSet-nested.html"
"fieldset_label_valid_ruleunit", "test_mixed_1.html"
"a_text_purpose_ruleunit", "A-nonTabable.html"
"a_text_purpose_ruleunit", "A-slot-text-error2.html"
"a_text_purpose_ruleunit", "A-slot-text-error1.html"
"target_spacing_sufficient_ruleunit", "link_in_text.html"
"target_spacing_sufficient_ruleunit", "block_element_inline.html"
"aria_landmark_name_unique_ruleunit", "example_0_fail.html"

Steps to reproduce

Will be easier once the java checker is merged. From the java-accessibility-checker directory, can run:

mvn test -Dtest="AccessibilityCheckerSeleniumFFTest"

Will have to comment out the skip items to see them fail.

[shunguoy]

Chrome and FF with different results: http://127.0.0.1:5500/test/v2/checker/accessibility/rules/aria_role_redundant_ruleunit/Fail.html
potential causes: (1) the browser removes/fixes some attributes; (2) some functions may be different; (3) potential rule issue.

[shunguoy]

Investigate the root issues, then open bugs on each finding.

[shunguoy]

Here are the Chrome vs. Firefox test results:

"aria_role_redundant_ruleunit", "Fail.html" Same
"style_focus_visible_ruleunit", "CSS-used.html". Different
"object_text_exists_ruleunit", "act_fail_3.html" Same
"aria_attribute_conflict_ruleunit", "aria-hidden.html". Same
"aria_attribute_valid_ruleunit", "ValidRoleSpecified.html" Different
"aria_attribute_valid_ruleunit", "ValidAttribute.html" Different
"aria_attribute_valid_ruleunit", "ValidRoleSpecifiedValidAttribute.html". Different
"aria_attribute_valid_ruleunit", "InvalidAttribute.html" Different
"aria_attribute_valid_ruleunit", "InValidRoleInvalidAttribute.html" Different
"aria_attribute_valid_ruleunit", "InValidRoleSpecified.html" Different
"aria_attribute_valid_ruleunit", "ValidRoleSpecifiedInvalidAttribute.html". Different
"aria_attribute_valid_ruleunit", "ValidRoleSpecifiedMultiple.html" Different
"aria_attribute_valid_ruleunit", "InValidRoleSpecifiedMultiple.html" Different
"aria_attribute_valid_ruleunit", "area_element_test.html" Same
"aria_attribute_valid_ruleunit", "elementsWithSupportingAttributes.html". Different
"aria_role_valid_ruleunit", "area_no_href.html" Same
"element_scrollable_tabbable_ruleunit", "act_fail2.html" Different, known
"element_scrollable_tabbable_ruleunit", "textarea_pass2.html" Same
"element_scrollable_tabbable_ruleunit", "act_fail1.html" Different, known
"fieldset_label_valid_ruleunit", "FieldSet-hasarialabel.html" Same
"fieldset_label_valid_ruleunit", "FieldSet-nested.html" Same
"fieldset_label_valid_ruleunit", "test_mixed_1.html" Same
"a_text_purpose_ruleunit", "A-nonTabable.html" Different
"a_text_purpose_ruleunit", "A-slot-text-error2.html" Different
"a_text_purpose_ruleunit", "A-slot-text-error1.html" Different
"target_spacing_sufficient_ruleunit", "link_in_text.html" Same
"target_spacing_sufficient_ruleunit", "block_element_inline.html". Different
"aria_landmark_name_unique_ruleunit", "example_0_fail.html" Same