add/selenium-tests: selenium tests that replicate the existing e2e tests #3664
Conversation
|
I'm aware Cypress has limitations but I personally don't think we should use several frameworks for our tests. If we were to change, I think we should change all our tests and from previous experiences, Selenium adds a lot of burden and maintainability issues because of random errors. I think we should just hold on testing those areas for now or try to test the smaller bits using unit tests. Not saying we shouldn't look for alternatives for Cypress in the future if we find it doesn't do the job properly but It has proven to be stable and pretty easy to maintain and work with IMO. |
I have already ported all the existing Cypress tests so we can switch to Selenium completely. It was actually a pretty nice experience. The main point of trickiness was loading up the new page because I had to dismiss a confirmation popup and ensure that the editor was set to visual mode. Things are more verbose but I enjoy the control that it gives you. The only errors I have received so far have not been "random" but were due to changes in page structure (ie changed class names) and they would have also needed to be changed in Cypress.
We're already hit major limitations in Cypress. @ephox-mogran tried to test keyboard navigation with it and found it could not send a Tab key ( cypress-io/cypress#299 - it looks like this may have been fixed now). I tried to test selection with it and found that typing does not set the selection realistically. I can not test the behavior of Autocomplete with it after spending a day trying and it took me 20minutes to get the test working using Selenium. Weirdly Cypress does not actually seem to be testing the behaviour of Chrome - it is testing the behavior of Cypress which just happens to run in Chrome... Cypress is rapidly evolving at the moment. It may turn out to be a great environment for testing in future but it is insufficient at the moment. Another counterpoint is that Cypress only tests Chrome (and derivatives). The majority of the developers already test in Chrome every day because that is the primary browser used for development so we really need automated tests that can run on other browsers like Edge, Firefox and Safari. |
Codecov Report
@@ Coverage Diff @@
## master #3664 +/- ##
==========================================
- Coverage 39.19% 37.45% -1.75%
==========================================
Files 300 277 -23
Lines 7095 6707 -388
Branches 1307 1222 -85
==========================================
- Hits 2781 2512 -269
+ Misses 3620 3536 -84
+ Partials 694 659 -35
Continue to review full report at Codecov.
|
- Modified quote test to account for new <cite> tag - Match "Code Editor" & "Visual Editor" menu items with regex to ignore the keyboard shortcut - Add a delay to login so the onload javascript does not change focus while typing
|
Valid points about cypress here, we're going with Google puppeteer right now. Let's give it try and see how it goes. Thanks for your time trying alternatives. |
Description
Since Cypress only supports Chrome (and derivatives) currently and it has some other limitations to do with tab navigation and selection I decided to explore Selenium testing.
It still has limitations - notably actions like Shift+Click or key combinations like Ctrl+a don't seem to be possible on Firefox because Selenium web-driver for node does not support the new W3C web driver specification yet.
Run in Chrome:
Run in Firefox:
Note in theory you should be able to change the test_browser environment variable to specify other browsers like ie, edge and safari but I have not tried it yet.
How Has This Been Tested?
The tests have been run on Firefox and Chrome on Linux Mint.
Screenshots (jpeg or gifs if applicable):
Types of changes
New feature.
Checklist: