Automated smoke test with selenium against release tarball

[PVince81]

To be able to catch critical issues like owncloud/core#27670 early, we should implement selenium UI tests that do simple things like:

• logging in
• uploading a file
• checking the list
• downloading
• check if settings page loads
• logging out

Repeat these with the following matrix dimensions:

• "deployed in root" vs "deployed in subdir"
• "reverse proxy" vs "no proxy"
• "theme enabled" vs "no theme enabled" (can be tested with enterprise theme)

These automated tests should run at least on master and the stable branches, not on all PRs.

That would already be a good start.

Then if needed we can more items from the smoke test plan.

@owncloud/qa

[PVince81]

• also test appliance with this

[PVince81]

reuse this ticket for making it easier to run the existing acceptance @smokeTest tagged tests on a release tarball

[PVince81]

in fact, as long as we have the test suites available we could run more than just the smoke tests.

the goal here is to have a set of scripts to make this easier

[phil-davis]

yes, we can run whatever we like - smokeTest tag would most often be useful.

The thing to design/provide is an automated way to get (or make) a prospective "release" tarball, unpack it somewhere, install it, fire up Apache, then run the selected acceptance tests against the installed system. Have this as a drone "job" that can somehow be "easily" kicked off when we want.

[PVince81]

@individual-it can you add steps here how to run this locally ?

[individual-it]

1. clone git stable10
2. install oC from tarball
3. install oC from tarball again to an other folder (to be able to test fed-sharing tests)
4. in the tarball installation install the testing app
5. from the git clone directory run the tests with BEHAT_FILTER_TAGS='@smokeTest' e.g. TEST_SERVER_URL=http://172.17.0.1/owncloud-10.1.1/owncloud TEST_SERVER_FED_URL=http://172.17.0.1/owncloud-10.1.1/owncloud-fed BEHAT_FILTER_TAGS='@smokeTest' make test-acceptance-webui

[individual-it]

Look like most work is done here. Should we keep it open for the proxy work?
There is an other issue about running the tests explicitly against the release tarball #611

[individual-it]

last thing to do is theme enabled

[individual-it]

proposing to postpone this after we have the starlak implementation of drone

CC @phil-davis

[phil-davis]

I put this in the next sprint backlog. It will be useful to be able to "easily" run part or all the core acceptance test suites against a proposed core tarball.

[individual-it]

no need to test with theme for now, but we need a way to easily run tests against the tarball in CI

[phil-davis]

Note: I am working on the drone starlark infrastructure to make this easy to use in core. owncloud/core#38622

[phil-davis]

PR #38437 added the functions that can download a tarball and setup the oC10 server using the tarball as the system-under-test.

But it did not provide any "hooks" for using those functions. We need code in .drone.star that lets someone specify the tarball version to use in the top config section, and that there is code in the starlark that "knows what to do" to install the server-under-test from the specified tarball.

See PR #38622 for code that I made a start on.

I suggest that the acceptance tests config section can support something like:

'testAgainstCoreTarball': True, # true or false, default is false
'coreTarball': 'daily-master-qa', # the "name" of the tarball to use, like we do in oc10 apps - "daily-master-qa", "latest" etc

Then you can test against a tarball by simply setting testAgainstCoreTarball to True, and you will be testing against daily-master-qa by default.

I have unassigned myself. Someone can take this over if possible.

[dpakach]

@phil-davis I'll take over this issue

[phil-davis]

Finished in owncloud/core#38622
Docs issue: owncloud/docs#3559

Issue is done.