4.20.0 : Smoke Tests

[xpkoala]

Goal: easy-to-follow test steps for sanity checking a release manually

Fleet version (Head to the "My account" page in the Fleet UI or run fleetctl version):

Web browser (e.g. Chrome 88.0.4324):

Important reference data

1. fleetctl preview setup
2. permissions documentation
3. premium tests require license key (needs renewal) fleetctl preview --license-key=eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJGbGVldCBEZXZpY2UgTWFuYWdlbWVudCBJbmMuIiwiZXhwIjoxNjQwOTk1MjAwLCJzdWIiOiJkZXZlbG9wbWVudCIsImRldmljZXMiOjEwMCwibm90ZSI6ImZvciBkZXZlbG9wbWVudCBvbmx5IiwidGllciI6ImJhc2ljIiwiaWF0IjoxNjIyNDI2NTg2fQ.WmZ0kG4seW3IrNvULCHUPBSfFdqj38A_eiXdV_DFunMHechjHbkwtfkf1J6JQJoDyqn8raXpgbdhafDwv3rmDw
4. premium tests require license key (active - Expires Sunday, January 1, 2023 12:00:00 AM) fleetctl preview --license-key=eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJGbGVldCBEZXZpY2UgTWFuYWdlbWVudCBJbmMuIiwiZXhwIjoxNjcyNTMxMjAwLCJzdWIiOiJGbGVldCBEZXZpY2UgTWFuYWdlbWVudCIsImRldmljZXMiOjEwMCwibm90ZSI6ImZvciBkZXZlbG9wbWVudCBvbmx5IiwidGllciI6InByZW1pdW0iLCJpYXQiOjE2NDI1MjIxODF9.EGHQjIzM73YyMbnCruswzg360DEYCsDi9uz48YcDwQHq90BabGT5PIXRiculw79emGj5sk2aKgccTd2hU5J7Jw

Smoke Tests

Smoke tests are limited to core functionality and serve as a sanity test. If smoke tests are failing, a release cannot proceed.

Prerequisites

1. fleetctl preview is set up and running the desired test version using --tag parameters.
2. Unless you are explicitly testing older browser versions, browser is up to date.
3. Certificate & flagfile are in place to create new host.
4. In your browser, clear local storage using devtools.

Instructions

Test name	Step instructions	Expected result	pass/fail
$Name	{what a tester should do}	{what a tester should see when they do that}	pass/fail
Update flow	remove all fleet processes/agents/etc using fleetctl preview reset for a clean slate run fleetctl preview with no tag for latest stable create a host/query to later confirm upgrade with STOP fleet-preview-server instances in containers/apps on Docker run fleetctl preview with appropriate testing tag	All previously created hosts/queries are verified to still exist	pass
Login flow	navigate to the login page and attempt to login with both valid and invalid credentials to verify some combination of expected results. navigate to the login page and attempt to login with both valid and invalid sso credentials to verify expected results.	text fields prompt when blank correct error message is "authentication failed" forget password link prompts for email valid credentials result in a successful login. valid sso credentials result in a successful login	pass
Query flow	Create, edit, run, and delete queries.	permissions regarding creating/editing/deleting queries are up to date with documentation syntax errors result in error messaging queries can be run manually	pass
Host Flow	Verify a new host can be added and removed following modal instructions using your own device.	Host is added via command line Host serial number and date added are accurate Host is not visible after it is deleted Warning and informational modals show when expected and make sense	pass

Notes

Blocking issues:

• On Home page, issues with Munki card #7642

What has not been tested:

[xpkoala]

reed@reed fleet % FLEET_VERSION_A=v4.19.0 FLEET_VERSION_B=main go test ./test/upgrade 
ok  	github.com/fleetdm/fleet/v4/test/upgrade	51.473s
reed@reed fleet % FLEET_VERSION_A=v4.19.1 FLEET_VERSION_B=main go test ./test/upgrade
ok  	github.com/fleetdm/fleet/v4/test/upgrade	48.442s