Migrate integration tests to Go testing framework #1839

pracucci · 2019-11-19T12:06:42Z

What this PR does:
This PR introduces an easy-to-use framework to write Cortex integration tests in Go. The current integration tests written in Bash have been ported (and enriched) into Go, and a couple of new tests have been added:

Backward compatibility test
Ingester transfer with blocks storage

Worth to note:

In CircleCI, the configs DB integration tests and the new tests have been splitted into 2 pipelines, to run them concurrently
I've tried to fine-tune the Minio and Consul config to reduce the startup time (I haven't found any possibile optimization on DynamoDB local)
make test doesn't run tests in the integration/ folder
Being based on the Go testing framework, I can run integration tests directly in the IDE (VSCode) as far as the cortex:latest Docker image is updated. On my local machine a single integration test takes about 10s and re-iterating on it (while developing it) looks pretty handy

Which issue(s) this PR fixes:
No issue

Checklist

Tests updated
Documentation added
CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

bboreham · 2019-11-19T17:43:05Z

I support the idea of moving to Go, but Compose seems to be adding a lot of weight - is it justified?
It looks like we will need another 50-line Compose file for each test?

pracucci · 2019-11-19T17:51:26Z

Compose seems to be adding a lot of weight - is it justified?
It looks like we will need another 50-line Compose file for each test?

@bboreham It's a good question. My idea (which is not visible in this PoC) is that you need a compose file for each scenario, not test suite. A scenario is a cluster setup (ie. storage based on Dynamo+S3 or another scenario with gossip ring instead of consul, etc), upon which you can run multiple tests. Each test suite runs against a scenario and within a single suite you can have multiple semi-isolated tests running each test with a different tenant ID (so the "framework" may give each test a unique tenant ID to use).

pstibrany

I like this a lot. Tests are readable and checks make more sense. They are also easy to run using standard go test tools. Big thumbs up! Thanks Marco for your effort.

pstibrany · 2020-02-06T09:53:11Z

integration/framework/scenario.go

+	}
+
+	// Setup the docker network
+	if out, err := RunCommandAndGetOutput("docker", "network", "create", s.networkName); err != nil {


Would it be possible to detect Error response from daemon: network with name cortex-1 already exists message and ignore it? Or perhaps try to delete it first. It's the most common issue I've run into when trying to run tests (esp. when I cancel the test in the middle).

The problem is not removing or reusing the network if already exists, but after that error you will get an error for already existing containers. You get this error if the previous test run panicked (ie. test timeout) and automatically cleaning up containers in a safe way is non trivial.

Please let me know if you ended up into this situation without a panic.

It was pretty common to run into problem with container left behind, when I didn't have CORTEX_IMAGE and CORTEX_INTEGRATION_DIR set correctly.

With unset CORTEX_INTEGRATION_DIR, docker starts minio container, but minio then fails. However, since this is reported as error in the StartService code, service is not appended to list of started services, and is then not cleaned up.

After talking to Peter, we actually found an issue. If service.Start() did return error (for any reason) the container may have been created but it was not cleaned up. I've pushed a fix for this (commit).

In the previous version I force-removed all containers and the network before running each scenario.
(That is what stop-components.sh did)

Ok. I've added the logic to cleanup the Docker containers + network before each test, so that we can now automatically recover from a spurious situation left from the previous run. The change is in this commit.

integration/framework/service.go

integration/framework/scenario.go

integration/framework/service.go

bboreham

Some nits below; more importantly I haven't had a clean run yet, and have broadly no idea how to debug.

$ CORTEX_IMAGE=quay.io/cortexproject/cortex:latest go test -timeout 300s -v -count=1 ./integration
=== RUN   TestBackwardCompatibilityWithChunksStorage
Starting dynamodb
Starting consul
dynamodb: Initializing DynamoDB Local with the following configuration:
dynamodb: Port:	8000
dynamodb: InMemory:	true
dynamodb: DbPath:	null
dynamodb: SharedDb:	true
dynamodb: shouldDelayTransientStatuses:	false
dynamodb: CorsParams:	*
consul: ==> Starting Consul agent...
consul: ==> Consul agent running!
consul: Version: 'v0.9.4'
consul: Node ID: '09f38a58-45f1-6e51-66e2-88b92af6b841'
consul: Node name: 'consul'
consul: Datacenter: 'dc1' (Segment: '<all>')
consul: Server: true (Bootstrap: false)
consul: Client Addr: 0.0.0.0 (HTTP: 8500, HTTPS: -1, DNS: 8600)
consul: Cluster Addr: 127.0.0.1 (LAN: 8301, WAN: 8302)
consul: Encrypt: Gossip: false, TLS-Outgoing: false, TLS-Incoming: false
consul: ==> Log data will now stream in as it occurs:
Starting table-manager
Starting ingester-1
Starting distributor
consul: ==> Newer Consul version available: 1.6.3 (currently running: 0.9.4)
Starting ingester-2
Stopping ingester-1
ingester-1: level=warn ts=2020-02-07T11:36:00.848025258Z caller=transfer.go:423 msg="transfer attempt failed" err="cannot find ingester to transfer chunks to: no pending ingesters" attempt=1 max_retries=10
Starting querier
querier: 2020/02/07 11:36:01 Request.HasNextPage deprecated. Use Pagination type for configurable pagination of API operations
Stopping querier
Starting querier
querier: 2020/02/07 11:36:02 Request.HasNextPage deprecated. Use Pagination type for configurable pagination of API operations
Stopping querier
Killing ingester-2
ingester-2: level=warn ts=2020-02-07T11:36:02.827192046Z caller=transfer.go:422 msg="transfer attempt failed" err="cannot find ingester to transfer chunks to: no pending ingesters" attempt=1 max_retries=10
Killing distributor
Killing table-manager
Killing consul
Killing dynamodb
--- PASS: TestBackwardCompatibilityWithChunksStorage (7.94s)
=== RUN   TestIngesterFlushWithChunksStorage
Starting dynamodb
Starting consul
dynamodb: Initializing DynamoDB Local with the following configuration:
dynamodb: Port:	8000
dynamodb: InMemory:	true
dynamodb: DbPath:	null
dynamodb: SharedDb:	true
dynamodb: shouldDelayTransientStatuses:	false
dynamodb: CorsParams:	*
consul: ==> Starting Consul agent...
consul: ==> Consul agent running!
consul: Version: 'v0.9.4'
consul: Node ID: 'db75608b-b074-d963-8c06-eabdc3d6a4b0'
consul: Node name: 'consul'
consul: Datacenter: 'dc1' (Segment: '<all>')
consul: Server: true (Bootstrap: false)
consul: Client Addr: 0.0.0.0 (HTTP: 8500, HTTPS: -1, DNS: 8600)
consul: Cluster Addr: 127.0.0.1 (LAN: 8301, WAN: 8302)
consul: Encrypt: Gossip: false, TLS-Outgoing: false, TLS-Incoming: false
consul: ==> Log data will now stream in as it occurs:
Starting table-manager
Starting ingester-1
Starting querier
Starting distributor
querier: 2020/02/07 11:36:08 Request.HasNextPage deprecated. Use Pagination type for configurable pagination of API operations
Stopping ingester-1
Killing distributor
Killing querier
Killing table-manager
Killing consul
Killing dynamodb
--- PASS: TestIngesterFlushWithChunksStorage (6.15s)
=== RUN   TestIngesterHandOverWithBlocksStorage
Starting minio
Starting consul
minio: Attempting encryption of all config, IAM users and policies on MinIO backend
minio: API: SYSTEM()
minio: Time: 11:36:10 UTC 02/07/2020
minio: DeploymentID: 69cd1dbf-5fa9-4fda-a7cd-2223e17064cc
minio: Error: disk path full
minio: 8: cmd/fs-v1-helpers.go:320:cmd.fsCreateFile()
minio: 7: cmd/fs-v1.go:1013:cmd.(*FSObjects).putObject()
minio: 6: cmd/fs-v1.go:927:cmd.(*FSObjects).PutObject()
minio: 5: cmd/config-common.go:62:cmd.saveConfig()
minio: 4: cmd/config-encrypted.go:305:cmd.migrateConfigPrefixToEncrypted()
minio: 3: cmd/config-encrypted.go:109:cmd.handleEncryptedConfigBackend()
minio: 2: cmd/server-main.go:208:cmd.initSafeMode()
minio: 1: cmd/server-main.go:373:cmd.serverMain()
minio: API: SYSTEM()
minio: Time: 11:36:10 UTC 02/07/2020
minio: DeploymentID: 69cd1dbf-5fa9-4fda-a7cd-2223e17064cc
minio: Error: disk path full
minio: 8: cmd/fs-v1-helpers.go:320:cmd.fsCreateFile()
minio: 7: cmd/fs-v1.go:1013:cmd.(*FSObjects).putObject()
minio: 6: cmd/fs-v1.go:927:cmd.(*FSObjects).PutObject()
minio: 5: cmd/config-common.go:62:cmd.saveConfig()
minio: 4: cmd/config-encrypted.go:305:cmd.migrateConfigPrefixToEncrypted()
minio: 3: cmd/config-encrypted.go:109:cmd.handleEncryptedConfigBackend()
minio: 2: cmd/server-main.go:208:cmd.initSafeMode()
minio: 1: cmd/server-main.go:373:cmd.serverMain()
consul: ==> Starting Consul agent...
consul: ==> Consul agent running!
consul: Version: 'v0.9.4'
consul: Node ID: '5ce784bb-6181-052a-18b3-620571ab9a86'
consul: Node name: 'consul'
consul: Datacenter: 'dc1' (Segment: '<all>')
consul: Server: true (Bootstrap: false)
consul: Client Addr: 0.0.0.0 (HTTP: 8500, HTTPS: -1, DNS: 8600)
consul: Cluster Addr: 127.0.0.1 (LAN: 8301, WAN: 8302)
consul: Encrypt: Gossip: false, TLS-Outgoing: false, TLS-Incoming: false
consul: ==> Log data will now stream in as it occurs:
consul: ==> Newer Consul version available: 1.6.3 (currently running: 0.9.4)
Killing consul
Killing minio
minio: Exiting on signal: TERMINATED
--- FAIL: TestIngesterHandOverWithBlocksStorage (16.41s)
    ingester_hand_over_test.go:18: 
        	Error Trace:	ingester_hand_over_test.go:18
        	            				ingester_hand_over_test.go:54
        	            				ingester_hand_over_test.go:14
        	Error:      	Received unexpected error:
        	            	the service minio is not ready
        	Test:       	TestIngesterHandOverWithBlocksStorage
=== RUN   TestIngesterHandOverWithChunksStorage
Starting dynamodb
Starting consul
dynamodb: Initializing DynamoDB Local with the following configuration:
dynamodb: Port:	8000
dynamodb: InMemory:	true
dynamodb: DbPath:	null
dynamodb: SharedDb:	true
dynamodb: shouldDelayTransientStatuses:	false
dynamodb: CorsParams:	*
consul: ==> Starting Consul agent...
consul: ==> Consul agent running!
consul: Version: 'v0.9.4'
consul: Node ID: 'd5d51e32-0e52-e71c-0c3a-391411e89b86'
consul: Node name: 'consul'
consul: Datacenter: 'dc1' (Segment: '<all>')
consul: Server: true (Bootstrap: false)
consul: Client Addr: 0.0.0.0 (HTTP: 8500, HTTPS: -1, DNS: 8600)
consul: Cluster Addr: 127.0.0.1 (LAN: 8301, WAN: 8302)
consul: Encrypt: Gossip: false, TLS-Outgoing: false, TLS-Incoming: false
consul: ==> Log data will now stream in as it occurs:
Starting table-manager
Starting ingester-1
Starting querier
Starting distributor
querier: 2020/02/07 11:36:31 Request.HasNextPage deprecated. Use Pagination type for configurable pagination of API operations
Starting ingester-2
Stopping ingester-1
ingester-1: level=warn ts=2020-02-07T11:36:31.426057707Z caller=transfer.go:422 msg="transfer attempt failed" err="cannot find ingester to transfer chunks to: no pending ingesters" attempt=1 max_retries=10
Killing ingester-2
ingester-2: level=warn ts=2020-02-07T11:36:31.843151654Z caller=transfer.go:422 msg="transfer attempt failed" err="cannot find ingester to transfer chunks to: no pending ingesters" attempt=1 max_retries=10
Killing distributor
Killing querier
Killing table-manager
Killing consul
Killing dynamodb
--- PASS: TestIngesterHandOverWithChunksStorage (6.69s)
FAIL
FAIL	github.com/cortexproject/cortex/integration	37.188s
FAIL

bboreham · 2020-02-07T10:57:12Z

.circleci/config.yml

+    # Make sure the working directory is within the GOPATH, otherwise
+    # "go test" will not download module dependencies.


We should really turn on -mod=vendor to avoid this. Not part of this PR.

integration/framework/scenario.go

integration/framework/service.go

integration/framework/scenario.go

Signed-off-by: Marco Pracucci <marco@pracucci.com>

…tex:latest Signed-off-by: Marco Pracucci <marco@pracucci.com>

Signed-off-by: Marco Pracucci <marco@pracucci.com>

pracucci · 2020-02-07T14:48:11Z

more importantly I haven't had a clean run yet

@bboreham According to the output, I can see:

        	Error:      	Received unexpected error:
        	            	the service minio is not ready
        	Test:       	TestIngesterHandOverWithBlocksStorage

And looking at minio logs (starting with minio:) I can see:

minio: Error: disk path full

Not sure about the root cause tho, unless the machine where you run Docker host has disk full (I assume that's not the case).

Signed-off-by: Marco Pracucci <marco@pracucci.com>

bboreham

I got the tests to run through cleanly; I think my disk did fill up and I just didn't notice the error message.
Then, as an experiment, I made a breaking change in cortex.proto and although unit tests passed the integration tests failed, so they work!

Great job, thanks @pracucci

bboreham mentioned this pull request Feb 2, 2020

Add integration test using older image #2064

Closed

pracucci added the keepalive Skipped by stale bot label Feb 3, 2020

pracucci force-pushed the setup-integration-tests-with-docker-compose-and-golang branch 4 times, most recently from 71418db to 3aa987f Compare February 4, 2020 16:51

pracucci changed the title ~~Proposal: integration tests based on docker-compose and go testing framework~~ Migrate integration tests to Go testing framework Feb 4, 2020

pracucci marked this pull request as ready for review February 4, 2020 17:09

pracucci force-pushed the setup-integration-tests-with-docker-compose-and-golang branch from 456213c to 3de4cd0 Compare February 5, 2020 09:40

pracucci mentioned this pull request Feb 6, 2020

Upgrade gogo/protobuf to 1.3.0 #2055

Merged

2 tasks

pstibrany approved these changes Feb 6, 2020

View reviewed changes

pracucci force-pushed the setup-integration-tests-with-docker-compose-and-golang branch from 3de4cd0 to 502bc57 Compare February 6, 2020 11:05

bboreham reviewed Feb 7, 2020

View reviewed changes

pracucci added 9 commits February 7, 2020 15:35

Introduced integration tests in Go

f7487d7

Signed-off-by: Marco Pracucci <marco@pracucci.com>

Deleted old (bash-based) integration tests

8f8fa60

Signed-off-by: Marco Pracucci <marco@pracucci.com>

Enrich error on docker network creation failure

c538919

Signed-off-by: Marco Pracucci <marco@pracucci.com>

Fixed linter

7d07e07

Signed-off-by: Marco Pracucci <marco@pracucci.com>

Fixed Service.WaitMetric()

ae1fc29

Signed-off-by: Marco Pracucci <marco@pracucci.com>

Fixed issues spotted during code review

d1dc1cf

Signed-off-by: Marco Pracucci <marco@pracucci.com>

Ensure the container is removed if service.Start() return error

27933da

Signed-off-by: Marco Pracucci <marco@pracucci.com>

Changed default image from cortex:latest to quay.io/cortexproject/cor…

d79fe4e

…tex:latest Signed-off-by: Marco Pracucci <marco@pracucci.com>

Added a comment to explain what's Minio

c78d894

Signed-off-by: Marco Pracucci <marco@pracucci.com>

pracucci force-pushed the setup-integration-tests-with-docker-compose-and-golang branch from 7d7be2d to c78d894 Compare February 7, 2020 14:38

Fixed backoff logic in service.WaitStarted()

b3fd9c1

Signed-off-by: Marco Pracucci <marco@pracucci.com>

pracucci added 2 commits February 7, 2020 17:24

Cleanup the Docker containers and network before starting a new test

f81d282

Signed-off-by: Marco Pracucci <marco@pracucci.com>

Fixed linter

0695da2

Signed-off-by: Marco Pracucci <marco@pracucci.com>

bboreham approved these changes Feb 11, 2020

View reviewed changes

pracucci merged commit 6ecd6f6 into cortexproject:master Feb 12, 2020

pracucci mentioned this pull request Feb 12, 2020

Fixed linter #2117

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Migrate integration tests to Go testing framework #1839

Migrate integration tests to Go testing framework #1839

pracucci commented Nov 19, 2019 •

edited

Loading

bboreham commented Nov 19, 2019

pracucci commented Nov 19, 2019

pstibrany left a comment

pstibrany Feb 6, 2020

pracucci Feb 6, 2020

pstibrany Feb 6, 2020

pracucci Feb 6, 2020 •

edited

Loading

bboreham Feb 7, 2020

pracucci Feb 7, 2020

bboreham left a comment

bboreham Feb 7, 2020

pracucci commented Feb 7, 2020 •

edited

Loading

bboreham left a comment

		# Make sure the working directory is within the GOPATH, otherwise
		# "go test" will not download module dependencies.

Migrate integration tests to Go testing framework #1839

Migrate integration tests to Go testing framework #1839

Conversation

pracucci commented Nov 19, 2019 • edited Loading

bboreham commented Nov 19, 2019

pracucci commented Nov 19, 2019

pstibrany left a comment

Choose a reason for hiding this comment

pstibrany Feb 6, 2020

Choose a reason for hiding this comment

pracucci Feb 6, 2020

Choose a reason for hiding this comment

pstibrany Feb 6, 2020

Choose a reason for hiding this comment

pracucci Feb 6, 2020 • edited Loading

Choose a reason for hiding this comment

bboreham Feb 7, 2020

Choose a reason for hiding this comment

pracucci Feb 7, 2020

Choose a reason for hiding this comment

bboreham left a comment

Choose a reason for hiding this comment

bboreham Feb 7, 2020

Choose a reason for hiding this comment

pracucci commented Feb 7, 2020 • edited Loading

bboreham left a comment

Choose a reason for hiding this comment

pracucci commented Nov 19, 2019 •

edited

Loading

pracucci Feb 6, 2020 •

edited

Loading

pracucci commented Feb 7, 2020 •

edited

Loading