Migration to terraform-plugin-testing causes panic in existing test suite

[jacobbednarz]

terraform-plugin-testing version

$ go list -m github.com/hashicorp/terraform-plugin-testing/...
github.com/hashicorp/terraform-plugin-testing v1.1.0

Relevant provider source code

// n/a

Terraform Configuration Files

// n/a

Expected Behavior

$ make test
go test -i $(go list ./...) || exit 1
?   	github.com/cloudflare/terraform-provider-cloudflare	[no test files]
echo $(go list ./...) | \
		xargs -t -n4 go test  -timeout=30s -parallel=4
go test -timeout=30s -parallel=4 github.com/cloudflare/terraform-provider-cloudflare github.com/cloudflare/terraform-provider-cloudflare/internal/consts github.com/cloudflare/terraform-provider-cloudflare/internal/framework/modifiers/defaults github.com/cloudflare/terraform-provider-cloudflare/internal/framework/provider
?   	github.com/cloudflare/terraform-provider-cloudflare	[no test files]
?   	github.com/cloudflare/terraform-provider-cloudflare/internal/consts	[no test files]
?   	github.com/cloudflare/terraform-provider-cloudflare/internal/framework/modifiers/defaults	[no test files]
ok  	github.com/cloudflare/terraform-provider-cloudflare/internal/framework/provider	1.282s [no tests to run]
go test -timeout=30s -parallel=4 github.com/cloudflare/terraform-provider-cloudflare/internal/framework/service/example github.com/cloudflare/terraform-provider-cloudflare/internal/sdkv2provider github.com/cloudflare/terraform-provider-cloudflare/internal/utils
ok  	github.com/cloudflare/terraform-provider-cloudflare/internal/framework/service/example	1.049s [no tests to run]
?   	github.com/cloudflare/terraform-provider-cloudflare/internal/utils	[no test files]
ok  	github.com/cloudflare/terraform-provider-cloudflare/internal/sdkv2provider	0.942s

This is reproducible by checking out the master branch that has not migrated to the terraform-plugin-testing repository and still uses the SDKv2 references.

Actual Behavior

$ make test

go test -i $(go list ./...) || exit 1
?   	github.com/cloudflare/terraform-provider-cloudflare	[no test files]
echo $(go list ./...) | \
		xargs -t -n4 go test  -timeout=30s -parallel=4
go test -timeout=30s -parallel=4 github.com/cloudflare/terraform-provider-cloudflare github.com/cloudflare/terraform-provider-cloudflare/internal/consts github.com/cloudflare/terraform-provider-cloudflare/internal/framework/modifiers/defaults github.com/cloudflare/terraform-provider-cloudflare/internal/framework/provider
?   	github.com/cloudflare/terraform-provider-cloudflare	[no test files]
?   	github.com/cloudflare/terraform-provider-cloudflare/internal/consts	[no test files]
?   	github.com/cloudflare/terraform-provider-cloudflare/internal/framework/modifiers/defaults	[no test files]
ok  	github.com/cloudflare/terraform-provider-cloudflare/internal/framework/provider	(cached) [no tests to run]
go test -timeout=30s -parallel=4 github.com/cloudflare/terraform-provider-cloudflare/internal/framework/service/example github.com/cloudflare/terraform-provider-cloudflare/internal/sdkv2provider github.com/cloudflare/terraform-provider-cloudflare/internal/utils
ok  	github.com/cloudflare/terraform-provider-cloudflare/internal/framework/service/example	(cached) [no tests to run]
?   	github.com/cloudflare/terraform-provider-cloudflare/internal/utils	[no test files]
/var/folders/36/zlscnhfn27n1yxx52cr1kdmr0000gp/T/go-build1935981002/b191/sdkv2provider.test flag redefined: sweep
panic: /var/folders/36/zlscnhfn27n1yxx52cr1kdmr0000gp/T/go-build1935981002/b191/sdkv2provider.test flag redefined: sweep

goroutine 1 [running]:
flag.(*FlagSet).Var(0xc0000a2180, {0x1f4c920, 0xc0001d3960}, {0x1d60fcd, 0x5}, {0x1d99c6b, 0x29})
	/Users/jacob/.asdf/installs/golang/1.20.1/go/src/flag/flag.go:982 +0x2f9
flag.(*FlagSet).StringVar(...)
	/Users/jacob/.asdf/installs/golang/1.20.1/go/src/flag/flag.go:847
flag.(*FlagSet).String(0x18938b1?, {0x1d60fcd, 0x5}, {0x0, 0x0}, {0x1d99c6b, 0x29})
	/Users/jacob/.asdf/installs/golang/1.20.1/go/src/flag/flag.go:860 +0xac
flag.String(...)
	/Users/jacob/.asdf/installs/golang/1.20.1/go/src/flag/flag.go:867
github.com/hashicorp/terraform-plugin-testing/helper/resource.init()
	/Users/jacob/.asdf/installs/golang/1.20.1/packages/pkg/mod/github.com/hashicorp/terraform-plugin-testing@v1.1.0/helper/resource/testing.go:53 +0x45
FAIL	github.com/cloudflare/terraform-provider-cloudflare/internal/sdkv2provider	0.880s
FAIL
make: *** [test] Error 1

Steps to Reproduce

1. Checkout tests: swap to provider agnostic test plugin cloudflare/terraform-provider-cloudflare#2272
2. Run make test

References

I'm not seeing any differences between sdkv2 (https://github.com/hashicorp/terraform-plugin-sdk/blob/main/helper/resource/testing.go#L48) and this repository (https://github.com/hashicorp/terraform-plugin-testing/blob/main/helper/resource/testing.go#L53) that would be causing this. I suspect it may be a bug in either my find and replace that was done as part of the migration or a bug in our existing test suite that we are now privvy to but would love some guidance on where to start with this one.

I've tried running this one through git bisect to see where it's falling apart but the cross repository interactions and merging is making it a little tough without more understanding of where I should focus my effort.

[bendbennett]

Hi @jacobbednarz 👋

Sorry you ran into trouble here.

The panic suggests that the sweep flag is being redefined:

panic: /var/folders/36/zlscnhfn27n1yxx52cr1kdmr0000gp/T/go-build1935981002/b191/sdkv2provider.test flag redefined: sweep

The sweep flag is defined within /helper/resource/testing.go.

After cloning and checking out cloudflare/terraform-provider-cloudflare#2272 a search for /helper/resource reveals that both github.com/hashicorp/terraform-plugin-sdk/v2/helper/resource and github.com/hashicorp/terraform-plugin-testing/helper/resource are being imported for instance:

• https://github.com/cloudflare/terraform-provider-cloudflare/blob/swap-to-testing-lib/internal/sdkv2provider/resource_cloudflare_custom_hostname.go#L15
• https://github.com/cloudflare/terraform-provider-cloudflare/blob/swap-to-testing-lib/internal/sdkv2provider/data_source_access_identity_provider_test.go#L10

Can you check that all of the required updates when switching to using terraform-plugin-testing have been made as outlined in the migration guide.

[jacobbednarz]

thanks for the pointer @bendbennett.

the migration guide only calls out swapping test files (_test.go) suffix. should this instead be extended to all files? i'm assuming it's 1:1 however, seems a little odd to be requiring a test related library outside of the tests. is this the intended behaviour?

[bendbennett]

@jacobbednarz you are quite right. The migration guide only refers to _test.go files. Will discuss with the team and get back to you.

[jacobbednarz]

appreciate it! 🙇

[bflad]

github.com/hashicorp/terraform-plugin-sdk/v2@v2.26.1 is now available with separated helper/retry and helper/id packages so non-testing helper/resource code can fully migrated away in the older dependency (thereby removing the duplicate TestMain definitions). The website migration guide content has been updated in this repository via #96 and will be published soon with the terraform-plugin-testing version 1.2.0 release. 👍

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.