Auto-install and run standalone test proxy server per test package (#…

…21168)

* Auto-install and run standalone test proxy server per test package

* Update recording tests to use standalone proxy

* Simplify proxy binary switch statement

* Add test proxy auto-install docs

* Fix up recording test coverage

* Add StopTestProxy note about go process handling

* Proxy restore/race condition handling. Force ignore PROXY_MANUAL_START in internal tests

* Fix recording readme error handling

.gitignore

@@ -53,3 +53,6 @@ vendor/
 
 # Default Test Proxy Assets restore directory
 .assets
+
+# Default Test Proxy tools install directory
+.proxy

documentation/developer_setup.md

@@ -135,7 +135,7 @@ Testing is built into the Go toolchain as well with the `testing` library. The t
 | playback | `$ENV:AZURE_RECORD_MODE="playback"` | Running tests against recording HTTP interactiosn |
 | live | `$ENV:AZURE_RECORD_MODE="live"` | Bypassing test proxy, running against live service, and not recording HTTP interactions (used by live pipelines) |
 
-To get started first [install test-proxy][test_proxy_install] via the standalone executable. Then to start the proxy, from the root of the repository, run the command `test-proxy start`.
+By default the [recording](recording_package) package will automatically install and run the test proxy server. If there are issues with auto-install or the proxy needs to be run standalone, it can be run manually instead. To get started first [install test-proxy][test_proxy_install] via the standalone executable, then to start the proxy, from the root of the repository, run the command `test-proxy start`. When invoking tests, set the environment variable `PROXY_MANUAL_START` to `true`.
 
 ### Test Mode Options
 
@@ -380,3 +380,4 @@ This creates the pipelines that will verify future PRs. The `azure-sdk-for-go` i
 [autorest_intro]: https://github.com/Azure/autorest/blob/main/docs/readme.md
 [autorest_directives]: https://github.com/Azure/autorest/blob/main/docs/generate/directives.md
 [test_resources]: https://github.com/Azure/azure-sdk-tools/tree/main/eng/common/TestResources
+[recording_package]: https://github.com/Azure/azure-sdk-for-go/tree/main/sdk/internal/recording

sdk/internal/CHANGELOG.md

@@ -4,6 +4,8 @@
 
 ### Features Added
 
+* Add support for auto-installing the test proxy standalone tooling in the test recording package
+
 ### Breaking Changes
 
 ### Bugs Fixed

sdk/internal/perf/recording_test.go

@@ -4,20 +4,37 @@
 package perf
 
 import (
+	"fmt"
 	"net/http"
+	"os"
 	"regexp"
 	"testing"
 
 	"github.com/stretchr/testify/require"
+
+	"github.com/Azure/azure-sdk-for-go/sdk/internal/recording"
 )
 
 func TestRecordingHTTPClient_Do(t *testing.T) {
+	// Ignore manual start in pipeline tests, we always want to exercise install
+	os.Setenv(recording.ProxyManualStartEnv, "false")
+
+	proxy, err := recording.StartTestProxy("", nil)
+	require.NoError(t, err)
+	defer func() {
+		err := recording.StopTestProxy(proxy)
+		if err != nil {
+			panic(err)
+		}
+	}()
+
 	req, err := http.NewRequest("POST", "https://www.bing.com", nil)
 	require.NoError(t, err)
 
+	proxyURL := fmt.Sprintf("https://localhost:%d", proxy.Options.ProxyPort)
 	client := NewProxyTransport(&TransportOptions{
 		TestName: t.Name(),
-		proxyURL: "https://localhost:5001/",
+		proxyURL: proxyURL,
 	})
 	require.NotNil(t, client)
 
@@ -32,7 +49,7 @@ func TestRecordingHTTPClient_Do(t *testing.T) {
 	require.NoError(t, err)
 	resp, err = client.Do(req)
 	require.NoError(t, err)
-	require.Equal(t, "https://localhost:5001", resp.Request.URL.String())
+	require.Equal(t, proxyURL, resp.Request.URL.String())
 	require.Contains(t, resp.Request.Header.Get(upstreamURIHeader), "https://www.bing.com")
 	require.Equal(t, resp.Request.Header.Get(modeHeader), "record")
 	require.Equal(t, resp.Request.Header.Get(idHeader), client.recID)
@@ -42,7 +59,7 @@ func TestRecordingHTTPClient_Do(t *testing.T) {
 	require.NoError(t, err)
 	resp, err = client.Do(req)
 	require.NoError(t, err)
-	require.Equal(t, "https://localhost:5001", resp.Request.URL.String())
+	require.Equal(t, proxyURL, resp.Request.URL.String())
 	require.Contains(t, resp.Request.Header.Get(upstreamURIHeader), "https://www.bing.com")
 	require.Equal(t, resp.Request.Header.Get(modeHeader), "playback")
 	require.Equal(t, resp.Request.Header.Get(idHeader), client.recID)

sdk/internal/recording/README.md

@@ -16,9 +16,33 @@ After you've set the `AZURE_RECORD_MODE`, set the `PROXY_CERT` environment varia
 $ENV:PROXY_CERT="C:/ <path-to-repo> /azure-sdk-for-go/eng/common/testproxy/dotnet-devcert.crt"
 ```
 
+## Running the test proxy
+
+Recording and playing back tests relies on the [Test Proxy](https://github.com/Azure/azure-sdk-tools/blob/main/tools/test-proxy/Azure.Sdk.Tools.TestProxy/README.md) to intercept traffic. The recording package can automatically install and run an instance of the test-proxy server per package. The following code needs to be added to test setup and teardown in order to achieve this:
+
+```golang
+func TestMain(m *testing.M) {
+	proxy, err := recording.StartTestProxy(nil)
+	if err != nil {
+		panic(err)
+	}
+
+    ... all other test code, including proxy recording setup ...
+
+	code := m.Run()
+
+	err = recording.StopTestProxy(proxy)
+	if err != nil {
+		panic(err)
+	}
+
+	os.Exit(code)
+}
+```
+
 ## Routing Traffic
 
-The first step in instrumenting a client to interact with recorded tests is to direct traffic to the proxy through a custom `policy`. In these examples we'll use testify's [`require`](https://pkg.go.dev/github.com/stretchr/testify/require) library but you can use the framework of your choice. Each test has to call `recording.Start` and `recording.Stop`, the rest is taken care of by the `recording` library and the [`test-proxy`](https://github.com/Azure/azure-sdk-tools/tree/main/tools/test-proxy)
+The first step in instrumenting a client to interact with recorded tests is to direct traffic to the proxy through a custom `policy`. In these examples we'll use testify's [`require`](https://pkg.go.dev/github.com/stretchr/testify/require) library but you can use the framework of your choice. Each test has to call `recording.Start` and `recording.Stop`, the rest is taken care of by the `recording` library and the [`test-proxy`](https://github.com/Azure/azure-sdk-tools/tree/main/tools/test-proxy).
 
 The snippet below demonstrates an example test policy:
 

sdk/internal/recording/matchers.go

@@ -9,6 +9,7 @@ package recording
 import (
 	"bytes"
 	"encoding/json"
+	"fmt"
 	"io"
 	"net/http"
 	"strings"
@@ -76,7 +77,8 @@ func SetDefaultMatcher(t *testing.T, options *SetDefaultMatcherOptions) error {
 		return nil
 	}
 	options.fillOptions()
-	req, err := http.NewRequest("POST", "http://localhost:5000/Admin/SetMatcher", http.NoBody)
+	url := fmt.Sprintf("%s/Admin/SetMatcher", defaultOptions().baseURL())
+	req, err := http.NewRequest("POST", url, http.NoBody)
 	if err != nil {
 		panic(err)
 	}