Clean up knative-trusted-prow from config

[michelle192837]

Seeing frequent errors in the oss-prow logs like:

jsonPayload: {
component: "prow-controller-manager"
error: "failed to construct manager for cluster knative-prow-trusted: Get "https://35.226.143.163/api?timeout=32s": dial tcp 35.226.143.163:443: i/o timeout"
file: "k8s.io/test-infra/prow/cmd/prow-controller-manager/main.go:189"
func: "main.main"
level: "error"
msg: "Failed to construct build cluster managers. Please check that the kubeconfig secrets are correct, and that RBAC roles on the build cluster allow Prow's service account to list pods on it."
}

and

jsonPayload: {
error: "Get "https://35.226.143.163/api?timeout=32s": dial tcp 35.226.143.163:443: i/o timeout"
level: "error"
msg: "Failed to get API Group-Resources"
stacktrace: "sigs.k8s.io/controller-runtime/pkg/cluster.New
	sigs.k8s.io/controller-runtime@v0.12.3/pkg/cluster/cluster.go:160
sigs.k8s.io/controller-runtime/pkg/manager.New
	sigs.k8s.io/controller-runtime@v0.12.3/pkg/manager/manager.go:322
k8s.io/test-infra/prow/flagutil.(*KubernetesOptions).BuildClusterManagers.func2
	k8s.io/test-infra/prow/flagutil/kubernetes_cluster_clients.go:422"
ts: 1689367353.4081764
}

knative-prow-trusted used to be referenced in the repo but doesn't seem to be anymore; I think we can clean it up and clean up these errors along with it.