Clean up knative-trusted-prow from config

Seeing frequent errors in the oss-prow logs like: ``` jsonPayload: { component: "prow-controller-manager" error: "failed to construct manager for cluster knative-prow-trusted: Get "https://35.226.143.163/api?timeout=32s": dial tcp 35.226.143.163:443: i/o timeout" file: "k8s.io/test-infra/prow/cmd/prow-controller-manager/main.go:189" func: "main.main" level: "error" msg: "Failed to construct build cluster managers. Please check that the kubeconfig secrets are correct, and that RBAC roles on the build cluster allow Prow's service account to list pods on it." } ``` and ``` jsonPayload: { error: "Get "https://35.226.143.163/api?timeout=32s": dial tcp 35.226.143.163:443: i/o timeout" level: "error" msg: "Failed to get API Group-Resources" stacktrace: "sigs.k8s.io/controller-runtime/pkg/cluster.New sigs.k8s.io/controller-runtime@v0.12.3/pkg/cluster/cluster.go:160 sigs.k8s.io/controller-runtime/pkg/manager.New sigs.k8s.io/controller-runtime@v0.12.3/pkg/manager/manager.go:322 k8s.io/test-infra/prow/flagutil.(*KubernetesOptions).BuildClusterManagers.func2 k8s.io/test-infra/prow/flagutil/kubernetes_cluster_clients.go:422" ts: 1689367353.4081764 } ``` knative-prow-trusted used to be referenced in the repo but doesn't seem to be anymore; I think we can clean it up and clean up these errors along with it.