Configuring Knative to use a local gateway in another namespace

[ca-scribner]

I am trying to use Knative Serving via the Knative Operator to make ksvcs available via a local gateway (not a service mesh). The Gateway I have is shared with other applications, and the workload pods for that gateway are in a separate namespace. I'm having trouble getting the local side of the gateway to work properly, specifically with the istio-proxy deployment deployed in a different namespace from serving/any ksvc. I wonder if there's a missing/misconfigured service that the net istio controller needs to generate.

I have:

• deployed istio in namespace-1, including
  • an image: istio/proxy deployment working as a gateway deployed with selector istio: ingressgateway
  • an external Gateway pointing to the above proxy called my-gateway
  • a svc called istio-ingressgateway-workload that points at the above proxy via a selector
• deployed the Knative Operator
• created a KnativeServing object in namespace/knative-serving using:

apiVersion: v1
kind: Namespace
metadata:
  name: knative-serving
---
apiVersion: v1
kind: Namespace
metadata:
  name: knative-serving
---
apiVersion: operator.knative.dev/v1beta1
kind: KnativeServing
metadata:
  name: knative-serving-app
  namespace: knative-serving
spec:
  version: "1.7.1"
  config:
    istio:
      gateway.namespace-1.my-gateway: "istio-ingressgateway-workload.namespace-1.svc.cluster.local"
      local-gateway.knative-serving.knative-local-gateway: "knative-local-gateway.knative-serving.svc.cluster.local"
    domain:
      10.64.140.43.nip.io: ""

Once I have these, I deploy a cloudevents player with:

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: cloudevents-player-1
spec:
  template:
    metadata:
      annotations:
        autoscaling.knative.dev/minScale: "1"
    spec:
      containers:
        - image: quay.io/ruben/cloudevents-player:0.2

Once up, I can connect to the ksvc through the external gateway no problem using:

curl -v http://cloudevents-player-1.namespace-1.10.64.140.43.nip.io -H "Content-Type: application/json" -H "Ce-Id: foo" -H "Ce-Specversion: 1.0" -H "Ce-Type: dev.example.events" -H "Ce-Source: curl-source" -d '{"msg":"Can you hear me?"}'

which yields a 202 Accepted. However, if I try the local ingress, I get connection refused:

curl -v http://cloudevents-player-1.namespace-1.svc.cluster.local -H "Content-Type: application/json" -H "Ce-Id: foo" -H "Ce-Specversion: 1.0" -H "Ce-Type: dev.example.events" -H "Ce-Source: curl-source" -d '{"msg":"Can you hear me?"}'

which yields a Failed to connect ... port 80: Connection refused.

I think this is because the svc/knative-local-gateway which is generated by the istio controller tries to point to the proxy via a selector, but the proxy is in a different namespace. We see the service (truncated a bit):

apiVersion: v1
kind: Service
metadata:
  labels:
  name: knative-local-gateway
  namespace: knative-serving
spec:
  ports:
  - name: http2
    port: 80
    protocol: TCP
    targetPort: 8081
  selector:
    istio: ingressgateway
  type: ClusterIP

which is in namespace/knative-serving and tries to select pods by selector: {istio: ingressgateway}. Problem is that there is no istio: ingressgateway in namespace/knative-serving, it is instead deployed in namespace/namespace-1.

Am I configuring this incorrectly? I don't see any other settings that might change this. Maybe the intent is for me to point the local gateway at my own service, but then I don't understand why knative would create a svc/knative-local-gateway at all. The existing svc/istio-ingressgateway-workload that I have would be unsuitable as it is forwarding port 80 to 8080 (eg: the external route through the gateway). I'd need another svc to pass to port 8081 (like the svc/knative-local-gateway does).

If everything is configured correctly, I think that svc/knative-local-gateway needs to either be created in the namespace of the istio proxy pod, or be an ExternalName svc that resolves to a service in the istio proxy pod's namespace and points at the istio proxy pod.

[ca-scribner]

Clarifying one thing, by default Istio's Gateway objects do work across namespaces (Gateway in namespace-1 can refer by selector to a deployment of image: istio/proxy in namespace-2), so the Gateway objects themselves I think are fine above. The problem is the service that serves as the entry way into the cluster-local gateway - that service cannot bridge the namespace boundaries by itself.

[ca-scribner]

After some more tweaking, I found that setting:

local-gateway.knative-serving.knative-local-gateway: "knative-local-gateway.namespace-1.svc.cluster.local"

resulted in:

• creation of gateway/knative-ingress-gateway and gateway/knative-local-gateway. These seem to always be created, regardless of whether they're cited in the config, and control pods named by selector: {istio: ingressgateway}. They open ports 80 and 8081, respectively.
• serving using the gateway/knative-local-gateway from the previous bullet, because it is cited in the config
• the creation of svc/knative-local-gateway in namespace/namespace-1. This service forwards traffic to selector: {istio: ingressgateway} port 8081. Since this svc is located in the namespace of the gateway pods it points to, it actually works out of the box as I hoped

Is this the expected behaviour? Maybe this is mainly a documentation problem, as it didn't feel clear this would happen and I only found it by accident.

Oddly, if I change the config to:

local-gateway.namespace-1.knative-local-gateway: "knative-local-gateway.namespace-1.svc.cluster.local"

it then creates the svc/knative-local-gateway in the namespace/istio-system, which seems to be a recurrance of knative/operator#565. Maybe what I'm describing is in the domain of the operator rather than net-istio? I'm not sure

[houshengbo]

@nak3 Could you take a look at this issue? Coz I am not sure if this is a configuration issue for operator, or something we need to support in ingress or serving.

[houshengbo]

@ca-scribner I have fixed the issue of #1216. I am not sure if that would help with this issue.

You said "by default Istio's Gateway objects do work across namespaces". That's fine.
If you use Istio's Gateway objects, which are non-default, what is difference from the default ones???
Did you change the selector? Because #1216 reports the issue, regarding service's selector is not changed, even if gateway's selector is changed.

[github-actions]

This issue is stale because it has been open for 90 days with no
activity. It will automatically close after 30 more days of
inactivity. Reopen the issue with /reopen. Mark the issue as
fresh by adding the comment /remove-lifecycle stale.

[knative-prow-robot]

This issue or pull request is stale because it has been open for 90 days with no activity.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close

/lifecycle stale

[dprotaso]

FWIW the knative-local-gateway is defined here as part of install net-istio

https://github.com/knative-sandbox/net-istio/blob/3df122bf9b0881092d8a58c814a163651cae2dea/config/203-local-gateway.yaml#L18

If you have your own internal gateway I think all you would want to do is configure this config map property

https://github.com/knative-sandbox/net-istio/blob/3df122bf9b0881092d8a58c814a163651cae2dea/config/400-config-istio.yaml#L65