Bug/Help Needed: Private Endpoint Exposes only one NIC IP address

[mehighlow]

Describe the bug

Private Endpoint Integration for CosmosDB exposes only one NIC IP address. However, a Private Endpoint for CosmosDB might create more than one NIC + IP address.

Automation can be leveraged to create just one DNS record for PrimaryNicPrivateIpAddress:

│   ├── OperatorSpec: *Object (3 properties)
│   │   ├── ConfigMapExpressions: *core.DestinationExpression[]
│   │   ├── ConfigMaps: *Object (1 property)
│   │   │   └── PrimaryNicPrivateIpAddress: *genruntime.ConfigMapDestination

although both IP addresses remain valid for establishing connections:

The issue arises using CosmosDB client (Azure SDK for .NET). By default, client has the LimitToEndpointProperty set to false. This configuration instructs the SDK to automatically discover write and read regions and use them when the configured application region is unavailable. In my case, the application opted to use the regional FQDN, even though the endpoint connection was configured as global. This behavior can be addressed by setting the LimitToEndpointProperty to true and ConnectionMode set to Gateway. But this adjustment requires knowing what to look for.

Azure Service Operator Version: 2.9.0

Expected behavior

Both regional and global IP addresses are exposed, required FQDN records exposed.

To Reproduce

Deploy CosmosDB with Private Endpoint integration. Use the Azure SDK for .NET to establish a connection to the database with default settings. Observe that the client selects the regional endpoint, even though the supplied database FQDN is global.

Screenshots

N/A

Additional context

N/A

[matthchr]

Does the Status of the PrivateEndpoint contain the IPs that you want?

If so, can you use the new expressions feature to output all of the URLs/IPs you want to a configmap?

If Status does have the FQDNs you want, if you share its shape here I can help you to craft a CEL expression that will output the one(s) you want.

[theunrepentantgeek]

Any update @mehighlow ?

[mehighlow]

@matthchr , @theunrepentantgeek

status indeed contains all IP addresses and FQDNs.

  conditions:
  - lastTransitionTime: "2024-12-18T06:03:57Z"
    observedGeneration: 1
    reason: Succeeded
    status: "True"
    type: Ready
  customDnsConfigs:
  - fqdn: dbname.documents.azure.com
    ipAddresses:
    - 10.10.16.4
  - fqdn: dbname-westus2.documents.azure.com
    ipAddresses:
    - 10.10.16.5

I’ll create a PR once this is done. Thanks!

[matthchr]

I don't think you need to send us a PR, you should be able to do this purely in CEL expressions in your resource now.

Here's an example

So something like:

spec:
  operatorSpec:
    configMapExpressions:
    - name: my-configmap
      key: global
      value: self.status.customDnsConfigs.filter(config, matches(config.fqdn, "\w+.documents"))[0].ipAddresses[0]
    - name: my-configmap
      key: regional
      value: self.status.customDnsConfigs.filter(config, matches(config.fqdn, "\w+-\w+.documents"))[0].ipAddresses[0]

Note that you can avoid the filter + matches and just do self.status.customDnsConfigs[0] and self.status.customDnsConfigs[1] if you always know that there will be 2 and they will be in global followed by regional order.

If there may only be 1 (or 0), you'll need to guard against the filter returning nothing, which can be done like this (not beautiful I know, but it does the job):
size(self.status.customDnsConfigs.filter(config, matches(config.fqdn, "\\w+-\\w+.documents"))) == 0 ? "" : self.status.customDnsConfigs.filter(config, matches(config.fqdn, "\\w+-\\w+.documents"))[0].ipAddresses[0]