Multiple kOps clustes in one OpenStack project breaks validation

[Wieneo]

/kind bug

1. What kops version are you running? The command kops version, will display
this information.
Client version: 1.28.1

2. What Kubernetes version are you running? kubectl version will print the
version if a cluster is running or provide the Kubernetes version specified as
a kops flag.
Server Version: v1.26.10

3. What cloud provider are you using?
OpenStack

4. What commands did you run? What is the simplest way to reproduce this issue?
kops update cluster --name ${CLUSTER} --yes --admin
kops validate cluster --wait 15m --name ${CLUSTER}

5. What happened after the commands executed?
Rollout got stuck with the following error:

Machine	4cea4af0-28de-4372-ba89-5a7160318e71	machine "4cea4af0-28de-4372-ba89-5a7160318e71" has not yet joined cluster
Machine	66bab00e-c0d4-43a5-bbdd-6e4f26c60cfd	machine "66bab00e-c0d4-43a5-bbdd-6e4f26c60cfd" has not yet joined cluster
Machine	6fd5b072-16e6-4011-afe5-f0375d3f274d	machine "6fd5b072-16e6-4011-afe5-f0375d3f274d" has not yet joined cluster
Machine	88d9bff5-412d-420e-b55e-ab196ce25ee2	machine "88d9bff5-412d-420e-b55e-ab196ce25ee2" has not yet joined cluster
Machine	96f254df-8cfa-4f8f-91fe-66cb898cfea2	machine "96f254df-8cfa-4f8f-91fe-66cb898cfea2" has not yet joined cluster
Machine	979f7c28-52ff-4e2e-87c2-e13629337fb5	machine "979f7c28-52ff-4e2e-87c2-e13629337fb5" has not yet joined cluster
Machine	a6270167-e960-459a-802e-f51fced38305	machine "a6270167-e960-459a-802e-f51fced38305" has not yet joined cluster
Machine	a6fe5623-2865-4f89-aaf0-29b7eb6a258a	machine "a6fe5623-2865-4f89-aaf0-29b7eb6a258a" has not yet joined cluster
Machine	c0d67b64-1324-4f7a-9c02-fbfa45ecfa12	machine "c0d67b64-1324-4f7a-9c02-fbfa45ecfa12" has not yet joined cluster
Machine	d6389009-b72b-4f32-8ca2-7980cdd7fcb7	machine "d6389009-b72b-4f32-8ca2-7980cdd7fcb7" has not yet joined cluster
Machine	dd5ed215-a656-4cfe-9cfd-f1cbe41cb59a	machine "dd5ed215-a656-4cfe-9cfd-f1cbe41cb59a" has not yet joined cluster

6. What did you expect to happen?
Validation would succeed

7. Anything else do we need to know?
This bug only occurs if multiple kOps Clusters are running in one OpenStack project.
#16022 seems to break the separation of instances between kOps clusters, causing kops validate to think instances from other clusters need to join the current cluster.
This bug also occurs without the serverGroupName annotation being set.

Downgrading to 1.28.0 fixes the issue.

[hakman]

FYI @zetaab

[zetaab]

could you please provide some more information like: are you using similarly named instance groups in two different kops clusters in same project? When I am reading code that can be one issue and I do not have (yet) good idea to solve that.

could you run kops get ig to all clusters in same project

[zetaab]

@Wieneo forgot to ping ^