Fleet Test Flag Cannot Find Match

[atsai1220]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

We are attempting to utilize the fleet CLI test command to validate our fleet.yaml and generated content in our pipeline. Currently, at v0.5.3, running the fleet test command against a fleet.yaml that has one item under targetCustomizations will fail unless we specify the name of the one target. I would expect to match on "default" if no targets are passed or if "default" is specifically targetted.

Expected Behavior

I expect the default target to be selected or have the ability to select the default target if there are items under targetCustomizations.

Steps To Reproduce

1. Download the fleet CLI at v0.5.3.

wget https://github.com/rancher/fleet/releases/download/v0.5.3/fleet-linux-amd64
chmod +x fleet-linux-amd64

2. Save this fleet.yaml in the same directory as fleet-linux-amd64

defaultNamespace: opencost
name: opencost-helm
helm:
  chart: opencost
  repo: https://opencost.github.io/opencost-helm-chart
  values:
    foo: bar
targetCustomizations:
- name: openstack
  clusterSelector:
    matchLabels:
      cloud-provider: openstack
  kustomize:
    dir: overlays/openstack

3. Run against the above fleet.yaml without a target or with default as target

$ ./fleet-linux-amd64 test -f ./fleet.yaml -t default -q
FATA[0005] no match found

$ ./fleet-linux-amd64 test -f ./fleet.yaml -q
FATA[0005] no match found

4. Success with target on the item under targetCustomizations

$ ./fleet-linux-amd64 test -f ./fleet.yaml -t openstack -q
# Matched: openstack

Environment

- Architecture: amd64
- Fleet Version: v0.5.3
- Cluster:
  - Provider:
  - Options:
  - Kubernetes Version:

Logs

No response

Anything else?

Thank you! Much appreciate all the community engagement we have.

[manno]

Indeed, the test command can only match on clustername/group/labels (-n, -g, -l, -L) or target (-t).

In the first mode, if no argument is passed, it uses "default" as a cluster group.

However, 'fleet test' doesn't look at actual clusters. Furthermore, when called with -f, it doesn't use the bundlereader, so the default target is not added. Actually a lot of processing is missing.

You can load a full bundle with -b, which gives you access to Spec.Target and Spec.TargetRestrictions and might result in better tests.

In general, I'm still not sure what fleet test is good for. It combines code from 'fleet apply' with 'fleet controller and is apparently undocumented.
If testing targeting and bundle rendering is important to users, we should rethink this tool.

[manno]

Also see #527

[strowi]

With 0.7.0 it still seems like if a targetCustomization is defined, it is not possible to test against the default.

[skaven81]

If testing targeting and bundle rendering is important to users, we should rethink this tool.

Yeah this is super important, and a glaring gap in Fleet today. The fact that one has to commit changes to a Bundle and then actually attempt to deploy it to a test cluster, just to validate that changes to fleet.yaml are working as expected, is not great.

As OP noted, we really need a comprehensive way of rendering the output of a Fleet Bundle for a particular cluster, when in an isolated test environment (or e.g. on your workstation). This would then allow us to write tests around that, to ensure that targetCustomizations are being matched as expected, and that the resulting resources contain the desired changes. It also would let us test the full Kustomize/Helm rendering pipeline and not just "hope" that it works once deployed.

[manno]

I'm experimenting with two new fleet CLI sub-commands: "target" and "deploy".
Target reads a bundle from a file and works with a live cluster to print out the bundledeployment
/content resource, which fleetcontroller would create. It takes a namespace as an argument, so it can look in that namespace for e.g. cluster resources. It can also dump the data structure which is used during "targeting", so decisions taken regarding labels and cluster names can be checked.

The second command "deploy" takes the output of target, or a dumped bundledeployment/content resource and deploys it to a cluster, just like fleet-agent would. It supports a dry run mode, to print out the resources which would be created, instead of installing them with helm. Since the command doesn't create the input resources, a running fleet-agent would likely garbage collect the deployment. I'm still investigating how to deal with manually created deployments. I think adopt or ignore are the most useful options.

Both commands are currently prototypes, but they should behave like the controllers do.

[manno]

fleet target and fleet deploy --dry-run have been merged and will be in 2.9.1