CNI commands aren't executed

[deric]

With flannel CNI plugin:

Error: /Stage[main]/Kubernetes::Kube_addons/Exec[Install cni network provider]: Could not evaluate: Could not find command 'get'
Error: /Stage[main]/Kubernetes::Kube_addons/Exec[Install Kubernetes dashboard]: Could not evaluate: Could not find command 'kubectl get pods --field-selector="status.phase=Running" -n kubernetes-dashboard | grep kubernetes-dashboard-'

The issue has been probably introduced in hardening efforts #592, cc @LukasAud

Instead of executing whole command:

kubectl -n kube-system get daemonset | egrep '(flannel|weave|calico-node|cilium)'

only kubectl alone is executed which returns help message.

here's debug output

Debug: Exec[Install cni network provider](provider=posix): Executing check 'kubectl -n kube-system get daemonset | egrep '(flannel|weave|calico-node|cilium)''
Debug: Executing: 'kubectl -n kube-system get daemonset | egrep '(flannel|weave|calico-node|cilium)''
Debug: Exec[Install cni network provider](provider=posix): Executing check 'kubectl'
Debug: Executing: 'kubectl'
Debug: /Stage[main]/Kubernetes::Kube_addons/Exec[Install cni network provider]/onlyif: kubectl controls the Kubernetes cluster manager.
Debug: /Stage[main]/Kubernetes::Kube_addons/Exec[Install cni network provider]/onlyif: 
Debug: /Stage[main]/Kubernetes::Kube_addons/Exec[Install cni network provider]/onlyif:  Find more information at: https://kubernetes.io/docs/reference/kubectl/overview/
Debug: /Stage[main]/Kubernetes::Kube_addons/Exec[Install cni network provider]/onlyif: 
Debug: /Stage[main]/Kubernetes::Kube_addons/Exec[Install cni network provider]/onlyif: Basic Commands (Beginner):
Debug: /Stage[main]/Kubernetes::Kube_addons/Exec[Install cni network provider]/onlyif:   create          Create a resource from a file or from stdin
Debug: /Stage[main]/Kubernetes::Kube_addons/Exec[Install cni network provider]/onlyif:   expose          Take a replication controller, service, deployment or pod and expose it as a new Kubernetes service
Debug: /Stage[main]/Kubernetes::Kube_addons/Exec[Install cni network provider]/onlyif:   run             Run a particular image on the cluster
Debug: /Stage[main]/Kubernetes::Kube_addons/Exec[Install cni network provider]/onlyif:   set             Set specific features on objects
Debug: /Stage[main]/Kubernetes::Kube_addons/Exec[Install cni network provider]/onlyif: 
Debug: /Stage[main]/Kubernetes::Kube_addons/Exec[Install cni network provider]/onlyif: Basic Commands (Intermediate):
Debug: /Stage[main]/Kubernetes::Kube_addons/Exec[Install cni network provider]/onlyif:   explain         Get documentation for a resource
Debug: /Stage[main]/Kubernetes::Kube_addons/Exec[Install cni network provider]/onlyif:   get             Display one or many resources
Debug: /Stage[main]/Kubernetes::Kube_addons/Exec[Install cni network provider]/onlyif:   edit            Edit a resource on the server
Debug: /Stage[main]/Kubernetes::Kube_addons/Exec[Install cni network provider]/onlyif:   delete          Delete resources by file names, stdin, resources and names, or by resources and label selector
Debug: /Stage[main]/Kubernetes::Kube_addons/Exec[Install cni network provider]/onlyif: 
Debug: /Stage[main]/Kubernetes::Kube_addons/Exec[Install cni network provider]/onlyif: Deploy Commands:
Debug: /Stage[main]/Kubernetes::Kube_addons/Exec[Install cni network provider]/onlyif:   rollout         Manage the rollout of a resource
Debug: /Stage[main]/Kubernetes::Kube_addons/Exec[Install cni network provider]/onlyif:   scale           Set a new size for a deployment, replica set, or replication controller
Debug: /Stage[main]/Kubernetes::Kube_addons/Exec[Install cni network provider]/onlyif:   autoscale       Auto-scale a deployment, replica set, stateful set, or replication controller
Debug: /Stage[main]/Kubernetes::Kube_addons/Exec[Install cni network provider]/onlyif: 
Debug: /Stage[main]/Kubernetes::Kube_addons/Exec[Install cni network provider]/onlyif: Cluster Management Commands:
Debug: /Stage[main]/Kubernetes::Kube_addons/Exec[Install cni network provider]/onlyif:   certificate     Modify certificate resources.
Debug: /Stage[main]/Kubernetes::Kube_addons/Exec[Install cni network provider]/onlyif:   cluster-info    Display cluster information
Debug: /Stage[main]/Kubernetes::Kube_addons/Exec[Install cni network provider]/onlyif:   top             Display resource (CPU/memory) usage
Debug: /Stage[main]/Kubernetes::Kube_addons/Exec[Install cni network provider]/onlyif:   cordon          Mark node as unschedulable
Debug: /Stage[main]/Kubernetes::Kube_addons/Exec[Install cni network provider]/onlyif:   uncordon        Mark node as schedulable
Debug: /Stage[main]/Kubernetes::Kube_addons/Exec[Install cni network provider]/onlyif:   drain           Drain node in preparation for maintenance
Debug: /Stage[main]/Kubernetes::Kube_addons/Exec[Install cni network provider]/onlyif:   taint           Update the taints on one or more nodes
Debug: /Stage[main]/Kubernetes::Kube_addons/Exec[Install cni network provider]/onlyif: 
Debug: /Stage[main]/Kubernetes::Kube_addons/Exec[Install cni network provider]/onlyif: Troubleshooting and Debugging Commands:
Debug: /Stage[main]/Kubernetes::Kube_addons/Exec[Install cni network provider]/onlyif:   describe        Show details of a specific resource or group of resources
Debug: /Stage[main]/Kubernetes::Kube_addons/Exec[Install cni network provider]/onlyif:   logs            Print the logs for a container in a pod
Debug: /Stage[main]/Kubernetes::Kube_addons/Exec[Install cni network provider]/onlyif:   attach          Attach to a running container
Debug: /Stage[main]/Kubernetes::Kube_addons/Exec[Install cni network provider]/onlyif:   exec            Execute a command in a container
Debug: /Stage[main]/Kubernetes::Kube_addons/Exec[Install cni network provider]/onlyif:   port-forward    Forward one or more local ports to a pod
Debug: /Stage[main]/Kubernetes::Kube_addons/Exec[Install cni network provider]/onlyif:   proxy           Run a proxy to the Kubernetes API server
Debug: /Stage[main]/Kubernetes::Kube_addons/Exec[Install cni network provider]/onlyif:   cp              Copy files and directories to and from containers
Debug: /Stage[main]/Kubernetes::Kube_addons/Exec[Install cni network provider]/onlyif:   auth            Inspect authorization
Debug: /Stage[main]/Kubernetes::Kube_addons/Exec[Install cni network provider]/onlyif:   debug           Create debugging sessions for troubleshooting workloads and nodes
Debug: /Stage[main]/Kubernetes::Kube_addons/Exec[Install cni network provider]/onlyif: 
Debug: /Stage[main]/Kubernetes::Kube_addons/Exec[Install cni network provider]/onlyif: Advanced Commands:
Debug: /Stage[main]/Kubernetes::Kube_addons/Exec[Install cni network provider]/onlyif:   diff            Diff the live version against a would-be applied version
Debug: /Stage[main]/Kubernetes::Kube_addons/Exec[Install cni network provider]/onlyif:   apply           Apply a configuration to a resource by file name or stdin
Debug: /Stage[main]/Kubernetes::Kube_addons/Exec[Install cni network provider]/onlyif:   patch           Update fields of a resource
Debug: /Stage[main]/Kubernetes::Kube_addons/Exec[Install cni network provider]/onlyif:   replace         Replace a resource by file name or stdin
Debug: /Stage[main]/Kubernetes::Kube_addons/Exec[Install cni network provider]/onlyif:   wait            Experimental: Wait for a specific condition on one or many resources
Debug: /Stage[main]/Kubernetes::Kube_addons/Exec[Install cni network provider]/onlyif:   kustomize       Build a kustomization target from a directory or URL.
Debug: /Stage[main]/Kubernetes::Kube_addons/Exec[Install cni network provider]/onlyif: 
Debug: /Stage[main]/Kubernetes::Kube_addons/Exec[Install cni network provider]/onlyif: Settings Commands:
Debug: /Stage[main]/Kubernetes::Kube_addons/Exec[Install cni network provider]/onlyif:   label           Update the labels on a resource
Debug: /Stage[main]/Kubernetes::Kube_addons/Exec[Install cni network provider]/onlyif:   annotate        Update the annotations on a resource
Debug: /Stage[main]/Kubernetes::Kube_addons/Exec[Install cni network provider]/onlyif:   completion      Output shell completion code for the specified shell (bash, zsh or fish)
Debug: /Stage[main]/Kubernetes::Kube_addons/Exec[Install cni network provider]/onlyif: 
Debug: /Stage[main]/Kubernetes::Kube_addons/Exec[Install cni network provider]/onlyif: Other Commands:
Debug: /Stage[main]/Kubernetes::Kube_addons/Exec[Install cni network provider]/onlyif:   alpha           Commands for features in alpha
Debug: /Stage[main]/Kubernetes::Kube_addons/Exec[Install cni network provider]/onlyif:   api-resources   Print the supported API resources on the server
Debug: /Stage[main]/Kubernetes::Kube_addons/Exec[Install cni network provider]/onlyif:   api-versions    Print the supported API versions on the server, in the form of "group/version"
Debug: /Stage[main]/Kubernetes::Kube_addons/Exec[Install cni network provider]/onlyif:   config          Modify kubeconfig files
Debug: /Stage[main]/Kubernetes::Kube_addons/Exec[Install cni network provider]/onlyif:   plugin          Provides utilities for interacting with plugins
Debug: /Stage[main]/Kubernetes::Kube_addons/Exec[Install cni network provider]/onlyif:   version         Print the client and server version information
Debug: /Stage[main]/Kubernetes::Kube_addons/Exec[Install cni network provider]/onlyif: 
Debug: /Stage[main]/Kubernetes::Kube_addons/Exec[Install cni network provider]/onlyif: Usage:
Debug: /Stage[main]/Kubernetes::Kube_addons/Exec[Install cni network provider]/onlyif:   kubectl [flags] [options]
Debug: /Stage[main]/Kubernetes::Kube_addons/Exec[Install cni network provider]/onlyif: 
Debug: /Stage[main]/Kubernetes::Kube_addons/Exec[Install cni network provider]/onlyif: Use "kubectl <command> --help" for more information about a given command.
Debug: /Stage[main]/Kubernetes::Kube_addons/Exec[Install cni network provider]/onlyif: Use "kubectl options" for a list of global command-line options (applies to all commands).
Error: /Stage[main]/Kubernetes::Kube_addons/Exec[Install cni network provider]: Could not evaluate: Could not find command 'get'
Error: /Stage[main]/Kubernetes::Kube_addons/Exec[Install Kubernetes dashboard]: Could not evaluate: Could not find command 'kubectl get pods --field-selector="status.phase=Running" -n kubernetes-dashboard | grep kubernetes-dashboard-'

Environment

• Kubernetes 1.24.9
• Debian 11
• puppetlabs-kubernetes from main branch, ref. 7238ba0
• puppet-agent 7.21.0

[deric]

I guess the problem is here:

      exec { 'Install cni network provider':
        command     => $provider_command,
        onlyif      => ['kubectl', 'get', 'nodes'],
        unless      => $provider_unless,
        environment => $env,
      }

['kubectl', 'get', 'nodes'] doesn't give the same output as ['kubectl get nodes']

      exec { 'Install cni network provider':
        command     => $provider_command,
        onlyif      => ['kubectl get nodes'],
        unless      => $provider_unless,
        environment => $env,
      }

I don't see the security benefits in this change. There's no string interpolation happening on this line.

[deric]

@chelnak This is related to hardening changes. Maybe we don't have to revert everything as done in here.

But it's definitely not covered by tests.

[chelnak]

That branch is very much WIP right now.. though I will be reverting some of the changes introduced by the hardening efforts. In some cases potential issues had already been addressed so there is no value in altering these.

Also thanks for raising awareness 👍