[8.x](backport #6381) [k8s] embed hints inputs in agent container image

[mergify]

What does this PR do?

This PR embeds the hints-based autodiscovery templates directly into the Elastic Agent container image. These templates, previously fetched by an init container, are now included in the image under /usr/share/elastic-agent/hints.inputs.d. The Elastic Agent has been updated to optionally use these embedded templates based on the Kubernetes hints configuration in the user's settings. If hints-based autodiscovery is enabled, the Elastic Agent dynamically reads from the embedded templates directory, ensuring seamless integration without external dependencies.

Additionally:

• Introduced a new mechanism for discovering hints-based inputs dynamically from the embedded directory.
• Updated CI to package the templates into the container image during the build process.
• Modified the Helm chart to remove the dependency on an init container for downloading the templates.

Why is it important?

This change improves the robustness and efficiency of Elastic Agent deployment in Kubernetes environments by:

• Eliminating the need for an init container to fetch and extract the templates during pod initialisation.
• Reducing potential errors due to external network issues or offline environments.
• Accelerating the pod startup process by preloading the templates into the container image.
• Supporting environments with strict network policies or without external internet access.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in ./changelog/fragments using the changelog tool
• I have added an integration test or an E2E test

Disruptive User Impact

This change should not cause any disruption for users. For users who enable hints-based autodiscovery, the behavior remains consistent, with improved reliability. For users who do not enable this feature, the embedded templates will remain unused, ensuring no impact on performance or functionality.

How to test this PR locally

mage integration:auth
PLATFORMS=linux/arm64 EXTERNAL=true SNAPSHOT=true PACKAGES=docker mage -v package 
INSTANCE_PROVISIONER=kind STACK_PROVISIONER=stateful K8S_VERSION=v1.31.1 SNAPSHOT=true mage integration:kubernetes

Related issues

• Closes Investigate the copy of autodiscovery templates inside Elastic Agent image #5661

  ---

  This is an automatic backport of pull request [k8s] embed hints inputs in agent container image #6381 done by Mergify.

[mergify]

Cherry-pick of a631362 has failed:

On branch mergify/bp/8.x/pr-6381
Your branch is up to date with 'origin/8.x'.

You are currently cherry-picking commit a631362d5.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	new file:   changelog/fragments/1734512365-embed-hints-inputs-in-agent-container-image.yaml
	modified:   deploy/helm/elastic-agent/templates/integrations/_kubernetes/_kubernetes.tpl
	modified:   deploy/helm/elastic-agent/templates/integrations/_kubernetes/_kubernetes_hints.tpl
	modified:   deploy/helm/elastic-agent/templates/integrations/_kubernetes/_preset_pernode.tpl
	modified:   dev-tools/packaging/package_test.go
	modified:   dev-tools/packaging/packages.yml
	modified:   internal/pkg/agent/application/application.go
	modified:   internal/pkg/composable/providers/kubernetes/config.go
	new file:   internal/pkg/composable/providers/kubernetes/config_test.go
	modified:   internal/pkg/config/loader.go

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   deploy/helm/elastic-agent/examples/kubernetes-hints-autodiscover/rendered/manifest.yaml
	both modified:   deploy/helm/elastic-agent/examples/multiple-integrations/rendered/manifest.yaml

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

[elastic-sonarqube]

Quality Gate passed

Issues
1 New issue
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
43.8% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

[mergify]

This pull request has not been merged yet. Could you please review and merge it @pkoutsovasilis? 🙏