[Fleet] Use cache-control headers on EPM get routes

[joshdover]

I experimented with adding a cache-control: max-age=600 (ten minutes) header to some of our EPM routes used to power the Integration UI and found it made exploring our integrations much snappier. However there are some adjustments we'd need to make to make this an acceptable solution and not serve the user stale data. For example, any request that relies on fetching Saved Objects that are mutable by the user should not be cached.

Implementation checklist

• Add cache-control: max-age=600 header to GET /api/fleet/epm/categories API endpoint (source)
• Add cache-control: max-age=600 header to GET /api/fleet/epm/packages API endpoint (source)
  • Add support for query parameter includeInstallStatus to /api/fleet/epm/packages API endpoint that defaults to false
    • When set to true, the installation status for the given packages is included in the response, when set to false (default), the installation status is excluded from the response
    • Update Open API docs to document this query parameter
• Add cache-control: max-age=600 header to GET /api/fleet/epm/packages/<name>/<version>/<file path> API endpoint (source)

In scope

• Fetch categories GET /api/fleet/epm/categories no issues, fully cacheable ✅
• Fetch package list GET /api/fleet/epm/packages minor issue, likely cacheable ✅
  • We current append the installed / not installed status to these objects which we will want to ensure is always up to date. This is used to power the "Installed integrations" tab.
  • We could move installed status to a separate endpoint or exclude it when query parameter is supplied to make the main use case for the UI cacheable.
  • This is likely the biggest gain for the overall user experience of the Integrations app. Navigating back to the grid from a details page results in running this request very frequently.
• Fetch package file GET /api/fleet/epm/packages/<name>/<version>/<file path> no issues, fully cacheable ✅
  • This fetches immutable assets from Saved Objects which never change for a given package version

Deferred

• Fetch single package GET /api/fleet/epm/packages/<name>/<version> more problems, likely cacheable ⚠️
  • We fetch the installed package from Saved Objects (if present) to provide install state and keep_policies_up_to_date setting
  • I believe this info is only needed for the "Settings" tab of the integration detail page and not the main tab. Could we exclude this information when a specific query parameter is supplied and fetch it separately when needed?

Related to #118751

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)