add cf task metrics

[risicle]

Hi,

We noticed cf_exporter was missing metrics for cf tasks. This information would have proved useful when we had a tenant spinning up nightly-but-never-ending tasks using ever accumulating amounts of resources.

Firstly this required a bump to the vendored go-cfclient to include the Relationships field in Task structs, added in cloudfoundry/go-cfclient#309

Secondly note how this collector is a little different from others because it performs some amount of aggregation up-front. This is because all potential labels that would allow identification of individual tasks are likely to be high cardinality ephemeral identifiers which could lead to the dreaded "cardinality explosion".

Of the information we can access about a task, the guid is unique for each task launched - an hourly task running 24/7 would produce a lot of these. sequence_id on its own is low-ish cardinality but when multiplied against the application guids would cause similar problems. A new droplet_guid is generated every time an application is re-staged (correct me?), and so a team doing a release a day would again start to generate a lot of unique combinations. Though a task's name could be heavily reused, in practise I suspect most users fall back to the default, which is randomly generated.

So that just leaves us with application_id and state as labels.

This uses the v3 task interface, so this is hidden behind the "v3 enabled" flag much like IsolationSegmentsCollector.

[ArthurHlt]

Hello, those mettrics are available in firehose_exporter, for the use case you mentioned (when you have task never ending), you can use metric firehose_value_metric_cc_tasks_running_count for example.

you can also get the drop-in replacement (more close too a drop-in) of firehose_exporter in PR here cloudfoundry/firehose_exporter#63 if you have a big cluster (we use it in production)

[risicle]

I may be missing something, but I don't think those firehose metrics would have allowed us to break it down by application, and thus ultimately discover its owner.

[risicle]

☝️ Have added another metric, the created_at of the oldest task per group. This allows long-running tasks to be spotted while still avoiding the high-cardinality labels that would be needed to identify individual tasks.

[risicle]

Have rebased following the merge of #57.

Most of what I've written above is still applicable.

This now provisionally depends on cloudfoundry/cli#2343 to allow me to make calls to the /tasks endpoint using MakeListRequest. As a temporary measure to ease review I've manually hacked the change in to the vendored dependency but would need to bump our cli dependency to a version that included that before merging.

[psycofdj]

thanks @risicle for this very usefull contribution !

[a-b]

LGTM

[risicle]

I think I've thoroughly failed to get cloudfoundry/cli#2343 merged, so if anyone ever wants to see this functionality in cf exporter they're probably going to have to take on the task (pun intended) of getting that merged themselves.

[github-actions]

This PR is stale because it has been open 60 days with no activity. Comment or this will be closed in 5 days.

[github-actions]

This PR was automatically closed because it has been stalled for 5 days with no activity.

[benjaminguttmann-avtq]

@risicle it happened, your CLI PR is merged :D

[risicle]

I noticed - I'm still in shock.

Have to rebase this now...

[risicle]

Tested again running as a cf app, works fine.

outdated, requested review from gmlt and/or mdimiceli

[mdimiceli]

Looks good to me, thanks @risicle !

[gmllt]

Good for me, thanks @risicle for the contribution

[risicle]

🎂 Happy anniversary! 🎂