[MetricBeat] [AWS] events can't set TargetGroup tags

[kwinstonix]

For confirmed bugs, please report:

• Version: 7.8.1

• Steps to Reproduce:

  • set aws ALB/NLB targetgroup tags
  • run metricbeat with this example config

metricbeat.modules:
  - module: aws
    aws_partition: aws
    period: 300s
    access_key_id:  'your aws key'
    secret_access_key: 'your aws key'
    metricsets:
      - cloudwatch
    metrics:
      - namespace: AWS/ApplicationELB
        statistic: ['Minimum']
        name: ['HealthyHostCount']
        tags.resource_type_filter: elasticloadbalancing

• current result:

events only contain Loadbalancer tags

• expected result:

events contain both LoadBalancer and TargetGroup tags

---

aws alb and nlb metrics structure

// CLI output ： aws resourcegroupstaggingapi get-resources --resource-type-filters elasticloadbalancing
        {
            "Namespace": "AWS/ApplicationELB",
            "MetricName": "UnHealthyHostCount",
            "Dimensions": [
                {
                    "Name": "TargetGroup",
                    "Value": "targetgroup/tg-name/12345678abc"
                },
                {
                    "Name": "LoadBalancer",
                    "Value": "app/lb-name/12345678abc"
                }
            ]
        },

        {
            "Namespace": "AWS/NetworkELB",
            "MetricName": "HealthyHostCount",
            "Dimensions": [
                {
                    "Name": "TargetGroup",
                    "Value": "targetgroup/tg-name/012345678abcdef"
                },
                {
                    "Name": "LoadBalancer",
                    "Value": "net/lb-name/012345678abcdef"
                }
            ]
        },

• aws resourcetagginggroup result examle

{
    "ResourceARN": "arn:aws:elasticloadbalancing:ap-northeast-2:123456789012:loadbalancer/app/alb-name/12345678abc",
        "Tags": [
        {
            "Key": "K1",
            "Value": "V1"
        },
}
{
    "ResourceARN": "arn:aws:elasticloadbalancing:ap-northeast-2:123456789012:loadbalancer/net/nlb-name/2233445566",
        "Tags": [
        {
            "Key": "K2",
            "Value": "V2"
        },
}

{
    "ResourceARN": "arn:aws:elasticloadbalancing:ap-northeast-2:123456789012:targetgroup/tg-name/012345678abcdef"
        "Tags": [
        {
            "Key": "K3",
            "Value": "V3"
        },
}

comparing above two example results, we can figure out that ALB and NLB LoadBalancer dimension value is not same with resource id field of ARN, and TargetGroup dimension value is same with resource id field of ARN.

• the reason

GetResourcesTags func has proccessed the resourcegroupstaggingapi result, the resourceTagMap is like this, so we can not get TargetGroup tags in insertTags function .

{
   "tg-name/012345678abcdef":   {
            "Key": "K3",
            "Value": "V3"
        },
  "net/nlb-name/2233445566" :  {
            "Key": "K2",
            "Value": "V2"
   }
}

beats/x-pack/metricbeat/module/aws/utils.go

Lines 189 to 197 in ea63f05

	for _, resourceTag := range output.ResourceTagMappingList {
	identifier, err := FindIdentifierFromARN(*resourceTag.ResourceARN)
	if err != nil {
	err = errors.Wrap(err, "error FindIdentifierFromARN")
	return nil, err
	}
	resourceTagMap[identifier] = resourceTag.Tags
	}

• suggestion

I think we should use both short resource identifier and whole resource identifier of ARN, just like this

{
   "tg-name/012345678abcdef":   {
            "Key": "K3",
            "Value": "V3"
        },
  "net/nlb-name/2233445566" :  {
            "Key": "K2",
            "Value": "V2"
   },

   "targetgroup/tg-name/012345678abcdef":   {
            "Key": "K3",
            "Value": "V3"
        },
  "loadbalancer/net/nlb-name/2233445566" :  {
            "Key": "K2",
            "Value": "V2"
   }
}

[elasticmachine]

Pinging @elastic/integrations-platforms (Team:Platforms)

[kaiyan-sheng]

@kwinstonix Thanks for creating this issue. Good point, it would be good to add collection for TargetGroup tags. I won't be able to start working on this for a while because of other priorities on my list 😬 Please feel free to take over!

[kwinstonix]

OK, I will open a PR

[kaiyan-sheng]

@kwinstonix Awesome!! Thank you!! Just ping me for the PR and I will test it 😄