Prometheus Counter has 0 value in CW

[vkuzniet]

I'm emitting a counter metric to count the number of bytes fetched for the container layer. If I call the prometheus metrics endpoint, I get the following:

# HELP stargz_fs_layer_fetched_size_bytes Total fetched size of the layer
# TYPE stargz_fs_layer_fetched_size_bytes counter
stargz_fs_layer_fetched_size_bytes{digest="sha256:b351995b18968208fc25eebd647205694d04876db60d4d79a67d1cc8ff50f4ff",mountpoint="/var/lib/containerd-stargz-grpc/snapshotter/snapshots/1/fs"} 775792
stargz_fs_layer_fetched_size_bytes{digest="sha256:c95ee4aba5b02db7bbf46410c26754b42ea8ce1cfbbdc40764f96fc6f044c30b",mountpoint="/var/lib/containerd-stargz-grpc/snapshotter/snapshots/2/fs"} 680

CW shows it as zero:

{
    "_aws": {
        "CloudWatchMetrics": [
            {
                "Namespace": "estargz_snapshotter",
                "Dimensions": [
                    [
                        "digest",
                        "mountpoint"
                    ],
                    [],
                    [
                        "digest"
                    ],
                    [
                        "mountpoint"
                    ]
                ],
                "Metrics": [
                    {
                        "Name": "stargz_fs_layer_fetched_size_bytes",
                        "Unit": "Bytes"
                    },
                    {
                        "Name": "stargz_fs_layer_size_bytes",
                        "Unit": "Bytes"
                    }
                ]
            }
        ],
        "Timestamp": 1625671639164
    },
    "digest": "sha256:b351995b18968208fc25eebd647205694d04876db60d4d79a67d1cc8ff50f4ff",
    "mountpoint": "/var/lib/containerd-stargz-grpc/snapshotter/snapshots/1/fs",
    "stargz_fs_layer_fetched_size_bytes": 0,
    "stargz_fs_layer_size_bytes": 0
}

Please see the Otel log and config files attached.

aws-otel-collector.log
aws-otel-collector.txt

[mxiamxia]

The logs seems not bring any useful info. Does your counter metrics value generated in an incremental way? EMF Exporter currently calculate the delta (current value - previous value) for counter metrics. If you metrics values are not incremental, you could see negative or 0 in EMF logs.

Should the number of bytes fetched be sent as gauge metrics?

[vkuzniet]

The way it is implemented in stargz snapshotter now is that layer fetch size and layer size are being exposed by implementing Collector interface and these are constant values per layer sha. See https://github.com/containerd/stargz-snapshotter/tree/main/fs/metrics/layer for more details. It was implemented as counter, since this seems to be the way it's done in containerd project as well.

I totally agree with the value going to 0 from timestamp 2 and going forward, however I'm having hard time to understand why there's no meaningful value for timestamp 1 (the first data point for that metric).

[github-actions]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 30 days.