promtail: relabel_configs does not transform the filename label

[rgl]

Describe the bug

relabel_configs does not transform the filename label.

To Reproduce

Steps to reproduce the behavior:

1. Started Promtail 2.2.1

scrape_configs:
  - job_name: logwrite
    static_configs:
      - labels:
          __path__: /host/var/log/*.log
    relabel_configs: &relabel_configs
      # XXX this fails to remove the /host/ prefix from the filename.
      - source_labels: [filename]
        regex: /host(/.+)
        replacement: $1
        target_label: filename
        action: replace

Expected behavior

expected the filename to be transformed from, e.g., /host/var/log/kmsg.log to /var/log/kmsg.log.

Screenshots, Promtail config, or terminal output

+ docker run --rm --name promtail-dry-run -v /home/rgl/Projects/linuxkit-vagrant/promtail-dry-run/log:/host/var/log:ro -v /home/rgl/Projects/linuxkit-vagrant/promtail-dry-run/..:/var/run/promtail:ro grafana/promtail:2.2.1 --dry-run -config.file /var/run/promtail/promtail-config.yml
Clients configured:
----------------------
url: '@@loki_push_url@@'
batchwait: 1s
batchsize: 1048576
backoff_config:
  min_period: 500ms
  max_period: 5m0s
  max_retries: 10
timeout: 10s
tenant_id: ""

2021-06-05T14:45:20	{filename="/host/var/log/getty.out.log"}2021-06-04T17:03:37Z getty.out getty: starting getty for tty1
2021-06-05T14:45:20	{filename="/host/var/log/dhcpcd.log"}	2021-06-04T17:03:37Z dhcpcd eth0: waiting for carrier
2021-06-05T14:45:20	{filename="/host/var/log/onboot.001-rngd1.log"}	2021-06-04T17:03:36Z onboot.001-rngd1 2021/06/04 17:03:36 No random source available
2021-06-05T14:45:20	{filename="/host/var/log/kmsg.log"}	2021-06-04T17:03:37Z kmsg (189) - 2021-06-04T17:03:34.958455265Z: acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge.
2021-06-05T14:45:20	{filename="/host/var/log/kmsg.log"}	2021-06-04T17:03:37Z kmsg  SUBSYSTEM=acpi
2021-06-05T14:45:20	{filename="/host/var/log/kmsg.log"}	2021-06-04T17:03:37Z kmsg  DEVICE=+acpi:PNP0A03:00
2021-06-05T14:45:20	{filename="/host/var/log/kmsg.log"}	2021-06-04T17:03:37Z kmsg (190) - 2021-06-04T17:03:34.958844265Z: acpiphp: Slot [3] registered

Alternative

I can workaround this by using a pipeline_stages, but relabel_configs should have worked.

here's the workaround:

scrape_configs:
  - job_name: logwrite
    static_configs:
      - labels:
          __path__: /host/var/log/*.log
    pipeline_stages:
      # remove the /host/ prefix from the filename.
      - regex:
          source: filename
          expression: '/host(?P<filename>/.+)'
      - labels:
          filename:

PS

This seems related to #775, which I didn't understood why it was closed.

[richarddwoods]

Am I right in thinking that a pipeline is run once per logged line, while relabel mapping happens once per configuration load?

[rgl]

Yes, you are. This was clarified at more-or-less related issue at #3812 (comment) and #3813 (comment).

[kavirajk]

I guess this can be closed!. Feel free to re-open if its still an issue!

[victor-sudakov]

I have encountered a similar situation (Promtail 2.4.1) and cannot understand why the config below is not working (label "namespc" is not being created):

scrape_configs:
  - job_name: system
    pipeline_stages:
      - docker: {}
    static_configs:
      - targets:
          - localhost
        labels:
          job: varlogs
          __path__: /var/log/**/*.log
    relabel_configs:
      - action: replace
        source_labels:
          - filename
        regex: /var/log/pods/([0-9a-z-]+?)_
        target_label: namespc

The explanation in #3812 (comment) did not make much sense to me as I'm not removing any label, I'm trying to extract part of log filename and convert it into a new label.

Does anyone care to explain what's incorrect in my config?

[sushantkumar-amagi]

This comment should be helpful @victor-sudakov Link