Flow prometheus remote write on Windows fails to replay WAL after restart

[dawei-nh]

What's wrong?

After restarting the Flow service or rebooting the Windows server, the agent will attempt to replay the WAL but get stuck in loop trying to replay the WAL:

ts=2023-05-10T00:22:46.5112732Z component=prometheus.remote_write.mimir subcomponent=rw level=info remote_name=d9bd36 url=https://my_mimir_url.com/api/v1/push msg="Replaying WAL" queue=d9bd36
ts=2023-05-10T00:22:46.5112732Z component=prometheus.remote_write.mimir subcomponent=rw level=error remote_name=d9bd36 url=https://my_mimir_url.com/api/v1/push msg="error tailing WAL" err="readCheckpoint: checkpointNum: invalid checkpoint dir string: C:\\ProgramData\\Grafana Agent Flow\\data\\prometheus.remote_write.mimir\\wal\\checkpoint.00000064"

After that, it will error ever minute with this:

ts=2023-05-10T00:23:46.5993796Z component=prometheus.remote_write.mimir subcomponent=rw level=error remote_name=d9bd36 url=https://my_mimir_url.com/api/v1/push msg="error tailing WAL" err="readCheckpoint: checkpointNum: invalid checkpoint dir string: C:\\ProgramData\\Grafana Agent Flow\\data\\prometheus.remote_write.mimir\\wal\\checkpoint.00000064"

And no metrics will be written to Mimir.

Stopping the agent and removing the WAL folder and starting it up again resolves it, but this is hard to maintain over a large amount of servers when doing updates.

Steps to reproduce

1. Configure Grafana Agent Flow on Windows with a prometheus remote_write component
2. Let the WAL accumulate
3. Restart the Grafana Agent Flow service

System information

Windows Server 2019 1809

Software version

Grafana Agent Flow v0.33.1

Configuration

logging {
	level = "info"
}

prometheus.exporter.windows "this" {
  enabled_collectors = ["adfs","cpu","cs","logical_disk","net","os","process","system"]
}

prometheus.scrape "windows" {
  targets    = prometheus.exporter.windows.this.targets
  forward_to = [prometheus.relabel.this.receiver]
  
  scrape_interval = "15s"
}

prometheus.relabel "this" {
  forward_to = [prometheus.remote_write.mimir.receiver]
  
  rule {
    action       = "replace"
	target_label = "instance"
	replacement  = constants.hostname
  }
}

prometheus.remote_write "mimir" {
  endpoint {
    url = "https://my-mimir-url.com/api/v1/push"
  basic_auth {
    username = "foo"
    password = "bar"
    }
  }
}

### Logs

_No response_

[dawei-nh]

After some investigating, this looks to be caused by the upstream Prometheus when trying to split the directory for the checkpoint.

https://github.com/prometheus/prometheus/blob/bd98fc8c45524609590ef757a5f32ee7f0f0d1c7/tsdb/wlog/watcher.go#L691-L697

It splits the directory string on the period. This works on a unix path but not Windows as path.Base() does not properly split the path in Windows, returning the full path to the folder/file instead.

This can be worked around by not using periods in any folder names for the WAL directory.

[rfratto]

Hi, thanks for reporting and investigating, your discover is really useful. I'll open a PR upstream to fix, and we'll see what we can do to bring it downstream to fix the issue.

This can be worked around by not using periods in any folder names for the WAL directory.

This isn't something that can be done without a code change unfortunately, the component name (which always contains a .) is hard-coded to be in the WAL directory.