Promtail missing short-lived pod logs

[nmiculinic]

Describe the bug

We have a short-lived container. It's created, it executes quite quickly (thing echo "sss"; exit 1) and when the pod phase reaches completed a controller deletes the pod.

The promtail hasn't shipped the pod's logs to the loki. It hasn't added it as a target, nor started tailing the log file before the log file was deleted (we're using docker with json-log logging driver)

To Reproduce
Steps to reproduce the behavior:

1. Started Promtail (SHA or version) to tail '...': v2.1.0
2. Start short-lived container

Expected behavior
Promtail tailing the short-lived container logfile and shipping it out

Environment:

• Infrastructure: [e.g., Kubernetes, bare-metal, laptop]: k8s, EKS
• Deployment tool: [e.g., helm, jsonnet]: helm, promtail

[stale]

This issue has been automatically marked as stale because it has not had any activity in the past 30 days. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[ipleten]

I wonder why it was closed as it's still an issue. I assume it's due to sync time with k8s API. I wonder if it's possible to get data from logs and parse them after Pod was terminated?
Having this logs is extra helpful to investigate pod startup failures (like Jenkins JNLP agent, DNS resolution troubles etc.)
Update: see #775