containers-rootlessport does not exit, keeps holding on to port

[denniskempin]

/kind bug

Description

When running podman with a port to --publish, after the container exits, the port remains occupied by the containers-rootlessport process.

Steps to reproduce the issue:

1. podman run -p 5000:5000 hello-world

Describe the results you received:

Even after the container has exited, the port remains occupied by the containers-rootlessport process:

$ netstat -tulpn | grep :5000
tcp6       0      0 :::5000                 :::*                    LISTEN      99047/containers-ro

$ ps aux | grep containers-ro
dennisk+   99047  0.0  0.0 2369872 45844 pts/1   Sl   20:18   0:00 containers-rootlessport

Describe the results you expected:

containers-rootlessport should exit and free the port.

Additional information you deem important (e.g. issue happens only occasionally):

containers-rootlessport only seems to linger around when using --publish. Without it, the process exists as expected.

Output of podman version:

Version:      3.0.1
API Version:  3.0.0
Go Version:   go1.15.8
Built:        Wed Dec 31 16:00:00 1969
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.19.6
  cgroupManager: cgroupfs
  cgroupVersion: v2
  conmon:
    package: 'conmon: /usr/bin/conmon'
    path: /usr/bin/conmon
    version: 'conmon version 2.0.25, commit: unknown'
  cpus: 56
  distribution:
    distribution: debian
    version: rodete
  eventLogger: journald
  hostname: [...]
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 89939
      size: 1
    - container_id: 1
      host_id: 624288
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 164358
      size: 1
    - container_id: 1
      host_id: 624288
      size: 65536
  kernel: 5.7.17-1rodete5-amd64
  linkmode: dynamic
  memFree: 116288745472
  memTotal: 135163117568
  ociRuntime:
    name: crun
    package: 'crun: /usr/bin/crun'
    path: /usr/bin/crun
    version: |-
      crun version 0.17
      commit: 0e9229ae34caaebcb86f1fde18de3acaf18c6d9a
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/user/164358/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    selinuxEnabled: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: 'slirp4netns: /usr/bin/slirp4netns'
    version: |-
      slirp4netns version 1.0.1
      commit: 6a7b16babc95b6a3056b33fb45b74a6f62262dd4
      libslirp: 4.4.0
  swapFree: 137350868992
  swapTotal: 137350868992
  uptime: 1h 22m 8.62s (Approximately 0.04 days)
registries: {}
store:
  configFile: /usr/local/google/home/denniskempin/.config/containers/storage.conf
  containerStore:
    number: 5
    paused: 0
    running: 0
    stopped: 5
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: 'fuse-overlayfs: /usr/bin/fuse-overlayfs'
      Version: |-
        fusermount3 version: 3.10.1
        fuse-overlayfs: version 1.4
        FUSE library version 3.10.1
        using FUSE kernel interface version 7.31
  graphRoot: /usr/local/google/home/denniskempin/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 110
  runRoot: /run/user/164358/containers
  volumePath: /usr/local/google/home/denniskempin/.local/share/containers/storage/volumes
version:
  APIVersion: 3.0.0
  Built: 0
  BuiltTime: Wed Dec 31 16:00:00 1969
  GitCommit: ""
  GoVersion: go1.15.8
  OsArch: linux/amd64
  Version: 3.0.1

Package info (e.g. output of rpm -q podman or apt list podman):

Listing... Done
podman/now 3.0.1+dfsg1-1 amd64 [installed,local]
[...]

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide?

Yes, I have installed the latest debian sid package. The issue does not seem to be mentioned in the troubleshooting guide unfortunately.

Additional environment details (AWS, VirtualBox, physical, etc.):

A couple of things that stand out when running with debug log-level:

WARN[0000] The cgroupv2 manager is set to systemd but there is no systemd user session available
WARN[0000] For using systemd, you may need to login using an user session
WARN[0000] Alternatively, you can enable lingering with: `loginctl enable-linger 164358` (possibly as root)
WARN[0000] Falling back to --cgroup-manager=cgroupfs

I have found some other bugs on this, but have not been able to resolve this yet.

INFO[0000] Failed to add conmon to cgroupfs sandbox cgroup: error creating cgroup path /libpod_parent/conmon: open /sys/fs/cgroup/cgroup.subtree_control: permission denied
[conmon:d]: failed to write to /proc/self/oom_score_adj: Permission denied

I am not sure if these are relevant.

Syslog has some dbus errors when the container is running, but also not sure if they are relevant as they are triggered by the root user, so it might just be concidental.

dbus-daemon[105475]: [session uid=0 pid=105472] org.freedesktop.DBus.Error.AccessDenied: Failed to set fd limit to 65536: Operation not permitted
dbus-daemon[105475]: Cannot setup inotify for '/root/.local/share/dbus-1/services'; error 'Permission denied'

Any advice on how to further debug the issue would be greatly appreciated.

[mheon]

@giuseppe @AkihiroSuda PTAL

[AkihiroSuda]

Is this a recent regression?

[AkihiroSuda]

Can't repro on my host (b5827d8 , Ubuntu 20.10)

[denniskempin]

Thank you for having a look @AkihiroSuda!

I am not sure if this is a regression. I have only noticed the issue today as it is the first time I was using podman with --publish.

I would not be surprised if the issue is some permission issue with the linux distribution I am running, it is a customized company internal distribution based on debian sid. However, so far I cannot tell where the issue might be coming from so I am not sure where to look.

Do you have any recommendations on what I could do to further investigate the issue? Is it possible to get verbose syslogs from the containers-rootlessport process?

[sdhoward]

related to #9447 ?

[mheon]

I would guess so. If anyone experiencing this can test with a newer Conmon and see if it resolves the issue, it would be greatly appreciated.

[sdhoward]

this seems to be resolved for me with conmon 2.0.27

[mheon]

Alright. Going to close as a dupe of #9447 then.