Podman Not Respecting volume_path in containers.conf

[jkonowitch]

/kind bug

Description
Editing the volume_path in any of the containers.conf files do not seem to change the location of where new pod volumes are created.

Steps to reproduce the issue:

1. Change the value of volume_path in containers.conf

2. Create a new volume

3. See that the new volume has not been placed in the new location

Describe the results you received:

jeff@jeff-test-2:~$ cat /usr/share/containers/containers.conf | grep volume_path
volume_path = "/home/jeff/my-volumes"
jeff@jeff-test-2:~$ cat /etc/containers/containers.conf | grep volume_path
volume_path = "/home/jeff/my-volumes"
jeff@jeff-test-2:~$ cat /home/jeff/.config/containers/containers.conf | grep volume_path
volume_path = "/home/jeff/my-volumes"

jeff@jeff-test-2:~$ podman volume create test
test
jeff@jeff-test-2:~$ podman volume inspect test
[
     {
          "Name": "test",
          "Driver": "local",
          "Mountpoint": "/home/jeff/.local/share/containers/storage/volumes/test/_data",
          "CreatedAt": "2020-07-21T14:57:21.791073235Z",
          "Labels": {
               
          },
          "Scope": "local",
          "Options": {
               
          },
          "UID": 0,
          "GID": 0,
          "Anonymous": false
     }
]

Describe the results you expected:

Expected the new volume to be created under /home/jeff/my-volumes

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

Version:      2.0.2
API Version:  1
Go Version:   go1.14.2
Built:        Thu Jan  1 00:00:00 1970
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.15.0
  cgroupVersion: v1
  conmon:
    package: 'conmon: /usr/libexec/podman/conmon'
    path: /usr/libexec/podman/conmon
    version: 'conmon version 2.0.18, commit: '
  cpus: 4
  distribution:
    distribution: ubuntu
    version: "20.04"
  eventLogger: file
  hostname: jeff-test-2
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.4.0-40-generic
  linkmode: dynamic
  memFree: 3342041088
  memTotal: 4126892032
  ociRuntime:
    name: runc
    package: 'containerd.io: /usr/bin/runc'
    path: /usr/bin/runc
    version: |-
      runc version 1.0.0-rc10
      commit: dc9208a3303feef5b3839f4323d9beb36df0a9dd
      spec: 1.0.1-dev
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  rootless: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: 'slirp4netns: /usr/bin/slirp4netns'
    version: |-
      slirp4netns version 1.1.4
      commit: unknown
      libslirp: 4.2.0
      SLIRP_CONFIG_VERSION_MAX: 2
  swapFree: 0
  swapTotal: 0
  uptime: 23m 43s
registries:
  search:
  - docker.io
  - quay.io
store:
  configFile: /home/jeff/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: vfs
  graphOptions: {}
  graphRoot: /home/jeff/.local/share/containers/storage
  graphStatus: {}
  imageStore:
    number: 0
  runRoot: /run/user/1000/containers
  volumePath: /home/jeff/.local/share/containers/storage/volumes
version:
  APIVersion: 1
  Built: 0
  BuiltTime: Thu Jan  1 00:00:00 1970
  GitCommit: ""
  GoVersion: go1.14.2
  OsArch: linux/amd64
  Version: 2.0.2

Package info (e.g. output of rpm -q podman or apt list podman):

podman/unknown,now 2.0.2~2 amd64 [installed]

[mheon]

This is an important value, and as such we cache it in the database to prevent unintended consequences on change (like all your existing volumes suddenly existing in the database, but not on disk). There should be a warning message in your debug logs about a mismatch in this path between configuration files and the database.

The recommended fix is to use podman system reset to purge all on-disk state (WARNING: this will remove all existing containers, pods, images, and volumes), but preserve and then restore the /home/jeff/.config/containers/containers.conf config file. Subsequent Podman invocations will use recreate the DB with the new value, and your volumes will be where you expected.

[jkonowitch]

Got it. That does indeed fix the issue - thank you for the speedy response!

PR to update docs here: #7041