Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CI: sys: fix "kube play url" flake #21632

Closed
wants to merge 1 commit into from
Closed

CI: sys: fix "kube play url" flake #21632

wants to merge 1 commit into from

Conversation

edsantiago
Copy link
Member

Nasty flakes. This one has been hitting us hard, and only
f38 root (both CNI and netavark).

Signed-off-by: Ed Santiago santiago@redhat.com

None

@edsantiago
CI: sys: fix "kube play url" flake
32dba2b 
Nasty flakes. This one has been hitting us hard, and only
f38 root (both CNI and netavark).

Signed-off-by: Ed Santiago <santiago@redhat.com>
@openshift-ci OpenShift CI
Copy link
Contributor

openshift-ci bot commented Feb 13, 2024

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: edsantiago

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Feb 13, 2024
@edsantiago
Copy link
Member Author

[sys] [700] podman kube play - URL

x x x x x x
sys(9) remote(7) fedora-38(9) root(9) host(9) boltdb(9)
podman(2)

@edsantiago edsantiago added the flakes Flakes from Continuous Integration label Feb 13, 2024
@edsantiago edsantiago marked this pull request as draft February 13, 2024 12:10
@openshift-ci openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Feb 13, 2024
@edsantiago
Copy link
Member Author

Converting to draft. This does not fix anything! In my local hammer testing I just got:

[05:08:09.449963892] # /home/esm/src/atomic/2018-02.podman/libpod/bin/podman run -d --name myyaml -p 5776:80 -v /tmp/podman_bats.1SlOzR/test.yaml:/var/www/testpod.yaml:Z -v /tmp/podman_bats.1SlOzR/ready:/var/www/ready:Z -w /var/www quay.io/libpod/testimage:20240123 /bin/busybox-extras httpd -f -p 80
[05:08:09.681115870] 36a45377fe03f9eeba9d73cc54690a5a4f70f09633a96b0c85154d9a9b56e951                                                                        
# curl -s -S http://127.0.0.1:5776/ready
curl: (7) Failed to connect to 127.0.0.1 port 5776 after 3098 ms: Couldn't connect to server
...repeat repeat repeat
#/vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
#| FAIL: Timed out waiting for 'curl -s -S http://127.0.0.1:5776/ready' to return 'READY'
#\^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Okay, this is not something I can fix.

@edsantiago edsantiago closed this Feb 13, 2024
@edsantiago edsantiago deleted the kube-play-url-flake branch February 13, 2024 12:12
@Luap99
Copy link
Member

Luap99 commented Feb 13, 2024

Weird one, FYI the wait_for_port is useless here as podman run -d binds the port on the host always before starting the container.
If it is only as root then it could be some firewall rule issue, however we have many tests that forward a port and connect to it (without flakes?) so why is this one different.

Of course there is the potential race that the http server in the container is not ready but than this patch should have fixed it.

@edsantiago
Copy link
Member Author

Added post-flake debug, ps and inspect. Will look at it tomorrow.

  #/vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
   #| FAIL: Timed out waiting for 'curl -s -S http://127.0.0.1:5457/ready' to return 'READY'
   #\^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   
   [14:22:17.553668027] # /home/esm/src/atomic/2018-02.podman/libpod/bin/podman ps -a
   [14:22:17.608806684] CONTAINER ID  IMAGE                              COMMAND               CREATED             STATUS             PORTS                 NAME
S
   72a55a54fe5a  quay.io/libpod/testimage:20240123  /bin/busybox-extr...  About a minute ago  Up About a minute  0.0.0.0:5457->80/tcp  myyaml
   
   [14:22:17.615893042] # /home/esm/src/atomic/2018-02.podman/libpod/bin/podman container inspect myyaml
   [14:22:17.668337800] [
        {
             "Id": "72a55a54fe5a7bb74cab8cbc504749405c33c90401e1790df96f4540ad1ab211",
             "Created": "2024-02-13T14:21:05.622396682-07:00",
             "Path": "/bin/busybox-extras",
             "Args": [
                  "httpd",
                  "-f",
                  "-p",
                  "80"
             ],
             "State": {
                  "OciVersion": "1.1.0+dev",
                  "Status": "running",
                  "Running": true,
                  "Paused": false,
                  "Restarting": false,
                  "OOMKilled": false,
                  "Dead": false,
                  "Pid": 3480362,
                  "ConmonPid": 3480360,
                  "ExitCode": 0,
                  "Error": "",
                  "StartedAt": "2024-02-13T14:21:05.791046692-07:00",
                  "FinishedAt": "0001-01-01T00:00:00Z",
                  "CgroupPath": "/machine.slice/libpod-72a55a54fe5a7bb74cab8cbc504749405c33c90401e1790df96f4540ad1ab211.scope",
                  "CheckpointedAt": "0001-01-01T00:00:00Z",
                  "RestoredAt": "0001-01-01T00:00:00Z"
},                                                                                                                              14:22:18 [265/4864]
             "Image": "1f6acd4c4a1d4d39395870e1c12dbb68da27dd432afecf86e7cb6e23cf4b75d0",
             "ImageDigest": "sha256:17823456328e5f38b2c0dc7f9453aee0196654dba39ae2220f869717f159159a",
             "ImageName": "quay.io/libpod/testimage:20240123",
             "Rootfs": "",
             "Pod": "",
             "ResolvConfPath": "/run/containers/storage/overlay-containers/72a55a54fe5a7bb74cab8cbc504749405c33c90401e1790df96f4540ad1ab211/userdata/resolv.conf
",
             "HostnamePath": "/run/containers/storage/overlay-containers/72a55a54fe5a7bb74cab8cbc504749405c33c90401e1790df96f4540ad1ab211/userdata/hostname",
             "HostsPath": "/run/containers/storage/overlay-containers/72a55a54fe5a7bb74cab8cbc504749405c33c90401e1790df96f4540ad1ab211/userdata/hosts",
             "StaticDir": "/var/lib/containers/storage/overlay-containers/72a55a54fe5a7bb74cab8cbc504749405c33c90401e1790df96f4540ad1ab211/userdata",
             "OCIConfigPath": "/var/lib/containers/storage/overlay-containers/72a55a54fe5a7bb74cab8cbc504749405c33c90401e1790df96f4540ad1ab211/userdata/config.j
son",
             "OCIRuntime": "crun",
             "ConmonPidFile": "/run/containers/storage/overlay-containers/72a55a54fe5a7bb74cab8cbc504749405c33c90401e1790df96f4540ad1ab211/userdata/conmon.pid",
             "PidFile": "/run/containers/storage/overlay-containers/72a55a54fe5a7bb74cab8cbc504749405c33c90401e1790df96f4540ad1ab211/userdata/pidfile",
             "Name": "myyaml",
             "RestartCount": 0,
             "Driver": "overlay",
             "MountLabel": "system_u:object_r:container_file_t:s0:c702,c790",
             "ProcessLabel": "system_u:system_r:container_t:s0:c702,c790",
             "AppArmorProfile": "",
             "EffectiveCaps": [
                  "CAP_CHOWN",
                  "CAP_DAC_OVERRIDE",
                  "CAP_FOWNER",
                  "CAP_FSETID",
                  "CAP_KILL",
                  "CAP_NET_BIND_SERVICE",
                  "CAP_SETFCAP",
                  "CAP_SETGID",
                  "CAP_SETPCAP",
                  "CAP_SETUID",
                  "CAP_SYS_CHROOT"
             ],
             "BoundingCaps": [
                  "CAP_CHOWN",
                  "CAP_DAC_OVERRIDE",
                  "CAP_FOWNER",
                  "CAP_FSETID",
                  "CAP_KILL",
                  "CAP_NET_BIND_SERVICE",
                  "CAP_SETFCAP",
                  "CAP_SETGID",
                  "CAP_SETPCAP",
                  "CAP_SETUID",
                  "CAP_SYS_CHROOT"
             ],
             "ExecIDs": [],
             "GraphDriver": {
                  "Name": "overlay",
                  "Data": {
                       "LowerDir": "/var/lib/containers/storage/overlay/7ffe79913b6cd452b42b4aef7f51ff4eeecbdde76a464ef55c321a345faaa4a6/diff:/var/lib/container
s/storage/overlay/25a964f60e84b2d24f8098376c985a5c057bea74d699ee50c93e417c26516cc3/diff",
                       "MergedDir": "/var/lib/containers/storage/overlay/3eaf2b080947ac9b2014cfea9370ae414dcbc7cf661ddd4850efaf87ff36f792/merged",
                       "UpperDir": "/var/lib/containers/storage/overlay/3eaf2b080947ac9b2014cfea9370ae414dcbc7cf661ddd4850efaf87ff36f792/diff",
                       "WorkDir": "/var/lib/containers/storage/overlay/3eaf2b080947ac9b2014cfea9370ae414dcbc7cf661ddd4850efaf87ff36f792/work"
                  }
             },
             "Mounts": [
{
                       "Type": "bind",
                       "Source": "/tmp/podman_bats.mmlLZN/ready",
                       "Destination": "/var/www/ready",
                       "Driver": "",
                       "Mode": "",
                       "Options": [
                            "nosuid",
                            "nodev",
                            "rbind"
                       ],
                       "RW": true,
                       "Propagation": "rprivate"
                  },
                  {
                       "Type": "bind",
                       "Source": "/tmp/podman_bats.mmlLZN/test.yaml",
                       "Destination": "/var/www/testpod.yaml",
                       "Driver": "",
                       "Mode": "",
                       "Options": [
                            "nosuid",
                            "nodev",
                            "rbind"
                       ],
                       "RW": true,
                       "Propagation": "rprivate"
                  }
             ],
             "Dependencies": [],
             "NetworkSettings": {
                  "EndpointID": "",
                  "Gateway": "10.88.0.1",
                  "IPAddress": "10.88.15.157",
                  "IPPrefixLen": 16,
                  "IPv6Gateway": "",
                  "GlobalIPv6Address": "",
                  "GlobalIPv6PrefixLen": 0,
                  "MacAddress": "0e:54:b1:5a:65:e9",
                  "Bridge": "",
                  "SandboxID": "",
                  "HairpinMode": false,
                  "LinkLocalIPv6Address": "",
                  "LinkLocalIPv6PrefixLen": 0,
                  "Ports": {
                       "80/tcp": [
                            {
                                 "HostIp": "",
                                 "HostPort": "5457"
                            }
                       ]
                  },
                  "SandboxKey": "/run/netns/netns-a684a72e-74d9-00bd-7dcf-ace252d050e2",
                  "Networks": {
                       "podman": {
"EndpointID": "",                                                                                                                   
                            "Gateway": "10.88.0.1",
                            "IPAddress": "10.88.15.157",
                            "IPPrefixLen": 16,
                            "IPv6Gateway": "",
                            "GlobalIPv6Address": "",
                            "GlobalIPv6PrefixLen": 0,
                            "MacAddress": "0e:54:b1:5a:65:e9",
                            "NetworkID": "podman",
                            "DriverOpts": null,
                            "IPAMConfig": null,
                            "Links": null,
                            "Aliases": [ 
                                 "72a55a54fe5a"
                            ]
                       }
                  }
             },
             "Namespace": "",
             "IsInfra": false,
             "IsService": false,
             "KubeExitCodePropagation": "invalid",
             "lockNumber": 260,
             "Config": {
                  "Hostname": "72a55a54fe5a",
                  "Domainname": "",
                  "User": "",
                  "AttachStdin": false,
                  "AttachStdout": false, 
                  "AttachStderr": false, 
                  "Tty": false,
                  "OpenStdin": false,
                  "StdinOnce": false,
                  "Env": [
                       "container=podman",
                       "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                       "HOME=/root",
                       "HOSTNAME=72a55a54fe5a"
                  ],
                  "Cmd": [
                       "/bin/busybox-extras",
                       "httpd",
                       "-f",
                       "-p",
                       "80"
                  ],
                  "Image": "quay.io/libpod/testimage:20240123",
                  "Volumes": null,
                  "WorkingDir": "/var/www",
                  "Entrypoint": "",
                  "OnBuild": null,
                  "Labels": {
"created_at": "2024-01-24T18:38:28Z",
                       "created_by": "test/system/build-testimage",
                       "io.buildah.version": "1.34.1-dev"
                  },
                  "Annotations": {
                       "io.container.manager": "libpod",
                       "org.opencontainers.image.stopSignal": "15"
                  },
                  "StopSignal": 15,
                  "HealthcheckOnFailureAction": "none",
                  "CreateCommand": [
                       "/home/esm/src/atomic/2018-02.podman/libpod/bin/podman",
                       "run",
                       "-d",
                       "--name",
                       "myyaml",
                       "-p",
                       "5457:80",
                       "-v",
                       "/tmp/podman_bats.mmlLZN/test.yaml:/var/www/testpod.yaml:Z",
                       "-v",
                       "/tmp/podman_bats.mmlLZN/ready:/var/www/ready:Z",
                       "-w",
                       "/var/www",
                       "quay.io/libpod/testimage:20240123",
                       "/bin/busybox-extras",
                       "httpd",
                       "-f",
                       "-p",
                       "80"
                  ],
                  "Umask": "0022",
                  "Timeout": 0,
                  "StopTimeout": 10,
                  "Passwd": true,
                  "sdNotifyMode": "container"
             },
             "HostConfig": {
                  "Binds": [
                       "/tmp/podman_bats.mmlLZN/ready:/var/www/ready:rw,rprivate,nosuid,nodev,rbind",
                       "/tmp/podman_bats.mmlLZN/test.yaml:/var/www/testpod.yaml:rw,rprivate,nosuid,nodev,rbind"
                  ],
                  "CgroupManager": "systemd",
                  "CgroupMode": "private",
                  "ContainerIDFile": "", 
                  "LogConfig": {
                       "Type": "journald",
                       "Config": null,
                       "Path": "",
                       "Tag": "",
                       "Size": "0B"
                  },
"NetworkMode": "bridge",
                  "PortBindings": {
                       "80/tcp": [
                            {
                                 "HostIp": "",
                                 "HostPort": "5457"
                            }
                       ]
                  },
                  "RestartPolicy": {
                       "Name": "",
                       "MaximumRetryCount": 0
                  },
                  "AutoRemove": false,
                  "VolumeDriver": "",
                  "VolumesFrom": null,
                  "CapAdd": [],
                  "CapDrop": [],
                  "Dns": [],
                  "DnsOptions": [],
                  "DnsSearch": [],
                  "ExtraHosts": [],
                  "GroupAdd": [],
                  "IpcMode": "shareable",
                  "Cgroup": "",
                  "Cgroups": "default",
                  "Links": null,
                  "OomScoreAdj": 0,
                  "PidMode": "private",
                  "Privileged": false,
                  "PublishAllPorts": false,
                  "ReadonlyRootfs": false,
                  "SecurityOpt": [],
                  "Tmpfs": {},
                  "UTSMode": "private",
                  "UsernsMode": "",
                  "ShmSize": 65536000,
                  "Runtime": "oci",
                  "ConsoleSize": [
                       0,
                       0
                  ],
                  "Isolation": "",
                  "CpuShares": 0,
                  "Memory": 0,
                  "NanoCpus": 0,
                  "CgroupParent": "",
                  "BlkioWeight": 0,
                  "BlkioWeightDevice": null,
                  "BlkioDeviceReadBps": null,
                  "BlkioDeviceWriteBps": null,
                  "BlkioDeviceReadIOps": null,
                  "BlkioDeviceWriteIOps": null,
                  "CpuPeriod": 0,
                  "CpuQuota": 0,
"CpuRealtimePeriod": 0,
                  "CpuRealtimeRuntime": 0,
                  "CpusetCpus": "",
                  "CpusetMems": "",
                  "Devices": [],
                  "DiskQuota": 0,
                  "KernelMemory": 0,
                  "MemoryReservation": 0,
                  "MemorySwap": 0,
                  "MemorySwappiness": 0, 
                  "OomKillDisable": false,
                  "PidsLimit": 2048,
                  "Ulimits": [
                       {
                            "Name": "RLIMIT_NPROC",
                            "Soft": 4194304,
                            "Hard": 4194304
                       }
                  ],
                  "CpuCount": 0,
                  "CpuPercent": 0,
                  "IOMaximumIOps": 0,
                  "IOMaximumBandwidth": 0,
                  "CgroupConf": null
             }
        }
   ]

@stale-locking-app stale-locking-app bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label May 14, 2024
@stale-locking-app stale-locking-app bot locked as resolved and limited conversation to collaborators May 14, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers
No reviews
Assignees
No one assigned
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. flakes Flakes from Continuous Integration locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. release-note-none
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@edsantiago @Luap99