Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

kube play URL : connection refused #21649

Closed
edsantiago opened this issue Feb 14, 2024 · 19 comments · Fixed by #21846
Closed

kube play URL : connection refused #21649

edsantiago opened this issue Feb 14, 2024 · 19 comments · Fixed by #21846
Labels
flakes Flakes from Continuous Integration locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments.

Comments

@edsantiago
Copy link
Member 

[+1437s] not ok 634 [700] podman kube play - URL
...
<+029ms> # # podman-remote --url unix:///var/tmp/podman_tmp_bKYC kube play http://127.0.0.1:5547/testpod.yaml
<+048ms> # Error: Get "http://127.0.0.1:5547/testpod.yaml": dial tcp 127.0.0.1:5547: connect: connection refused

It doesn't seem to be a wait-for-ready thing: adding a curl-retry loop times out.

Root only. Only f38 in CI, but reproduces within O(minutes) on my f39 laptop.

x x x x x x
sys(9) remote(7) fedora-38(9) root(9) host(9) boltdb(9)
podman(2)
@edsantiago edsantiago added the flakes Flakes from Continuous Integration label Feb 14, 2024
@edsantiago
Copy link
Member Author

For anyone who wishes to play along at home:

--- a/test/system/700-play.bats
+++ b/test/system/700-play.bats
@@ -511,15 +511,29 @@ _EOF
     TESTDIR=$PODMAN_TMPDIR/testdir
     mkdir -p $TESTDIR
     echo "$testYaml" | sed "s|TESTDIR|${TESTDIR}|g" > $PODMAN_TMPDIR/test.yaml
+    echo READY                                      > $PODMAN_TMPDIR/ready
 
     HOST_PORT=$(random_free_port)
     SERVER=http://127.0.0.1:$HOST_PORT
 
     run_podman run -d --name myyaml -p "$HOST_PORT:80" \
                -v $PODMAN_TMPDIR/test.yaml:/var/www/testpod.yaml:Z \
+               -v $PODMAN_TMPDIR/ready:/var/www/ready:Z \
                -w /var/www \
                $IMAGE /bin/busybox-extras httpd -f -p 80
+
     wait_for_port 127.0.0.1 $HOST_PORT
+    # FIXME: 2024-02-14 TEMPORARY: this and 'if' block below are
+    # instrumentation for #21649.
+    defer-assertion-failures
+    wait_for_command_output "curl -s -S $SERVER/ready" "READY"
+    if [[ "$output" != "READY" ]]; then
+        run_podman ps -a
+        run_podman container inspect myyaml
+        # INSERT YOUR OWN DEBUG COMMANDS HERE
+        immediate-assertion-failures
+    fi
+    immediate-assertion-failures
 
     run_podman kube play $SERVER/testpod.yaml
     run_podman inspect test_pod-test --format "{{.State.Running}}"

Then:

$ while :;do hack/bats --root 700:URL || break;done

@Luap99
Copy link
Member

Luap99 commented Feb 14, 2024

Maybe a silly question given you are able to reproduce on f39 but is it time to drop all f38 testing? It is not like we plan to ship 5.0 there anyway and CNI tests are now also gone so I don't think there is much reason to keep them around?

That may not fix the flake here but at least it prevents CI from flaking more often.

@edsantiago
Copy link
Member Author

Not a silly question. Right now f38 is the only place where we test boldtb and VFS. It could be possible to test those in a different f39 instance, but that would require mucking with Cirrus job titles, and that is always a nightmare

@rhatdan
Copy link
Member

rhatdan commented Feb 14, 2024

I would prefer to drop f38, when f40 branches and we can test on it. Podman 5.0 is not going to ship on F39 either. Although we should probably package it up, but never release it.

@edsantiago
Copy link
Member Author

I understand the desire to drop f38 but I repeat: serious, heavy work will be needed on Cirrus titles. Given the current daily churn on .cirrus.yml, and the frequent CI breaks, I am absolutely not going to touch Cirrus stuff until it all calms down.

@rhatdan
Copy link
Member

rhatdan commented Feb 14, 2024

SGTM

@edsantiago
Copy link
Member Author

edsantiago commented Feb 14, 2024 •
edited
Loading

Could this be an IPv4/v6 thing? nope, never mind, I should've been checking conmon not podman.

[test fails]
# /bin/podman container inspect --format {{.State.Pid}} myyaml
860736
# lsof -p 860736
COMMAND      PID USER   FD   TYPE DEVICE SIZE/OFF      NODE NAME
busybox-e 860736 root  cwd    DIR   0,73       34 123847742 /var/www
busybox-e 860736 root  rtd    DIR   0,73       12 123847715 /
busybox-e 860736 root  txt    REG   0,73   104168 121770097 /bin/busybox-extras
busybox-e 860736 root  mem    REG   0,73   104168 121770097 /bin/busybox-extras
busybox-e 860736 root  mem    REG   0,73   649832 121770332 /lib/ld-musl-x86_64.so.1
busybox-e 860736 root    0u   CHR    1,3      0t0         5 /dev/null
busybox-e 860736 root    1w  FIFO   0,14      0t0 195288364 pipe
busybox-e 860736 root    2w  FIFO   0,14      0t0 195288365 pipe
busybox-e 860736 root    3u  sock    0,8      0t0 195289294 protocol: TCPv6

edsantiago added a commit to edsantiago/libpod that referenced this issue Feb 15, 2024
@edsantiago
CI: sys: instrument "kube play url" flake
7526d36 
Instrumentation to try to track down containers#21649

Nasty flake. This one has been hitting us hard, and only
f38 root (both CNI and netavark. Also multiple versions
of netavark).

Signed-off-by: Ed Santiago <santiago@redhat.com>
@edsantiago
Copy link
Member Author

Odd. Just saw it flake with no route to host (instead of the usual connection refused).

@edsantiago
Copy link
Member Author

In case the problem is iptables or netavark, this is a diff between "calm system" and "system with a nonresponsive podman":

--- /tmp/iptables-baseline	2024-02-15 10:02:35.873328903 -0700
+++ -	2024-02-15 10:17:31.472311795 -0700
@@ -14,10 +14,13 @@
 DROP       0    --  0.0.0.0/0            0.0.0.0/0            ctstate INVALID
 ACCEPT     0    --  0.0.0.0/0            10.89.1.0/24         ctstate RELATED,ESTABLISHED
 ACCEPT     0    --  10.89.1.0/24         0.0.0.0/0           
+ACCEPT     0    --  0.0.0.0/0            10.88.0.0/16         ctstate RELATED,ESTABLISHED
+ACCEPT     0    --  10.88.0.0/16         0.0.0.0/0           
 
 Chain NETAVARK_INPUT (1 references)
 target     prot opt source               destination         
 ACCEPT     17   --  10.89.1.0/24         0.0.0.0/0            udp dpt:53
+ACCEPT     17   --  10.88.0.0/16         0.0.0.0/0            udp dpt:53
 
 Chain NETAVARK_ISOLATION_2 (1 references)
 target     prot opt source               destination         
@@ -25,4 +28,5 @@
 Chain NETAVARK_ISOLATION_3 (0 references)
 target     prot opt source               destination         
 DROP       0    --  0.0.0.0/0            0.0.0.0/0           
+DROP       0    --  0.0.0.0/0            0.0.0.0/0           
 NETAVARK_ISOLATION_2  0    --  0.0.0.0/0            0.0.0.0/0

And yeah, this one is failing with EHOSTUNREACH:

# telnet 127.0.0.1 5435
Trying 127.0.0.1...
telnet: connect to address 127.0.0.1: No route to host

@Luap99
Copy link
Member

Luap99 commented Feb 15, 2024

@edsantiago Just to be sure the additions (green) is the version where it does not work, right?
Can you also add the full iptables output here.

@edsantiago
Copy link
Member Author

Yes. Baseline is "no podman running", additions (green) are when I have an unresponsive kube container running.

Baseline shown below (iptables -L -n):

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
NETAVARK_INPUT  0    --  0.0.0.0/0            0.0.0.0/0            /* netavark firewall rules */

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         
NETAVARK_FORWARD  0    --  0.0.0.0/0            0.0.0.0/0            /* netavark firewall rules */

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

Chain NETAVARK_FORWARD (1 references)
target     prot opt source               destination         
DROP       0    --  0.0.0.0/0            0.0.0.0/0            ctstate INVALID
ACCEPT     0    --  0.0.0.0/0            10.89.1.0/24         ctstate RELATED,ESTABLISHED
ACCEPT     0    --  10.89.1.0/24         0.0.0.0/0           

Chain NETAVARK_INPUT (1 references)
target     prot opt source               destination         
ACCEPT     17   --  10.89.1.0/24         0.0.0.0/0            udp dpt:53

Chain NETAVARK_ISOLATION_2 (1 references)
target     prot opt source               destination         

Chain NETAVARK_ISOLATION_3 (0 references)
target     prot opt source               destination         
DROP       0    --  0.0.0.0/0            0.0.0.0/0           
NETAVARK_ISOLATION_2  0    --  0.0.0.0/0            0.0.0.0/0

@Luap99
Copy link
Member

Luap99 commented Feb 15, 2024

If that is your baseline with no containers running then some clean-up failed before as there should not be any 10.89.1.0 references in that.

A clean baseline if containers were running before should look like this:

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
NETAVARK_INPUT  0    --  0.0.0.0/0            0.0.0.0/0            /* netavark firewall rules */

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         
NETAVARK_FORWARD  0    --  0.0.0.0/0            0.0.0.0/0            /* netavark firewall rules */

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

Chain NETAVARK_FORWARD (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DROP       0    --  *      *       0.0.0.0/0            0.0.0.0/0            ctstate INVALID

Chain NETAVARK_INPUT (1 references)
 pkts bytes target     prot opt in     out     source               destination         

Chain NETAVARK_ISOLATION_2 (1 references)
 pkts bytes target     prot opt in     out     source               destination         

Chain NETAVARK_ISOLATION_3 (0 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 NETAVARK_ISOLATION_2  0    --  *      *       0.0.0.0/0            0.0.0.0/0

But none of this should effect port forwarding really, you need to check sudo iptables -nvL -t nat to see any port forwarding rules.

@edsantiago
Copy link
Member Author

Grumble. Okay, I've restarted the test loop. It should hang within a minute or two. (clarification: "hang" because I instrumented with sleep 9999 so I can investigate with the container running). ITM here's my baseline nat:

Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
  488 28504 NETAVARK-HOSTPORT-DNAT  0    --  *      *       0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
15575 1105K NETAVARK-HOSTPORT-DNAT  0    --  *      *       0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type LOCAL

Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
3904K  327M NETAVARK-HOSTPORT-MASQ  0    --  *      *       0.0.0.0/0            0.0.0.0/0           
   34  3694 NETAVARK-C93D28F2CF205  0    --  *      *       10.89.0.0/24         0.0.0.0/0           
    7  1022 NETAVARK-194D34E8D615B  0    --  *      *       10.89.1.0/24         0.0.0.0/0           

Chain NETAVARK-194D34E8D615B (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 ACCEPT     0    --  *      *       0.0.0.0/0            10.89.1.0/24        
    0     0 MASQUERADE  0    --  *      *       0.0.0.0/0           !224.0.0.0/4         

Chain NETAVARK-C93D28F2CF205 (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 ACCEPT     0    --  *      *       0.0.0.0/0            10.89.0.0/24        
    0     0 MASQUERADE  0    --  *      *       0.0.0.0/0           !224.0.0.0/4         

Chain NETAVARK-DN-0212CE89B293B (0 references)
 pkts bytes target     prot opt in     out     source               destination         

Chain NETAVARK-DN-1866B9FCB39F7 (0 references)
 pkts bytes target     prot opt in     out     source               destination         

Chain NETAVARK-DN-194D34E8D615B (2 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 NETAVARK-HOSTPORT-SETMARK  6    --  *      *       10.89.1.0/24         0.0.0.0/0            tcp dpt:5412
    0     0 NETAVARK-HOSTPORT-SETMARK  6    --  *      *       127.0.0.1            0.0.0.0/0            tcp dpt:5412
    0     0 DNAT       6    --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:5412 to:10.89.1.3:80
    0     0 NETAVARK-HOSTPORT-SETMARK  6    --  *      *       10.89.1.0/24         127.0.0.1            tcp dpts:8080:8082
    0     0 NETAVARK-HOSTPORT-SETMARK  6    --  *      *       127.0.0.1            127.0.0.1            tcp dpts:8080:8082
    0     0 DNAT       6    --  *      *       0.0.0.0/0            127.0.0.1            tcp dpts:8080:8082 to:10.89.1.3:8080-8082/8080

Chain NETAVARK-DN-1D8721804F16F (3 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 NETAVARK-HOSTPORT-SETMARK  6    --  *      *       10.88.0.0/16         127.0.0.1            tcp dpt:42089
    7   420 NETAVARK-HOSTPORT-SETMARK  6    --  *      *       127.0.0.1            127.0.0.1            tcp dpt:42089
    7   420 DNAT       6    --  *      *       0.0.0.0/0            127.0.0.1            tcp dpt:42089 to:10.88.10.158:5000
    0     0 NETAVARK-HOSTPORT-SETMARK  6    --  *      *       10.88.0.0/16         0.0.0.0/0            tcp dpt:5412
    0     0 NETAVARK-HOSTPORT-SETMARK  6    --  *      *       127.0.0.1            0.0.0.0/0            tcp dpt:5412
    0     0 DNAT       6    --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:5412 to:10.88.15.67:80
    0     0 DNAT       6    --  *      *       0.0.0.0/0            127.0.0.1            tcp dpts:8080:8082 to:10.88.15.67:8080-8082/8080

Chain NETAVARK-DN-2B0AC2CFAD473 (0 references)
 pkts bytes target     prot opt in     out     source               destination         

Chain NETAVARK-DN-34AE53115BF02 (0 references)
 pkts bytes target     prot opt in     out     source               destination         

Chain NETAVARK-DN-39532C6767264 (0 references)
 pkts bytes target     prot opt in     out     source               destination         

Chain NETAVARK-DN-71E246834E221 (0 references)
 pkts bytes target     prot opt in     out     source               destination         

Chain NETAVARK-DN-AB864D580210A (0 references)
 pkts bytes target     prot opt in     out     source               destination         

Chain NETAVARK-DN-AFD48F84636BE (0 references)
 pkts bytes target     prot opt in     out     source               destination         

Chain NETAVARK-DN-C8B87F0922FBF (0 references)
 pkts bytes target     prot opt in     out     source               destination         

Chain NETAVARK-DN-C93D28F2CF205 (2 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 NETAVARK-HOSTPORT-SETMARK  6    --  *      *       10.89.0.0/24         0.0.0.0/0            tcp dpt:5499
    0     0 NETAVARK-HOSTPORT-SETMARK  6    --  *      *       127.0.0.1            0.0.0.0/0            tcp dpt:5499
    0     0 DNAT       6    --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:5499 to:10.89.0.3:80
    0     0 NETAVARK-HOSTPORT-SETMARK  6    --  *      *       10.89.0.0/24         127.0.0.1            tcp dpts:8080:8082
    0     0 NETAVARK-HOSTPORT-SETMARK  6    --  *      *       127.0.0.1            127.0.0.1            tcp dpts:8080:8082
    0     0 DNAT       6    --  *      *       0.0.0.0/0            127.0.0.1            tcp dpts:8080:8082 to:10.89.0.3:8080-8082/8080

Chain NETAVARK-DN-D082943377E18 (0 references)
 pkts bytes target     prot opt in     out     source               destination         

Chain NETAVARK-DN-D9B9BDE897335 (0 references)
 pkts bytes target     prot opt in     out     source               destination         

Chain NETAVARK-DN-F4B54CC537909 (0 references)
 pkts bytes target     prot opt in     out     source               destination         

Chain NETAVARK-DN-F9BB1A4E812BF (0 references)
 pkts bytes target     prot opt in     out     source               destination         

Chain NETAVARK-HOSTPORT-DNAT (2 references)
 pkts bytes target     prot opt in     out     source               destination         
    7   420 NETAVARK-DN-1D8721804F16F  6    --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:42089 /* dnat name: podman id: 02900bfad45dec5687a37d5b35ba304ebf5e9da967c10580b21ad2d6efe2c0e9 */
    0     0 NETAVARK-DN-C93D28F2CF205  6    --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:5499 /* dnat name: mytestnetworkqDS6zCbv id: 6e5b7d8d2e98d6938fb965ac6a83bd1a8a76d10c6981a74a6bfd421b4730aa2f */
    0     0 NETAVARK-DN-C93D28F2CF205  6    --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpts:8080:8082 /* dnat name: mytestnetworkqDS6zCbv id: 6e5b7d8d2e98d6938fb965ac6a83bd1a8a76d10c6981a74a6bfd421b4730aa2f */
    0     0 NETAVARK-DN-194D34E8D615B  6    --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:5412 /* dnat name: mytestnetwork6dwsVOMy id: b48eaac9afeb369ecb86a18977867b28a3b67655d65162d7f4c2ff8f49eaec51 */
    0     0 NETAVARK-DN-194D34E8D615B  6    --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpts:8080:8082 /* dnat name: mytestnetwork6dwsVOMy id: b48eaac9afeb369ecb86a18977867b28a3b67655d65162d7f4c2ff8f49eaec51 */
    0     0 NETAVARK-DN-1D8721804F16F  6    --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:5412 /* dnat name: podman id: b48eaac9afeb369ecb86a18977867b28a3b67655d65162d7f4c2ff8f49eaec51 */
    0     0 NETAVARK-DN-1D8721804F16F  6    --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpts:8080:8082 /* dnat name: podman id: b48eaac9afeb369ecb86a18977867b28a3b67655d65162d7f4c2ff8f49eaec51 */

Chain NETAVARK-HOSTPORT-MASQ (1 references)
 pkts bytes target     prot opt in     out     source               destination         
 4106  246K MASQUERADE  0    --  *      *       0.0.0.0/0            0.0.0.0/0            /* netavark portfw masq mark */ mark match 0x2000/0x2000

Chain NETAVARK-HOSTPORT-SETMARK (12 references)
 pkts bytes target     prot opt in     out     source               destination         
 4782  287K MARK       0    --  *      *       0.0.0.0/0            0.0.0.0/0            MARK or 0x2000

@edsantiago
Copy link
Member Author

And here's the diff

--- /tmp/iptables-baseline	2024-02-15 10:02:35.873328903 -0700
+++ -	2024-02-15 10:53:49.560626270 -0700
@@ -1,28 +1,121 @@
-Chain INPUT (policy ACCEPT)
-target     prot opt source               destination         
-NETAVARK_INPUT  0    --  0.0.0.0/0            0.0.0.0/0            /* netavark firewall rules */
-
-Chain FORWARD (policy ACCEPT)
-target     prot opt source               destination         
-NETAVARK_FORWARD  0    --  0.0.0.0/0            0.0.0.0/0            /* netavark firewall rules */
-
-Chain OUTPUT (policy ACCEPT)
-target     prot opt source               destination         
-
-Chain NETAVARK_FORWARD (1 references)
-target     prot opt source               destination         
-DROP       0    --  0.0.0.0/0            0.0.0.0/0            ctstate INVALID
-ACCEPT     0    --  0.0.0.0/0            10.89.1.0/24         ctstate RELATED,ESTABLISHED
-ACCEPT     0    --  10.89.1.0/24         0.0.0.0/0           
-
-Chain NETAVARK_INPUT (1 references)
-target     prot opt source               destination         
-ACCEPT     17   --  10.89.1.0/24         0.0.0.0/0            udp dpt:53
-
-Chain NETAVARK_ISOLATION_2 (1 references)
-target     prot opt source               destination         
-
-Chain NETAVARK_ISOLATION_3 (0 references)
-target     prot opt source               destination         
-DROP       0    --  0.0.0.0/0            0.0.0.0/0           
-NETAVARK_ISOLATION_2  0    --  0.0.0.0/0            0.0.0.0/0           
+Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
+ pkts bytes target     prot opt in     out     source               destination         
+  488 28504 NETAVARK-HOSTPORT-DNAT  0    --  *      *       0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type LOCAL
+
+Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
+ pkts bytes target     prot opt in     out     source               destination         
+
+Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
+ pkts bytes target     prot opt in     out     source               destination         
+15740 1115K NETAVARK-HOSTPORT-DNAT  0    --  *      *       0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type LOCAL
+
+Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
+ pkts bytes target     prot opt in     out     source               destination         
+3904K  327M NETAVARK-HOSTPORT-MASQ  0    --  *      *       0.0.0.0/0            0.0.0.0/0           
+   36  3896 NETAVARK-C93D28F2CF205  0    --  *      *       10.89.0.0/24         0.0.0.0/0           
+    7  1022 NETAVARK-194D34E8D615B  0    --  *      *       10.89.1.0/24         0.0.0.0/0           
+    1   146 NETAVARK-1D8721804F16F  0    --  *      *       10.88.0.0/16         0.0.0.0/0           
+
+Chain NETAVARK-194D34E8D615B (1 references)
+ pkts bytes target     prot opt in     out     source               destination         
+    0     0 ACCEPT     0    --  *      *       0.0.0.0/0            10.89.1.0/24        
+    0     0 MASQUERADE  0    --  *      *       0.0.0.0/0           !224.0.0.0/4         
+
+Chain NETAVARK-1D8721804F16F (1 references)
+ pkts bytes target     prot opt in     out     source               destination         
+    0     0 ACCEPT     0    --  *      *       0.0.0.0/0            10.88.0.0/16        
+    0     0 MASQUERADE  0    --  *      *       0.0.0.0/0           !224.0.0.0/4         
+
+Chain NETAVARK-C93D28F2CF205 (1 references)
+ pkts bytes target     prot opt in     out     source               destination         
+    0     0 ACCEPT     0    --  *      *       0.0.0.0/0            10.89.0.0/24        
+    0     0 MASQUERADE  0    --  *      *       0.0.0.0/0           !224.0.0.0/4         
+
+Chain NETAVARK-DN-0212CE89B293B (0 references)
+ pkts bytes target     prot opt in     out     source               destination         
+
+Chain NETAVARK-DN-1866B9FCB39F7 (0 references)
+ pkts bytes target     prot opt in     out     source               destination         
+
+Chain NETAVARK-DN-194D34E8D615B (2 references)
+ pkts bytes target     prot opt in     out     source               destination         
+    0     0 NETAVARK-HOSTPORT-SETMARK  6    --  *      *       10.89.1.0/24         0.0.0.0/0            tcp dpt:5412
+    0     0 NETAVARK-HOSTPORT-SETMARK  6    --  *      *       127.0.0.1            0.0.0.0/0            tcp dpt:5412
+    0     0 DNAT       6    --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:5412 to:10.89.1.3:80
+    0     0 NETAVARK-HOSTPORT-SETMARK  6    --  *      *       10.89.1.0/24         127.0.0.1            tcp dpts:8080:8082
+    0     0 NETAVARK-HOSTPORT-SETMARK  6    --  *      *       127.0.0.1            127.0.0.1            tcp dpts:8080:8082
+    0     0 DNAT       6    --  *      *       0.0.0.0/0            127.0.0.1            tcp dpts:8080:8082 to:10.89.1.3:8080-8082/8080
+
+Chain NETAVARK-DN-1D8721804F16F (4 references)
+ pkts bytes target     prot opt in     out     source               destination         
+    0     0 NETAVARK-HOSTPORT-SETMARK  6    --  *      *       10.88.0.0/16         127.0.0.1            tcp dpt:42089
+    7   420 NETAVARK-HOSTPORT-SETMARK  6    --  *      *       127.0.0.1            127.0.0.1            tcp dpt:42089
+    7   420 DNAT       6    --  *      *       0.0.0.0/0            127.0.0.1            tcp dpt:42089 to:10.88.10.158:5000
+    0     0 NETAVARK-HOSTPORT-SETMARK  6    --  *      *       10.88.0.0/16         0.0.0.0/0            tcp dpt:5412
+    0     0 NETAVARK-HOSTPORT-SETMARK  6    --  *      *       127.0.0.1            0.0.0.0/0            tcp dpt:5412
+    0     0 DNAT       6    --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:5412 to:10.88.15.67:80
+    0     0 DNAT       6    --  *      *       0.0.0.0/0            127.0.0.1            tcp dpts:8080:8082 to:10.88.15.67:8080-8082/8080
+    0     0 NETAVARK-HOSTPORT-SETMARK  6    --  *      *       10.88.0.0/16         0.0.0.0/0            tcp dpt:5680
+    0     0 NETAVARK-HOSTPORT-SETMARK  6    --  *      *       127.0.0.1            0.0.0.0/0            tcp dpt:5680
+    0     0 DNAT       6    --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:5680 to:10.88.16.133:80
+
+Chain NETAVARK-DN-2B0AC2CFAD473 (0 references)
+ pkts bytes target     prot opt in     out     source               destination         
+
+Chain NETAVARK-DN-34AE53115BF02 (0 references)
+ pkts bytes target     prot opt in     out     source               destination         
+
+Chain NETAVARK-DN-39532C6767264 (0 references)
+ pkts bytes target     prot opt in     out     source               destination         
+
+Chain NETAVARK-DN-71E246834E221 (0 references)
+ pkts bytes target     prot opt in     out     source               destination         
+
+Chain NETAVARK-DN-AB864D580210A (0 references)
+ pkts bytes target     prot opt in     out     source               destination         
+
+Chain NETAVARK-DN-AFD48F84636BE (0 references)
+ pkts bytes target     prot opt in     out     source               destination         
+
+Chain NETAVARK-DN-C8B87F0922FBF (0 references)
+ pkts bytes target     prot opt in     out     source               destination         
+
+Chain NETAVARK-DN-C93D28F2CF205 (2 references)
+ pkts bytes target     prot opt in     out     source               destination         
+    0     0 NETAVARK-HOSTPORT-SETMARK  6    --  *      *       10.89.0.0/24         0.0.0.0/0            tcp dpt:5499
+    0     0 NETAVARK-HOSTPORT-SETMARK  6    --  *      *       127.0.0.1            0.0.0.0/0            tcp dpt:5499
+    0     0 DNAT       6    --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:5499 to:10.89.0.3:80
+    0     0 NETAVARK-HOSTPORT-SETMARK  6    --  *      *       10.89.0.0/24         127.0.0.1            tcp dpts:8080:8082
+    0     0 NETAVARK-HOSTPORT-SETMARK  6    --  *      *       127.0.0.1            127.0.0.1            tcp dpts:8080:8082
+    0     0 DNAT       6    --  *      *       0.0.0.0/0            127.0.0.1            tcp dpts:8080:8082 to:10.89.0.3:8080-8082/8080
+
+Chain NETAVARK-DN-D082943377E18 (0 references)
+ pkts bytes target     prot opt in     out     source               destination         
+
+Chain NETAVARK-DN-D9B9BDE897335 (0 references)
+ pkts bytes target     prot opt in     out     source               destination         
+
+Chain NETAVARK-DN-F4B54CC537909 (0 references)
+ pkts bytes target     prot opt in     out     source               destination         
+
+Chain NETAVARK-DN-F9BB1A4E812BF (0 references)
+ pkts bytes target     prot opt in     out     source               destination         
+
+Chain NETAVARK-HOSTPORT-DNAT (2 references)
+ pkts bytes target     prot opt in     out     source               destination         
+    7   420 NETAVARK-DN-1D8721804F16F  6    --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:42089 /* dnat name: podman id: 02900bfad45dec5687a37d5b35ba304ebf5e9da967c10580b21ad2d6efe2c0e9 */
+    0     0 NETAVARK-DN-C93D28F2CF205  6    --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:5499 /* dnat name: mytestnetworkqDS6zCbv id: 6e5b7d8d2e98d6938fb965ac6a83bd1a8a76d10c6981a74a6bfd421b4730aa2f */
+    0     0 NETAVARK-DN-C93D28F2CF205  6    --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpts:8080:8082 /* dnat name: mytestnetworkqDS6zCbv id: 6e5b7d8d2e98d6938fb965ac6a83bd1a8a76d10c6981a74a6bfd421b4730aa2f */
+    0     0 NETAVARK-DN-194D34E8D615B  6    --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:5412 /* dnat name: mytestnetwork6dwsVOMy id: b48eaac9afeb369ecb86a18977867b28a3b67655d65162d7f4c2ff8f49eaec51 */
+    0     0 NETAVARK-DN-194D34E8D615B  6    --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpts:8080:8082 /* dnat name: mytestnetwork6dwsVOMy id: b48eaac9afeb369ecb86a18977867b28a3b67655d65162d7f4c2ff8f49eaec51 */
+    0     0 NETAVARK-DN-1D8721804F16F  6    --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:5412 /* dnat name: podman id: b48eaac9afeb369ecb86a18977867b28a3b67655d65162d7f4c2ff8f49eaec51 */
+    0     0 NETAVARK-DN-1D8721804F16F  6    --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpts:8080:8082 /* dnat name: podman id: b48eaac9afeb369ecb86a18977867b28a3b67655d65162d7f4c2ff8f49eaec51 */
+    0     0 NETAVARK-DN-1D8721804F16F  6    --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:5680 /* dnat name: podman id: 91b21ef1a4fbbc0e4f87b8937007f192b5f41b015e3daca7c5af88448f6a41e3 */
+
+Chain NETAVARK-HOSTPORT-MASQ (1 references)
+ pkts bytes target     prot opt in     out     source               destination         
+ 4106  246K MASQUERADE  0    --  *      *       0.0.0.0/0            0.0.0.0/0            /* netavark portfw masq mark */ mark match 0x2000/0x2000
+
+Chain NETAVARK-HOSTPORT-SETMARK (14 references)
+ pkts bytes target     prot opt in     out     source               destination         
+ 4947  297K MARK       0    --  *      *       0.0.0.0/0            0.0.0.0/0            MARK or 0x2000

@Luap99
Copy link
Member

Luap99 commented Feb 15, 2024

Well sure you have leaked iptables in there so this is no sane state at all and reproducing with this does not help much. Let me guess it only fails when the random port is 5412? Because the old iptables here will redirect to a wrong target.

The issue is that some previous cleanup failed and left now no longer accurate iptables rule behind. So the real catch here is to figure out who leaks it. To figure that out it would make sense to log the iptables -nvL -nat in teardown in CI and see where it starts to leak to narrow it down to hopefully a specific test.

@edsantiago
Copy link
Member Author

No, I haven't kept track of the ports, but my command-line history shows 5435 also.

All this testing is on my laptop. Is there a clean safe way for me to reset iptables state to something clean? I don't feel like rebooting right now but I could.

@Luap99
Copy link
Member

Luap99 commented Feb 15, 2024

https://serverfault.com/questions/200635/best-way-to-clear-all-iptables-rules

@edsantiago
Copy link
Member Author

Reproduced after iptables -F -t nat.

# podman ps
CONTAINER ID  IMAGE                              COMMAND               CREATED        STATUS        PORTS                 NAMES
de43e47ea1b6  quay.io/libpod/testimage:20240123  /bin/busybox-extr...  4 minutes ago  Up 4 minutes  0.0.0.0:5748->80/tcp  myyaml

# date;telnet 127.0.0.1 5748;date
Thu Feb 15 11:58:49 AM MST 2024
Trying 127.0.0.1...
telnet: connect to address 127.0.0.1: No route to host
Thu Feb 15 11:58:52 AM MST 2024   <<<<<<<<< takes 3 seconds
--- /tmp/iptables-nat-baseline	2024-02-15 11:49:38.498691603 -0700
+++ -	2024-02-15 11:57:26.209477247 -0700
@@ -1,18 +1,27 @@
 Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
  pkts bytes target     prot opt in     out     source               destination         
+    0     0 NETAVARK-HOSTPORT-DNAT  0    --  *      *       0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type LOCAL
 
 Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
  pkts bytes target     prot opt in     out     source               destination         
 
 Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
  pkts bytes target     prot opt in     out     source               destination         
+  204 12240 NETAVARK-HOSTPORT-DNAT  0    --  *      *       0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type LOCAL
 
 Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
  pkts bytes target     prot opt in     out     source               destination         
+ 1066 67810 NETAVARK-HOSTPORT-MASQ  0    --  *      *       0.0.0.0/0            0.0.0.0/0           
+    2   292 NETAVARK-1D8721804F16F  0    --  *      *       10.88.0.0/16         0.0.0.0/0           
 
 Chain NETAVARK-194D34E8D615B (0 references)
  pkts bytes target     prot opt in     out     source               destination         
 
+Chain NETAVARK-1D8721804F16F (1 references)
+ pkts bytes target     prot opt in     out     source               destination         
+    0     0 ACCEPT     0    --  *      *       0.0.0.0/0            10.88.0.0/16        
+    0     0 MASQUERADE  0    --  *      *       0.0.0.0/0           !224.0.0.0/4         
+
 Chain NETAVARK-C93D28F2CF205 (0 references)
  pkts bytes target     prot opt in     out     source               destination         
 
@@ -25,8 +34,11 @@
 Chain NETAVARK-DN-194D34E8D615B (0 references)
  pkts bytes target     prot opt in     out     source               destination         
 
-Chain NETAVARK-DN-1D8721804F16F (0 references)
+Chain NETAVARK-DN-1D8721804F16F (1 references)
  pkts bytes target     prot opt in     out     source               destination         
+    0     0 NETAVARK-HOSTPORT-SETMARK  6    --  *      *       10.88.0.0/16         0.0.0.0/0            tcp dpt:5748
+    0     0 NETAVARK-HOSTPORT-SETMARK  6    --  *      *       127.0.0.1            0.0.0.0/0            tcp dpt:5748
+    0     0 DNAT       6    --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:5748 to:10.88.16.202:80
 
 Chain NETAVARK-DN-2B0AC2CFAD473 (0 references)
  pkts bytes target     prot opt in     out     source               destination         
@@ -64,11 +76,14 @@
 Chain NETAVARK-DN-F9BB1A4E812BF (0 references)
  pkts bytes target     prot opt in     out     source               destination         
 
-Chain NETAVARK-HOSTPORT-DNAT (0 references)
+Chain NETAVARK-HOSTPORT-DNAT (2 references)
  pkts bytes target     prot opt in     out     source               destination         
+    0     0 NETAVARK-DN-1D8721804F16F  6    --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:5748 /* dnat name: podman id: de43e47ea1b6dcc881c519329abc130b756d226944f0b93da4f3b5db7309c0be */
 
-Chain NETAVARK-HOSTPORT-MASQ (0 references)
+Chain NETAVARK-HOSTPORT-MASQ (1 references)
  pkts bytes target     prot opt in     out     source               destination         
+    0     0 MASQUERADE  0    --  *      *       0.0.0.0/0            0.0.0.0/0            /* netavark portfw masq mark */ mark match 0x2000/0x2000
 
-Chain NETAVARK-HOSTPORT-SETMARK (0 references)
+Chain NETAVARK-HOSTPORT-SETMARK (2 references)
  pkts bytes target     prot opt in     out     source               destination         
+  204 12240 MARK       0    --  *      *       0.0.0.0/0            0.0.0.0/0            MARK or 0x2000

@stale-locking-app stale-locking-app bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label May 28, 2024
@stale-locking-app stale-locking-app bot locked as resolved and limited conversation to collaborators May 28, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
flakes Flakes from Continuous Integration locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

systests: kube play URL: workaround for ECONNREFUSED edsantiago/libpod
3 participants
@edsantiago @rhatdan @Luap99