Bug: massiv latency impact when using dnsname (workaround in comment 3) #55

dschier-wtd · 2021-02-05T10:59:05Z

Hi,

thanks for the very cool work and effort you are putting into podman. I have identified some very weird behavior, when using podman in combination with dnsname plugin.

It seems like there is a huge performance impact (x ~150 slower response), when using podman dnsname, instead of IPs or dns servers.

Step by Step

create a test environment (this case rootful)

$ sudo podman network create test01
/etc/cni/net.d/test01.conflist

$ sudo podman network inspect test01 | grep dns

                "domainName": "dns.podman",
                "type": "dnsname"

$ sudo podman container run -dt -P --name web01 --network test01 httpd

$ sudo podman container ls

CONTAINER ID  IMAGE                           COMMAND           CREATED         STATUS             PORTS                  NAMES
3261a7db67f6  docker.io/library/httpd:latest  httpd-foreground  13 seconds ago  Up 12 seconds ago  0.0.0.0:39323->80/tcp  web01

Testing container -> host -> container communication via IP

$ sudo podman container run --rm --network test01 fedora:33 bash -c "time curl 192.168.178.106:39323"

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    45  100    45    0     0  45000      0 --:--:-- --:--:-- --:--:-- 45000
<html><body><h1>It works!</h1></body></html>

real	0m0.004s
user	0m0.001s
sys	0m0.002s

Testing container -> container communication via IP

$ sudo podman inspect web01 | grep IPAddress
            "IPAddress": "",
                    "IPAddress": "10.89.0.11",

$ sudo podman container run --rm --network test01 fedora:33 bash -c "time curl 10.89.0.11"
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    45  100    45    0     0  45000      0 --:--:-- --:--:-- --:--:-- 45000
<html><body><h1>It works!</h1></body></html>

real	0m0.004s
user	0m0.000s
sys	0m0.004s

testing container -> host -> container communication via dns

$ sudo podman container run fedora:33 bash -c "time curl nb01:39323"
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    45  100    45    0     0  22500    <html><body><h1>It works!</h1></body></html>
  0 --:--:-- --:--:-- --:--:-- 22500

real	0m0.006s
user	0m0.001s
sys	0m0.004s

testing container -> container communication via dnsname

$ sudo podman container run --rm --network test01 fedora:33 bash -c "time curl web01"
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    45  100    45    0     0     54      0 --:--:-- --:--:-- --:--:--    54
<html><body><h1>It works!</h1></body></html>

real	0m0.835s
user	0m0.003s
sys	0m0.003s

As you can see, steps 1 - 4 seem ok, but step 5 shows a real time x150+ slower than the other examples. This is reproducible with all kinds of traffic, as soon as dnsname name resolving is involved. Adding this up in a construct like Nextcloud, you will see huge impacts.

user -> traefik -> nextcloud-web -> nextcloud-php  -> nextcloud-db

Additonal information

$ podman --version
podman version 2.2.1

$ rpm -qa | grep podman
podman-2.2.1-1.fc33.x86_64
podman-docker-2.2.1-1.fc33.noarch
podman-plugins-2.2.1-1.fc33.x86_64

$ rpm -qa | grep dnsmasq
dnsmasq-2.83-1.fc33.x86_64

It would be awesome to get some insights here. Maybe I am doing it wrong? Are there additional parameters needed?

Please also feel free to reach out to me for any additonal information.

The text was updated successfully, but these errors were encountered:

dschier-wtd · 2021-02-05T15:06:51Z

Update for a redo in the same container:

$ sudo podman run -it --name client01 --network test01 fedora:33 bash
[root@534fd4a9ce91 /]# time curl web01
<html><body><h1>It works!</h1></body></html>

real	0m0.948s
user	0m0.003s
sys	0m0.011s
[root@534fd4a9ce91 /]# time curl web01
<html><body><h1>It works!</h1></body></html>

real	0m0.954s
user	0m0.007s
sys	0m0.004s
[root@534fd4a9ce91 /]# time curl web01
<html><body><h1>It works!</h1></body></html>

real	0m0.813s
user	0m0.006s
sys	0m0.005s

dschier-wtd · 2021-02-06T08:59:33Z

Additional update / workaround:

Using the internal fqdn (excluding the search domain) solves the issue:

[root@95b42f3e3572 /]# time curl web01
<html><body><h1>It works!</h1></body></html>

real	0m0.974s
user	0m0.006s
sys	0m0.006s

[root@95b42f3e3572 /]# time curl web01.dns.podman
<html><body><h1>It works!</h1></body></html>

real	0m0.008s
user	0m0.003s
sys	0m0.005s

For me, this is good enough, but maybe worth an inspection how dnsname/dnsmasq are resolving search domains/priorities these. Maybe internet resolving is tried first and times out or so. Not sure.

dschier-wtd · 2021-02-06T09:13:42Z

@baude Dunno, if this may impact the docker-compose functionality of podman3.0, but it may be worth a look. it is very common to define multiple networks in docker-compose and communicate via hostnames.

Luap99 · 2021-02-23T17:23:24Z

@daniel-wtd can you try with --dns-search dns.podman for the podman run command

dschier-wtd · 2021-02-24T08:27:50Z

Hi,

I started both of the containers with --dns-search dns.podman. Please find the results below. Looking good.

sudo podman container run --rm --network test01 --dns-search dns.podman fedora:33 bash -c "time curl web01"

real    0m0.005s
user    0m0.002s
sys     0m0.003s

sudo podman container run --rm --network test01 --dns-search dns.podman fedora:33 bash -c "time curl example.com"

real    0m0.227s
user    0m0.002s
sys     0m0.004s

Luap99 · 2021-02-24T09:00:44Z

OK I think we should add this automatically when dnsname is used. In order to do so dnsname has to add the dns search domain to the cni result and podman has to read the search domain and add it to resolv.conf.

dschier-wtd · 2021-02-24T09:13:21Z

Sounds like a plan. There may be the situation like:

networks

proxy-net
dns=proxy.podman
app-net
dns=app.podman
db-net
dns=db.podman

containers

proxy01
networks=proxy-net
app01
networks=proxy-net, app-net
db01
networks=app-net, db-net

And I am not sure, if there are limitations in the resolvers. (count of dns search entries, dns server entries)

Luap99 · 2021-02-24T09:26:41Z

Note that dnsname currently only works for one attached network, see containers/podman#8399, containers/podman#9492 and #12

Luap99 · 2021-02-24T09:53:36Z

#57 and containers/podman#9501 should fix this

rhatdan · 2021-03-08T19:11:14Z

@Luap99 can we close this issue now?

Luap99 · 2021-03-08T19:58:50Z

Yes

dschier-wtd · 2021-03-10T22:35:06Z

Thanks a lot everybody :)

akash0x53 · 2022-10-09T13:54:35Z

Why this issue occurs? Any ways to reproduce this issue?

dschier-wtd changed the title ~~Bug: massiv latency impact when using dnsname~~ Bug: massiv latency impact when using dnsname (workaround in comment 3) Feb 6, 2021

Luap99 mentioned this issue Feb 24, 2021

Add dns search domain to cni result #57

Merged

Luap99 mentioned this issue Feb 24, 2021

Add dns search domains from cni response to resolv.conf containers/podman#9501

Merged

rhatdan closed this as completed Mar 9, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bug: massiv latency impact when using dnsname (workaround in comment 3) #55

Bug: massiv latency impact when using dnsname (workaround in comment 3) #55

dschier-wtd commented Feb 5, 2021 •

edited

Loading

dschier-wtd commented Feb 5, 2021

dschier-wtd commented Feb 6, 2021 •

edited

Loading

dschier-wtd commented Feb 6, 2021

Luap99 commented Feb 23, 2021

dschier-wtd commented Feb 24, 2021

Luap99 commented Feb 24, 2021

dschier-wtd commented Feb 24, 2021 •

edited

Loading

Luap99 commented Feb 24, 2021

Luap99 commented Feb 24, 2021 •

edited

Loading

rhatdan commented Mar 8, 2021

Luap99 commented Mar 8, 2021

dschier-wtd commented Mar 10, 2021

akash0x53 commented Oct 9, 2022

Bug: massiv latency impact when using dnsname (workaround in comment 3) #55

Bug: massiv latency impact when using dnsname (workaround in comment 3) #55

Comments

dschier-wtd commented Feb 5, 2021 • edited Loading

Step by Step

Additonal information

dschier-wtd commented Feb 5, 2021

dschier-wtd commented Feb 6, 2021 • edited Loading

dschier-wtd commented Feb 6, 2021

Luap99 commented Feb 23, 2021

dschier-wtd commented Feb 24, 2021

Luap99 commented Feb 24, 2021

dschier-wtd commented Feb 24, 2021 • edited Loading

networks

containers

Luap99 commented Feb 24, 2021

Luap99 commented Feb 24, 2021 • edited Loading

rhatdan commented Mar 8, 2021

Luap99 commented Mar 8, 2021

dschier-wtd commented Mar 10, 2021

akash0x53 commented Oct 9, 2022

dschier-wtd commented Feb 5, 2021 •

edited

Loading

dschier-wtd commented Feb 6, 2021 •

edited

Loading

dschier-wtd commented Feb 24, 2021 •

edited

Loading

Luap99 commented Feb 24, 2021 •

edited

Loading