httpx specifices :80 and :443 in the Host header, causing false negatives

[laluka]

httpx version:

Current Version: v1.1.4

Current Behavior:

httpx sends http/https probes with the port specified within the http Host header even for ports :80 and :443 which causes false negatives when server / reverse proxy targeted validated a strict vhost is used, without port information.

Expected Behavior:

Implement one of:

• Probe with AND without :PORT for the most common http ports such as 80, 443, 8080, 8440
• Add an option to have all specified ports scanned both with vhost:port and vhost within the Host http header

Steps To Reproduce:

# curl -k https://127.0.0.1/
# ncat -lnvp 443 --ssl

Ncat: Version 7.80 ( https://nmap.org/ncat )
Ncat: Generating a temporary 2048-bit RSA key. Use --ssl-key and --ssl-cert to use a permanent one.
Ncat: SHA-1 fingerprint: 44DD 4CEF 4E03 1541 B6FB 79EC 61C8 99CD AFE7 C098
Ncat: Listening on :::443
Ncat: Listening on 0.0.0.0:443
Ncat: Connection from 127.0.0.1.
Ncat: Connection from 127.0.0.1:48334.
GET / HTTP/1.1
Host: 127.0.0.1
User-Agent: curl/7.68.0
Accept: */*

# echo 127.0.0.1 | httpx -ports 443 -no-fallback-scheme (same behavior without -no-fallback-scheme)
# ncat -lnvp 443 --ssl
Ncat: Version 7.80 ( https://nmap.org/ncat )
Ncat: Generating a temporary 2048-bit RSA key. Use --ssl-key and --ssl-cert to use a permanent one.
Ncat: SHA-1 fingerprint: 925C B498 3CF6 4D33 EF6C FDB4 3820 92FF BADB 001B
Ncat: Listening on :::443
Ncat: Listening on 0.0.0.0:443
Ncat: Connection from 127.0.0.1.
Ncat: Connection from 127.0.0.1:48336.
GET / HTTP/1.1
Host: 127.0.0.1:443
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36
Accept-Charset: utf-8
Accept-Encoding: gzip
Connection: close

Anything else:

I've had something like 10% false negative over 20k+ hosts, so I think this is worth patching 🌹
Thanks so much for all the awesome tools and automation mindset! ;)

[yabeow]

Hi @laluka,

I'm interested in this issue and I think we should replicate the behaviour of curl, something like this:

ncat -lnvp 443 --ssl
curl -k https://localhost:443/aaaa

GET /aaaa HTTP/1.1
Host: localhost
User-Agent: curl/7.77.0
Accept: */*

ncat -lnvp 8080 --ssl
curl -k https://localhost:8080/aaaa

GET /aaaa HTTP/1.1
Host: localhost:8080
User-Agent: curl/7.77.0
Accept: */*

You can see that curl don't specify the port part in the Host header for only 80 and 443.

[yabeow]

Also, could you provide some webserver examples that cause false negatives for httpx, @laluka?

[laluka]

Sadly no as they were work-related, but this behavior is highly dependent on the server configuration, I had issues with nginx, apache, and more..

I do agree with @yabeow that the curl behavior should be reproduced, only add an extra case for ports 80 and 443 would cover most of the cases :)

[yabeow]

Thanks for your reply, @laluka. I'm looking at the code now, I will submit a PR if I've found a way to fix this.

[yabeow]

I've found a quick hack for this issue - modify https://github.com/projectdiscovery/httpx/blob/master/runner/runner.go#L683 like this:

h := ""
if protocol == httpx.HTTP && port == 80 {
	h = fmt.Sprintf("http://%s", target)
} else if protocol == httpx.HTTPS && port == 443 {
	h = fmt.Sprintf("https://%s", target)
} else {
	h, _ = urlutil.ChangePort(target, fmt.Sprint(port))
}

I'm not familiar with the behaviour of urlutil.ChangePort (which resides in https://github.com/projectdiscovery/urlutil/) so I left this message here for anyone who does. Please take a look at it if you can 👍

[laluka]

@yabeow well, if it works, merge & run ! 🔥

[yabeow]

Actually, I'm not a maintainer :D

@ehsandeep @Ice3man543 @Mzack9999 Can you guys take a look at this?

[ehsandeep]

@laluka @yabeow this is now supported in dev branch with #471, as default or when using ports flag, now port information is trimmed from host header for port http/80 and https/443, this default behavior can be also disabled with optional ldp flag when using ports.