url validator, exotic _wildcard_ issue

[jeroengui]

I use the validators library in a URL scanning project. But this URL that work in the browser doesn't get recognized as a URL (replaced the tt by xx because it's a malicious sample):

hxxps://wildcard.jetsett.in/wwwmicrfofst/pohwernetsharaven8726skkksusukjs9916192629900177771017000172610720027111113/

I suppose the _wildcard_ part of the domain has something to do with it...

[yozachar]

$ python -c "from validators import __version__; print(__version__)"
0.22.0

$ python -c "from validators import url; print(url('hxxps://wildcard.jetsett.in/wwwmicrfofst/pohwernetsharaven8726skkksusukjs9916192629900177771017000172610720027111113'))" 
ValidationError(func=url, args={'value': 'hxxps://wildcard.jetsett.in/wwwmicrfofst/pohwernetsharaven8726skkksusukjs9916192629900177771017000172610720027111113'})

$ python -c "from validators import url; print(url('https://wildcard.jetsett.in/wwwmicrfofst/pohwernetsharaven8726skkksusukjs9916192629900177771017000172610720027111113'))"
True

I do not understand, it works as expected. hxxp / hxxps is a convention, not a valid protocol.

[jeroengui]

I get that hxxp/hxxps is just a convention. I only wanted to raise an issue about the http version.

The Markdown syntax in this issue removed the underscores from the url with _wildcard_, sorry for that.

import validators
print(validators.url('https://_wildcard_.jetsett.in/wwwmicrfofst/pohwernetsharaven8726skkksusukjs9916192629900177771017000172610720027111113/'))

When I run this I get this output:

ValidationError(func=url, args={'value': 'https://_wildcard_.jetsett.in/wwwmicrfofst/pohwernetsharaven8726skkksusukjs9916192629900177771017000172610720027111113/'})

[yozachar]

Hey @jeroengui the domain _wildcard_.jetsett.in seems to be invalid.

Because the following is valid according to rfc_2782 (TLDR: underscores are prepended not appended).

$ python -c "from validators import url; print(url('https://_wildcard._jetsett.in/wwwmicrfofst/pohwernetsharaven8726skkksusukjs9916192629900177771017000172610720027111113', rfc_2782=True))"
True

Are you sure it's not _wildcard._jetsett.in instead?

[jeroengui]

I'm not sure about the standard and how more I read how more confused I get, but the thing is:

Most browsers are accepting urls with subdomains like _wildcard_.jetsett.in and malicious actors are using them in phishing campaigns etc.

At the moment it poses a real threat, because a lot of (antivirus) companies use validators for urls just like this one.
The urls never get in their blocklists, exposing their users to more risk because these malicious urls work fine in their browsers.

I don't know if this is something that should be fixed in the browsers, in the standard or in the validators, and the issue is definitely bigger than just this implementation.

So to be clear:

We are figuring out if http://_wildcard_.domain.com is a valid url.

We agree that http://_wildcard.jetsett.in and https://my_sarisari_store.typepad.com/ are valid urls.

So an underscore can be used in any place of the subdomain except the one right before the dot?

Stackoverflow has had some discussion on the topic also: https://stackoverflow.com/questions/2180465/can-domain-name-subdomains-have-an-underscore-in-it

EDIT: I added an example DNS record on my own domain that seems to work fine in most browsers : here (https://_test_.jeroengui.be/)

[jeroengui]

Some more information: It seems like SSL certificates cannot be generated for _wildcard_.domain.com but a wildcard certificate like *.domain.com works just fine for _wildcard_.domain.com

https://cabforum.org/2018/11/12/ballot-sc12-sunset-of-underscores-in-dnsnames/

[yozachar]

• rfc_1034 does not mention anything about underscore.
• rfc_2782 talks about prepending underscores.

So appending underscore to sub/domain name is not encouraged, but it is technically valid.

This patch

- + r"(?:[a-zA-Z0-9-_]{0,61}[A-Za-z0-9])?\.)"
+ + rf"(?:[a-zA-Z0-9-_]{{0,61}}[A-Za-z0-9{'_'if rfc_2782 else ''}])?\.)"

in

validators/src/validators/domain.py

Line 45 in 854375f

+ r"(?:[a-zA-Z0-9-_]{0,61}[A-Za-z0-9])?\.)"

will work.

You'll have to add ("example_.com", False, True) to pytest parameter list of test_returns_true_on_valid_domain() in test_domain.py.

PR is welcome.