Change host parser for non-special schemes

[annevk]

new URL("unknown://†/") does not result in "Punycode" (results in %E2%80%A0) and new URL("notspecial://H%4fSt/path") does not remove the URL encoding (results in H%4fSt not host). It seems all implementations are in agreement here so I'm not sure it's a change worth pursuing. Barring any objections I plan on changing the standard.

Acknowledge: Alex Christensen.

[achristensen07]

In these cases, what we humans considered the host was actually the path.

<script>
try {
    var url = new URL("notspecial://H%4fSt/path");
    alert('host: "' + url.host + '", pathname: "' + url.pathname + '", toString: "' + url.toString() + '"');
    // Firefox and Chrome have an empty host and the pathname is "//H%4fSt/path"
} catch (e) { alert("threw"); }
<script>

When trying the same with "notspecial://" Safari agrees with Chrome and Firefox. I think these are similar cases.

[achristensen07]

Given that different browsers preserve the case of notspecial://CamelCase for different reasons, I think we should keep the spec as it is. I think I'm willing to change WebKit to punycode encode nonspecial hosts (changing uppercase ASCII to lowercase) to continue to have a spec that has hosts in nonspecial schemes. Alternatives would be to percent encode nonspecial schemes, which I think is strange and shouldn't be done, or considering asdf://host to have a path of "//host", which shouldn't be done.

[annevk]

Ah right, non-Safari browsers basically do not parse non-special schemes at all. The standard aligned with Safari in b266a43 as it better matches the intent of URLs and the original RFCs. Closing this then. Thanks!

[achristensen07]

I've seen some bad compatibility issues with changing behavior with such URLs. I think we should change the spec after all.

I propose that we keep the case of hosts of URLs with non-special schemes and not add a '/' when serializing them. I also propose we continue to punycode-encode hosts of URLs with non-special schemes if they contain a non-ASCII character after being percent-decoded, and not add a '/' when serializing them. Having the same host be percent-encoded with a non-special scheme and punycode-encoded with a special scheme is strange.

All browsers would still have to change, but the serialization of URLs like "asdf://HoSt" would be compatible with existing browsers, which agree for differing reasons.

[annevk]

That seem very specific somewhat strange rules. In particular not normalizing ASCII-only in any way, but normalizing if there's any non-ASCII is rather weird. I guess we'd only not add / if the path is empty?

[achristensen07]

I agree that the rules are strange, but they're the most compatible and logical rules I can think of. We need to keep the case sensitivity which matches all browsers, we need to keep the no '/' which matches all browsers (yes, only if the path is empty), and we need to do something with non-ASCII code points. I haven't seen any such URLs that have non-ASCII code points, so I'm not sure what the compatibility problems might be if there are any. Safari fails to parse such URLs, while Chrome and Firefox percent-encode them because they are considered to be in the path.

An alternative would be to match the behavior of Chrome and Firefox with all non-special URLs. That would be replacing this:
Otherwise, if remaining starts with an "/", set state to path or authority state, and increase pointer by one.
with this:
Otherwise, if url’s scheme is a special scheme and remaining starts with a "/", set state to path state. Otherwise, if url’s scheme is not a special scheme and remaining starts with a "/", set state to path or authority state, and increase pointer by one.
Which would match behavior of Chrome, Firefox, and Safari for some reason (which I think should change) with urls like a:// but with URLs like a://b/c Safari treats b as the host and /c as the path, while Chrome and Firefox treats //b/c as the path. Maybe I'm biased in thinking Safari has more reasonable behavior here. I don't like that Chrome and Firefox don't seem to allow having a host in non-special URLs. If we aligned the spec with this behavior it would lead to strange behavior, like developers asking why are the host and path of a://b/c so different from http://b/c. I'm curious if there is a reason Chrome and Firefox have such behavior. The compatibility issues I've run into only uses the entire serialized URL, and I wonder if Chrome or Firefox have any tests that verify non-special URLs don't have a host.

[annevk]

The specification used to align with non-Safari. It changed in b266a43 based on https://www.w3.org/Bugs/Public/show_bug.cgi?id=27233. Rationale being that it would be better to deviate less from the original RFCs and Safari showed it was possible.

I think I'm okay with your suggested changes, but it would be good to hear from @valenting @sleevi or others as to what Firefox and Chrome might be willing to do here and on what timeframe. Having one browser lead the charge is great, but not enough for success.

[achristensen07]

I guess either way, the behavior of "a://B" (which needs to be left as-is) will be different than the behavior of "http://B" (which is canonicalized to "http://b/"). I'm curious what other browsers think we should do about the existence of a host and non-ASCII code points.

[achristensen07]

This problem gets worse with IPv4 and IPv6 addresses that are hosts of unrecognized schemes that need canonicalization or fail parsing, like asdf://123456 or asdf://[asdf]

[annevk]

So non-special schemes get no host parsing whatsoever basically? Somewhat sad and I guess that means that if we ever get another DNS-based scheme we'll have to grow the list of special schemes. 😟

[achristensen07]

Basically, except we need to do something with non-ASCII code points. I haven't come across any actual uses of unrecognized schemes with non-ASCII code points in the hosts, but its behavior should be defined and I think percent-encoding is a bad idea.

[annevk]

Only percent-encoding seems reasonable given that we cannot normalize for ASCII.

[achristensen07]

It's strange, but behavior here is going to be strange no matter what, and percent-encoding matches existing behavior of the canonicalized URL string. I'm wondering if Chrome and Firefox are willing to change their behavior of not having a host for non-special schemes, though

[annevk]

@sleevi?

I think I can say that Firefox is willing to change behavior here, though I don't think it'll be necessarily short term.

[sleevi]

@annevk Sorry, somehow I missed these pings in my inbox due to accidentally muting the thread (likely fat fingered something)

I admit, I haven't read the WHATWG spec that closely for the context of the issue - much of our GURL implementation reflected (or tried to) RFC3986. However, where I suspect the incompat has arisen is with respect to standard URLs (e.g. those with authority components in a structured form). Despite https://tools.ietf.org/html/rfc3986#section-1.1.1 remarking that

It thus defines the syntax and semantics needed to implement a scheme-
independent parsing mechanism for URI references, by which the
scheme-dependent handling of a URI can be postponed until the
scheme-dependent semantics are needed.

namely, that all unrecognized schemes can be assumed generic unless/until they're supported, the GURL implementation treats all unrecognized schemes as non-standard (specifically, https://cs.chromium.org/chromium/src/url/url_util.cc?rcl=1478435172&l=105 ). That is, if it's not a scheme GURL explicitly knows how to parse, it treats it as opaque, and everything as the path.

This would likely explain the divergence here, as well as the rationale. This decision has then cascaded into a number of design decisions throughout Chrome with respect to non-standard schemes it exposes (e.g. chrome-extension://, chrome://, chrome-guest://, etc), which all assume they can safely be extended as non-standard schemes (omitting authority, and following any number of internal structural rules)

As such, it's beyond my ken to know what the extent of complexity or negative affect would be - much of it exists in those consuming our low-level URL parser, but I don't have good knowledge about those or their implementation implications. In order for us to get that desired behaviour, it would effectively mean treating unrecognized schemes (asdf://) as generic/standard schemes, so that they get the same behaviours - but because we don't 'force' internal callers to pre-register their schemes (... something we definitely should do, to prevent the unknown-unknowns like this), I don't know who would break, and I don't personally have the time to explore making the change and seeing what breaks and guiding it. I suspect it'd be non-trivial, but I suspect that the above code really is as simple as changing that logic so that non-standard schemes are explicitly registered as such, in line with RFC 3986.

[annevk]

Note that RFC 3986 (and the URL Standard to a lesser extent due to compat) require all schemes to be parsed in the same way. Only post-parsing can you do scheme-specific dispatch. The whole idea of "standard URL" vs "non-standard URL" is something that sounds more like it's copied from Gecko (and perhaps by extension older RFCs) than the latest RFC.

[achristensen07]

I'm ok with percent-encoding non-ascii values in hosts of URLs with non-special schemes. If the scheme is changed to a special scheme, the percent-encoded values will be percent-decoded again before being punycode encoded, so no information is lost. Switching from special to a non-special scheme could be strange, though.

[annevk]

@achristensen07 I'd be in favor of forbidding such drastic scheme changes. Especially if they require re-parsing the host when done. That seems bad.

[annevk]

FWIW, those kind of scheme changes are not allowed by the URL Standard. And haven't been for quite a while.

[annevk]

@achristensen07 when looking at paths I found other problems too, some of which crept into your implementation. E.g., new URL("///x", "foo://y/") should probably result in foo:///x, not foo://x. It seems special and non-special are not clearly separated there as they should have been.

[achristensen07]

Ignoring extra beginning slashes is new in WebKit. I haven't seen any compatibility problems. I don't have strong feelings about it.

[achristensen07]

How do we feel about the port of such URLs? I'm seeing some compatibility issues with URLs like "asdf://host:123456789123456789/" that used to be valid
Should we allow larger ports with custom schemes? Should we consider those numbers part of the host? Should I open another issue?

[annevk]

@achristensen07 that should be a new issue. I guess at some point we have to decide how far we go to make this work.