Allow special characters on URI template expressions

[jmdacruz]

Defining something like this:

## Resources [/resources{?$filter}]

throws the following error:

URI template expression "?$filter" contains invalid characters.
Allowed characters for expressions are A-Z a-z 0-9 _ and
percent encoded characters. 

But, the use of such parameters is actually quite common in the industry, for instance see: http://docs.oasis-open.org/odata/odata/v4.0/odata-v4.0-part2-url-conventions.html

Moreover, the use of '$' is allowed by the URI specification (see: http://tools.ietf.org/html/rfc3986#section-2). This means that the '$' must not be URL encoded. A similar issue was also addressed by swagger earlier this year: swagger-api/swagger-editor#308

[zdne]

Ah this is unfortunate. According to URI Template RFC 6570 – $ cannot be used as variable name see http://tools.ietf.org/html/rfc6570#section-2.3 while you are correct it can be in URL.

Furthermore it is in list of reserved in section 1.5.

However reading on:

The expression syntax specifically excludes use of the dollar ("$") and parentheses ["(" and ")"] characters so that they remain available for use outside the scope of this specification.

I guess we can exclude $ safely ... what can possibly go wrong ... 😜

Thoughts?

[jmdacruz]

I think you might have found a small mismatch between RFC6570 and the OData spec, because my interpretation of section 2.2 is that the reason $, (, and ) are excluded is so that you could, for instance, apply a macro pre-processor (e.g., replace $macro with value) before you process it as a URI template. This seems to imply that following RFC6570 should lead to URIs without $, but since section 2.2 explicitly leaves those characters outside of scope that means that theoretically a URI template parser should ignore $ and let it through, potentially allowing for a macro pre-processor to be applied after the fact. The lack of clarity on whether this can be applied before or after is what causes the confusion in my opinion.

As you mention correctly RFC6570 section 2.3 excludes $ from the variable specification, and that's in clear conflict with the OData specification. That said, I don't think excluding $ from the URI template parser's verification would cause any conflict since it's ultimately Apiary's interpretation of the symbol (which RFC6570 pretty much leaves open to interpretation in section 2.2, as explained above)

I'll try reaching out to the editors of RFC6570 so that they can shed some light on this.

[jmdacruz]

I reached out to one of the editors of RFC6570, and the problem is that the dollar sign is defined as reserved for future use (e.g., handling $() expressions as macros). This means that it should not be ignored (i.e., letting it just passthrough) and what you are doing right now (throwing an exception) is correct in the context of this spec.

That said, dollar signs are still valid characters in URIs (RFC3986), which means that there should be a way for you to define an "escaped" versions of it on the URI template, and having the template engine interpret them and transform them back into $. RFC6570 only discusses literals (see https://tools.ietf.org/html/rfc6570#section-2.1) outside of the context of an expression ({filter} is an expression), which means that there is really no rule on literals inside expressions. Moreover, section 2.3 it clearly states that if you use pct-encoded triplets on variable names inside expressions these should not be decoded, since they are assumed to be an integral part of the variable name (e.g., %24var is not the same variable as $var).

My final thought on this: RFC6570 is flawed unless there is a way for me to define variable names that have $ as a prefix on them (in which the $ is an integral part of the variable name, not just a separator of some sort). I can't seem to find a way to do this within the confines of the spec, but I'm pretty sure I'm missing something and you guys will solve it :-)

Unless if you find a way to do this within the spec, my take is you should treat $ as part of the variable name (assuming that if you ever had a macro pre-processor running before the template parser then that step would replace macros, if any).

[zdne]

Hey @jmdacruz thanks for the comprehensive research!

RFC6570 is flawed

We have hit many boundaries there so many times.

I'm pretty sure I'm missing something and you guys will solve it

Thank for the trust! Now, we cannot fail you! :)

Unless if you find a way to do this within the spec, my take is you should treat $ as part of the variable name (assuming that if you ever had a macro pre-processor running before the template parser then that step would replace macros, if any).

This is possible to do. I am more thinking about following getting rid of URI templates altogether (unless explicitly used).

So you could be writing just:

# Resource [/r]
- parameters
    - $filter (string, query)

This could solve the issue for query parameters. Not sure how to get around path segments.

Maybe just to use Express JS / Sinatra syntax for path variables:

# Resource [/r/:id]
- parameters
    - $filter (string, query)
    - id (number, path) 

But I am sure we would run into a new whole world of problems with :pathparam syntax.

Thoughts?

[jmdacruz]

I think your proposal should work. Since the order to query string parameters doesn't really matter, I think it's perfectly fine to have them listed on the "parameters" section.

On the path parameters, you obviously need something to indicate where the parameter would be located in the path, and the :pathparam syntax seems nice and well understood.

Note that the difference between query and path parameters is that the former indicates a variable name that will be visible in the final URL (e.g., /r/123?$filter=exampleFilter), whereas the later doesn't (id or :id will never be part of the final URL, but you'll replace that by, e.g., 123). This means that you'll never run into a situation where the path parameter syntax will give you a hard time, since the path variable name is an identifier that only lives within the API Blueprint spec (you can enforce restrictions on :pathparam variable names within API Blueprint without necessarily imposing restrictions on the values themselves)

That said, I also think that the URI template syntax is nice and makes query string parameters more visible in the URL while writing the blueprint (the end result, the documentation and sample code, will be identical in any case)

[BreiteSeite]

I have a similar problem when using an URI template with multiple composite parameters:

Something like this

## Resources [/resources{?foo*,bar*}]

causes a URI template expression "?foo*,bar*" contains invalid characters. Allowed characters for expressions are A-Z a-z 0-9 _ and percent encoded characters (warning code 12) warning, which is wrong in my opinion... or am i missing something?

[zdne]

@BreiteSeite you are correct – ?foo*,bar* should be legal according to RFC6570. I have created a tracking issue in the parser apiaryio/snowcrash#357

Note, in Apiary, they the explode modifier is not supported (see https://docs.apiary.io/api_101/uri-templates/#unsupported)