Support deep objects for query parameters with deepObject style

[bajtos]

Background

Many applications expect deeply nested objects in input parameters, see the discussion in swagger-ui starting from this comment: swagger-api/swagger-ui#4064 (comment) In LoopBack, we are running into this problem too, see loopbackio/loopback-next#1679.

Consider a filter parameter defined as follows:

parameters:
 filterParam:
   in: query
   name: filter
   schema:
     type: object
   style: deepObject
   explode: true
   description: Options for filtering the results
   required: false

Let's say the user wants to provide the following value, for example by entering a JSON into the text area rendered by swagger-ui:

{
  "name": "ivan",
  "birth-date": {
    "gte": "1970-01-01"
  }
}

At the moment, the OpenAPI Specification v3 does not describe how to encode such value in a query string. As a result, OpenAPI clients like swagger-ui don't know how to handle this input - see the discussion in swagger-api/swagger-js#1385 for an example.

Proposal

The following query string should be created by swagger-js client for the input value shown above.

filter[name]=ivan&filter[birth-date][qte]=1970-01-01

The proposed serialization style is supported by https://www.npmjs.com/package/qs, which is used by http://expressjs.com as the default query parser, which means that a vast number of Node.js API servers are already expecting this serialization style.

I am not sure if there is any existing formal specification of this format, I am happy to look into that once my proposal gets considered as acceptable in principle.

Additional information

Existing issues that are touching a similar topic:

• style: form + object looses the parameter name? style: form + object looses the parameter name? #1006
• Are indexes in the query parameter array representable? Are indexes in the query parameter array representable? #1501

Two older comments from swagger-js that may be relevant:

swagger-api/swagger-js#1140

Limitations:
deepObject does not handle nested objects. The specification and swagger.io documentation does not provide an example for serializing deep objects. Flat objects will be serialized into the deepObject style just fine.

swagger-api/swagger-js#1140 (comment)

As for deepObject and nested objects - that was explicitly left out of the spec, and it's ok to just Not Support It™.

[earth2marsh]

Are there any other examples of these nested deepObjects outside of Express? The more widespread a pattern, the more likely it is to be considered. Myself, I have an aversion to passing such a complicated object in the query string. Any insight into why Express even landed on this pattern?

[darrelmiller]

We were a bit stuck when allowing the deepObject style. We understood that users wanted this capability but there is no standard definition of what that serialization format looks like. We had a few choices, allow it and define our own standard and hope implementations followed it. Don't allow it because there are no standards, or add it in, say nothing about its format and hope that a default serialization format emerges.

If we can get some confidence that the qs package is becoming a defacto standard and we can create an accurate description of the serialization, then I have no issue recommending that we include that description in a future minor release of the spec.

[rmunix]

What about the URL max length limits? I think one of the reasons why a standard for object serialization in the URL is hard to materialize is because of the URL max length problem, the URL just wasn't intended to pass data in this way. Depending on the browser and/or server software being used the URL max length varies but in general it is small when compared to how much data can be transmitted with other methods like POST or PUT. It certainly would work for small objects but people tend to inadvertently abuse these kinds of APIs by passing bigger than allowed payloads.

[handrews]

@rmunix regarding URL max length problems, I was very happy to see the HTTP SEARCH method draft RFC revived last month: https://tools.ietf.org/html/draft-snell-search-method-01

[louisl]

Are there any other examples of these nested deepObjects outside of Express? The more widespread a pattern, the more likely it is to be considered. Myself, I have an aversion to passing such a complicated object in the query string. Any insight into why Express even landed on this pattern?

http://esbenp.github.io/2016/04/15/modern-rest-api-laravel-part-2/

I use a modified version of https://github.com/esbenp/bruno referenced in the article above in a few apis, it's extremely useful for including related data and search filtering. I'm not really sure how to define those in a spec. I do appreciate that the url strings could get silly long to the point of failing if abused, but without this sort of thing searches and such would have to be actioned as POST requests or limited to basic GET request params. It seems to me no ones really come up with a holy grail API solution for complex search queries so it's a bit of a free for all at the moment.

[bajtos]

Ruby on Rails

Seems to use the same approach.

Reference: https://edgeapi.rubyonrails.org/classes/Hash.html#method-i-to_query
Source code: https://github.com/rails/rails/blob/b5302d5a820b078b6488104dd695a679e5a49623/activesupport/lib/active_support/core_ext/object/to_query.rb#L61-L86

Example code:

require "activesupport"

data = {
  "name" => "David",
  "nationality" => "Danish",
  "address" => {
    "street" => "12 High Street",
    "city" => "London",
  },
  "location" => [10, 20],
}
print data.to_query("person")

Produces the following query string, I have urldecoded and reformatted it for better readability:

person[address][city]=London&
person[address][street]=12+High+Street&
person[location][]=10&
person[location][]=20&
person[name]=David&
person[nationality]=Danish

Notice that array items are using an empty index, i.e. person[location][]=10, instead of person[location][0]=10.

[bajtos]

Python 2.7

AFAICT, Python does not support nested objects in query parameters.

Example code:

from urllib import urlencode
from urlparse import parse_qs

data = {
  'person': {
    'name': 'David',
    'nationality': 'Danish',
    'address': {
      'street': '12 High Street',
      'city': 'London',
    },
    'location': [10, 20],
  }
}

print urlencode(data)

Produces the following query string, I have urldecoded and reformatted it for better readability:

person={
  'nationality':+'Danish',+
  'location':+[10,+20],+
  'name':+'David',+
  'address':+{
    'city':+'London',+
    'street':+'12+High+Street'
  }
}

Interestingly enough, the roundtrip does not preserve the original data.

print parse_qs(urlencode(data))

Outcome:

{'person': ["{'nationality': 'Danish', 'location': [10, 20], 'name': 'David', 'address': {'city': 'London', 'street': '12 High Street'}}"]}

Another example:

print parse_qs('foo[bar]=1&foo[baz]=2')
# {'foo[baz]': ['2'], 'foo[bar]': ['1']}

[louisl]

Not API specific, but jQuery can generate nested array url params from objects.

http://api.jquery.com/jquery.param/

[Stratus3D]

JSONAPI sparse fieldsets require this: https://jsonapi.org/format/#fetching-sparse-fieldsets

[benhaynes]

Hey @earth2marsh and @darrelmiller ... We (Directus team) have been trying to use OpenAPI 3.0 for a while now, but the lack of support for nested deepObjects has kept us from using this spec. We have a dynamic API that allows for relatively complex filtering, for example: filter[<field-name>][<operator>]=<value>

GET /items/users?filter[category][eq]=vip

Our API Reference for this filtering

Is there any hope for this being supported in the future or should we "move on"?

[ewrayjohnson]

@bajtos: On October 11, 2018 you wrote "I'll try to find some time to fix swagger-js in the next few weeks." What is your status on this?

[bajtos]

Eh, I didn't even started 😢 Feel free to contribute the fix yourself.

[darrelmiller]

@benhaynes Sorry if this comes across as snarky, it's not intended to, it's just I'm in a rush and I don't know how to ask this in a friendly/sincere way.

What would you like us to do? Pick a winner from the many options? Design our own format? If we pick a format that is incompatible with what you currently use, would you switch? Should we find a way of supporting many different formats?

[benhaynes]

Hey @darrelmiller — not snarky at all, I sincerely appreciate the response as it maintains some momentum in the discussion!

We're certainly not trying to force the spec to follow our format, and understand your position of not wanting to blaze ahead without a standard to follow. To answer your question honestly, if the option you choose is incompatible with our method, then we wouldn't be able to use OpenAPI since we can't introduce a breaking change into our platform's filtering. Still, we'd support your team's decision if they think a different direction is a better solution.

I'm not sure how "extensible" your spec/codebase is, but support for multiple (even optional) serialization formats seems the most promising. Perhaps leaving these as unofficial until a "winner" eventually surfaces. In our experience, industry-leading frameworks offering solutions is the most efficient way for a de facto standard to emerge.

Our proposal is to support a deeply nested structure, such as: param[nest1][nest2]=value, where there can be 1,2,3,n levels of nesting. The comments here might be biased, but it seems that most are either already using this pattern or are recommending it.

Thanks again. I'd love to hear your (or anyone else's) thoughts on this approach!

[wellingguzman]

Hey @darrelmiller, I would like to understand the OpenAPI position on this. Is the reason to not support the outcome /items/users?filter[category][eq]=vip because it's not a standard or because the parameter definition format is not part of the standard?

Also, in the technical side, will this bring a complex/breaking change and it requires much more time?

In term of where else this format is supported I would like to add another example.

PHP

Example:

<?php

$data = [
  'person' => [
    'name' => 'David',
    'nationality' => 'Danish',
    'address' => [
      'street' => '12 High Street',
      'city' => 'London',
    ],
    'location' => [10,20],
  ]
];

echo http_build_query($data);

Result:

person[name]=David&
person[nationality]=Danish
&person[address][street]=12+High+Street
&person[address][city]=London
&person[location][0]=10
&person[location][1]=20

The result is urldecoded.

Also PHP automatically parse these parameters into an array. Passing the result above into query string will result in the original array.

[ardalis]

I think I'm running into this same issue. I have a simple C# type that needs two integers. I can define an API endpoint like this just fine:
public IActionResult Foo(int x, int y) {}

and it works but if I use my binding model type with those same two integers as properties:
public IActionResult Foo(FooModel model) {}

then my /swagger endpoint wants to generate JSON+Patch stuff and has no idea how to generate a URL with the appropriate querystring values to bind to that model. Will complex / deep object support help me in this (very simple) case? If not is there a known good way to support it currently?

[apimon]

Arrrgh, hit that wall, too.... great example of a not so obvious limitation that might kill your whole project dev setup and workflow.

[okirmis]

Ran into the same problem (using Rails), also with a parameter allowing for dynamic filtering.

The funny thing is, that swagger-editor does generate deeper nested objects as placeholder text for the parameters text area when clicking "try it out":

      - in: query
        name: filter
        schema:
          type: object
          properties:
            boolean_param:
              type: boolean
              nullable: true
            some_enum_of_types:
              type: array
              items:
                type: string
              nullable: true
          default: {}
        style: deepObject

results in

{
  "boolean_param": true,
  "some_enum_of_types": [
    "string"
  ]
}

which will be silently ignored. I know that this is actually a bug in swagger-editor, but it shows that it would be more consistent to allow deeper nested objects.

[rijkvanzanten]

What needs to be done to move on with this @earth2marsh? You asked for a couple examples outside of Express, which I think have been provided in the messages above. I can help out writing some of the needed documents for this in a PR if that helps.

[handrews]

I think the simplest way to allow folks to experiment with different approaches would be to implement #1502 (Support for arbitrary query strings). That would let folks define entirely custom serialization, perhaps based on media types as @darrelmiller suggests, without having to work around the spec's mechanisms for supporting more common serialization forms.

[POMXARK]

https://github.com/abbasudo/laravel-purity

This is extremely necessary! I can't convert to format query parameter

filters[is_publish][$contains]=true

Terrible. An outrageous flaw.
And no one has undertaken to fix this with at least a third-party package

[dafeder]

@POMXARK let's keep it respectful, the maintainers don't owe you anything. You're welcome to submit a PR or make and share a third-party package if this is so important to you, we're all just doing our best here.

FWIW I'm not a JS developer but I would still strongly vote for standardizing around "qs style" as it's very intuitive and close enough to other serialization methods to work with in different languages. It should be recognized though that there is really no way to do nested query objects in a GET query string that will be straightforward to document.

[handrews]

@dafeder thanks for your comment!

It should be recognized though that there is really no way to do nested query objects in a GET query string that will be straightforward to document.

Yeah, this is why I'm pushing for #1502 in version 3.2 of the spec. It's not ideal, but it gets the OpenAPI spec details out of the way and lets folks do their own serialization for the entire query string, including by just setting the whole thing to content: {application/x-www-form-urlencoded: {...}} and serializing the way we do application/x-www-form-urlencoded bodies. Which would also allow creating 3rd-party extensions to handle alternate query string formats, which is not possible right now because there is no place in any Object where you could easily put such an extension.

[dafeder]

@handrews yes that makes a lot of sense, the problem is really more on the SwaggerUI side. As a spec I can see this being the right way to do it (and clarifying this will probably help Swagger handle it better).

[DustinCai]

Any updates on this?

[handrews]

@DustinCai see my comment above about 3.2. We're getting 3.0.4 and 3.1.1 out the door right now, then 3.2 should be a relatively quick release as we want to keep it small (and then do a 3.3 if 3.2 is well-received). I'd hoped to get 3.2 done by the end of the year, but 3.0.4 and 3.1.1 have taken a bit longer so maybe January or (hopefully) at worst February.

I know that solution (#1502) might not be what you're looking for, but there are too many possible different formats and too many competing demands. Allowing people to define their own query string formats (media types) for the whole query string will unblock experimentation, particularly if combined with a media type registry per #3771.

[zachdaniel]

EDIT: no need to read, I was making an implementation mistake.

Is there actually a situation where someone wants the query parameter specified as a deepObject to be keys with url encoded values that are themselves JSON? I feel like I'm misunderstanding the issue, because I've never heard of that as being something that anyone actually wants. foo[bar][baz]=buz has been the standard way of encoding "nested" query parameters for pretty much my entire career. At a minimum, it would serve every modern web setup better if that was the one option chosen as opposed to foo=%7B%22bar%22:%20%22buz%22%7D which is just wildly strange to me.

The other flaw with that is that it's ambiguous. For instance, our filters support name="{Fred" and name[eq]="Fred". But I can't tell if the user is actually typing json into the search box, to look for a thing with the name {"eq": "Fred"}, or if they are asking for the thing with the name equal to "Fred". So I can't even add server side logic to properly handle this.

If the default was foo[bar]=baz, I could detect which format was which, translating that to foo={"bar": "baz"} in some preprocessor.

I think I must be understanding because I can't really conceive of how thousands of other people haven't hit on this same issue up to now. Is there a solution I'm not aware of?

EDIT: nevermind. I was missing style being set to :form in OpenApiSpex the tool I'm using to generate the open api spec.

[handrews]

@zachdaniel it's mostly that deepObject is intentionally under-specified, which was intended to allow experimentation and possibly future expansion. But there isn't really an obvious way to have tools try out different things here.

In any event, please take a look at the just-released-yesterday 3.0.4 (release notes) and 3.1.1 (release notes) releases, as they both contain a substantial rewrite and dramatic expansion (with multiple new appendices added to the spec) of the parameter and form serialization rules.

[zachdaniel]

Will check it out. I also may have been wrong about the form style. I think I still haven't figured out how to get the format I'm looking for yet.

[zachdaniel]

I still don't see something hinting at what, to me, would be the most common option. What is the way to tell it that a query parameter with nested schemas should be encoded as foo[bar][baz]=buz. I cannot find it. Or are you saying there is just no standard way to do this and individual tools (swagger UI) in this case will just do whatever they want?

[handrews]

@zachdaniel That's because deepObject does not specify what to do with, um... deeper objects 🤦 The name is kind-of misleading as it's actually only one level deeper than normal.

In 3.0.4 and 3.1.1 we added the following clarification to deepObject:

The representation of array or object properties is not defined.

Unsatisfying, I know, but I dug through the history and there was no sign of any intention to specify how to handle these things. While certain frameworks have "standards" they don't all agree (you can look through other issues in this repo to find conflicting requests). In 3.2, I'm hoping to fix #1502 which would allow people to define their own alternative query string formats and not try to contort style/explode/etc. into doing someting they weren't really designed to do.

[zachdaniel]

I get it from the spec perspective, but ultimately as someone maintaining a framework offering automatic swagger UI generation, this one singular sticking point issue is causing users to decide not to use swagger UI at all. It has (for them) a sort of all-or-nothing effect because they want to expose this to API consumers. I may be able to figure something out along the lines of detecting a request coming through swagger UI's "try it out" feature (maybe some way to put in a header to all of those requests for example), which would allow me to then do special query parameter decoding.

I think I'm primarily taking umbrage with SwaggerUI at this point, not the spec. I can't imagine a world where anyone wants a query parameter to be URI encoded JSON. I get it that it's unspecified, but thats a weird choice. Not much left to do here on this issue, and looking forward to a resolution to #1502 as potentially opening a path here.

EDIT: nevermind, that won't work either because I still can't tell the difference between the value being a string of valid JSON and a query parameter encoded by swagger UI 😢

[handrews]

I think I'm primarily taking umbrage with SwaggerUI at this point, not the spec.

Yeah we can't help you with that, I'm afraid!

I can't imagine a world where anyone wants a query parameter to be URI encoded JSON.

Oh, I've seen it. I can't say I'd recommend it, but I've definitely seen it. I've also seen JSON documents embedded in JSON strings. People do some astonishing things...

EDIT: nevermind, that won't work either because I still can't tell the difference between the value being a string of valid JSON and a query parameter encoded by swagger UI 😢

I'm not quite sure where the JSON complaint comes in, though. Is SwaggerUI treating deepObject as an instruction to encode like JSON? It is definitely not supposed to do that.

[zachdaniel]

As far as I can tell, it's when I have a deepObject where a value is an object. It ends up encoded as filter[field]="{"some": "json"}"

[handrews]

@zachdaniel huh. Well, the spec does not define behavior for that case, so on the one hand, we can't say that's wrong. But also if they've chosen to do that, that's thier own decision and they don't need anything from us if they want to make a different decision. As far as the spec is concerned, the behavior of deepObject with properties that are themselves objects or arrays is undefined, so anything that a tool does is entirely under their own control (and unlikely to be portable).

[handrews]

@zachdaniel although if this is an application/x-www-form-urlencoded request body, they might be applying the Encoding Object's logic for serializing JSON objects as embedded JSON? That's the only thing I can think of that might be going on (and, despite having extensively researched and mostly re-written that entire part of the spec over the last few months, i can't recall exactly how those requirements would interact).